Half of website traffic coming from bots and crawlers

About half of all web traffic comes from bots and crawlers, costing companies a lot of money, according to a new report released by DeviceAtlas.

Around 48% of non-human sources accounted for traffic to the sites analysed for DeviceAtlas’s Mobile Web Intelligence Report for Q1 2016.

The company, which offers software for companies to detect the devices being used by visitors to their websites, said this included legitimate search-engine crawlers as well as automated scrapers and bots generated by hackers and spammers.

“Allowing web crawlers to scan your site is a necessity if you want your web pages to appear in Google, Bing or other search results. But at the same time, excessive traffic caused by non-human visitors can be costly in terms of bandwidth, website stability and even potential outages,” the company wrote in a blog post.

DeviceAtlas said the web crawlers that automatically scan online content are deeply ingrained in the online world so that you may be unaware of the amount of web traffic these ‘machines’ generate.

At the same time, Google Analytics doesn’t report on bots and crawlers by default, meaning you may not be able to see the entire share of non-human traffic.

According to DeviceAtlas’s analytics, the most active bot was a search engine — Majestic-12 — which exceeded the amount of traffic from any other bot, including Google and Bing.

In a Computerworld article, Frank Scavo, president of Computer Economics, is quoted to have said that nearly all web analytics tools attempt to filter out non-human traffic but it is difficult to achieve.

“Fraudsters go to great lengths to make their traffic appear to be human-generated,” Scavo reportedly said. “Moreover, ad sellers and marketing agencies may not be particularly interested in seeing their web traffic numbers reduced.

“If you’re advertising on a per-impression or per-click basis, you need to closely scrutinise your analytics.”

In the article, Scavo added that companies “never” underpay and, if possible, it is best to link web marketing expenses to concrete business results like conversions, rather than impressions or clicks.

Image courtesy of medithIT under CC-BY-2.0