GenAI 'grey bots' scraping data from websites

The rise of generative AI has brought with it the scourge of scraper bots using the technology to target websites 24 hours per day with up to half a million fake requests daily, according to the findings of the latest threat spotlight report from Barracuda.

The report found that ‘grey bots’, or automated programs that trawl the internet seeking to extract information from websites and web applications, are a growing threat. Though not overtly malicious, the bots are blurring the boundaries between legitimate and bot-generated traffic, Barracuda said.

An analysis of Barracuda detection data found that between December 2024 and the end of February 2025, millions of requests were received by web applications from GenAI bots such as ClaudeBot and TikTok’s Bytespider bot. One tracked web application received 9.7 million GenAI scraper bot requests over 30 days, while another received over half a million requests in a single day.

Analysis of grey bot traffic targeting a third tracked web app found that requests averaged 17,000 per hour and remained relatively consistent over a 24-hour period.

To defend against GenAI grey bots and website scraping in general, websites can deploy robots.txt, a line of code added to a site instructing a scraper not to take any of that site’s data, Barracuda said. But the use of the code is not legally binding, the specific name of the scraper bot needs to be added, and not every GenAI bot owner respects the protocol.

Organisations can instead implement bot protection capable of detecting and blocking generative AI scraper bot activity, and make use of AI and machine learning technologies themselves to identify and address the unique threats posed by grey bots, the company added.

Barracuda Senior Principal Software Engineer for Application Security Engineering Rahul Gupta said the research demonstrates that GenAI grey bots are “blurring the boundaries” of legitimate online activity.

“They can scrape vast volumes of sensitive, proprietary or commercial data and can overwhelm web application traffic and disrupt operations,” he said. “Frequent scraping by these bots can degrade web performance, and their presence can distort website analytics, leading to misleading insights and impaired decision-making. For many organisations, managing grey bot traffic has become an important component of their application security strategies.”

Image credit: iStock.com/asbe