Rise of AI crawlers and bots causing web traffic havoc
Friday, 22 August 2025 ()
A new report from edge cloud platform provider Fastly reveals what it called “a striking shift in the nature of automated web traffic” with a recent analysis of traffic indicating that AI crawlers make up close to 80% of the AI bot traffic observed. Meta generated more than half, eclipsing both Google and OpenAI combined.
These results are derived from traffic analyzed between April 16 and July 15 across two of the company’s offerings, Next Gen WAF and Bot Management, and illustrate, the company said, how AI-driven automation is reshaping online traffic.
For the purpose of the report, its authors categorized AI bots into two types: Crawlers and Fetchers. Crawler bots, they wrote, “ operate similarly to search engine crawlers — they systematically scan websites to collect content for building searchable indexes or training language models. This process is a precondition to the model’s ‘training’ phase.”
Fetcher bots, on the other hand, they said, “access website content in response to user actions. For example, when a user requests up-to-date information on a specific topic, a fetcher bot retrieves the relevant page in real time. They are also used to help surface website links that match a user’s search query, directing them to the most relevant content. Crawler bots contribute nearly 80% of the total AI bot request volume, with fetcher bots making up the remaining 20%.”
*‘Massive surge in AI bot traffic’*
Real-time fetching by AI bots, the report states, is a greater challenge than peak crawler rate. Analysis revealed that in one case, a single crawler reached a peak of around 1,000 requests per minute.
Real-time fetching, on the other hand, is significantly more aggressive: in one instance, a fetcher bot made 39,000 requests per minute to a single website at peak load. “This traffic volume can overwhelm origin servers, strain server resources, consumer bandwidth, and cause expensive DDoS-like effects even without malicious intent,” the report noted.
Other key findings from the
|
||||
|
|
||||
You Might Like |
||||
