Artificial intelligence is sucking up the web at high speed: +49% of robot traffic in nine months

Since the advent of artificial intelligence, many online media publishers fear losing traffic to their websites. Last April, the New York Times reportedly lost 36% of its traffic from Google, even before the arrival of new features announced at the Google I/O event last May.

And while some media outlets have already expressed concern about the massive arrival of artificial intelligence, it may well be that a point of no return has already been reached…

Robot traffic explodes in a few months

As one might imagine, with artificial intelligence still in its infancy, robots tasked with "scraping" the web, that is, navigating web pages to extract information, are growing rapidly.

According to the latest report from TollBit entitled "State of the Bots, AI bot traffic is said to have increased by 49% in just three quarters. With the arrival of advanced search features from ChatGPT, Perplexity, and Claude d’Anthropic, and a growing number of AI users, this observation is understandable, but it is also worrying...

Crawl rules ignored by AI robots?

Just like search engine robots, AI robots are subject to crawl rules that allow or prevent them from browsing certain pages of a given website.

To do this, website publishers use in particular the robots.txt file, which allows them to both guide AI robots and ask them to ignore certain web pages.

Unfortunately, as we can see in the TollBit report, robots seem to be increasingly ignoring the rules dictated by publishers. Indeed, the report shows that 12.9% of robots no longer follow crawl rules, whereas this figure was only 3.3% last quarter.

Beneficial consequences for brands?

Although some website publishers have expressed concern about the development of artificial intelligence, fearing that their traffic will plummet in the coming months, side effects are also worth noting.

According to TollBit, 22% of the companies analyzed have seen an increase in their traffic thanks to artificial intelligence, and 26% are mentioned more by AI than before.

However, while these consequences are beneficial for some brands, others could well sink if they are neglected by the responses of artificial intelligence...