While we are trying to hang on to the A.I. hype train some topics go unnoticed. I just listened to the latest episode of Chaosradio, the CCC podcast (I hope it's okay to call it like that). They were talking about A.I. crawlers, tiny user agents that crawl the web for content to train or to use for research. Usually I try to avoid phrases like "...back then, when the internet was a better place..." but these arguments come to mind when you learn, how ruthless these agents behave (or better: how they are engineered to behave).
Today in some cases A.I. crawling agents are responsible for 50% of some of the traffic that is hitting smaller websites. Even Cloudflare is trying to create a business model out of the business model of scraping content by introducing a crawl control. The behaviour is ruthless, because these agents change origin and source so quickly that sustained identification is not possible.
Can the bubble burst, please.