Devs say AI crawlers dominate traffic, forcing blocks on entire countries

Viewed 91
With the rise of aggressive AI crawlers impacting website traffic and reliability, developers are taking notice and expressing concerns. Many have experienced significant drops in legitimate visitors, leading some to shut down websites due to server overload. Comments suggest a need for better management of AI scraping practices, including rate limiting and improved collaboration between companies. Some developers propose innovative solutions, such as serving random data to AI bots to offset their negative impact. Meanwhile, the security measures like blocking entire countries are viewed as a lazy response to the problem. There are ongoing debates about the ethics of web crawling, particularly when it affects non-profit sites versus for-profit ones, subtly hinting at larger issues with capitalism and ownership in the tech landscape.
0 Answers