The Robots.txt War: Why News Publishers Are Blocking AI Crawlers
The digital landscape is changing rapidly, with artificial intelligence (AI) rapidly taking center stage. One of the most notable developments in this evolving environment is the emergence of AI web crawlers, sophisticated programs designed to scour the internet for information to fuel the development of powerful AI models. However, this access to vast amounts of digital content has ignited a fierce battle between publishers and AI companies, with robots.txt, a seemingly obscure file, becoming a key battleground.
Robots.txt is a text file that instructs web crawlers (both human and AI-powered) on which parts of a website they can or cannot access. While seemingly simple, robots.txt is gaining significant importance as the primary tool for news publishers to control the access and potential use of their content by AI.
The Rise of the AI Web Crawler
AI web crawlers are essential for the development of large language models (LLMs), the powerful engines behind popular AI applications like ChatGPT. These models learn from vast amounts of text data gathered from the internet, enabling them to generate human-like text, translate languages, write different kinds of creative content, and more.
However, the insatiable appetite of LLMs for online content has raised concerns among publishers who see the potential for their work to be used without their consent, leading to concerns about copyright infringement and diminished revenue.
A Growing Trend: Publishers Blocking AI Crawlers
Data journalist Ben Welsh has been tracking the evolving battle between publishers and AI crawlers. His research shows that a significant portion of news websites are blocking access to Applebot-Extended, Apple’s dedicated AI crawler. While 53% of the sites block OpenAI’s bot, and nearly 43% block Google-Extended, the blocking of Applebot-Extended suggests a growing awareness and apprehension about AI crawlers.
"A bit of a divide has emerged among news publishers about whether or not they want to block these bots," says Welsh. "I don’t have the answer to why every news organization made its decision. Obviously, we can read about many of them making licensing deals, where they’re being paid in exchange for letting the bots in – maybe that’s a factor."
The Power of Partnership, or the Price of Blocking
The emergence of licensing deals between AI companies and publishers reveals a potential path forward. Several news organizations, including The New York Times and Condé Nast, have partnered with companies like OpenAI and Perplexity, allowing access to their content in exchange for financial compensation.
Jon Gillham, founder of Originality AI, believes that these partnerships are a strategic move. "A lot of the largest publishers in the world are clearly taking a strategic approach," he says. "I think in some cases, there’s a business strategy involved — like, withholding the data until a partnership agreement is in place."
The example of Condé Nast is telling. Initially blocking OpenAI’s crawlers, the company unblocked them after announcing a partnership with OpenAI. Similarly, Buzzfeed spokesperson Juliana Clifton stated that they put all unidentified AI crawlers on their blocklist unless a paid partnership is in place.
The Robots.txt Dilemma: A Manual Battle in an Automated World
While clear, the process of managing robots.txt is far from seamless. Manually editing robots.txt to accommodate the constantly evolving landscape of AI web crawlers can be a daunting task.
"People just don’t know what to block," says Gavin King, founder of Dark Visitors, a company that automates robots.txt updates. King notes that many publishers are relying on his service due to concerns about copyright infringement.
The Rise of the Media Executive as Webmaster
The significance of robots.txt extends beyond the realm of webmasters. It has become a critical element in the decision-making process of media executives. WIRED reported that CEO’s of major media companies are directly involved in deciding which bots to block, highlighting the importance of this seemingly technical tool for the future of digital publishing.
Protecting the Value of Published Work
Many publishers, including Vox Media, have explicitly stated that they block AI crawlers as a means of protecting the value of their work.
"We’re blocking Applebot-Extended across all of Vox Media’s properties, as we have done with many other AI scraping tools when we don’t have a commercial agreement with the other party," says Lauren Starke, Vox Media’s senior vice president of communications. "We believe in protecting the value of our published work."
The Future of the Robots.txt War
The battle over robots.txt is likely to intensify as AI continues to evolve. While AI crawlers offer immense potential for innovation, publishers are rightfully concerned about protecting their content and earning a fair return on their work.
The ideal solution may lie in fostering a balance between access and control. This could involve the development of more transparent and ethical AI web crawlers, along with comprehensive frameworks for data licensing and compensation.
The robots.txt war is a complex battle with no easy answers. But by understanding the forces at play, publishers can navigate this evolving landscape and ensure that their valuable content is used responsibly in the AI age.
