AI startup Anthropic is under fire for alleged breaches of anti-scraping regulations, particularly on websites like Freelancer and iFixit. Claims suggest that Anthropic’s AI tool, ClaudeBot, has been used to extract vast amounts of data against these websites’ wishes, violating the robots.txt rules. This controversy has sparked serious debate about web etiquette and legality.
Freelancer’s CEO, Matt Barrie, and iFixit’s CEO, Kyle Wiens, reported a significant surge in bot hits. Barrie stated that ClaudeBot was responsible for about 3.5 million hits within just four hours on Freelancer, while iFixit saw a million hits in a single day. This bot activity has raised eyebrows in the digital industry, leading to discussions about unauthorized content use and the drain on DevOps resources.
An issue not unique to Anthropic, similar accusations have been leveled against another AI company, Perplexity. The strain on network maintenance and system development caused by such bot activities has caused industry leaders to push aggressively for substantial solutions.
In response to these allegations, Freelancer and iFixit blocked Anthropic’s crawler by ignoring robots.txt signals.
Accusations of Anthropic breaching web regulations
A move intended to protect users’ content and business intelligence, it effectively prevents Anthropic from gathering any proprietary data from their platforms. Despite this proactive measure, the firms accused, including Anthropic and OpenAI, have yet to address these accusations officially.
Both Freelancer and iFixit have criticized Anthropic’s scraping activities as disruptive and detrimental to their business. To combat scraping, iFixit implemented high-traffic alarms, while Freelancer initiated stricter access rules and IP tracking to identify potential scrapers. These measures highlight the industry’s commitment to maintaining the integrity of its digital properties and concern over privacy and data protection.
In a press release, Anthropic denied these accusations, stating that it respects robots.txt and had adjusted its crawl pace following iFixit enabling the file. Furthermore, bots like ClaudeBot are part of an ongoing debate about collecting website content to improve AI products while avoiding copyright infringement. OpenAI, for instance, has made agreements with publishers to prevent such legal complications. Similarly, iFixit’s Wiens has indicated a willingness to negotiate a content licensing deal with Anthropic.
