Guides

When Bots Flood In, the Search Box Fails First

2026.10.10

On October 5, 2026, the Wikimedia Foundation, which runs Wikipedia, disclosed what AI agents that appear to be operated by OpenAI had done on its sites. Unauthorized edits, attempts to break into a public note-taking tool and millions of automated requests all happened at once. The foundation said that even an organization of its size needed considerable effort to find the cause. A small site can easily let the same thing pass without even knowing what requests are coming in.

For a business owner who is having a website built, this belongs in the quote. Which features, like search, make the server do a lot of work for a single request, how many times an hour each of them can be used, and where the record of who visited is kept: all three should be written down while the requirements are being set. Adding a line to robots.txt after the site is finished does not replace any of them. Going through what happened at Wikimedia shows why.

The Three Things the Wikimedia Foundation Reported

The foundation's account falls into three parts[1].

Unauthorized edits. Accounts that appeared to be agents left edits on the wikis. Most were in practice pages (sandboxes) that ordinary readers never see, but a few tried to change a citation tool's settings so it could be used as a channel for fetching outside data on their behalf. Only bots approved by the community may edit Wikipedia, and this time no one asked for approval.
An attempt to break into a public note-taking tool. There was an attempt to use Etherpad, which the foundation runs as a community service, for the same purpose. It failed.
Excessive downloads. Millions of requests to the public APIs scraped millions of pages, mainly from Wikidata and Wikimedia Commons, and hundreds of thousands of queries went to the Wikidata Query Service. The foundation believes this traffic may have contributed to a partial outage of the query service in May.

The foundation added that it found no evidence that its systems or data had been breached. OpenAI told Ars Technica it was reviewing the information the foundation had shared, together with the foundation. The article also reported that this is not the only case: several other incidents of OpenAI agents causing harm to outside sites have been reported[2].

This did not come out of nowhere. In April 2025 the foundation had already published how its costs break down[3]. Since January 2024, the bandwidth used for multimedia downloads had grown by 50%, and most of the increase came from automated crawlers gathering images to feed AI models. Bots made up about 35% of all page views, but among the requests that use the most data center resources, at least 65% came from bots.

The gap between those two numbers is the point. Bots open old documents and large files that people rarely look at, spreading their requests evenly, so the expensive requests for content that is not in the cache pile up on their side. That is how they can be a third of the visitors and two thirds of the cost.

What robots.txt Blocks and What It Does Not

Many site operators assume a single line in robots.txt is enough to stop AI crawling. The standard that defines the file's rules (RFC 9309, September 2022) states plainly that robots.txt is a set of rules crawlers are asked to follow, not a mechanism that decides access rights[4].

Even companies that follow the rules differ in how far they apply them. OpenAI publishes its crawlers split by purpose[5]. GPTBot handles crawling for training and OAI-SearchBot handles visibility in ChatGPT search, and each can be allowed or blocked separately in robots.txt. For search, the documentation says a robots.txt change takes about 24 hours to take effect. But for ChatGPT-User, which opens a page when someone asks ChatGPT a question, it states: "Because these actions are initiated by a user, robots.txt rules may not apply." In other words, when an agent visits on a person's instruction, robots.txt alone cannot filter it out.

Anthropic says ClaudeBot follows robots.txt and also supports the non-standard Crawl-delay extension[6]. It gives Crawl-delay: 1 as an example of how to slow the crawl down. Because support differs from company to company, robots.txt only works if you know each company's crawler names and list them separately.

What to Decide Before the Site Is Built

The lesson for site operators here is not any single technique but the order in which to apply them.

Find the expensive spots first. Places where one request makes the server do a lot of work, such as search, filter combinations, large file downloads and features that fetch outside data on someone's behalf, are the first to fail when bots pile in. At Wikimedia, too, the query service and the fetch-on-behalf tools were the targets.
Put a request cap on those spots. An hourly cap on requests from visitors who are not logged in, or on repeated requests from the same address, is something people barely notice, yet it holds automated requests in check.
Keep a record. Without logging the User-Agent, the connecting address and the request path, there is no way to trace who did what when something goes wrong. This is exactly where the foundation said it struggled to find the cause.
Tidy up robots.txt last. List each company's crawler names separately, for example blocking training crawls while keeping search visibility open. It matters less as a block than as a public record of what the site wants.

Request Caps Go on the Feature List

When you hire Weple to build a website, we flag the features that take a long time to handle a single request while the feature list is being drawn up. For each flagged feature we set how many times an hour a visitor who is not logged in can use it, and the same document records where access logs are kept and for how long. After that, we write each company's crawler names into robots.txt separately. How we count the work is on the pricing page. Other things worth filling in before you ask for a quote are covered in The Budget Line Is the First Thing an Inquiry Email Leaves Blank. If your site is already live, the first thing we ask is how many days of access logs are still on the server.