71% of Top Sites Allowing AI Crawlers Cancel Their Own robots.txt Rules

A robots.txt file is a note you tape to your front door.

Everyone is welcome. Stay out of the bedroom. Don't open the fridge.

There's no lock. Polite guests read the note and behave. The others were never going to.

Then you add one friendly line: Cousin, you're welcome here!

Your cousin takes things literally and reads only the part with their name on it. "Stay out of the bedroom" was addressed to everyone, and your cousin isn't everyone. Your cousin is family.

So the fridge is open now. We can't say whether anyone went in. The note just says it's fine.

GPTBot is the cousin. Now the numbers.

Of the 479 top-10,000 websites that allow a named AI crawler and also keep site-wide robots.txt rules, 340 (71.0%) have exempted that crawler from at least one of those rules. On 214 of them (44.7%), the crawler's own group has no Disallow rule at all, so none of what the site wrote for everyone else applies to it.

We went looking for this because we found it on a store we built. One line added to welcome AI crawlers had switched off the rest of the file for them, and nothing warned us.

So on 6 November 2026 we read the robots.txt of every domain in the Tranco top 10,000. The data and code are public, including a free checker you can run on your own site.

The numbers

• 10,000 domains tested from the Tranco ranking of 5 November 2026

• 5,707 had a robots.txt we could read

• 1,573 (27.6%) of those named at least one of the 17 AI crawlers we tracked

• 479 allowed an AI crawler and also had wildcard Disallow rules

• 340 (71.0%) of those let a crawler escape at least one rule

• 214 (44.7%) were the clearest case, where the crawler's group has no Disallow rule

• 14 rules canceled on the median affected site, and 606 at the highest

Lower-ranked sites did worse:

• Ranks 1 to 1,000: 38 of 64 (59.4%)

• Ranks 1,001 to 5,000: 136 of 194 (70.1%)

• Ranks 5,001 to 10,000: 166 of 221 (75.1%)

We left out 43 sites whose wildcard group blocks everything and allows AI crawlers by exception. Those are deliberate allow-lists.

Why one Allow line cancels the rest

A robots.txt file is not read top to bottom as one list. A crawler picks one group and ignores the others. RFC 9309 says it plainly:

"If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present."

So the * group is a fallback. Once a crawler has a group of its own, the fallback no longer applies to it. Google's documentation is even shorter:

"User agent specific groups and global groups (*) are not combined."

Here is the pattern we kept finding:

User-agent: *
Disallow: /cart/
Disallow: /admin/
Disallow: /search

User-agent: GPTBot
Allow: /

It reads like "and also let GPTBot in." What it means is "GPTBot may ignore everything above." The three Disallow rules still apply to any crawler without its own group. They no longer apply to GPTBot.

Nobody typed that second block to break anything. Nobody ever does. That's the cousin's whole charm.

What it looks like on real sites

We read these files by hand and fetched them again after the scan. Each one matched the scanned copy exactly.

• hp.com gives GPTBot Allow: /. That cancels all 170 of its wildcard rules for GPTBot, including /*LogonForm and /*api/. Somebody wrote those 170 rules. We think about that person sometimes.

• weather.com gives ChatGPT-User Allow: /, which cancels all 65 wildcard rules.

• wordpress.org gives GPTBot Allow: /, which cancels all 4 wildcard rules, including /wp-admin/ and /search.

• forbes.com gives OAI-SearchBot Allow: /, which cancels all 20 wildcard rules.

• visitdubai.com gives GPTBot two rules, Allow: /*.md$ and Allow: /llms.txt. That looks like "you may use the guest bathroom." But nothing says to stay out of the other rooms, so 37 of the 40 wildcard rules stop applying to it.

• nytimes.com shows the partial version. Amazonbot has one rule of its own, Disallow: /wirecutter/, which leaves 71 of 86 wildcard rules out of its group.

We can't tell you which of these were deliberate. A publisher may well want GPTBot on every path. What the files can't show is whether the owner meant /wp-admin/ and /search to be part of that decision.

Which crawlers escape most often

Across the 340 affected sites, these crawlers escaped at least one rule:

• GPTBot: 193 sites

• OAI-SearchBot: 186 sites

• ChatGPT-User: 184 sites

• PerplexityBot: 159 sites

• ClaudeBot: 154 sites

This isn't only an AI issue. Googlebot escapes wildcard rules in the same way on 225 sites in the sample. We didn't study how many of those are intentional. Googlebot has been family for years, and nobody says anything.

On 235 of the 340 sites, at least one canceled rule contains a word such as admin, login, cart, checkout, api or search. That is a word match, not a review of each path, so treat it as a rough signal.

It happened to us too

A Next.js store we built failed this test on the day of the scan. Nine AI crawlers escape 14 wildcard rules there, including /api/ and /_next/. Googlebot and Bingbot escape them too, because they also have their own groups. This blog passed, but only because we had already fixed it in September.

We also ran two small regional samples. Of 30 MENA SEO agencies, 24 had a readable file, 5 allowed an AI crawler with wildcard rules, and 4 of those 5 failed. Of 30 Gulf brands, 21 had a readable file and 2 had this setup, and both failed. Those samples are too small for a percentage, but agencies are the people clients ask to set this up. We're not naming them. Everyone has a cousin.

How to check and fix yours

To check one site, download the repository and run this. It needs Node.js and nothing else:

node scripts/check.mjs yourdomain.com

It lists each wildcard rule that each named AI crawler no longer has to follow. You can also do this by eye: look for any User-agent line naming a crawler, and compare its rules with your * group.

To fix it, choose one of these:

• If the crawler should follow your normal rules, delete its group. It falls back to User-agent: * and it's still allowed everywhere your rules permit.

• If it needs its own rules, repeat every Disallow you still want inside its group. It needs the full list, not only the differences.

Here is the earlier file with the second fix applied:

User-agent: *
Disallow: /cart/
Disallow: /admin/
Disallow: /search

User-agent: GPTBot
Disallow: /cart/
Disallow: /admin/
Disallow: /search

If you'd rather write the file visually, Analytics by Ghaith includes a robots.txt builder with AI crawler presets. It's available to trial users, active web development or SEO clients with free basic accounts, and paid subscribers. It builds files and doesn't run bulk scans.

Don't treat any of this as protection. The note was never a lock, and RFC 9309 is direct: "These rules are not a form of access authorization." OpenAI also notes that for ChatGPT-User, which acts on a person's request, "robots.txt rules may not apply." Anything private needs real authentication.

How far to trust these numbers

• It's a snapshot. Files change, and these results describe 6 November 2026.

• We measured files, not behaviour. Nothing here shows that any crawler visited any path, and a canceled rule is not leaked data.

• Coverage isn't complete. 2,150 domains were unreachable, mostly infrastructure hosts or names with no DNS record. A slow retest of 200 of them read 8 files, so we likely missed about 4% of that group. Another 1,586 returned no file, including 850 that answered 403.

• Each rule was tested with one sample path. Complex wildcard rules can behave differently on other paths.

• We checked ourselves with a second parser. A separate library, robots-parser, confirmed 345 of 347 flagged sites across all samples. The other two have crawler groups with no rules, where parsers disagree. We followed the RFC.

• Our scanner failed twice. One site's file has 88 asterisks in a row, and our scanner stopped to think about that and never came back. Then DNS overload wrongly marked sites as unreachable. We fixed both and rescanned, which took readable files from 4,811 to 5,707. The README has the details.

Get the data

The repository has one row for every domain, the raw file for every flagged site with its hash, the scripts behind each number here, and the frozen Tranco list. The data is free to reuse with credit. If you think we got a site wrong, tell us and we'll check it.

Now go look at your fridge.