Search Intelligence71% of Top Sites Allowing AI Crawlers Cancel Their Own robots.txt Rules
We read robots.txt across the Tranco top 10,000. Of 479 sites that allow an AI crawler and keep site-wide rules, 340 exempted that crawler from at least one rule.








