robots.txt for AI Crawlers: What to Allow and Block

Most sites that vanish from ChatGPT answers did it to themselves. Someone read a headline about AI scraping, pasted a block of Disallow: / rules into robots.txt, and shut out the bots that fetch pages when a customer asks a question.

The fix takes ten minutes, but you have to know that each AI company runs more than one bot, and they do different jobs.

Table of contents

Training bots and answer bots are different

Every major AI company now splits its crawling into purposes. I checked each vendor’s own documentation before writing this.

Two things worth knowing about that table. Google-Extended has no user agent string of its own, so it never shows up in your server logs. And ChatGPT-User is the odd one out: because a human asked for the page, OpenAI says it may not follow robots.txt at all.

The decision: three setups

1. Open door. Allow everything. This is what I run on this site, with every bot above set to Allow: /. I want to be cited, and I’d rather be quoted than protected.

2. Opt out of training, stay in the answers. Disallow GPTBot, ClaudeBot and Google-Extended. Allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. This suits publishers who sell their archive or worry about reuse, and still want to appear when someone asks a question.

3. Closed door. Disallow all of them. Only do this if AI visibility has no value to you, for example a paywalled research product that sells by sales call. Everyone else is trading away discoverability for a feeling of control.

For an agency client or a local business, option 1 or 2 is almost always right. Setup 3 is rarely a decision anyone has actually thought through.

The file for option 2

Add these lines to your existing robots.txt. Don’t replace the file, because your sitemap line and your Googlebot and Bingbot rules need to stay.

# Training: opted out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search and live answers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

If you’d rather not edit it by hand, the AI robots.txt generator has a toggle per bot and gives you the finished lines to paste in.

Check it actually works

1. Open https://yourdomain.com/robots.txt in a browser. It must load as plain text from the root of the domain, once per subdomain.
2. Run your site through the AI crawler access checker to see which bots your rules let in.
3. Wait. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.
4. Check that your firewall or CDN isn’t blocking the bots anyway. A robots.txt that says Allow doesn’t help if a bot-protection rule returns a 403 to the same crawler. This is the failure I’d look for first when a site “allows” AI bots and still gets no citations.

What not to do

• Don’t block by IP address. Anthropic’s documentation warns that IP blocking may not reliably opt you out, because it can stop the bot from reading your robots.txt in the first place.
• Don’t treat robots.txt as security. It’s a request that named, well-behaved bots honor. Scrapers that ignore it need server-side rules.
• Don’t use robots.txt to hide pages you want out of Google. That’s a noindex job.
• Don’t forget Crawl-delay is non-standard. Anthropic says it supports it for ClaudeBot. Don’t assume every bot does.

One more file worth publishing

Once the doors are open, tell the bots where the good pages are. That’s what llms.txt is for, and my write-up on whether it actually moves citations covers what I’d expect from it. To see whether any of this is paying off, measure your AI search traffic.