Does blocking GPTBot remove my site from ChatGPT answers?
Not by itself. GPTBot is OpenAI’s training crawler — blocking it keeps your content out of future model training runs that honor robots.txt. ChatGPT’s live answers come from OAI-SearchBot (the ChatGPT Search index) and ChatGPT-User (real-time fetches when a user asks about a page). Block those two and you disappear from ChatGPT’s browsing answers; block only GPTBot and you stay citable today.
What is the difference between training, search-index, and user-fetch crawlers?
Training crawlers (GPTBot, ClaudeBot, Google-Extended…) collect content to train future models. Search-index crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build the indexes AI search products answer from. User-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page live because a user asked about it. Each category has a different cost to block: training affects future models, the other two affect your visibility right now.
Why does the checker say a bot is allowed when my robots.txt never mentions it?
robots.txt is opt-out, not opt-in. A crawler that finds no group matching its user-agent — and no * group — is allowed everywhere by default. "Allowed" often just means "you never said no."
Do AI crawlers actually obey robots.txt?
The named bots from OpenAI, Anthropic, Google, Apple, and Perplexity document their tokens and state they honor robots.txt. But it is voluntary compliance, not enforcement — plenty of scrapers ignore it entirely. If you need a hard block, do it at the server or WAF level; robots.txt is a policy signal.
How does this checker decide allowed vs blocked?
It follows the longest-match rules Google documents for robots.txt: the most specific matching user-agent group wins, then the longest matching Allow/Disallow path wins, with * wildcards and $ end-anchors supported and ties going to Allow. It checks the path you enter (default /) — a site can allow its homepage while blocking /blog/, so test the paths you actually care about.