Does blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude answers?

Not by itself. OpenAI and Anthropic each document separate user agents for training, for search, and for fetches a person asks for, and robots.txt controls each one separately. To keep pages out of training while leaving them reachable for AI search, block the training agent and allow the search agent.

What is the difference between Google-Extended and Googlebot?

Googlebot is the crawler that fetches pages for Google Search. Google-Extended is a robots.txt token with no crawler of its own: it tells Google whether content it has already crawled may be used for Gemini models. Google states that Google-Extended does not affect inclusion or ranking in Google Search.

What is llms.txt and do I need one?

llms.txt is a proposed convention: a Markdown file at the root of a site that lists the pages you consider most useful, with a short description of each. It is optional. There is no guarantee that any AI system reads it, so treat it as a low-cost experiment rather than a requirement.

Is robots.txt enough to stop AI crawlers?

robots.txt is a request, not a lock. Crawlers that follow the standard will respect it, but nothing forces a crawler to read it. To block traffic outright, use rules at your server, CDN, or firewall. This tool reads your rules; it does not test whether any crawler obeys them.

AI Crawler Access Checker and llms.txt Generator

Enter a URL to see which AI and search crawlers your robots.txt allows or blocks, at the site root and at that page, with the exact rule that matched. We also check llms.txt, llms-full.txt, ai.txt, and noai tags. Then generate robots.txt rules and an llms.txt file to copy.

We fetch the page once as HelloSEOToolsBot, plus robots.txt, llms.txt, llms-full.txt, and ai.txt from the same origin. Public hostnames only, 15 seconds at most. Results are cached for 5 minutes; nothing is stored beyond that.

Generate robots.txt rules

Tick what you want to block for each vendor. Nothing is blocked by default. Blocking search and answer crawlers removes the site from that vendor's AI search answers and the links they show, so most sites leave those allowed.

Vendor	Block training crawlers	Block search and answer crawlers

Googlebot and Bingbot are not offered here: blocking them removes the site from Google Search and Bing. Google-Extended is the token Google offers for Gemini training, and Applebot-Extended is Apple's equivalent.

Add these lines to the robots.txt at your site root. A crawler follows only the most specific group that names it, so a group added here replaces your User-agent: * rules for that crawler. If you have existing rules for one of these user agents, merge them into one group.

Generate llms.txt

Save this as a plain text file named llms.txt at your site root, so it loads at https://yoursite.com/llms.txt. Fields are filled from the page you checked when it has a title and meta description; edit anything before you copy.

What these crawlers do

Each company runs one or more user agents, and the same company often uses different ones for different jobs. The three groups in the results are:

Training: crawlers that collect pages a company may use to train or improve AI models. Blocking them is the usual choice when you do not want your content used for training; it does not by itself take the site out of search results.
Search and answers: crawlers that build the indexes search engines and AI assistants use to find pages and cite sources. Blocking them means your pages can stop appearing, or being linked, in those results.
User-initiated fetch: agents that load a page when a person asks an assistant about it. They act for that person instead of crawling on a schedule, and some vendors state that robots.txt may not apply to them.

These companies publish their user agents and what each one is for:

Google documents Googlebot for Search and Google-Extended, a robots.txt token with no crawler of its own that controls use of crawled content for Gemini.
OpenAI documents GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for fetches a person asks for.
Anthropic documents ClaudeBot for training, Claude-SearchBot for search, and Claude-User for fetches a person asks for. The older anthropic-ai token still appears in many robots.txt files, so we check it too.
Perplexity documents PerplexityBot for search and Perplexity-User for fetches a person asks for.

The other tokens in the table (Bingbot, Applebot-Extended, CCBot, Amazonbot, Bytespider, meta-externalagent, DuckAssistBot, cohere-ai, YouBot) are common in robots.txt files. Their purposes here are short labels; check each operator's own documentation before relying on them.

Limitations

robots.txt is advisory. It asks crawlers to stay out; it cannot force them. We report what your rules say, and we do not verify how any crawler actually behaves.
We evaluate robots.txt the way Google describes: groups with one or more User-agent lines, case-insensitive user agent matching, longest matching rule wins, Allow wins a tie, and * and $ wildcards. Other crawlers may parse edge cases differently.
A missing robots.txt (a 4xx response) means everything is allowed. When robots.txt cannot be read (a 5xx, 429, or timeout) we show the result as unknown, because the standard tells crawlers to assume a full block in that case.
llms.txt is a proposed convention. There is no guarantee that any AI system reads it or uses it. ai.txt and the noai and noimageai values are also non-standard, and support for them varies.
We fetch as HelloSEOToolsBot from Cloudflare's network. Sites that block unknown bots or serve different files to different crawlers can give a different answer than a real crawler gets.

Frequently asked questions

Does blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude answers?: Not by itself. OpenAI and Anthropic each document separate user agents for training, for search, and for fetches a person asks for, and robots.txt controls each one separately. To keep pages out of training while leaving them reachable for AI search, block the training agent and allow the search agent.
What is the difference between Google-Extended and Googlebot?: Googlebot is the crawler that fetches pages for Google Search. Google-Extended is a robots.txt token with no crawler of its own: it tells Google whether content it has already crawled may be used for Gemini models. Google states that Google-Extended does not affect inclusion or ranking in Google Search.
What is llms.txt and do I need one?: llms.txt is a proposed convention: a Markdown file at the root of a site that lists the pages you consider most useful, with a short description of each. It is optional. There is no guarantee that any AI system reads it, so treat it as a low-cost experiment rather than a requirement.
Is robots.txt enough to stop AI crawlers?: robots.txt is a request, not a lock. Crawlers that follow the standard will respect it, but nothing forces a crawler to read it. To block traffic outright, use rules at your server, CDN, or firewall. This tool reads your rules; it does not test whether any crawler obeys them.

API

The same result is available as JSON, free, no key required, cached for 5 minutes (add &fresh=1 to bypass the cache):

GET https://helloseotools.com/api/ai-crawlers?url=https://example.com/

Response: { ok, origin, path, finalUrl, pageTitle, pageDescription, robots: { url, status, httpStatus, lines }, crawlers: [{ name, vendor, purpose, category, allowedRoot, allowedPath, rule, ruleLine, group }], llms: { txt, full }, aiTxt, noai: { noai, noimageai, sources }, checkedAt }. group is specific when a group names the crawler, * when the catch-all group applied, none when no group applied, and unknown when robots.txt could not be read. Errors return { ok: false, error }.

Please keep automated use under a few requests per second. Heavy or abusive use will be rate limited.

Related tools

Full SEO Report: one URL in, a graded report out, including an AI readiness category.
Indexability Checker: robots.txt, noindex, and canonical checks for Googlebot.
Schema Generator: build valid JSON-LD structured data.