Is your website blocking ChatGPT?

If you have ever wondered whether ChatGPT can read your website, the answer is sitting in a plain text file you can open right now. Type your domain into a browser, add /robots.txt on the end, press enter. Whatever appears is the entirety of what you are telling AI crawlers. A 404 means there is no file, which means nothing is blocked. That is a perfectly valid state, not a fault.

Blocking is not the common case. In an audit of 3,241 small business homepages run between 11 July and 16 August 2026, 3.1% of the 1,934 live Croydon borough sites disallowed the crawlers used by AI assistants, and 3.3% did across the rest of the sample. The trouble is what happens when you are in that few per cent: nothing. No warning in a dashboard, no dip you could spot, no email. The site simply stops being readable to the tools a growing number of people now ask for a recommendation.

The crawler names worth knowing

There are dozens of them, but four companies account for almost everything that matters to a UK local business, and each one publishes the exact token to use in robots.txt. These are those tokens, taken from the companies' own documentation.

Tokens and behaviour taken from each company's own crawler documentation, checked 25 September 2026. Three rows need a footnote: OpenAI says robots.txt rules may not apply to user initiated ChatGPT-User fetches, Perplexity says Perplexity-User generally ignores robots.txt, and Google-Extended has no user agent string of its own, existing purely as a robots.txt control.

Two decisions people muddle into one

"Should AI companies train on my writing" and "do I want to be named when someone asks an assistant for a plumber in Croydon" are different questions, with different answers, and the crawler names are what keep them apart. Block the wrong one and you answer a question you never meant to ask.

• Training. GPTBot and ClaudeBot are the ones their owners describe as collecting content that may be used to train models. Disallowing them is a licensing decision about your own words and photographs.
• Citation. OAI-SearchBot, Claude-SearchBot and PerplexityBot exist to find pages and link to them inside answers. Disallowing these is what takes you out of the answer itself.
• Gemini. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Blocking it withholds your content from Gemini training and grounding and costs you nothing in ordinary search.
• The user agents. ChatGPT-User and Perplexity-User fetch a page because a person asked for it in that moment. Both companies say robots.txt may not stop them, which is worth knowing before you assume a line in a text file has settled the matter.

For most local businesses, the sensible position is the obvious one. Let the search and citation crawlers in, because being named in an answer is free advertising to someone who has already described exactly the job they need doing. Then decide about the training crawlers on your own terms, knowing that the two choices are genuinely independent.

The trap: robots.txt groups do not add up

This is the one that catches people, including people who write robots.txt for a living. Rules live in groups, each headed by a User-agent line. A crawler finds the single most specific group that matches its name and follows that group only. Google's specification says it in one sentence: "User agent specific groups and global groups (*) are not combined."

So the moment you write a named group for a crawler, every rule you put under the star group stops applying to it. Not merged, not inherited as a baseline. Ignored entirely. Three worked examples, all for GPTBot, all with the same star group most sites have:

The same logic applies to every named crawler, not only GPTBot. Read the robots.txt specification if you want the precise matching rules.

The practical consequence is worth spelling out. If you have pages that should never turn up inside an AI answer, a checkout, a client preview, a form handler, then those Disallow lines have to be repeated inside every named group in the file. Leave them out of one group and that crawler walks straight past them, and your file looks completely reasonable while it does.

How to check your own site in five minutes

• Open yourdomain.co.uk/robots.txt in a browser. Read all of it, not just the first group. It is usually shorter than a shopping list.
• Look for a Disallow line with nothing after the slash. On its own it closes the entire site to whichever agent heads that group.
• Count the named groups. If any crawler has its own heading, check it carries every restriction you meant to apply to it, because it is not inheriting any of them.
• On WordPress, open Settings then Reading and check that "Discourage search engines from indexing this site" is unticked. Since version 5.3 it writes a noindex, nofollow robots tag into the head of every page, and older versions wrote a blanket Disallow into the generated robots.txt. It is the single commonest way a live site ends up hidden, usually left switched on after a rebuild.
• View the page source and search for the word noindex. A meta robots tag is a second, separate way of being excluded, and robots.txt has no say over it.
• Check whatever sits in front of your site. A firewall or bot protection rule can refuse a crawler by IP or user agent before your robots.txt is ever read, and it will not show up in the file.

This site's own file is at jonesdigital.co.uk/robots.txt if you want to see a deliberate one. Everything is allowed, two paths are closed, and those same two Disallow lines are repeated inside every named crawler group, for exactly the reason above.

Blocking is a legitimate choice, as long as it is a choice

None of this is an argument that you must let everything in. If the words and photographs on your pages are the product, a magazine, a research firm, a photographer with a portfolio, then disallowing the training crawlers is a reasonable commercial position and plenty of publishers have taken it. What is not reasonable is discovering three years later that a checkbox from a rebuild has been quietly keeping you out of every AI answer in your trade.

Once you know the file says what you meant it to say, the next question is whether an assistant has any reason to name you. That is a longer job and mostly not a technical one: how to get your business recommended by ChatGPT and AI search covers the four things that genuinely differ from ordinary SEO.