Your site might be invisible to AI search because of one file nobody checks

Not because of content quality. Not because your competitors have better backlinks. Because of a line in a text file sitting quietly at the root of your domain that was written years before ChatGPT existed.

robots.txt tells crawlers what they can and cannot access. Most robots.txt files were written with traditional search bots in mind. The AI crawlers (GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended) arrived later, and a lot of existing robots.txt configurations either block them explicitly or catch them in a wildcard rule that was never meant to include them.

The result: AI engines cannot read your site, cannot cite your content, and will not mention your business in answers, regardless of how good that content is.

Why it happens

A common legacy practice in SEO is to add a blanket rule that blocks all unrecognised bots. It looks like this:

User-agent: *
Disallow: /

This was originally used to block scraper bots, spam crawlers, and anything that was not Google or Bing. It worked fine when the crawler landscape was predictable. It does not work now.

Every major AI search engine deploys its own crawler with its own user-agent string. If your robots.txt blocks all user agents by default, or if it was written with an aggressive deny-all fallback, every one of those AI bots is blocked silently, with no error, no notification, and no way to know unless you go looking.

How to check your robots.txt

Open a browser and go to: yourdomain.com/robots.txt

You are looking for any of the following user-agent strings, either blocked specifically or caught by a wildcard rule:

A blocked configuration looks like one of these:

# Blocks all crawlers including AI bots
User-agent: *
Disallow: /

# Or explicitly blocked
User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

If you see any of these, AI engines cannot access your site.

The fix

To allow AI crawlers, either remove the blocking directives entirely or add explicit allow rules. The cleanest approach for most sites is to allow specific AI crawlers while keeping any other restrictions you have in place:

# Allow AI crawlers explicitly
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

If your robots.txt has a blanket User-agent: * / Disallow: / rule, add the above blocks before the wildcard rule: robots.txt is read top to bottom and the first matching rule wins.

This is a file change. No deployment needed on most platforms, no code change, no plugin. Hand it to your developer with the exact directives above and it is done in under five minutes.

Verify the fix and check for everything else while you are there

Once you have updated robots.txt, confirm the change took effect by reloading yourdomain.com/robots.txt and checking that the blocking directives are gone.

The robots.txt block is one of the most common AI visibility issues, but it is rarely the only one. Sites that block AI crawlers also tend to have missing structured data, incomplete E-E-A-T signals, and no llms.txt file : all signals AI engines use to decide whether to cite a source.

AngryRobot's Site Audit checks all of this across 70+ signals in about 60 seconds. It flags crawler permission issues, structured data gaps, and everything else that affects your GAIO Score, the 0 to 100 measure of how well your site is positioned for AI search. Free to start, no card required.

If you manage client sites, the scan output comes with a client-ready report you can share the same day.