Can AI see your website?

Four things decide whether an AI system can see your website: the permissions in your robots.txt file, the firewall or security service in front of your host, whether your words are in the HTML or only appear after JavaScript runs, and a single setting called noindex that tells every crawler to leave. Each one is a door. A crawler has to get through all four, in that order, before a single sentence of yours can end up in an answer. Most agent websites have at least one of them shut, and almost none of the agents know.

Door one: does robots.txt let AI crawlers in?

robots.txt is a small text file at yoursite.com/robots.txt that lists which crawlers may fetch which pages. It has been around for over 30 years and was written down as a formal standard in RFC 9309 in 2022. AI companies publish the names their crawlers use, so you can allow or block each one: OpenAI documents GPTBot, ChatGPT-User and OAI-SearchBot, Anthropic documents ClaudeBot, Perplexity documents PerplexityBot, and Google uses Google-Extended as the switch for its AI products. The checker tests 8 of these names against your file and gives partial credit per crawler, because a site that welcomes 6 of 8 is in a very different position from one that blocks them all.

The common surprise: a well-meaning SEO plugin or a hosting provider added a block for “AI bots” in 2023, when that seemed prudent, and nobody has looked at the file since.

Door two: is a firewall turning AI away anyway?

robots.txt is a request, not a lock. The standard itself is blunt about this:

These rules are not a form of access authorization.

The lock is your firewall. Cloudflare sits in front of millions of websites and, since July 2025, blocks AI crawlers by default for new customers; other security services and managed WordPress hosts ship similar rules. When that rule is on, your robots.txt can say welcome all day and GPTBot still gets a 403 or a “checking your browser” screen. Nothing in your dashboard shows it, because you are visiting with a browser. The checker catches it by fetching your page twice, once as a browser and once announcing itself as GPTBot, and comparing what came back. A site that serves the browser a page and the crawler a wall fails the AI-crawler firewall check no matter what robots.txt says.

Door three: are your words actually in the HTML?

Open your site, right-click, choose View Page Source, and look for a sentence from your home page. If you can find it, good. If the source is a few hundred lines of script tags and an empty <div id="root">, your content is assembled in the visitor’s browser by JavaScript, and most AI crawlers never run it. They read the raw HTML the server sends and move on. Single-page-app builders and some brokerage platforms work this way. So do review widgets: a testimonial section that loads through a third-party script is invisible in the source, which is why the checker looks for review markup in the served HTML and reports it missing when only the widget is there. The fix is server-side rendering, which for most agents means a different theme or platform rather than a setting.

Door four: is the noindex switch on?

A single line, <meta name="robots" content="noindex">, or the same instruction sent as an X-Robots-Tag header, tells every crawler to drop the page. On WordPress it is one checkbox on the Reading settings screen, labelled “Discourage search engines from indexing this site,” and it is routinely switched on while a site is being built and forgotten at launch. The MDN reference for the robots meta name lists the values. A site with noindex on can look flawless, pass every design review and sit invisible for 6 months before anyone notices. Unticking the box takes about 10 seconds. It is the highest-weighted single line in the Search Foundations category for exactly that reason.

How do I check all four in one go?

Run your address through the checker. The first category on the report, AI Crawler Access, covers doors one, two and three with the evidence for each: which crawlers robots.txt blocks, what the GPTBot fetch returned compared with the browser fetch, and how many words of text were in the raw HTML. The noindex check sits at the top of the Search Foundations category. Fix whatever failed, re-run, and the re-check line shows the difference about 15 seconds later. Three of the four doors can usually be opened in under 30 minutes; the JavaScript door is the one that can take weeks, because it usually means a new theme or platform.