Robots.txt isn't why most dental practices are invisible to AI

Here's a question most dental practice owners assume they already know the answer to. If an AI assistant can't see your website, are you the one blocking it? We went and checked, at national scale. 11.4% of dental websites block at least one AI or data crawler in their robots.txt file. That leaves the large majority not blocking anything — which is exactly where this video gets interesting.

One thing before we go further. This is a separate study from the 527-practice scan we've published elsewhere on this channel. Different month, different sample, different question. That earlier scan asked whether ChatGPT or Perplexity would actually name a practice. This one asks something narrower and more mechanical: nationally, what does a dental website's robots.txt file say about AI crawlers? Keep the two apart — we'll bring one number back in for comparison later, and we'll say so clearly when we do.

In July 2026, we pulled robots.txt from a national list of dental practice domains — 7,632 of them. 6,497 came back readable. The remainder hit a server error, a timeout, or simply went quiet, and we dropped those instead of guessing at what they meant. A domain we couldn't check isn't a domain that's blocking anyone, and treating the two the same would inflate this finding.

Think of robots.txt as a note taped to a website's front door, listing which visitors are welcome. It lives at the root of the site, and a handful of companies — OpenAI, Anthropic, Perplexity, Google — tell the public exactly which of their automated readers are looking for that note. A crawler that plays by the rules reads the note first. See a no with its name on it, and it walks away without asking for anything else. Small file, real consequences — which is the entire premise behind this video.

Here's the headline number, in full. Run the check across every readable site in that pool of 6,497, and 11.4% of them turn away at least one of the thirteen AI and data crawlers we tested. Thirteen, not two or three — the list includes crawlers that gather training data, crawlers that power a company's own search product, and a few that don't belong to any assistant a patient is likely to be using at all.

Blocking one of those thirteen doesn't mean the same thing every time. A site can block a crawler that only collects training data while leaving every assistant's actual answer-generating crawler wide open — or the other way around. That difference is the entire reason we didn't stop at one number.

So we narrowed it. Drop everything except the crawlers that OpenAI, Anthropic and Perplexity themselves operate — the three companies behind ChatGPT, Claude and Perplexity. Narrowed that way, the block rate drops to 7.6%. Roughly one dental website in thirteen.

Even inside that narrower group, not every crawler does the same job. OpenAI and Anthropic each run a training crawler and a separate one that's closer to answering a live question. Perplexity's crawler is the one that surfaces and links pages in its results, not a training tool at all. A practice can block a training crawler out of caution and never affect whether it gets cited — or block the wrong one and quietly opt out of being cited at all. The 7.6% doesn't tell you which situation any single site is in.

Two numbers, then: 11.4% and 7.6%. They are not interchangeable, and we're going to say the qualifier every time we use them. 11.4% blocks at least one of thirteen tested crawlers. 7.6% blocks specifically a crawler that OpenAI, Anthropic or Perplexity runs. If you see either number repeated without that phrase attached to it, the qualifier got lost — not the finding.

Turn the number around and the more useful fact is the one we didn't put in the title: the large majority of the dental websites we could read aren't blocking any AI crawler at all. Whatever is keeping most dental practices out of AI answers, for most of them, it is not a locked door.

So does an open door mean an AI assistant actually mentions you? We can check that against a different, smaller sample. In a separate scan of 527 dental practices across seven US cities this past June, robots.txt was already open to AI crawlers for nearly every one of them. And in that same 527-practice sample, 96.2% — 507 of 527 practices — still got zero mentions from ChatGPT or Perplexity. Different sample, different month, and the shape repeats anyway: an open door did not translate into a mention.

Say that plainly, because it's easy to slide past. An open robots.txt file does not cause an AI assistant to mention a practice. Being reachable is a precondition, not a guarantee — the practice still has to look like the right answer to whatever question got asked. The 96.2% figure from that other scan is exactly why we're not claiming otherwise.

Run the logic the other direction and the same caution applies. A blocked crawler doesn't explain most of this gap either. Only 11.4% of the sites in this study block anything, and narrowed to the crawlers that actually power an assistant's answers, it's 7.6%. That's not a large enough share to account for how rarely dental practices show up in AI answers overall. Where blocking happens, it's a real problem — it just isn't the main story.

One more limit, and we'd rather flag it than let you assume otherwise. We read the file a site publishes. We didn't test whether a server actually honors that file when a real crawler shows up asking for a page — that's a different question, and we looked at it separately in another study on this channel, because the two don't always agree. This page only tells you what a practice declared, not what happens at the door when someone actually knocks.

If you run a dental practice site, the file is worth five minutes. Open your own domain, add slash robots.txt, and read what's there. Look for a line that blocks everything, or that names a crawler you don't recognize. If nothing there says no to OpenAI, Anthropic or Perplexity's crawlers, this particular door is already open — and now you know that being open is not the same as being found.

To be clear about what this video is not saying. We're not saying blocking is common — for most sites, it isn't. We're not saying an open file gets a practice recommended — on its own, it doesn't. And we're not saying robots.txt is the reason most dental practices are invisible to AI, because nothing in either dataset supports that claim.

So, to recap. We read robots.txt on 6,497 dental websites out of 7,632 requested. 11.4% block at least one of thirteen tested AI and data crawlers. Narrowed to just the crawlers OpenAI, Anthropic and Perplexity run themselves, that's 7.6%. And in a separate, earlier scan of 527 dental practices, 96.2% still got zero mentions from ChatGPT or Perplexity — despite most of them already having the door open.

Every number in this video, the full methodology, and both datasets are open at citevio.com/data. We'd rather be corrected than quoted wrong — if a figure here looks off, the contact is right there on the page. This channel publishes measured, dated research on how AI search engines find, or don't find, dental practices — not guaranteed rankings. Thanks for watching.