Of 499 dental practice websites whose robots.txt welcomes AI crawlers, 72 (14.4%) were refused by their own server anyway. The same homepage that loads in a browser came back to an AI crawler as a block or a bot challenge. In 63 of those cases, 12.6% of the sample, Googlebot was served normally in the same run. This is not a general bot wall. It is AI-specific.
In our experience, advice about AI crawler access stops at robots.txt. Open the file, add the bots, done. That advice is incomplete, because robots.txt is a note pinned to the door and the server is the door. In July 2026 we tested how often the two disagree on sites that had already opened the file. On one in seven, they disagreed.
This is not 14.4% of US dental practices. The sample was drawn only from sites whose robots.txt already lets AI crawlers reach the homepage. Practices that block in robots.txt were left out on purpose, because the question here is what happens after the declaration says yes. Every figure on this page carries that denominator.
What did we test, and on which sites?
We drew 550 US dental practice domains at random from a frame of 5,019 whose robots.txt does not block AI crawlers at the root, then asked each one for its homepage eight times: once as an ordinary browser, six times as AI-related crawlers, and once as Googlebot for control. After dropping sites where the plain browser request also failed, 499 remained.
The frame came from a separate study we ran the same week. Across 7,632 US dental practice domains, 6,497 of which returned a readable answer, 11.4% block at least one AI crawler in robots.txt and another 11.4% publish no robots.txt at all. The numbers behind that one are in are dental websites blocking AI crawlers. This page is about the group in the middle: the sites whose file says come in.
| Test design | Value |
|---|---|
| Question | Does a site whose robots.txt allows AI crawlers actually serve them? |
| Frame | 5,019 US dental practice domains whose robots.txt leaves the root open to AI crawlers |
| Where the frame came from | Our 22 July 2026 robots.txt study of 7,632 dental domains, 6,497 readable |
| Sampling | Deterministic random draw, seed 20260721, first 550 domains |
| Requests per domain | 8: one browser baseline, six AI-related crawlers, one Googlebot control |
| Valid n | 499 |
| Dropped before scoring | 51, because the plain browser request failed too |
Those 51 sites matter more than they look. If a site turns away an ordinary Chrome visitor, it is not treating AI crawlers differently, it is turning away everyone, and counting it as an AI block would inflate the finding. So we removed them before scoring.
| Why we dropped a domain | Domains |
|---|---|
| Baseline returned HTTP 403 | 27 |
| Baseline request failed (DNS, TLS, timeout or reset) | 14 |
| Baseline was itself a challenge page, so the site challenges every visitor | 7 |
| Baseline returned HTTP 404 | 2 |
| Baseline returned HTTP 429 | 1 |
| Total dropped | 51 |
How many sites refused an AI crawler?
Seventy-two of the 499 sites whose robots.txt welcomes AI crawlers, or 14.4% of that group. Of those 72, sixty-nine (13.8%) returned an outright block status such as 403 or 429, and three (0.6%) returned a bot challenge page. A further 41 sites served every crawler but sent a different page body to at least one of them, which we treat as a signal worth flagging rather than a finding.
| What the server did | Sites | Share of the 499 |
|---|---|---|
| Served all six AI-related crawlers the page | 386 | 77.4% |
| Refused at least one AI crawler | 72 | 14.4% |
| Served every crawler, but returned a different page body to at least one | 41 | 8.2% |
Counted per crawler-and-site pair rather than per site, 79 of the refusals were an HTTP 403 and 38 were an HTTP 429. The bottom row deserves its own caution: a different page body can mean a legitimate variation as easily as it can mean cloaking, so we did not publish it as a rate. It is in the table because leaving it out would make the picture look tidier than the data is.
Is this a general bot wall, or is it aimed at AI?
It is aimed at AI. In 63 of the 72 cases, 12.6% of the 499 sites whose robots.txt welcomes AI crawlers, the Googlebot control request came back clean: Google's crawler was served the same page a browser gets while at least one AI crawler was not. Across all 499 sites, only 6 refused Googlebot at all.
This control is the reason the finding holds up. Without it, anyone could answer that these practices simply block every bot and AI is not being singled out. The measurement says otherwise, and the gap is wide enough that it is hard to read another way.
| Control reading | Sites | Share of the 499 |
|---|---|---|
| Googlebot served normally, at least one AI crawler refused | 63 | 12.6% |
| Googlebot refused anywhere in the sample (a wall against every bot) | 6 | 1.2% |
Broken down by crawler, two names absorb most of the refusals. ClaudeBot was refused by 9.2% of the sample and GPTBot by 8.0%, while ChatGPT-User, the identity OpenAI uses for requests a person triggers inside ChatGPT, was refused by 1.4%. Googlebot sits at the bottom of the same list, which is the comparison that matters.
| Crawler | Refused | Share of the 499 |
|---|---|---|
| ClaudeBot (Anthropic) | 46 | 9.2% |
| GPTBot (OpenAI) | 40 | 8.0% |
| OAI-SearchBot (OpenAI) | 17 | 3.4% |
| PerplexityBot (Perplexity) | 10 | 2.0% |
| ChatGPT-User (OpenAI) | 7 | 1.4% |
| Bingbot (Microsoft) | 6 | 1.2% |
| Googlebot — control | 6 | 1.2% |
Could we have triggered those rate limits ourselves?
No, and the request order rules it out. Every domain received the same eight requests in the same fixed sequence. If our own pace had caused the 429 responses, the effect would build up and the last identities in the sequence would be hit hardest. The opposite happened: 37 of the 38 rate-limit responses landed on the first and fourth crawlers, and the seventh, our Googlebot control, took none of them.
| Order | Crawler | HTTP 429 | HTTP 403 |
|---|---|---|---|
| 1 | GPTBot | 21 | 16 |
| 2 | OAI-SearchBot | 0 | 17 |
| 3 | ChatGPT-User | 0 | 7 |
| 4 | ClaudeBot | 16 | 27 |
| 5 | PerplexityBot | 0 | 10 |
| 6 | Bingbot | 1 | 2 |
| 7 | Googlebot (control) | 0 | 3 |
A cumulative rate limit cannot leave positions two, three and five at zero and then produce 16 rate limits at position four. Something in front of these sites reads the name on the request and decides. That is what an AI-specific rule looks like in a log file.
Why would a server block a crawler that robots.txt allows?
Because they are two different systems. robots.txt is a published request that a well-behaved crawler chooses to obey. Your hosting, CDN or security plugin sits in front of the file and decides whether the request is answered at all. When a bot-management rule matches an AI user agent, the crawler never gets far enough to read the permission you wrote.
Nearly a third of the sites in our sample, 147 of 499, were behind Cloudflare. We did not record which product produced each refusal and we are not claiming Cloudflare blocked these practices, but the direction the industry moved is documented. On 1 July 2025 Cloudflare announced it was "changing the default to block AI crawlers unless they pay creators for their content," and its AI Crawl Control product lets a site owner "set allow or block rules for individual crawlers."
Sources: Cloudflare, 1 July 2025; Cloudflare AI Crawl Control documentation; Citevio live-access study, July 2026.
For a dental practice, this is rarely a decision anyone remembers making. A rule like it can arrive as a default on a new hosting plan, as a security plugin's "block bad bots" checkbox, or as a line a developer added years ago and nobody revisited. The owner is told the website is fine, and by every measure they can see, it is.
Why does this matter for a dental practice?
Because an AI assistant can only work with what it is allowed to read, and it names very few practices to begin with. SOCi's 2026 Local Visibility Index, covering roughly 350,000 locations across 2,751 brands, found that 1.2% of locations get recommended on ChatGPT and 7.4% on Perplexity, against 35.9% in Google's local 3-pack. Which crawler your server turns away decides how much of that you give up.
That last part is worth spelling out, because not every refusal costs the same thing. The two identities refused most often in our sample are training crawlers: OpenAI documents GPTBot as the crawler that gathers content for training its models, and Anthropic describes ClaudeBot as its general-purpose crawler for training data. The crawlers closer to answer time were refused far less. OAI-SearchBot, which OpenAI says powers ChatGPT's search, was refused by 3.4%, and Bingbot, which builds the index ChatGPT searches for local questions, by 1.2%.
Sources: OpenAI crawler documentation; Anthropic, April 2026; ChatGPT's use of Bing for local queries per Damian Rollison, Search Engine Land, May 2025.
So a single refused crawler rarely erases a practice from an answer on its own. What it does is shrink the material these companies build their picture of you from, quietly, on a site whose owner has already checked the one file everybody tells them to check.
Access is the entry condition, not the promise. Opening the door does not get a practice named, and we have said so plainly in why your dental practice is not showing up in ChatGPT, and city by city in reports like our AI visibility report for Charlotte dentists. What this study adds is that a practice can believe the door is open, point at its own robots.txt as proof, and still be turned away at the step nobody checks.
Source: SOCi Local Visibility Index, 2026, via Search Engine Land.
What this study does not show
It is one snapshot of one segment, taken on 22 July 2026, and the segment is only sites whose robots.txt already welcomes AI crawlers. We identified our requests by user agent rather than by crawler IP, we tested homepages only, and we ran from a single exit address outside the US. Where these limits bias the result, they bias it downward: the true refusal rate is more likely to be above 14.4% than below it.
- User agent, not IP. Real AI crawlers arrive from published address ranges. Sites that filter by IP rather than by name would refuse them and still serve us, so we cannot see those blocks.
- Homepages only. Inner pages can behave differently, and a rule that spares the homepage may still shut a crawler out of your service pages.
- Network errors were not counted as refusals. Silent drops are scored as "no evidence", which lowers our number rather than raising it.
- One point in time. Bot-management settings change weekly, sometimes with a plan renewal nobody announces internally.
- No state breakdown. No single state in this sample reached the 300-site threshold we require before publishing a regional figure, so we are not publishing one.
How do I test my own site in five minutes?
Ask your own homepage for the same page twice, once as a browser and once as an AI crawler, and compare the status codes. A 200 for the browser and a 403, 429 or 503 for the crawler means your server is refusing it. If the two match, your server is not refusing by name, though a rule that filters by IP address instead would look the same from here.
- Run the crawler request. From a computer outside your practice network, run the command below. It prints a status code and nothing else.
- Run the same request without the crawler identity to get your browser baseline, then compare the two numbers.
- Read the result. 200 means served. 403, 429, 503 or a page full of "checking your browser" text means refused. Repeat for each crawler you care about.
- Take it to the right person. This is a hosting, CDN or security setting, so it goes to whoever runs your website infrastructure, not to whoever writes your content.
- Ask them one question: do we have a rule that blocks AI crawler user agents, and can we allow OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot?
curl -A "GPTBot" -s -o /dev/null -w "%{http_code}\n" https://yourpractice.com/
curl -s -o /dev/null -w "%{http_code}\n" https://yourpractice.com/One thing to expect in that conversation: a good host may answer that they block fake crawlers, and they are right to. Anyone can put "GPTBot" in a user agent string, which is exactly what our test did. The answer is not to trust the name but to verify the address, and OpenAI publishes the IP ranges its crawlers use for precisely that purpose. A rule can then allow the real crawler and keep refusing the impostor.
Source: OpenAI crawler documentation, accessed July 2026.
See whether your own site is refusing AI crawlers
The manual test above is the definitive check for one crawler at a time. The free check below is the fast version: it reads your site and scores its AI readiness, starting with whether AI crawlers can reach it at all. No email, no call.
Want to see what the engines currently say about you? Follow how to check what ChatGPT says about your practice.
Common questions
My robots.txt allows AI crawlers. Can my site still block them?
Yes, and it is common enough to be worth checking. robots.txt is a request that a well-behaved crawler reads before it asks for a page. Your server, CDN or security plugin is the door itself, and it can refuse the request no matter what the file says. In our July 2026 test of 499 dental practice sites whose robots.txt welcomed AI crawlers, 14.4% refused at least one of them at the server.
How do I check whether my server is refusing an AI crawler?
Ask for your own homepage twice: once with an ordinary browser user agent, once with an AI crawler's. If the browser request returns 200 and the crawler request returns 403, 429 or 503, or comes back as a challenge page, your server is treating that crawler differently. The exact user agent strings are published in OpenAI's crawler documentation.
Does refusing AI crawlers also hurt my Google ranking?
Usually not, because these are separate crawlers governed by separate rules. In our July 2026 test only 6 of 499 sites, 1.2%, refused Googlebot at all, while 12.6% served Googlebot normally and still refused an AI crawler. That gap is the whole finding: a site can be fully open to Google and still be closed to the crawlers OpenAI, Anthropic and Perplexity send.
Who do I ask to fix an AI crawler block?
Whoever manages your hosting, CDN or website security, which is usually your web developer rather than your marketing agency. Ask one question: do we have a rule that blocks AI crawler user agents? Cloudflare, as one example, lets a site owner set allow or block rules for individual crawlers, so the fix is normally a setting rather than a rebuild.