Of 499 dental practice websites whose robots.txt welcomes AI crawlers, 72 (14.4%) were refused by their own server anyway. The same homepage that loads in a browser came back to an AI crawler as a block or a bot challenge. In 63 of those cases, 12.6% of the sample, Googlebot was served normally in the same run. This is not a general bot wall. It is AI-specific.
In our experience, advice about AI crawler access stops at robots.txt. Open the file, add the bots, done. That advice is incomplete, because robots.txt is a note pinned to the door and the server is the door. In July 2026 we tested how often the two disagree on sites that had already opened the file. On one in seven, they disagreed. Server access is one piece of a larger picture — see why robots.txt isn't why most dental practices are invisible to AI.
This is not 14.4% of US dental practices. The sample was drawn only from sites whose robots.txt already lets AI crawlers reach the homepage. Practices that block in robots.txt were left out on purpose, because the question here is what happens after the declaration says yes. Every figure on this page carries that denominator.
What did we test, and on which sites?
We drew 550 US dental practice domains at random from a frame of 5,019 whose robots.txt does not block AI crawlers at the root, then asked each one for its homepage eight times: once as an ordinary browser, six times as AI-related crawlers, and once as Googlebot for control. After dropping sites where the plain browser request also failed, 499 remained.
The frame came from a separate study we ran the same week. Across 7,632 US dental practice domains, 6,497 of which returned a readable answer, 11.4% block at least one AI crawler in robots.txt and another 11.4% publish no robots.txt at all. The numbers behind that one are in are dental websites blocking AI crawlers. This page is about the group in the middle: the sites whose file says come in.
The source trail for that wider frame continues in the permanent archive under DOI 10.5281/zenodo.22016876.
| Test design | Value |
|---|---|
| Question | Does a site whose robots.txt allows AI crawlers actually serve them? |
| Frame | 5,019 US dental practice domains whose robots.txt leaves the root open to AI crawlers |
| Where the frame came from | Our 22 July 2026 robots.txt study of 7,632 dental domains, 6,497 readable |
| Sampling | Deterministic random draw, seed 20260721, first 550 domains |
| Requests per domain | 8: one browser baseline, six AI-related crawlers, one Googlebot control |
| Valid n | 499 |
| Dropped before scoring | 51, because the plain browser request failed too |
Those 51 sites matter more than they look. If a site turns away an ordinary Chrome visitor, it is not treating AI crawlers differently, it is turning away everyone, and counting it as an AI block would inflate the finding. So we removed them before scoring.
| Why we dropped a domain | Domains |
|---|---|
| Baseline returned HTTP 403 | 27 |
| Baseline request failed (DNS, TLS, timeout or reset) | 14 |
| Baseline was itself a challenge page, so the site challenges every visitor | 7 |
| Baseline returned HTTP 404 | 2 |
| Baseline returned HTTP 429 | 1 |
| Total dropped | 51 |
Citevio's dental market research is published openly under CC-BY-4.0 with a DOI, so the numbers can be checked and reused; client data is never published.
How many sites refused an AI crawler?
Seventy-two of the 499 sites whose robots.txt welcomes AI crawlers, or 14.4% of that group. Of those 72, sixty-nine (13.8%) returned an outright block status such as 403 or 429, and three (0.6%) returned a bot challenge page. A further 41 sites served every crawler but sent a different page body to at least one of them, which we treat as a signal worth flagging rather than a finding.
| What the server did | Sites | Share of the 499 |
|---|---|---|
| Served all six AI-related crawlers the page | 386 | 77.4% |
| Refused at least one AI crawler | 72 | 14.4% |
| Served every crawler, but returned a different page body to at least one | 41 | 8.2% |
Counted per crawler-and-site pair rather than per site, 79 of the refusals were an HTTP 403 and 38 were an HTTP 429. The bottom row deserves its own caution: a different page body can mean a legitimate variation as easily as it can mean cloaking, so we did not publish it as a rate. It is in the table because leaving it out would make the picture look tidier than the data is.
Is this a general bot wall, or is it aimed at AI?
It is aimed at AI. In 63 of the 72 cases, 12.6% of the 499 sites whose robots.txt welcomes AI crawlers, the Googlebot control request came back clean: Google's crawler was served the same page a browser gets while at least one AI crawler was not. Across all 499 sites, only 6 refused Googlebot at all.
This control is the reason the finding holds up. Without it, anyone could answer that these practices simply block every bot and AI is not being singled out. The measurement says otherwise, and the gap is wide enough that it is hard to read another way.
| Control reading | Sites | Share of the 499 |
|---|---|---|
| Googlebot served normally, at least one AI crawler refused | 63 | 12.6% |
| Googlebot refused anywhere in the sample (a wall against every bot) | 6 | 1.2% |
Broken down by crawler, two names absorb most of the refusals. ClaudeBot was refused by 9.2% of the sample and GPTBot by 8.0%, while ChatGPT-User, the identity OpenAI uses for requests a person triggers inside ChatGPT, was refused by 1.4%. Googlebot sits at the bottom of the same list, which is the comparison that matters.
| Crawler | Refused | Share of the 499 |
|---|---|---|
| ClaudeBot (Anthropic) | 46 | 9.2% |
| GPTBot (OpenAI) | 40 | 8.0% |
| OAI-SearchBot (OpenAI) | 17 | 3.4% |
| PerplexityBot (Perplexity) | 10 | 2.0% |
| ChatGPT-User (OpenAI) | 7 | 1.4% |
| Bingbot (Microsoft) | 6 | 1.2% |
| Googlebot — control | 6 | 1.2% |
Could we have triggered those rate limits ourselves?
No, and the request order rules it out. Every domain received the same eight requests in the same fixed sequence. If our own pace had caused the 429 responses, the effect would build up and the last identities in the sequence would be hit hardest. The opposite happened: 37 of the 38 rate-limit responses landed on the first and fourth crawlers, and the seventh, our Googlebot control, took none of them.
| Order | Crawler | HTTP 429 | HTTP 403 |
|---|---|---|---|
| 1 | GPTBot | 21 | 16 |
| 2 | OAI-SearchBot | 0 | 17 |
| 3 | ChatGPT-User | 0 | 7 |
| 4 | ClaudeBot | 16 | 27 |
| 5 | PerplexityBot | 0 | 10 |
| 6 | Bingbot | 1 | 2 |
| 7 | Googlebot (control) | 0 | 3 |
A cumulative rate limit cannot leave positions two, three and five at zero and then produce 16 rate limits at position four. Something in front of these sites reads the name on the request and decides. That is what an AI-specific rule looks like in a log file.
Why would a server block a crawler that robots.txt allows?
Because they are two different systems. robots.txt is a published request that a well-behaved crawler chooses to obey. Your hosting, CDN or security plugin sits in front of the file and decides whether the request is answered at all. When a bot-management rule matches an AI user agent, the crawler never gets far enough to read the permission you wrote.
Nearly a third of the sites in our sample, 147 of 499, were behind Cloudflare. We did not record which product produced each refusal and we are not claiming Cloudflare blocked these practices, but the direction the industry moved is documented. On 1 July 2025 Cloudflare announced it was "changing the default to block AI crawlers unless they pay creators for their content," and its AI Crawl Control product lets a site owner "set allow or block rules for individual crawlers."
Sources: Cloudflare, 1 July 2025; Cloudflare AI Crawl Control documentation; Citevio live-access study, July 2026.
For a dental practice, this is rarely a decision anyone remembers making. A rule like it can arrive as a default on a new hosting plan, as a security plugin's "block bad bots" checkbox, or as a line a developer added years ago and nobody revisited. The owner is told the website is fine, and by every measure they can see, it is.
Why does this matter for a dental practice?
Because an AI assistant can only work with what it is allowed to read, and it names very few practices to begin with. SOCi's 2026 Local Visibility Index, covering roughly 350,000 locations across 2,751 brands, found that 1.2% of locations get recommended on ChatGPT and 7.4% on Perplexity, against 35.9% in Google's local 3-pack. Which crawler your server turns away decides how much of that you give up.
That last part is worth spelling out, because not every refusal costs the same thing. The two identities refused most often in our sample are training crawlers: OpenAI documents GPTBot as the crawler that gathers content for training its models, and Anthropic describes ClaudeBot as its general-purpose crawler for training data. The crawlers closer to answer time were refused far less. OAI-SearchBot, which OpenAI says powers ChatGPT's search, was refused by 3.4%, and Bingbot, which builds the index ChatGPT searches for local questions, by 1.2%.
Sources: OpenAI crawler documentation; Anthropic, April 2026; ChatGPT's use of Bing for local queries per Damian Rollison, Search Engine Land, May 2025.
So a single refused crawler rarely erases a practice from an answer on its own. What it does is shrink the material these companies build their picture of you from, quietly, on a site whose owner has already checked the one file everybody tells them to check.
Access is the entry condition, not the promise. Opening the door does not get a practice named, and we have said so plainly in why your dental practice is not showing up in ChatGPT, and city by city in reports like our AI visibility report for Charlotte dentists. What this study adds is that a practice can believe the door is open, point at its own robots.txt as proof, and still be turned away at the step nobody checks.
Source: SOCi Local Visibility Index, 2026, via Search Engine Land.
What this study does not show
It is one snapshot of one segment, taken on 22 July 2026, and the segment is only sites whose robots.txt already welcomes AI crawlers. We identified our requests by user agent rather than by crawler IP, we tested homepages only, and we ran from a single exit address outside the US. Where these limits bias the result, they bias it downward: the true refusal rate is more likely to be above 14.4% than below it.
- User agent, not IP. Real AI crawlers arrive from published address ranges. Sites that filter by IP rather than by name would refuse them and still serve us, so we cannot see those blocks.
- Homepages only. Inner pages can behave differently, and a rule that spares the homepage may still shut a crawler out of your service pages.
- Network errors were not counted as refusals. Silent drops are scored as "no evidence", which lowers our number rather than raising it.
- One point in time. Bot-management settings change weekly, sometimes with a plan renewal nobody announces internally.
- No state breakdown. No single state in this sample reached the 300-site threshold we require before publishing a regional figure, so we are not publishing one.
How do I test my own site in five minutes?
Ask your own homepage for the same page twice, once as a browser and once as an AI crawler, and compare the status codes. A 200 for the browser and a 403, 429 or 503 for the crawler means your server is refusing it. If the two match, your server is not refusing by name, though a rule that filters by IP address instead would look the same from here.
- Run the crawler request. From a computer outside your practice network, run the command below. It prints a status code and nothing else.
- Run the same request without the crawler identity to get your browser baseline, then compare the two numbers.
- Read the result. 200 means served. 403, 429, 503 or a page full of "checking your browser" text means refused. Repeat for each crawler you care about.
- Take it to the right person. This is a hosting, CDN or security setting, so it goes to whoever runs your website infrastructure, not to whoever writes your content.
- Ask them one question: do we have a rule that blocks AI crawler user agents, and can we allow OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot?
curl -A "GPTBot" -s -o /dev/null -w "%{http_code}\n" https://yourpractice.com/
curl -s -o /dev/null -w "%{http_code}\n" https://yourpractice.com/One thing to expect in that conversation: a good host may answer that they block fake crawlers, and they are right to. Anyone can put "GPTBot" in a user agent string, which is exactly what our test did. The answer is not to trust the name but to verify the address, and OpenAI publishes the IP ranges its crawlers use for precisely that purpose. A rule can then allow the real crawler and keep refusing the impostor.
Source: OpenAI crawler documentation, accessed July 2026.
We walk through this same three-identity check on video in the ChatGPT-blocking video walkthrough, using the July 2026 sample.
See whether your own site is refusing AI crawlers
The manual test above is the definitive check for one crawler at a time. The free check below is the fast version: it reads your site and scores its AI readiness, starting with whether AI crawlers can reach it at all. No email, no call.
Want to see what the engines currently say about you? Follow how to check what ChatGPT says about your practice.
Common questions
My robots.txt allows AI crawlers. Can my site still block them?
Yes, and it is common enough to be worth checking. robots.txt is a request that a well-behaved crawler reads before it asks for a page. Your server, CDN or security plugin is the door itself, and it can refuse the request no matter what the file says. In our July 2026 test of 499 dental practice sites whose robots.txt welcomed AI crawlers, 14.4% refused at least one of them at the server.
How do I check whether my server is refusing an AI crawler?
Ask for your own homepage twice: once with an ordinary browser user agent, once with an AI crawler's. If the browser request returns 200 and the crawler request returns 403, 429 or 503, or comes back as a challenge page, your server is treating that crawler differently. The exact user agent strings are published in OpenAI's crawler documentation.
Does refusing AI crawlers also hurt my Google ranking?
Usually not, because these are separate crawlers governed by separate rules. In our July 2026 test only 6 of 499 sites, 1.2%, refused Googlebot at all, while 12.6% served Googlebot normally and still refused an AI crawler. That gap is the whole finding: a site can be fully open to Google and still be closed to the crawlers OpenAI, Anthropic and Perplexity send.
Who do I ask to fix an AI crawler block?
Whoever manages your hosting, CDN or website security, which is usually your web developer rather than your marketing agency. Ask one question: do we have a rule that blocks AI crawler user agents? Cloudflare, as one example, lets a site owner set allow or block rules for individual crawlers, so the fix is normally a setting rather than a rebuild.
Which AI crawlers were refused most often on the sites tested in this study?
ClaudeBot led at 9.2% of the 499 sites, followed by GPTBot at 8.0%. ChatGPT-User, the identity OpenAI uses for requests a person triggers inside ChatGPT, was refused by only 1.4%, well below the training crawlers and close to Googlebot's 1.2% control rate.
Could Citevio's own testing have triggered these blocks by hitting rate limits?
No, and the request order rules it out. Every domain received the same eight requests in the same fixed sequence, so a cumulative rate limit caused by our own pace would build up and hit the last identities hardest. Instead, 37 of the 38 rate-limit responses landed on the first and fourth crawlers in the sequence, and our seventh-position Googlebot control took none of them, a shape only a rule that matches specific user agents can produce.
Why would a server block a crawler that robots.txt already allows?
Because they are two different systems. robots.txt is a published request that a well-behaved crawler chooses to obey, while your hosting, CDN or security plugin sits in front of that file and decides whether the request is answered at all. When a bot-management rule matches an AI user agent, the crawler never gets far enough to read the permission you wrote.
Does refusing a crawler always mean losing citations, or could it just be an opt-out from training?
It depends which crawler is refused. GPTBot and ClaudeBot are training crawlers, documented as such by OpenAI and Anthropic, so refusing them is closer to a training opt-out. OAI-SearchBot, which OpenAI says powers ChatGPT's search, and Bingbot, which builds the index ChatGPT uses for local questions, were refused far less often in our sample, at 3.4% and 1.2%, and refusing those sits closer to the crawlers that shape what an assistant can cite.
How many of the tested sites were running behind Cloudflare?
147 of the 499 sites, nearly a third. We did not record which product produced each individual refusal and are not claiming Cloudflare blocked these specific practices, but the industry direction is documented: Cloudflare announced on 1 July 2025 that it was changing its default to block AI crawlers unless they pay creators, and its AI Crawl Control product lets a site owner set allow or block rules for individual crawlers.
What share of local businesses actually get recommended by ChatGPT and Perplexity, once you're readable?
Small. SOCi's 2026 Local Visibility Index, covering roughly 350,000 locations across 2,751 brands, found that 1.2% of locations get recommended on ChatGPT and 7.4% on Perplexity, against 35.9% in Google's local 3-pack. Being readable removes one obstacle; it does not change how selective an engine is about who it names.
What command can I run myself to test whether my server is blocking a crawler?
Run curl -A "GPTBot" -s -o /dev/null -w "%{http_code}\n" https://yourpractice.com/ and compare the status code it prints against the same command without the crawler identity. A 200 for the plain request and a 403, 429 or 503 for the crawler-identified one means your server is treating that crawler differently.
If my host says they already block fake crawlers, should I trust that?
It is a reasonable thing for them to do, since anyone can put "GPTBot" in a user agent string, which is exactly what our test did. The fix is not to trust the name but to verify the address, since OpenAI publishes the IP ranges its real crawlers use; a rule can then allow the genuine crawler while still refusing the impostor.
Is the 14.4% figure the same as the share of all US dental practices blocking AI, or just this open-robots.txt subset?
Just this subset. The sample was drawn only from sites whose robots.txt already lets AI crawlers reach the homepage, so 14.4% is not 14.4% of US dental practices; practices that block in robots.txt were deliberately left out, because this study asks what happens after the declaration already says yes. Every figure on this page carries that denominator.
How many sites were dropped from this study before scoring, and why?
51 of the original 550 domains, because their homepage also failed for a plain browser request. A site that turns away an ordinary visitor is not treating AI crawlers differently, it is turning away everyone, and counting it as an AI-specific block would have inflated the finding: 27 returned an outright HTTP 403, 14 failed to connect at all, 7 challenged every visitor including the browser, and the rest returned other errors.
Does this study cover pages other than the homepage?
No, homepages only. Inner pages can behave differently, and a rule that spares the homepage may still shut a crawler out of your service pages, which is one of the limits we flag rather than paper over.
Could a site that blocks crawlers by IP address rather than by name have gone undetected in this study?
Yes. We identified our requests by user agent, and real AI crawlers arrive from published address ranges. A site that filters by IP rather than by declared name would refuse the genuine crawler and still serve our test, so blocks of that kind are not visible to this method, which means the true refusal rate is more likely to sit above 14.4% than below it.