Of 499 dental practice websites whose robots.txt welcomes AI crawlers, 72 (14.4%) were refused by their own server anyway. The same homepage that loads in a browser came back to an AI crawler as a block or a bot challenge. In 63 of those cases, 12.6% of the sample, Googlebot was served normally in the same run. These responses differed by the crawler identity we declared; they do not establish how verified crawlers were treated.
Our requests used crawler user-agent names, not verified crawler IP addresses. Throughout this page, a crawler refusal means a refusal of our test request carrying that name. In our experience, advice about AI crawler access stops at robots.txt. Open the file, add the bots, done. That advice is incomplete, because robots.txt is a note pinned to the door and the server is the door. In July 2026 we tested how often the two disagree on sites that had already opened the file. On one in seven, they disagreed. Server access is one piece of a larger picture — see why robots.txt isn't why most dental practices are invisible to AI.
This is not 14.4% of US dental practices. The sample was drawn only from sites whose robots.txt already lets AI crawlers reach the homepage. Practices that block in robots.txt were left out on purpose, because the question here is what happens after the declaration says yes. Every figure on this page carries that denominator.
What did we test, and on which sites?
We drew 550 US dental practice domains at random from a frame of 5,019 whose robots.txt does not block AI crawlers at the root, then asked each one for its homepage eight times: once as an ordinary browser, six times as AI-related crawlers, and once as Googlebot for control. After dropping sites where the plain browser request also failed, 499 remained.
The frame came from a separate study we ran the same week. Across 7,632 US dental practice domains, 6,497 of which returned a readable answer, 11.4% block at least one AI crawler in robots.txt and another 11.4% returned no usable robots.txt file; 463 of those 739 returned HTTP 404, and the rest returned other status codes. The numbers behind that one are in are dental websites blocking AI crawlers. This page is about the group in the middle: the sites whose file says come in.
The source trail for that wider frame continues in the permanent archive under DOI 10.5281/zenodo.23143214.
| Test design | Value |
|---|---|
| Question | Does a site whose robots.txt allows AI crawlers actually serve them? |
| Frame | 5,019 US dental practice domains whose robots.txt leaves the root open to AI crawlers |
| Where the frame came from | Our 22 July 2026 robots.txt study of 7,632 dental domains, 6,497 readable |
| Sampling | Deterministic random draw, seed 20260721, first 550 domains |
| Requests per domain | 8: one browser baseline, six AI-related crawlers, one Googlebot control |
| Valid n | 499 |
| Dropped before scoring | 51, because the plain browser request failed too |
Those 51 sites matter more than they look. If a site turns away an ordinary Chrome visitor, it is not treating AI crawlers differently, it is turning away everyone, and counting it as an AI block would inflate the finding. So we removed them before scoring.
| Why we dropped a domain | Domains |
|---|---|
| Baseline returned HTTP 403 | 27 |
| Baseline request failed (DNS, TLS, timeout or reset) | 14 |
| Baseline was itself a challenge page, so the site challenges every visitor | 7 |
| Baseline returned HTTP 404 | 2 |
| Baseline returned HTTP 429 | 1 |
| Total dropped | 51 |
Citevio's dental market research is published openly under CC-BY-4.0 with a DOI, so the numbers can be checked and reused; client data is never published.
How many sites refused an AI crawler?
Seventy-two of the 499 sites whose robots.txt welcomes AI crawlers, or 14.4% of that group. Of those 72, sixty-nine (13.8%) returned an outright block status such as 403 or 429, and three (0.6%) returned a bot challenge page. A further 41 sites served every crawler but sent a different page body to at least one of them, which we treat as a signal worth flagging rather than a finding.
| What the server did | Sites | Share of the 499 |
|---|---|---|
| Served all six AI-related crawlers the page | 386 | 77.4% |
| Refused at least one AI crawler | 72 | 14.4% |
| Served every crawler, but returned a different page body to at least one | 41 | 8.2% |
Counted per crawler-and-site pair rather than per site, 79 of the refusals were an HTTP 403 and 38 were an HTTP 429. The bottom row deserves its own caution: a different page body can mean a legitimate variation as easily as it can mean cloaking, so we did not classify it as a refusal. It is in the table because leaving it out would make the picture look tidier than the data is.
Is this a general bot wall, or is it aimed at AI?
The responses differed by declared identity. In 63 of the 72 cases, 12.6% of the 499 sites whose robots.txt welcomes AI crawlers, the Googlebot control request came back clean: our request declaring Google's crawler identity was served normally while at least one AI-identified request was not. Across all 499 sites, only 6 refused Googlebot at all.
The Googlebot-named control shows that these sites did not treat all tested identities alike. It does not identify the security rule responsible or prove that a verified AI crawler would receive the same response.
| Control reading | Sites | Share of the 499 |
|---|---|---|
| Googlebot served normally, at least one AI crawler refused | 63 | 12.6% |
| Googlebot-named request refused anywhere in the sample | 6 | 1.2% |
Broken down by crawler, two names absorb most of the refusals. ClaudeBot was refused by 9.2% of the sample and GPTBot by 8.0%, while ChatGPT-User, the identity OpenAI uses for requests a person triggers inside ChatGPT, was refused by 1.4%. Googlebot sits at the bottom of the same list, which is the comparison that matters.
| Crawler | Refused | Share of the 499 |
|---|---|---|
| ClaudeBot (Anthropic) | 46 | 9.2% |
| GPTBot (OpenAI) | 40 | 8.0% |
| OAI-SearchBot (OpenAI) | 17 | 3.4% |
| PerplexityBot (Perplexity) | 10 | 2.0% |
| ChatGPT-User (OpenAI) | 7 | 1.4% |
| Bingbot (Microsoft) | 6 | 1.2% |
| Googlebot — control | 6 | 1.2% |
Could we have triggered those rate limits ourselves?
The sequence does not establish the cause of the rate limits. Every domain received the same eight requests in the same fixed sequence. A simple accumulating limit would tend to affect later requests, but the observed pattern differed: 37 of the 38 rate-limit responses landed on the first and fourth crawlers, and the seventh, our Googlebot control, took none of them.
| Order | Crawler | HTTP 429 | HTTP 403 |
|---|---|---|---|
| 1 | GPTBot | 21 | 16 |
| 2 | OAI-SearchBot | 0 | 17 |
| 3 | ChatGPT-User | 0 | 7 |
| 4 | ClaudeBot | 16 | 27 |
| 5 | PerplexityBot | 0 | 10 |
| 6 | Bingbot | 1 | 2 |
| 7 | Googlebot (control) | 0 | 3 |
These results are consistent with identity-specific rules, but do not prove that explanation. Randomizing request order, varying the delay and inspecting the host security logs would help separate identity filtering from timing and other security checks.
Why would a server block a crawler that robots.txt allows?
Because they are two different systems. robots.txt is a published request that a well-behaved crawler chooses to obey. Your hosting, CDN or security plugin sits in front of the file and decides whether the request is answered at all. When a bot-management rule matches an AI user agent, the crawler never gets far enough to read the permission you wrote.
Nearly a third of the sites in our sample, 147 of 499, were behind Cloudflare. We did not record which product produced each refusal and we are not claiming Cloudflare blocked these practices, but the direction the industry moved is documented. On 1 July 2025 Cloudflare announced it was "changing the default to block AI crawlers unless they pay creators for their content," and its AI Crawl Control product lets a site owner "set allow or block rules for individual crawlers."
Sources: Cloudflare, 1 July 2025; Cloudflare AI Crawl Control documentation; Citevio live-access study, July 2026.
For a dental practice, this is rarely a decision anyone remembers making. A rule like it can arrive as a default on a new hosting plan, as a security plugin's "block bad bots" checkbox, or as a line a developer added years ago and nobody revisited. The owner is told the website is fine, and by every measure they can see, it is.
Why does this matter for a dental practice?
Direct access to your site and being recommended are separate questions. An assistant may also use information from other sources. In a separate multi-location business study, SOCi's 2026 Local Visibility Index, covering roughly 350,000 locations across 2,751 brands, found that 1.2% of locations get recommended on ChatGPT and 7.4% on Perplexity, against 35.9% in Google's local 3-pack. Those figures describe that study, not the effect of crawler blocking on dental practices.
That last part is worth spelling out, because not every refusal costs the same thing. The two identities refused most often in our sample are training crawlers: OpenAI documents GPTBot as the crawler that gathers content for training its models, and Anthropic describes ClaudeBot as its general-purpose crawler for training data. The crawlers closer to answer time were refused far less. OAI-SearchBot, which OpenAI says powers ChatGPT's search, was refused by 3.4%, and Bingbot, Microsoft's search crawler, by 1.2%. This experiment did not test which search providers ChatGPT used.
Sources: OpenAI crawler documentation; Anthropic, April 2026; Historical commentary, rather than current provider documentation: Damian Rollison, Search Engine Land, May 2025.
So a single refused crawler rarely erases a practice from an answer on its own. This experiment measured responses to test requests, not a change in citations or recommendations. Training, search and user-triggered access should be reviewed separately.
Access is the entry condition, not the promise. Opening the door does not get a practice named, and we have said so plainly in why your dental practice is not showing up in ChatGPT, and city by city in reports like our AI visibility report for Charlotte dentists. What this study adds is that a practice can believe the door is open, point at its own robots.txt as proof, and still be turned away at the step nobody checks.
Sources: SOCi Local Visibility Index, 2026: SOCi recommendation rates; SOCi sample and scope; also reported by Search Engine Land.
What this study does not show
It is one snapshot of one segment, taken on 22 July 2026, and the segment is only sites whose robots.txt already welcomes AI crawlers. We identified our requests by user agent rather than by crawler IP, we tested homepages only, and we ran from a single exit address outside the US. These limits can affect the result in either direction. A host may reject our imitation while allowing a verified crawler, or serve us while rejecting real crawler traffic. This test does not determine the refusal rate for verified crawlers.
- User agent, not IP. Real AI crawlers arrive from published address ranges. A host can allow verified crawlers while refusing an imitation, or do the reverse. Our test cannot resolve either case without verified crawler traffic and server logs.
- Homepages only. Inner pages can behave differently, and a rule that spares the homepage may still shut a crawler out of your service pages.
- Network errors were not counted as refusals. Silent drops are scored as "no evidence"; they do not establish whether a verified crawler could get through.
- One point in time. The results are a historical snapshot; later bot-management settings were not measured.
- No state breakdown. No single state in this sample reached the 300-site threshold we require before publishing a regional figure, so we are not publishing one.
How do I test my own site in five minutes?
Ask your own homepage for the same page twice, once as a browser and once as an AI crawler, and compare the status codes. Compare both the status codes and returned content. A difference flags these test requests for investigation; matching codes do not prove matching content or access for a verified crawler.
- Run the crawler request. From a computer outside your practice network, run the command below. It prints a status code and saves the returned page so you can inspect it for a challenge.
- Run the browser-identified request below to get a comparison response, then compare the codes and saved page content.
- Read the result. A 200 response may still contain a challenge rather than your content. A 403, 429, 503 or challenge page warrants investigation; verify genuine crawler traffic in server logs before changing security rules.
- Take it to the right person. This is a hosting, CDN or security setting, so it goes to whoever runs your website infrastructure, not to whoever writes your content.
- Ask them one question: are verified search crawlers getting the intended content, and can we manage search access separately from training access?
curl -L -A "GPTBot" -s -o crawler-response.html -w "%{http_code}\n" https://yourpractice.com/
curl -L -A "Mozilla/5.0" -s -o browser-response.html -w "%{http_code}\n" https://yourpractice.com/One thing to expect in that conversation: a good host may answer that they block fake crawlers, and they are right to. Anyone can put "GPTBot" in a user agent string, which is exactly what our test did. The answer is not to trust the name but to verify the address, and OpenAI publishes the IP ranges its crawlers use for precisely that purpose. A rule can then allow the real crawler and keep refusing the impostor.
Source: OpenAI crawler documentation, accessed July 2026.
We walk through this same three-identity check on video in the ChatGPT-blocking video walkthrough, using the July 2026 sample.
See whether your own site is refusing AI crawlers
The manual comparison is a diagnostic starting point. The free check below provides another access signal; neither establishes how verified crawlers are treated without server-log evidence. No email, no call.
Want to see what the engines currently say about you? Follow how to check what ChatGPT says about your practice.
Common questions
My robots.txt allows AI crawlers. Can my site still block them?
Yes, and it is common enough to be worth checking. robots.txt is a request that a well-behaved crawler reads before it asks for a page. Your server, CDN or security plugin is the door itself, and it can refuse the request no matter what the file says. In our July 2026 test of 499 dental practice sites whose robots.txt welcomed AI crawlers, 14.4% refused at least one of them at the server.
How do I check whether my server is refusing an AI crawler?
Ask for your own homepage twice: once with an ordinary browser user agent, once with an AI crawler's. If the browser request returns 200 and the crawler request returns 403, 429 or 503, or comes back as a challenge page, these test requests received different treatment. Check the returned content and verified crawler traffic in server logs before deciding whether a genuine crawler is blocked. The exact user agent strings are published in OpenAI's crawler documentation.
Does refusing AI crawlers also hurt my Google ranking?
Usually not, because these are separate crawlers governed by separate rules. In our July 2026 test only 6 of 499 sites, 1.2%, refused Googlebot at all, while 12.6% served Googlebot normally and still refused an AI crawler. That gap is the whole finding: a site can be fully open to Google and still be closed to the crawlers OpenAI, Anthropic and Perplexity send.
Who do I ask to fix an AI crawler block?
Whoever manages your hosting, CDN or website security, which is usually your web developer rather than your marketing agency. Ask one question: do we have a rule that blocks AI crawler user agents? Cloudflare, as one example, lets a site owner set allow or block rules for individual crawlers, so the fix is normally a setting rather than a rebuild.
Which AI crawlers were refused most often on the sites tested in this study?
ClaudeBot led at 9.2% of the 499 sites, followed by GPTBot at 8.0%. ChatGPT-User, the identity OpenAI uses for requests a person triggers inside ChatGPT, was refused by only 1.4%, well below the training crawlers and close to Googlebot's 1.2% control rate.
Could Citevio's own testing have triggered these blocks by hitting rate limits?
The fixed request sequence does not rule that out. Every domain received the same eight requests in the same order. In the observed results, 37 of the 38 rate-limit responses landed on the first and fourth crawlers in the sequence, and our seventh-position Googlebot control took none of them, a pattern consistent with identity-specific rules but insufficient to prove the cause without varying the order and inspecting security logs.
Why would a server block a crawler that robots.txt already allows?
Because they are two different systems. robots.txt is a published request that a well-behaved crawler chooses to obey, while your hosting, CDN or security plugin sits in front of that file and decides whether the request is answered at all. When a bot-management rule matches an AI user agent, the crawler never gets far enough to read the permission you wrote.
Does refusing a crawler always mean losing citations, or could it just be an opt-out from training?
It depends which crawler is refused. GPTBot and ClaudeBot are training crawlers, documented as such by OpenAI and Anthropic, so refusing them is closer to a training opt-out. OAI-SearchBot, which OpenAI says powers ChatGPT's search, and Bingbot were refused far less often in our sample, at 3.4% and 1.2%. The former is documented for ChatGPT search; this experiment did not establish which search providers ChatGPT used or measure any effect on citations.
How many of the tested sites were running behind Cloudflare?
147 of the 499 sites, nearly a third. We did not record which product produced each individual refusal and are not claiming Cloudflare blocked these specific practices, but the industry direction is documented: Cloudflare announced on 1 July 2025 that it was changing its default to block AI crawlers unless they pay creators, and its AI Crawl Control product lets a site owner set allow or block rules for individual crawlers.
What share of local businesses actually get recommended by ChatGPT and Perplexity, once you're readable?
Small. SOCi's 2026 Local Visibility Index, covering roughly 350,000 locations across 2,751 brands, found that 1.2% of locations get recommended on ChatGPT and 7.4% on Perplexity, against 35.9% in Google's local 3-pack. Being readable removes one obstacle; it does not change how selective an engine is about who it names.
What command can I run myself to test whether my server is blocking a crawler?
Use the paired commands in the manual test section to compare browser-identified and crawler-identified requests. Inspect the saved content as well as the status codes: a 200 can contain a challenge. Differences warrant investigation; only verified crawler traffic in server logs can establish what happened to a genuine crawler.
If my host says they already block fake crawlers, should I trust that?
It is a reasonable thing for them to do, since anyone can put "GPTBot" in a user agent string, which is exactly what our test did. The fix is not to trust the name but to verify the address, since OpenAI publishes the IP ranges its real crawlers use; a rule can then allow the genuine crawler while still refusing the impostor.
Is the 14.4% figure the same as the share of all US dental practices blocking AI, or just this open-robots.txt subset?
Just this subset. The sample was drawn only from sites whose robots.txt already lets AI crawlers reach the homepage, so 14.4% is not 14.4% of US dental practices; practices that block in robots.txt were deliberately left out, because this study asks what happens after the declaration already says yes. Every figure on this page carries that denominator.
How many sites were dropped from this study before scoring, and why?
51 of the original 550 domains, because their homepage also failed for a plain browser request. A site that turns away an ordinary visitor is not treating AI crawlers differently, it is turning away everyone, and counting it as an AI-specific block would have inflated the finding: 27 returned an outright HTTP 403, 14 failed to connect at all, 7 challenged every visitor including the browser, and the rest returned other errors.
Does this study cover pages other than the homepage?
No, homepages only. Inner pages can behave differently, and a rule that spares the homepage may still shut a crawler out of your service pages, which is one of the limits we flag rather than paper over.
Could a site that blocks crawlers by IP address rather than by name have gone undetected in this study?
Yes. We identified our requests by user agent, and real AI crawlers arrive from published address ranges. A site that filters by IP rather than by declared name would refuse the genuine crawler and still serve our test, so those blocks can be missed. The reverse is also possible: a host may reject our imitation while allowing verified crawler traffic. This test cannot establish the direction of the overall bias.