- 0:00 The two gates: what robots.txt does not control
- 0:19 The measured result: 72 of 499 sites, 14.4%
- 0:55 How the sample was built — and what was dropped
- 1:39 The Googlebot control: 6 of 499, 1.2%
- 2:04 The sharpest comparison: 12.6% served Googlebot, refused AI
- 2:54 What caused it? We do not know
- 3:16 The one explanation this does rule out
- 3:33 The practice-level check: three identities
- 4:47 Do not stop at the status code — compare the body
- 5:30 What to ask your host, and what not to authorize
- 6:15 Should every crawler be allowed?
- 6:32 The causal limit: access is not a citation
- 7:11 Why there is no time promise here
- 7:31 The practical conclusion
- 7:53 Where the open data lives
What is the finding, in one paragraph?
Your robots.txt file can say yes while your server still says no. In Citevio's live crawler access study on 22 July 2026, 72 of 499 permissive US dental practice websites, 14.4%, refused at least one AI crawler at the server, even though their own robots.txt already allowed it. Only 6 of the 499, 1.2%, refused Googlebot at all, and 63, 12.6%, served Googlebot normally while refusing an AI crawler. This is an access finding, not a promise about being named by AI.
What did the measured result show?
Among 499 dental practice websites whose robots.txt already welcomes AI crawlers, 72, 14.4%, were refused by their own server anyway. That refusal happens after the policy question has already been settled: every site in the sample allowed AI crawlers in robots.txt, and every homepage also loaded for a plain browser request. The server, not the published policy, made the final call.
| Control reading | Sites | Share of the 499 |
|---|---|---|
| Refused at least one AI crawler at the server | 72 | 14.4% |
| Served Googlebot normally, refused an AI crawler | 63 | 12.6% |
| Refused Googlebot anywhere in the sample | 6 | 1.2% |
Source: Citevio live crawler access study, 22 July 2026, n=499. Full methodology and raw data at citevio.com/data.
The declaration baseline comes from a separate, wider study: 6,497 readable US dental practice robots.txt files, read on 22 July 2026 across 13 AI and data crawlers.
Is this a general bot wall, or is it aimed at AI?
It looks aimed at AI, not at automated traffic in general. Only 6 of 499 sites, 1.2%, refused Googlebot at all. In 63 of the 499, 12.6%, Googlebot was served normally while at least one AI crawler was refused on the same homepage, in the same run. A general bot wall would be expected to catch Googlebot too; this pattern mostly did not.
A single browser check would miss this entirely. It only proves that the ordinary request succeeded; it says nothing about the response delivered to a different identity. Reproducing the comparison requires preserving the request headers, response status, final URL and a small body signature for each identity tested. Without that record, a diagnosis is still only a guess.
What caused the difference, and what does this rule out?
We do not know what caused it. The study recorded responses, not configuration histories, so it cannot say whether a host, a content delivery network, a security product, a plugin, or a human decision created the rule. Naming one as the cause would turn an observation into a diagnosis this instrument cannot support.
The finding still rules out one explanation. The refusal cannot be explained by the public robots.txt file alone, because that file permitted the crawler by design in every site in this sample. Whatever created the refusal acted after the file had already answered.
How do I test whether my own site is blocking an AI crawler?
Do not ask only whether robots.txt looks open. Ask whether the same homepage is actually delivered when the requester identifies itself differently. A useful comparison needs a baseline and a control, so start with the same three identities used in this study.
- Ordinary browser first. If this request fails, the result does not isolate crawler treatment; it could be a general outage or a broken page. Establish that the homepage can be served before blaming the crawler-specific layer.
- Then an AI crawler identity. Keep the page, the moment and the request method the same. Record the returned status and the response body; a statement that someone "checked the setting" is not the same as the record you need.
- Then Googlebot, as a control. If the browser and Googlebot both receive the homepage while the AI crawler is refused, the pattern becomes specific rather than general. You still do not know the cause, but you know which layer deserves inspection.
Why shouldn't you stop at the status code?
A challenge page can arrive with a superficially successful response. Compare the page body, the content length and the visible title, not just the number. A useful body check looks for a stable piece of the real page, such as its visible practice name or a known heading, then a sanity review: a matching fragment does not prove full rendering, and a changed fragment does not automatically prove blocking.
Once you have that record, ask the host for the exact rule that produced the difference. The question is not "are we fine?" It is "which rule handled this user agent, and can we see it?" If support cannot identify the decision point, ask them to reproduce the requests beside you and retain the result. Do not authorize a broad security change merely to make a test pass; the narrow goal is to understand the existing behavior, choose the intended policy, and verify that the live response now matches that choice.
Which OpenAI user agents should I look for in my robots.txt?
OpenAI's bot documentation lists four: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. It states that "each setting is independent of the others", so a rule for one does not decide the others. Only OAI-SearchBot is tied to ChatGPT search results.
- OAI-SearchBot. OpenAI says it "is used to surface websites in search results in ChatGPT's search features" and that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." This is the rule that bears on appearing in ChatGPT search.
- OAI-AdsBot. OpenAI says it is "used to validate the safety of web pages submitted as ads on ChatGPT", so it concerns pages submitted as ads, not search or training.
- GPTBot. OpenAI describes it as crawling "content that may be used in training our generative AI foundation models", and says disallowing it "indicates a site's content should not be used in training." Blocking it is a training choice, and OpenAI documents it as a separate setting from search.
- ChatGPT-User. OpenAI says it is used "for certain user actions in ChatGPT and Custom GPTs" and that "because these actions are initiated by a user, robots.txt rules may not apply." A robots.txt line is therefore a weak signal for this agent, which is one more reason to test what your server actually delivers.
Source: OpenAI, crawler documentation, accessed 29 September 2026. The per-crawler breakdown in Citevio's published dataset lists GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Bingbot, with Googlebot as the control.
Should every crawler be allowed?
That is a business and policy choice for the practice, not something this measurement can decide. Some crawlers may be refused intentionally. The finding does not tell a practice what its policy should be; it tells a practice whether the published choice and the enforced choice are actually the same.
This is one day's measurement: hosting and content delivery rules change without notice, and repeating the test on another day will not reproduce this file exactly. It is not a causal test: this study measured whether a homepage was served, not whether removing a refusal causes an assistant to name, cite or recommend a practice, and it did not measure how long any access change takes to appear in an assistant's answer. The layer that produced any given refusal, host, CDN, security product, plugin or human choice, was not recorded, so naming one would turn an observation into a diagnosis this study cannot support. And this access-study percentage is a separate reading from Citevio's broader robots.txt-declaration survey; the two are not interchangeable and should not be averaged together.
Being readable is a prerequisite, not a guarantee. An engine may fetch a page and still omit the practice; it may use different sources or rebuild the answer on another run. A successful request removes one obstacle, and it does not purchase a result no vendor can honestly offer.
What is the practical conclusion?
Read the policy file, then test the delivered page. Keep the ordinary browser and Googlebot as controls. If the responses differ, ask for the exact enforced rule instead of asking whether everything is fine. Do not infer a visibility outcome from an access result; confirm what the server actually does.
How did we measure this?
In Citevio's live crawler access study, 550 US dental practice websites whose robots.txt already allowed AI crawlers were sampled, and each homepage was requested eight times in a fixed order on 22 July 2026: once as an ordinary browser, six times under AI crawler identities, and once as Googlebot for control. Sites whose homepage failed even for the plain browser request were dropped, which left a denominator of 499. The aggregated counts, method and known limitations are published at citevio.com/data. No practice name or domain is published in the aggregate package, and no client data is included.
Citevio's dental market research is published openly under CC-BY-4.0 with a DOI, so the numbers can be checked and reused; client data is never published.
The permanent identifier for the underlying research archive is DOI 10.5281/zenodo.23143214.
Common questions
Can my robots.txt allow AI crawlers while my server still blocks them?
Yes. In Citevio's live crawler access study on 22 July 2026, 72 of 499 dental practice websites whose robots.txt already allowed AI crawlers, 14.4%, were refused by their own server anyway. The published policy file is only the first gate; the server applies a separate rule at the second gate.
Is this a general bot wall, or is it aimed specifically at AI crawlers?
It looks identity-specific rather than general. Only 6 of 499 sites, 1.2%, refused Googlebot at all, while 63 of 499, 12.6%, served Googlebot normally and still refused an AI crawler. The gap between those two numbers is what makes the pattern look aimed at AI rather than at automated traffic in general.
Does passing this access check mean ChatGPT will recommend my practice?
No. This study measured whether a homepage was served to an AI crawler, not whether an assistant names, cites or recommends the practice afterward. Being reachable is a precondition, not a guarantee, and no causal link between access and a citation has been tested here.
How do I test whether my own dental website is blocking an AI crawler?
Request your homepage under three identities, an ordinary browser, an AI crawler, and Googlebot as a control, and compare the status code, response body and visible title for each. If the browser and Googlebot get the real page while the AI crawler does not, ask your host for the exact rule that handled that user agent.
Why do you test Googlebot as well as an AI crawler when checking my own site?
Because it tells you whether the block is aimed at AI specifically or is a general bot wall. In Citevio's live crawler access study on 22 July 2026, only 6 of 499 permissive sites, 1.2%, refused Googlebot at all, while 63, 12.6%, served Googlebot normally and still refused an AI crawler on the same homepage. If your own test shows Googlebot getting through while an AI crawler does not, the block is specific rather than general.
Why shouldn't I stop at checking the HTTP status code?
Because a challenge page can arrive disguised as a success. Compare the response body, content length and visible title as well as the status code, looking for a stable piece of the real page such as your practice name or a known heading. A matching fragment does not prove full rendering, and a changed fragment does not automatically prove blocking, so record what actually arrived before deciding what it means.
What should I ask my host if I find a difference between identities?
Ask which specific rule handled that user agent, not whether everything is fine, since reassurance without a result you can see is not evidence. If support cannot identify the decision point, ask them to reproduce the requests beside you and retain the result. Do not authorize a broad security change just to make a test pass; the goal is to understand the existing behavior, choose the intended policy, and verify the live response matches it.
Should I try to block every crawler, or leave AI crawlers open?
That is a business and policy choice for your practice, not something this measurement can decide. Some crawlers may be refused intentionally. What the finding tells you is only whether your published choice and your enforced choice actually match, not what that choice should be.
What caused the refusal on the sites in this study, and can you tell me the cause on my site?
We do not know, and this study cannot tell you either. It recorded responses, not configuration histories, so it cannot say whether a host, a CDN, a security product, a plugin or a human decision created any given rule. What it does rule out is the public robots.txt file alone, because every site in the sample had already allowed the crawler there; whatever created the refusal acted after the file had already answered.
If my site passes this check today, will it definitely still pass next week?
No. This is one day's measurement, hosting and content delivery rules change without notice, and repeating the test on another day will not reproduce this exact result. Treat a passing test as a snapshot, not a permanent guarantee, and re-check periodically rather than once.
Does this video tell me how long a fix takes to show up in ChatGPT's answers?
No. This study measured whether a homepage was served, not how long any access change takes to appear in an assistant's answer, and no time-to-citation distribution was measured. A retesting schedule can be chosen; a duration promise cannot be derived from anything measured here.
Is there a faster way to check this than running the manual test myself?
Yes. The free checker embedded on this page reads your site and scores its AI readiness in seconds, crawler access included. The manual three-identity test above remains the definitive check if you want to see the exact status codes and response bodies yourself.
What information should I record when I test my own site?
The request headers, the response status, the final URL and a small body signature for each identity you test, an ordinary browser, an AI crawler and Googlebot as a control. Without that record, a diagnosis is still only a guess, and it is what lets your host, or anyone else, reproduce the comparison.
Full transcript
Your robots.txt file can say yes while your server still says no. That is the answer to "is my dental website blocking ChatGPT?" The public policy file is only the first gate. The homepage request meets a second gate, where the server applies a different rule.
Here is the measured result. In Citevio's live crawler access study on 22 July 2026, 72 of 499 permissive US dental practice websites, or 14.4%, refused at least one AI crawler at the server. That is an access finding, not a promise about being named by AI. This is one day's measurement. Hosting and content delivery rules change without notice, and a repeat on another day will not reproduce this file exactly.
That design starts after the policy question has already been settled. Every site in the final sample allowed AI crawlers in robots.txt. Every homepage also loaded for a plain browser request. We then changed the identity making the request and recorded whether the server treated the crawler identity differently.
Why separate those layers? Because robots.txt is a statement. A server response is enforcement. The statement can be permissive while a firewall, hosting rule, security layer, or another system refuses the request later. The public rule and the delivered page can disagree without either record disappearing.
The control request makes the finding easier to interpret. In Citevio's live crawler access study on 22 July 2026, 6 of 499 permissive US dental practice websites, or 1.2%, refused Googlebot. That gives us a way to distinguish a general bot wall from treatment aimed at a particular identity.
The strongest comparison is narrower. In the Citevio crawler access study on 22 July 2026, 12.6% of those 499 permissive US dental practice websites refused an AI crawler while serving Googlebot normally. The same homepage gave a different answer depending on who was asking for it. This is why a single browser check is weak evidence. It proves only that the ordinary request succeeded. It says nothing about the response delivered to another identity. Preserve the request headers, response status, final URL, and a small body signature so another person can reproduce the comparison. If the evidence cannot be reproduced, the diagnosis is still only a guess.
What caused the difference? We do not know. The study recorded responses, not configuration histories. It cannot tell us whether a host, content delivery network, security product, plugin, or human choice created the rule. Naming one as the cause would turn an observation into a diagnosis our instrument cannot support.
The finding still rules out one explanation. The refusal cannot be explained by the public robots.txt file alone, because that file permitted the crawler by design. Whatever created the refusal acted after that file had already answered.
Now translate that into a practice-level check. Do not ask only whether robots.txt looks open. Ask whether the same homepage is actually delivered when the requester identifies itself differently. A useful comparison needs a baseline and a control, so start with the same three identities.
First, request the homepage as an ordinary browser. If that request fails, the result does not isolate crawler treatment. It could be a general outage, a network failure, or a broken page. Establish that the homepage can be served before blaming the crawler-specific layer.
Second, request the same homepage under an AI crawler identity. Keep the page, the moment, and the request method the same. Record the returned status and the response body. A statement that someone "checked the setting" is not the same as the record you need.
Third, use Googlebot as a control. If the ordinary browser and Googlebot receive the homepage while the AI crawler is refused, the pattern becomes specific rather than general. You still do not know the cause, but you know which layer deserves inspection.
Do not stop at the status code. A challenge page can arrive with a superficially successful response. Compare the page body, content length, and visible title. The question is whether the crawler received the real homepage, so preserve enough of each response to compare the requests rather than the number alone. A useful body check looks for a stable piece of the real page, such as its visible practice name or a known heading. That check also needs a sanity review. A matching fragment does not prove full rendering, and a changed fragment does not automatically prove blocking. Record what arrived before deciding what it means.
Then ask the host for the exact rule that produced the difference. The question is not "are we fine?" It is "which rule handled this user agent, and can we see it?" A rule can be reviewed and changed. Reassurance without a result you can see is not evidence. If support cannot identify the decision point, ask them to reproduce the requests beside you and retain the result. Do not authorize a broad security change merely to make a test pass. The narrow goal is to understand the existing behavior, choose the intended policy, and verify that the live response now matches that choice.
Should every crawler be allowed? That is a business and policy choice for the practice. Some crawlers may be refused intentionally. The measurement does not tell you what your policy should be. It tells you the published choice and the enforced choice should be the same.
Here is the causal limit. We did not measure whether removing a refusal causes an assistant to name, cite, or recommend a practice. We measured whether a homepage was served. We have not measured whether changing this access state changes an AI visibility outcome in a controlled experiment.
Being readable is a prerequisite. It is not a guarantee. An engine may fetch a page and still omit the practice. It may use different sources or rebuild the answer on another run. A successful request removes one obstacle; it does not purchase a result that no vendor can honestly offer.
The same caution applies to time. We did not measure how long any access change takes to appear in an assistant answer, and we did not measure a time-to-citation distribution. A schedule for retesting can be chosen. A duration promise cannot be derived from anything measured in this access study.
So the practical conclusion is small and testable. Read the policy file, then test the delivered page. Keep the ordinary browser and Googlebot controls. If the responses differ, ask for the exact enforced rule. Do not infer a visibility outcome. Confirm what the server actually does.
The aggregated counts, method, and known limitations are published at citevio.com/data. No practice name or domain is published in the aggregate package, and no client data is included. If you want to challenge the denominator or the definition of refusal, you will find the complete record there.
Narration in this video is synthesized; the data, methodology and limitations are our own and are linked below.
Prefer to run the check by hand? See how to test if AI can read your dental website.