To test if AI can read your dental website, check three things in order: whether crawlers can reach your server at all, whether your actual content appears in the raw HTML (not just after JavaScript runs), and whether your robots.txt file matches what your server does in practice. Passing all three means a crawler can read your site — it does not mean an AI engine will mention your practice.
A normal browser view does not show every condition an automated crawler may encounter. These three tests help you check whether the server responds, whether key content is in the raw HTML, and whether the server treats crawler requests differently.
What exactly are you testing for?
"Can AI read my site" is really three separate questions, and each one fails for a different reason. A site can pass one test and fail another — so testing only one gives you a false sense of security.
- Access — can the crawler's request reach your server and get a response, rather than a timeout, an error, or a firewall block?
- Render — once the crawler gets a response, is your actual page content sitting in that raw HTML, or does it only appear after JavaScript builds the page in a browser?
- Instruction — does your robots.txt file actually match what your server does? A robots.txt that allows a crawler means nothing if the server itself blocks or challenges that crawler's requests.
These three failures are independent. A site can have a wide-open robots.txt and still be invisible to AI because its content is rendered client-side. A site can have perfectly server-rendered content and still fail because its robots.txt blocks the crawler. Test all three.
Test 1 — can the crawler reach your site at all?
This is the most basic check. Before worrying about rendering or robots.txt, confirm the site responds to a plain, unauthenticated browser request. Test crawler user-agents separately in Test 2.
- Load your homepage and a few interior pages in a private/incognito browser window with no login session, to rule out anything gated behind an account.
- Check that the page loads over HTTPS without certificate warnings — a broken certificate can cause automated requests to fail even when browsers quietly work around it.
- Confirm the server returns a normal response rather than a security-challenge page (some bot-protection or CDN configurations issue a JavaScript challenge or CAPTCHA to any request that doesn't look like a standard browser — which includes crawlers).
If your site sits behind aggressive bot protection, that protection may be treating legitimate AI crawlers the same way it treats malicious bots. That's a server-and-firewall-level setting, separate from anything in your page content.
Test 2 — is your content in the raw HTML?
No major AI crawler runs JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot read the raw HTML only, so a page whose content is drawn by JavaScript can rank on Google and still be blank to ChatGPT. Google Gemini is the exception, because it uses Google's rendering service.
This problem may not be visible in a normal browser, because browsers run JavaScript automatically and can show a complete page even when the original HTML is nearly empty.
- Right-click your page and choose "View Page Source" (or use
curl/ a similar raw-fetch tool). Treat this as a first approximation of the response a non-JavaScript crawler may receive, not as proof of what every crawler receives. - Search that raw source for your practice's actual text: your services, your address, your provider names, your differentiators. If it's not in that raw HTML, a crawler that doesn't execute JavaScript won't see it.
- Check content loaded through interactive widgets, tabs, or "load more" sections separately; it may be JavaScript-only even when the rest of the site is server-rendered.
Then verify the response with crawler user-agents. Use a command-line HTTP client or ask your developer to request the same URL once with a standard browser user-agent and separately with GPTBot, ClaudeBot, and PerplexityBot. For each response, record the HTTP status code and save the response body; compare both with the browser-user-agent response. A different status code, challenge page, or missing key content means the server is returning a different version. This tests user-agent-specific server behavior; it does not prove that a request came from a verified crawler IP.
For a fuller explanation of why this happens on dental sites specifically and what "server-rendered" actually means in practice, see how JavaScript makes dental websites invisible to AI.
Test 3 — does your robots.txt say one thing and your server another?
robots.txt is a set of instructions, not an enforcement mechanism — it tells a well-behaved crawler what it's allowed to request, but it doesn't guarantee the server will actually honor that when the request arrives. The two can disagree.
- Open
yoursite.com/robots.txtdirectly in a browser and check whether it explicitly disallows any of the AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, and similar). - Use the status-code and response-body comparison above for each crawler user-agent — a robots.txt that says "allow" is not proof the server agrees, particularly if a CDN, WAF, or bot-management layer is configured independently of the robots.txt file.
- If you find a mismatch — robots.txt open, server blocking — that's a specific and fixable pattern. See robots.txt open but AI bots blocked for how to identify and correct it.
| What was measured | Share blocked |
|---|---|
| At least one of 13 AI and data crawlers | 12.5% of 3,929 dental practice domains |
| At least one of the crawlers used by ChatGPT, Claude or Perplexity | 8.3% of 3,929 dental practice domains |
Those numbers cover the robots.txt / access layer only. For the crawler access context and the crawlers included, see which dental websites are blocking AI crawlers. For a separate figure on structured data, see the anatomy study below.
The national declaration study behind this access test examined 6,497 readable US dental practice robots.txt files on 22 July 2026 against 13 AI and data crawlers.
| Signal | Result |
|---|---|
| robots.txt in acceptable ("OK") condition | 492 of 527 (93.4%) |
| No structured data (schema) present | 284 of 527 (53.9%) |
Citevio's dental market research is published openly under CC-BY-4.0 with a DOI, so the numbers can be checked and reused; client data is never published.
These three tests tell you whether a crawler can reach and read your pages. They do not tell you whether an engine will name your practice — we have not measured a link between passing these tests and being cited. Passing them removes a barrier; it is not a guarantee of appearing in an AI answer, and Citevio does not offer that guarantee for any site, tested or not.
What comes after an access test?
These tests answer an access question: can a crawler reach and read the page? They do not inventory the information on that page. For the separate, numbered content inventory — including what was and was not measured in 527 dental practice websites — see the dental website checklist for AI search.
Does schema markup fix this?
No, and it is not useless either — both of those answers are sold, and both are wrong. Schema does not buy a citation. What it does is restate, in a form a machine reads without guessing, information that is already visible on the page. Writing a fact into schema that does not appear on the page does not create the fact; it creates a mismatch.
That last point is the practical rule, and it is the one most often broken by plugins: the markup has to match the visible text. Hours in the schema that disagree with hours on the page give a machine two answers, which is worse than giving it one imperfect one.
Google is direct about the "special AI schema" pitch that has grown up around this: "There's also no special schema.org structured data that you need to add." There is no AI-specific vocabulary to buy or install. Ordinary, accurate markup describing a local business and its services is the whole of it.
Source: Google Search Central, AI features and your website, read 18 August 2026.
So why bother at all? Because the measured position is not "schema is optional", it is "more than half of these sites have none". In Citevio's study of 527 US dental practice websites, over half had no structured data at all and just 18 sites had it complete. Whatever schema is worth, most dental sites are not in a position to find out — and going from nothing to accurate basic markup is a small job with no downside, which is a different proposition from a plugin that promises citations.
Whether adding schema will make an engine start recommending you specifically is answered separately, and the answer is not the one plugin marketing gives, in why AI recommends some dentists. What your site platform does and does not let you control here is on what platform dental websites run on.
Common questions
Does HTTPS or uptime affect whether AI crawlers can read my site?
Yes, at a basic level. If a crawler cannot connect to your site — because it is down, times out, or has a broken SSL certificate — it cannot read anything on it, regardless of how the content is built. This is separate from the rendering and robots.txt issues covered above; it is simply whether the site answers at all when a crawler requests a page.
Does it matter whether I test the mobile or desktop version of my site?
Test the version your site actually serves to automated requests, since that is what a crawler receives. If your site serves the same HTML to mobile and desktop visitors, one check is enough. If it serves different content by device, test both versions, because a crawler could see one version and not the other.
Can a popup or cookie banner block an AI crawler from reading my page?
A popup that only appears after the page loads in a browser does not block a crawler, because crawlers read the raw HTML rather than interacting with the page the way a visitor does. What can cause a problem is a consent wall or interstitial that is required server-side before the underlying content is served — in that case the crawler may receive the interstitial instead of the actual page content.
What exactly are the three things I'm testing when I check whether AI can read my site?
Access, render, and instruction — three separate questions that each fail for a different reason. Access is whether a crawler's request can reach your server at all. Render is whether your actual content sits in the raw HTML or only appears after JavaScript builds the page. Instruction is whether your robots.txt file matches what your server actually does with a crawler's request.
Can my site pass one of these three tests and still fail the others?
Yes. A site can have a wide-open robots.txt and still be invisible to AI because its content is rendered client-side, or it can have perfectly server-rendered content and still fail because robots.txt blocks the crawler. The three failures are independent, which is why testing only one gives a false sense of security.
Should I test my site while logged in, or in a private/incognito window?
Test in a private or incognito browser window with no login session. Loading your homepage and a few interior pages this way rules out anything that's accidentally gated behind an account, which is closer to how an anonymous crawler would encounter the site.
Could a broken SSL certificate stop an AI crawler even if my site loads fine in a normal browser?
Yes. A broken certificate can cause automated requests to fail even when a browser quietly works around it and loads the page anyway. Confirming your site loads over HTTPS without certificate warnings is one of the most basic checks, done before worrying about rendering or robots.txt.
What does it mean if my site shows a CAPTCHA or "verify you're human" screen to an automated request?
It usually means your bot-protection or CDN configuration is issuing a security challenge to any request that doesn't look like a standard browser — and that includes AI crawlers. If that's happening, it's a server-and-firewall-level setting, separate from anything in your actual page content.
How do I check whether my page's actual content is sitting in the raw HTML?
Right-click the page and choose "View Page Source" (or use curl or a similar raw-fetch tool), then search that raw source for your practice's actual text — services, address, provider names, differentiators. If it isn't in that raw HTML, a crawler that doesn't execute JavaScript won't see it.
If I can read my content by viewing page source, does that prove every AI crawler can see it?
No. Viewing page source is only a first approximation of what a non-JavaScript crawler may receive, not proof of what every crawler actually gets. To confirm, you need to request the same URL with the actual crawler user-agents — GPTBot, ClaudeBot, PerplexityBot — and compare the response.
Are tabs, "load more" sections, or interactive widgets tested the same way as the rest of my page?
Check them separately. Content loaded through interactive widgets, tabs, or "load more" sections may be JavaScript-only even when the rest of the site is server-rendered, so passing the raw-HTML test on your main content doesn't mean everything on the page passed it.
How do I test what GPTBot or ClaudeBot specifically receive from my server?
Use a command-line HTTP client, or ask your developer, to request the same URL once with a standard browser user-agent and separately with GPTBot, ClaudeBot, and PerplexityBot. Record the HTTP status code and save the response body for each, then compare them against the browser-user-agent response.
Does testing with a crawler's user-agent name prove the request actually came from that crawler?
No. This method tests user-agent-specific server behavior — how your server treats a request that claims to be GPTBot or ClaudeBot — but it does not prove a request came from a verified crawler IP address.
Where do I find my site's robots.txt file to check it myself?
Open yoursite.com/robots.txt directly in a browser. Check whether it explicitly disallows any of the AI crawler user-agents, such as GPTBot, ClaudeBot, or PerplexityBot.
Does passing all three tests guarantee an AI engine will mention my practice?
No. These tests tell you whether a crawler can reach and read your pages — they do not tell you whether an engine will name your practice, and no link between passing them and being cited has been measured. Passing them removes a barrier; it isn't a guarantee of appearing in an AI answer.
What share of the dental practice websites Citevio studied had their robots.txt in acceptable condition?
492 of 527 (93.4%) had robots.txt in acceptable ("OK") condition, according to Citevio's dental practice web anatomy study covering 6–26 June 2026. That study covered 527 practices across 7 US metros plus 9 smaller nearby towns.
What happens if I put a fact into schema markup that isn't written anywhere on my visible page?
It creates a mismatch, not a fact. Schema is meant to restate, in a form a machine reads without guessing, information that's already visible on the page — writing something into schema that the page itself doesn't say doesn't make it true, it just gives a crawler two conflicting signals.
Do I need a special AI-specific schema type that ChatGPT looks for?
No. Google's own documentation states there's no special schema.org structured data you need to add for AI features — no AI-specific vocabulary to buy or install. Ordinary, accurate markup describing a local business and its services is the whole of it.
Do I need to create an llms.txt file for AI engines to find my practice?
No. Google's documentation is direct on this: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." A file like that can be maintained for other reasons, but it isn't a Google visibility measure and shouldn't be sold as one.
Should each treatment I offer have its own separate page, or is one combined page fine?
We don't know — that specific question hasn't been measured. Whether a separate page per treatment performs better than one combined page is one of several questions this research takes no position on, and any confident answer to it elsewhere in the industry is an opinion, not a finding.
Sources: OpenAI crawler documentation; Anthropic crawler documentation; Perplexity crawler documentation; Google Search Central rendering documentation; Citevio dental practice web anatomy study, 6–26 June 2026 (own data, methodology: how we measure AI visibility).
For anyone repeating these checks, the supporting archive is citable as DOI 10.5281/zenodo.23143214.