How do I test whether AI can actually read my dental website?

A normal browser view does not show every condition an automated crawler may encounter. These three tests help you check whether the server responds, whether key content is in the raw HTML, and whether the server treats crawler requests differently.

What exactly are you testing for?

"Can AI read my site" is really three separate questions, and each one fails for a different reason. A site can pass one test and fail another — so testing only one gives you a false sense of security.

• Access — can the crawler's request reach your server and get a response, rather than a timeout, an error, or a firewall block?
• Render — once the crawler gets a response, is your actual page content sitting in that raw HTML, or does it only appear after JavaScript builds the page in a browser?
• Instruction — does your robots.txt file actually match what your server does? A robots.txt that allows a crawler means nothing if the server itself blocks or challenges that crawler's requests.

These three failures are independent. A site can have a wide-open robots.txt and still be invisible to AI because its content is rendered client-side. A site can have perfectly server-rendered content and still fail because its robots.txt blocks the crawler. Test all three.

Test 1 — can the crawler reach your site at all?

This is the most basic check. Before worrying about rendering or robots.txt, confirm the site responds to a plain, unauthenticated browser request. Test crawler user-agents separately in Test 2.

1. Load your homepage and a few interior pages in a private/incognito browser window with no login session, to rule out anything gated behind an account.
2. Check that the page loads over HTTPS without certificate warnings — a broken certificate can cause automated requests to fail even when browsers quietly work around it.
3. Confirm the server returns a normal response rather than a security-challenge page (some bot-protection or CDN configurations issue a JavaScript challenge or CAPTCHA to any request that doesn't look like a standard browser — which includes crawlers).

If your site sits behind aggressive bot protection, that protection may be treating legitimate AI crawlers the same way it treats malicious bots. That's a server-and-firewall-level setting, separate from anything in your page content.

Test 2 — is your content in the raw HTML?

This problem may not be visible in a normal browser, because browsers run JavaScript automatically and can show a complete page even when the original HTML is nearly empty.

1. Right-click your page and choose "View Page Source" (or use curl / a similar raw-fetch tool). Treat this as a first approximation of the response a non-JavaScript crawler may receive, not as proof of what every crawler receives.
2. Search that raw source for your practice's actual text: your services, your address, your provider names, your differentiators. If it's not in that raw HTML, a crawler that doesn't execute JavaScript won't see it.
3. Check content loaded through interactive widgets, tabs, or "load more" sections separately; it may be JavaScript-only even when the rest of the site is server-rendered.

Then verify the response with crawler user-agents. Use a command-line HTTP client or ask your developer to request the same URL once with a standard browser user-agent and separately with GPTBot, ClaudeBot, and PerplexityBot. For each response, record the HTTP status code and save the response body; compare both with the browser-user-agent response. A different status code, challenge page, or missing key content means the server is returning a different version. This tests user-agent-specific server behavior; it does not prove that a request came from a verified crawler IP.

For a fuller explanation of why this happens on dental sites specifically and what "server-rendered" actually means in practice, see how JavaScript makes dental websites invisible to AI.

Test 3 — does your robots.txt say one thing and your server another?

robots.txt is a set of instructions, not an enforcement mechanism — it tells a well-behaved crawler what it's allowed to request, but it doesn't guarantee the server will actually honor that when the request arrives. The two can disagree.

1. Open yoursite.com/robots.txt directly in a browser and check whether it explicitly disallows any of the AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, and similar).
2. Use the status-code and response-body comparison above for each crawler user-agent — a robots.txt that says "allow" is not proof the server agrees, particularly if a CDN, WAF, or bot-management layer is configured independently of the robots.txt file.
3. If you find a mismatch — robots.txt open, server blocking — that's a specific and fixable pattern. See robots.txt open but AI bots blocked for how to identify and correct it.

Those numbers cover the robots.txt / access layer only. For the crawler access context and the crawlers included, see which dental websites are blocking AI crawlers. For a separate figure on structured data, see the anatomy study below.

Common questions

Does HTTPS or uptime affect whether AI crawlers can read my site?

Does it matter whether I test the mobile or desktop version of my site?

Can a popup or cookie banner block an AI crawler from reading my page?

Sources: OpenAI crawler documentation; Anthropic crawler documentation; Perplexity crawler documentation; Google Search Central rendering documentation; Citevio dental practice web anatomy study, 6–26 June 2026 (own data, methodology: how we measure AI visibility).