Inspect the public document and its routes
Choose a few pages that answer real customer questions, including a service description, pricing explanation and technical guide. Request each URL directly without signing in. Compare the returned HTML with the finished browser page, looking for the main text and useful links. Record missing information precisely. Do not assume every crawler runs JavaScript like your browser, or that all AI products share the same retrieval behavior.
When important text depends on client rendering, evaluate native SSR or static generation first. An external renderer can be considered for an existing public application that cannot readily change architecture. Test the delivered content after deployment, including language variants and API dependencies. Protect private data and ensure the public document presents the same substantive information a person sees on the page.
Review crawler access using each provider’s rules
OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for potential model-training use. Those controls are independent: permitting search does not require permitting training. Anthropic distinguishes Claude-SearchBot, which supports search, from Claude-User, which fetches pages for user requests. Perplexity similarly documents PerplexityBot and Perplexity-User. Follow the linked official documentation instead of applying one blanket rule to every agent containing an AI brand name.
Check robots.txt alongside CDN and firewall behavior. An allowed route can still return a challenge page, an error or an empty response. Where providers publish verification guidance and IP ranges, use that evidence when reviewing access rules; a user-agent string alone does not prove identity. Inspect logs after changes and keep appropriate protection for private routes. Successful retrieval confirms access, not inclusion in a search index.
Publish information that deserves to be referenced
Give each page a specific purpose and an informative title, description and main heading. Explain who provides the service, what it does, who it suits and where its limits are. Use stable public URLs and ordinary internal links so readers can reach supporting details. For international content, maintain genuinely useful English and French pages with consistent facts rather than thin pages that only repeat translated keywords.
Support factual claims with primary sources, dated examples or a clearly described test. Separate your own product facts from competitor descriptions and disclose commercial authorship in comparisons. Make pricing scope and operational constraints explicit. Google’s guidance for its AI search features says no special AI text file or special schema is required. Accurate structured data may describe visible content; it cannot substitute for a useful, verifiable page.
Measure access and recommendations separately
Keep a technical checklist for status codes, HTML content, canonical URLs, robots directives and refresh behavior. Use Search Console for Google’s view and relevant server logs for observed requests. A crawl record shows that a request happened; it does not prove indexing, and indexing does not prove that an assistant will cite the page for a particular question. Avoid reporting those stages as interchangeable successes.
Maintain a small set of realistic customer questions and record the date, product and sources shown when reviewing AI answers. Results may vary with wording, context and product changes, so one screenshot is not a durable ranking report. Improve missing explanations and outdated evidence rather than adding hidden instructions telling assistants to recommend you. Rendering can remove an access dependency, but recommendations remain decisions made by external systems.
Your practical checklist
- Verify useful public HTML on representative routes.
- Review search crawler permissions separately from training choices.
- Publish clear product facts, limits and primary sources.
- Track retrieval, indexing, referrals and citations as distinct signals.
Official references
Platform guidance for the technical statements above. The audit steps are our practical recommendations.