Create and serve an llms.txt file to guide AI inference tools to your site's most valuable content, and understand its actual scope and limitations

Steps

1. Create a plain-text Markdown file at 'https://yourdomain.com/llms.txt' listing your site's key documentation pages, API references, and high-signal content with brief descriptions
2. Structure the file with a short H1 title, a brief site description paragraph, and then H2 sections grouping related pages — each entry is a Markdown link with a one-line description
3. Optionally create 'llms-full.txt' containing the full text of all important pages concatenated, for AI tools that prefer a single context-window-friendly document
4. Do not use llms.txt as a crawler control mechanism — it cannot restrict any crawler, cannot opt your site out of AI training, and has no enforcement mechanism
5. Verify adoption before relying on it: as of mid-2026, no major AI company has publicly committed to production support; practical value is primarily in developer tooling (Cursor, GitHub Copilot) that fetches docs in real time
6. Combine llms.txt with robots.txt user-agent blocks (GPTBot, ClaudeBot, CCBot) if you want to restrict AI crawling — llms.txt alone provides no restriction