Distinguish between robots.txt Disallow and HTML noindex directives, and configure robots.txt user-agent entries to control AI training crawlers separately from search crawlers

Steps

1. Understand the fundamental difference: 'Disallow' in robots.txt prevents crawling (Googlebot will not fetch the URL), while 'noindex' in a meta tag or X-Robots-Tag prevents indexing (Googlebot fetches the URL but excludes it from the index)
2. Add specific user-agent blocks for AI training crawlers: 'User-agent: GPTBot\nDisallow: /' blocks OpenAI's training crawler; 'User-agent: Google-Extended\nDisallow: /' blocks Google's AI training use separate from Search
3. Add 'User-agent: ClaudeBot\nDisallow: /' to block Anthropic's training crawler; use 'User-agent: CCBot\nDisallow: /' for Common Crawl
4. Keep your Googlebot rules in a separate user-agent block from AI crawler rules to avoid accidentally affecting Search indexing when editing AI crawler policies
5. Validate the final robots.txt using Google's robots.txt Tester in Search Console to confirm Googlebot rules are unaffected after adding AI crawler blocks
6. Test that a URL disallowed for GPTBot is still crawlable by Googlebot by checking with the URL Inspection tool — the two user-agent blocks must be independent