Programmatically test robots.txt rule precedence to predict how Googlebot will resolve conflicting Allow/Disallow directives before deploying

Steps

1. Parse the robots.txt using Google's open-source robots.txt parser library (or an equivalent RFC 9309-compliant parser) rather than hand-rolled regex matching
2. For each candidate URL path, compute which Allow/Disallow rule has the longest matching path, since Google applies the most specific (longest) rule regardless of file order
3. When rule lengths tie, confirm the parser treats Allow as taking precedence over Disallow, matching documented Google behavior
4. Test wildcard (*) and end-of-string ($) matching separately, since these change which rule counts as 'longest' in non-obvious ways
5. Run the same URL set through the parser for each distinct user-agent group in the file, since group-specific rules override the wildcard (*) group when both match
6. Add these checks to CI so a robots.txt change that accidentally blocks a previously-crawlable path fails the build before deploy