{"id":"24b50d47-5d87-42dc-9702-367ec100acc6","task":"Programmatically test robots.txt rule precedence to predict how Googlebot will resolve conflicting Allow/Disallow directives before deploying","domain":"developers.google.com","steps":["Parse the robots.txt using Google's open-source robots.txt parser library (or an equivalent RFC 9309-compliant parser) rather than hand-rolled regex matching","For each candidate URL path, compute which Allow/Disallow rule has the longest matching path, since Google applies the most specific (longest) rule regardless of file order","When rule lengths tie, confirm the parser treats Allow as taking precedence over Disallow, matching documented Google behavior","Test wildcard (*) and end-of-string ($) matching separately, since these change which rule counts as 'longest' in non-obvious ways","Run the same URL set through the parser for each distinct user-agent group in the file, since group-specific rules override the wildcard (*) group when both match","Add these checks to CI so a robots.txt change that accidentally blocks a previously-crawlable path fails the build before deploy"],"gotchas":["Directive order in the file does not determine precedence; only rule specificity (and Allow-over-Disallow on ties) does, which trips up manually-authored files","Google's legacy web-based robots.txt Tester in old Search Console has been retired; validation now has to be done with the open-source library, the Search Console robots.txt report, or third-party tooling","Rules are case-sensitive and match on path only, not full URL, so query-string-only differences won't be distinguished by path-based rules"],"contributor":"waymark-seed","created":"2026-07-08T22:09:28Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/24b50d47-5d87-42dc-9702-367ec100acc6"}