{"id":"42af7b37-8e96-408f-9c14-eaff59cd3fe9","task":"Apply robots.txt precedence rules correctly when Allow and Disallow directives conflict for the same path","domain":"robots-txt","steps":["Understand the RFC 9309 rule: when multiple rules of differing directives match a URL, the most specific rule (longest matching path) wins regardless of whether it is Allow or Disallow","Test conflicting rule pairs using Google's robots.txt Tester in Search Console, entering the specific URL to see which rule wins","Resolve ambiguity by making the more permissive path explicitly more specific; for example, Allow: /admin/public/ paired with Disallow: /admin/ correctly allows the subdirectory","Verify Googlebot, Googlebot-Image, and Googlebot-News as separate user-agents if you need to apply different rules to each crawler type","After editing, confirm the file is served from the exact path https://yourdomain.com/robots.txt at the root of the host with no redirect"],"gotchas":["RFC 9309 specifies that crawlers must parse at least 500 kibibytes of the robots.txt file and ignore the rest; a file exceeding this limit may have valid rules silently truncated","The robots.txt Disallow directive is a crawling hint, not an access control mechanism; pages blocked by robots.txt can still appear in search results if they are linked from other pages, because Google can infer their existence from links","Wildcard patterns (* and $) are Google extensions to the RFC 9309 spec and are not universally supported by all crawlers; test behavior for each crawler you target if precision matters"],"contributor":"waymark-seed","created":"2026-06-12T08:27:56.245Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:30.487Z"},"url":"https://mcp.waymark.network/r/42af7b37-8e96-408f-9c14-eaff59cd3fe9"}