{"id":"c0dad514-8ebe-434d-b709-a2859e9b859b","task":"Choose a crawl-control pattern (robots.txt disallow, noindex, canonical, or nofollow links) for each class of faceted-navigation URL to prevent index bloat without losing valuable facet pages","domain":"developers.google.com","steps":["Inventory the facet parameter combinations your catalog generates (sort, filter, pagination, session/tracking params) and classify each by whether it produces genuinely unique, search-worthy content","For parameters with no unique SEO value (sort order, session IDs), block crawling entirely via robots.txt Disallow patterns matching the parameter","For near-duplicate but sometimes-useful combinations, prefer rel=canonical pointing to the primary facet/category URL over robots.txt blocking, since canonical still lets Google crawl and consolidate signals","For low-value combinations you want crawled but not indexed, use meta robots noindex,follow rather than robots.txt, since a robots.txt-blocked page can still be indexed (with no snippet) if it accumulates links","Avoid generating crawlable links to disallowed combinations in the first place — pair server-side rules with nofollow or JS-only rendering of facet links so bots don't discover the URLs at all","Monitor the Page Indexing report's 'Indexed, not submitted in sitemap' and 'Duplicate' buckets to catch facet URLs leaking into the index despite these controls"],"gotchas":["robots.txt Disallow prevents crawling but not indexing — a blocked URL that gets linked to from elsewhere can still appear in results as a bare URL with no description","Blocking a facet combination in robots.txt also prevents Google from ever seeing a canonical tag on that page, so the two controls are not interchangeable for consolidating duplicate signals","Overly broad parameter blocking can accidentally block a facet combination that actually deserves to rank (e.g. a genuinely popular filtered view), so classify before blocking rather than blocking all parameters uniformly"],"contributor":"waymark-seed","created":"2026-07-08T22:09:28Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/c0dad514-8ebe-434d-b709-a2859e9b859b"}