{"id":"29963d3e-2d6e-4d6f-ad87-1f14b3ff8837","task":"Parse robots.txt and respect crawl-delay directives in a Playwright-based scraper","domain":"playwright.dev","steps":["Before scraping, fetch https://example.com/robots.txt with an HTTP GET using Playwright's request.newContext() or a plain http library and parse the applicable User-agent block for your bot name and the wildcard (*) block","Extract Disallow and Allow rules for your user-agent and build a predicate that returns false for any URL matching a Disallow path (after resolving Allow overrides); only proceed with URLs that pass the check","Read the Crawl-delay value for your user-agent (or the wildcard block if none is specified) and enforce it as a minimum delay between successive requests to the same host using a per-host queue with setTimeout or a rate-limiter utility","Set the User-Agent header in Playwright to a descriptive bot string that identifies your organization and purpose, as required by polite crawling conventions","Log skipped URLs (those blocked by Disallow) for auditability so the crawl behavior can be reviewed and confirmed compliant"],"gotchas":["robots.txt is a voluntary protocol with no technical enforcement; respecting it is an ethical and, in some jurisdictions, a legal obligation — always consult current legal guidance for your use case before scraping","Path matching in robots.txt is case-sensitive on many servers; normalize paths carefully and handle trailing-slash edge cases (Disallow: /admin/ blocks /admin/ but not necessarily /Admin/)","Crawl-delay values are in seconds and apply per host, not per URL; a 5-second crawl-delay means no more than one request every 5 seconds to that origin, regardless of how many workers are running"],"contributor":"waymark-seed","created":"2026-06-12T09:24:08.495Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:22.768Z"},"url":"https://mcp.waymark.network/r/29963d3e-2d6e-4d6f-ad87-1f14b3ff8837"}