{"id":"0d8486c1-4a1f-4e9b-9d98-85a67d79c518","task":"Configure permitted automation to operate transparently within a site's published rate limits instead of evading bot detection","domain":"playwright.dev","steps":["Read the target site's robots.txt and terms of service first, and only automate paths and rates it explicitly allows","Identify the client honestly — keep a real, descriptive User-Agent rather than spoofing a different browser to hide automated traffic","Throttle request concurrency and add delays to stay within any published rate limit or crawl-delay directive, instead of rotating IPs/proxies to defeat limits","Prefer an official API or data feed over scraping HTML when the site offers one, since that's the intended automated-access path","Honor 429/Retry-After responses by backing off, rather than retrying immediately with a different fingerprint"],"gotchas":["Anti-bot vendors treat User-Agent spoofing, fingerprint randomization, and headless-flag masking as evasion signals — using them turns otherwise-permitted automation into a ToS violation","Rate limits are often per-IP or per-account; running many parallel browser contexts from one IP can still trip limits designed around request rate, not context count","Ethical/permitted automation is fundamentally a policy question, not a technical one — no header or timing configuration substitutes for actual permission from the site operator"],"contributor":"waymark-seed","created":"2026-07-08T22:09:28Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/0d8486c1-4a1f-4e9b-9d98-85a67d79c518"}