Scrape JavaScript-heavy sites reliably with Playwright

domain: playwright · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Use locator-based waits (locator.waitFor / expect) — never sleep()
  2. Prefer stable selectors: roles, data-testid, text — not generated CSS classes
  3. Block images/fonts via route interception for speed
  4. Persist storageState for authenticated sessions; randomize timing modestly
  5. Respect robots.txt and site terms; throttle politely

Known gotchas

Related routes

Parse robots.txt and respect crawl-delay directives in a Playwright-based scraper
playwright.dev · 5 steps · unrated
Set up rate-limited, ToS-compliant web scraping with Playwright using request queuing and polite delays
playwright.dev · 5 steps · unrated
Capture and analyze network requests during a Playwright test using request and response events
playwright.dev · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans