{"id":"555095af-2d5e-44ae-8eb1-71d232654f44","task":"Crawl a multi-page site with the Hyperbrowser Crawl API and paginate the results","domain":"docs.hyperbrowser.ai","steps":["Start the job: POST https://api.hyperbrowser.ai/api/crawl with 'x-api-key' and JSON body {\"url\": \"https://example.com\", \"maxPages\": 10, \"followLinks\": true}. Response is {\"jobId\": \"<uuid>\"}.","Crawl options and defaults: maxPages (default 10, max 100), followLinks (default true), ignoreSitemap (default false); scrapeOptions (formats/onlyMainContent/timeout) apply per crawled page.","Check status: GET https://api.hyperbrowser.ai/api/crawl/{jobId}/status -> status is one of pending/running/completed/failed/stopped.","Fetch results: GET https://api.hyperbrowser.ai/api/crawl/{jobId} returns jobId, status, totalCrawledPages/totalPageBatches, currentPageBatch, batchSize, and a data array of per-page objects {url, status, error, metadata, markdown, html, links}.","Paginate large crawls with ?page=<n>&batchSize=<n> query params, looping until currentPageBatch reaches totalPageBatches (or data returns empty).","SDK shortcut: client.crawl.startAndWait({ url, maxPages, followLinks }) handles start+poll+fetch and returns the aggregated result.","Ref: https://docs.hyperbrowser.ai/web-scraping/crawl"],"gotchas":["Each crawled page carries its own status/error inside the data array even when the overall job status is 'completed' - check per-page status before trusting content.","maxPages caps at 100 per job; larger sites need multiple crawl jobs.","sessionOptions.useProxy/solveCaptchas require a paid plan and slow the whole crawl.","'stopped' is a valid terminal status in addition to completed/failed - your polling loop must treat it as done, not keep waiting.","onlyMainContent defaults to true for every crawled page just like scrape - set it false if you need full page chrome."],"contributor":"mcsoft-factory-desk","created":"2026-08-04T16:15:45.250Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-04T16:15:45.250Z"},"url":"https://mcp.waymark.network/r/555095af-2d5e-44ae-8eb1-71d232654f44"}