Documented steps

Start the job: POST https://api.hyperbrowser.ai/api/crawl with 'x-api-key' and JSON body {"url": "https://example.com", "maxPages": 10, "followLinks": true}. Response is {"jobId": "<uuid>"}.
Crawl options and defaults: maxPages (default 10, max 100), followLinks (default true), ignoreSitemap (default false); scrapeOptions (formats/onlyMainContent/timeout) apply per crawled page.
Check status: GET https://api.hyperbrowser.ai/api/crawl/{jobId}/status -> status is one of pending/running/completed/failed/stopped.
Fetch results: GET https://api.hyperbrowser.ai/api/crawl/{jobId} returns jobId, status, totalCrawledPages/totalPageBatches, currentPageBatch, batchSize, and a data array of per-page objects {url, status, error, metadata, markdown, html, links}.
Paginate large crawls with ?page=<n>&batchSize=<n> query params, looping until currentPageBatch reaches totalPageBatches (or data returns empty).
SDK shortcut: client.crawl.startAndWait({ url, maxPages, followLinks }) handles start+poll+fetch and returns the aggregated result.
Ref: https://docs.hyperbrowser.ai/web-scraping/crawl

Known gotchas

Each crawled page carries its own status/error inside the data array even when the overall job status is 'completed' - check per-page status before trusting content.
maxPages caps at 100 per job; larger sites need multiple crawl jobs.
sessionOptions.useProxy/solveCaptchas require a paid plan and slow the whole crawl.
'stopped' is a valid terminal status in addition to completed/failed - your polling loop must treat it as done, not keep waiting.
onlyMainContent defaults to true for every crawled page just like scrape - set it false if you need full page chrome.

google-search-console · 5 steps · unrated

Paginate large SOQL query results using nextRecordsUrl

developer.salesforce.com · 5 steps · unrated

Paginate through a large FHIR search result Bundle using next-page links

fhir · 6 steps · unrated

Give your agent this knowledge — and 16,400+ more routes

One MCP install gives any agent live access to the full route map across 5,800+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Crawl a multi-page site with the Hyperbrowser Crawl API and paginate the results

Documented steps

Known gotchas

Related routes

Give your agent this knowledge — and 16,400+ more routes

Need this verified for your stack — or a route we don't have yet?