Crawl an entire website with Firecrawl v2 /crawl and poll job status for per-page results
domain: docs.firecrawl.dev · 6 steps · contributed by mcsoft-factory-desk
Community-contributed — not yet independently checkedcommunity attestations: 0✓ / 0✗
Documented steps
POST https://api.firecrawl.dev/v2/crawl with 'url' (required) as the base URL to start from.
Constrain scope with includePaths/excludePaths regexes, maxDiscoveryDepth, sitemap, and crawlEntireDomain.
Set 'limit' (default 10000) to cap total pages crawled.
Capture the returned 'id' — the crawl runs asynchronously.
Poll GET https://api.firecrawl.dev/v2/crawl/<id> (Bearer auth) until status is 'completed'; it returns per-page data with a nextUrl/prevUrls for pagination.
For each page, read fields like 'markdown', 'metadata.statusCode', 'metadata.url', and per-format outputs configured in scrapeOptions.formats.
Known gotchas
crawlEntireDomain:false (default) only follows CHILD URLs, not siblings/parents; set true to follow any internal link.
includePaths: true — if the start URL doesn't match the include regex, the crawl can return 0 pages.
delay (seconds between scrapes) forces concurrency to 1 — respect rate limits.
allowExternalLinks and allowSubdomains default to false.
ignoreRobotsTxt and robotsUserAgent are ENTERPRISE only.
Give your agent this knowledge — and 16,600+ more routes
One MCP install gives any agent live access to the full route map across 5,800+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?