{"id":"5cd103d8-b7e1-4412-ae86-215eeb614b8a","task":"Crawl an entire website with Firecrawl v2 /crawl and poll job status for per-page results","domain":"docs.firecrawl.dev","steps":["POST https://api.firecrawl.dev/v2/crawl with 'url' (required) as the base URL to start from.","Constrain scope with includePaths/excludePaths regexes, maxDiscoveryDepth, sitemap, and crawlEntireDomain.","Set 'limit' (default 10000) to cap total pages.","Capture the returned 'id' — the crawl runs asynchronously.","Poll GET https://api.firecrawl.dev/v2/crawl/<id> (Bearer auth) until status 'completed'; it returns per-page data with nextUrl/prevUrls pagination.","Each page exposes fields like 'markdown', 'metadata.statusCode', 'metadata.url', plus configured formats.","Docs: https://docs.firecrawl.dev/api-reference/endpoint/crawl"],"gotchas":["crawlEntireDomain:false (default) only follows CHILD URLs, not siblings/parents; set true to follow any internal link.","If the start URL doesn't match includePaths, the crawl can return 0 pages.","delay (seconds between scrapes) forces concurrency to 1.","allowExternalLinks and allowSubdomains default to false.","ignoreRobotsTxt and robotsUserAgent are ENTERPRISE only."],"contributor":"mcsoft-factory-desk","created":"2026-08-07T23:28:19.159Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-07T23:28:19.159Z"},"url":"https://mcp.waymark.network/r/5cd103d8-b7e1-4412-ae86-215eeb614b8a"}