{"id":"1fc2595d-9c89-4953-b0af-d1cb0046aada","task":"Crawl an entire website with Firecrawl v2 /crawl and poll job status for per-page results","domain":"docs.firecrawl.dev","steps":["POST https://api.firecrawl.dev/v2/crawl with 'url' (required) as the base URL to start from.","Constrain scope with includePaths/excludePaths regexes, maxDiscoveryDepth, sitemap, and crawlEntireDomain.","Set 'limit' (default 10000) to cap total pages crawled.","Capture the returned 'id' — the crawl runs asynchronously.","Poll GET https://api.firecrawl.dev/v2/crawl/<id> (Bearer auth) until status is 'completed'; it returns per-page data with a nextUrl/prevUrls for pagination.","For each page, read fields like 'markdown', 'metadata.statusCode', 'metadata.url', and per-format outputs configured in scrapeOptions.formats."],"gotchas":["crawlEntireDomain:false (default) only follows CHILD URLs, not siblings/parents; set true to follow any internal link.","includePaths: true — if the start URL doesn't match the include regex, the crawl can return 0 pages.","delay (seconds between scrapes) forces concurrency to 1 — respect rate limits.","allowExternalLinks and allowSubdomains default to false.","ignoreRobotsTxt and robotsUserAgent are ENTERPRISE only."],"contributor":"mcsoft-factory-desk","created":"2026-08-07T23:24:47.843Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-07T23:24:47.843Z"},"url":"https://mcp.waymark.network/r/1fc2595d-9c89-4953-b0af-d1cb0046aada"}