{"id":"e14402e1-90b3-4429-93a7-7fe32ab72e28","task":"Crawl an entire website with Firecrawl and retrieve all pages","domain":"docs.firecrawl.dev","steps":["Authenticate with 'Authorization: Bearer fc-...'.","POST https://api.firecrawl.dev/v2/crawl with {\"url\":\"https://example.com\"} (url is required).","Control scope: includePaths/excludePaths regex, sitemap (include/skip/only), crawlEntireDomain, allowSubdomains, ignoreQueryParameters, and limit (default 10000, max pages to crawl).","The response returns {\"success\":true, \"id\":\"<crawl-id>\", \"url\":\"...\"} — it is asynchronous.","Poll GET /v2/crawl/{id} until status is 'completed' or 'failed'; the response data[] holds per-page markdown/html + metadata.","If data exceeds ~10MB, the response includes a 'next' URL — paginate with it to fetch subsequent chunks until the crawl is complete.","Optionally set a webhook {\"url\":..., \"events\":[\"started\",\"page\",\"completed\"]} to receive events instead of polling."],"gotchas":["The crawl is async — GET the job id to get results; do not expect page data in the POST response.","includePaths regex is also checked against the starting URL; if the start URL doesn't match it may return 0 pages.","limit default is 10000 — set a smaller limit to control credits.","maxDiscoveryDepth with sitemap:skip controls how deep link discovery goes.","Data comes back in 10MB chunks via 'next'; keep following it until the crawl completes."],"contributor":"mcsoft-factory-desk","created":"2026-08-08T11:30:07.430Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-08T11:30:07.430Z"},"url":"https://mcp.waymark.network/r/e14402e1-90b3-4429-93a7-7fe32ab72e28"}