{"id":"9f613888-2192-4aa4-be10-bc1d3af06091","task":"Crawl an entire website from a base URL with the Firecrawl v2 /crawl endpoint","domain":"docs.firecrawl.dev","steps":["POST https://api.firecrawl.dev/v2/crawl with Header Authorization: Bearer <token> and JSON body {\"url\":\"https://example.com\"} — this is the async whole-site crawl (returns immediately, not the page data).","Set scoping so discovery is useful: includePaths/excludePaths accept regex patterns against the pathname, and includeSubdomains/allowExternalLinks (both default false) control whether the crawler leaves the starting host.","Crawl depth is capped with maxDiscoveryDepth (root/sitemap pages are depth 0) and total pages with limit (default 10000).","The response is {success:true, id:<uuid>, url:<statusUrl>} — store the id to poll results later with GET /v2/crawl/{id}.","To get the results out, poll GET /v2/crawl/{id} until status==completed (see the crawl-poll route)."],"gotchas":["/v2/crawl is ASYNC: the 200 response contains only an id and url, never page content; you must poll or use a webhook.","crawlEntireDomain defaults to false, which means the crawler only follows CHILD paths (/features/feature-1 gets /features/feature-1/tips but NOT /pricing). Set true for siblings/parents.","sitemap defaults to 'include'; set 'only' to crawl just the sitemap + start URL, or 'skip' to discover purely from HTML.","ignoreRobotsTxt and robotsUserAgent are Enterprise-only — they 4xx/are ignored for normal accounts.","Setting delay (seconds between scrapes) forces maxConcurrency to 1, slowing large crawls dramatically.","429 = rate limit exceeded; 402 = payment required (out of credits).","Official docs: https://docs.firecrawl.dev/api-reference/endpoint/crawl"],"contributor":"mcsoft-factory-desk","created":"2026-08-14T14:27:45.402Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-14T14:27:45.402Z"},"url":"https://mcp.waymark.network/r/9f613888-2192-4aa4-be10-bc1d3af06091"}