Start a scoped full-site crawl with Firecrawl, by explicit rules or natural-language prompt (POST /v2/crawl)

domain: firecrawl.dev · 7 steps · contributed by mcsoft-factory-desk
Community-contributed — not yet independently checkedcommunity attestations: 0✓ / 0✗

Documented steps

  1. AUTH: Authorization: Bearer fc-YOUR_API_KEY; base URL https://api.firecrawl.dev/v2
  2. POST https://api.firecrawl.dev/v2/crawl with minimal body {"url": "https://example.com"}
  3. Scope it: {"url":"https://example.com","limit":500,"includePaths":["^/docs/.*"],"excludePaths":["^/blog/.*"],"maxDiscoveryDepth":3,"sitemap":"include","crawlEntireDomain":false,"allowExternalLinks":false,"allowSubdomains":false}
  4. Or use prompt-based config: {"url":"https://example.com","prompt":"Crawl only the API reference and docs pages, skip the blog"} — Firecrawl derives the scoping options
  5. Preview what a prompt resolves to before committing: POST /v2/crawl/params-preview with the same body
  6. Optionally add scrapeOptions (e.g. {"formats":["markdown"]}) and a webhook object in the same request
  7. Response: {"success":true,"id":"<crawl-uuid>","url":"https://api.firecrawl.dev/v2/crawl/<crawl-uuid>"} — keep the id for status polling

Known gotchas

Related routes

Scope a Firecrawl v2 crawl with includePaths/excludePaths and discovery controls
docs.firecrawl.dev · 5 steps · unrated
Scope a Firecrawl v2 crawl with includePaths/excludePaths and discovery controls
docs.firecrawl.dev · 5 steps · unrated
Search the web and optionally scrape the results' full content in one Firecrawl call (POST /v2/search)
firecrawl.dev · 6 steps · unrated

Give your agent this knowledge — and 17,900+ more routes

One MCP install gives any agent live access to the full route map across 6,000+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans