firecrawl.dev

Authenticate to the Firecrawl API and handle 429 rate limits, 402 credit exhaustion, and concurrency caps correctly

Run agent-driven structured extraction across many pages with Firecrawl Extract, then poll for results (POST /v2/extract)

Receive Firecrawl crawl/batch results via webhooks instead of polling, with HMAC signature verification

Cancel a running Firecrawl crawl and inspect failed or robots-blocked pages (DELETE /v2/crawl/{id}, GET /v2/crawl/{id}/errors)

Poll a Firecrawl crawl job for status and drain paginated results (GET /v2/crawl/{id})

Start a scoped full-site crawl with Firecrawl, by explicit rules or natural-language prompt (POST /v2/crawl)

Extract structured JSON from a page with a schema or prompt using Firecrawl's json format (v2 scrape)

Search the web and optionally scrape the results' full content in one Firecrawl call (POST /v2/search)

Enumerate a website's URLs quickly without scraping content using Firecrawl Map (POST /v2/map)

Batch scrape many URLs asynchronously with Firecrawl and poll for completion (POST /v2/batch/scrape)

Scrape a page that needs browser interaction first (click, scroll, wait, type) using Firecrawl scrape actions

Scrape a single URL into clean markdown/HTML/screenshot output with the Firecrawl v2 API (POST /v2/scrape)