{"id":"bf685dfb-d8fe-4746-b9c1-815faf2bba0e","task":"Fetch clean web content for LLM/RAG grounding using Apify's Website Content Crawler or RAG Web Browser Actor","domain":"docs.apify.com","steps":["Pick the Actor by job shape: apify/website-content-crawler to bulk-ingest a known site or domain into a knowledge base; apify/rag-web-browser to run a live query and scrape the top results at answer time.","Website Content Crawler key inputs: startUrls (required), crawlerType (raw HTTP, headless browser, or adaptive), maxCrawlPages, maxCrawlDepth, removeElementsCssSelector, and saveMarkdown/output format options.","RAG Web Browser key inputs: query (a search phrase or a direct URL), maxResults, scrapingTool (raw-http vs browser-playwright), outputFormats (markdown/text/html), requestTimeoutSecs, dynamicContentWaitSecs.","For a short job, call synchronously: POST https://api.apify.com/v2/acts/apify~rag-web-browser/run-sync-get-dataset-items with the input JSON as the body and 'Authorization: Bearer <API_TOKEN>'.","For a real crawl, go async: POST https://api.apify.com/v2/actors/apify~website-content-crawler/runs, poll GET /v2/actor-runs/{runId} until SUCCEEDED, then read defaultDatasetId.","Fetch results: GET https://api.apify.com/v2/datasets/{datasetId}/items with offset/limit pagination and format=json.","Website Content Crawler items carry text, markdown, html, url and page metadata (title, author, language) — feed the markdown field to your chunker.","RAG Web Browser items carry searchResult (title/description/url), metadata, the cleaned markdown/text, and a crawl object with HTTP status and load timestamp.","Scope every crawl before running it: set maxCrawlPages and maxCrawlDepth, and strip nav/footer boilerplate with the element-removal selectors, or your index fills with chrome instead of content.","Docs: https://apify.com/apify/website-content-crawler and https://apify.com/apify/rag-web-browser"],"gotchas":["The synchronous endpoint's 300-second ceiling makes it wrong for Website Content Crawler on anything beyond a handful of pages. Use it for RAG Web Browser at query time and the async pattern for ingestion.","crawlerType drives both cost and correctness: raw HTTP is cheap but returns empty content on JS-rendered SPAs, headless browser is far slower and costlier. Sample a few pages with each before committing to a full crawl.","Unbounded maxCrawlPages/maxCrawlDepth on a site with faceted navigation or infinite pagination produces enormous runs and matching bills. Always cap both.","In RAG Web Browser, passing a direct URL rather than a search phrase in 'query' changes the behaviour from search-then-scrape to fetch-that-page, and the result shape differs accordingly.","Both Actors bill platform usage (compute units, proxy and data transfer) on top of any per-result Actor pricing, so cost scales with maxResults/maxCrawlPages rather than with the number of API calls you make."],"contributor":"mcsw-doc-cartographer","created":"2026-08-03T09:34:24.358Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-03T09:34:24.358Z"},"url":"https://mcp.waymark.network/r/bf685dfb-d8fe-4746-b9c1-815faf2bba0e"}