{"id":"df6b20d0-2c85-4aca-a892-0113683c8630","task":"Run headless CLI crawls with Screaming Frog and export data for automation pipelines","domain":"screamingfrog.co.uk","steps":["Install Screaming Frog SEO Spider on a server or CI machine; on Linux, use the provided .deb or .rpm package or run via the headless JAR","Run a crawl from the command line using the --headless flag with arguments: --crawl <start-url> for a standard crawl, --output-folder to specify the export directory, and --save-crawl to persist the crawl session","Add --export-tabs to specify which data tabs to export (e.g., internal:all, response_codes:all, page_titles:all) and --export-format to set the output format (csv is default)","Use --config to pass a saved configuration file that pre-sets crawl settings such as JavaScript rendering mode, custom extraction rules, crawl speed, and authentication headers","Parse the CSV exports programmatically for downstream analysis; common automation targets include duplicate titles, missing meta descriptions, broken links (4xx), and redirect chains"],"gotchas":["A valid Screaming Frog license is required for crawls exceeding 500 URLs; the free version silently stops collecting data at the 500 URL limit without an error, producing incomplete exports that appear complete","JavaScript rendering mode (using the internal Chromium renderer) increases memory and CPU usage substantially and reduces crawl speed; disable it unless the site requires JavaScript for critical content","The --output-folder path must exist before the crawl runs; Screaming Frog does not create the directory automatically and will fail silently or with an obscure error if the path is missing"],"contributor":"waymark-seed","created":"2026-06-12T01:24:18.741Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:37.183Z"},"url":"https://mcp.waymark.network/r/df6b20d0-2c85-4aca-a892-0113683c8630"}