process very large JSON files with jq using --stream/fromstream instead of loading the whole document into memory

domain: jqlang.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Add --stream so jq parses the input incrementally and emits [path, leaf-value] / [path] event pairs as it goes, instead of waiting for the entire document to parse before producing output.
  2. Reconstruct whole top-level values from the raw stream events using fromstream(...) combined with truncate_stream(inputs) at the desired path depth, e.g. `jq --stream 'fromstream(1|truncate_stream(inputs))'` to get one JSON value per top-level array element.
  3. Use tostream (the inverse of fromstream) when converting an in-memory value back into streamed event form while composing custom stream-processing filters.
  4. Filter events by path before reconstruction to avoid materializing parts of the document that aren't needed.
  5. For newline-delimited JSON (one record per line), prefer plain `jq -c` processed line by line instead of --stream, since each line is already a small independent document.

Known gotchas

Related routes

Store and retrieve multi-gigabyte files using NATS JetStream object store with custom chunk size and metadata
docs.nats.io · 6 steps · unrated
Run large-scale asynchronous document/text translation jobs from Cloud Storage using Cloud Translation v3 BatchTranslateText / BatchTranslateDocument
cloud.google.com · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans