{"id":"9a62a383-f60f-4b52-ab96-29b7a5143633","task":"process very large JSON files with jq using --stream/fromstream instead of loading the whole document into memory","domain":"jqlang.org","steps":["Add --stream so jq parses the input incrementally and emits [path, leaf-value] / [path] event pairs as it goes, instead of waiting for the entire document to parse before producing output.","Reconstruct whole top-level values from the raw stream events using fromstream(...) combined with truncate_stream(inputs) at the desired path depth, e.g. `jq --stream 'fromstream(1|truncate_stream(inputs))'` to get one JSON value per top-level array element.","Use tostream (the inverse of fromstream) when converting an in-memory value back into streamed event form while composing custom stream-processing filters.","Filter events by path before reconstruction to avoid materializing parts of the document that aren't needed.","For newline-delimited JSON (one record per line), prefer plain `jq -c` processed line by line instead of --stream, since each line is already a small independent document."],"gotchas":["Streaming mode's raw output (path/leaf-value pairs) is not directly usable as ordinary JSON; nearly every --stream use case requires piping through fromstream/truncate_stream, and skipping this step produces confusing low-level path arrays.","--stream trades memory efficiency for CPU overhead and more complex filter logic; for files that comfortably fit in memory, ordinary non-streaming jq filters are simpler and often faster to write correctly.","The truncation depth passed to truncate_stream must match the nesting level of the values being reconstructed; an incorrect depth yields fragments of values or several nested values merged incorrectly."],"contributor":"waymark-seed","created":"2026-07-08T16:31:32.019Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/9a62a383-f60f-4b52-ab96-29b7a5143633"}