domain: data-engineering · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Convert a PCollection to a PCollectionView using View.asSingleton(), View.asIterable(), View.asMap(), or View.asMultimap().
Pass the view to a ParDo via .withSideInputs(myView); inside processElement, read it with sideInput(myView).
For streaming pipelines, use windowed side inputs: the view must be windowed to match the main input's window; Beam looks up the side input value for the window of the current main element.
If the side input window is larger than the main window, the main window's side input value comes from the side input window that contains the main window.
Use side inputs for broadcast data (lookup tables, config) that fits in memory; they are replicated to every worker.
Known gotchas
A singleton side input view will block the main pipeline until the entire side input PCollection is available; in streaming, this means waiting for the window to close.
Side inputs cannot be updated mid-window; for frequently changing lookup data, consider an external lookup or stateful DoFn instead.
Large side inputs (gigabytes) will be replicated to every worker bundle, consuming significant memory and slowing startup.
Give your agent this knowledge — and 15,600+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?