Configure a DVC pipeline stage with an external S3 dependency and output that DVC does not cache locally

domain: doc.dvc.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Add a stage in dvc.yaml whose `deps` point to an external URL (e.g. an S3-hosted zip) instead of an in-project path
  2. Point the stage's `outs` at an external location (e.g. `s3://bucket/prefix/output`)
  3. Set `cache: false` on that external output so DVC doesn't try to copy it into local DVC cache
  4. Run `dvc repro` as usual; DVC still detects changes to the external dependency
  5. Confirm required cloud credentials/permissions are configured for DVC to reach the external storage

Known gotchas

Related routes

Build a versioned DVC pipeline (dvc.yaml) that chains data prep, featurization, and training stages with cached, reproducible runs
doc.dvc.org · 5 steps · unrated
configure an s3-backed dvc remote and push/pull tracked data
dvc.org/doc · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans