Configure GitLab CI needs: keyword with DAG to eliminate sequential stage bottlenecks and fan out parallel jobs

domain: docs.gitlab.com · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Replace the default stage-based ordering by adding needs: to jobs that should run as soon as their dependencies complete, regardless of stage: job-b: stage: test; needs: [job-a] — job-b starts immediately after job-a succeeds without waiting for all stage: build jobs
  2. To receive artifacts from a dependency, specify needs with artifacts: true: needs: [{job: build-frontend, artifacts: true}] so the artifacts from that specific job are downloaded before the dependent job starts
  3. For jobs that should run without any dependency (i.e., immediately at pipeline start), set needs: [] explicitly — without this, a job in a later stage waits for all prior-stage jobs even if they are unrelated
  4. Model a fan-out/fan-in pattern: one build job produces an artifact, multiple parallel test jobs declare needs: [build] with artifacts: true, and a final report job declares needs over all test job names
  5. Validate the DAG structure with the GitLab CI Lint tool (Repository > CI/CD > CI Lint or API endpoint /api/v4/ci/lint) which visualizes the job dependency graph
  6. Set rules: with if: $CI_PIPELINE_SOURCE to conditionally include or exclude branches of the DAG without restructuring the entire pipeline YAML

Known gotchas

Related routes

Configure a GitLab CI DAG pipeline using needs: with artifacts to fan out parallel jobs and aggregate results, then use rules: with if and changes to skip unchanged components
GitLab CI · 6 steps · unrated
Configure GitLab CI parent-child pipelines where a parent pipeline dynamically generates a child pipeline YAML based on changed paths and triggers it with the artifact strategy
GitLab CI · 6 steps · unrated
Configure Airflow pools and priority weights to control concurrency and prioritize critical DAG tasks
airflow.apache.org · 6 steps · unrated

Give your agent this knowledge — and 15,600+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans