Building a CI/CD Pipeline That People Trust

A CI/CD pipeline people trust has five properties, and everything else in this guide is detail underneath them. It is fast enough that an engineer watches it finish instead of walking away. It is honest, so a red build always means a real defect and never a flaky test or an expired token. It is reproducible, so the same commit produces the same artifact on any runner. It is promotable, so the exact bytes that passed staging are the bytes that reach production, never a rebuild. And it is reversible, so any deploy can be undone in a couple of minutes by whoever happens to be on call, without reading a runbook they have never opened.

Most pipelines fail on honesty and reversibility long before they fail on speed. A team can live with a twelve-minute build. A team cannot live with a suite that fails one run in six for no reason, because after a month nobody reads the failure, they just press the retry button, and the pipeline has stopped being a signal. Same with rollback: a plan that exists only as a paragraph in a wiki is not a rollback plan, it is a hope.

This guide assumes you already ship something and now have to make shipping boring. If you are earlier than that, the CI/CD pipeline primer covers the vocabulary, and what DevOps actually means covers why any of this exists. Examples use GitHub Actions because it is the most common starting point, but every idea here transfers; the GitHub Actions guide goes deeper on that runner specifically.

What Makes a Pipeline Trustworthy

Trust is measurable, and the four DORA metrics are the usual measuring stick: deployment frequency, lead time from commit to production, change failure rate, and time to restore service. A pipeline that improves the first two while quietly worsening the last two has not made anything better. Speed without reversibility is just a faster way to be broken.

The internal test is simpler than the metrics. Ask three questions after your next incident. Did anyone hesitate to deploy because they were unsure they could undo it? Did anyone re-run a failed job hoping it would pass? Did anyone need a person, rather than a document, to explain how to ship? Each yes points at a specific defect, and each defect has a fix in the sections below.

CI/CD Pipeline Stages, in the Order That Matters

Sort stages by expected cost divided by probability of catching something. Formatting and type checks take seconds and fail constantly, so they go first. Integration tests take minutes and fail rarely, so they go later. The goal is that the most common failure is reported in about ninety seconds, not at minute eleven behind a queue of things that were going to pass anyway.

name: ci

on:
  push:
    branches: [main]
  pull_request:

# One run per branch. A new push cancels the old run, except on main.
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

# Least privilege by default; jobs opt into more.
permissions:
  contents: read

jobs:
  check:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: npm          # caches ~/.npm keyed on the lockfile
      - run: npm ci           # ci, not install: honours the lockfile exactly
      - run: npm run lint
      - run: npx tsc --noEmit

Two details in that file do more work than they look like they do. concurrency with cancel-in-progress stops a busy branch from occupying every runner with results nobody will read. And npm ci instead of npm install means the lockfile is authoritative, so the build cannot quietly float to a new patch version between the run that passed and the run that shipped.

Caching: Where the Minutes Actually Go

Two rules make caching safe. Cache inputs, such as downloaded packages and compiler intermediates, never outputs the build's correctness depends on. And build the key from a hash of the file that determines the content, which is the lockfile, with a shorter prefix in restore-keys so a partial hit still saves most of the download.

A cache key that is too specific never hits. A key that is too loose hits with stale content and produces a build that behaves differently from a clean one, which is worse than slow. The pattern that works everywhere is one exact key plus a ladder of prefixes: try the precise match, fall back to any cache for this lockfile, fall back to any cache for this operating system.

      - name: Cache build output
        uses: actions/cache@v4
        with:
          path: |
            .turbo
            node_modules/.cache
          # Exact: same lockfile, same commit
          key: build-${{ runner.os }}-${{ hashFiles('package-lock.json') }}-${{ github.sha }}
          # Fallbacks, tried in order, longest prefix first
          restore-keys: |
            build-${{ runner.os }}-${{ hashFiles('package-lock.json') }}-
            build-${{ runner.os }}-

Container builds have their own caching model, and the biggest win is free: order the Dockerfile so that the layers that change least often come first. Copy the manifest and lockfile, install dependencies, and only then copy the source. Change one line of application code and the dependency layer is still valid.

FROM node:22-slim AS build
WORKDIR /app

# Dependency layer: invalidated only when the lockfile changes
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

# Source layer: changes on every commit, and that is fine
COPY . .
RUN npm run build

FROM gcr.io/distroless/nodejs22-debian12
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER nonroot
CMD ["dist/server.js"]

Parallelizing Tests Without Making Them Flaky

Wall-clock time for a sharded suite is the time of the slowest shard, so balancing matters more than shard count. Split by recorded per-test duration, not by file name or file count. Most runners support this directly: Jest, Vitest, and Playwright take --shard=N/T, and pytest has -n auto for process-level parallelism and pytest-split for duration-balanced groups.

Adding shards has sharply diminishing returns because every shard pays the fixed cost of checkout, dependency install, and service startup again. Four shards on a ten-minute suite is usually a good trade. Sixteen shards on the same suite often runs slower in aggregate and costs four times the runner minutes.

  test:
    needs: check
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      # Do not cancel shard 3 because shard 1 failed:
      # you want the whole picture in one run.
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22', cache: npm }
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/4 --reporter=junit --outputFile=junit-${{ matrix.shard }}.xml
      - uses: actions/upload-artifact@v4
        if: always()      # upload the report even when the tests failed
        with:
          name: junit-${{ matrix.shard }}
          path: junit-${{ matrix.shard }}.xml

# Processes on one machine. 'auto' matches the CPU count.
# loadgroup keeps tests marked with the same xdist_group together.
pytest -n auto --dist loadgroup

# Record how long each test takes; commit the file.
pytest --store-durations

# Then split across CI machines by measured time, not file count.
pytest --splits 4 --group "$SHARD" --durations-path .test_durations

# Browser tests shard the same way.
npx playwright test --shard=1/4

Parallelism is where hidden test coupling becomes visible. Tests that shared a database row, a fixed port, a temp file path, or a frozen clock pass in sequence and fail at random once they run at the same time. That is not a reason to go back to serial execution; it is the suite telling you which tests were lying about their independence.

For deciding what belongs in each tier of the suite in the first place, the frontend testing guide covers where unit, component, and end-to-end tests each earn their runtime.

Artifacts: Build Once, Promote the Same Bytes

Build the artifact exactly once, publish it addressed by its content digest, and have every subsequent environment deploy that digest. Rebuilding per environment means production runs bytes that no test ever executed, and it reintroduces every source of nondeterminism the pipeline was supposed to remove: a new base image, a floated transitive dependency, a different runner.

The mechanical version of the rule is that a tag is a pointer and a digest is an identity. ghcr.io/acme/api:latest can mean something different tomorrow. ghcr.io/acme/api@sha256:9f2a... cannot. Deploy manifests should carry digests. Human-facing tags are for people reading a registry listing, not for deciding what runs.

  build:
    needs: check
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write        # required to sign the attestation
      attestations: write
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/acme/api:${{ github.sha }}
          provenance: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - uses: actions/attest-build-provenance@v2
        with:
          subject-name: ghcr.io/acme/api
          subject-digest: ${{ steps.push.outputs.digest }}
          push-to-registry: true

One consequence of building once is that configuration cannot be baked in. If the staging build has a staging API URL compiled into the bundle, you no longer have one artifact, you have two, and the promotion story is fiction. Environment differences belong in runtime configuration: environment variables, mounted config, a settings service. For browser bundles where a value genuinely has to be compiled in, either serve the config from an endpoint at boot or accept that the frontend is a separate artifact with its own promotion path, and say so out loud.

Environment Promotion and Approval Gates

Promotion is a deploy of an artifact that already exists, parameterized by environment. Staging should be automatic and unattended, because an environment that needs a human to update it drifts, and a drifted staging environment produces confident green results that mean nothing. Production gates should be few, fast, and informative: a gate whose approver clicks yes without reading is a delay, not a control.

Keep the environment definitions in the same repository as the code that deploys into them, and keep the infrastructure itself in code as well so that "what is different about staging" has a diffable answer. The Terraform guide covers that half; the point here is that promotion only means something when the target environments are shaped alike.

  deploy-staging:
    needs: [build, test]
    runs-on: ubuntu-latest
    environment: staging          # no reviewers configured: deploys itself
    steps:
      - run: ./deploy.sh staging ghcr.io/acme/api@${{ needs.build.outputs.digest }}
      - name: Smoke test the deployed instance
        run: ./smoke.sh https://staging.acme.dev

  deploy-production:
    needs: deploy-staging
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    # Reviewers and a wait timer are configured on the environment
    # in repo settings, not here. Secrets are scoped to it too.
    environment:
      name: production
      url: https://api.acme.dev
    steps:
      # Same digest that staging ran. No rebuild anywhere in this job.
      - run: ./deploy.sh production ghcr.io/acme/api@${{ needs.build.outputs.digest }}

Database changes are where promotion gets genuinely hard, because schema is shared state and cannot be rolled back by pointing at an older image. The pattern that survives is expand and contract, sometimes called parallel change. Deploy the additive schema change first, on its own. Ship code that writes to both the old and new shape and reads the new one with a fallback. Backfill. Only once the previous release is fully retired do you ship the migration that drops the old column, in a separate later deploy. Three deploys instead of one, and every intermediate state is rollback-safe.

Secrets: Stop Storing Cloud Keys in CI

Long-lived access keys stored as CI variables are the most common serious weakness in an otherwise decent pipeline. They do not expire, they are readable by anything that can run a step, and they survive an employee's departure. OIDC federation replaces them: the runner presents a short-lived signed token describing the repository, branch, and environment, and the cloud provider exchanges it for credentials that live for minutes.

    permissions:
      id-token: write     # mint the OIDC token
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-production
          aws-region: us-east-1
      - run: aws ecs update-service --cluster app --service api --force-new-deployment

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      // Pin to one repo AND one environment. A wildcard here
      // lets any branch in the repo assume the production role.
      "token.actions.githubusercontent.com:sub": "repo:acme/api:environment:production"
    }
  }
}

The sub condition is the part people get wrong. A policy that matches repo:acme/api:* means a pull request from a branch called fix-typo can assume the production deployment role. Pin it to the environment, and configure that environment to only accept deployments from protected branches. Cloud IAM shape is covered further in the AWS IAM guide.

Deployment Strategies: Rolling, Blue-Green, and Canary

Rolling replaces instances a few at a time and is the sensible default for stateless services. Blue-green runs two full environments and flips traffic at once, buying an instant switch back at the cost of double capacity. Canary sends a small traffic slice to the new version and widens only if error rate and latency hold, which catches problems that only appear under real load.

Kubernetes gives you rolling updates without any extra tooling: set maxSurge and maxUnavailable, write an honest readiness probe, and the control plane will not send traffic to a pod that has not said it is ready. Canary needs more. The common answer is a progressive delivery controller that owns the rollout and can abort on its own.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  replicas: 10
  strategy:
    canary:
      # Analysis runs continuously against the canary pods.
      # A failed run aborts and shifts traffic back automatically.
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: { duration: 30m }

A canary is only as good as the question it asks. "Did the pods start" is not a canary, it is a health check. The analysis should compare the canary against the stable version on the metrics users feel: error rate, latency at the ninety-fifth percentile, and one business number such as checkout completions. That requires the telemetry to exist first, which is why the observability guide is a prerequisite for progressive delivery rather than a nice extra.

Rollback That Works Under Pressure

Rollback means redeploying an artifact that already exists, never reverting the commit and rebuilding. A rebuild takes as long as the original pipeline, can fail on the way, and requires a person to make judgement calls while the site is down. The correct rollback is one command, needs no compilation, and has been rehearsed at least once on a calm afternoon.

# What is running, and what ran before it?
kubectl rollout history deployment/api

# Go back one revision. No rebuild, no pipeline run, no registry push.
kubectl rollout undo deployment/api

# Confirm it landed. Non-zero exit if it did not, so scripts can branch on it.
kubectl rollout status deployment/api --timeout=120s

# Or pin an exact known-good digest, which is safer than "one back"
# when several deploys went out in the last hour.
kubectl set image deployment/api api=ghcr.io/acme/api@sha256:9f2a1c...

The commands are the easy part. What makes rollback fail in real incidents is state that moved forward while the code moved back.

One more habit worth adopting: make rollback the default response to a production alert during a deploy window, not a decision to be debated. Roll back first, diagnose after, from logs and traces rather than from a live broken system. Teams that treat rollback as an admission of failure end up debugging in production for forty minutes while users are affected. Teams that treat it as the cheapest available action recover in three.

The Failure Modes You Will Actually Hit

None of these are exotic. Every one of them is common enough that most engineers reading this have lived through at least three.

CI/CD Tools Compared, Honestly

Tool choice matters far less than pipeline design, and every tool below can express everything in this guide. The real differences are where the work runs, how much of the runner you own, and how painful it is to reproduce a failure locally.

A defensible default for a small team: the CI that ships with your code host, containers as the artifact format, OIDC for cloud access, and a separate deployment controller once you are on Kubernetes. Revisit only when a specific pain has a name and a cost.

When Not to Build Any of This

Pipeline work absorbs unlimited engineering attention, and a sophisticated pipeline around a product with no users is a hobby with a YAML file.

The Short Version

The bottom line: order stages so the cheapest, most frequent failure reports in about ninety seconds. Cache inputs on a lockfile-derived key with a fallback ladder, and make sure dropping every cache changes nothing but the clock. Shard tests by measured runtime and quarantine flakes instead of retrying them. Build one artifact, address it by digest, and let promotion mean deploying that digest into a new environment. Replace stored cloud keys with OIDC pinned to a repository and an environment. Pick the rollout strategy that matches the blast radius you can accept, and make rollback a single rehearsed command with no rebuild in it. A pipeline earns trust the same way a person does: by being predictable, and by being reversible when it is wrong.

Frequently Asked Questions

What is the difference between continuous delivery and continuous deployment?

Continuous delivery means every commit that passes the pipeline produces an artifact that could go to production at any moment, and a human decides when. Continuous deployment removes the human: passing the pipeline is the decision, and the change ships automatically. The engineering work is identical up to that last step, which is why the useful question is not which one you practice but whether the approval adds information. If the approver clicks yes without looking, the gate is theatre and you are paying the cost of continuous deployment without the speed.

How long should a CI pipeline take?

Keep the part that blocks a merge under about ten minutes, which is roughly the window in which an engineer will wait and watch rather than switch tasks. Past fifteen minutes people start batching changes to amortise the wait, which makes each change bigger and each failure harder to diagnose. The three levers, in order of payoff: move work that does not gate the merge into a post-merge or nightly job, cache dependency installs and build output on a lockfile-derived key, and split the test suite across shards balanced by measured runtime rather than file count.

Should I use blue-green or canary deployments?

Blue-green runs two complete environments and flips all traffic at once, so the switch is instant and so is the switch back, at the cost of double capacity during the changeover. Canary sends a small slice of live traffic to the new version, watches error rate and latency, then widens if the numbers hold. Canary catches problems that only show up under real traffic and limits the blast radius to the slice, but it needs traffic splitting and automated metric analysis to be worth anything. Blue-green when the switch back must be instantaneous and you can afford the capacity; canary when you have enough traffic that a five percent slice gives a clear signal within minutes. For a service running a handful of instances, a plain rolling update is usually the right answer, and the Kubernetes guide covers the probe settings that make it safe.

How do I roll back a deployment safely?

Redeploy a previously published artifact digest rather than reverting the commit and rebuilding, because a rebuild takes as long as the original pipeline and can fail on the way. On Kubernetes that is kubectl rollout undo deployment/name, followed by kubectl rollout status to confirm, or kubectl set image with an exact digest when several deploys went out recently. What makes rollback fail in practice is state: a migration that dropped a column, a cache entry the old code cannot parse, a message format the old consumer does not understand. Use expand-and-contract migrations so the previous release still runs against the new schema, keep the contract step in a separate later deploy, and rehearse the whole path once a quarter when nothing is broken.

Reference documentation: GitHub Actions Docs, Security Hardening with OpenID Connect, Kubernetes Deployments, Argo Rollouts: Canary, Docker Build Cache