DevOps
Reliable Git Deployment Pipelines with Production Verification
A release pipeline design that turns a reviewed commit into a traceable artifact, controlled deployment, observable verification, and rehearsed rollback.
Author: Fahad Bin Shakir · Published: · Updated: · 14 min read
Introduction
A Git push is a source control event, not evidence that production is healthy. Between the reviewed commit and a user's response sit dependency resolution, asset compilation, artifact transfer, environment configuration, database compatibility, process startup, edge caches, DNS, and third party services. A reliable pipeline records and verifies those transitions.
The goal is not a long chain of green icons. It is a release whose contents are known, whose authority is bounded, whose effects are observed through the public path, and whose rollback decision can be made without reconstructing what happened.
Define the release identity once
Tie every build to an immutable commit and produce one versioned artifact. Install dependencies from lockfiles, compile production assets, remove development material, and record checksums plus toolchain versions. Promote that same artifact between environments rather than rebuilding from the branch at each step; otherwise staging and production may contain different dependency resolutions under the same commit label.
Generate provenance that an operator can read: repository, commit, workflow run, artifact digest, build time, and intended environment. Do not place secrets in the artifact or provenance. A small public release identifier can help support teams correlate a page with a deployment, while detailed records remain in protected operational systems.
Make pull request checks representative
Run formatting or lint checks, type analysis, unit and integration tests, dependency review, secret scanning, and a production build before merge. Add route, migration, accessibility, or image validation that reflects the application rather than relying on a generic template. Fail when generated route shells, sitemap entries, or metadata fall out of sync with source data.
Tests should be deterministic enough to trust. Quarantine and repair flaky checks instead of normalizing reruns until green. Cache dependencies carefully: a cache can accelerate installation, but the lockfile and integrity checks still decide what enters the artifact.
Use a production shaped preview for changes that affect routing, headers, generated pages, or environment behavior. It should remain protected from indexing and use disposable credentials, but it must run the built artifact behind the same important proxy assumptions. Reviewers can then inspect the real output rather than a development server whose fallback routing and response headers differ from deployment.
Expose the preview release identifier to reviewers and retire the environment promptly when its branch closes.
- Test direct entry to every public route, not only client navigation from the homepage.
- Reject unexpected large assets, source maps, credentials, and debug files.
- Validate database changes against both the current and next application version.
- Keep production only credentials unavailable to pull request jobs.
Protect the production environment
Restrict production deployment to a protected branch or approved tag. Use environment controls for required review, concurrency, and narrowly scoped secrets. A deployment job should receive credentials only after its protection rules pass, and those credentials should be limited to the target environment and operation.
Serialize changes that cannot overlap safely. Two workflows writing the same release pointer or running migrations concurrently can each succeed locally and leave production inconsistent. Use a concurrency group or deployment lock, make operations idempotent, and define what happens to a queued older release when a newer commit arrives.
Deploy with an atomic boundary
Upload into a new release location, connect environment configuration, warm required caches, and run preflight checks before changing traffic. Switch with a platform release primitive, symlink, or load balancer target rather than copying files over the active application. Keep the previous compatible release available for a bounded rollback window.
Database work needs an expand and contract sequence. Add backward compatible structures first, deploy code that can use both shapes, migrate data in controlled batches, and remove the old form in a later release. A code rollback after destructive schema change is not a reliable recovery plan.
Verify the public outcome immediately
A process health endpoint confirms only that a process answers. After the switch, request the canonical public domain through its CDN and reverse proxy. Verify homepage and direct route status, redirect behavior, critical assets, application APIs, forms without sending repeated real messages, robots, sitemap, canonicals, structured data, security headers, and an unknown path.
Record deployment markers in monitoring and compare error rate, latency, saturation, queue depth, and a small set of user journeys. Verification must have a deadline and explicit rollback thresholds. A partial release that serves HTML but loses its CSS or contact API should not remain live because one health check returned 200.
release=$(git rev-parse HEAD)
npm ci
npm run verify
# Build once, record the artifact digest, deploy it, then probe the canonical public routes.Treat rollback and audit as normal operations
Automate rollback to the previous known artifact and rehearse it before an incident. Decide how new writes, queued jobs, and expanded schemas behave when old code returns. For content heavy sites, rolling forward with a small correction may be safer than reversing a data change, but that decision should use documented thresholds rather than optimism.
Keep an append only record of who approved, what artifact moved, which checks ran, when traffic switched, what probes observed, and whether rollback occurred. Review failed and emergency releases. The useful outcome is a smaller gap between what the pipeline claims and what the public system actually delivered.
Deployment checklist
- Build one immutable artifact from an approved commit and record its digest.
- Keep secrets out of artifacts, logs, and untrusted pull request jobs.
- Run representative tests, production build, metadata checks, and asset budgets.
- Protect production with branch rules, reviews, scoped credentials, and concurrency.
- Use backward compatible migrations and an atomic traffic switch.
- Probe the canonical public domain, APIs, assets, errors, and security headers.
- Define rollback thresholds and rehearse the previous release path.
- Retain a readable audit trail from commit through production verification.