Deployment Patterns

Production deployment workflows and CI/CD best practices.

When to Activate

• Setting up CI/CD pipelines
• Dockerizing an application
• Planning deployment strategy (blue-green, canary, rolling)
• Implementing health checks and readiness probes
• Preparing for a production release
• Configuring environment-specific settings

Deployment Strategies

#### Rolling Deployment (Default)

Replace instances gradually, old and new versions run simultaneously during rollout.

Instance 1: v1 → v2  (update first)
Instance 2: v1        (still running v1)
Instance 3: v1        (still running v1)

Instance 1: v2
Instance 2: v1 → v2  (update second)
Instance 3: v1

Instance 1: v2
Instance 2: v2
Instance 3: v1 → v2  (update last)

Pros: Zero downtime, gradual rollout Cons: Two versions run simultaneously, requires backward-compatible changes Use when: Standard deployments, backward-compatible changes

#### Blue-Green Deployment

Run two identical environments. Switch traffic atomically.

Blue  (v1) ← traffic
Green (v2)   idle, running new version

# After verification:
Blue  (v1)   idle (becomes standby)
Green (v2) ← traffic

Pros: Instant rollback (switch back to blue), clean cutover Cons: Requires 2x infrastructure during deployment Use when: Critical services, zero-tolerance for issues

#### Canary Deployment

Route a small percentage of traffic to the new version first.

v1: 95% of traffic
v2:  5% of traffic  (canary)

# If metrics look good:
v1: 50% of traffic
v2: 50% of traffic

# Final:
v2: 100% of traffic

Pros: Catches issues with real traffic before full rollout Cons: Requires traffic splitting infrastructure, monitoring Use when: High-traffic services, risky changes, feature flags

Docker

#### Multi-Stage Dockerfile (Node.js)

# Stage 1: Install dependencies
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --production=false

# Stage 2: Build
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
RUN npm prune --production

# Stage 3: Production image
FROM node:22-alpine AS runner
WORKDIR /app

RUN addgroup -g 1001 -S appgroup && adduser -S appuser -u 1001
USER appuser

COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/package.json ./

ENV NODE_ENV=production
EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "dist/server.js"]

#### Multi-Stage Dockerfile (Go)

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /server ./cmd/server

FROM alpine:3.19 AS runner
RUN apk --no-cache add ca-certificates
RUN adduser -D -u 1001 appuser
USER appuser

COPY --from=builder /server /server

EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s CMD wget -qO- http://localhost:8080/health || exit 1
CMD ["/server"]

#### Multi-Stage Dockerfile (Python/Django)

FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir uv
COPY requirements.txt .
RUN uv pip install --system --no-cache -r requirements.txt

FROM python:3.12-slim AS runner
WORKDIR /app

RUN useradd -r -u 1001 appuser
USER appuser

COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY . .

ENV PYTHONUNBUFFERED=1
EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=3s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health/')" || exit 1
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "4"]

#### Docker Best Practices

# GOOD practices
- Use specific version tags (node:22-alpine, not node:latest)
- Multi-stage builds to minimize image size
- Run as non-root user
- Copy dependency files first (layer caching)
- Use .dockerignore to exclude node_modules, .git, tests
- Add HEALTHCHECK instruction
- Set resource limits in docker-compose or k8s

# BAD practices
- Running as root
- Using :latest tags
- Copying entire repo in one COPY layer
- Installing dev dependencies in production image
- Storing secrets in image (use env vars or secrets manager)

CI/CD Pipeline

#### GitHub Actions (Standard Pipeline)

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test -- --coverage
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage
          path: coverage/

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    # The "production" environment must have a GitHub environment protection rule
    # with required reviewers, so this job pauses for an explicit human approval
    # before it runs. Merge to main must never auto-deploy to prod without that gate.
    environment: production
    steps:
      - name: Deploy to production
        run: |
          # Platform-specific deployment command
          # Railway: railway up
          # Vercel: vercel --prod
          # K8s: kubectl set image deployment/app app=ghcr.io/${{ github.repository }}:${{ github.sha }}
          echo "Deploying ${{ github.sha }}"

Gate the production deploy behind an explicit approval. Either protect the production GitHub environment with required reviewers (as above), or move production promotion to a separate workflow_dispatch trigger that a human runs on demand. Merging to main may build the image and deploy to staging automatically, but it must not promote to production without that explicit gate.

#### Pipeline Stages

PR opened:
  lint → typecheck → unit tests → integration tests → preview deploy

Merged to main:
  lint → typecheck → unit tests → integration tests → build image → deploy staging → smoke tests
  → [explicit manual approval gate] → deploy production

Everything up to and including the staging smoke tests runs automatically on merge. The final deploy-production step is a separate, explicitly approved gate (an environment protection rule with required reviewers, or a manually triggered workflow), not an automatic link in the merge chain.

Health Checks

#### Health Check Endpoint

// Simple health check
app.get("/health", (req, res) => {
  res.status(200).json({ status: "ok" });
});

// Detailed health check (for internal monitoring)
app.get("/health/detailed", async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    externalApi: await checkExternalApi(),
  };

  const allHealthy = Object.values(checks).every(c => c.status === "ok");

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? "ok" : "degraded",
    timestamp: new Date().toISOString(),
    version: process.env.APP_VERSION || "unknown",
    uptime: process.uptime(),
    checks,
  });
});

async function checkDatabase(): Promise<HealthCheck> {
  try {
    await db.query("SELECT 1");
    return { status: "ok", latency_ms: 2 };
  } catch (err) {
    return { status: "error", message: "Database unreachable" };
  }
}

#### Kubernetes Probes

livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 2

startupProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 0
  periodSeconds: 5
  failureThreshold: 30    # 30 * 5s = 150s max startup time

Environment Configuration

#### Twelve-Factor App Pattern

# All config via environment variables — never in code
DATABASE_URL=postgres://user:pass@host:5432/db
REDIS_URL=redis://host:6379/0
API_KEY=${API_KEY}           # injected by secrets manager
LOG_LEVEL=info
PORT=3000

# Environment-specific behavior
NODE_ENV=production          # or staging, development
APP_ENV=production           # explicit app environment

#### Configuration Validation

import { z } from "zod";

const envSchema = z.object({
  NODE_ENV: z.enum(["development", "staging", "production"]),
  PORT: z.coerce.number().default(3000),
  DATABASE_URL: z.string().url(),
  REDIS_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
  LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"),
});

// Validate at startup — fail fast if config is wrong
export const env = envSchema.parse(process.env);

Rollback Strategy

#### Instant Rollback

# Docker/Kubernetes: point to previous image
kubectl rollout undo deployment/app

# Vercel: promote previous deployment
vercel rollback

# Railway: redeploy previous commit
railway up --commit <previous-sha>

# Database: rollback migration (if reversible)
npx prisma migrate resolve --rolled-back <migration-name>

#### Rollback Checklist

• [ ] Previous image/artifact is available and tagged
• [ ] Database migrations are backward-compatible (no destructive changes)
• [ ] Feature flags can disable new features without deploy
• [ ] Monitoring alerts configured for error rate spikes
• [ ] Rollback tested in staging before production release

Production Readiness Checklist

Before any production deployment:

#### Application

• [ ] All tests pass (unit, integration, E2E)
• [ ] No hardcoded secrets in code or config files
• [ ] Error handling covers all edge cases
• [ ] Logging is structured (JSON) and does not contain PII
• [ ] Health check endpoint returns meaningful status

#### Infrastructure

• [ ] Docker image builds reproducibly (pinned versions)
• [ ] Environment variables documented and validated at startup
• [ ] Resource limits set (CPU, memory)
• [ ] Horizontal scaling configured (min/max instances)
• [ ] SSL/TLS enabled on all endpoints

#### Monitoring

• [ ] Application metrics exported (request rate, latency, errors)
• [ ] Alerts configured for error rate > threshold
• [ ] Log aggregation set up (structured logs, searchable)
• [ ] Uptime monitoring on health endpoint

#### Security

• [ ] Dependencies scanned for CVEs
• [ ] CORS configured for allowed origins only
• [ ] Rate limiting enabled on public endpoints
• [ ] Authentication and authorization verified
• [ ] Security headers set (CSP, HSTS, X-Frame-Options)

#### Operations

• [ ] Rollback plan documented and tested
• [ ] Database migration tested against production-sized data
• [ ] Runbook for common failure scenarios
• [ ] On-call rotation and escalation path defined

GitHub Actions Standards

#### OIDC Authentication (No Static Credentials)

Use OIDC to authenticate to cloud providers, never store long-lived credentials in CI secrets:

jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/deploy-role
          aws-region: eu-west-1

#### Job-Level Permissions Block (Required)

Every job must declare minimum required permissions explicitly:

jobs:
  build:
    permissions:
      contents: read        # checkout
      packages: write       # push to GHCR
      id-token: write       # OIDC
      pull-requests: write  # PR comments

Never use permissions: write-all at workflow or job level.

#### Pipeline Linting

Add actionlint as a required CI check on all workflow changes:

- name: Lint GitHub Actions workflows
  uses: rhysd/actionlint@v1

#### Failure Notifications

On any pipeline failure, notify the PR author and a shared Slack channel:

- name: Notify on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Pipeline failed: ${{ github.workflow }} · ${{ github.job }}",
        "blocks": [{
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*Pipeline:* ${{ github.workflow }}\n*Job:* ${{ github.job }}\n*Branch:* ${{ github.ref_name }}\n*SHA:* ${{ github.sha }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View logs>"
          }
        }]
      }

#### Matrix Build Configuration

strategy:
  fail-fast: true    # PR builds: stop on first failure
  # fail-fast: false  # Release validation: test all combinations
  matrix:
    os: [ubuntu-latest, windows-latest]
    java: ['21', '23']

Use fail-fast: true for PR builds; disable for release validation matrices.

#### Pipeline SLA

• CI (lint + test + build): <= 15 minutes
• Deployment pipeline (staging + production): <= 10 minutes
• If a pipeline exceeds SLA, raise it as a tech-debt issue within 2 business days

Canary Deployment

#### Traffic Rollout Schedule

Increment canary traffic in three steps with a health gate between each:

Abort and roll back automatically if error rate exceeds baseline by > 1% or p99 latency increases > 20% during any soak period.

#### Automated Rollback

Automated rollback is a safety reflex and is acceptable even while production promotion stays a manual, approved gate: rolling back returns the system to the last known-good state, it does not push a new untested release forward.

Configure an automated rollback job triggered on health check failure or error rate spike detected within 60 seconds of traffic shift:

- name: Check canary health
  run: |
    ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_requests_total{status=~'5..'}[1m])" | jq '.data.result[0].value[1]' -r)
    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate $ERROR_RATE exceeds threshold — rolling back"
      kubectl rollout undo deployment/my-app
      exit 1
    fi

kubectl rollout undo is a last-resort manual fallback only. Rollbacks in production must be triggered by the automated pipeline, not manually by engineers.

Kubernetes Cluster Standards

#### Pod Security Context (Required on Every Deployment)

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: [ALL]

#### PodDisruptionBudget (Required for All Production Deployments)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1      # or maxUnavailable: 1 — never allow all pods down simultaneously
  selector:
    matchLabels:
      app: my-app

#### NetworkPolicy, Default Deny

Every namespace must have a default-deny NetworkPolicy. Allowances are explicit:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-ingress
spec:
  podSelector:
    matchLabels:
      app: my-app
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: ingress-controller
      ports:
        - protocol: TCP
          port: 8080

#### Autoscaling

# HPA for request-driven scaling
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Use KEDA for event-driven workloads (queue depth, Kafka lag). Use VPA for resource right-sizing in non-production environments.

#### Resource Governance

Every namespace must have ResourceQuota and LimitRange:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi

#### Observability

Every service must expose Prometheus metrics via a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      path: /actuator/prometheus
      interval: 15s

#### Policy Enforcement

Use Kyverno (preferred) or OPA Gatekeeper to enforce cluster policies:

• Require securityContext.runAsNonRoot: true
• Require resource requests/limits on all containers
• Block images without a digest or from non-approved registries
• Require PodDisruptionBudget for Deployments with replicas > 1

#### Stateful Workload Protection

# VolumeSnapshot before any stateful upgrade
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pre-upgrade-snapshot
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: my-data-pvc

Set reclaimPolicy: Retain on all production PersistentVolumes to prevent accidental data loss.

Helm Chart Standards

#### Required Files

Every Helm chart must contain:

• Chart.yaml: name, version, appVersion, description, maintainers
• values.yaml: all configurable values with inline comments
• templates/_helpers.tpl: shared named templates
• NOTES.txt: post-install instructions

#### Value Validation

Use required and fail functions to validate critical values at render time:

# templates/deployment.yaml
image:
  repository: {{ required "image.repository is required" .Values.image.repository }}
  tag: {{ required "image.tag is required" .Values.image.tag }}

#### Distribution & Deployment

• Distribute charts via an OCI registry (helm push, helm pull oci://)
• Commit Chart.lock to VCS
• Deploy with --atomic flag (auto-rollback on failure)
• Never store secrets as plaintext in chart YAML: use External Secrets Operator or Sealed Secrets
• Security scan charts with Trivy + Checkov before publishing

Cloud Provider Standards

#### IAM, Least Privilege

• No wildcard permissions (*) on any resource
• One service account per workload: never share between services
• Rotate service account keys every 90 days (or use Workload Identity / OIDC instead)

#### Network Isolation

• All databases and internal services in private subnets: no public endpoints
• Bastion/jump host or VPN required for admin access

#### Infrastructure as Code

• Remote Terraform state with locking: S3 + DynamoDB (AWS) or GCS (GCP)
• Never commit .tfstate files to VCS

#### Resource Tagging (Required)

Every cloud resource must have these tags:

#### Cost Controls

• Budget alerts at 80% and 100% of monthly budget
• Quota monitoring with alerts before hitting service limits

#### Audit & Security Posture

• CloudTrail (AWS) / Cloud Audit Logs (GCP) enabled in all accounts
• GuardDuty (AWS) / Security Command Center (GCP) enabled
• CDN configuration for all static assets

#### Disaster Recovery

• Multi-region DR with documented RTO and RPO targets
• Annual DR drill with documented results
• Automated failover for RTO < 1 hour; manual runbook for RTO >= 1 hour