Your single source of truth for CI/CD pipelines, infrastructure automation, monitoring, and incident response.
This runbook is designed to empower our engineering teams to ship code faster, safer, and with operational excellence. It standardizes our tooling, codifies our best practices, and provides clear procedures for troubleshooting common failure modes.
|
This document is a living artifact. If you discover a workaround that is not documented here, please submit a Pull Request to this repository. Tribal knowledge is a liability; documented knowledge is an asset. |
Our standard pipeline enforces code quality, runs security scans, builds immutable artifacts, and deploys them incrementally through environments.
All pipelines must be defined in the root of the repository.
GitHub Actions: .github/workflows/ci.yml
GitLab CI: .gitlab-ci.yml
Jenkins: Jenkinsfile (Declarative syntax preferred)
Trigger Strategy:
* push to develop → Triggers Build + Unit Tests + Deploy to Dev.
* push to main / release/* → Triggers Build + Integration Tests + Deploy to Staging + Manual Approval Gate for Production.
All microservices must be containerized using Docker. We enforce a Distroless or Slim Alpine base image to minimize the attack surface.
# Stage 1: Builder
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /service .
# Stage 2: Final Image (Distroless)
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /service /service
USER nonroot:nonroot
ENTRYPOINT ["/service"]|
Always set |
We do not simply kubectl apply in production. We use progressive delivery:
Canary Deployments: Roll out to 10% of traffic for 30 minutes to monitor error rates.
Blue/Green: Zero-downtime switching by updating the Service selector.
Rollbacks: Immediately revert to the previous successful Helm release using helm rollback.
|
Database migrations must be backward-compatible before the new application version is deployed. Always run |
We treat our infrastructure like software. All cloud resources are defined declaratively and stored in version control.
We use Terraform Cloud (or AWS S3 backend) with state locking.
State Bucket: s3://<company>-terraform-states
DynamoDB Lock Table: Prevents concurrent modifications.
Workspace Strategy: We use separate workspaces for dev, staging, and prod.
Naming Conventions:
* All resources must have a Name tag in the format: {Project}-{Environment}-{ResourceType}.
* Example: payment-api-prod-rds-mysql
modules/
├── networking/
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
├── compute/
│ └── ec2/
└── database/
└── rds/|
Never commit |
For pet-servers (legacy) and VM provisioning, we use Ansible.
Playbook Execution: Always run in --check mode first (Dry-run) to validate changes before actual enforcement.
Secrets: We use ansible-vault to encrypt sensitive variables. The vault password is retrieved from AWS Secrets Manager at runtime.
If it isn’t measured, it doesn’t exist. We utilize the three pillars: Metrics, Logs, and Traces.
Every service must expose a /metrics endpoint.
Critical Golden Signals to Alert On:
1. Latency: P99 response time (e.g., histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 500ms).
2. Traffic: Request volume per second (e.g., rate(http_requests_total[1m])).
3. Errors: Error percentage (e.g., rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05).
4. Saturation: CPU/Memory usage of the pod (e.g., container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85).
Format: All logs must be output in JSON format to facilitate structured querying.
Trace ID: Every log line must include a trace_id and span_id (correlation with APM).
Retention: Dev logs retained 7 days; Prod logs retained 30 days in hot storage.
| Severity | Definition | Response Time (SLA) |
|---|---|---|
P0 (Critical) |
Service is down. Customer impact confirmed. |
Acknowledge within 5 mins. Establish bridge call. |
P1 (High) |
Service degraded. High latency. Partial outage. |
Acknowledge within 15 mins. |
P2 (Medium) |
Non-critical component failing. Disk space warning. |
Acknowledge within 2 hours. |
P3 (Low) |
Info. Metrics anomalies with no user impact. |
Acknowledge within 24 hours. |
Security is embedded in the developer workflow, not bolted on at the end.
Rule #1: Never hardcode secrets in environment variables.
Rule #2: Never store .env files in Git.
Runtime Injection: Secrets are pulled via the Vault Agent Sidecar Injector or AWS SDK at application startup.
secretName := "prod/payment-api/db-password"
svc := secretsmanager.New(session.New())
input := &secretsmanager.GetSecretValueInput{SecretId: &secretName}
output, _ := svc.GetSecretValue(input)
password := *output.SecretStringAll dependencies are scanned for known vulnerabilities.
Trigger: Runs automatically on every Pull Request.
Tool: Snyk / Trivy / Dependabot.
Action: If a Critical severity CVE is found, the PR is automatically blocked from merging. A security champion must review and whitelist it with a justification.
|
Be aware of Dependency Confusion attacks. Configure your package managers (NPM, Pip, Maven) to use your internal private registry as the primary source, with public registries as a fallback. |
Create a branch from main: hotfix/description.
Cherry-pick the specific commit that fixes the critical issue.
Run the pipeline and manually approve the "Deploy to Prod" step.
Immediately merge the hotfix back into develop to prevent regression in the next release.
Retrieve the events: kubectl describe pod <pod-name>.
Check if it’s an OOM (Out of Memory) kill (look for OOMKilled: true).
Check the previous logs: kubectl logs <pod-name> --previous.
Verify if the ConfigMap mounted matches the expected file structure.
Generate a new access key in the IAM console.
Store it in Vault under a new version.
Restart the services that rely on this secret (or wait for the sidecar to refresh).
Revoke the old access key 24 hours later to ensure all services have rotated.
Term |
Definition |
Artifact |
A packaged immutable binary (e.g., JAR, Docker Image) produced by a pipeline. |
Immutable Infrastructure |
Servers are never modified after deployment; they are replaced entirely during updates. |
Idempotency |
An operation that produces the same result regardless of how many times it is executed. |
MTTR |
Mean Time To Recover. The average time taken to restore service after an outage. |
GitOps |
Declarative infrastructure and application management powered by Git as the single source of truth (e.g., ArgoCD). |
|
Document Metadata - Last Reviewed: 2026-08-15 - Version: v2.3.1 - Maintainer: Platform Engineering (Slack: #ask-platform) |