Self-service AWS CodePipeline platform — developers ship compliant CI/CD pipelines in minutes via dashboard, CLI, CDK, or AI prompt, while platform teams enforce policy-as-code guardrails, governance, and per-team isolation.
Day-2 procedures for a running Pipeline Builder deployment: preflight, secret generation & rotation, backups & disaster recovery, and teardown. See deploy/README.md for the target map and bring-up flow.
Every entrypoint should assert its tools up front (preflight <tools…> in deploy/bin/common.sh) so a missing dependency fails fast instead of deep into a 30–60 min provision. Typical needs:
| Target | Tools |
|---|---|
| docker | docker, openssl, jq |
| minikube | minikube, kubectl, openssl, envsubst, jq |
| ec2 (bootstrap) | aws, docker, jq, openssl |
| eks | aws, eksctl, kubectl, openssl, jq |
Backup/restore additionally need pg_dump/psql (postgres client) and mongodump/mongorestore (mongo database tools).
Generation is automatic and per-deploy. On first bring-up, .env is seeded from .env.example and its CHANGE_ME credentials are filled with fresh random values by pb_gen_env_secrets (deploy/bin/gen-env-secrets.sh), which then asserts no CHANGE_ME remains in a required secret. The MongoDB replica-set keyfile is generated by pb_ensure_mongo_keyfile (deploy/bin/mongo-keyfile.sh). Neither .env nor mongodb-keyfile is tracked in git.
The keyfiles that were previously committed have been
git rm --cacheded. A fresh checkout ships none — setup generates them. Existing environments should rotate their keyfile (it was shared/public): generate a new one, restart mongod on each member with the new key.
--rotate flag)A naive “regenerate .env” would rewrite passwords out of sync with the running databases and break them — the password in .env must match what the DB actually accepts. Rotate per-secret, in order:
JWT_SECRET / REFRESH_TOKEN_SECRET (stateless). Rotating invalidates all issued tokens (users re-login). Safe path: set the new secret, roll the services. For zero-downtime, run a dual-key overlap window if the platform supports a secondary verification key; otherwise accept a brief re-auth.POSTGRES_PASSWORD / DB_PASSWORD. Change the password in Postgres first, then update the secret, then roll: ALTER USER "$POSTGRES_USER" WITH PASSWORD '<new>'; → update the k8s Secret / .env → kubectl rollout restart deploy/postgres and the app deployments. Do NOT just rewrite .env.MONGO_INITDB_ROOT_PASSWORD + MONGODB_URI. db.changeUserPassword() in Mongo first, then update the secret + URI, then roll.jwt-keys.sh) — regenerate, re-run store-token so image pulls don’t 401 (see the CodeBuild cold-start note).Always update the k8s Secret (not just .env) on the k8s targets, then kubectl rollout restart the affected Deployments.
Each target ships its own bin/backup.sh / bin/restore.sh (e.g. deploy/local/minikube/bin/backup.sh) — same names across targets, common.sh stays shared in deploy/bin/. The three kubectl targets (minikube / ec2 / eks) run the port-forward variant: it stands up short-lived kubectl port-forwards to the in-cluster postgres/mongodb (+minio), rewrites the connection env to the tunnels, dumps, and tears them down — so the in-cluster service names don’t need to be host-reachable (DRY_RUN=1 and restore.sh --list skip the forwards and need no cluster). The docker variant connects directly. They dump and restore Postgres + Mongo (to/from S3; restore.sh requires --confirm-destructive), and optionally mirror the MinIO buckets (attachments/registry/loki/thanos) when MINIO_ENDPOINT is set. They are not scheduled by default on any target — wire them:
deploy/aws/eks/backup/backup-cronjob.yaml (kept out of the kustomize overlay so it never auto-applies) after (1) provisioning an encrypted + versioned S3 bucket, (2) granting the db-backup ServiceAccount s3:PutObject (Pod Identity / IRSA — the current eks setup role grants only SES + CodePipeline, so add this), (3) pointing image: at a backup image with pg_dump/mongodump/aws/mc, and (4) setting BACKUP_BUCKET. (Minikube is local and has no bucket to write to; run deploy/local/minikube/bin/backup.sh manually there if you point it at reachable object storage.)bootstrap.sh installs pipeline-backup.timer disabled. To enable it you must install the DB clients, give the host a path to the ClusterIP DBs (port-forward/NodePort), provision the bucket + s3:PutObject, and set BACKUP_BUCKET.Bucket hardening: enable SSE-KMS, versioning, and a bucket lifecycle retention policy (not the app’s client-side RETENTION_DAYS prune, which a compromised role could bypass).
MinIO object storage (plugin images, message attachments, logs) is backed up by the same script: set MINIO_ENDPOINT + MINIO_ROOT_USER/PASSWORD + a durable MINIO_BACKUP_TARGET_URL and its *_ACCESS_KEY/*_SECRET_KEY. backup.sh runs mc mirror (additive — never deletes from the backup, so a source delete can’t wipe it; pair the target with versioning for point-in-time). Restore with restore.sh --minio --confirm-destructive (reverse mirror; standalone, does not touch the DBs). Skipping this (leaving MINIO_ENDPOINT unset) is a deliberate opt-out — a DB-only restore can’t rebuild a working platform without the blobs.
DR drill: periodically restore the latest backup into a scratch namespace/instance and verify — an untested backup is not a backup.
Several stateful services store into MinIO (S3-compatible), each with its own bucket + a per-service, bucket-scoped key (never the root credentials) — all created by the minio-init bootstrap (a compose service / a k8s Job):
| Bucket | Consumer | Key |
|---|---|---|
message-attachments |
message service (attachments) | message-svc |
registry |
Docker registry (S3 storage driver — now stateless, no PVC) | registry-svc |
loki |
Loki (chunks + index; /loki is now ephemeral scratch) |
loki-svc |
thanos |
Thanos sidecar (Prometheus 2h TSDB blocks, long-term) — READ back via the store-gateway + querier | thanos-svc |
HA / topology:
minio Service (round-robin); peers resolve via the minio-headless Service.mc admin replicate.Back up the MinIO drives as part of DR (EKS: the 4 data-minio-* PVCs; ec2: minio-data/{1..4}; dev: ./data/minio-data). Fresh install — nothing to migrate.
Long-term metrics (Thanos) read path. The sidecar only uploads Prometheus’ 2h blocks to the thanos bucket; querying them back is served by two components (thanos-query.yaml on the k8s targets, equivalent services in docker-compose): a store-gateway (exposes the archived blocks over the Thanos StoreAPI, gRPC 10901; local index cache is ephemeral) and a querier (Prometheus-compatible HTTP 9090 that fans out to the sidecar + store-gateway and de-duplicates). PROMETHEUS_URL points platform’s Observability query endpoint at the querier (http://thanos-query:9090) so PromQL spans recent + archived history; set it back to http://prometheus:9090 for recent-only. KEDA autoscaling deliberately still targets Prometheus directly (recent-only, lower latency).
All targets run an Istio ambient mesh (STRICT mTLS + identity authz). Verify + operate:
kubectl get pods -n istio-system # istiod, ztunnel, istio-cni Ready
istioctl analyze -n pipeline-builder # policy sanity
istioctl ztunnel-config workloads # every pod PROTOCOL=HBONE (enrolled)
sa/<name> is missing from the callee’s
AuthorizationPolicy in k8s/istio.yaml — add it and re-apply. Every scraped app
service must list prometheus; every API must list nginx.8080 on aws;
8080+8443 on local).kubectl delete -k k8s/ removes the mesh policies but leaves
istio-system installed; istioctl install is idempotent so re-runs are safe.
minikube delete (local/ec2) / eksctl delete cluster (eks) wipe everything.See Service Mesh for the full troubleshooting table.
docker compose down (data persists in data/); reset = down && rm -rf data/.bin/shutdown.sh does a graceful minikube stop — it halts the VM but PRESERVES its disk and the full cluster state (workloads, PVCs, data), so a restart brings everything back with no re-provisioning. It deliberately does NOT delete the namespace/manifests. Bring it back with bin/startup.sh (fast resume + reconnect port-forwards; no re-install/re-apply). To wipe instead: minikube delete --profile=pipeline-builder (a clean rebuild = delete then re-run bin/setup.sh, or RECREATE=y bin/setup.sh). Data location: minikube stores all hostPath data (postgres, mongodb, minio buckets, …) on the VM’s own persistent /data disk, not the host deploy/local/minikube/data/ folder — that folder stays empty (minikube reserves /data for its persistent disk, which shadows a host mount there, and DB data on a 9p mount is unreliable). Data survives minikube stop/start; minikube delete wipes it. For host-side copies use deploy/local/minikube/bin/backup.sh (dumps via kubectl port-forward — mongodump / pg_dump / mc mirror).bin/shutdown.sh (as root) removes the iptables DNAT rules, then a graceful minikube stop — same as minikube, it PRESERVES the VM disk + cluster state; bin/startup.sh brings it back. It does NOT touch the EC2 instance (tear that down by deleting the CloudFormation stack). Wipe the cluster with sudo -u minikube minikube delete --profile=pipeline-builder.shutdown.sh (types the cluster name to confirm; --delete-volumes to also remove the Retained EBS/EFS). Without --domain, eks leaves the ACM cert / Route 53 alias / SES resources behind (warned).Lean deploy (LEAN=1) — when the full stack + the Istio mesh exceeds ~8 vCPU (an ~8-core laptop, or a smaller EC2 instance), LEAN=1 brings up the core stack + mesh only: it omits the optional observability/admin services (prometheus, thanos, loki, promtail, jaeger, alertmanager, mongo-express, pgadmin) and collapses every workload to a single replica. Supported on minikube (LEAN=1 deploy/local/minikube/bin/setup.sh) and ec2 — at launch via the CFN Lean param (LEAN=1 deploy/aws/ec2/bin/setup.sh, or pipeline-manager infra provision --target ec2 --lean), or on the box (LEAN=1 sudo -E bash deploy/aws/ec2/bin/startup.sh; -E preserves the env through sudo). It lets ec2 run on a t3.xlarge instead of a t3.2xlarge. Both targets drive the same lean_filter. Full stack (all observability) is the default for larger machines; eks is unaffected. See Service Mesh: LEAN mode.
Lifecycle scripts (minikube + ec2) — both single-node targets share a setup.sh / startup.sh / shutdown.sh triad. setup.sh provisions (CREATE cluster + install mesh/KEDA + apply manifests); startup.sh is the fast resume of a stopped cluster (reconnects port-forwards on minikube; re-mounts host data + iptables on ec2 — no re-install/re-apply); shutdown.sh is a graceful minikube stop.
Sizing overrides — setup.sh (minikube) / startup.sh (ec2) take env-var overrides: DISK_SIZE=60g (VM disk; default 30g minikube / 40g ec2), ISTIO_VERSION=…, LEAN=1. CPU, memory, and disk size are applied only at cluster CREATE. When an existing cluster is found, setup.sh/startup.sh resume it (data preserved) unless you ask to recreate: on a TTY they prompt (default: keep), or set RECREATE=y to rebuild non-interactively (minikube: this WIPES /data; ec2: rebuilds the cluster but host $DATA_DIR data survives — clear it to truly wipe). Back up first with backup.sh if needed. On the docker driver the disk is bounded by Docker Desktop’s virtual-disk limit; eks node disk is managed by the Auto Mode NodeClass, not DISK_SIZE.
Destructive resets print raw one-liners today — dump first (backup.sh) before wiping data you might want.