Deploy Operations Runbook

Day-2 procedures for a running Pipeline Builder deployment: preflight, secret generation and rotation, backups and disaster recovery, object storage, the mesh, and teardown.

See deploy/README.md for the target map and bring-up flow.

Highlights

Nothing is scheduled by default on any target. Until you wire a backup, your RPO is “whenever someone last ran backup.sh by hand”.
Postgres and MongoDB are each a single instance with no standby. Losing a volume means a restore, not a failover.
There is no point-in-time recovery. The dumps are logical, so you can restore to a dump boundary and nothing in between.
The biggest hole is .env and certs/. Lose SECRET_ENCRYPTION_KEY and a perfect database dump is still partially unreadable, forever. Copy them into your secret manager.
There is deliberately no blind --rotate flag. Regenerating .env would put passwords out of sync with the running databases and break them. Rotate per secret, in order, database first.
The backup CronJob is enabled by default and will alert until its image exists. It is in both AWS kustomizations, on BACKUP_SCHEDULE (02:00 UTC), with BACKUP_ENABLED=true — so a deploy with no working backup is red rather than quietly green. Build BACKUP_IMAGE, or set BACKUP_ENABLED=false to suspend it deliberately.
The backup job is bucket-scoped and cannot delete. It used to run as the object store’s root credential — admin over every bucket, with read and delete. It now holds a db-backups-only user with PutObject/GetObject/ListBucket and no delete verb, which is possible because retention is the bucket’s own lifecycle rule rather than a client-side prune. A stolen backup credential cannot destroy the backups it was stolen from.
An untested backup is not a backup, and a drill that restores only the dumps proves less than it looks.

Overview

This page is for whoever operates a deployed instance. It assumes the platform is already up — the bring-up itself is in AWS Deployment and the target READMEs.

The single most important thing on it: read What the data tier actually is before you commit to an RPO, because the topology is the constraint, not the backup script.

How it works

Backup and restore are one implementation

deploy/bin/backup.sh and deploy/bin/restore.sh do the work. Each target’s bin/backup.sh / bin/restore.sh is a thin wrapper that picks the connection mode.

Target	Connect mode	How it reaches the data
minikube / ec2 / eks	`--connect k8s`	Short-lived `kubectl port-forward`s to the in-cluster Postgres, MongoDB and object store, with the connection env rewritten to the tunnels and torn down on exit — so in-cluster service names don’t need to be host-reachable
docker	`--connect direct`	Straight to the containers

DRY_RUN=1 and restore.sh --list skip the port-forwards and need no cluster at all.

What it moves: the Postgres dump and the Mongo dump, to and from S3 (restore.sh requires --confirm-destructive). Optionally it also mirrors every object-storage bucket with rclone, when S3_BACKUP_TARGET_URL is set — the source side (S3_ENDPOINT plus root credentials) is already in every target’s .env, so only the destination is opt-in.

The buckets mirrored are the canonical PB_OBJECTSTORE_BUCKETS list in deploy/bin/common.sh: message-attachments, registry, loki, thanos, plugins, plugin-quarantine, audit-heads. A deploy-contract test keeps that list equal to what each target’s rustfs-init bootstrap Job creates and what the EKS CronJob mirrors.

Secret generation is automatic and per-deploy

On first bring-up, .env is seeded from .env.example and its CHANGE_ME credentials are filled with fresh random values by pb_gen_env_secrets (deploy/bin/gen-env-secrets.sh), which then asserts no CHANGE_ME remains in a required secret. The MongoDB replica-set keyfile is generated by pb_ensure_mongo_keyfile (deploy/bin/mongo-keyfile.sh).

Neither .env nor mongodb-keyfile is tracked in git.

The keyfiles that were previously committed have been git rm --cacheded. A fresh checkout ships none — setup generates them. Existing environments should rotate their keyfile, since it was shared publicly: generate a new one, then restart mongod on each member with the new key.

Configuration

Preflight

Every entrypoint should assert its tools up front (preflight <tools…> in deploy/bin/common.sh) so a missing dependency fails fast instead of deep into a 30–60 minute provision.

Target	Tools
docker	`docker`, `openssl`, `jq`
minikube	`minikube`, `kubectl`, `openssl`, `envsubst`, `jq`
ec2 (bootstrap)	`aws`, `docker`, `jq`, `openssl`
eks	`aws`, `eksctl`, `kubectl`, `openssl`, `jq`

Backup and restore additionally need pg_dump / psql (Postgres client) and mongodump / mongorestore (Mongo database tools).

EKS — enabling the nightly CronJob

deploy/aws/eks/k8s/backup-cronjob.yaml is applied by the normal deploy on both AWS targets, from one implementation, and ships suspended (BACKUP_ENABLED=false). One REPLACE_ME remains: BACKUP_IMAGE, which must carry pg_dump, mongodump, the aws CLI, rclone, openssl and curl, and run as a non-root uid. Build it, set BACKUP_IMAGE, set BACKUP_ENABLED=true, and re-run the target’s startup.sh — the schedule is BACKUP_SCHEDULE (default 0 2 * * *, 02:00 UTC).

The job writes to RustFS, the in-cluster object store (BACKUP_BUCKET=db-backups, S3_ENDPOINT=http://rustfs:9000), using the bucket-scoped backup-access-key/backup-secret-key from rustfs-secret — so no S3 bucket, IAM role, IRSA or Pod Identity is needed. rustfs-init creates db-backups with versioning, a BACKUP_RETENTION_DAYS expiry rule and a noncurrent expiry, and attaches a policy with no delete. Understand the trade before relying on it: those dumps live in the same cluster, account and region as the databases, so they survive a dropped table or a bad migration but not loss of the cluster, the AZ or the account. deploy/bin/{backup,restore}.sh honour S3_ENDPOINT for the dump path, so repointing both at a real off-cluster bucket is the only change required if that is ever revisited.

Create the bucket with SSE-KMS, versioning, and — recommended — Object Lock in governance mode.

Put retention on a bucket lifecycle rule, not on a client-side prune: a compromised backup role can skip a client-side prune but cannot shorten a lifecycle rule. This is how the shipped RustFS path already works — rustfs-init sets the rule and the job holds no delete verb. Mirror it if you repoint at real S3.
Create the IAM role. This is the part the deploy does not do for you — the EKS setup role grants only SES and CodePipeline today, so this is an addition.

Make a role the db-backup ServiceAccount can assume, via EKS Pod Identity:
```
aws eks create-pod-identity-association \
  --cluster-name <cluster> --namespace pipeline-builder \
  --service-account db-backup --role-arn <role>
```
Or via IRSA, then uncomment the eks.amazonaws.com/role-arn annotation on the ServiceAccount in the manifest. Minimum policy:
```
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::<backup-bucket>/*" },
    { "Effect": "Allow",
      "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
      "Resource": "<the bucket's KMS key ARN>" }
  ]
}
```
Grant no s3:DeleteObject: retention belongs to the lifecycle rule, so nothing needs it. The shipped RustFS policy is the model — PutObject, GetObject/GetObjectVersion, ListBucket/ListBucketVersions, scoped to the backup bucket and nothing else.
Point image: at a suitable image — one that ships pg_dump, mongodump, the aws CLI, rclone and bash (the job script uses set -o pipefail, which dash does not have), and that runs as non-root. The pod sets runAsNonRoot: true, runAsUser: 65532, so a root-by-default image such as the official postgres is rejected by the kubelet.
Set the values. BACKUP_BUCKET, and keep ENV_NAME matching what backup.sh / restore.sh use — they read the same s3://<bucket>/<env>/<YYYY/MM/DD>/ layout. Then either complete the S3_BACKUP_TARGET_* block plus an objectstore-backup-target Secret, or unset S3_BACKUP_TARGET_URL to skip the object-storage mirror.

Apply and verify. Force one run rather than waiting for 02:00:

kubectl apply -f deploy/aws/eks/k8s/backup-cronjob.yaml
kubectl -n pipeline-builder create job --from=cronjob/db-backup db-backup-manual

Check both the pod logs and that the objects actually landed in the bucket.

Test a restore into a scratch namespace: deploy/aws/eks/bin/restore.sh --confirm-destructive, plus --object-store for object storage. Until this passes you have a CronJob, not a backup.

EC2 — the same CronJob, not a host timer

ec2 and eks run one implementation. k8s/backup-cronjob.yaml is applied by the normal deploy on both targets, on the same BACKUP_SCHEDULE, suspended until BACKUP_ENABLED=true. Enabling it is identical to eks: build the image, set BACKUP_IMAGE, flip BACKUP_ENABLED, re-run startup.sh.

ec2 used to carry a second, separate schedule — bootstrap.sh installed a host pipeline-backup.timer firing backup.sh at 03:30 UTC while the CronJob ran at 03:00. The two knew nothing about each other, so enabling both produced two full dumps a night against the same destination half an hour apart, and the two AWS targets could not be reasoned about together. The timer has been removed.

What bootstrap.sh still installs on the host is the client tools — pg_dump, mongodump and rclone — because the manual path runs there and needs them:

deploy/aws/ec2/bin/backup.sh          # dumps over kubectl port-forwards; no image needed

That manual path is also the answer while BACKUP_IMAGE does not yet exist: it works today, the CronJob does not.

Bucket hardening is the same: SSE-KMS, versioning, and a bucket lifecycle retention policy rather than a client-side prune a compromised role could bypass. On the in-cluster default, rustfs-init already does the versioning and lifecycle parts.

Local targets

Minikube and docker are local and have no bucket to write to. Run their bin/backup.sh by hand against reachable object storage if you want copies.

Object-storage mirror

Plugin images, message attachments and logs are backed up by the same script. The source side (S3_ENDPOINT plus RUSTFS_ROOT_ACCESS_KEY / SECRET_KEY) is already in .env; set a durable S3_BACKUP_TARGET_URL and its S3_BACKUP_TARGET_ACCESS_KEY / _SECRET_KEY to opt in.

backup.sh runs rclone copy — additive, never deleting from the backup, so a source delete can’t wipe it. Pair the target with versioning for point-in-time. (That copy rather than sync has this property was verified live before choosing it here.)

Restore with restore.sh --object-store --confirm-destructive, a reverse mirror that is standalone and does not touch the databases.

Leaving S3_BACKUP_TARGET_URL unset is a deliberate opt-out — but a DB-only restore can’t rebuild a working platform without the blobs.

Secret rotation

There is deliberately NO blind --rotate flag. A naive “regenerate .env” would rewrite passwords out of sync with the running databases and break them — the password in .env must match what the DB actually accepts. Rotate per secret, in order.

Application secrets — the ES256 user-token signing key, the per-service internal signing keys, SECRET_ENCRYPTION_KEY, the alert-relay bearer and the image-registry signing key each have a zero-downtime overlap window and a step-by-step procedure, with verification and rollback, in Secret Rotation. Use pb_rotate_env_secret / pb_finish_env_rotation from deploy/bin/gen-env-secrets.sh rather than hand-editing .env; the SecretRotationPreviousLingering alert fires while a rotation is left half-finished. The datastore credentials below are the ones that have no overlap mechanism.

Secrets with an overlap window

User-token signing key (ES256, platform only). Rotate by kid: publish the incoming key alongside the retiring one, switch signing, then drop the old kid. Nobody is logged out, and no other service needs a restart or a config change because they all read /.well-known/jwks.json. Full procedure.

The stored machine credentials are opaque service-account keys, not JWTs, so the events Lambda and CodeBuild are unaffected — nothing to re-mint. The overlap must outlive the longest refresh token (REFRESH_TOKEN_EXPIRES_IN, 30 days by default) or devices that have not refreshed are signed out.
Per-service internal signing keys (internal service tokens only, stateless). One ES256 key per service, mounted into that service alone. Rotate with deploy/bin/service-signing-keys.sh --rotate <service>; the overlap is the retiring public key staying in the shared bundle. Service tokens live 5 minutes, so the window is short and no session is affected. Roll the public bundle out BEFORE the private key.
Registry signing keypair (jwt-keys.sh). Rotate via a two-cert trust bundle, new cert first, so in-flight registry tokens keep verifying: full procedure. Nothing to re-issue — CodeBuild presents a registry:push service-account key, and the signing keypair only affects the tokens image-registry mints.

Datastore credentials — database first, always

POSTGRES_PASSWORD / DB_PASSWORD. Change the password in Postgres first, then update the secret, then roll:
```
ALTER USER "$POSTGRES_USER" WITH PASSWORD '<new>';
```
→ update the k8s Secret / .env → kubectl rollout restart deploy/postgres and the app deployments. Do not just rewrite .env.
ECOSYSTEM_PUBLIC_READER_PASSWORD (the public plugin directory’s view-only login). Same order:
```
ALTER ROLE ecosystem_public_reader WITH PASSWORD '<new>';   -- as the superuser
```
→ update .env / the postgres-secret key → restart pgbouncer, whose userlist is seeded at startup (compose: docker compose up -d --force-recreate pgbouncer; k8s: kubectl rollout restart deploy/pgbouncer) → restart the plugin service.

The role can only read the two public views, so a leak exposes nothing that isn’t already public — but rotate it like any credential.
MONGO_INITDB_ROOT_PASSWORD + MONGODB_URI. db.changeUserPassword() in Mongo first, then update the secret and URI, then roll.
Mongo keyfile. Requires a rolling restart of the replica set with the new key, since members must share it. Plan a maintenance window.

Always update the k8s Secret — not just .env — on the k8s targets, then kubectl rollout restart the affected Deployments.

What the data tier actually is

Read this before sizing an RPO, because the topology is the constraint:

Store	Topology	Standby / failover	Consequence
Postgres	single instance, `replicas: 1`, `strategy: Recreate`, one RWO volume	none — no hot standby, no replica, no WAL archiving	Losing the volume loses everything written since the last dump. A pod restart is a brief outage; a lost volume is a restore.
MongoDB	single instance, `replicas: 1`, running as a ONE-MEMBER replica set (`rs0`)	none — `rs0` exists so drivers can use transactions and change streams, not for redundancy	Same: one member, one volume, no second copy.
Redis	3 + 3 Sentinel (eks/ec2), single pod (minikube/docker)	Sentinel failover on the AWS targets	Not backed up at all, by design — see below.
Object store (RustFS)	4-pod erasure-coded StatefulSet (eks) · single-node (ec2/minikube/docker)	Pod-fault tolerance on eks only. ec2/minikube/docker rely on the EBS volume or hostPath disk plus snapshots — a real simplification from the old 4-directory MinIO layout on ec2, which was never actual drive-fault tolerance anyway, since all four directories lived on the same EBS volume	Mirrored by `backup.sh` when `S3_BACKUP_TARGET_URL` is set.

Redis HA removed the old data-tier single point of failure; Postgres is now the one that remains, with MongoDB beside it. Neither is replicated on any target, including eks.

RPO and RTO

	Value	Why
RPO, Postgres + Mongo	up to 24 hours	The only scheduled backup is the nightly CronJob at 02:00 UTC. Everything written since the last successful dump is lost.
RPO, Postgres + Mongo, no image built	∞ — total loss	The AWS targets schedule the CronJob by default, but it cannot run until `BACKUP_IMAGE` exists — it fails nightly and `CronJobRunFailed` fires. The local targets schedule nothing at all. Until a dump has actually landed and been restored, the honest RPO is total loss.
Point-in-time recovery	not possible	The dumps are LOGICAL (`pg_dump` / `mongodump`). There is no WAL archiving, no `pg_basebackup`, no oplog tailing — you can restore to a dump boundary and to nothing in between.
RPO, metrics	~2 hours	The Thanos sidecar uploads Prometheus’ TSDB blocks every 2h; the not-yet-uploaded block is lost with the pod.
RPO, logs	minutes	Loki flushes chunks to the object store continuously; the in-pod WAL is an `emptyDir` and is lost with the pod.
RTO	restore time + rollout	There is nothing to fail over TO. Recovery is: provision, restore the dumps, restore the object-store mirror, re-create the secrets, roll the deployments.

Tightening the RPO below a day means either running backup.sh more often — change the CronJob schedule:, it is cheap and the dumps are small — or introducing real replication, which this deployment does not ship.

What is NOT backed up

backup.sh and the CronJob cover exactly three things: the Postgres dump, the Mongo dump, and — only when S3_BACKUP_TARGET_URL is set — an rclone copy mirror of the object-store buckets. Everything below is outside that, and some of it is unrecoverable rather than merely inconvenient.

.env, and every generated key — the largest hole, and the one that is not recoverable by re-provisioning:

Location	Holds
`deploy/<target>/.env`	`SECRET_ENCRYPTION_KEY`
`deploy/<target>/certs/`	The ES256 user-token key, the per-service internal signing keys, the image-registry token keypair, the gateway TLS material
`deploy/<target>/mongodb-keyfile`	The replica-set key

All are gitignored and none is in the backup.

Lose SECRET_ENCRYPTION_KEY and every encrypted column stays encrypted forever — stored AI provider keys, IdP client secrets, TOTP secrets and the SAML SP private keys. A restored database is then partially unreadable even though the dump was perfect.
Lose the user-token signing key and every session ends at once. Recoverable — people sign in again.
Copy .env, certs/ and mongodb-keyfile into your secret manager as part of provisioning, and treat them as part of the backup set. See Secret Rotation.

Also outside the backup:

Prometheus’ local TSDB (--storage.tsdb.retention.time=7d). Not backed up, and does not need to be if the thanos bucket is in the object-store mirror — the sidecar has already uploaded everything older than ~2h. Skip the mirror and you have no metric history at all after a rebuild.
Loki’s log store. Same shape: chunks and index live in the loki object-store bucket and are covered only by the mirror. /loki in the pod is ephemeral scratch.
Grafana (/var/lib/grafana). Datasources are re-provisioned from the grafana-datasources ConfigMap, so those come back; dashboards, users, API keys and annotations created through the UI do not. Keep dashboards in source control if they matter.
Alertmanager state (silences plus the notification log). After a rebuild every silence is gone and previously-notified alerts re-notify once.
Jaeger traces. All-in-one, non-durable by design.
Redis. Deliberately not backed up: it holds BullMQ queues, the durable audit spool, session and step-up state, and idempotency keys. Losing it drops in-flight plugin builds (BullMQ retries what it still has) and any audit events still in the spool that had not flushed to Mongo.
Plugin build scratch and the buildkit layer cache. emptyDir or a named volume; ephemeral by contract, rebuilt on the next build.
The cluster itself. No etcd backup, no manifest snapshot. Recovery is re-running the target’s setup.sh against the restored data, which is the supported path.

DR drill

Periodically restore the latest backup into a scratch namespace or instance and verify — an untested backup is not a backup. For the cheap check you can run often, restore.sh --verify is read-only: it resolves the keys, downloads each archive and runs the same integrity checks a real restore runs, then stops before anything is dropped — so it needs no --confirm-destructive and no scratch database. It answers “is there a restorable archive?”, not “does the data come back correctly”; only a real drill answers the second. A drill that restores only the dumps proves less than it looks: include the object-store mirror and a .env / certs/ restore, or you have not tested the parts that fail hardest.

Object storage (RustFS)

Several stateful services store into RustFS (S3-compatible, on every deploy target), each with its own bucket and a per-service, bucket-scoped key — never the root credentials. All are created by the rustfs-init bootstrap Job, a compose service or a k8s Job.

Bucket	Consumer	Key
`message-attachments`	message service (attachments)	`message-svc`
`registry`	Docker registry (S3 storage driver — now stateless, no PVC)	`registry-svc`
`loki`	Loki (chunks + index; `/loki` is now ephemeral scratch)	`loki-svc`
`thanos`	Thanos sidecar (Prometheus 2h TSDB blocks, long-term) — read back via the store-gateway and querier	`thanos-svc`

See deploy/aws/eks/k8s/rustfs.yaml and each other target’s k8s/rustfs.yaml for the full reasoning.

Topology by target

EKS (production) — distributed RustFS: a StatefulSet of 4 pods, one pb-ebs PVC each, erasure-coded so it tolerates 2 pod or drive losses, spread across nodes via anti-affinity. Clients hit the rustfs Service (round-robin); peers resolve via the rustfs-headless Service. Topology was verified live — the rc CLI, RustFS’s mc equivalent, and Object Lock enforcement both checked against a real container — before the manifest was written.
ec2 (single-node prod-style) — single-node RustFS, one hostPath directory. A deliberate simplification from MinIO’s former 4-directory SNMD layout, which bought bit-rot detection and healing, not drive-fault tolerance: all four directories sat on the same EBS data volume regardless. Durability is unchanged — the EBS volume plus its daily DLM snapshots and DeletionPolicy / UpdateReplacePolicy: Snapshot in template.yaml. True drive or node HA needs separate volumes and nodes, i.e. EKS.
docker / minikube (local dev) — single-node RustFS, single directory, no HA.

Back up the object-store drives as part of DR: EKS the 4 data-rustfs-* PVCs, ec2 rustfs-data, dev ./data/rustfs-data.

Long-term metrics (Thanos) read path

The sidecar only uploads Prometheus’ 2h blocks to the thanos bucket. Querying them back is served by two components — thanos-query.yaml on the k8s targets, equivalent services in docker-compose:

a store-gateway, exposing the archived blocks over the Thanos StoreAPI (gRPC 10901; its local index cache is ephemeral);
a querier, a Prometheus-compatible HTTP endpoint on 9090 that fans out to the sidecar plus store-gateway and de-duplicates.

PROMETHEUS_URL points platform’s Observability query endpoint at the querier (http://thanos-query:9090) so PromQL spans recent plus archived history; set it back to http://prometheus:9090 for recent-only. KEDA autoscaling deliberately still targets Prometheus directly — recent-only, lower latency.

Service mesh (Istio ambient)

All targets run an Istio ambient mesh with STRICT mTLS and identity authz.

kubectl get pods -n istio-system            # istiod, ztunnel, istio-cni Ready
istioctl analyze -n pipeline-builder        # policy sanity
istioctl ztunnel-config workloads           # every pod PROTOCOL=HBONE (enrolled)

Symptom	Cause
A service 403s another	The caller’s `sa/<name>` is missing from the callee’s `AuthorizationPolicy` in `k8s/istio.yaml` — add it and re-apply. Every scraped app service must list `prometheus`; every API must list `nginx`.
Ingress broken after STRICT	The nginx external port is not carved out — `8080` on aws, `8080` + `8443` on local.

Teardown: kubectl delete -k k8s/ removes the mesh policies but leaves istio-system installed; istioctl install is idempotent so re-runs are safe. minikube delete (local/ec2) or eksctl delete cluster (eks) wipe everything.

See Service Mesh for the full troubleshooting table.

Teardown

docker — docker compose down (data persists in data/). Reset with down && rm -rf data/.
minikube — bin/shutdown.sh does a graceful minikube stop: it halts the VM but PRESERVES its disk and the full cluster state (workloads, PVCs, data), so a restart brings everything back with no re-provisioning. It deliberately does NOT delete the namespace or manifests. Bring it back with bin/startup.sh (fast resume plus reconnected port-forwards; no re-install, no re-apply). To wipe instead: minikube delete --profile=pipeline-builder — a clean rebuild is delete then re-run bin/setup.sh, or RECREATE=y bin/setup.sh.
ec2 — bin/shutdown.sh as root removes the iptables DNAT rules, then does a graceful minikube stop, preserving the VM disk and cluster state. bin/startup.sh brings it back. It does NOT touch the EC2 instance — tear that down by deleting the CloudFormation stack. Wipe the cluster with sudo -u minikube minikube delete --profile=pipeline-builder.
eks — shutdown.sh (types the cluster name to confirm; --delete-volumes to also remove the Retained EBS/EFS). Without --domain, eks leaves the ACM cert, Route 53 alias and SES resources behind, and warns that it did.

Minikube data location. Minikube stores all hostPath data — Postgres, MongoDB, RustFS buckets — on the VM’s own persistent /data disk, not the host deploy/local/minikube/data/ folder, which stays empty. Minikube reserves /data for its persistent disk, which shadows a host mount there, and DB data on a 9p mount is unreliable. Data survives minikube stop/start; minikube delete wipes it. For host-side copies use deploy/local/minikube/bin/backup.sh.

Lean deploy (`LEAN=1`)

When the full stack plus the Istio mesh exceeds ~8 vCPU — an ~8-core laptop, or a smaller EC2 instance — LEAN=1 brings up the core stack and mesh only. It omits the optional observability and admin services (prometheus, thanos, loki, promtail, jaeger, alertmanager, mongo-express, pgadmin, grafana, kiali) and collapses every workload to a single replica.

Supported on:

minikube — LEAN=1 deploy/local/minikube/bin/setup.sh
ec2 — at launch via the CFN Lean param (LEAN=1 deploy/aws/ec2/bin/setup.sh, or pipeline-manager infra provision --target ec2 --lean), or on the box with LEAN=1 sudo -E bash deploy/aws/ec2/bin/startup.sh (-E preserves the env through sudo)

It lets ec2 run on a t3.xlarge instead of a t3.2xlarge. Both targets drive the same lean_filter. Full stack is the default for larger machines; eks is unaffected. See Service Mesh: LEAN mode.

Lifecycle scripts (minikube + ec2)

Both single-node targets share a setup.sh / startup.sh / shutdown.sh triad:

Script	Does
`setup.sh`	Provisions — CREATE cluster, install mesh and KEDA, apply manifests
`startup.sh`	The fast resume of a stopped cluster: reconnects port-forwards on minikube, re-mounts host data and iptables on ec2. No re-install, no re-apply
`shutdown.sh`	A graceful `minikube stop`

Sizing overrides. setup.sh (minikube) and startup.sh (ec2) take env-var overrides: DISK_SIZE=60g (VM disk; default 30g minikube, 40g ec2), ISTIO_VERSION=…, LEAN=1.

CPU, memory and disk size are applied only at cluster CREATE. When an existing cluster is found, the scripts resume it with data preserved, unless you ask to recreate: on a TTY they prompt (default: keep), or set RECREATE=y to rebuild non-interactively. On minikube that WIPES /data; on ec2 it rebuilds the cluster but host $DATA_DIR data survives, so clear it to truly wipe. Back up first with backup.sh if needed.

On the docker driver the disk is bounded by Docker Desktop’s virtual-disk limit; eks node disk is managed by the Auto Mode NodeClass, not DISK_SIZE.

Destructive resets print raw one-liners today — dump first with backup.sh before wiping data you might want.

Operator consoles (`/grafana/`, `/kiali/`)

The three kubernetes targets serve two admin consoles through nginx subpaths, the same pattern as /pgadmin/:

Grafana — dashboards over Thanos → Prometheus → Loki → Jaeger. Thanos is the default datasource because it fans out to the object-store blocks that Prometheus alone drops.
Kiali — the Istio mesh console.

Both are dropped by LEAN=1.

Access and authentication

On AWS (ec2, eks) the consoles — and /pgadmin/, /mongo-express/ — are off by default, with their routes returning 404. Turned on with ADMIN_UIS_ENABLED=true, they sit behind an nginx auth_request to platform’s superadmin + AAL2 check (GET /admin/console-check; see AWS: Access Points).

The RustFS object-store console is covered by the same flag, but is served on its own port :9001 at /rustfs/console/ instead of a gateway path — its UI resolves its API endpoints from window.location and points its S3 client at the bare origin — ListBuckets is GET /, objects are /<bucket>/... — so it needs an origin whose whole path space is RustFS’s. :9000 is the S3 data plane and has no UI at all. On ec2 that port is published by a second ALB listener; on eks and the local targets reach it on localhost (kubectl port-forward svc/nginx 9001:9001, or the port-forward minikube’s startup script already creates).

Locally (docker, minikube) nginx applies no auth of its own.

Each carries its own login as well, and both read data with no org scoping — unlike the tenant-facing /dashboard/observability pages, which are org-scoped through the platform’s PromQL proxy. They are therefore admin-only, and nothing tenant-facing links to them.

Kiali additionally runs view_only_mode with read-only RBAC (no create, update or delete verbs anywhere) and auth.strategy: token, so signing in needs a ServiceAccount token:

kubectl -n pipeline-builder create token kiali

Two limits worth knowing

Under Istio ambient without waypoint proxies, Kiali’s graph is L4 only. Measured on a live cluster: ztunnel emits 553 istio_* series and istio_requests_total is zero. So you get who talks to whom, how much, and whether it is mTLS — but no HTTP rates, latency or status codes.
On docker only Grafana ships. That target runs no service mesh, so Kiali would render an empty graph.

On minikube both are also port-forwarded by startup.sh (Grafana localhost:3001, Kiali localhost:20001) and exposed on NodePorts 30300 / 30201.

Provisioned dashboards

Grafana loads every dashboard JSON under config/grafana/dashboards/ (docker: config/grafana/provisioning/dashboards/) into a read-only Pipeline Builder folder. On the kubernetes targets the setup scripts turn that directory into the grafana-dashboards ConfigMap, mounted at /etc/grafana/provisioning/dashboards.

Today that is Plugin ecosystem (uid: plugin-ecosystem): the moderation queue by kind and status, oldest pending request per lane, SLA breaches, manager decisions and latency, auto-approvals per rule, separation-of-duties and Verified-eligibility refusals, Ecosystem Manager headcount, re-sign progress and failures, and ecosystem notice sends and drops.

The file is identical on every target — a platform test enforces it — so edit it in source control, not in the UI. After changing it on kubernetes, re-run the setup script’s ConfigMap step (or kubectl create configmap grafana-dashboards … --dry-run=client -o yaml | kubectl apply -f -) and restart Grafana.

Plugin-ecosystem alerts

In alert-rules.yml on every target. Runbook: Ecosystem Moderation.

Alert	Fires when	Severity
`EcosystemSecurityLaneSLABreach`	a security-fix request has waited over 4 h (5 min)	critical
`EcosystemStandardLaneSLABreach`	a standard request has waited over 48 h (30 min)	warning
`EcosystemResignFailures`	more than 3 image re-signs failed in the last hour	warning
`EcosystemResignStalled`	re-sign jobs are queued but nothing was re-signed for 2 h	warning
`EcosystemApproverShortage`	fewer than 2 system-org members hold `plugins:moderate` (1 h)	warning
`EcosystemNotificationsDropped`	a queued ecosystem notice was given up on after its retries	warning
`SubmissionBacklogHigh`	more than 25 anonymous plugin submissions awaiting gates or moderation for 1 h	warning
`SubmissionGateFailureSpike`	more than 10 anonymous submissions failed the same gate in an hour	warning
`OfficialAutoApprovalAnomaly`	the Official catalog auto-approval rule is near its 50/day cap or far above its usual hourly rate	warning

The ecosystem gauges (ecosystem_requests_pending, ecosystem_requests_sla_breached, ecosystem_approvers, …) are sampled every minute by every plugin replica, so the rules take max() and never read a stale former leader. ecosystem_approvers is read from platform at most every 5 minutes and simply isn’t reported while platform can’t answer, so a platform outage never looks like an approver shortage.

Storage alerts

In alert-rules.yml on the three Kubernetes targets. Fed by two scrape jobs against the kubelet (kubelet and kubelet-cadvisor in prometheus.yml), which need the nodes/metrics RBAC in k8s/prometheus.yaml; egress on :10250 was already granted by allow-kube-api-egress. Docker has no kubelet and ships none of these.

Alert	Fires when	Severity
`PersistentVolumeFillingUp`	a PVC is under 15% free for 15 min	warning
`PersistentVolumeCriticallyFull`	a PVC is under 5% free for 5 min	critical
`PersistentVolumeFillingUpFast`	extrapolating 6 h, a PVC under 40% free runs out within 4 h	warning
`NodeDiskFillingUp`	a node’s root filesystem is under 15% free for 15 min	warning
`NodeDiskCriticallyFull`	a node’s root filesystem is under 5% free for 5 min	critical

Which of these has data depends on the target, and that is a property of the volumes, not a setting. The PersistentVolume rules read kubelet_volume_stats_*, which the kubelet emits only for volumes whose CSI driver implements metrics — so they have data on eks (pb-ebs, ebs.csi.eks.amazonaws.com). On ec2 and minikube the PVs are manual hostPath (storageClassName: "") and the kubelet reports nothing for them; verified live, where the collector had run 20,832 times and produced zero kubelet_volume_stats_* series against two bound PVCs. There, the node-disk rules are the real signal — and the better one, because every hostPath mount (RustFS’s buckets, the Postgres and Mongo data directories, the registry) shares that single filesystem and they all stop together when it fills.

Remedies differ accordingly:

eks — pb-ebs sets allowVolumeExpansion: true, so expansion is online:
```
kubectl -n pipeline-builder patch pvc postgres-data \
  -p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}'
```
Volumes only ever grow: neither EBS nor Kubernetes can shrink one, and EBS refuses a second modification of the same volume for roughly 6 hours — so size with headroom rather than nudging upward. The StatefulSet claims (data-rustfs-{0..3}, data-redis-{0..2}) come from volumeClaimTemplates, which is immutable on a live StatefulSet: patch each PVC and edit the manifest, re-applying with kubectl delete sts <name> --cascade=orphan so the pods survive. Skip the manifest edit and a replacement replica silently comes back at the old size.
ec2 — there is no CSI volume to expand. Grow the instance’s EBS volume (aws ec2 modify-volume), then growpart and resize2fs on the instance itself. Patching a PVC does nothing, because the PV is a directory on that disk.
minikube — minikube stop, grow the VM disk, minikube start; or free space (docker system prune inside the node, old Loki/Thanos blocks).

Scheduled-job alerts

In alert-rules.yml on the three Kubernetes targets. Fed by kube-state-metrics (k8s/kube-state-metrics.yaml), deployed on aws/eks and aws/ec2 only. Docker has no Kubernetes API and ships neither rule; minikube ships the rules but not the exporter, so they are inert there.

Alert	Fires when	Severity
`CronJobRunFailed`	a Job in the namespace reached its `Failed` condition in the last 6 h (5 min)	warning
`CronJobNotScheduled`	an unsuspended CronJob’s next scheduled run is over 2 h in the past (30 min)	critical

Why an exporter was needed at all. A CronJob’s outcome lives in the API server, not in any pod’s /metrics, so nothing in the stack could see it. The two scheduled jobs here each fail in a direction nothing noticed:

db-backup POSTs to Alertmanager when it runs and fails, which leaves the worse case uncovered — a job that is suspended, deleted or never scheduled posts nothing, because nothing is left running to post. DatabaseBackupStale partly covers that through Pushgateway, but only once a backup has succeeded at least once to establish a series that can go stale.
ask-model-up could not self-report at all: its image is registry.k8s.io/kubectl, which is distroless and ships no shell and no HTTP client, so there was nothing to run a curl from. A GPU node that could not be provisioned (no capacity, a vCPU quota) left Ask unavailable for the day with the first signal being a user’s question failing.

Both rules are keyed on CronJobs in the namespace rather than on job names, so anything scheduled later is covered without a new rule.

Suspended CronJobs are deliberately excluded from CronJobNotScheduled. Setting ASK_SCHEDULE_ENABLED=false legitimately applies both ask-model scalers suspended, and paging for a supported configuration is how an alert gets permanently muted.

Least privilege. Unlike the upstream deployment, this one watches a single namespace (--namespaces=pipeline-builder), which makes a namespaced Role sufficient rather than a cluster-wide ClusterRole over every object kind. It is granted list/watch only, and not secrets or configmaps — kube-state-metrics does not export their values but does export their names and metadata, and Grafana here applies no org scoping. The --resources flag and the Role must stay in exact lockstep: a resource watched without permission logs failed to list on every resync and its metric family is silently absent, so an alert built on it never fires. A contract test enforces both the lockstep and that every kube_* family the rules read is actually exported.

kube-state-metrics itself is covered by ServiceDown — it is discovered by the same kubernetes-pods scrape job as everything else, so if it stops answering, that fires.

Automatic PVC expansion (eks only)

With PVC_AUTOEXPAND_ENABLED=true, Alertmanager also POSTs PersistentVolumeFillingUp / PersistentVolumeCriticallyFull to platform’s /observability/pvc-autoexpand, which raises the claim’s requested size. The route is additive (continue: true), so the ops-team Slack notification still happens — automation must never be the only thing that knows a volume is filling.

Off by default, and eks-only. Expansion is irreversible: neither EBS nor Kubernetes can shrink a volume. ec2 and minikube bind manual hostPath PVs with no CSI driver to resize, and the endpoint refuses a claim with no storageClassName rather than issuing a patch the API server would accept and silently ignore.

Setting	Default	Meaning
`PVC_AUTOEXPAND_ENABLED`	`false`	Arms the expander
`PVC_AUTOEXPAND_STEP_PERCENT`	`50`	Growth per expansion, rounded up to a whole GiB
`PVC_AUTOEXPAND_CEILING_MULTIPLE`	`4`	Hard ceiling, as a multiple of the original request
`PVC_AUTOEXPAND_MAX_GI`	`500`	Absolute cap whatever the multiple says
`PVC_AUTOEXPAND_COOLDOWN_SECONDS`	`21600`	Minimum gap per claim (EBS refuses a second modify for ~6 h)

The ceiling is measured from the size pinned in pipeline-builder.io/autoexpand-original on first expansion, so it cannot compound. On reaching it expansion stops and PvcAutoExpandAtCeiling fires (critical): a volume that keeps growing past 4× is usually an unarchived WAL or a log loop, not real demand, and the right response is to find the writer.

Operational properties worth knowing:

Fails closed without Redis. The cooldown and the cross-replica lock are one SET NX EX; with no Redis there is neither, and the failure mode would be repeated EBS modification attempts on every repeat_interval.
Least privilege. Platform holds a namespaced Role (get, patch on persistentvolumeclaims in pipeline-builder only) — not a ClusterRole, and no create/delete: deleting a Retain-policy claim would orphan a billing EBS volume.
StatefulSet claims are expanded too, and that leaves the manifest behind, because volumeClaimTemplates is immutable. PvcAutoExpandTemplateDrift fires so it cannot rot silently — update rustfs.yaml / redis.yaml and re-apply with --cascade=orphan.

Automatic data-volume expansion (ec2 only)

The ec2 counterpart, and a different actuator because there is no claim to patch: this target binds manual hostPath PVs, so the databases, the RustFS buckets and the registry all share one EBS volume (/dev/xvdf, whole-disk ext4 labelled pipeline-data, mounted at /opt/pipeline). With NODE_DISK_AUTOEXPAND_ENABLED=true, NodeDiskFillingUp also POSTs to platform’s /observability/node-disk-autoexpand, which calls ec2:ModifyVolume and then runs an SSM document to grow the filesystem.

Why SSM, and why it is not a back door. resize2fs needs root on the host, which no pod has. The instance role is granted ssm:SendCommand on exactly one document — <stack>-resize-data-fs, which takes no parameters and resizes a fixed device — and on exactly one instance. A workload that reaches IMDS and steals instance credentials can therefore trigger that one resize and nothing else. A grant on AWS-RunShellScript, or a parameterised document, would hand it arbitrary root commands on a box that also builds untrusted plugins.

ec2:ModifyVolume is additionally conditioned on the pipeline-builder-data-volume tag, so the instance cannot grow — or bill for — any other volume in the account, including its own root volume.

There is no growpart: UserData formats the volume whole-disk with no partition table, so the filesystem starts at sector 0 and resize2fs alone is correct. That omits the step most likely to destroy data. The document does wait for the EBS modification to settle first — resize2fs against a volume still being optimized grows to the old size and reports success.

The volume is found by asking IMDS for this instance’s id and filtering DescribeVolumes on attachment plus the data-volume tag, rather than by a configured name: the volume’s Name tag interpolates the domain, and a tag-value lookup could match a different stack’s volume in the same account.

Ceiling, cooldown and fail-closed-without-Redis behave exactly as the eks path, with NodeDiskAutoExpandAtCeiling (critical) and NodeDiskAutoExpandFailing watching the actuator. The ceiling is measured from the size first seen, recorded as a tag on the volume so it cannot compound.

Blast radius is larger than anything on eks, which is why it ships off: this is the single disk the whole deployment runs on.

Plugin uploads are accepted, not built

load-plugins.sh uploads each plugin and gets HTTP 202 Accepted. The image build, the cosign signature, the grype scan and PLUGIN_VULN_GATE all run after that, and the version row is written only once they pass — so a plugin whose build is blocked leaves no version at all. It is not listed as broken; it is simply absent from the catalog.

So a successful load means every upload was accepted, not that every plugin is usable. Check the builds separately.

The loader used to poll until every accepted plugin had a version and fail the load otherwise. That was removed: it could not distinguish a build that FAILED from one that had not finished. Loading the full catalog onto a cold cluster lands builds at roughly one a minute, so any fixed budget expires mid-run, and the loader declared ~100 healthy plugins failed while naming PLUGIN_VULN_GATE as the likely cause — a confident wrong diagnosis, and a red that carried no information.

Build outcomes are reported where they are actually known:

the Builds page (Queue / Failed-Triage) groups failures by category and shows the CVE and the version that fixes it;
the per-org security notifications N30/N31 name a blocked version;
the plugin service log carries the same detail.

This is a stage of init, not all of it. A blocked build leaves the admin user, the pipeline templates and the compliance rules in place — the deploy banner says INITIALIZATION INCOMPLETE and points at the stage that stopped, rather than claiming nothing loaded.

AWS Deployment — deploying to EC2 and EKS in the first place
Secret Rotation — the per-secret procedures with overlap windows
Service Mesh — the full mesh troubleshooting table and LEAN mode
Environment Variables — every configuration variable
Ecosystem Moderation — the queue the alerts above watch