Software Factories on AWS

Self-hosted platform for standing up and running software factories on AWS: complete DevSecOps environments with secure delivery pipelines, supply-chain controls, compliance evidence and multi-tenant governance — in minutes, not months.

AWS Deployment

Two targets: EC2 (a single Minikube instance) or EKS (managed Kubernetes, Auto Mode).

Highlights

Overview

This guide is for operators standing up Pipeline Builder on AWS. It covers both targets in either public (internet-facing ALB) or private (internal ALB, the default) mode, plus email via SES, platform initialization, execution reporting and drift detection.

Both targets deploy the full stack: app services, databases, observability (Prometheus + Loki, surfaced in-app) and admin tools.

What you get for observability

Five dashboards — Platform Overview, Plugin Builds, Queue Health, Registry Activity, Audit Activity — are seeded into the database at platform cold start as public rows owned by the system org (org_id = the configured SYSTEM_ORG_ID, default 000000000000000000000001), so they appear automatically for any logged-in org at /dashboard/observability/<id>.

Who Sees
Org member Plugin Builds, and the org-scoped part of Platform Overview
Org admin The above plus Audit Activity, reading the MongoDB audit trail scoped to their org
System admin Everything, including Queue Health and Registry Activity

Panels backed by fleet-wide queries are shown only to system admins, and a dashboard with no panel the caller can render is hidden entirely. Audit Activity also has a dedicated page at /dashboard/observability/audit-activity.

How it works

  1. Pick a target and a mode — EC2 or EKS; public or private. Both need --domain and --hosted-zone-id.
  2. Provision. pipeline-manager infra provision is recommended; bin/setup.sh or raw CloudFormation work too. It requests a DNS-validated ACM cert and fronts the private compute with an ALB.
  3. The cert validates mid-deploy, then instances or pods pass health checks a few minutes later.
  4. The URL is https://<your-domain> — publicly resolvable in public mode, in-VPC only in private mode.
  5. Initialization registers the first admin and, with the right flags, loads the plugin, compliance and sample catalogs.
  6. Post-deploy wiring makes AWS-specific things work — the stored service credentials and the EventBridge reporting path.
  7. Execution events flow back to the reporting service, which is what fills the Reports page and DORA.

Configuration

  1. Have a domain in a public Route 53 hosted zone. Without it there is no cert and no deployment.
  2. Choose the mode. Private (default) puts the ALB inside the VPC; public makes it internet-facing. See Deployment modes.
  3. Provision with infra provision, or the target’s bin/setup.sh. Use --plan first to see what will happen.
  4. Decide about email. --email is the default on both AWS targets and provisions the whole SES path; --no-email opts out. See Email (SES).
  5. Complete the post-deploy steps — they are not optional: store the three service credentials, then deploy the EventBridge reporting infrastructure.
  6. Verify at the access points, and turn the admin consoles on deliberately: they are off by default on AWS and sit behind a superadmin + AAL2 check when enabled.
  7. Wire a backup before you need one — nothing is scheduled by default. See Deploy Operations.

Table of Contents

  EC2 EKS
Runtime Minikube on EC2 EKS Auto Mode (Karpenter-scaled EC2 nodes)
Infra 1 CloudFormation stack eksctl cluster + Kubernetes manifests
TLS ACM cert at the ALB ACM cert at the ALB Ingress
Public surface ALB only (instance private) ALB Ingress only (nodes private)
Storage hostPath PVCs on EBS EBS (RWO) + EFS (RWX) via CSI
Scaling Vertical (instance resize) Horizontal (Karpenter nodes + pod autoscaling)
Cost ~$610-680/mo at the default m5.4xlarge; ~$145-215 on t3.xlarge ~$850-950/mo measured, 24/7
Best for Dev/staging Production

AI-assisted install (infra provision)

The recommended way to install the platform is pipeline-manager infra provision. It picks the target, runs prerequisite checks (AWS CLI + working credentials for EC2/EKS — plus kubectl, openssl, and envsubst for EKS (eksctl is auto-installed by setup.sh); Docker etc. for local), assembles the exact, validated setup.sh command (secrets masked, missing inputs reported rather than guessed), prints the plan, and then deploys it — gated by confirmation prompts (--yes to auto-accept for CI; --json prints the plan and runs nothing). With an AI key configured it also parses a natural-language goal and diagnoses CloudFormation failures.

npm install -g @pipeline-builder/pipeline-manager

# Deploy (shows the plan, then confirms; add --yes for non-interactive CI):
pipeline-manager infra provision --target eks \
  --domain pipeline.example.com --hosted-zone-id Z123 --ghcr-token ghp_xxx --email

# Inspect the plan as JSON without running anything:
pipeline-manager infra provision --target eks --json \
  --domain pipeline.example.com --hosted-zone-id Z123 --ghcr-token ghp_xxx --email

# Or describe the goal (needs an AI key — see Environment Variables):
pipeline-manager infra provision --prompt "deploy to EKS in us-east-1 with email enabled"

# Diagnose a failed deploy:
pipeline-manager infra provision --target eks --diagnose ./stack-events.txt

Always deploys (gated). infra provision checks, assembles, prints the plan, and runs the deploy — it refuses on failed prerequisites or missing inputs, asks for confirmation before deploying (--yes auto-accepts for CI), streams the deploy to your terminal, then verifies /health + /ready on the application URL. On the AWS targets the deploy self-inits (EC2 on first boot; EKS in setup.sh’s final phase), so infra provision surfaces it rather than running it separately; on local/minikube infra provision runs init-platform for you. --json is the only non-executing mode — it prints the plan and exits (for tooling).

On failure it troubleshoots. It matches known CloudFormation signatures and prints the likely cause + fix — and for a few it can auto-fix and retry (e.g. an existing SES identity → re-run with --skip-ses-identity; an ACM/DNS-propagation timeout → resume). Retries are gated and bounded by --retries <n> (default 1; the scripts are idempotent so a re-run resumes). With an AI key it adds a free-form diagnosis on top. When SES is enabled, a successful deploy prints DKIM/sandbox next-steps.

Flags: --yes auto-approves (CI), --retries <n> auto-fix/retry budget, --init <mode> controls post-deploy initialization (auto default / manual / skip — see below), --skip-ses-identity for an already-verified SES domain, --stack-name <name> (EC2) / --cluster-name <name> (EKS) to deploy/manage a second environment.

Init mode (--init <mode>). One flag controls how the platform initializes after deploy:

  • auto (default) — init-platform runs once the platform is up, registering the admin (with the default password) and loading plugins/compliance/samples. The AWS targets self-run it as part of the deploy: EC2 on first boot (UserData → on the box as the minikube user — watch with aws ssm start-session … && sudo tail -f /var/log/user-data.log); EKS in setup.sh’s final phase, reaching the cluster over a kubectl port-forward. local/minikube run it from infra provision.
  • manual — don’t init; infra provision surfaces the exact step for you to run yourself (do this to set real admin credentials PLATFORM_IDENTIFIER/PLATFORM_PASSWORD instead of the default).
  • skip — don’t initialize at all (no register, no loads).

Teardown. Add --teardown to remove a deployment. local/minikube stop the stack (on-disk / PVC data persists). EC2 DELETEs its CloudFormation stack and EKS runs bin/shutdown.sh (deletes the cluster, EFS, ACM cert + Route 53 alias) — both irreversible — so the destructive path is gated harder than deploy: you must type the resource id to confirm (a y/N is too easy to fat-finger), and --yes alone does not bypass it — only --force does (for CI). When you pass a custom --stack-name <name> (EC2) or --cluster-name <name> (EKS), the confirmation binds to that name — you type the stack/cluster name, not the target id, so a wrong name can’t be confirmed by habit. The region comes from --region / AWS_REGION. As always, bin/shutdown.sh (local/minikube/EKS) and aws cloudformation delete-stack (EC2) can be run directly.

# Teardown — prints the destroy plan, then prompts (type "eks" to confirm):
pipeline-manager infra provision --target eks --teardown

Bootstrap a fresh machine (--repo). Without a checkout, --repo git-clones the platform repo first and runs from it. The clone is sparse + partial — git clone --filter=blob:none --no-checkout + cone sparse-checkout (git ≥ 2.27; older git falls back to a full clone) — so it materializes only the deploy folders the selected target + options need, not the whole repo (packages/, api/, frontend/, … are never downloaded). The common base is just deploy/bin; each target adds its own folder (deploy/local/docker, deploy/local/minikube — self-contained — deploy/aws/ec2, deploy/aws/eks), and each post-install load adds its folder. Re-syncs are additive (sparse-checkout add), so one --workdir can accumulate multiple targets. Override with --repo <url>, --ref <branch|tag>, --workdir <dir>. (--ref is a branch/tag; arbitrary SHAs may not fetch under the shallow clone.)

Post-install steps. After deploy + health, infra provision registers the admin (non-interactive with --admin-email/--admin-password, which set PLATFORM_IDENTIFIER/PLATFORM_PASSWORD) and runs opt-in loads — each also pulls its folder into the sparse clone: --with-plugins (build + load plugins; adds deploy/plugins + deploy/codebuild), --with-compliance (deploy/compliance), --with-samples (deploy/samples), or --with-all. Also --build-bootstrap (CodeBuild bootstrap image), --with-smoke-test (read-only API check), --with-events (EC2/EKS event ingestion — a four-step bundle: three infra store-token runs writing the platform, registry:push and reporting:ingest service-account keys to Secrets Manager at the pipeline-builder/{orgId}/… pattern, then infra setup-events --scoped-ingest deploying the EventBridge → SQS → Lambda that reads the ingest key; all pull AWS creds from the standard env / ~/.aws chain, and need PLATFORM_PASSWORD for the step-up each key write requires), and repeatable --post-step "<cmd>". The default is register-only (minimal clone); the loads are deterministic + idempotent, so re-running with more options just layers them on. On the AWS targets these loads run deploy-side by default (so infra provision doesn’t prompt for them locally) — EC2 on first boot, EKS in setup.sh’s final phase over a kubectl port-forward. Pass --init manual to drive them yourself.

# Fresh box → sparse-clone just deploy/bin + deploy/local/docker, deploy, register, load samples:
pipeline-manager infra provision --target docker --repo --with-samples --yes \
  --admin-email admin@acme.com --admin-password 's3cret'

The underlying bin/setup.sh scripts remain the source of truth and can always be run directly — the rest of this guide documents them.

On EKS, bin/setup.sh runs ten phases and accepts --from-phase / --to-phase / --only-phase, so a re-deploy does not have to re-walk the ~25 minutes of cluster, EFS and ACM work in phases 1-3. bin/startup.sh is the shorthand for phases 4-8 — secrets, IAM, operators, workloads and the Route 53 alias — which is what a config change, a new image tag or an edited manifest actually needs, and it gives EKS the startup.sh the EC2 and minikube targets already have. It does not run init-platform.sh or the smoke checks; both are separate scripts and neither follows from re-applying config.

Service mesh (Istio ambient)

Both AWS targets run an Istio ambient service mesh (STRICT mTLS + identity-based L4 authorization between all services). bin/setup.sh (EKS) / bin/startup.sh (EC2) install it right after KEDA; policies live in k8s/istio.yaml. See Service Mesh for the full model. AWS-target specifics:

Deployment modes (public vs private)

Either target (EC2 or EKS) deploys in one of two modes. Both put the compute in private subnets and terminate TLS at an ALB with a publicly-trusted, DNS-validated ACM cert — so both require --domain + --hosted-zone-id (the public Route 53 zone is where ACM validates the cert). The mode flips only the ALB scheme and the DNS record:

  private (inside-AWS-only, default) public
ALB scheme internal, private subnets internet-facing, public subnets
Compute (instance / tasks) private subnet, no public IP private subnet, no public IP
DNS Route 53 private zone alias → internal ALB public Route 53 alias → ALB
Reachable from inside the VPC (peered / VPN / Direct Connect) the public internet
CodeBuild VPC-attached (PIPELINE_VPC_ID / SUBNET_IDS / SECURITY_GROUP_IDS) AWS-managed network, reaches the ALB over the internet
Plugin pull https://<domain>/v2/ (resolves in-VPC) https://<domain>/v2/ (public)

In private mode, EC2 folds the VPC interface endpoints (S3, Logs, Secrets Manager, KMS, STS, CodeBuild, ECR) and the Route 53 private-zone alias to the internal ALB into its single stack, gated on DeployMode=private (no separate prereqs stack). EKS sets the ALB Ingress to scheme: internal and aliases the domain to it; eksctl provisions the cluster VPC (public + private subnets, NAT). Both request a DNS-validated ACM cert and alias the domain to the ALB (public alias or private zone). For VPC-attached CodeBuild plugin pulls, supply PIPELINE_VPC_ID / SUBNET_IDS and build-dependency egress (NAT / internal mirrors).

Use the matching quickstart below: Public or Private.

Public deployment (quickstart)

A public deployment uses an internet-facing ALB so the dashboard, API, and plugin registry are reachable from the internet over HTTPS (the compute still stays private behind it). See Deployment modes for the full comparison.

Prerequisites (both targets)

1. Deploy in public mode

Pick the target. Both take --deploy-mode public; everything else matches the private flow.

# EC2 — single Minikube instance behind an internet-facing ALB
cd deploy/aws/ec2
bash bin/setup.sh --deploy-mode public \
  --key-pair my-keypair \
  --domain pipeline.example.com \
  --hosted-zone-id Z1234567890 \
  --ghcr-token ghp_xxxxxxxxxxxx

# EKS — managed Kubernetes (Auto Mode) behind an internet-facing ALB Ingress
cd deploy/aws/eks
bash bin/setup.sh --deploy-mode public \
  --domain pipeline.example.com \
  --hosted-zone-id Z1234567890 \
  --ghcr-token ghp_xxxxxxxxxxxx

The ACM cert DNS-validates during the deploy, so expect setup.sh to wait a few minutes for the cert to reach ISSUED (EC2: while CloudFormation is CREATE_IN_PROGRESS; EKS: aws acm wait certificate-validated). setup.sh runs from your machine with your credentials. The EC2 Deploy section also shows the raw-CloudFormation equivalent.

2. Get the URL

The URL is simply https://<your-domain> (the value you passed to --domain), reachable once the Route 53 alias resolves and the target(s) pass health checks — a few minutes after the stack completes, while the instance bootstraps / tasks start. To read it back from the stack outputs:

# EC2 — ApplicationURL output is the full https:// URL
aws cloudformation describe-stacks --stack-name pipeline-builder \
  --query 'Stacks[0].Outputs[?OutputKey==`ApplicationURL`].OutputValue' --output text

# EKS — the URL is https://<your-domain> (the Route 53 alias setup.sh creates → the ALB Ingress).
# Confirm the ALB hostname the Ingress was assigned:
kubectl get ingress pb-ingress -n pipeline-builder \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

3. Initialize the platform

Public install is otherwise identical to private — by default the admin user is registered and plugins loaded automatically (EC2 self-inits on first boot; EKS self-inits in setup.sh’s final phase over a kubectl port-forward). To do it yourself, deploy with --init manual and follow Post-Deploy Steps.

Note: “public” exposes only the ALB. The instance/nodes have no public IP and no inbound SSH; admin access is still SSM Session Manager (EC2) or kubectl (EKS). To make a deployment internal-only later, redeploy with --deploy-mode private (default). See Deployment mode (DEPLOY_MODE) for the full mode comparison.

Private deployment (quickstart)

A private deployment uses an internal-scheme ALB — reachable only from inside your AWS network (the VPC, peered VPCs, or via VPN / Direct Connect), never the public internet. This is the default mode. See Deployment modes for the full comparison.

Prerequisites (both targets)

Identical to the Public quickstart above: AWS CLI, a registered domain + public Route 53 hosted zone, a GitHub PAT (--ghcr-token), and — EC2 only — an EC2 key pair.

The public Route 53 hosted zone is still required even in private mode: ACM validates the cert via a public DNS record. The private hosted zone (for in-VPC resolution of your domain) is created automatically by the stack — you don’t supply it.

1. Deploy in private mode

private is the default, so --deploy-mode private is optional (shown for clarity). Same flags as public, minus the public exposure.

# EC2 — single Minikube instance behind an internal ALB
cd deploy/aws/ec2
bash bin/setup.sh --deploy-mode private \
  --key-pair my-keypair \
  --domain pipeline.example.com \
  --hosted-zone-id Z1234567890 \
  --ghcr-token ghp_xxxxxxxxxxxx

# EKS — managed Kubernetes (Auto Mode) behind an internal ALB Ingress
cd deploy/aws/eks
bash bin/setup.sh --deploy-mode private \
  --domain pipeline.example.com \
  --hosted-zone-id Z1234567890 \
  --ghcr-token ghp_xxxxxxxxxxxx

In private mode, EC2 also folds the VPC interface endpoints (S3, Logs, Secrets Manager, KMS, STS, CodeBuild, ECR) and the Route 53 private-zone alias to the internal ALB into its single stack (gated on DeployMode=private, no separate prereqs stack). EKS sets the ALB Ingress to scheme: internal and aliases the domain to it. Either way the ACM cert still DNS-validates during the deploy, so expect a few minutes of waiting for it to issue.

2. Get the URL

The URL is the same https://<your-domain>, but it resolves only from inside the VPC (via the private hosted zone) — it will not resolve from your laptop or the public internet. For EC2, read it back from the stack output (ApplicationURL); for EKS the URL is https://<your-domain> from the Route 53 alias, and you can confirm the ALB hostname as in the public step 2.

3. Initialize the platform

By default this happens automatically. EC2 runs init-platform.sh ec2 on first boot (as the minikube user, in-VPC); EKS runs init-platform.sh eks from setup.sh’s final phase over a kubectl port-forward to svc/nginx. Both register the admin (default password) and load plugins/compliance/samples — watch EC2 with sudo tail -f /var/log/user-data.log (after SSM).

If you deployed with --init manual (or want to re-run / set real admin creds):

Then load plugins per Post-Deploy Steps.

Note: private mode also wires CodeBuild into the VPC (PIPELINE_VPC_ID / SUBNET_IDS from the foundation VPC) so it can reach the internal ALB and pull plugin images over https://<domain>/v2/. You still supply egress (NAT / package mirrors) for build dependencies. See Deployment mode (DEPLOY_MODE) for the full comparison.

EC2

Single hardened EC2 instance running Minikube with all services.

Prerequisites

Deploy

For the one-command happy path, use the Public or Private quickstart — both run bin/setup.sh from your machine with your credentials (so the instance role needs no CloudFormation permissions). An ALB fronts the always-private instance and terminates TLS with an ACM cert the template DNS-validates against your zone, so --domain + --hosted-zone-id are required; setup.sh refuses to start without them.

Manual alternative (raw CloudFormation). Deploys the same single stack — nothing to follow up with. The ACM cert DNS-validates during stack creation, so expect a few minutes in CREATE_IN_PROGRESS:

cd deploy/aws/ec2

aws cloudformation deploy \
  --stack-name pipeline-builder \
  --template-file template.yaml \
  --parameter-overrides \
    DeployMode=private \
    DomainName=pipeline.example.com \
    HostedZoneId=Z1234567890 \
    KeyPairName=my-keypair \
    GhcrToken=ghp_xxxxxxxxxxxx \
  --capabilities CAPABILITY_IAM
# (DeployMode=public for an internet-facing ALB; domain + zone still required.)

Get the URL:

aws cloudformation describe-stacks --stack-name pipeline-builder \
  --query 'Stacks[0].Outputs[?OutputKey==`ApplicationURL`].OutputValue' --output text

Parameters

Parameter Required Default Description
KeyPairName Yes — EC2 key pair (serial-console/break-glass; routine access is via SSM)
GhcrToken Yes — GHCR token for pulling images
DomainName Yes — FQDN — ACM cert + Route 53 alias to the ALB
HostedZoneId Yes — Public Route 53 zone ID (ACM DNS validation + alias)
InstanceType No m5.4xlarge EC2 instance type (16 vCPU / 64 GiB) — the smallest size on which every HPA can reach maxReplicas alongside the mesh and the self-hosted ask-model. t3.2xlarge (8 vCPU / 32 GiB) runs the stack at steady state with less headroom; use t3.xlarge only with Lean=true.
Lean No false When true, omit the optional observability/admin services + single-replica everything so the core stack + mesh fits a t3.xlarge. Pair with InstanceType=t3.xlarge. See Service Mesh: LEAN mode.
EbsVolumeSize No 60 Root volume size in GiB (OS, binaries)
DataVolumeSize No 500 Data volume size in GiB (/opt/pipeline, gp3 encrypted) — Docker, plugins, registry, databases. Minimum 440, which is 2x what the stack commits to this volume: 140 GiB of declared PersistentVolume claims (7 services x 20 GiB) plus the 80 GiB minikube node disk, which lives here because Docker’s data-root does.
GitRepo No (this repo) Git repository URL
GitBranch No main Branch to deploy

Storage Requirements

The EC2 deployment uses two EBS volumes:

Volume Default Mount Contents
Root 60 GiB / OS, Docker/minikube binaries, app code
Data 500 GiB /opt/pipeline Docker layers, plugin artifacts, registry, databases, logs

Data volume breakdown:

Component build_image prebuilt prebuilt + –cleanup
Docker build cache + images 20-30 GB 60-90 GB 60-90 GB
Plugin artifacts (image.tar + plugin.zip) 0 GB 130-190 GB 0 GB
Registry (pushed images) 40-60 GB 40-60 GB 40-60 GB
PostgreSQL + MongoDB 5-15 GB 5-15 GB 5-15 GB
Minikube + logs + metrics 15-25 GB 15-25 GB 15-25 GB
Total 80-130 GB 250-380 GB 120-190 GB

The figures above are OBSERVED usage. The volume is sized from what the stack commits instead, at a 2x minimum — 140 GiB of declared PV claims (every hostPath in k8s/ is under /opt/pipeline/pipeline-data, each claiming 20 GiB) plus the 80 GiB minikube node disk, so the floor is 440 GiB. Postgres and MongoDB using 5-15 GB today does not lower it: the claims are what a CSI would enforce if this target ever stopped using hostPaths, and the node disk is a real file on this volume either way.

Recommendations:

Plugin strategy Data volume Notes
build_image (default) 440 GB The floor. Builds from Dockerfile at upload time
prebuilt with --cleanup 500 GB Removes artifacts after upload
prebuilt without cleanup 700 GB Keeps 130-190 GB of artifacts for re-runs

Daily runtime operations (after initial plugin load) add ~1-5 GB/month from database growth and logs. Add a weekly Docker prune cron to reclaim build cache:

# /etc/cron.weekly/docker-prune
docker system prune -af --filter "until=168h"

Expanding EBS Volume

If you need more storage after deployment (e.g., switching to prebuilt), expand the data volume live — no reboot required:

# 1. Find the data volume ID
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(curl -s -X PUT \
  http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600')" \
  http://169.254.169.254/latest/meta-data/instance-id)

VOL_ID=$(aws ec2 describe-volumes \
  --filters "Name=attachment.instance-id,Values=$INSTANCE_ID" "Name=tag:Name,Values=*data*" \
  --query 'Volumes[0].VolumeId' --output text)

# 2. Expand to desired size (e.g., 500 GiB for prebuilt)
aws ec2 modify-volume --volume-id $VOL_ID --size 500

# 3. Wait for modification to complete (~30s)
watch -n5 "aws ec2 describe-volumes-modifications --volume-ids $VOL_ID \
  --query 'VolumesModifications[0].ModificationState' --output text"
# Wait until it shows "optimizing" or "completed"

# 4. Grow the partition and filesystem (on the EC2 instance)
DEVICE=$(lsblk -no PKNAME $(findmnt -n -o SOURCE /opt/pipeline))
PART=$(lsblk -no PARTNUM $(findmnt -n -o SOURCE /opt/pipeline) 2>/dev/null)
[ -n "$PART" ] && sudo growpart /dev/$DEVICE $PART
sudo xfs_growfs /opt/pipeline    # XFS filesystem
# or: sudo resize2fs $(findmnt -n -o SOURCE /opt/pipeline)   # ext4 filesystem

# 5. Verify
df -h /opt/pipeline

Deploy with a larger data volume upfront:

aws cloudformation deploy \
  --stack-name pipeline-builder \
  --template-file template.yaml \
  --parameter-overrides \
    DataVolumeSize=500 \
    KeyPairName=my-key \
    GhcrToken=ghp_xxx \
  --capabilities CAPABILITY_IAM

What Happens

  1. CloudFormation creates the VPC (2 AZs: public + private subnets), NAT gateway, ALB + ACM cert, security groups, the private EC2 instance, and the Route 53 alias to the ALB
  2. EC2 UserData clones the repo and runs bootstrap.sh, which:
    • Updates OS, installs fail2ban, disables SSH password auth
    • Installs Docker, Minikube, kubectl
    • Generates .env with random secrets (JWT keys, DB passwords)
    • Runs startup.sh — creates Minikube (fresh, no cluster yet), installs the mesh/KEDA, deploys all K8s manifests
    • Sets one iptables bridge: instance :30080 → Minikube NodePort 30080 (the ALB target). TLS is terminated at the ALB (ACM) — no cert on the box.

First boot always CREATES a fresh cluster — startup.sh finds no existing profile, so it never prompts (the RECREATE prompt is TTY-gated and only reached on an existing cluster; a non-interactive UserData run has no TTY). If you later SSH onto the box and re-run startup.sh against a running/stopped cluster, it resumes by default; force a cluster rebuild with RECREATE=y sudo -u minikube deploy/aws/ec2/bin/startup.sh. The EC2 data lives on the host ($DATA_DIR), so a rebuild re-mounts the same data — to truly wipe, clear $DATA_DIR on the host first.

Post-Deploy

The instance is private (no public IP); use SSM:

# Watch bootstrap progress
aws ssm start-session --target <instance-id>   # then: sudo tail -f /var/log/user-data.log

# Check pods
aws ssm start-session --target <instance-id>   # then: sudo -u minikube kubectl get pods -n pipeline-builder

The ALB target reports unhealthy (503) until the instance finishes bootstrapping Minikube + services — expected; it self-heals.

Scripts

All in deploy/aws/ec2/bin/. On the instance the repo is checked out under the data volume, so the scripts live at /opt/pipeline/pipeline-builder/deploy/aws/ec2/bin/.

Script Purpose Run as
setup.sh Deploy the stack (private mode folds endpoints + private zone into it) — from your machine operator
bootstrap.sh Full EC2 setup (runs automatically via UserData) root
startup.sh Start Minikube + deploy K8s manifests + the ALB-target iptables bridge root (sudo)
shutdown.sh Stop Minikube + remove iptables rules root (sudo)

The instance has no public IP / no SSH — connect with SSM Session Manager first:

aws ssm start-session --target <instance-id>   # then, on the instance:

# Start (after bootstrap or reboot)
sudo bash /opt/pipeline/pipeline-builder/deploy/aws/ec2/bin/startup.sh

# Stop
sudo bash /opt/pipeline/pipeline-builder/deploy/aws/ec2/bin/shutdown.sh

# Check pod status
sudo -u minikube kubectl get pods -n pipeline-builder

Security

TLS

TLS is terminated at the ALB with an ACM certificate the template requests and DNS-validates against HostedZoneId. ACM auto-renews it; there is no certbot/Let’s Encrypt and no cert on the instance. nginx serves plain HTTP behind the ALB.

Deployment mode (DEPLOY_MODE)

See Deployment modes for the public/private comparison and what each changes. DEPLOY_MODE defaults to private; pass --deploy-mode public (or DEPLOY_MODE=public) for the internet-facing posture. The instance is always private and TLS is always ACM-at-the-ALB regardless of mode; the private-mode VPC endpoints + private-zone alias are folded into the single stack (gated on DeployMode=private) — no separate prereqs stack.

DEPLOY_MODE and the VPC identity (PIPELINE_VPC_ID / PIPELINE_SUBNET_IDS) are injected into the instance .env automatically by bootstrap.sh (exported from the template’s UserData, from the stack’s VPC + private subnets) — and passed through to the first-boot init — so the synthesized CodeBuild attaches to the VPC and init-platform.sh’s private-mode preflight passes with no manual step. (If you run init-platform.sh by hand on the box, the values are already in .env.)

Teardown

aws cloudformation delete-stack --stack-name pipeline-builder
aws cloudformation wait stack-delete-complete --stack-name pipeline-builder

EKS

Managed Kubernetes on Amazon EKS Auto Mode — AWS-managed, Karpenter-scaled EC2 nodes with the AWS Load Balancer Controller, EBS CSI, and CoreDNS built in (EFS CSI added by the deploy). One orchestrator script stands up the cluster and applies the same Kubernetes workloads as the minikube/ec2 targets, tuned for multi-node (PVC storage, ALB Ingress).

Why EKS Auto Mode? Plugin/base images are built with rootless BuildKit, which needs an unconfined seccomp profile to create its user namespace — only possible on EC2-backed Kubernetes nodes (securityContext.seccompProfile: Unconfined). Auto Mode keeps node management hands-off while running on EC2, so BuildKit works and the proven k8s manifests are reused as-is.

Prerequisites

Deploy

Use the Public or Private quickstart for the one-command path — bin/setup.sh runs all phases end to end. It requests a DNS-validated ACM cert for --domain and terminates TLS at the ALB Ingress (no certbot, no self-signed cert), so --domain + --hosted-zone-id are required in both modes. The cert validates mid-deploy (aws acm wait certificate-validated), so expect a few minutes of waiting there.

Parameters

Parameter Required Default Description
--domain Yes — FQDN — ACM cert + Route 53 alias to the ALB Ingress
--hosted-zone-id Yes — Public Route 53 zone ID (ACM DNS validation + alias)
--ghcr-token Yes — GHCR token for pulling the service images
--deploy-mode No private public (internet-facing ALB) or private (internal)
--cluster-name No pipeline-builder EKS cluster name (set a second one to run multiple environments)
--no-email No — Skip SES (transactional email is provisioned by default)
--email-from No noreply@<domain> From address SES sends as
--email-from-name No pipeline-builder Display name on outbound email
--no-create-ses-identity No — Skip creating the SES identity (domain already verified in this account)
--alert-email No — Subscribe an address to the SES bounce/complaint SNS topic
--region No us-east-1 AWS region

Deployment mode (DEPLOY_MODE)

See Deployment modes for the public/private comparison. DEPLOY_MODE defaults to private; set it in the env before setup.sh or use --deploy-mode public. On EKS it controls only the ALB Ingress scheme — internal (private) vs internet-facing (public) — and the Route 53 record. For VPC-attached CodeBuild plugin pulls in private mode, supply PIPELINE_VPC_ID / SUBNET_IDS (the eksctl cluster VPC).

Phases

bin/setup.sh runs these in order — there are no per-component CloudFormation stacks (eksctl manages the cluster’s own stacks under the hood):

Phase Contents
1. Cluster eksctl creates the EKS Auto Mode cluster (cluster/cluster.yaml) + the aws-efs-csi-driver addon
2. EFS Encrypted EFS filesystem + security group (NFS from the nodes) + mount targets in the private subnets → the pb-efs (RWX) StorageClass
3. ACM DNS-validated ACM cert for --domain (publishes the validation record to Route 53, waits for ISSUED)
4. Secrets Namespace + the secret/ConfigMap set the manifests expect (JWT, DB creds, registry token keypair, app-env, DB init, observability configs) — same layout as the ec2 target
5. Pod Identity SES ses:SendEmail/ses:SendRawEmail association for the platform ServiceAccount (when email is enabled)
6. KEDA Installs the KEDA operator (the plugin ScaledObject autoscaler — Auto Mode doesn’t bundle it)
7. Workloads kubectl kustomize k8s | kubectl apply — all services: Nginx, Platform, Pipeline, Plugin, Quota, Billing, Message, Reporting, Compliance, Frontend, the in-cluster image-registry, observability (Prometheus, Loki, Alertmanager — surfaced via /dashboard/observability), admin tools (PgAdmin, Mongo Express), and the ALB Ingress
8. Route 53 A-alias --domain → the ALB the Ingress provisions

Post-Deploy

setup.sh applies the manifests, then the pods need a minute or two to pull, start, and pass readiness. The ALB target group reports unhealthy (503) until nginx and the platform are ready — expected; it self-heals.

# Watch the rollout reach a steady state
kubectl get pods -n pipeline-builder -w

# Wait on the gateway + core services
kubectl rollout status deploy/nginx deploy/platform deploy/pipeline deploy/plugin -n pipeline-builder

# Shell into a running pod (platform)
kubectl exec -it deploy/platform -n pipeline-builder -- /bin/sh

Plugin base images are seeded by init-platform.sh eks (the post-deploy step) — built by the in-cluster rootless buildkitd and pushed to the in-cluster registry; setup.sh itself doesn’t build them. By default infra provision runs that init for you over a kubectl port-forward; with the raw script, run ./deploy/bin/init-platform.sh eks once the registry is up (see Post-Deploy Steps). It works from anywhere with kubectl access — no VPC-attached host needed, even in private mode.

Storage Requirements

Persistent state lives on PVCs provisioned by the EBS/EFS CSI drivers — no EBS volumes to hand-manage. PostgreSQL, MongoDB, Redis, and the rest run as Kubernetes workloads (the postgres/mongo/redis images), not as RDS/DocumentDB/ElastiCache. Plugin images are built by an in-cluster rootless BuildKit sidecar and pushed to the in-cluster registry (there is no ECR dependency).

Resource Storage class Size Notes
PostgreSQL pb-ebs (RWO) 5-15 GB Pipelines, plugins, compliance, messages
MongoDB pb-ebs (RWO) 10-20 GB Quota + billing records
Prometheus / Alertmanager / PgAdmin pb-ebs (RWO) 1-10 GB each Metrics, alert state, admin UI
In-cluster registry none — RustFS — Stateless: images go to the registry bucket via the S3 storage driver
Loki none — RustFS — Chunks + index ship to the loki bucket
Redis pb-ebs (RWO) 1-5 GB Sentinel HA StatefulSet (3 Redis + 3 Sentinel) — queues + cache
Plugin builds / uploads pb-efs (RWX) per-pod BuildKit layer cache + upload staging (shared in-pod with the sidecar)

Recommendations:

Resource Setting
pb-ebs PVCs gp3, ReclaimPolicy: Retain — data survives a PVC/pod delete (clean up orphans manually)
pb-efs Elastic — grows automatically; no pre-provisioning
Registry growth Prune old plugin image tags from the in-cluster registry periodically

Monthly cost — from the AWS Pricing API (us-east-1, 2026-10-09), at the template’s own defaults.

The default InstanceType is m5.4xlarge, the TOP of the range this doc used to quote as “$140-265/mo” — so a deploy that changes nothing pays the high end, not the low one. Storage was omitted entirely: EbsVolumeSize=60 + DataVolumeSize=500 is 560 GiB of gp3, about $45/mo on its own.

Shape Compute 560 GiB gp3 Total 24/7
Default — m5.4xlarge ~$561 ~$45 ~$610-680/mo
t3.2xlarge ~$243 ~$45 ~$290-360/mo
t3.xlarge (LEAN, smaller volumes) ~$121 ~$20 ~$145-215/mo

Ranges include the ALB, EIP and data transfer, which vary with use. As with EKS, stopping the instance when idle beats every other saving — this target is a single instance, so stopping it costs only the EBS.

Monthly cost — MEASURED, not estimated.

Taken from AWS Cost Explorer on 2026-10-09 for the cluster as it actually runs (6 nodes: 4x c6a.xlarge + 2x m5a.xlarge, ~34 workloads). The figures that used to sit here were a bottom-up estimate totalling ~$150-400/mo, and they were low by roughly 2-3x — the node line in particular assumed far fewer nodes than Karpenter actually schedules for this workload.

Resource Cost
EC2 compute (Karpenter, on-demand) ~$560-640
EC2 - Other (EBS, NAT, data transfer) ~$190-240
EKS control plane + Auto Mode ~$73-80
ALB ~$10-30
VPC, WAF, Secrets Manager, CloudWatch ~$15-25
Total, running 24/7 ~$850-950/mo

Two things move this number far more than any tuning:

Node cost is the dominant, workload-dependent term: Karpenter scales nodes to fit scheduled pods, and the NodePool ceiling in cluster/nodepool.yaml (48 vCPU / 96Gi) is what bounds the worst case rather than the typical one.

Expanding EKS Storage

pb-ebs PVCs (postgres / mongodb / prometheus / …): the gp3 StorageClass allows volume expansion, so grow a volume by raising the PVC request and letting the EBS CSI driver expand it online:

kubectl patch pvc postgres-data -n pipeline-builder \
  -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
# the EBS CSI driver expands the volume + filesystem online (gp3) — no pod restart needed

pb-efs (plugin build uploads) — no expansion needed: EFS is elastic and grows automatically. (The registry, Loki, and message attachments now live in RustFS (this target’s S3-compatible object store), not EFS — the registry is stateless S3, Loki ships chunks/index to S3; only plugin build uploads still use pb-efs.) Cap object-store growth via bucket lifecycle rules + Loki retention_period; check EFS usage via aws efs describe-file-systems.

Cluster capacity: node capacity is managed by Karpenter (Auto Mode) — it provisions and removes EC2 nodes to fit scheduled pods, so there is no instance to resize.

EKS vs the other k8s targets

EKS reuses the same Kubernetes manifests as minikube/ec2, with these AWS-managed substitutions:

minikube / ec2 EKS
hostPath volumes EBS (RWO) + EFS (RWX) PVCs via CSI
NodePort + iptables bridge ALB Ingress (target-type: ip → nginx:8080)
Single node Karpenter-scaled EC2 nodes (Auto Mode)
EC2 instance role (SES) EKS Pod Identity association
Self-managed addons Auto Mode: AWS LB Controller, EBS CSI, CoreDNS built in (EFS CSI added)

Scripts

All in deploy/aws/eks/bin/. Run from your local machine (or via infra provision).

Script Purpose
setup.sh Full deploy: cluster → EFS → ACM → secrets → KEDA → manifests → Route 53
shutdown.sh Teardown: Ingress/ALB → Route 53 → EFS → cluster → ACM cert

Monitoring

# Pod / service status
kubectl get pods,svc -n pipeline-builder

# Tail logs for a service
kubectl logs -f deploy/nginx -n pipeline-builder

TLS Renewal

The ACM cert is DNS-validated and auto-renews — nothing to do (ACM rotates it as long as the validation CNAME stays in the hosted zone). The ALB Ingress picks up the renewed cert automatically.

Teardown

cd deploy/aws/eks
bash bin/shutdown.sh --cluster-name pipeline-builder --region us-east-1 \
  --domain pipeline.example.com --hosted-zone-id Z123 --yes

Deletes the Ingress/ALB, the Route 53 alias, the EFS filesystem, the cluster (eksctl delete cluster), and the ACM cert — in dependency order.

EBS volumes on the pb-ebs (Retain) StorageClass are not auto-deleted (they’re reported at the end). Remove leftovers manually if you don’t need the data.

Email (SES)

The platform sends transactional email (invitations, email verification, password resets) via Amazon SES. It’s enabled by default — every AWS deploy provisions it; pass --no-email to skip it:

# EC2 — SES is provisioned by default
bash bin/setup.sh --key-pair my-keypair --domain pipeline.example.com \
  --hosted-zone-id Z123 --ghcr-token ghp_xxx

# EKS — pass --no-email to opt out
bash bin/setup.sh --domain pipeline.example.com \
  --hosted-zone-id Z123 --ghcr-token ghp_xxx --no-email

By default the deploy wires up everything in one shot:

Flag Default Purpose
--no-email — Skip SES (it is provisioned by default: identity + DKIM + role grant + app env)
--email-from noreply@<domain> From address SES sends as
--email-from-name pipeline-builder Display name on outbound email
--no-create-ses-identity — Skip identity creation when --domain is already a verified SES identity in this account/region (avoids a “already exists” rollback); IAM + env are still wired
--alert-email — Subscribe this address to the bounce/complaint SNS topic (you must confirm the email AWS sends)

Region matters: the SES identity is regional and must match the deploy region. The deploy pins SES_REGION to it automatically (EC2 derives it from the stack region in bootstrap.sh, not the static .env default).

Verification & the SES sandbox

Two things happen after the stack completes, and both need your attention:

  1. DKIM verification is asynchronous — Route 53 → SES propagation takes minutes to hours. Sends before the domain verifies fail gracefully (the platform logs it and continues). Check status at SES console → Verified identities.
  2. New SES accounts are sandboxed — you can only send to verified recipients, max 200/day. To send to arbitrary users, request production access (SES console → Account dashboard). CloudFormation can’t do this for you. To smoke-test while sandboxed, verify a real recipient address — never admin@internal (it bounces, and sandbox bounces hurt the reputation AWS reviews for production approval).

Bounce & complaint tracking

SES enforces sender reputation at the account level — above ~5% bounce or ~0.1% complaint it puts the account under review and can pause all sending (including password resets). To make that visible instead of a silent outage, the deploy provisions a configuration set that every send routes through (SES_CONFIGURATION_SET on the platform), with an SNS topic receiving every bounce, complaint, and reject (pipeline-builder-email-events on EC2; <cluster-name>-email-events on EKS).

Pass --alert-email you@example.com to subscribe an address at deploy time (confirm the subscription email AWS sends), or subscribe the topic later from the console. Without a subscription the topic still collects events — you just won’t be alerted. Reputation rates are also on the SES console Account dashboard.

Going to production

The deploy leaves SES verified but sandboxed. These steps finish the job; do 1–6 before requesting production access, because AWS weighs bounce handling and domain authentication when it reviews the request. REGION, TOPIC_ARN and the addresses are yours to substitute.

  1. A mailbox on your domain. SES sends; it is not a mailbox. Verification links, SNS confirmations and DMARC reports all need an address that receives mail, so point the domain’s MX at a mail host (Zoho Mail, Google Workspace, Microsoft 365, …) and publish its SPF and DKIM records. Amazon WorkMail is not an option for new setups: it took no new customers after April 30, 2026 and reaches end of support on March 31, 2027. The mail host’s SPF lives on the root domain; SES’s lives on its own MAIL FROM subdomain (step 2), so the two never conflict.
  2. Custom MAIL FROM — aligns SPF with your domain: 
    aws sesv2 put-email-identity-mail-from-attributes --email-identity <domain> \
  --mail-from-domain mail.<domain> --behavior-on-mx-failure USE_DEFAULT_VALUE --region REGION

    then publish mail.<domain> MX 10 feedback-smtp.REGION.amazonses.com and mail.<domain> TXT "v=spf1 include:amazonses.com ~all".

  3. DMARC — _dmarc.<domain> TXT "v=DMARC1; p=none; rua=mailto:<you>@<domain>". Easy DKIM already aligns, so this passes from day one; tighten to p=quarantine after a few weeks of clean reports.
  4. Verify a real recipient for sandbox testing: aws sesv2 create-email-identity --email-identity <you>@<domain> --region REGION. To resend the link for a pending address, use the v1 API: aws ses verify-email-identity --email-address <you>@<domain>.
  5. Subscribe and confirm bounce alerts. 
    aws sns subscribe --topic-arn TOPIC_ARN --protocol email \
  --notification-endpoint <you>@<domain> --region REGION

    Click the confirmation link. Re-running subscribe for an address that is still pending does not reliably resend it, and a pending subscription cannot be deleted (it expires after 3 days). Use Request confirmation on the subscription in the SNS console, or subscribe a plus-address (<you>+alerts@<domain>) if your mail host supports sub-addressing.

  6. A public privacy policy on your site. Reviewers look for one; link it in the request.
  7. Request production access (SES console → Account dashboard, or aws sesv2 put-account-details --production-access-enabled --mail-type TRANSACTIONAL …). The first answer is often an automated “needs more information” asking for: website URL, email type, expected volume, how recipients opt in, bounce and complaint handling, and a sample email. Reply on the case in Support Center — replying to the notification email bounces, because it comes from a no-reply address — and do not submit a second request. AWS uses the volume you give to set your initial sending quota, so give a realistic transactional estimate; you can request an increase later.

Superadmin email. If you replace admin@internal in BOOTSTRAP_SUPERADMIN_EMAILS with a real address so notifications reach a person, add it rather than replacing (fresh installs need admin@internal), and register that account yourself first: any account that registers with a listed email is promoted to superadmin on registration.

Post-Deploy Steps

After deploying (EC2 or EKS), complete these steps to initialize the platform and enable reporting.

1. Initialize the Platform

Register the admin user and load pre-built plugins and sample pipeline templates:

cd deploy

# Prompts for build strategy + categories. Admin creds come from PLATFORM_IDENTIFIER /
# PLATFORM_PASSWORD (defaulting if unset — export real values on ec2/eks)
bash bin/init-platform.sh ec2         # EC2 (resolves URL from the pipeline-builder stack)
bash bin/init-platform.sh eks         # EKS (port-forwards svc/nginx via kubectl)
bash bin/init-platform.sh docker       # Docker Compose
bash bin/init-platform.sh minikube    # Minikube

# Non-interactive
export PLATFORM_BASE_URL=https://pipeline.example.com
export PLATFORM_IDENTIFIER=admin@internal
export PLATFORM_PASSWORD='<a strong secret you generate>'
bash bin/init-platform.sh ec2

# Non-interactive with prebuilt images
PLUGIN_BUILD_STRATEGY=prebuilt bash bin/init-platform.sh ec2

# Non-interactive with prebuilt + specific categories
PLUGIN_BUILD_STRATEGY=prebuilt PLUGIN_CATEGORY=infrastructure,language bash bin/init-platform.sh ec2

# Control upload parallelism (default: 4, auto-lowered to 1 for prebuilt)
PARALLEL_JOBS=2 bash bin/init-platform.sh docker

# Force rebuild all prebuilt images even if image.tar exists
PLUGIN_BUILD_STRATEGY=prebuilt FORCE_REBUILD=true bash bin/init-platform.sh ec2

# Force a from-scratch rebuild of the base images AND the CodeBuild bootstrap
# image, bypassing the docker-cache / registry-tag skips (equivalent to
# FORCE_REBUILD=true plus --force/FORCE_PUSH on the bootstrap build)
bash bin/init-platform.sh --force ec2

# Clean up plugin.zip and image.tar after upload (reclaim disk space)
./deploy/bin/init-platform.sh --cleanup docker
./deploy/bin/load-plugins.sh --rebuild --cleanup

# EC2 with sudo (required for minikube user context)
sudo -u minikube PLATFORM_BASE_URL=https://your-ip bash /opt/pipeline/pipeline-builder/deploy/bin/init-platform.sh ec2
sudo -u minikube PLATFORM_BASE_URL=https://your-ip bash /opt/pipeline/pipeline-builder/deploy/bin/init-platform.sh --cleanup ec2

init-platform.sh does: health check → register admin → login → select build strategy → load plugins → load pipelines.

Environment Variables

Variable Default Description
PLATFORM_BASE_URL auto-detected Platform API URL (skips CloudFormation lookup when set)
PLATFORM_IDENTIFIER admin@internal Admin email
PLATFORM_PASSWORD Pipeline-Builder-Dev-2026! (local targets only) Admin password. The dev default is REFUSED on ec2/eks — set a strong secret there.
PLUGIN_BUILD_STRATEGY build_image build_image or prebuilt
PLUGIN_CATEGORY all Comma-separated categories (e.g., language,security)
PARALLEL_JOBS 4 (1 for prebuilt) Upload concurrency. Passed through to load-plugins.sh. Override with --parallel N on CLI.
FORCE_REBUILD false Force rebuild all prebuilt image.tar files
PLUGIN_S3_CLEAR false Clear S3 bucket before upload (S3 strategy only)

Use --cleanup flag on init-platform.sh or load-plugins.sh to remove plugin.zip and image.tar files after upload. Useful on EC2 where prebuilt images can consume 25-75GB of disk.

Use --force on init-platform.sh to rebuild the base images and the CodeBuild bootstrap image from scratch, ignoring the docker-cache / registry-tag idempotency skips (drives FORCE_REBUILD for the base images and --force + FORCE_PUSH for the bootstrap image). Use it after changing a base Dockerfile or the bootstrap image contents, when the cached tag would otherwise be reused.

Script Purpose
init-platform.sh Register admin + select build strategy + load plugins + pipelines (interactive)
build-plugin-images.sh Pre-build Docker images for plugins (prebuilt strategy)
load-plugins.sh Upload plugins from deploy/plugins/
load-templates.sh Upload pipeline templates from deploy/samples/templates/
test-plugins.sh Validate plugin specs and Dockerfiles
build-codebuild-bootstrap.sh Build + publish the CodeBuild bootstrap image (CODEBUILD_DEFAULT_IMAGE fallback runtime)

Build the CodeBuild bootstrap image (build-codebuild-bootstrap.sh)

The bootstrap image backs CODEBUILD_DEFAULT_IMAGE — the fallback runtime CodeBuild uses for the synth step and for any plugin step whose own image isn’t resolved. build-codebuild-bootstrap.sh builds it with Docker and publishes it to the platform’s registry. provision --build-bootstrap runs this for you; run it directly to refresh the image after a CLI/Dockerfile change.

The bootstrap image bakes in aws-cdk + esbuild + pnpm so synth bundles the PluginLookup Lambda locally (no Docker-in-Docker). If you instead run cdk synth / pipeline deploy on your own machine, you need those same tools on PATH — otherwise bundling fails with Could not resolve "axios". See Pipeline Manager → local deploy prerequisites.

cd deploy/bin

# Build (if missing) + publish to the EC2 platform registry
DEPLOY_TARGET=ec2 ./build-codebuild-bootstrap.sh

# Force a rebuild even if the image is cached, and republish over an existing tag
DEPLOY_TARGET=ec2 FORCE_PUSH=true ./build-codebuild-bootstrap.sh --force

# Pin the pipeline-manager version baked into the image (avoids the @latest
# layer-cache footgun that can ship stale CLI code)
DEPLOY_TARGET=ec2 PIPELINE_MANAGER_VERSION=3.4.131 ./build-codebuild-bootstrap.sh --force

# Cut a NEW image version instead of refreshing 1.0 in place
DEPLOY_TARGET=ec2 BOOTSTRAP_IMAGE_TAG=pipeline-bootstrap:1.1 ./build-codebuild-bootstrap.sh

Environment variables

Variable Default Description
DEPLOY_TARGET docker Push transport: docker | minikube | ec2 | eks. ec2/minikube push via a kubectl run crane pod.
FORCE_PUSH false Republish even when the remote tag already exists.
BOOTSTRAP_IMAGE_TAG pipeline-bootstrap:1.0 Image tag to build/push. Must match the deployment’s CODEBUILD_DEFAULT_IMAGE — a mismatch fails CodeBuild with BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE.
PIPELINE_MANAGER_VERSION latest npm dist-tag/version baked in; auto-resolved to a concrete version so the Docker layer cache is correct.
PUBLISH_PLATFORM linux/amd64 Build/push platform; set linux/arm64 for an all-Graviton stack.

--force rebuilds the local image; without it an already-cached image skips straight to publish.

Run it on the EC2 instance — the ec2 transport runs a kubectl run crane pod, so it needs both Docker and a kubeconfig that reaches the cluster. Two gotchas:

2. Store Service Credentials

The Lambdas and CodeBuild read service-account keys from Secrets Manager — machine identities owned by the org, not a person’s token. infra store-token provisions the account and issues the key:

# First sign in (browser device flow — prints a code, no password on the CLI).
# The session is stored per-platform, so store-token picks it up automatically.
pipeline-manager auth login --no-verify-ssl

# Issuing a service-account key is step-up gated, so the command needs your
# password as well as your session. Keep it out of shell history.
export PLATFORM_PASSWORD='…'

# 1. The org's full-privilege automation credential (synth/deploy, plugin lookup)
pipeline-manager infra store-token --days 30 --schedule --region us-east-1

# 2. The CI registry credential CodeBuild presents as Basic auth (REQUIRED:
#    synth wires this secret into every build image's pull credentials)
pipeline-manager infra store-token --scope registry:push --schedule --region us-east-1

# 3. The event-ingestion credential
pipeline-manager infra store-token --scope reporting:ingest --schedule --region us-east-1

Each run writes its own secret and its own rotation stack:

Secret Service account Key scope
pipeline-builder/{orgId}/platform platform-automation none (org admin Roles)
pipeline-builder/{orgId}/registry-push registry-push registry:push
pipeline-builder/{orgId}/reporting-ingest reporting-ingest reporting:ingest

On a headless host with no browser, export PLATFORM_IDENTIFIER / PLATFORM_PASSWORD instead: store-token still has a non-interactive password login of its own for exactly that case (it is provisioning a machine credential, with nobody present to approve anything). PLATFORM_PASSWORD is required either way — it is the step-up factor for both writes.

By default infra store-token only writes the secret — you must re-run it before the key expires (audit tokens warns you in advance). To avoid that, add --schedule to also deploy a small daily key-rotation stack (pipeline-builder-token-renew):

# Custom rotation time (5-field cron; minimum every 15 minutes):
pipeline-manager infra store-token --schedule --cron '0 3 * * *' --region us-east-1

The rotation stack is a scheduled Lambda that mints a sibling key on the same account, writes it to the secret, and only then retires its predecessor — in that order, so a failure at any step leaves a working credential behind. It installs nothing at runtime. See Authentication → Self-rotation. (The --with-events provision bundle opts into --schedule automatically, since the event-ingestion Lambda depends on its key.)

3. Deploy EventBridge Reporting Infrastructure

Set up pipeline execution reporting to track success rates, stage performance, and build analytics:

export PLATFORM_BASE_URL=https://pipeline.example.com

pipeline-manager infra setup-events --region us-east-1

This creates a CloudFormation stack (pipeline-builder-events) containing:

What is (and isn’t) forwarded to the platform

The Lambda runs inside your AWS account and forwards only pipeline-execution telemetry — enough to compute success rates, stage/action timing, and DORA metrics.

Not forwarded — stays in your AWS account:

  • Your AWS account number — explicitly stripped from every event (delete detail.account).
  • The pipeline ARN (arn:aws:codepipeline:<region>:<account>:<name>) — built only as a transient handle to resolve the pipeline’s pb.pipeline-id tag via codepipeline:ListTagsForResource, then discarded; never stored or sent.
  • AWS credentials / IAM and any account-identifying details.

The platform stores no AWS account id anywhere (schemas, JWTs, and APIs are account-id-free by design), so there is nothing to mask.

Forwarded payload (POST /api/reports/events, batched):

Field Notes
pipelineId The platform pipeline id (from the pb.pipeline-id tag) — not the ARN
eventSource / eventType codepipeline · PIPELINE/STAGE/ACTION
status CodePipeline state (SUCCEEDED/FAILED/…)
executionId · stageName · actionName Execution GUID + stage/action names
errorMessage Human-readable failure summary (capped), on failures
startedAt · completedAt · durationMs Timing
commitSha · commitRef · commitTimestamp · commitCount Source revision + commit time/count for measured lead time (DORA), when --with-dora is enabled
environment The deploy stage’s env, from the pipeline’s pb.deploys tag
detail Raw CodePipeline event detail with account removed (log URL / error code for drill-down)

Reference: @pipeline-builder/pipeline-events.

4. Verify Reporting

# Check the EventBridge stack
aws cloudformation describe-stacks --stack-name pipeline-builder-events \
  --query 'Stacks[0].StackStatus' --output text

# Check the EventBridge rule
aws events describe-rule --name pipeline-builder-codepipeline-events

# Check the Lambda
aws lambda get-function --function-name pipeline-builder-event-ingestion \
  --query 'Configuration.LastModified'

How Reporting Works

Synth  → CDK tags the CodePipeline `pb.pipeline-id=<pipelineId>` (+ `pb.deploys`) (stable, set at creation)
Deploy → pipeline-manager registers the pipeline (by pipelineId) in pipeline_registry
Execute → CodePipeline runs → EventBridge captures state changes
Ingest  → SQS → Lambda resolves the pb.pipeline-id tag → POST /api/reports/events (keyed by pipelineId)
Store   → Reporting API matches the registry by pipelineId → inserts into pipeline_events
View    → Dashboard Reports page or GET /api/reports/...

The pipeline ARN and AWS account number never leave AWS — the Lambda attributes events via the pipeline’s pb.pipeline-id tag (= the opaque pipelineId), so nothing sensitive is stored and there is no masking key to manage. The Lambda’s execution role needs codepipeline:ListTagsForResource.

Plugin Docker builds are captured automatically by the plugin service (no EventBridge needed).

Drift Detection (audit stacks)

The pipeline_registry table is written only when pipeline-manager pipeline deploy succeeds. CloudFormation stacks can be created or destroyed outside of that path — manual aws cloudformation delete-stack, console operations, side-channel deploys — and over time the registry can drift from reality.

The audit stacks command joins the registry against live CloudFormation stacks tagged pipeline-builder and surfaces two categories of drift:

Finding Meaning Typical cause
Orphaned stack Tagged stack exists in CloudFormation, but no matching row in pipeline_registry Pipeline was deleted from the dashboard but the CDK stack stayed in AWS
Missing stack Registry row exists, but no matching CloudFormation stack Stack was deleted manually (e.g. aws cloudformation delete-stack) without going through the platform

Usage

# Scan all orgs in the default region
pipeline-manager audit stacks --region us-east-1

# Scan one org, JSON output (suitable for piping into jq / cron alerting)
pipeline-manager audit stacks --org acme --region us-east-1 --json

# With a specific AWS profile
pipeline-manager audit stacks --profile production --region us-east-1

Flags:

Flag Purpose
--region <region> AWS region to scan. Defaults to AWS_REGION env, then CDK_DEFAULT_REGION, then us-east-1.
--org <orgId> Restrict both the registry fetch and the stack scan to a single org.
--profile <profile> AWS CLI profile (default: default).
--json Emit a single JSON document instead of human output.

Exit codes

The command is designed to be cron-friendly:

Exit code Meaning
0 No drift
1 One or more findings (orphaned and/or missing stacks)
2 AWS error or scan failure

A typical alerting setup runs the audit nightly and pages on non-zero exit:

# /etc/cron.d/pipeline-builder-audit
0 6 * * * deploy-bot pipeline-manager audit stacks --region us-east-1 --json > /var/log/pb-audit.json || alert-on-call "pipeline-builder drift detected"

Remediation

Drift is not auto-fixed — the command only reports. Reconciliation is manual and depends on the cause:

What it doesn’t catch

Report API Endpoints

All endpoints require authentication and org context. Time range defaults to last 30 days.

Pipeline Execution Reports

Endpoint Description Query Params
GET /api/reports/execution/count Execution count per pipeline with status breakdown —
GET /api/reports/execution/success-rate Pass/fail rate over time interval, from, to
GET /api/reports/execution/timeline Execution timeline (alias for success-rate) interval, from, to
GET /api/reports/execution/duration Average/min/max/p95 execution duration from, to
GET /api/reports/execution/stage-failures Stage failure heatmap from, to
GET /api/reports/execution/stage-bottlenecks Slowest stages per pipeline from, to
GET /api/reports/execution/action-failures Action/step failure rate from, to
GET /api/reports/execution/errors Error categorization (top N) from, to, limit

Plugin Reports

Endpoint Description Query Params
GET /api/reports/plugins/summary Plugin inventory (total/active/public/private) —
GET /api/reports/plugins/distribution Type and compute distribution —
GET /api/reports/plugins/versions Version counts per plugin name —
GET /api/reports/plugins/build-success-rate Docker build success rate over time interval, from, to
GET /api/reports/plugins/build-duration Build time per plugin from, to
GET /api/reports/plugins/build-failures Build failure reasons (top N) from, to, limit
GET /api/reports/plugins/runtime-success-rate Runtime success rate per plugin version (pipeline runs) from, to, name, publisher, version
GET /api/reports/plugins/runtime-duration Runtime p50/p95 duration per plugin version from, to, name, publisher, version

Plugin runtime telemetry. The runtime reports describe how plugins behave when your pipelines run them. The build reports cover building the plugin images. In the dashboard, Reports → Plugins → Runs shows both runtime reports as one table per plugin version (runs, success rate, p50/p95 duration, last run), with CSV export. Here is how the data gets there:

  1. When pipeline-manager pipeline deploy synthesizes a pipeline, it records a step manifest: which plugin (publisher, name, version, image digest) each CodePipeline stage and action runs. It writes it as pb-step-manifest.json in the cloud assembly.
  2. After the deploy, the manifest is sent with the POST /api/pipelines/registry registration. The platform re-reads name, version and digest from the plugin row (the CLI’s claim isn’t trusted), then replaces the pipeline’s rows in pipeline_step_manifests.
  3. Event ingest joins each ACTION/BUILD event on (pipeline, stage, action). It stamps plugin_publisher, plugin_name and plugin_version onto pipeline_events.

Common query parameters:

Param Values Default
interval day, week, month week
from ISO 8601 timestamp 30 days ago
to ISO 8601 timestamp now
limit integer 20

Access Points

After deployment, access services at:

Service Path
Application /
Reports Dashboard /dashboard/reports
Observability (native) /dashboard/observability
Registry UI /dashboard/registry (system-admin only)
PgAdmin /pgadmin/ — only with ADMIN_UIS_ENABLED=true
Mongo Express /mongo-express/ — same
Grafana /grafana/ — same (then its own login, GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD)
Kiali (mesh graph) /kiali/ — same (read-only)
RustFS console (object store) :9001, not a path — https://<domain>:9001/rustfs/console/ on ec2; on eks kubectl port-forward -n pipeline-builder svc/nginx 9001:9001 (same ADMIN_UIS_ENABLED gate)

The admin consoles are off by default on AWS: their routes 404. With ADMIN_UIS_ENABLED=true in .env (re-run setup), every request to them first passes an nginx auth_request to platform GET /admin/console-check — a live session of a platform administrator at AAL2 — with the token taken from the pb_admin_console cookie and stripped before the console sees the request. For occasional use prefer kubectl -n pipeline-builder port-forward svc/grafana 3000.

The RustFS object-store console rides the same flag and the same auth_request, but on its own port rather than a path, because its UI resolves its API endpoints from window.location, and its S3 client uses the bare origin — ListBuckets is GET / and objects are /<bucket>/..., with the admin API at /rustfs/admin/v3/*. Mounted under /rustfs/console/ on the gateway its assets would load and every one of those calls would hit the platform instead. On ec2 the ALB gets a second HTTPS listener on 9001 (same ACM cert) bridged to the nginx NodePort 30901, so it is https://<domain>:9001/rustfs/console/. On eks there is deliberately no such listener: an ALB rule cannot be scoped to a listener, and the alternatives (group.name IngressGroup, or a second LoadBalancer Service) are unverified on EKS Auto Mode’s own ALB controller — a silently ignored group.name would provision a second ALB on a DNS name the cert does not cover. nginx still serves the port, so reach it with kubectl port-forward -n pipeline-builder svc/nginx 9001:9001. Either way it can LIST and DELETE objects in the registry, Loki, Thanos, plugin-build and audit-chain buckets, so treat it as at least as powerful as /pgadmin/.

After every provision the setup scripts run deploy/bin/post-provision-smoke.sh: a test alert through Alertmanager to Slack, a test email through platform, a CodePipeline credential dry-run from the pipeline pod, and a probe that a connection the NetworkPolicies deny is actually denied. Its warnings are non-fatal — read the summary line.

File Structure

EC2 deployment files ``` deploy/aws/ec2/ ├── template.yaml # CloudFormation stack ├── .env.example # Reference config ├── postgres-init.sql # Schema + RLS (identical in every target) ├── mongodb-init.js # Mongo indexes (identical in every target) ├── services.txt # Generated by projen: ├── bin/ │ ├── setup.sh # Deploy the stack (from your machine) │ ├── bootstrap.sh # EC2 setup + hardening │ ├── startup.sh # Minikube + K8s deploy + ALB-target iptables bridge │ └── shutdown.sh # Teardown ├── k8s/ # Kubernetes manifests │ └── kustomization.yaml # Kustomize entry point ├── nginx/ │ ├── nginx.conf # Nginx config (routes; drift-checked against the other targets) │ ├── jwt.js # NJS access-log claim decode (identical in every target) │ ├── metrics.js # NJS gateway request counters (identical in every target) │ └── registry-auth.js # NJS registry token-realm rewrite └── config/ # Prometheus, Promtail, Grafana, Loki, Alertmanager, Thanos ``` This target's own copies, read straight out of this tree: `postgres-init.sql`, `services.txt`, `mongodb-init.js`, `nginx/jwt.js`, `nginx/metrics.js` and `config/{loki,alertmanager,thanos}`. They are identical in every target, and `test/deploy-contracts` fails the build if a copy drifts. </details>
EKS deployment files ``` deploy/aws/eks/ ├── bin/ │ ├── setup.sh # Full deploy orchestrator (cluster → … → Route 53) │ └── shutdown.sh # Teardown (Ingress/ALB → Route 53 → EFS → cluster → ACM) ├── cluster/ │ └── cluster.yaml # eksctl ClusterConfig (Auto Mode + aws-efs-csi-driver) ├── k8s/ │ ├── kustomization.yaml # Standalone manifests (not shared with ec2/minikube) │ ├── storageclasses.yaml# pb-ebs (RWO) + pb-efs (RWX) │ ├── ingress.yaml # ALB Ingress → nginx:8080 (ACM TLS at the ALB) │ └── *.yaml # Full workload set, PVC-tuned for multi-node ├── config/ # Prometheus, Promtail, Grafana, Loki, Alertmanager, Thanos ├── nginx/ # nginx.conf, admin-uis*.conf, registry-auth.js, jwt.js, metrics.js ├── .env.example ├── postgres-init.sql # Schema + RLS (identical in every target) ├── mongodb-init.js # Mongo indexes (identical in every target) ├── services.txt # Generated by projen: └── mongodb-keyfile # generated per deploy, gitignored ``` postgres-init.sql, services.txt, mongodb-init.js, jwt.js/metrics.js and the Loki/Alertmanager/Thanos configs are this target's own copies, held identical across targets by `test/deploy-contracts`. </details> --- ## Troubleshooting **Pods stuck Pending (EC2):** Check CPU requests vs instance capacity. `kubectl describe pod ` shows scheduling failures. **ImagePullBackOff (EC2):** Verify GHCR credentials and that iptables rules aren't intercepting minikube's outbound traffic. Authenticate with the GitHub Container Registry first if you haven't already: ```bash echo $YOUR_PAT | docker login ghcr.io -u USERNAME --password-stdin ``` `YOUR_PAT` is a GitHub Personal Access Token with the `read:packages` scope. The `bootstrap.sh` and `startup.sh` scripts pick up `GHCR_TOKEN` and `GHCR_USER` env vars to create the in-cluster `ghcr-secret` automatically. **`GhcrToken` rejected with `unauthorized` or `denied`:** The pre-built images at `ghcr.io/mwashburn160/*` are **public** — anonymous pulls succeed — but `GhcrToken` is still requested by the CFN templates because anonymous GHCR pulls are subject to a low per-IP rate limit (60 req/hr) that will trip mid-deploy when EC2 pulls all 10 service images concurrently. Authenticated pulls raise the limit to 5,000 req/hr. **Use your own GitHub Personal Access Token — do not copy a value from documentation, an example command, or another user's deployment.** Tokens that aren't yours will fail (or worse, succeed temporarily and break later when the original owner rotates them). To create your own: - **Classic PAT** (simplest): https://github.com/settings/tokens → "Generate new token (classic)" → check only the `read:packages` scope. - **Fine-grained PAT** (recommended): https://github.com/settings/personal-access-tokens → "Generate new token" → resource owner = your account → permissions: `Packages: Read` (account permissions, not repo). Pass it as the `GhcrToken` CFN parameter or export it as `GHCR_TOKEN` for `bootstrap.sh`/`startup.sh`. There is no username to set — ghcr.io validates only the token for PAT auth, so the deploy uses a fixed internal value. If you intentionally want to skip auth for a small test deploy, leave `GhcrToken` empty and the bootstrap scripts will fall back to anonymous pulls — expect occasional 429s on retry-storms across all 10 services. **CrashLoopBackOff on observability pods (EC2):** Usually hostPath permission issues. Check pod logs. Init containers handle `chown` for loki (10001) and prometheus (65534). **Pods stuck Pending / no nodes (EKS):** Karpenter provisions nodes on demand — a brief Pending is normal at cold start. If it persists, check `kubectl describe pod ` for scheduling reasons and `kubectl get events -n pipeline-builder`. A pb-ebs (RWO) PVC is AZ-pinned, so its pod must schedule in the volume's AZ. **ALB Ingress has no address (EKS):** The AWS Load Balancer Controller provisions the ALB from `ingress.yaml`. Check `kubectl describe ingress pb-ingress -n pipeline-builder` for controller events, and that the ACM cert reached `ISSUED`. The Route 53 alias is only written once the Ingress reports a hostname. **Certificate / stack hangs in CREATE_IN_PROGRESS:** The ACM cert DNS-validates during stack creation (a few minutes). If it never issues, the `--hosted-zone-id` is wrong or not authoritative for `--domain`. Check ACM status: `aws acm describe-certificate --certificate-arn ` (look for `DomainValidationOptions[].ValidationStatus`). **No reporting data after deploy:** 1. Verify `pipeline-manager infra store-token` was run 2. Check Lambda logs: `aws logs tail /aws/lambda/pipeline-builder-event-ingestion --follow` 3. Check SQS DLQ for failed events 4. Verify pipeline was deployed after `infra setup-events` (it must have a `pipeline_registry` row)