# Deploying on AWS (/docs/operations/deploy-aws) > The reference deployment of the control plane on AWS, as three CloudFormation stacks rendered by aws deploy-templates, from the VPC to the ECS Fargate services. `@better-compute/aws` renders the reference deployment of the control plane on AWS as three CloudFormation JSON templates. They lay out the [split-roles topology](/docs/operations/deployment#split-roles) on AWS services: the api behind an L7 load balancer, the gateway behind an L4 load balancer with TLS passthrough, workers with no listener, and Aurora PostgreSQL with IAM authentication. Nothing here is required: the templates are a starting point you can read, diff and change, not an installer. ```bash better-compute aws deploy-templates --out cfn # network.json, data.json, control-plane.json better-compute aws deploy-templates --out cfn --zones 2 --nat single --readers 0 --name bc-stage ``` The command makes no AWS call. It writes only new files, and refuses before writing anything if one exists. ## The three stacks [#the-three-stacks] | Stack | What it creates | Imports | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `network` | A VPC across 2 or 3 zones with a public and a private /20 per zone, NAT per zone (or one), the S3 gateway endpoint, interface endpoints for Secrets Manager, KMS, ECR, CloudWatch Logs and STS, VPC flow logs | none | | `data` | Aurora PostgreSQL 16 or later on Serverless v2: IAM database authentication, TLS forced, `btree_gist` allowed, storage under its own KMS key, the master password managed by Secrets Manager, a writer and 0 to 2 readers | `network` | | `control-plane` | ECS Fargate services `api`, `worker` and `gateway` plus a `migrate` task; an ALB with an ACM certificate and WAF; an NLB in TCP passthrough; task roles; the deployment secret and the sealed PKI bundle; one KMS key; S3 buckets; an EventBridge bus; log groups and alarms | `network`, `data` | Each stack exports what the next one needs as `-` (for example `bc-prod-network-PrivateSubnetIds`), and the later stacks take the earlier stack names as the `NetworkStack` and `DataStack` parameters. Structural choices (zones, NAT mode, readers, the deployment name) are command flags; everything that differs between environments (the image, the domains, the certificate, task counts, sizes) is a stack parameter. ### Processes [#processes] | Service | Command | In front | Security group admits | | ---------------- | ------------------------------------------------------------ | -------------------------------------- | ------------------------------------------------------------------ | | `api` | `serve --roles api --port 3000 --no-migrate` | ALB, HTTPS 443, health check `/readyz` | 3000 from the ALB only | | `worker` | `serve --roles controllers,scheduler,sinks,iam --no-migrate` | none | nothing | | `gateway` | `serve --roles gateway --no-migrate` | NLB, TCP 443 to 8443 | 8443 from the NLB; the internal API port from api and worker tasks | | `migrate` (task) | `migrate` | none | nothing | Every task runs in the private subnets without a public address, with a read-only root file system, all Linux capabilities dropped, `awslogs` in non-blocking mode, and the deployment circuit breaker with rollback. All of them also wear the data stack's client security group, the only group the database admits on 5432. The ALB uses the `ELBSecurityPolicy-TLS13-1-2-2021-06` policy, redirects HTTP to HTTPS, drops invalid headers, and has a WAF web ACL in front: a per-IP rate limit (`WafRateLimit` requests per five minutes), the AWS IP reputation list, the common rule set, and known bad inputs. The common rule set's 8 KiB body rule only counts, since API bodies can be larger. The idle timeout is an hour, for interactive tunnels. The NLB never terminates TLS: the gateway authenticates each daemon by its client certificate, which a terminating balancer would strip. Its target group turns client IP preservation off, so tasks only need to admit the NLB's own security group. ### Secrets and keys [#secrets-and-keys] The templates take no secret as a parameter (`lintTemplate` refuses `NoEcho` parameters): * `BETTER_COMPUTE_SECRET` is generated by Secrets Manager (64 characters) and reaches the tasks as an ECS secret. * The **sealed PKI bundle** (the JSON `better-compute pki init` writes) goes into the `PkiBundleSecret` secret, which the tasks get as `BC_PKI_BUNDLE`. The stack creates it with a random placeholder, and you put the real bundle into it. * The database master password is generated and kept by RDS in Secrets Manager. The processes never use it: they connect as an IAM-authenticated user with `rdsAuthTokenProvider`. One rotating KMS key encrypts the two secrets, the blobs, snapshots and audit buckets, every log group and the alarm topic. The database has a key of its own. Both keys are kept when a stack is deleted. ### Restoring under a new stack [#restoring-under-a-new-stack] The secrets, the platform key, the log groups and the blobs, snapshots and audit buckets outlive a deleted stack, and a restored database depends on all of them: data sealed with the old `BETTER_COMPUTE_SECRET` (the PKI bundle too) is unreadable under any other, snapshot records point at `s3:///…`, and stored objects are encrypted with the old key. So a new control-plane stack adopts them instead of creating new ones: `ExistingDeploymentSecretArn`, `ExistingPkiBundleSecretArn`, `ExistingPlatformKeyArn`, `ExistingBlobsBucket`, `ExistingSnapshotsBucket`, `ExistingAuditBucket` and `AdoptLogGroups=true`. Every role may then decrypt with the old key, api and worker may also write with it, and the adopted buckets get the new stack's TLS-only policy. A restore keeps the deployment name: tenant secrets live under `better-compute//`, the only prefix the task roles may read. The repository's runbook has the command. ### Task roles [#task-roles] Every role trusts only `ecs-tasks.amazonaws.com` calling from this account. A base policy, shared by all three, allows `rds-db:connect` as the one database user, `secretsmanager:GetSecretValue` under `better-compute//*`, `kms:Decrypt` with the platform key, and reading snapshots (NodeSpec signs snapshot download URLs wherever it renders). The secret prefix carries the deployment name, so a staging deployment in the same account and region can neither read nor overwrite production's tenant secrets; the config module passes the same prefix to `awsSecretsManagerStore`. On top of that: | Role | Adds | | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | api | Encrypting with the platform key; writing secrets under `better-compute//*`; reading the downloads bucket; reading and writing blobs and snapshots; `events:PutEvents` on the bus | | worker | Encrypting with the platform key; writing secrets; blobs and snapshots; `s3:PutObject` and `s3:PutObjectRetention` on the audit bucket; `events:PutEvents` | | gateway | Nothing | The gateway reads tenant secrets because `serve --roles gateway` runs the secret broker in process: a daemon's `SecretRequest` is answered there, and the broker releases a ref only when it is bound to work the asking node is doing right now, sealed to that node. It never writes secrets, objects or events, and never encrypts. Every role that reads a bucket also has `s3:ListBucket` on it. Without it S3 answers a read of a missing key with 403 AccessDenied instead of 404, so a download that was never published would be an error rather than not found. On the blobs and snapshots buckets the listing is confined to the stores' prefixes (`s3:prefix` like `blobs/*` and `snapshots/*`, passed to the tasks as `BC_BLOBS_PREFIX` and `BC_SNAPSHOTS_PREFIX`), so no task enumerates keys outside them. Only the downloads bucket, which holds public bcd binaries, is listed whole. No statement uses a wildcard resource or a service-wide action. The migrate task has a role of its own that allows only `rds-db:connect`: an upgrade runs the next image's migrate step before that image serves anything, so it gets no secret, bucket, bus or KMS permission. A configuration whose loading calls AWS under `migrate` needs those permissions added. ### Buckets [#buckets] | Bucket | Encryption | Notes | | --------- | ---------- | ------------------------------------------------------------------------------------------------------- | | downloads | SSE-S3 | bcd binaries. Read only by its CloudFront distribution (origin access control), never by the bucket URL | | blobs | SSE-KMS | Support bundles and other blobs. Never public, never behind the CDN | | snapshots | SSE-KMS | `s3SnapshotStore` | | audit | SSE-KMS | Object Lock in compliance mode with `AuditRetentionDays` default retention, for `s3AuditSink` | All of them block public access, enforce bucket-owner ownership, are versioned, and refuse any request that is not made over TLS. Blobs, snapshots and audit are kept when the stack is deleted, and a restore adopts them. ## Deploying [#deploying] The [deployment guide in the repository](https://github.com/better-compute/better-compute/tree/main/deploy/aws) has the commands. In short: 1. **Image.** Build an image whose entrypoint is the `better-compute` CLI and which contains your configuration module (below), push it to ECR, and note its digest. The template accepts only `@sha256:`. 2. **Certificate.** Request an ACM certificate for the API domain in the stack's region. 3. **Network and data.** Deploy `network`, then `data` with `NetworkStack` set and `DataApi=Enabled`. 4. **Database user.** With the Data API and the master secret, create the IAM user once: `CREATE ROLE bc_app LOGIN`, `GRANT rds_iam TO bc_app`, `GRANT CREATE ON DATABASE better_compute TO bc_app`, `GRANT CREATE, USAGE ON SCHEMA public TO bc_app`. Then update the stack with `DataApi=Disabled`. 5. **Control plane at zero.** Deploy `control-plane` with `ApiCount=0 WorkerCount=0 GatewayCount=0`. 6. **PKI bundle.** Put the sealed bundle into the `PkiBundleSecretArn` secret. 7. **Migrate.** Run the migrate task definition once (`aws ecs run-task`, with the `TaskSubnetIds` and `MigrateSecurityGroupIds` outputs). 8. **Scale up and point DNS.** Update the stack with the real counts. With `HostedZoneId` set, the stack writes alias records for both domains; otherwise point them at `ApiLoadBalancerDnsName` and `GatewayLoadBalancerDnsName`. Each later release is a new image digest, and its [expand migrations](/docs/operations/upgrades) run before any process of it: set `MigrateImage` to the new digest (only the migrate task definition changes), run the migrate task, then set `Image` to it and clear `MigrateImage`. The services roll with the circuit breaker, which rolls back a release whose tasks never become healthy. ## The configuration in the image [#the-configuration-in-the-image] The tasks pass everything a configuration needs as environment variables: `AWS_REGION`, `BC_PUBLIC_URL`, `BC_GATEWAY_PUBLIC_HOST`, `BC_GATEWAY_INTERNAL_PORT`, `BC_DATABASE_HOST`, `BC_DATABASE_PORT`, `BC_DATABASE_NAME`, `BC_DATABASE_USER`, `BC_EVENT_BUS`, the four bucket names (`BC_DOWNLOADS_BUCKET`, `BC_BLOBS_BUCKET`, `BC_SNAPSHOTS_BUCKET`, `BC_AUDIT_BUCKET`), the blob and snapshot store prefixes (`BC_BLOBS_PREFIX`, `BC_SNAPSHOTS_PREFIX`), `BC_AUDIT_RETENTION_DAYS`, `BC_KMS_KEY_ARN`, `BC_DEPLOYMENT` and `BC_ROLE`, plus the secrets `BETTER_COMPUTE_SECRET` and `BC_PKI_BUNDLE`. A configuration on AWS wires them into the platform-service factories: ```ts import { readFileSync } from 'node:fs'; import pg from 'pg'; import { defineConfig } from '@better-compute/server'; import { postgres } from '@better-compute/postgres'; import { vps } from '@better-compute/vps'; import { ambientCredentials } from '@better-compute/aws-core'; import { awsSecretsManagerStore, eventBridgeSink, rdsAuthTokenProvider, s3AuditSink, s3BlobStore, s3SnapshotStore, } from '@better-compute/aws'; export default defineConfig(async ({ command, env = process.env } = {}) => { const aws = { region: env.AWS_REGION!, credentials: ambientCredentials({ region: env.AWS_REGION }) }; const sse = { kind: 'aws:kms', keyId: env.BC_KMS_KEY_ARN } as const; // The task roles may list these buckets only under these prefixes. const snapshots = s3SnapshotStore({ ...aws, bucket: env.BC_SNAPSHOTS_BUCKET!, prefix: env.BC_SNAPSHOTS_PREFIX!, sse }); // A restore rendered before the download credentials load cannot be fetched: serve only once they have. if (command === 'serve' && !(await snapshots.warm())) throw new Error('snapshot download credentials did not load'); const host = env.BC_DATABASE_HOST!; const port = Number(env.BC_DATABASE_PORT); const user = env.BC_DATABASE_USER!; return { database: postgres({ pool: new pg.Pool({ host, port, user, database: env.BC_DATABASE_NAME, // The RDS certificate bundle, shipped in the image. ssl: { ca: readFileSync('/etc/better-compute/rds-global-bundle.pem', 'utf8'), rejectUnauthorized: true }, password: rdsAuthTokenProvider({ ...aws, hostname: host, port, username: user }), }), }), secret: env.BETTER_COMPUTE_SECRET!, baseURL: env.BC_PUBLIC_URL!, // The prefix the task roles may use: better-compute/. secrets: [awsSecretsManagerStore({ ...aws, prefix: `better-compute/${env.BC_DEPLOYMENT}` })], sinks: { events: [eventBridgeSink({ ...aws, eventBusName: env.BC_EVENT_BUS })], audit: [s3AuditSink({ ...aws, bucket: env.BC_AUDIT_BUCKET!, prefix: 'audit', sse })], }, downloads: { blob: s3BlobStore({ ...aws, bucket: env.BC_DOWNLOADS_BUCKET! }) }, supportBundles: { blob: s3BlobStore({ ...aws, bucket: env.BC_BLOBS_BUCKET!, prefix: env.BC_BLOBS_PREFIX!, sse }) }, modules: [vps({ snapshots })], // …your tenancy, IAM, PKI (from env.BC_PKI_BUNDLE) and modules, as in examples/postgres. }; }); ``` `ambientCredentials` (from `@better-compute/aws-core`) reads the task role through the ECS container credentials endpoint. It ships with the aws-core credential sources of the AWS foundation phase (A0); `@better-compute/aws`'s own `envCredentialSource` reads only `AWS_ACCESS_KEY_ID` and its siblings, so it cannot use a task role. Name the file `better-compute.config.ts` (the example is TypeScript), or drop the `!` and `as const` for a `.mjs` file. The snapshot store signs restore URLs inside NodeSpec's transaction, from credentials it holds in memory. Until they load, a restore rendered for a host keeps its `s3://` identifier, which bcd cannot fetch, and the instance fails. So the example builds the store first and, for `serve`, refuses to start when `warm()` resolves `false`; ECS then replaces the task. `migrate` renders no NodeSpec and skips the wait. The `host` export's `gateway` should advertise each task's own address as its internal URL (from the ECS task metadata endpoint), listen on `BC_GATEWAY_INTERNAL_PORT`, and name `BC_GATEWAY_PUBLIC_HOST`. The [observability on AWS](/docs/operations/aws) page covers CloudWatch metrics through EMF: the task logs already go to CloudWatch Logs, so `emfMetricsExporter` needs no extra permission. ## Alarms [#alarms] The stack publishes to its SNS topic (optionally with an `AlarmEmail` subscription) when the api answers 5xx responses, an api or gateway target is unhealthy, no worker task is running, or Aurora runs near its maximum capacity. Add your own on the same topic. ## Access logs [#access-logs] Set `AccessLogsBucket` to a bucket in the same account and region, prepared for log delivery (its policy lets the Elastic Load Balancing log delivery principal for your region and `logging.s3.amazonaws.com` write to it), and the stack turns on ALB access logs under `alb/-api/` and S3 server access logs for the four buckets under `s3/-/`. Empty (the default) turns them off. The gateway's NLB has no access logs to give: it is a TCP passthrough, and NLB access logs cover TLS listeners only. Keep the logs bucket outside the stack, so its retention is yours and deleting the stack never deletes the evidence. ## Limits and choices [#limits-and-choices] * `control-plane.json` is larger than CloudFormation's 51,200-byte inline limit, pretty-printed or minified with `--compact`: deploy it with `--s3-bucket`. The command says when a body is over, and offers `--compact` only when the minified body would fit. * The templates use the `AWS::Partition`, `AWS::Region` and `AWS::URLSuffix` pseudo parameters for ARNs, but they are tested against the commercial `aws` partition only. CloudFront and some interface endpoint names differ in `aws-cn`. * One set of stacks is one region. Gateways in further regions (daemons dial the gateways of their own region) need a network of their own and a gateway-only variant of the control-plane stack, which the templates do not render yet; every region keeps writing to the one database. * Not rendered: AWS App Runner (an alternative to ECS for the API), AWS Global Accelerator in front of the gateways, RDS Proxy, and an AWS Budgets action. The first three are optional; the processes connect to the Aurora endpoint directly, and running them behind RDS Proxy is not verified (the migrator and some reconcilers take advisory locks, which pin proxy sessions). The live suite's README recommends a Budgets action for its sandbox account, set up by hand. * The templates are checked by `lintTemplate` and unit tests only. CloudFormation's own `ValidateTemplate` runs in the AWS live suite (`tests/live/aws/templates.live.test.ts`), which needs sandbox credentials. * Deletion protection is on for the database and both load balancers. Turn it off on purpose before deleting a stack.