Deploying on AWS

@better-compute/aws renders the reference deployment of the control plane on AWS as three CloudFormation JSON templates. They lay out the split-roles topology on AWS services: the api behind an L7 load balancer, the gateway behind an L4 load balancer with TLS passthrough, workers with no listener, and Aurora PostgreSQL with IAM authentication. Nothing here is required: the templates are a starting point you can read, diff and change, not an installer.

better-compute aws deploy-templates --out cfn            # network.json, data.json, control-plane.json
better-compute aws deploy-templates --out cfn --zones 2 --nat single --readers 0 --name bc-stage

The command makes no AWS call. It writes only new files, and refuses before writing anything if one exists.

The three stacks

Each stack exports what the next one needs as <stack name>-<output> (for example bc-prod-network-PrivateSubnetIds), and the later stacks take the earlier stack names as the NetworkStack and DataStack parameters. Structural choices (zones, NAT mode, readers, the deployment name) are command flags; everything that differs between environments (the image, the domains, the certificate, task counts, sizes) is a stack parameter.

Processes

Every task runs in the private subnets without a public address, with a read-only root file system, all Linux capabilities dropped, awslogs in non-blocking mode, and the deployment circuit breaker with rollback. All of them also wear the data stack's client security group, the only group the database admits on 5432.

The ALB uses the ELBSecurityPolicy-TLS13-1-2-2021-06 policy, redirects HTTP to HTTPS, drops invalid headers, and has a WAF web ACL in front: a per-IP rate limit (WafRateLimit requests per five minutes), the AWS IP reputation list, the common rule set, and known bad inputs. The common rule set's 8 KiB body rule only counts, since API bodies can be larger. The idle timeout is an hour, for interactive tunnels.

The NLB never terminates TLS: the gateway authenticates each daemon by its client certificate, which a terminating balancer would strip. Its target group turns client IP preservation off, so tasks only need to admit the NLB's own security group.

Secrets and keys

The templates take no secret as a parameter (lintTemplate refuses NoEcho parameters):

• BETTER_COMPUTE_SECRET is generated by Secrets Manager (64 characters) and reaches the tasks as an ECS secret.
• The sealed PKI bundle (the JSON better-compute pki init writes) goes into the PkiBundleSecret secret, which the tasks get as BC_PKI_BUNDLE. The stack creates it with a random placeholder, and you put the real bundle into it.
• The database master password is generated and kept by RDS in Secrets Manager. The processes never use it: they connect as an IAM-authenticated user with rdsAuthTokenProvider.

One rotating KMS key encrypts the two secrets, the blobs, snapshots and audit buckets, every log group and the alarm topic. The database has a key of its own. Both keys are kept when a stack is deleted.

Restoring under a new stack

The secrets, the platform key, the log groups and the blobs, snapshots and audit buckets outlive a deleted stack, and a restored database depends on all of them: data sealed with the old BETTER_COMPUTE_SECRET (the PKI bundle too) is unreadable under any other, snapshot records point at s3://<old snapshots bucket>/…, and stored objects are encrypted with the old key. So a new control-plane stack adopts them instead of creating new ones: ExistingDeploymentSecretArn, ExistingPkiBundleSecretArn, ExistingPlatformKeyArn, ExistingBlobsBucket, ExistingSnapshotsBucket, ExistingAuditBucket and AdoptLogGroups=true. Every role may then decrypt with the old key, api and worker may also write with it, and the adopted buckets get the new stack's TLS-only policy. A restore keeps the deployment name: tenant secrets live under better-compute/<name>/, the only prefix the task roles may read. The repository's runbook has the command.

Task roles

Every role trusts only ecs-tasks.amazonaws.com calling from this account. A base policy, shared by all three, allows rds-db:connect as the one database user, secretsmanager:GetSecretValue under better-compute/<name>/*, kms:Decrypt with the platform key, and reading snapshots (NodeSpec signs snapshot download URLs wherever it renders). The secret prefix carries the deployment name, so a staging deployment in the same account and region can neither read nor overwrite production's tenant secrets; the config module passes the same prefix to awsSecretsManagerStore. On top of that:

The gateway reads tenant secrets because serve --roles gateway runs the secret broker in process: a daemon's SecretRequest is answered there, and the broker releases a ref only when it is bound to work the asking node is doing right now, sealed to that node. It never writes secrets, objects or events, and never encrypts.

Every role that reads a bucket also has s3:ListBucket on it. Without it S3 answers a read of a missing key with 403 AccessDenied instead of 404, so a download that was never published would be an error rather than not found. On the blobs and snapshots buckets the listing is confined to the stores' prefixes (s3:prefix like blobs/* and snapshots/*, passed to the tasks as BC_BLOBS_PREFIX and BC_SNAPSHOTS_PREFIX), so no task enumerates keys outside them. Only the downloads bucket, which holds public bcd binaries, is listed whole.

No statement uses a wildcard resource or a service-wide action. The migrate task has a role of its own that allows only rds-db:connect: an upgrade runs the next image's migrate step before that image serves anything, so it gets no secret, bucket, bus or KMS permission. A configuration whose loading calls AWS under migrate needs those permissions added.

Buckets

All of them block public access, enforce bucket-owner ownership, are versioned, and refuse any request that is not made over TLS. Blobs, snapshots and audit are kept when the stack is deleted, and a restore adopts them.

Deploying

The deployment guide in the repository has the commands. In short:

1. Image. Build an image whose entrypoint is the better-compute CLI and which contains your configuration module (below), push it to ECR, and note its digest. The template accepts only <repository>@sha256:<digest>.
2. Certificate. Request an ACM certificate for the API domain in the stack's region.
3. Network and data. Deploy network, then data with NetworkStack set and DataApi=Enabled.
4. Database user. With the Data API and the master secret, create the IAM user once: CREATE ROLE bc_app LOGIN, GRANT rds_iam TO bc_app, GRANT CREATE ON DATABASE better_compute TO bc_app, GRANT CREATE, USAGE ON SCHEMA public TO bc_app. Then update the stack with DataApi=Disabled.
5. Control plane at zero. Deploy control-plane with ApiCount=0 WorkerCount=0 GatewayCount=0.
6. PKI bundle. Put the sealed bundle into the PkiBundleSecretArn secret.
7. Migrate. Run the migrate task definition once (aws ecs run-task, with the TaskSubnetIds and MigrateSecurityGroupIds outputs).
8. Scale up and point DNS. Update the stack with the real counts. With HostedZoneId set, the stack writes alias records for both domains; otherwise point them at ApiLoadBalancerDnsName and GatewayLoadBalancerDnsName.

Each later release is a new image digest, and its expand migrations run before any process of it: set MigrateImage to the new digest (only the migrate task definition changes), run the migrate task, then set Image to it and clear MigrateImage. The services roll with the circuit breaker, which rolls back a release whose tasks never become healthy.

The configuration in the image

The tasks pass everything a configuration needs as environment variables: AWS_REGION, BC_PUBLIC_URL, BC_GATEWAY_PUBLIC_HOST, BC_GATEWAY_INTERNAL_PORT, BC_DATABASE_HOST, BC_DATABASE_PORT, BC_DATABASE_NAME, BC_DATABASE_USER, BC_EVENT_BUS, the four bucket names (BC_DOWNLOADS_BUCKET, BC_BLOBS_BUCKET, BC_SNAPSHOTS_BUCKET, BC_AUDIT_BUCKET), the blob and snapshot store prefixes (BC_BLOBS_PREFIX, BC_SNAPSHOTS_PREFIX), BC_AUDIT_RETENTION_DAYS, BC_KMS_KEY_ARN, BC_DEPLOYMENT and BC_ROLE, plus the secrets BETTER_COMPUTE_SECRET and BC_PKI_BUNDLE. A configuration on AWS wires them into the platform-service factories:

import { readFileSync } from 'node:fs';
import pg from 'pg';
import { defineConfig } from '@better-compute/server';
import { postgres } from '@better-compute/postgres';
import { vps } from '@better-compute/vps';
import { ambientCredentials } from '@better-compute/aws-core';
import {
  awsSecretsManagerStore, eventBridgeSink, rdsAuthTokenProvider,
  s3AuditSink, s3BlobStore, s3SnapshotStore,
} from '@better-compute/aws';

export default defineConfig(async ({ command, env = process.env } = {}) => {
  const aws = { region: env.AWS_REGION!, credentials: ambientCredentials({ region: env.AWS_REGION }) };
  const sse = { kind: 'aws:kms', keyId: env.BC_KMS_KEY_ARN } as const;
  // The task roles may list these buckets only under these prefixes.
  const snapshots = s3SnapshotStore({ ...aws, bucket: env.BC_SNAPSHOTS_BUCKET!, prefix: env.BC_SNAPSHOTS_PREFIX!, sse });
  // A restore rendered before the download credentials load cannot be fetched: serve only once they have.
  if (command === 'serve' && !(await snapshots.warm())) throw new Error('snapshot download credentials did not load');
  const host = env.BC_DATABASE_HOST!;
  const port = Number(env.BC_DATABASE_PORT);
  const user = env.BC_DATABASE_USER!;
  return {
    database: postgres({
      pool: new pg.Pool({
        host, port, user, database: env.BC_DATABASE_NAME,
        // The RDS certificate bundle, shipped in the image.
        ssl: { ca: readFileSync('/etc/better-compute/rds-global-bundle.pem', 'utf8'), rejectUnauthorized: true },
        password: rdsAuthTokenProvider({ ...aws, hostname: host, port, username: user }),
      }),
    }),
    secret: env.BETTER_COMPUTE_SECRET!,
    baseURL: env.BC_PUBLIC_URL!,
    // The prefix the task roles may use: better-compute/<deployment name>.
    secrets: [awsSecretsManagerStore({ ...aws, prefix: `better-compute/${env.BC_DEPLOYMENT}` })],
    sinks: {
      events: [eventBridgeSink({ ...aws, eventBusName: env.BC_EVENT_BUS })],
      audit: [s3AuditSink({ ...aws, bucket: env.BC_AUDIT_BUCKET!, prefix: 'audit', sse })],
    },
    downloads: { blob: s3BlobStore({ ...aws, bucket: env.BC_DOWNLOADS_BUCKET! }) },
    supportBundles: { blob: s3BlobStore({ ...aws, bucket: env.BC_BLOBS_BUCKET!, prefix: env.BC_BLOBS_PREFIX!, sse }) },
    modules: [vps({ snapshots })],
    // …your tenancy, IAM, PKI (from env.BC_PKI_BUNDLE) and modules, as in examples/postgres.
  };
});

ambientCredentials (from @better-compute/aws-core) reads the task role through the ECS container credentials endpoint. It ships with the aws-core credential sources of the AWS foundation phase (A0); @better-compute/aws's own envCredentialSource reads only AWS_ACCESS_KEY_ID and its siblings, so it cannot use a task role. Name the file better-compute.config.ts (the example is TypeScript), or drop the ! and as const for a .mjs file.

The snapshot store signs restore URLs inside NodeSpec's transaction, from credentials it holds in memory. Until they load, a restore rendered for a host keeps its s3:// identifier, which bcd cannot fetch, and the instance fails. So the example builds the store first and, for serve, refuses to start when warm() resolves false; ECS then replaces the task. migrate renders no NodeSpec and skips the wait.

The host export's gateway should advertise each task's own address as its internal URL (from the ECS task metadata endpoint), listen on BC_GATEWAY_INTERNAL_PORT, and name BC_GATEWAY_PUBLIC_HOST. The observability on AWS page covers CloudWatch metrics through EMF: the task logs already go to CloudWatch Logs, so emfMetricsExporter needs no extra permission.

Alarms

The stack publishes to its SNS topic (optionally with an AlarmEmail subscription) when the api answers 5xx responses, an api or gateway target is unhealthy, no worker task is running, or Aurora runs near its maximum capacity. Add your own on the same topic.

Access logs

Set AccessLogsBucket to a bucket in the same account and region, prepared for log delivery (its policy lets the Elastic Load Balancing log delivery principal for your region and logging.s3.amazonaws.com write to it), and the stack turns on ALB access logs under alb/<name>-api/ and S3 server access logs for the four buckets under s3/<name>-<bucket>/. Empty (the default) turns them off. The gateway's NLB has no access logs to give: it is a TCP passthrough, and NLB access logs cover TLS listeners only. Keep the logs bucket outside the stack, so its retention is yours and deleting the stack never deletes the evidence.

Limits and choices

• control-plane.json is larger than CloudFormation's 51,200-byte inline limit, pretty-printed or minified with --compact: deploy it with --s3-bucket. The command says when a body is over, and offers --compact only when the minified body would fit.
• The templates use the AWS::Partition, AWS::Region and AWS::URLSuffix pseudo parameters for ARNs, but they are tested against the commercial aws partition only. CloudFront and some interface endpoint names differ in aws-cn.
• One set of stacks is one region. Gateways in further regions (daemons dial the gateways of their own region) need a network of their own and a gateway-only variant of the control-plane stack, which the templates do not render yet; every region keeps writing to the one database.
• Not rendered: AWS App Runner (an alternative to ECS for the API), AWS Global Accelerator in front of the gateways, RDS Proxy, and an AWS Budgets action. The first three are optional; the processes connect to the Aurora endpoint directly, and running them behind RDS Proxy is not verified (the migrator and some reconcilers take advisory locks, which pin proxy sessions). The live suite's README recommends a Budgets action for its sandbox account, set up by hand.
• The templates are checked by lintTemplate and unit tests only. CloudFormation's own ValidateTemplate runs in the AWS live suite (tests/live/aws/templates.live.test.ts), which needs sandbox credentials.
• Deletion protection is on for the database and both load balancers. Turn it off on purpose before deleting a stack.