Continuous deployment — factory self-deploy via a containerized runner

On every push to main, .github/workflows/deploy.yml first confirms on a GitHub-hosted runner that the commit passed the green-gate check (the verify-gate job — it queries the check-run rather than re-running the suite, so it costs seconds and refuses to deploy an ungated commit). Only then does the self-hosted runner that itself is a container on the docker host (mimi) safely redeploy the factory:

verify green-gate         # ungated commit => the deploy stops here, nothing is touched
docker compose build      # build the new image first — the running factory stays up if this fails
tag current image :prev   # for rollback
docker compose up -d      # swap
health-check the factory  # retry ~60s (over the shared network, by DNS)
rollback to :prev if the new container is unhealthy

Why a container runner (docker-out-of-docker)

The runner container mounts the host's docker socket, so when the deploy job runs factory.sh redeploy it builds and swaps the factory as a sibling container on the same host daemon — exactly as a manual factory.sh up would, on the host's ports. This is docker-out-of-docker (sharing the host daemon via its socket), not docker-in-docker (a nested daemon). It works cleanly here because the factory's compose uses a named volume (factory-data:/data), not host bind mounts, and the build context is streamed to the daemon — so nothing depends on paths inside the runner container.

The runner also joins the factory's shared network (FACTORY_NETWORK), so its health-check reaches the factory by its DNS alias on the container port. No SSH key and no deploy secret is ever exposed to GitHub, and no deploy credential ever lands in an agent-operated container.

One-time setup

1. Provide a GitHub registration credential

The runner registers itself with GitHub on first boot. Give it one of:

RUNNER_PAT — a repo-scoped GitHub personal access token (fine-grained: the aif repo with Administration: read & write, or a classic token with repo). The runner image uses it to mint registration tokens itself, and can re-register across restarts. Recommended.
RUNNER_TOKEN — a one-time registration token from repo → Settings → Actions → Runners → New self-hosted runner (valid ~1 hour, single use).

Create this token yourself in GitHub — never paste a token into chat, and never commit it.

Where to put it. Not in the main .env — that file is stored verbatim as the FACTORY_ENV deploy secret and shipped to the deploy job, which has no need for the runner's registration credential. Instead, put it in .env.runner (gitignored; copy .env.runner.example), which factory.sh runner sources on its own:

cp .env.runner.example .env.runner
#   set RUNNER_PAT=... (recommended) — or RUNNER_TOKEN=... for a one-time token

Or skip the file and pass it inline: RUNNER_PAT=... ./scripts/factory.sh runner up. A short-lived RUNNER_TOKEN is best passed inline rather than saved.

2. Run everything from your operator machine — no shell on mimi

factory.sh drives mimi's docker daemon over DOCKER_HOST=ssh://$DEPLOY_SSH, so every runner command runs from the same machine you already deploy the factory from. runner up creates the container on mimi over that SSH tunnel; you never open a shell on mimi. (Set FACTORY_DEPLOY_LOCAL=1 only if you actually run these on the host itself.)

The one thing that isn't a CLI knob is a Docker Desktop GUI setting on mimi — but it's on by default, so usually there's nothing to do:

The runner mounts /var/run/docker.sock. On Docker Desktop for Mac that path is a symlink to ~/.docker/run/docker.sock, and bind-mounting it into a container works by default (verified: a sibling container reaches the daemon through it). runner up runs a preflight that confirms this from your machine and fails with a clear message if it can't.
If that preflight fails, enable Settings → Advanced → "Allow the default Docker socket to be used" on mimi, or set DOCKER_SOCK to the socket path as it exists on mimi (an absolute path — not $HOME, which would expand on your machine, not mimi):
```
DOCKER_SOCK=/Users/<mimi-user>/.docker/run/docker.sock \
  RUNNER_PAT=<pat> ./scripts/factory.sh runner up
```
factory.sh runner honors DOCKER_SOCK (default /var/run/docker.sock).

3. Bring the runner up (from your operator machine)

With your .env present (it supplies DEPLOY_SSH, FACTORY_NAME, FACTORY_NETWORK, FACTORY_HOSTNAME, DASHBOARD_CONTAINER_PORT):

# repo-scoped PAT (recommended):
RUNNER_PAT=<your-github-pat> ./scripts/factory.sh runner up

# ...or a one-time registration token:
RUNNER_TOKEN=<registration-token> ./scripts/factory.sh runner up

./scripts/factory.sh runner logs     # watch it register ("Listening for Jobs")
./scripts/factory.sh runner ps

runner up preflights the socket mount, then starts the runner container on mimi on FACTORY_NETWORK, labeled mimi (the workflow targets runs-on: [self-hosted, mimi]), with the docker socket mounted. Manage it — all from your machine — with:

./scripts/factory.sh runner up|down|rm|logs|ps
#   down = stop (keep the work volume);  rm = stop + delete the volume

Confirm it appears Idle under repo → Settings → Actions → Runners.

4. Add the factory runtime env as a repo secret

The redeployed container must come up with the same runtime env it was first deployed with (same OAuth client, SERVICE_TOKEN, ports, FACTORY_NAME, network …) — otherwise the credentials/ports change out from under you. factory.sh redeploy refuses to run if GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET or SERVICE_TOKEN is missing from that env, because with no shared URL token left there would be no way back into the dashboard.

Store your factory .env verbatim as a single repo secret:

GitHub: repo → Settings → Secrets and variables → Actions → New repository secret
Name: FACTORY_ENV
Value: the full contents of the .env you deploy the factory with (it contains FACTORY_NAME, DASHBOARD_PORT, DASHBOARD_CONTAINER_PORT, GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, FACTORY_PUBLIC_URL, SERVICE_TOKEN, FACTORY_NETWORK, FACTORY_HOSTNAME, GH_TOKEN, GIT_AUTHOR_*, etc.).

The workflow writes it to .env at deploy time and removes it after. DEPLOY_SSH in that env is ignored — the workflow sets FACTORY_DEPLOY_LOCAL=1 so factory.sh redeploy uses the docker daemon reached through the mounted socket.

The health-check target is supplied by the runner container's environment (FACTORY_HEALTH_HOST = FACTORY_HOSTNAME, FACTORY_HEALTH_PORT = DASHBOARD_CONTAINER_PORT, set in runner/compose.yaml), so the deploy job reaches the factory by DNS over the shared network. The probe hits /health — the only unauthenticated route left; / now redirects an anonymous request to /login.

5. (Optional) Let the deploy notice a dead runner — `RUNNER_STATUS_PAT`

A wedged runner does not make a deploy fail; it makes it wait. The job sits queued until GitHub cancels it at its 24-hour ceiling, and because the running factory is untouched the whole time, it reads as "deploy pending" rather than as a problem. timeout-minutes does not help — that bounds a job which is running, not one waiting to be picked up (measured: a job with timeout-minutes: 2 on a label no runner had was still queued at 2m25s).

The runner-online job in deploy.yml closes that gap by asking GitHub whether any online runner carries the mimi label before the deploy is queued. That list needs repo Administration: read, which GITHUB_TOKEN cannot be granted, so it takes a PAT:

GitHub: repo → Settings → Secrets and variables → Actions → New repository secret
Name: RUNNER_STATUS_PAT
Value: a fine-grained token for this repo with Administration: Read (a classic repo token also works). The RUNNER_PAT in .env.runner already has the access — reuse it, or mint a read-only one.

It is optional by design. With no secret set, or with a token that cannot read the list, the job prints a warning and passes, so a factory that has not set one deploys exactly as it did before. Only a token that can read the list and finds no online mimi runner blocks the deploy — with the fix in the log:

./scripts/factory.sh runner down && ./scripts/factory.sh runner up

Note that is a recreate, not a restart. docker restart cannot fix a wedged runner — it is what causes it (see Notes / options).

6. The stage factory (AIF-653)

A merge to main deploys a stage factory; production changes only when someone promotes a commit. Stage is a second factory on the same host, made by the same factory.sh redeploy with a different env file, so everything that follows from FACTORY_NAME — container, named volume, image, DNS alias — is its own. It starts with fresh data and loads no project (the repo's projects ship as .yaml.txt), so it never drives a real repo.

Write .env.stage (git-ignored) from your production .env, changing: FACTORY_NAME=aif-stage, DASHBOARD_PORT, DEV_SSH_PORT and the STAGE_PROXY_PORT/STAGE_PROXY_PORT_END range to ports free on mimi, and FACTORY_PUBLIC_URL (if set) to stage's own address. Keep FACTORY_NETWORK (stage joins production's network under its own alias). Leave out credentials stage has no business holding — LINEAR_API_KEY, FLY_API_TOKEN, GH_TOKEN. FACTORY_NAME is the one that matters most: it is what keeps stage off production's container and data volume, and it is also stage's alias on the shared network — so it must be whatever stage's tunnel route points at. Left at production's value, the stage deploy rebuilds PRODUCTION with stage's env and its health check passes anyway. scripts/check-stage-env.sh refuses that (and any shared alias or host port); both the deploy and the next step run it. Give stage its own SESSION_COOKIE_NAME (e.g. fsid_stage), and keep its SESSION_COOKIE_DOMAIN to its own host. Production scopes its fsid cookie to the parent domain, so the browser sends it to stage as well; under the same name stage reads production's session first, finds no such session in its own database, and every sign-in lands back on /login. The check refuses that too. Leave production on the default — the iOS and desktop apps look for fsid by name.
./scripts/push-env.sh --stage — sets the FACTORY_STAGE_ENV secret.
To sign in to stage in a browser, give it a tunnel route, and add <stage url>/auth/google/callback to the Google OAuth client's redirect URIs.
The production factory's GH_TOKEN needs Actions: read & write on this repo, so the Deployments tab can dispatch the promotion and follow the run.

Until FACTORY_STAGE_ENV exists, a push to main fails at the stage redeploy with a message saying so. Production is untouched and can still be promoted.

Deploying merges straight to production (AIF-655). Where a merge goes is one repository variable, DEPLOY_ON_MERGE — flipped without a commit, in the repo's Settings → Secrets and variables → Actions → Variables, or:

gh variable set DEPLOY_ON_MERGE --body production   # merges deploy production
gh variable delete DEPLOY_ON_MERGE                  # back to stage (the default)

Or from the project's Deployments tab: the Merges deploy to switch (AIF-660) does exactly those two calls, because .factory/deployments.yaml names the variable under merge_target. It needs GH_TOKEN to have Variables: write on the repo, and it records who flipped it in the tab's recent actions.

Only production (any case) opts in; unset, stage or anything else means stage, so a typo never ships to production. A dispatch (Promote, or Run workflow) goes to its target input — production unless it says stage. A merge going to production takes the production path unchanged — verify-gate still has to pass, FACTORY_ENV is used, it queues with promotions, and a production deployment is recorded. The flip takes effect from the next run. While it is set, stage gets no merges, so the Deployments tab will show stage falling behind production. Each run's summary says which target it picked and the variable value behind it.

How a change ships

You submit an aif issue; the agent works it (branch → build/typecheck → PR pushed). You review and approve; the merge stage lands it on main.
The push to main triggers deploy-factory, which deploys stage (or production, when DEPLOY_ON_MERGE=production — see §6). verify-gate confirms the commit's green-gate check passed (following the commit back to its PR, since the check-run lives on the PR head). Then the mimi runner container checks out that commit, materialises .env from FACTORY_STAGE_ENV, and runs ./scripts/factory.sh redeploy against the host daemon via the mounted socket. A staging GitHub deployment is recorded.
Promote — the project's Deployments tab lists what is on stage but not on production; its Promote button dispatches deploy-factory with that exact sha. The run verifies the gate on that sha, checks it out, redeploys production from FACTORY_ENV, and records a production deployment — which is also when the factory's issues get their deployed mark (AIF-459). The tab also has Deploy latest → stage / production buttons (AIF-662): they dispatch deploy-factory with main's head and target=stage|production — for re-shipping after a failed merge deploy, or shipping main straight to production. Declared under deploy_latest in .factory/deployments.yaml.
Build fails → the running factory is untouched. New container unhealthy → it auto-rolls-back to the previous image. Healthy → you're on the new version.

Stage and production each have their own concurrency group, so a merge never queues behind a promotion. The contract the tab reads is .factory/deployments.yaml (format in factory/server/src/deployconfig.ts): it takes each environment's commit from the latest successful GitHub deployment in staging / production — the ones step 2 and 3 record — rather than calling either factory's /health.

Watch a deploy under the repo's Actions tab. You can also run deploy-factory by hand there (Run workflow): it deploys production at sha, or main's head when sha is blank. Roll back by promoting an older sha. The button takes a skip_gate_check input for the emergency case where you must redeploy a commit the gate never blessed.

Notes / options

A wedged runner is RECREATED, never restarted. A restart reuses the container's writable layer, so the registration from its first boot is still there — and the image's entrypoint clears .runner but not .credentials. That half-configured pair fails both ways (config.sh refuses with *"already configured"; removal fails with "Value cannot be null (configuredSettings)"*), the container exits 2, and restart: unless-stopped repeats it forever. So docker restart cannot fix this state — it is what produces it. Use ./scripts/factory.sh runner down && ./scripts/factory.sh runner up, which gives a fresh layer. runner/compose.yaml now clears both files in an entrypoint wrapper, so a host reboot re-registers on its own; the recreate is for a container created before that fix, or for any other stuck state.
Host runner instead of a container: if you'd rather run the runner as a raw host service, register it the usual way (GitHub's shown config.sh/svc.sh steps) with the label mimi and add its user to the docker group. The same workflow then health-checks the factory at 127.0.0.1:${DASHBOARD_PORT} (the published host port) — factory.sh redeploy defaults there when the container's FACTORY_HEALTH_* vars are absent.
Extra approval gate: add environment: production to the job and require reviewers on that environment for a second human click before each deploy. (The factory already gates via issue approval before merge, so this is optional.)
Config-only changes (tools/workflows YAML under config/) don't need a full redeploy — they take effect via factory.sh config-push/reload. The redeploy rebuilds the whole image, which also covers them.
Multiple factories on one host: give each its own runner (unique FACTORY_NAME → its own runner project/label) + FACTORY_ENV secret (or use GitHub Environments) and a workflow per instance.