Operations

Environments, deploys, migrations, backups and health.

How the system is deployed, checked, backed up and watched. The operational runbooks (docs/operations/) name the environments' resources and their current state, so this public site lists them by title and section only; the full text is in the repository. The browser suite's guide is included as it stands.

Environments

One Worker definition serves every environment, chosen by the mode passed to the cf CLI: cf deploy --mode staging or --mode production picks the Worker, its database, buckets, queues and domain from apps/web/cloudflare.config.ts. Any other mode is local development, with a placeholder database and no domain.

This documentation is its own Worker, chambers-docs, serving static files only (apps/docs/cloudflare.config.ts), at chambers-docs.whymelabs.com. It is deployed with production, by the same reviewed workflow run.

Deploying

A deploy applies the migrations to the environment's remote D1 first, then uploads the Worker with cf deploy --mode <environment>, then runs a smoke test against the health route, both sign-in pages and one API call. Secrets are set per environment with the cf CLI and never committed; local development reads them from apps/web/.dev.vars. Code is rolled back by redeploying an earlier version, data by D1 Time Travel or the daily backup.

The runbook, docs/operations/deploy.md: Deploying Chambers App. Sections: Email · Secrets (per environment) · Chamber addresses and brands · Uploaded media (R2 and Cloudflare Images) · News fetching (a Queue) · News analysis (Workers AI) · Business matching (Vectorize and a Queue) · Rate limits, Turnstile and security headers · Deploying · From GitHub Actions · The demo chamber on staging · The documentation site · Rolling back.

Continuous integration and the deploy workflow

Every push runs the checks, a dry-run deploy with no credentials, and the browser suite. Pushes to main deploy staging; production deploys only when run by hand through the production GitHub environment.

CI: .github/workflows/ci.yml

check

1. pnpm install --frozen-lockfile
2. pnpm typecheck
3. pnpm test
4. pnpm build

deploy-dry-run

The deploy steps with no Cloudflare credentials: every migration applied to a fresh local D1 (the same engine as Cloudflare's, which refuses some SQL that plain SQLite accepts), then each environment's Worker built and checked with --dry-run, uploading nothing. The real deploy is .github/workflows/deploy.yml.

1. pnpm install --frozen-lockfile
2. Apply every migration to a fresh local D1
3. Dry-run deploy, staging
4. Dry-run deploy, production
5. Build the documentation site
6. Dry-run deploy, documentation

e2e

The browser suite (apps/web/e2e, docs/operations/e2e.md): starts the local Worker on a fresh, seeded database and drives it in Chromium at phone width. Separate from `pnpm test`.

1. pnpm install --frozen-lockfile
2. npx playwright install --with-deps chromium
3. pnpm e2e

Deploy: .github/workflows/deploy.yml

Staging deploys on every push to main. Production deploys only when run by hand, through the "production" GitHub environment (configure required reviewers on it).

deploy

1. pnpm install --frozen-lockfile
2. pnpm typecheck
3. pnpm test
4. Check Cloudflare credentials
5. Dry run (no Cloudflare credentials)
6. Apply database migrations
7. Deploy
8. Smoke test

docs

The documentation site (apps/docs), its own static-assets Worker at chambers-docs.whymelabs.com. It is public, so it is published only with a production deploy: after the app's job succeeds, through the same reviewed "production" environment. Every other run (staging, or no credentials) builds it and dry-runs it. One concurrency group for every docs publish, so two runs never upload the docs Worker at once.

1. pnpm install --frozen-lockfile
2. Build the documentation site
3. Check Cloudflare credentials
4. Documentation dry run
5. Deploy the documentation site

Migrations

Migrations are plain SQL files in packages/core/migrations, applied in order with cf d1 migrations apply: locally with pnpm --filter web db:migrate:local, in CI to a fresh local D1 (which refuses some SQL that plain SQLite accepts), and to the remote database as the first step of every deploy, so a failing migration stops the deploy. Apply a migration to the remote database before deploying code that needs it. Capability tests apply every migration to a real SQLite on each run. 47 migrations today; see the list.

Backups and the restore drill

D1 Time Travel can restore a database to any minute in the last 30 days. Besides it, the daily scheduled backup writes every table as JSON Lines to a private R2 bucket under backups/<date>/, with a manifest written last, and keeps 30 days. Uploaded images are not in the backup. After any restore, every data deletion completed since the restore point is done again.

The runbook, docs/operations/restore.md: Backups and restoring. Sections: The daily backup · Restoring with Time Travel · Restoring a logical backup · Rehearsal.

Observability and health

GET /api/health runs system.health and answers 200 with the database's state, or 503; the deploy workflow's smoke test calls it. The operator's health page (/admin/health, capability operations.health) reads scheduled_runs for each piece of scheduled work, the outbox, news fetching and backups, and says what is overdue or failing. Logs are structured JSON, scrubbed of emails, phone numbers and tokens before they are written, and kept with invocation logs in Workers Logs. A capability that throws is logged with its name, actor id and operation id, and nothing personal.

The browser suite

A small committed suite in apps/web/e2e/ covers the paths that exist only in the interface, as the specs ask: signing in with a code, applying and approval (#16); publishing an event, registering in a wave, a waitlist offer, scanning a member card at the door and the showcase (#27); editing a profile, its member and Friend previews, and directory search and filters (#28); installability, the service worker, and turning push on from the inbox (#31). Everything else is tested at the capability seam (pnpm test); this suite is not part of pnpm test.

Running it

pnpm e2e                      # from the repo root (or `pnpm e2e` in apps/web)
pnpm --filter web e2e events      # one file

It needs Chromium for Playwright. Where it is not already installed (as it is in this workspace's image), run npx playwright install --with-deps chromium in apps/web once. The suite uses the full Chromium build in its new headless mode (channel: "chromium"): the lighter headless shell refuses notification permission.

Every run is self-contained (apps/web/e2e/start.ts, which Playwright starts as its web server):

1. Deletes and recreates apps/web/.e2e-state, a local state directory apart from the one pnpm dev uses (.cloudflare/state). The Vite config persists to it when CHAMBERS_STATE_DIR is set.
2. Applies the migrations to it (cf d1 migrations apply --local --persist-to .e2e-state).
3. Seeds it with e2e/seed.ts, which runs real capabilities against the local D1 file: the platform operator creates "E2E Chamber" and appoints its chamber admin, who creates the branches and committees and appoints the KL YEC Exco admin; members join by applying and being approved, and save their profiles; a Friend applies. One event that has already ended is made on a clock three days back, since events cannot be created in the past. Nobody appoints themselves. The people are listed in e2e/people.ts.
4. Starts vite dev on port 5370 (strictly that port), which Playwright stops when the run ends.

The suite runs one test at a time at 390px wide, and checks for sideways scrolling on each page it visits. Tracing is off; pass --trace on to record one. Results and failure snapshots go to apps/web/e2e/.results (ignored by git).

When writing a test, E2E_REUSE=1 pnpm --filter web e2e <file> runs against a server you started yourself with node --experimental-strip-types e2e/start.ts in apps/web, instead of starting a fresh one. State then accumulates between runs, and each address can ask for a limited number of codes per fifteen minutes, so restart the server when a test needs a clean slate.

How sign-in codes are read

Locally the email binding writes each email's text to a file under apps/web/.cloudflare/tmp/email/. The files do not name the recipient, so the fixture (emailedCode in e2e/fixtures.ts) notes the time just before it asks for a code and takes the newest file written since then that holds a sign-in code ("Your sign-in code is …" or "您的登录验证码是 …"). That is why the suite runs on one worker.

Each sign-in sends its own cf-connecting-ip header, so every sign-in comes from a different network as far as the edge limits and the Turnstile threshold are concerned. A session is kept for the rest of the run after the first sign-in (Playwright storage state, saved again when each page closes), so a person is asked for at most a few codes per run, under the five per fifteen minutes allowed.

What is stood in for

• Camera. The door test asks the API for the member's card code, draws it as a QR code into a Y4M video (e2e/fake-camera.ts), and launches Chromium with --use-fake-device-for-media-stream --use-file-for-fake-video-capture=<video>. The door page's own camera and decoder read it.
• Push service. Headless Chromium cannot reach a push service, so pushManager.subscribe always fails there. The push test replaces it with a stand-in subscription (a push-service address and a valid P-256 key). The permission grant, the inbox prompt, the request to /api/push (whose body must carry the page's language) and the device in notification settings are real. Real devices are checked by hand (launch-checklist.md).
• Installability. Chromium never offers to install from an incognito context, which every new Playwright context is, so that test uses a profile on disk (launchPersistentContext).

CI

The e2e job in .github/workflows/ci.yml installs dependencies, installs Chromium with npx playwright install --with-deps chromium, and runs pnpm e2e. It runs beside the check job, not inside it.

Demo data

Demo data lives in apps/web/demo.

Runbooks, launch checklist and legal drafts

Listed by title and section here; the full text is in the repository.