Skip to main content
Use this page when production is broken and the fix cannot wait for the normal promotion. It covers how each deployable ships and how to undo it, then the emergency change path and the review that follows it. The pipelines themselves are described in the CI/CD overview.

​
Restore service first

Undo what broke before you ship a fix. The fastest undo depends on the deployable: A rollback does not reverse database migrations (except the audit cutover rollback), data already written, messages already sent, provider calls already made, or builds already installed on devices. Those need a forward fix. External side effects are never undone by a deploy rollback; a compensating action is a new, deliberate change (async durability patterns). Production can also change without a merge to main. A workflow_dispatch run with environment=prod can deploy api-events or gcp-platform-signals, or apply Terraform, from any branch. The routes below that use it say so, and Known gaps explains why nothing restricts it.

​
Deployables

​
Web

How it deploys. web-ci-cd.yml runs its deploy job on pushes to dev, qa and main that change one of the workflow’s paths (mainly web/**), and on workflow_dispatch (no inputs). The GitHub environment is the branch name, with main mapped to prod. The deploy needs checks and unit-tests-merge to succeed. e2e-tests runs only for PRs into dev and pushes to dev. On pushes to qa and main, and on dispatch, it is skipped and the deploy still runs. The job builds web/docker/Dockerfile, pushes soundcheck-web:<env> to Azure Container Registry, and deploys that image to the App Service for the environment with azure/webapps-deploy. Rollback.
  • Revert the bad commit on a branch cut from dev, open the PR into dev, then promote it to qa and main the same way as any fix.
  • gh workflow run web-ci-cd.yml --ref main redeploys the current head of main. Use it to retry a failed deploy. It cannot return to an older commit: the workflow has no commit input, and each deploy overwrites the :<env> tag.
  • Pointing the App Service back at an earlier image is not in any workflow. See Known gaps.
Dispatch only from dev, qa or main. From any other branch the environment is named after that branch, and the config step falls through to the dev settings.

​
API (api-events) and hosted MCP

How it deploys. api-events-deploy.yml deploys on pushes to dev, qa and main that touch its paths (these include mcp-ts/**), and on workflow_dispatch with an environment input (dev, qa or prod). Pull requests run the gates only. On push, the set-env job maps main to prod, qa to qa, and anything else to dev. On dispatch it uses the environment input as given, with no check against the branch. The build-and-deploy job waits for checks, integration-tests, openapi-parity, encore-e2e and schema-gates to succeed, then:
  1. builds an Encore Docker image tagged <sha12>-<attempt> and pushes it to <region>-docker.pkg.dev/<project>/<env>-api-events;
  2. runs a blocking OpenAPI breaking-change check against the revision Cloud Run serves now;
  3. runs gcloud run deploy <env>-api-events with that image;
  4. verifies MCP OAuth discovery on the public host, then runs npm run smoke:deployed in mcp-ts/.
The projects are dev-soundcheck-live, qa-soundcheck-live and soundcheck-live (prod), all in us-central1. Terraform ignores the image field of the Cloud Run service (infra-gcp/modules/events/main.tf), so a later apply does not roll the CI image back. A push that changes Azure infra/ and api-events migrations together defers this deploy until terraform-apply.yml dispatches it, after the apply or, when the plan has no changes, straight away. A push that changes infra-gcp/ does not defer it: terraform-apply.yml does not watch infra-gcp/, and nothing dispatches this workflow after a GCP apply. If a new api-events revision depends on a GCP apply, an operator sequences the two by hand. Rollback, route 1: move traffic. gcloud run deploy sends traffic to the new revision, and no Terraform traffic block overrides that. Find the previous revision, then send traffic back to it:
Before you move traffic, confirm that the previous revision runs against the current schema. Moving traffic does not undo migrations. Rollback, route 2: redeploy an older commit (prod, emergency only). This route dispatches to prod from a branch with no PR, so it is an emergency change and needs the record described in Emergency change. Use it only when route 1 cannot restore service. Push the older commit to a new branch, then dispatch the workflow from that branch:
The dispatch runs the workflow file, and the gates, from the commit you pushed, so an older commit runs its own older workflow. Only the emergency owners dispatch with environment=prod, because nothing restricts which branch can (see Known gaps). Before you dispatch, confirm that the older revision runs against the current schema. No gate compares an older revision with the live database, and migrations are never reversed (see Database migrations). The breaking-change check compares OpenAPI specs only. The dispatched run repeats checks, openapi-parity, integration-tests, encore-e2e and schema-gates, then build-and-deploy. The required security check (gitleaks, npm audit and Semgrep) does not run, because they run only on pull requests. security.yml also runs weekly and on manual workflow_dispatch, but those runs produce the repository-wide report (full-history verified secrets, full Semgrep SARIF, open advisories), not the PR gates, and nothing starts them when you dispatch a deploy. Run it by hand (gh workflow run security.yml --ref <branch>) and link the run as evidence. Record that gap in the post-deployment review, and land the same commit through a PR afterwards so the checks run on it. The breaking-change step in build-and-deploy compares the older spec with the live revision, so an older API that drops fields the live API added can fail it. An intended break is accepted by adding its line to docs/openapi/breaking-changes-allowed.txt. The deploy waits behind any running deploy for the same environment, because api-events-deploy.yml and terraform-apply.yml share the backend-deploy-<env> concurrency group. MCP library. mcp-deploy.yml is CI only: lint, format, typecheck, tsc build and coverage. It has no deploy job, no publish step, and no dispatch inputs. The hosted MCP endpoint ships with the API, so roll it back with the API routes above. A revert in mcp-ts/ goes through the normal flow and ships with the next API deploy.

​
CheckAI

How it deploys. checkai-deploy.yml deploys only on pushes to dev, and only when the repository variable CHECKAI_VERCEL_DEPLOY is true. On 2026-10-08 that variable was not set (gh variable list), so no workflow deploys CheckAI today and the production alias changes only by hand in Vercel. When the variable is on, the job records the current production deployment, builds, creates a production deployment without moving the alias, promotes it to https://checkai-omega.vercel.app, and polls /eve/v1/health. If the health check fails after promotion and a previous production deployment was recorded, the job rolls back by itself. The workflow has no workflow_dispatch trigger. Once the variable is on, the CheckAI production alias follows dev, not main. Rollback.
  • While the variable is unset: use Vercel’s own rollback for the checkai project, with the command below and the previous deployment id from the Vercel dashboard.
  • Automatic, once the variable is on: the job rolls back to the deployment it recorded if the post-promotion health check fails.
  • Manual: the job logs previous production deployment: <id> in its “Record the current production deployment” step. Use that id:
  • Or revert the commit on dev. The alias moves on the next dev deploy.

​
gcp-platform-signals

How it deploys. gcp-platform-signals-deploy.yml deploys on pushes to dev, qa and main that touch the service or the release-age scripts, and on workflow_dispatch with an environment input (dev, qa or prod). A dispatch uses the input as given, with no check against the branch, the same as the API. The job runs the release-age gate and the Rust gates, builds the image tagged <sha12>-<attempt>, resolves its digest, and deploys that digest to <env>-gcp-platform-signals in us-central1. Terraform ignores the image field (infra-gcp/modules/platform-signals/cloud_run.tf). Rollback. Resolve the digest of the previous image tag, then redeploy it. The earlier run’s “Tag and push image” step shows the tag. Moving traffic to a previous revision works the same way as for the API.
A prod redeploy from a dispatch is an emergency change, as for the API route 2.

​
Mobile app (Expo and EAS)

How it deploys. ci-cd.yml (CI / CD) runs on pushes to qa and main that touch app code, config or shared inputs. Pushes to dev do nothing. The release route step (scripts/mobile-release-policy.mjs route) accepts only qa and main, so a dispatch from any other branch fails at that step.
  • qa: builds a fresh iOS binary with the qa profile and submits it to TestFlight. Android is fingerprint-based, and its store submission is manual.
  • main: publishes production OTA updates for both platforms when a compatible production binary exists. A native change blocks the OTA until a new binary is built and installed.
  • workflow_dispatch modes: smart_build, build_submit, promote (needs build_id) and ota_update. ota_update publishes the head of the branch it is dispatched from. Its update_branch input accepts only auto (the default) or the channel that matches the dispatched branch; any other value fails the route step (scripts/mobile-release-policy.mjs).
Rollback.
  • JavaScript-only regression: revert the commit through dev → qa → main. The push to main publishes a new OTA.
  • Faster: republish an earlier EAS update group with the EAS CLI (eas update:republish --group <group-id>). Check the flags with eas update:republish --help first, because the workflow installs eas-version: latest and the flags are not pinned here.
  • Native regression: a build already installed on a device cannot be pulled back from this repo. Halting a rollout in App Store Connect or Google Play happens outside the repo. See Known gaps. The forward fix:
    1. Land the fix through dev → qa → main.
    2. From main, dispatch ci-cd.yml with mode=smart_build or mode=build_submit to build a new production native binary.
    3. After approval, submit that exact build. mode=promote with platform and build_id submits an existing build, and its route step accepts main. For Android, also pass android_track=production; the input defaults to internal, which reaches testers only. iOS uploads need a separate App Review step before public release.
    4. Never submit the QA-channel binary to the public store, because its embedded QA channel would persist. The steps are in Production release from main.

​
Azure Terraform (infra/)

How it deploys. terraform-apply.yml runs terraform plan on pull requests into dev, qa and main that touch infra/**. A push to one of those branches applies the saved plan for infra/environments/<env>. Dispatch takes environment and run_apply (default true). The apply uses the environment you choose and the Terraform in the branch you dispatch from. Nothing checks that the branch matches the environment, and prod has no protection (see Known gaps). A dispatch with run_apply=false is a plan-only run from the chosen branch:
If the pushed commits change api-events schema or migration files (api-events/src/db/schema.prisma, api-events/src/db/migrations/, api-events/prisma.config.ts), the workflow dispatches api-events-deploy.yml after the apply, or on a push whose plan has no changes. The trigger is the commit diff, not the plan content. The dispatch runs on main for prod, qa for qa, and dev for dev. Rollback. Revert the change and move the revert through the promotion flow. Each merge plans and applies again. Before you merge the revert, read the Plan Summary step of the Terraform Plan job for the add, change and destroy counts. A revert restores configuration. It does not bring back data, secrets or state that a destroy removed. Some Azure changes are one-way. For example, enable_app_service = false in infra/environments/dev/main.tf and infra/environments/qa/main.tf destroys the retired Go App Service on apply, and reverting the setting creates a new App Service from configuration. It does not restore the destroyed one. Treat any destroy in the plan as a forward-fix decision.

​
GCP Terraform (infra-gcp/)

How it deploys. infra-gcp runs in Cloud Build, not GitHub Actions. Each environment project (dev-soundcheck-live, qa-soundcheck-live, soundcheck-live) has a tf-plan-<env> trigger for pull requests, which posts one sticky plan comment per root, and a tf-apply-<env> trigger for pushes to dev, qa and main. The apply trigger runs infra-gcp/terraform-runner/entrypoint.mjs with MODE=apply. The DNS root applies only after a merge to main. The apply trigger is disabled until enable_automatic_applies is true in that environment’s runner root. The runner refuses to apply a plan that deletes any of these: Cloud SQL instances, databases or users; Secret Manager secrets (except the legacy secret); Cloud DNS managed zones; Artifact Registry repositories; projects; networks; subnetworks; or storage buckets. It also refuses to delete a dead-letter topic or subscription unless its address is listed in infra-gcp/dlq-drain-allow.json on the last applied commit. Rollback. Revert through the normal flow. If the runner refuses a plan during an emergency, do not work around the refusal by deleting the protected resource. Keep the resource and fix the configuration forward.

​
Database migrations (api-events)

How they apply. Migrations live in api-events/src/db/migrations/, and the schema is api-events/src/db/schema.prisma. On boot, api-events/src/encore.service.ts calls migrateDeployOrThrow(), which runs prisma migrate deploy against the environment’s database. The first new revision that boots applies whatever is pending. CI does not run migrate deploy against shared environments. Rollback: forward-fix, with one rehearsed exception. Nothing in the repo runs down.sql files as a generic rollback. One legacy file exists under api-events/src/db/migrations/20250515223805_add_subscription_tables/, and no script runs it. For every migration except the audit cutover, fix forward. Exception: the audit cutover. Migration 20260825200000_audit_legacy_views turned seven legacy audit tables into views over audit_log. Its header documents a rollback (drop the views, rename the archives back). Do not run that rollback: it drops every audit row written after the cutover from the legacy readers. The corrected procedure is audit.rollback_cutover(), installed by migration 20260827190000_audit_cutover_rollback. It copies the post-cutover rows back into the archives and restores the seven tables in one transaction. Run it through api-events/scripts/audit/rollback-cutover.mjs:
  • Run --dry-run first. --confirm --source <table> limits the restore to one source.
  • The script refuses to run if audit.rollback_cutover(text) is not installed, so the target database must already have 20260827190000_audit_cutover_rollback.
  • It needs DATABASE_URL and a person. It is not a generic down migration. Rows with no legacy shape stay in audit_log only, and the ON DELETE CASCADE foreign keys dropped by 20260827180000_audit_guard_hardening are not restored.
  • After a rollback, do not run prisma migrate dev, because the diff proposes dropping the seven restored tables. To go forward again, the script header gives the steps: prisma migrate resolve --rolled-back for 20260825200000_audit_legacy_views and 20260825200100_audit_cutover_straggler_sweep, re-run api-events/scripts/audit/backfill.mjs, then redeploy.
For any other migration, fix forward:
  • Add a new migration that corrects the schema or data. Do not edit an applied migration.
  • New migration SQL must replay cleanly. Follow .cursor/rules/sql-migration-idempotence.mdc and the policy in api-events/src/db/migrationIdempotence.policy.json.
  • Lock-taking DDL follows migration lock safety.
If a migration fails, its row stays failed and every later boot fails with P3009 until someone runs prisma migrate resolve by hand. The recovery steps are in migration lock safety. That step needs database access and a person.

​
Emergency change

​
When it applies

Use the emergency path only when one of these is true:
  • Production is down, or a core customer flow is broken (sign-in, gigs, crew messages, payments).
  • Production exposes customer data or a secret, or a production dependency is compromised.
  • The rollback above cannot restore service, and waiting for the normal promotion would extend the outage.
If a rollback restores service, do the rollback and take the fix through the normal flow. Declare the emergency with four things: the condition that was met, the incident link, the rollback that was tried first, and the declarer.
  • With a PR, put them in the PR description.
  • With no PR (route 2 of the API rollback, or any environment=prod dispatch with no PR), write them as a comment on the incident’s Linear issue before you dispatch. Add the dispatch run link to that comment afterwards.
The declarers are the owners listed in .github/CODEOWNERS (today @steventt07 and @wskel). GitHub requires no approval for these changes (see Known gaps), so the declaration is the approval record: it shows that an owner chose the emergency path on purpose. The post-deployment reviewer must be someone other than the author.

​
Branch and PR

  • Name the branch <owner>/<slug>. Create the worktree with npm run task:new -- <slug>, which cuts <owner>/<slug> from origin/dev. The owner is TASK_OWNER or your git email.
  • Do not reuse a branch whose PR has already merged. The Claude Code guard blocks pushes to such a branch.
  • Open the PR into dev. Commits reach dev, qa and main through a PR, with two exceptions. The expo-version-updater app bypasses the ruleset for CI version and docs commits (see Known gaps). A workflow_dispatch can deploy prod from a branch with no PR: route 2 of the API rollback is the emergency exception to “a PR is required”, and its declaration above is the record.
  • Ruleset Protect default,dev,qa,main requires a pull request before merging, blocks force pushes and branch deletion, requires conversations to be resolved, and allows only merge commits.
  • The ruleset requires zero approvals, so the author can merge once the required checks pass. The post-deployment review below is the compensating control.
  • Do not use the expo-version-updater app’s token for an emergency change.

​
Required checks

The ruleset requires the checks below. Each one runs from the PR’s own head, so a PR can change the steps of a required job, and a check is only as strong as the review of its definition. Known gaps covers the .github/ review gap.
  • Git hooks. Pre-commit and pre-push run on your machine, and GitHub does not run them. Do not use --no-verify, core.hooksPath overrides or any other bypass. Fix the failure instead. The Claude Code guard (scripts/agent-hooks/guard-bash.mjs) refuses these bypasses in Claude Code sessions only.
  • Security check. The required security check runs on every PR into dev, qa and main, because security.yml has no path filter on pull requests. One job runs three scans, and each runs even when an earlier one failed:
    • gitleaks scans the PR’s commits for secrets.
    • The npm audit fails on new high or critical advisories in lockfiles the PR changes.
    • Semgrep fails on ERROR findings in the code the PR changes.
  • Other required checks. Enforce Branching Rules, Static Checks, Mobile - Quality, the api-events checks, and components / result, which covers checkai, mcp-ts, gcp-platform-signals and GCP Terraform.
  • Skipped means passing. A required check whose jobs were skipped by a path filter counts as passing. Confirm that the checks for the packages you changed actually ran.
  • Drafts. A draft PR skips api-events integration-tests and encore-e2e, and the required api-events / suites shows as deferred until the PR is marked ready. Mark it ready before you merge.
  • Promotion reuse. A dev → qa or qa → main PR can reuse the checks, unit-tests and api-events gates from a push run on the same tree, when that run passed. The security checks are not in that list, so they run on promotion PRs as well.
  • Terraform plans for main-targeted PRs. The plan job runs under the prod environment, which has no protection rules. See Known gaps.

​
Promotion

  1. Merge the fix PR into dev. The push deploys to dev.
  2. Open a PR from dev into qa. Enforce Branching Rules rejects a qa PR from any other source branch. The push deploys to qa.
  3. Open a PR from qa into main. The push deploys to production. Mobile publishes a production OTA unless the change needs a new native binary.
Keep closing keywords out of promotion PR text. Enforce Branching Rules rejects closing references to promotion PRs, and a commit or PR that says fix #N closes PR #N when it lands on dev. Write Follow-up for #123 or Addresses review feedback on #123 instead. Closing keywords belong on issues that should actually close. The branch rules offer no route that moves one fix from dev to qa, or from qa to main, without the rest of that branch. Before you open the promotion PR, check what else dev holds, because dev → qa promotes all of it.

​
Post-deployment review

Within two business days of the production deploy, someone other than the author reviews the change in production and records the review as evidence. The ruleset requires no approval, and a prod dispatch can deploy without a merge to main, so this review is the compensating control for changes the author merged and deployed. Record:
  • the PR and the merge commits on dev, qa and main, or, for a change with no PR, the rollback branch and the commit it points to;
  • the condition that justified the emergency path, the rollback attempted first, and the declarer;
  • the deploy run links for each environment the change reached (api-events-deploy.yml, web-ci-cd.yml, ci-cd.yml, checkai-deploy.yml, gcp-platform-signals-deploy.yml, terraform-apply.yml). For a route 2 or other prod dispatch, include the dispatch run link and the branch it ran from;
  • for every action taken outside a GitHub workflow (a Cloud Build tf-apply-<env> build, gcloud run services update-traffic, a Vercel rollback, eas update:republish, a store submission or halted rollout): the provider’s build, deployment or update link, the exact revision, deployment, update group or build ID it moved to, who ran it, and the UTC time;
  • which required checks ran, and any skipped check with its reason (draft, path filter or reuse);
  • whether migrations or Terraform applied, and whether a plan was refused;
  • what was checked in production after the change, and by whom;
  • follow-up issues.
Evidence sources are the PR, the deploy run links, and the incident record. The Linear release records from linear-release.yml are not evidence yet, because its automatic runs are dry-run (see Known gaps). Until the owners choose a system of record, record the review as a comment on the PR and link it from the incident’s Linear issue. For a change with no PR (route 2 or any other prod dispatch), record the review as a comment on the incident’s Linear issue, next to the declaration and the dispatch run link, and link it from the PR that later lands the same commit. When the review needs repository-wide scan evidence, link the latest weekly security.yml run (Mondays at 09:17 UTC). It scans the full history and reports open high and critical advisories and Semgrep findings for the whole repository.

​
Routine changes

The same gap applies outside emergencies: the ruleset requires no approval, so most ordinary PRs are merged by their author. For those, the review happens on each qa → main promotion PR instead of per change:
  • Within two business days of the promotion merge, and at least weekly while promotions happen, someone who authored none of the promoted PRs reviews them.
  • They leave one comment on the promotion PR that lists every promoted PR merged without another person’s approval, states what they checked (the diff, the required checks, the deploy runs), and names anything that needs a follow-up issue.
  • No emergency declaration or incident issue is needed. A change already covered by an emergency post-deployment review can be listed by link.

​
Known gaps

These were checked on 2026-10-08 against the GitHub API and the workflow files. They limit what the controls above guarantee.
  • Production can be deployed from any branch. api-events-deploy.yml and gcp-platform-signals-deploy.yml take environment from the dispatch input and do not check that the branch is main. terraform-apply.yml applies the chosen environment’s Terraform from the dispatched branch, and run_apply defaults to true. The prod GitHub environment has no protection rules and no deployment-branch policy. A prod deploy or a prod Terraform apply therefore does not need a merge to main, and route 2 of the API rollback uses this path. A merge to main does not gate these runs. The compensating control is the post-deployment review. The fix is a ref guard that allows environment=prod only from main, or environment protection on prod with required reviewers and a branch policy. Owners to decide.
  • Terraform plans for main-targeted PRs run under prod. For a pull_request whose base is main, terraform-apply.yml sets the environment of its terraform-plan job to prod. The job then logs in to Azure with OIDC (azure/login@v3, with no pull-request guard), passes the AZURE_* secrets, and has id-token: write. The OIDC token names the prod environment. Because prod has no protection rules and no deployment-branch policy, Terraform code in a branch of this repository, for example an external data source, runs in the prod environment without a merge to main. Whether that reaches production Azure resources depends on the federated credentials, which this repository does not show. Owners to decide: plan PRs under a secret-free environment, add a deployment-branch policy to prod, or run PR plans from the trusted base checkout.
  • Environments have no approval gate. The dev, qa and prod GitHub environments have no protection rules: no required reviewers and no wait timer. The CI/CD overview describes manual approval for prod Azure applies, but the environment does not enforce it.
  • No required approvals. The ruleset requires zero approvals and no code-owner review. The post-deployment review is the only independent check on changes the author merged.
  • .github/ and check definitions are not independently reviewed. A PR can change the workflow files that define its own required checks. The ruleset does not require code-owner review for /.github/, and .github/CODEOWNERS only requests review, so those changes are self-approved. Enabling code-owner review for /.github/ in the ruleset would close this. Until then, the post-deployment review covers it.
  • Local hooks are not enforced by the server. Nothing on GitHub runs the pre-commit or pre-push hooks. The Claude Code guard covers Claude Code sessions only.
  • Web checks are not merge-required. The checks, unit-tests and unit-tests-merge jobs of web-ci-cd.yml are not in the ruleset, so a PR that changes web/ can merge with them red. The web deploy job still needs checks and unit-tests-merge green on its push run, so that change does not deploy.
  • The expo-version-updater app bypasses the ruleset. The ruleset exempts the GitHub App expo-version-updater (bypass actor 2751420, bypass_mode: exempt), so its pushes reach dev, qa and main without a PR. It makes three kinds of commits:
    • the mobile version bump, made by the check-version action in ci-cd.yml (jobs smart-check-fingerprint, on qa pushes and smart_build dispatches, and build-submit-version-check, on build_submit dispatches). It runs only when a version bump is needed, commits with git commit --no-verify, and pushes the branch it checked out;
    • the OpenAPI spec sync (job sync-openapi-spec in docs-ci.yml), on pushes to dev, qa and main that change api-events;
    • the changelog sync (job changelog-sync in docs-ci.yml), on pushes that change the docs.
    It is not an emergency-change path, and its commits skip local hooks. Do not use it to land a fix.
  • CheckAI deploys by hand. CHECKAI_VERCEL_DEPLOY is unset, so no workflow deploys CheckAI and there is no deploy run to cite as evidence. Once it is set, a CheckAI fix reaches the production alias on the next dev push.
  • Linear release records are dry-run. linear-release.yml runs after each successful Web, api-events and mobile deploy, but automatic runs set dry_run to true, so they only log. The workflow says the rollout stays dry until the first qa deploys are verified in the logs. No Linear release is written for normal deploys yet.
  • No documented App Service image rollback. The web deploy overwrites the :<env> tag and records no digest, so the previous Azure image is not identified by any workflow.
  • GCP apply state is not visible in the repo. Each apply trigger stays disabled until its runner root sets enable_automatic_applies and that root is applied. Confirm the live trigger state in Cloud Build before you rely on an automatic apply.
  • Store rollouts are outside the repo. App Store Connect and Google Play can halt a rollout, but no workflow in this repository does.