> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soundchecklive.io/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD Overview

> Continuous integration and deployment for Soundcheck

Soundcheck uses GitHub Actions for CI/CD across mobile, web, api-events (Encore), MCP, and infrastructure.

The **public API / system of record** is **api-events**, deployed via `api-events-deploy.yml`. The Go `api/` tree and `api-deploy.yml` workflow are removed from this repository.

## All workflows

| Workflow | File | Purpose |
| - | - | - |
| **Mobile CI/CD** | `ci-cd.yml` | Build and deploy mobile apps to stores |
| **Web CI/CD** | `web-ci-cd.yml` | Build and deploy Next.js web app |
| **Web E2E Nightly** | `web-e2e-nightly.yml` | Full sharded Playwright workflow coverage + gate |
| **API Events Deploy** | `api-events-deploy.yml` | Checks, schema gates, OpenAPI parity, e2e, and deploy for Encore api-events (GCP) |
| **Components CI** | `components-ci.yml` | PR entry point for checkai, mcp-ts, gcp-platform-signals and GCP Terraform: one paths filter, then each component workflow through `workflow_call`, and the required `components / result` on PRs into qa and main |
| **MCP Deploy** | `mcp-deploy.yml` | `@soundcheck/mcp` library checks + tsc build (`mcp-ts/`). Hosted MCP deploys with api-events. |
| **Code Quality** | `code-quality.yml` | Repo-wide static checks (taxonomy, script tests, SQL scanner CLIs, shell/Dockerfile/PowerShell lint, CI coverage lint) + mobile format / type-check / lint / unit / Knip + advisory `Conventions` semantic lint |
| **Docs CI** | `docs-ci.yml` | One `Check Docs` job on PRs (link check plus the public changelog audience gate); OpenAPI spec + changelog sync on push |
| **Terraform Apply** | `terraform-apply.yml` | Azure infrastructure deployment |
| **GCP Terraform CI** | `gcp-terraform-validate.yml` | Format, validate, and unit-test `infra-gcp`. No GCP credentials and no apply. |
| **Branch Enforcement** | `branch-enforcement.yml` | PR naming conventions |
| **Linear Release** | `linear-release.yml` | Records QA/Production releases in Linear after successful Web, api-events, and mobile deploys |

### Consolidation rules

Keep the check list short and the runner bill honest:

* A check that is one script against an already-installed tree belongs as a **step** in the job that owns that install (`checks`, `schema-gates`, `openapi-parity`), not as a new workflow or job.
* Repo-wide one-second linters go in `code-quality.yml` `static-suite`. That job runs beside `gates` instead of after it; the thin `static-checks` job holds the required `Static Checks` name and fails if either one fails.
* New jobs are for genuinely different infrastructure (services, Docker builds, browsers) or for wall-time parallelism on the critical path.
* Size the runner by measured duration. Path filters, echo/compat jobs, coverage comments, lint-only, node script gates and the docs link check use `blacksmith-2vcpu-ubuntu-2404`. Typecheck and lint jobs, node gates that need an install, and validation-only builds use `blacksmith-4vcpu-ubuntu-2404`. Keep `blacksmith-8vcpu-ubuntu-2404` for long vitest coverage suites, integration, e2e, `next build`, Docker service stacks, browsers and Rust. GitHub/Blacksmith bill per started job, rounded up to a minute; 8vCPU costs 4× 2vCPU-minutes and 2× 4vCPU-minutes, so a job under two minutes on 8vCPU is usually cheaper one size down. Every 8vCPU job is listed in `.github/runner-allowlist.json`, which Lint Workflows checks with `scripts/check-runner-sizes.mjs`. 2vCPU jobs write `runs-on: ${{ vars.RUNNER_LIGHT || 'blacksmith-2vcpu-ubuntu-2404' }}`. `runner-routing.yml` runs daily at 00:17 UTC (and on demand) and sets `RUNNER_LIGHT` to `ubuntu-24.04` while under 98% of the org's 50,000 included GitHub minutes are used, then back to the Blacksmith label (routes in `.github/runner-routes.json`, script `scripts/runner-routing/route.mjs`). An unset variable means Blacksmith, and a failed usage read sets it back to Blacksmith. A missing or expired `RUNNER_ROUTING_TOKEN` fails the daily run and leaves the last label in place; delete the variable to force Blacksmith. The same checker rejects a plain label that has a route, unless the job is listed under `pinned` in `.github/runner-allowlist.json`. Gates, `static-suite`, `pr-build` and `schema-gates` are pinned to Blacksmith for the whole month: measured on 2026-11-03 they ran 1.2 to 1.7 times as fast there, and the shorter runs at Blacksmith's lower per-minute rate cost about the same as using included GitHub minutes first.
* Conditional workflows keep `paths:` on **push** only. On **pull\_request**, `api-events-deploy` always runs and its `changes` job (`dorny/paths-filter`, API call, no checkout) decides relevance; every other job is `if:`-gated on it. The small components (`checkai-deploy`, `mcp-deploy`, `gcp-platform-signals`, `gcp-terraform-validate`) have no `pull_request` trigger: `components-ci.yml` filters paths for all of them in one `components / changes` job and calls each one (`workflow_call`) only when the PR touches it. A PR that touches none of them starts one light job instead of five. Their PR checks are named `<caller> / <job>` (for example `checkai / checks`), so a job that runs on both events names itself `${{ github.event_name == 'pull_request' && 'checks' || 'checkai / checks' }}`: short when called, prefixed on push, so names stay unique across workflows. Do not put several cheap scans in separate jobs: each job is billed as at least a minute, so `security.yml` runs gitleaks, the npm audit and Semgrep (through `docker run`) in one `security` job. Dorny filters that use `!` exclusions must set `predicate-quantifier: some-with-excludes` — default `some` treats each `!` glob as its own matcher and marks every non-excluded file relevant. Give jobs unique display names (`<component> / <job>`) because check runs are matched by name across all workflows.

### Scheduled failure alerts

GitHub tells nobody when a scheduled run fails, and `Runner Routing` failed daily for 26 days before anyone saw it. `scheduled-failure-alert.yml` keeps one open issue per failing scheduled workflow, with the label `scheduled-failure`. A later failure updates the issue in place, and the first scheduled run to pass comments and closes it. It uses the workflow token, so it needs no secret. To add a scheduled workflow, add its name to the `workflows` list in that file; Lint Workflows fails until you do.

### Promotion PR references

The existing **Enforce Branching Rules** check also rejects GitHub closing-keyword references to promotion PRs. A target is a promotion only when GitHub confirms that its head and base belong to the same repository and the branch pair is `dev → qa` or `qa → main`. Titles and labels do not determine this classification; ordinary issue-closing references remain allowed.

Use neutral wording for promotion review work: `Follow-up for #123` or `Addresses review feedback on #123`. Closing keywords (`close/closes/closed`, `fix/fixes/fixed`, and `resolve/resolves/resolved`, case-insensitive and with an optional colon) belong with issues that should actually close.

The check inspects the current PR title, description, and full messages of commits newly introduced by the PR. It reads complete Git history, avoiding the pull-request commits API's 250-commit limit. Commits already reachable from the repository's default branch are excluded: replaying historical messages during `dev → qa → main` must not permanently block promotions. New feature-branch commit messages are still checked. The range starts at the original event base SHA, while the default-branch exclusion uses the fetched tip, so unrelated base-branch advances do not invalidate reruns. Metadata edits, new commits, reopen, and ready-for-review events rerun the check. A run checks the live title and description rather than the event payload, and checks again when they change during the run (up to three times). When the head commit or base branch moved, the run passes with a "Skipped" line and leaves the check to the run for that change. A source or target repository other than this one still fails.

Supported references are `#123`, `soundchecklive/repository#123`, and `https://github.com/soundchecklive/repository/issues/123` or `/pull/123`, with an optional trailing slash on URL references. Repeat the closing keyword for each reference. Lookups stay within this GitHub organization. References to other organizations are outside this guard's scope and are never fetched. Inaccessible or unverifiable same-organization targets fail closed; the job token may not be able to read another private repository. Check access or use neutral wording when no closure is intended.

The workflow uses only read permissions and does not use `pull_request_target`, repository secrets, or write credentials. Tests run with `node --test scripts/promotion-reference-guard/*.test.mjs` and need no npm dependencies. The rule lives in `scripts/promotion-reference-guard/check.mjs`.

To block merging, **Enforce Branching Rules** must remain a required check in the applicable branch rules. This change does not alter branch protection or bypass permissions. This human-error guard is not a security boundary: a PR can change its `pull_request` workflow or the checked-out guard script and intentionally bypass the check. Use independent review/protection for those files if resistance to deliberate bypass is required. The check also cannot inspect a merge message edited in GitHub's final merge dialog or direct pushes that bypass PRs; keep merge-time titles/messages neutral too. Existing merged commits are not rewritten.

### Promotion PR gate reuse

A dev to qa or qa to main PR re-runs on every push to its head branch, and the same push already ran the full gates on the same tree. In the three days before this change, those re-runs were 17% of modeled CI spend, about \$20 a day. `api-events-deploy.yml`, `web-ci-cd.yml` and three of the components in `components-ci.yml` (mcp-ts, openapi-operation-ids and Terraform) skip the duplicate gates when a push run already passed them. checkai and gcp-platform-signals always run on promotions, because their PRs run gates a push never does: `eve build`, and the Docker build, smoke test and Trivy scan. `code-quality.yml` and `security.yml` only run on PRs, so there is no push run to reuse; they run on every promotion push.

In `api-events-deploy.yml` the `api-events / changes` job decides. In `components-ci.yml` the `components / changes` job runs one decision per relevant reusable component in parallel and skips that component when it reuses. In `web-ci-cd.yml` a `promotion-reuse` job decides, and `checks`, `unit-tests` and `unit-tests-merge` depend on it. Both run `scripts/promotion-reuse/decide.mjs`, which reuses a push run only when all of these hold:

* The PR comes from a branch in this repository and is dev to qa or qa to main. The workflow jobs also check the base branch before they poll, so other PRs from `dev` or `qa` pay nothing.
* The base brought no change to a path the workflow watches since the merge base, so the merge result has the same watched tree as the head.
* A push run on the head branch is at or before the PR head, and nothing the workflow watches changed between that run's commit and the PR head. The watched paths are the `on.push.paths` list of the workflow, plus `EXTRA_WATCHED` for inputs the gates read that the filter omits. Today that is `scripts/api-resources/`, `checkai/` and `mcp-ts/` for web, which its client generators and SDK build consume.
* Every gate listed for the workflow in `WORKFLOWS` finished with `success` in that run. For `api-events-deploy.yml` they are `checks`, `integration-tests (suite)`, `encore-e2e (suite)` and `schema-gates`. For `web-ci-cd.yml` they are `checks`, both `unit-tests` shards and `unit-tests-merge`. For the components they are `mcp-ts / checks` and `Terraform CI`. A deploy or an e2e job that is still running, or that failed, does not matter, because promotion PRs never ran it.

The PR event fires while the push run is still in flight, so the deciding job polls every 30 seconds for up to 20 minutes on a 2 vCPU runner. A promotion PR whose head has no covering push run after four minutes runs every gate as before, as does any PR when a gate failed, was skipped or was cancelled, when `qa` changed watched paths, or when the script errors.

Reused gates are skipped. In `api-events-deploy.yml`, `integration-tests` and `encore-e2e` report through the required `api-events / suites` (`scripts/required-job-result.mjs`), which accepts a skip only when `changes` reports `reuse=true`. `components / result` applies the same rule to each component. `openapi-parity` and `pr-build` still run. `openapi-parity` compares the spec with the PR's base, so on a promotion PR it checks the breaking changes against what qa or main already serves, which the push run never saw. `pr-build` builds with the target environment's config and only runs on PRs. The web jobs are not required checks, so a skip needs no result job there, but `unit-tests-merge` carries the same condition because it would otherwise run without the unit-test artifacts. The script is checked out from the base commit, so a PR cannot edit its own check, and the first promotion after this lands runs every gate because the base does not have the script yet.

Run `node scripts/promotion-reuse/decide.mjs` with the `PR_*` variables from the workflow to see a decision locally. Tests: `node --test scripts/promotion-reuse/decide.test.mjs`.

### Ready PR gate reuse

Drafts skip integration and e2e, so marking a PR ready has to start the workflows again. That run used to repeat jobs that had already passed on the same commit, usually a few minutes earlier, at about \$32 a week. `web-ci-cd.yml` and `api-events-deploy.yml` now skip those repeats when the inputs are identical. `scripts/promotion-reuse/ready.mjs` makes the decision in the `promotion-reuse` job on web and the `api-events / changes` job on api-events, and it reuses an earlier run only when all of these hold:

* The event is the first attempt of a `ready_for_review` run, on a PR from a branch in this repository into `dev`. Promotion PRs keep the rules above. "Re-run all jobs" always runs the gates.
* An earlier run on the same head commit passed every gate in `READY_GATES`. For web they are `checks`, both `unit-tests` shards and `unit-tests-merge`. For api-events they are `checks`, `openapi-parity`, `pr-build` and `schema-gates`.
* That run targeted the same branch, and its merge commit has the same two parents as this run's: the same `dev` commit and the same head commit. Any move of `dev` runs the gates, because gates read files outside each workflow's path filter, such as `integrations/cleo-pay/`. Each `checks` job records its merge and target in a step named `Tested merge <sha> into <branch>`, because the Actions API leaves the PR details empty on most PR runs and never reports the merge.

Otherwise every gate runs, including when a job was cancelled or when the script errors. On api-events the ready steps inside `changes` use `continue-on-error`, so their failure cannot block the gates that depend on that job. A PR moved back to draft and marked ready again can reuse the same earlier run; push a commit or use "Re-run all jobs" to force the gates.

Integration and e2e still run, because a draft never ran them. On api-events they read the `reuse` output, which only the promotion check sets, and the reused gates read `ready_reuse`.

Tests: `node --test scripts/promotion-reuse/ready.test.mjs`.

### Semantic convention lint

`code-quality.yml` `Conventions (advisory)` and the husky pre-commit run `.github/semantic-lint.py`. The script sends the added diff hunks to the third-party classifier at `https://classifier.dev/v1/classify`, one call per convention (logging PII, swallowed errors, money as float, bare `Date`). Findings at confidence 0.9 or higher become one sticky PR issue comment (`<!-- semantic-lint -->`, posted by `conventions-comment`). The cited line is the first added line of the hunk. Lower-confidence findings go to the job summary only. The check is advisory: it is not required, and an unreachable classifier exits 0. Conventions live in the script's `CONVENTIONS` list. Review loops (`/x`, `/release-loop`) must read that comment; it is not a review thread, so a green check and an empty thread list do not mean it is clear. To skip it locally, set `SOUNDCHECK_SKIP_SEMANTIC_LINT=1`.

### Affected checks and Jev change triage

`scripts/affected/` chooses which checks a change needs. `graph.mjs` maps paths to packages and their downstream consumers: the OpenAPI spec feeds the web, mcp-ts, and mobile clients; mcp-ts feeds api-events and CheckAI; the web generated client, `web/src/lib/crew`, and `web/src/lib/api-core` feed mobile. `profiles.mjs` turns that into a plan of `<package>:<check>` ids. The drift test in `scripts/affected/affected.test.mjs` fails when a workflow `changes` filter lists a path the map does not cover.

In CI the planner always runs from a trusted checkout of the base commit (`.trusted-base`) and reads the PR only as git data (`--repo`). The key goes only to jobs that run no PR code: `Affected checks`, `e2e-plan` (web) and `api-events / encore-e2e-select`. Their npm upgrade reads the base commit's `.npmrc`, not the PR's. PR code therefore never sees `AI_GATEWAY_API_KEY` and cannot change its own plan. Changes to the planner, `scripts/lib/`, `.github/actions/`, or `code-quality.yml` always run every check.

When `JEV_CHANGE_TRIAGE` is `shadow` or `on`, the planner sends the PR title and description (or commit messages), the file list, and the redacted diff with three lines of context to Jev (`checkai/scripts/jev/change-triage.ts`). The change is split by package and each package gets its own call, run in parallel: up to six calls (smaller packages are folded into the sixth) with up to 20,000 diff characters each, inside a 24,000-character state. The answers merge strictly: the most cautious change type (at its lowest confidence), the highest risk, and the highest full-suite and e2e probabilities. If any call fails, the whole answer counts as missing. Jev answers a change type (docs, tests-only, refactor, bugfix, feature, schema-or-migration, api-contract, …), a risk level, and whether the full suite or e2e should run. In `on` mode the change type picks the checks from a fixed profile table:

* Jev can **add** checks at any confidence, for example the full unit suite for a refactor or integration and e2e runs for a schema change.
* Jev can **remove** checks only when the change type has probability ≥ 0.8 and the planner's own facts agree (a comment-only diff, test-only files), or when a trivial-risk fix (≥ 0.9) lets it drop downstream packages.
* **Floor** checks always stay: Prisma and migrations, SQL, lockfiles and `package.json`, auth, money, env and config, and deleted files. Any `.github/` change disables trimming.
* A timeout, missing key, or unreadable answer falls back to the map plan.

Escalation is per package: a part that Jev rates high risk (probability above 0.7), or that needs the full suite or e2e (above 0.7), escalates only its own packages and their downstream consumers.

**Product areas and e2e shrinking.** Jev also answers one yes/no question per product area in `scripts/affected/areas.mjs`: sign-in, gigs, staffing, money, clients, messaging, imports, org settings, and catalog. Each area lists its web Playwright specs and its `e2e/tests/` folders, and a test fails when a spec or folder belongs to no area. With `JEV_CHANGE_TRIAGE=on`, the `api-events / encore-e2e (suite)` job behind the required `api-events / suites` shrinks rather than skipping. On PRs it runs the folders for every area at probability 0.3 or higher, plus sign-in and any changed test folders, with route-coverage enforcement off. It runs the full suite when:

* Jev gave no area answers, or its change type is below 0.6 confidence
* the PR touches the schema, SQL, a lockfile, the e2e harness, `tools/`, or a workflow
* the planner step fails

Pushes to `dev` always run the full suite. In `shadow`, the `Affected checks` summary shows the selection without applying it.

Web e2e uses the same areas. With `JEV_CHANGE_TRIAGE=on`, the `e2e-tests` job in `web-ci-cd.yml` passes `--area-specs` to `run-local-e2e.mjs`, which keeps only the critical-tier rows tagged in the selected areas' spec files (sign-in always included). Changed specs still run in full. An empty or failed selection keeps the whole critical tier. `node web/scripts/e2e-critical-tier.mjs --list --area-specs=a.spec.ts,b.spec.ts` shows the rows a selection keeps.

A workflow change forces the full e2e run only when its changed lines fall in an e2e job (`e2e-tests`, `encore-e2e`, `changes`) or outside `jobs:` in `web-ci-cd.yml` or `api-events-deploy.yml`, or when an e2e setup action changes. Deploy-only edits no longer do.

The `e2e-plan` job in `web-ci-cd.yml` lists the PR's changed spec files and sets the e2e shard count. A PR that changes no spec runs one shard, and PRs that change specs and pushes to `dev` run two. Each shard pays about 80 seconds of setup (dependencies, containers, migrations, Encore boot) for a critical tier of about 13 tests, and 76% of web-e2e PRs change no spec, so this removes about 15% of PR web e2e job-minutes. The cost is about a minute of extra wall-clock time on those PRs.

The web e2e job also carries the Turbopack `next dev` compile cache (`web/.next/dev/cache`) between runs. Measured locally, a warm cache makes the dev server ready in 2.7 seconds instead of 9.1, and first page loads take 2.1 seconds instead of 11.8 across ten pages. The cache is large (1 GB cold) and grows with every run, so pushes to `dev` start clean and save a fresh cache per shard, and PRs only restore it. Turbopack writes the cache after a few idle seconds, so dev pushes set `E2E_TURBOPACK_FLUSH_MS` and global teardown pauses before Playwright stops the server. The job summary reports which cache was restored and its size.

Jev also answers nine signal questions, merged across package calls in the strict direction. Each drives one rule (thresholds in `SIGNAL_RULES`, `scripts/affected/profiles.mjs`):

| Signal | Rule |
| - | - |
| Changes an export another package imports | Below 0.15 (with a confident change type), downstream packages' checks are dropped. The OpenAPI spec and the shared web client paths always keep every client's type check. |
| Changes database queries | Above 0.7, adds api-events integration tests |
| UI-only | Above 0.8, no full-suite or e2e escalation |
| Behind a default-off flag | Above 0.8, high risk does not escalate |
| Generated code or rename | Above 0.9 (with a confident change type), trims to type check and lint |
| API shape changed without the spec | Above 0.6, adds OpenAPI parity and warns |
| Changes permissions | Above 0.5, forces the sign-in and staffing e2e areas and warns |
| User-facing | Above 0.6, suggests a changelog entry |
| Blast radius (0–2) | 1.5 or more, runs the full e2e suite |

To measure whether Jev can safely run fewer checks, `node scripts/affected/replay.mjs --prs 200` replays merged PRs. For each PR it fetches the CI jobs that ran (a job counts as a real failure only when every attempt on a commit failed), reads the failed e2e logs for the failing folders and specs, and runs live Jev triage on the PR diff. It reports failures in checks Jev's plan would skip, e2e failures outside the area selection, job-minutes saved by candidate skip policies, and failure rates by Jev's answers. Results are cached per PR in `.git/affected-replay.jsonl`; `--report` rebuilds the report from the cache.

To measure the automated review loop, `node scripts/review-loops/review-loops.mjs` collects every merged PR into `dev`, `qa`, and `main`. For each PR it records Greptile and Codex review rounds, their inline findings (severity, file, round), manual `@greptileai` triggers, and fix commits after the first review. It links promotion PRs to the fix ("sweep") PRs that reference them, and asks Jev to classify each finding: its category, whether a local check could have caught it, whether it depends on other files, and whether it is valid. The report covers rounds per PR over time and by size, package, and author; findings raised only in later rounds; category and package hotspots; and the promotion loop. Caches live in `.git/review-loops.jsonl` and `.git/review-findings.jsonl`; `--report` re-analyses them.

To stress-test the planner with live Jev calls, run `node scripts/affected/stress.mjs`. It runs synthetic scenarios with expected change types, failure injection (timeouts, a bad key, a 40,000-line diff, eight packages), a replay of recent merged PRs on `origin/dev`, and repeated runs of the same change to check stability. It exits non-zero only on a safety violation: a floor check missing, or a map check dropped without a trim decision.

Locally, the planner reads `AI_GATEWAY_API_KEY` and `JEV_CHANGE_TRIAGE` from the shell, or failing that from `web/.env` (the worktree's, then the main checkout's). CI reads only the repository variable and secret. `shadow` records what Jev would have run without applying it. Local decisions append to `.git/jev-triage.log`; CI writes them to the `Affected checks` job summary and the `affected-plan` artifact.

Where it runs:

* **Pre-commit** stays fast: lint and format on staged files, related tests (no coverage), the comment gate, and scanners. No Jev call.
* **Pre-push** (`.husky/pre-push`) plans the pushed range and runs the type checks, lint, and Knip for affected packages. `JEV_CHANGE_TRIAGE` defaults to `shadow` there. When the plan asks for a full unit suite (from the lockfile, auth, or money floors, or from Jev), pre-push leaves that suite to CI, which runs each affected package's full suite on the PR. `SOUNDCHECK_PREPUSH_FULL=1` ignores Jev and runs those suites locally. E2E, integration, and OpenAPI parity are left to CI. See [Git hooks](https://docs.soundchecklive.io/engineering/development/code-quality.md#git-hooks) for timings and workarounds.
* **Coverage is not measured in CI.** Unit suites run without coverage and no hook or check reads a coverage percentage. Run `npm run test:coverage` in a package for a local report.
* **CI** (`code-quality.yml`): the `Affected checks` job plans the PR (repository variable `JEV_CHANGE_TRIAGE`, default `shadow`; secret `AI_GATEWAY_API_KEY`). `Static Checks` skips PRs that touch none of its inputs. `Mobile - Quality` starts when the map says mobile is affected, or when a changed path still needs the comment gate, the test-date gate, or the root format check (`e2e/`, `scripts/`, `tools/`, and the root files that check covers, including `.editorconfig`). It skips its type check, lint, unit tests, and Knip when the map says mobile is unaffected. It follows the map only, never Jev, and runs everything if the plan job fails or the plan value is missing.

### CI rerun triage

`ci-rerun-triage.yml` runs when Web CI/CD, api-events CI/CD, Code Quality, checkai CI, or the MCP workflow fails on its first attempt. `scripts/ci-rerun/triage.mjs` fetches each failed job's log and extracts the failing step: its last lines, plus earlier lines that look like errors. Classification comes from two sources:

* **Known signatures** (`scripts/ci-rerun/signatures.mjs`):
  * Code failures are never rerun: type errors, format, lint, coverage, `KNOWN_GAPS`, OpenAPI parity, taxonomy, comments, schema drift, test assertions.
  * Infrastructure failures are rerun: Encore boot, network, upstream 5xx, registry or rate limits, runner loss.
* **Jev** (`checkai/scripts/jev/ci-failure.ts`) classifies the rest: the cause, whether a rerun would pass, and whether a person should look.

A job is rerun-safe when it matches an infrastructure signature, or when Jev says `infrastructure` or `flaky-test` at 0.75 or higher and "rerun passes" at 0.7 or higher. The run is rerun (failed jobs only, once) only when every failed job is rerun-safe and `vars.CI_RERUN_TRIAGE` is `on`. The default is `shadow`, which only reports.

The verdicts (rerun-safe, needs a fix with a fix hint, or unclear) go to the job summary and a sticky PR comment. Across 80 real failures from the replay cache, signatures classified 56 as code and 4 as infrastructure. Jev labelled most of the remaining 24 code, config, or test problems and approved no reruns on its own. Run `node scripts/ci-rerun/triage.mjs --run <id> --mode shadow` to triage any run locally.

### Review triage

`review-triage.yml` runs when Greptile or Codex submits a review; it can also be run by hand with a PR number. `scripts/review-triage/triage.mjs` reads the PR's review threads through one GraphQL query and triages the unresolved bot findings:

* **Duplicates:** a finding that another open finding already covers, either from a different reviewer within 3 lines of it or with similar wording in the same file, goes in "same as another finding". A finding similar to one already resolved stays in "fix now" and is marked as raised again.
* **Jev** (`checkai/scripts/jev/review-finding.ts --triage`) answers the category, validity, the action (fix now, batch, decline, discuss), and whether to sweep the diff for the same bug.
* **Severity:** P0 and P1 stay in "fix now" unless Jev says decline at 0.9 or higher. Other findings are declined at 0.75.

The result is one sticky PR comment. On #2257 (dev→qa) it grouped 8 open findings into 7 to fix now plus 1 duplicate. The duplicate was a Codex finding restating Greptile's on the same line.

Triage reads review threads only. Greptile also writes a summary into the PR description between `<!-- greptile_comment -->` and `<!-- /greptile_comment -->`. Notes it does not file as separate comments stay in that block, including when the Greptile check says `0 comments added`. Review loops (`/x`, `/release-loop`) read that block on the head SHA. A confidence score of 5 is not a merge requirement.

### Engineer setup for affected checks

1. Pull `dev` and run `npm ci` at the repo root. Husky's `prepare` script installs both hooks (pre-commit and pre-push).
2. Optional, to get Jev locally:

   * `cd checkai && npm ci`
   * add `AI_GATEWAY_API_KEY=` to `web/.env` (gitignored; see `web/.env.example`).

   Without both, the hooks use the dependency map only and say so in the plan summary.
3. Expect fast commits and slower pushes: pre-push runs the type checks and lint for the packages your push affects. Add the full unit suites with `SOUNDCHECK_PREPUSH_FULL=1 git push`. Never use `--no-verify`.
4. The pre-push hook runs Jev in `shadow` unless your shell sets `JEV_CHANGE_TRIAGE`. Decisions are appended to `.git/jev-triage.log` in each checkout.
5. In worktrees, `npm run worktree:setup -- . checkai` installs both packages.

### Required checks and path filters

GitHub required status checks must be reported on every PR. A workflow with `pull_request.paths:` that does not match the PR never runs, so a required check from it sits at "Expected — waiting for status to be reported" and blocks the merge forever. The same thing happened when bot commits carried `[skip ci]` (fixed in `f583f8ea2`). A poller workflow used to sit on a runner until path-filtered jobs finished; that is retired. Do not recreate it.

The supported pattern is the `changes` job above: the workflow always runs, irrelevant jobs are **skipped**, and GitHub treats a skipped required check as passing. Path logic then lives once, in the workflow.

### Required status checks

Ruleset **Protect default,dev,qa,main** requires (GitHub Actions):

* `Enforce Branching Rules`
* `Mobile - Quality`, `Static Checks`
* `api-events / changes`, `api-events / checks`, `api-events / suites`, `api-events / openapi-parity`, `api-events / schema-gates`, `api-events / pr-build`
* `components / result` (checkai, mcp-ts, gcp-platform-signals, GCP Terraform)
* `security`

The ruleset applies to PRs into `qa` and `main` only, so `components / result` runs only there. A component caller job skipped by its `if` reports only the caller name (`checkai`), never `checkai / checks`, so the per-component names cannot be required; the result job applies `scripts/required-job-result.mjs` to each component and fails on any relevant component that failed, was cancelled, or was skipped without a reuse.

**Draft PRs skip integration and e2e.** `api-events / integration-tests`, `api-events / encore-e2e`, `api-events / encore-e2e-select` and web `e2e-plan` / `e2e-tests` skip while a PR is a draft. Everything else still runs on a relevant draft push, including `api-events / checks`, `openapi-parity`, `schema-gates`, `pr-build`, web `checks` and `unit-tests`, `Mobile - Quality`, and the mcp-ts, checkai and gcp-platform-signals jobs. `api-events-deploy.yml` and `web-ci-cd.yml` also trigger on `ready_for_review`, so marking the PR ready runs them on the same head. A skipped check counts as passing, and a job skipped by its own `if` never evaluates a computed `name:`, so the required names come from separate result jobs. The suites report as `api-events / integration-tests (suite)` and `api-events / encore-e2e (suite)`. One `suites-result` job always runs and applies `scripts/required-job-result.mjs` to each suite.

On a draft it is named `api-events / suites (deferred)` and passes with a note to mark the PR ready. The required name `api-events / suites` is then not reported at all, so the PR shows them as expected rather than failed, and merge and auto-merge stay blocked until the `ready_for_review` run reports them. A skipped job would not do this: a skipped required check counts as passing.

Once the PR is ready, it reports the required name and:

* **Pass** when both suites passed, or when the `changes` job found nothing relevant to api-events.
* **Fail** on any other skip, failure or cancel, including a failed `changes` job.

The non-required deferred jobs keep their names. About half of PR pushes happen while a PR is a draft, so this removes roughly $200–250 a month of integration and e2e runner time at September 2026 volume. Code Quality's `Conventions comment` job also posts a sticky `<!-- ci-budget -->` comment once a feature PR passes 15 pushes in a UTC day. It counts the Code Quality runs on the head branch since the PR was opened (a run's `pull_requests` field lists every PR currently open on that branch, including for runs made before the PR existed, so it cannot scope the count), estimates cost at $0.55 per push, and never skips anything. Promotion PRs (`dev` → `qa`, `qa` → `main`) get no budget comment, because their pushes are merges, not iteration.

Why `api-events / changes` is itself required: a job that is skipped because its `needs` failed also reports `skipped`, so only a red `changes` job blocks that case. `components / result` covers the same case for the components, because it fails when `components / changes` did not succeed. Do not require `checkai / evals`; it is toggled by `vars.CHECKAI_EVAL_URL` and skipped-as-pass would hide it. Do not require path-filtered Web CI names or `api-events / set-env`. `Mobile - Quality` is a required check that skips when the plan says it has nothing to scan; a skipped required check still passes. When it starts, it includes Knip only if the map says mobile is affected. `checkai / checks` and `checkai / evals` restore `checkai/node_modules` via `setup-checkai-deps`. `Mobile - Quality` and the mobile jobs in `ci-cd.yml` restore root `node_modules` via `setup-mobile-deps` (same restore/save split as `setup-e2e-deps`).

***

## Wire enum changes

The web and the API deploy separately, in no fixed order, so a web build can read an API that has not changed yet. When a value of a wire enum changes, deploy the API first, or first ship a web build that accepts both values. Renaming `client` to `recipient` in `awaitingParties` (PR 4070) left the portal with no agreements for nine minutes on DEV because the web deployed first. Web services parse lists with `parseWireRows` (`web/src/lib/services/wireDrift.ts`): a row that does not match is dropped, the rest render, and Sentry gets the route, the field path and the received enum value. Search Sentry for the `wire_route` tag.

## Mobile app CI/CD

### Workflow overview

| Trigger | Action |
| - | - |
| Push to `qa` | Build and submit a fresh iOS binary to TestFlight; build Android only when needed |
| Push to `main` | Publish production OTA for both platforms after compatibility checks |
| Push to `dev` | No automatic mobile build or release |
| Manual dispatch | Custom build/submit workflows |

### Smart builds

The CI pipeline uses **fingerprint-based detection**:

* QA pushes always build and submit iOS, including JS-only changes; Android and manual smart builds skip compatible native builds
* Main pushes publish OTA only when both platforms have compatible production binaries; native changes block OTA until new production binaries are built separately.

Android and manual smart builds avoid unnecessary native rebuilds; QA iOS always produces a fresh TestFlight artifact.

### Native label on PRs

`native-fingerprint.yml` labels a PR `native` when it changes the Expo fingerprint, and removes the label when a later push no longer does. It compares the PR merge commit with its base parent (`scripts/native-fingerprint/check.mjs`) and lists the changed sources in the job summary. It is advisory and never fails the PR. A `native` PR on `dev` holds back the OTA for every JS change promoted with it until new production binaries exist, so schedule the build or batch those PRs on purpose.

The workflow runs only when a PR touches one of its `paths`. `scripts/native-fingerprint/paths.integration.test.mjs` (Code Quality, mobile-affected) fails when the fingerprint gains a repo file that the list does not cover. A PR that reverts its only native change stops matching the paths, so remove a stale label by hand.

### Version management

* **Version string**: Managed by EAS remotely (`appVersionSource: "remote"`)
* **Build numbers**: Auto-incremented by EAS (`autoIncrement`)
* **Auto-update**: CI updates patch version if iOS submission would fail

```
# Example: Version already submitted
1.1.0 → 1.1.1 (automatic)
```

***

## Workflow modes

The unified `ci-cd.yml` workflow supports multiple modes via `workflow_dispatch`:

| Mode | Purpose |
| - | - |
| `smart_build` | Fingerprint-based build (default) |
| `build_submit` | Force full build; submit only with `submit=true` |
| `promote` | Submit existing build to stores |
| `ota_update` | Publish OTA update only |

***

## Required secrets

### GitHub secrets

| Secret | Purpose |
| - | - |
| `EXPO_TOKEN` | Expo access token |
| `TESTFLIGHT_GROUPS` | (Optional) TestFlight group names |

### EAS credentials

Managed in EAS Dashboard or via `eas credentials`:

* iOS certificates and provisioning profiles
* Android keystore

***

## iOS submission

### Configuration

```json theme={null}
// eas.json
{
  "submit": {
    "production": {
      "ios": {
        "ascAppId": "REPLACE_WITH_ASC_APP_ID"
      }
    }
  }
}
```

### TestFlight groups

If `TESTFLIGHT_GROUPS` is set, CI auto-assigns builds to those groups:

```env theme={null}
TESTFLIGHT_GROUPS=Internal Testers,QA
```

***

## Android submission

### First-time setup

Before CI can publish to Internal:

**1. Create app in Play Console**

Package: `com.soundchecklive.app`

**2. Complete store listing**

Name, description, category, contact info, privacy policy

**3. Complete app content**

Data Safety, Target audience, Content rating questionnaire

**4. Add graphics**

512x512 icon, 1024x500 feature graphic, screenshots

**5. Configure Play App Signing**

EAS manages keystore, but Play needs first upload

### Submission options

1. **Preferred**: Configure Google Play service account in EAS once (non-interactive CI)
2. **Alternative**: Add `GOOGLE_SERVICE_ACCOUNT_JSON` to GitHub secrets

***

## Typical flows

### Branch and channel routing

| Git source | Build profile | OTA channel | EAS environment |
| - | - | - | - |
| `qa` | `qa` | `qa` | `production` |
| `main` | `production` | `production` | `production` |

QA retains production backend/auth. Its separate OTA channel keeps tester updates away from store binaries. The `qa` profile extends `production` with store distribution, no development client, and no simulator build. Other Git branches and tags are rejected by the release route guard.

### Native builds from qa

Promote code through a PR from `dev` to `qa`. Mobile changes on `qa` always build a fresh iOS binary with the `qa` profile and automatically submit its exact validated ID to TestFlight, including for JS-only changes. Android builds remain fingerprint-based and Android submission remains manual. QA pushes do not publish OTAs. The version-check action may still update the version and push its generated version commit to `qa`.

### Release to testers

QA iOS submissions happen automatically after a successful build. To retry a submission or submit Android, dispatch `promote` from `qa` with the platform and exact build ID. The build's project, platform, status, distribution, profile, and channel are checked before submission. QA Android submissions use only the internal track; iOS uploads go to App Store Connect/TestFlight. Alternatively, dispatch `build_submit` with `submit=true` to build and submit together. Build modes default to `submit=false`.

### Production release from main

After promoting approved code to `main`, manually dispatch `smart_build` or `build_submit` there when a new production native binary is needed. The production profile embeds the production channel. Submit that exact build after approval. Never promote the QA-channel binary to the public store; its embedded QA channel would persist. iOS uploads require a separate App Review step before public release.

### OTA updates

Merging mobile changes to `main` automatically publishes production OTAs for both platforms, provided compatible production binaries exist. For a manual retry or tester OTA, dispatch `ota_update` from `qa` for tester updates or `main` for production updates. Choose the platform and leave `update_branch=auto` to select the source's channel. This historical input selects an EAS channel, not a Git branch. Explicit cross-channel requests fail before publication.

The workflow compares each selected platform against finished builds with the matching profile, channel, and runtime. QA fingerprints and OTAs both use the production EAS environment. Fingerprint/API failures block publication. Only main pushes and manual `ota_update` publish OTAs. QA pushes, other branches, tags, and `smart_build` do not. A missing production binary blocks the entire automatic OTA; create and distribute that binary separately, then rerun the OTA.

Before the first QA build/release, create EAS branch `qa` and channel `qa` linked to it; keep channel `production` linked to branch `production`. Both lanes need a native build containing `expo-updates` before device verification.

***

## Project configuration

### eas.json

Key settings:

```json theme={null}
{
  "build": {
    "production": {
      "autoIncrement": true
    }
  },
  "cli": {
    "appVersionSource": "remote"
  }
}
```

### app.config.js

Runtime version and EAS Update endpoint:

```js theme={null}
{
  expo: {
    updates: {
      url: 'https://u.expo.dev/<projectId>',
    },
    runtimeVersion: {
      policy: 'fingerprint',
    },
  },
}
```

Build profiles in `eas.json` set `channel` (`development` / `preview` / `production`) so binaries pull updates from the matching EAS Update channel.

***

## Troubleshooting

> [accordion] Missing EXPO\_TOKEN
>
> Workflows fail early if `EXPO_TOKEN` is not set. Add it to GitHub Secrets.

> [accordion] iOS submission fails - version exists
>
> CI auto-updates the patch version. If you still see this, the auto-update ran but something else failed.

> [accordion] Android submission fails
>
> Ensure Google Play service account is configured or `GOOGLE_SERVICE_ACCOUNT_JSON` is set.

> [accordion] Build number issues
>
> EAS manages build numbers remotely. Never manually set `buildNumber` or `versionCode`.

***

## Version commands

```bash theme={null}
# Check current remote versions
eas build:version:get --platform ios
eas build:version:get --platform android

# Set version (major/minor releases only)
eas build:version:set --platform all --version 2.0.0
```

***

## Web app CI/CD

**Workflow**: `web-ci-cd.yml`

### Triggers

| Trigger | Action |
| - | - |
| Push to `dev`, `qa`, `main` | Lint, typecheck, build |
| Push to `dev` | + full Playwright suite (deploy gate) |
| PR to `dev` | + critical-tier Playwright tests (coverage-matrix rows marked `tier: critical`) plus any spec files the PR changed |
| Manual dispatch | Full workflow |

### Jobs

1. **Checks**: Lint, typecheck, build Next.js app
2. **E2E Tests**: Playwright with **Encore api-events on :4000** (not the retired Go API on `:8080`)
3. **Deploy** (push to `dev`, `qa`, `main`): one image build, pushed to the Azure registry and to the environment's Artifact Registry `<env>-web` repository. It deploys to Azure App Service, then to Cloud Run `<env>-web` by digest, then invalidates Cloud CDN on `<env>-web-url-map`. The Cloud Run steps need `GCP_WEB_WIF_PROVIDER_<ENV>` and `GCP_WEB_DEPLOYER_SA_<ENV>`; while both are unset the job deploys to Azure only. See `infra-gcp/modules/web/README.md` (CI deploys, Cutover runbook).

### Additive nightly coverage

`web-e2e-nightly.yml` is **additive** — it does not replace `web-ci-cd.yml` e2e.
It runs the complete Chromium suite nightly at 03:00 UTC (and via
`workflow_dispatch`), with isolated ephemeral API stacks per shard, Playwright
artifacts/JSON reports, and `web/scripts/e2e-coverage-gate.mjs`.
`--shard=N/M` is rewritten by `web/scripts/run-local-e2e.mjs` into a
duration-weighted file list (`web/scripts/playwright-duration-shards.mjs`) so
long specs do not pile onto one shard. Use it for
coverage-matrix enforcement and the alternate feature-state job; keep PR smoke
and push-to-`dev` full suite on the existing CI path.

**Alerting MVP:** GitHub Actions marks a failed nightly run red and retains the
Playwright artifacts for investigation. This workflow does not yet have a
dedicated Slack/webhook alert; add one when an owned on-call destination is
available rather than reusing the changelog or docs-only webhooks.

### Required secrets

| Secret | Purpose |
| - | - |
| `NEXT_PUBLIC_API_URL` | API endpoint |
| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | Clerk auth |
| `CLERK_SECRET_KEY` | Clerk server-side |
| `NEXT_PUBLIC_GOOGLE_MAPS_API_KEY` | Maps integration |

***

## API Events CI/CD

**Workflow**: `api-events-deploy.yml`

The Encore **api-events** service is the public API and system of record.

### Triggers

| Trigger | Action |
| - | - |
| Push to `dev`, `qa`, `main` | Checks + deploy when paths match |
| Changes under `api-events/**` | Build and deploy new revision |
| Manual dispatch | Select environment |

### Pipeline (summary)

1. **Set environment**: Resolve dev / qa / prod from branch or dispatch input
2. **Checks**: Lint, format, SQL scanner steps, typecheck, unit tests
3. **Parallel gates**: `schema-gates` (Prisma validate, layout, drift, ephemeral migrate, status, replay), `openapi-parity` (emitted spec, path templates, breaking changes, MCP coverage), `integration-tests`, `encore-e2e`, `pr-build` (Docker validation, PR only)
4. **Deploy**: Roll out new revision (shared env DB migrations apply on boot)

`encore-e2e` runs the suite twice on one runner: first against the provider stubs (Jev-selected folders when `JEV_CHANGE_TRIAGE=on`, with route-coverage enforcement on a full run), then the full suite against the provider emulators in `tools/emulators` after Encore restarts with `dev/e2e-emulators.env`. Every stateful service is reset between the passes: the Twilio and Clerk emulators restart, Azurite is recreated, and the database is reset and re-seeded. The emulator pass used to be a separate `encore-e2e-emulators` job that repeated the same setup on its own 8vCPU runner; folding it in saves about one billed minute per run, and an emulator-mode failure now blocks merge through the required `encore-e2e` check. It loads a Blacksmith-cached `clerk-emulator` image (`setup-docker-builder@v2` with `cache-key: tools/clerk-emulator/Dockerfile`, then `build-push-action`) and restores `e2e/node_modules` via `setup-e2e-deps`, the same node\_modules-cache pattern as `setup-api-events-deps`. Changes under `tools/clerk-emulator/**` (except markdown) make the job relevant.

### Concurrency

api-events deploy shares `backend-deploy-{env}` with `terraform-apply.yml`:

```yaml theme={null}
concurrency:
  group: backend-deploy-${{ env_name }}
  cancel-in-progress: false
```

This serializes infra and api-events deploy workflows per environment.

### Deploy chain

When Terraform changes affect migration-related infrastructure, `terraform-apply.yml` dispatches `api-events-deploy.yml` after apply so new instances boot-migrate pending schema changes.

> [!NOTE]
>
> The retired Go API Linux Web App (`infra/modules/backend`) is still provisioned in production. Dev and QA set `enable_app_service = false`, so Terraform destroys that App Service while leaving Postgres, blob storage, and the rest of the Azure stack in place. This repository no longer builds or deploys Go API images via `api-deploy.yml`.

***

## MCP CI/CD

**Workflow**: `mcp-deploy.yml`

Library CI for `@soundcheck/mcp`: one `checks` job (lint, format, typecheck, tsc build, unit tests). Hosted MCP deploys with api-events. The workflow no longer pushes a `soundcheck-mcp` image or deploys Azure App Service.

***

## Database schema CI

**Job**: `schema-gates` in `api-events-deploy.yml` (no shared env DB credentials)

Runs whenever `api-events/**` changes, in parallel with `checks`. The former standalone `api-migrations.yml` duplicated this job and was retired.

### Steps

1. `prisma validate` and migrations layout check
2. `migrate diff` drift check (schema.prisma vs migrations) against a shadow Postgres
3. `npm run db:migrate` against fresh GHA Postgres on `:5434` (`soundcheck_test`), then `db:migrate:status`
4. `npm run test:migration-idempotence` replays required migrations

There is **no** Go↔TS `schema:parity` gate and **no** `api/db/**` path trigger.

### Shared-env apply (not in this job)

Pending migrations for **dev/qa/prod** apply when **api-events starts**
(`migrateDeployOrThrow` in `encore.service.ts`). Explicit local apply:
`cd api-events && npm run db:migrate`.

> [!WARNING]
>
> Do not run `prisma migrate deploy` against shared dev/qa/prod from GitHub Actions.
> CI may apply migrations only to ephemeral/localhost Postgres; the running
> api-events process applies migrations on boot for real environments.

***

## Linear Releases

**Workflow**: `linear-release.yml`

Linear Releases answer "is ENG-123 live, and where?" by grouping issues into releases per pipeline. The workflow listens with `workflow_run` for successful runs of `Web CI/CD`, `api-events CI/CD`, and `CI / CD` on `qa` and `main`, then runs `linear/linear-release-action` to create one completed release per deploy. `dev` deploys are not tracked; a merge into `dev` leaves the issue in **ENG Review** until the `ticket-states` job below moves it.

### Pipelines

Six continuous pipelines in Linear (Settings → Releases). Names must match the workflow exactly because the secret name is derived from `<component>_<stage>`.

| Pipeline | Source run | Branch | Secret |
| - | - | - | - |
| Web QA | `Web CI/CD` | `qa` | `LINEAR_RELEASE_KEY_WEB_QA` |
| Web Production | `Web CI/CD` | `main` | `LINEAR_RELEASE_KEY_WEB_PRODUCTION` |
| API QA | `api-events CI/CD` | `qa` | `LINEAR_RELEASE_KEY_API_QA` |
| API Production | `api-events CI/CD` | `main` | `LINEAR_RELEASE_KEY_API_PRODUCTION` |
| Mobile QA | `CI / CD` | `qa` (TestFlight) | `LINEAR_RELEASE_KEY_MOBILE_QA` |
| Mobile Production | `CI / CD` | `main` (OTA) | `LINEAR_RELEASE_KEY_MOBILE_PRODUCTION` |

The commit path filter is not configured by hand. `scripts/linear-release-paths.mjs` reads the source workflow's `on.push.paths` list at the deployed revision and passes it as `include_paths`, so the release filter is the deploy trigger by construction. Leave the path filter empty on the Linear pipelines; a narrower filter there would only hide commits the deploy shipped.

Secrets are pipeline access keys generated per pipeline in Linear; personal API keys do not work. They live on the GitHub environments: the `*_QA` keys on `qa`, the `*_PRODUCTION` keys on `prod`. The `sync` job selects the environment from the branch. Add them as environment **secrets**, not variables.

Linear's team workflow settings keep one status automation: PR opened → **ENG Review**. The `ticket-states` job below owns **QA** and **Done**, so "PR merged into `dev` → QA" and "release completed in a Production pipeline → Done" must stay off: the first undoes the ENG Review that follows a merge, and the second would skip the device-label hold. Done therefore means verified and live for customers, not merged.

### Ticket states

The `linear / ticket states` job runs `scripts/linear-ticket-states/run.mjs` after the same `resolve` gate as `sync`, once per real qa or main deploy. It is stateless and idempotent: web, api-events and mobile each trigger it for the same head, and job-level concurrency (`linear-ticket-states-<branch>`) runs one pass per branch at a time.

| Deploy | Candidates (ENG) | Unheld, fully promoted | Held (`needs-revyl`, `needs-real-iphone`) |
| - | - | - | - |
| `qa` | ENG Review | → QA | stays; one comment "Reached qa in `<sha7>`; held until the device check passes." |
| `main` | ENG Review, QA | → Done | stays; one comment "Reached main in `<sha7>`; held until the device check passes." |

* **Fully promoted** means every merged GitHub PR attached to the issue shipped in every component it touches. A PR's files are matched against each deploy workflow's `on.push.paths` at the deployed revision (web, api-events, mobile, the same lists `sync` uses). A renamed file counts for both its old and new path. For each touched component, the newest qualifying run on the branch must contain the PR's `merge_commit_sha` (`git merge-base --is-ancestor`, full-history checkout); a commit missing from the checkout counts as not shipped. A qualifying run follows the same rules as `resolve`: its deploy job succeeded. Web and api-events runs, push or `workflow_dispatch` (such as the Terraform-dispatched API deploy), must also own a successful Deployment to `qa` or `prod`. Mobile counts push runs only, and also needs the publishing step: `Submit to App Store Connect` on `qa`, `Publish OTA update` on `main`. An OTA skipped for a native change therefore does not count, and mobile tickets wait for the next published update. A PR that touches no deploy path only needs to be in the deployed head. Closed, unmerged PRs and linked promotion PRs (`dev`→`qa`, `qa`→`main`) are ignored.
* **Fails closed.** The issue is left alone, so a partial ship never moves, when it has any of these: a linked PR that is still open, a merged PR not yet shipped, a PR from another repository or one that cannot be read, no PR attachment, or more than one page of labels, attachments or hold comments. Only the newest 30 successful runs per component are searched.
* **No stale writes.** Each move or comment re-reads the issue first and is dropped if the issue left the source state or the hold comment already exists. A `qa` pass planned before a `main` pass therefore never moves a Done ticket back to QA.
* **Hold comments** are deduped per branch by their text, so repeated deploys of `qa` comment once and the first `main` deploy comments once more.
* The decision rules live in `scripts/linear-ticket-states/decide.mjs`, covered by `decide.test.mjs` (run in Lint Workflows).

Run modes:

* No `LINEAR_API_KEY` environment secret: the job prints a notice and exits 0. The key is a Linear personal API key with write access, stored on the `qa` and `prod` environments (pipeline access keys cannot change issue state).
* Automatic runs are dry until the repository variable `LINEAR_TICKET_STATES` is `live`. A dry run writes the planned moves to the job summary and changes nothing.
* `workflow_dispatch` uses the same `branch` and `dry_run` inputs. Dispatch also runs `sync` with the same `dry_run`, so keep `dry_run=true` unless both should go live.

```bash theme={null}
# Preview the ticket moves the current qa head would make
gh workflow run linear-release.yml --ref dev -f component=mobile -f branch=qa -f dry_run=true
```

### What counts as a deploy

* The source run must conclude `success`, and the component's deploy job (`deploy`, `api-events / build-and-deploy`, `OTA Update`, or `Smart build - iOS`) must itself have succeeded. Runs where every job was skipped still report `success` and are ignored.
* The mobile version-bump commit (`chore: update version to X.Y.Z [skip mobile build]`) is rejected before a runner starts.
* Web and api-events deploy jobs declare `environment:`, so the workflow also requires a GitHub Deployment for `head_sha` in `qa` or `prod`. An api-events dispatch that targets a different environment than its branch is skipped.
* Web and api-events additionally require that the Deployment's success status was logged by this run, so an older deployment of the same sha does not count.
* Manual `CI / CD` dispatches (`smart_build`, `build_submit`, `promote`) are not recorded automatically; their submit behaviour depends on inputs the run payload does not expose. Record those with a dispatch of this workflow.
* Release version is the short SHA, so re-running a deploy re-syncs the same release instead of creating a second one.
* Any GitHub API failure in the gate fails the run; nothing is synced on uncertainty.
* Every checkout pins `workflow_run.head_sha`. `github.sha` under `workflow_run` is the default-branch tip and must not be used.

Release version is the short commit SHA. Mobile releases are named `v<expo.version> (TestFlight|OTA) · <sha>`, using the version that `check-version` pushed for that run when it bumped one.

### Dry run and backfill

`workflow_dispatch` takes `component`, `branch`, optional `base_ref`, and `dry_run` (default true). Dispatch works from any branch that contains the workflow; `workflow_run` only fires from the default branch's copy.

```bash theme={null}
# Preview what API Production would record for the last week of main
gh workflow run linear-release.yml --ref dev -f component=api -f branch=main -f dry_run=true -f base_ref=<sha>

# One-time backfill so the first real release is not a single commit
gh workflow run linear-release.yml --ref dev -f component=api -f branch=main -f dry_run=false -f base_ref=<sha>
```

Without `base_ref`, the action's first sync for a pipeline scans only `HEAD`. Run the backfill once per pipeline before the first automatic sync.

> [!WARNING]
>
> `scripts/linear-release-paths.mjs` must exist at the deployed revision. Until `dev` has been promoted through `qa` to `main`, syncs for older revisions fail with a clear error instead of guessing a path list.

***

## Infrastructure (Terraform)

### Azure

**Workflow**: `terraform-apply.yml`

Azure apply stays on GitHub-hosted runners until that stack is retired. It is not the GCP apply path.

### Triggers

| Trigger | Action |
| - | - |
| Push to `dev` | Apply to dev |
| Push to `qa` | Apply to QA |
| Push to `main` | Apply to prod (with approval) |
| PR | Plan only (no apply) |

### Pipeline steps

1. **Init**: Initialize Terraform with backend
2. **Security Scan**: Run tfsec
3. **Plan**: Generate execution plan
4. **Apply**: Apply changes (merge only)

### Required approvals

* **Dev/QA**: Automatic on merge
* **Prod**: Requires manual approval from infrastructure leads

### GCP

`gcp-terraform-validate.yml` checks formatting and runs `terraform validate` plus unit tests. It has no GCP credentials.

Plan and apply run in three Cloud Build deployments: dev in `dev-soundcheck-live`, QA in `qa-soundcheck-live`, and prod in `soundcheck-live`. Each project owns its connection, image repository, App-key secret, and plan/apply/builder identities. All three use one GitHub App. A pull request plans the target environment and posts one sticky comment per root. Once automatic applies are enabled, a merge to `dev`, `qa`, or `main` applies that environment. The DNS root (`infra-gcp/dns`) applies only after merge to `main`, and that apply also runs the prod environment so the new zone is visible. api-events deploy does not wait.

Operators bootstrap `infra-gcp/terraform-runner/environments/{dev,qa,prod}` independently, using the shared module in `infra-gcp/terraform-runner/terraform`. Each runner state lives at `terraform-runner/default.tfstate` in its project's dedicated `<project>-terraform-runner-tfstate` bucket. Existing environment and DNS state locations do not change. The apply trigger starts disabled; enable it in that environment's root only after the image, App key, and plan path are verified and the runner script has reached the target branch. Bootstrap and the destroy guard are in the GCP infra skill.

***

## Branch strategy

| Branch | Environment | Auto-deploy |
| - | - | - |
| `dev` | Development | ✅ |
| `qa` | QA/Staging | ✅ |
| `main` | Production | ✅ (with gates) |

### PR flow

```
feature → dev → qa → main
```

Each merge triggers deployment to the corresponding environment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.