CI/CD Overview

​

All workflows

​

Consolidation rules

• A check that is one script against an already-installed tree belongs as a step in the job that owns that install (checks, schema-gates, openapi-parity), not as a new workflow or job.
• Repo-wide one-second linters go in code-quality.yml static-suite. That job runs beside gates instead of after it; the thin static-checks job holds the required Static Checks name and fails if either one fails.
• New jobs are for genuinely different infrastructure (services, Docker builds, browsers) or for wall-time parallelism on the critical path.
• Size the runner by measured duration. Path filters, echo/compat jobs, coverage comments, lint-only, node script gates and the docs link check use blacksmith-2vcpu-ubuntu-2404. Typecheck and lint jobs, node gates that need an install, and validation-only builds use blacksmith-4vcpu-ubuntu-2404. Keep blacksmith-8vcpu-ubuntu-2404 for long vitest coverage suites, integration, e2e, next build, Docker service stacks, browsers and Rust. GitHub/Blacksmith bill per started job, rounded up to a minute; 8vCPU costs 4× 2vCPU-minutes and 2× 4vCPU-minutes, so a job under two minutes on 8vCPU is usually cheaper one size down. Every 8vCPU job is listed in .github/runner-allowlist.json, which Lint Workflows checks with scripts/check-runner-sizes.mjs. 2vCPU jobs write runs-on: ${{ vars.RUNNER_LIGHT || 'blacksmith-2vcpu-ubuntu-2404' }}. runner-routing.yml runs daily at 00:17 UTC (and on demand) and sets RUNNER_LIGHT to ubuntu-24.04 while under 98% of the org’s 50,000 included GitHub minutes are used, then back to the Blacksmith label (routes in .github/runner-routes.json, script scripts/runner-routing/route.mjs). An unset variable means Blacksmith, and a failed usage read sets it back to Blacksmith. A missing or expired RUNNER_ROUTING_TOKEN fails the daily run and leaves the last label in place; delete the variable to force Blacksmith. The same checker rejects a plain label that has a route, unless the job is listed under pinned in .github/runner-allowlist.json. Gates, static-suite, pr-build and schema-gates are pinned to Blacksmith for the whole month: measured on 2026-11-03 they ran 1.2 to 1.7 times as fast there, and the shorter runs at Blacksmith’s lower per-minute rate cost about the same as using included GitHub minutes first.
• Conditional workflows keep paths: on push only. On pull_request, api-events-deploy always runs and its changes job (dorny/paths-filter, API call, no checkout) decides relevance; every other job is if:-gated on it. The small components (checkai-deploy, mcp-deploy, gcp-platform-signals, gcp-terraform-validate) have no pull_request trigger: components-ci.yml filters paths for all of them in one components / changes job and calls each one (workflow_call) only when the PR touches it. A PR that touches none of them starts one light job instead of five. Their PR checks are named <caller> / <job> (for example checkai / checks), so a job that runs on both events names itself ${{ github.event_name == 'pull_request' && 'checks' || 'checkai / checks' }}: short when called, prefixed on push, so names stay unique across workflows. Do not put several cheap scans in separate jobs: each job is billed as at least a minute, so security.yml runs gitleaks, the npm audit and Semgrep (through docker run) in one security job. Dorny filters that use ! exclusions must set predicate-quantifier: some-with-excludes — default some treats each ! glob as its own matcher and marks every non-excluded file relevant. Give jobs unique display names (<component> / <job>) because check runs are matched by name across all workflows.

​

Scheduled failure alerts

​

Promotion PR references

​

Promotion PR gate reuse

• The PR comes from a branch in this repository and is dev to qa or qa to main. The workflow jobs also check the base branch before they poll, so other PRs from dev or qa pay nothing.
• The base brought no change to a path the workflow watches since the merge base, so the merge result has the same watched tree as the head.
• A push run on the head branch is at or before the PR head, and nothing the workflow watches changed between that run’s commit and the PR head. The watched paths are the on.push.paths list of the workflow, plus EXTRA_WATCHED for inputs the gates read that the filter omits. Today that is scripts/api-resources/, checkai/ and mcp-ts/ for web, which its client generators and SDK build consume.
• Every gate listed for the workflow in WORKFLOWS finished with success in that run. For api-events-deploy.yml they are checks, integration-tests (suite), encore-e2e (suite) and schema-gates. For web-ci-cd.yml they are checks, both unit-tests shards and unit-tests-merge. For the components they are mcp-ts / checks and Terraform CI. A deploy or an e2e job that is still running, or that failed, does not matter, because promotion PRs never ran it.

​

Ready PR gate reuse

• The event is the first attempt of a ready_for_review run, on a PR from a branch in this repository into dev. Promotion PRs keep the rules above. “Re-run all jobs” always runs the gates.
• An earlier run on the same head commit passed every gate in READY_GATES. For web they are checks, both unit-tests shards and unit-tests-merge. For api-events they are checks, openapi-parity, pr-build and schema-gates.
• That run targeted the same branch, and its merge commit has the same two parents as this run’s: the same dev commit and the same head commit. Any move of dev runs the gates, because gates read files outside each workflow’s path filter, such as integrations/cleo-pay/. Each checks job records its merge and target in a step named Tested merge <sha> into <branch>, because the Actions API leaves the PR details empty on most PR runs and never reports the merge.

​

Semantic convention lint

​

Affected checks and Jev change triage

• Jev can add checks at any confidence, for example the full unit suite for a refactor or integration and e2e runs for a schema change.
• Jev can remove checks only when the change type has probability ≥ 0.8 and the planner’s own facts agree (a comment-only diff, test-only files), or when a trivial-risk fix (≥ 0.9) lets it drop downstream packages.
• Floor checks always stay: Prisma and migrations, SQL, lockfiles and package.json, auth, money, env and config, and deleted files. Any .github/ change disables trimming.
• A timeout, missing key, or unreadable answer falls back to the map plan.

• Jev gave no area answers, or its change type is below 0.6 confidence
• the PR touches the schema, SQL, a lockfile, the e2e harness, tools/, or a workflow
• the planner step fails

• Pre-commit stays fast: lint and format on staged files, related tests (no coverage), the comment gate, and scanners. No Jev call.
• Pre-push (.husky/pre-push) plans the pushed range and runs the type checks, lint, and Knip for affected packages. JEV_CHANGE_TRIAGE defaults to shadow there. When the plan asks for a full unit suite (from the lockfile, auth, or money floors, or from Jev), pre-push leaves that suite to CI, which runs each affected package’s full suite on the PR. SOUNDCHECK_PREPUSH_FULL=1 ignores Jev and runs those suites locally. E2E, integration, and OpenAPI parity are left to CI. See Git hooks for timings and workarounds.
• Coverage is not measured in CI. Unit suites run without coverage and no hook or check reads a coverage percentage. Run npm run test:coverage in a package for a local report.
• CI (code-quality.yml): the Affected checks job plans the PR (repository variable JEV_CHANGE_TRIAGE, default shadow; secret AI_GATEWAY_API_KEY). Static Checks skips PRs that touch none of its inputs. Mobile - Quality starts when the map says mobile is affected, or when a changed path still needs the comment gate, the test-date gate, or the root format check (e2e/, scripts/, tools/, and the root files that check covers, including .editorconfig). It skips its type check, lint, unit tests, and Knip when the map says mobile is unaffected. It follows the map only, never Jev, and runs everything if the plan job fails or the plan value is missing.

​

CI rerun triage

• Known signatures (scripts/ci-rerun/signatures.mjs):
  • Code failures are never rerun: type errors, format, lint, coverage, KNOWN_GAPS, OpenAPI parity, taxonomy, comments, schema drift, test assertions.
  • Infrastructure failures are rerun: Encore boot, network, upstream 5xx, registry or rate limits, runner loss.
• Jev (checkai/scripts/jev/ci-failure.ts) classifies the rest: the cause, whether a rerun would pass, and whether a person should look.

​

Review triage

• Duplicates: a finding that another open finding already covers, either from a different reviewer within 3 lines of it or with similar wording in the same file, goes in “same as another finding”. A finding similar to one already resolved stays in “fix now” and is marked as raised again.
• Jev (checkai/scripts/jev/review-finding.ts --triage) answers the category, validity, the action (fix now, batch, decline, discuss), and whether to sweep the diff for the same bug.
• Severity: P0 and P1 stay in “fix now” unless Jev says decline at 0.9 or higher. Other findings are declined at 0.75.

​

Engineer setup for affected checks

1. Pull dev and run npm ci at the repo root. Husky’s prepare script installs both hooks (pre-commit and pre-push).
2. Optional, to get Jev locally:
   • cd checkai && npm ci
   • add AI_GATEWAY_API_KEY= to web/.env (gitignored; see web/.env.example).
   Without both, the hooks use the dependency map only and say so in the plan summary.
3. Expect fast commits and slower pushes: pre-push runs the type checks and lint for the packages your push affects. Add the full unit suites with SOUNDCHECK_PREPUSH_FULL=1 git push. Never use --no-verify.
4. The pre-push hook runs Jev in shadow unless your shell sets JEV_CHANGE_TRIAGE. Decisions are appended to .git/jev-triage.log in each checkout.
5. In worktrees, npm run worktree:setup -- . checkai installs both packages.

​

Required checks and path filters

​

Required status checks

• Enforce Branching Rules
• Mobile - Quality, Static Checks
• api-events / changes, api-events / checks, api-events / suites, api-events / openapi-parity, api-events / schema-gates, api-events / pr-build
• components / result (checkai, mcp-ts, gcp-platform-signals, GCP Terraform)
• security

• Pass when both suites passed, or when the changes job found nothing relevant to api-events.
• Fail on any other skip, failure or cancel, including a failed changes job.

​

Wire enum changes

​

Mobile app CI/CD

​

Workflow overview

​

Smart builds

• QA pushes always build and submit iOS, including JS-only changes; Android and manual smart builds skip compatible native builds
• Main pushes publish OTA only when both platforms have compatible production binaries; native changes block OTA until new production binaries are built separately.

​

Native label on PRs

​

Version management

• Version string: Managed by EAS remotely (appVersionSource: "remote")
• Build numbers: Auto-incremented by EAS (autoIncrement)
• Auto-update: CI updates patch version if iOS submission would fail

# Example: Version already submitted
1.1.0 → 1.1.1 (automatic)

​

Workflow modes

​

Required secrets

​

GitHub secrets

​

EAS credentials

• iOS certificates and provisioning profiles
• Android keystore

​

iOS submission

​

Configuration

// eas.json
{
  "submit": {
    "production": {
      "ios": {
        "ascAppId": "REPLACE_WITH_ASC_APP_ID"
      }
    }
  }
}

​

TestFlight groups

TESTFLIGHT_GROUPS=Internal Testers,QA

​

Android submission

​

First-time setup

​

Submission options

1. Preferred: Configure Google Play service account in EAS once (non-interactive CI)
2. Alternative: Add GOOGLE_SERVICE_ACCOUNT_JSON to GitHub secrets

​

Typical flows

​

Branch and channel routing

​

Native builds from qa

​

Release to testers

​

Production release from main

​

OTA updates

​

Project configuration

​

eas.json

{
  "build": {
    "production": {
      "autoIncrement": true
    }
  },
  "cli": {
    "appVersionSource": "remote"
  }
}

​

app.config.js

{
  expo: {
    updates: {
      url: 'https://u.expo.dev/<projectId>',
    },
    runtimeVersion: {
      policy: 'fingerprint',
    },
  },
}

​

Troubleshooting

​

Version commands

# Check current remote versions
eas build:version:get --platform ios
eas build:version:get --platform android

# Set version (major/minor releases only)
eas build:version:set --platform all --version 2.0.0

​

Web app CI/CD

​

Triggers

​

Jobs

1. Checks: Lint, typecheck, build Next.js app
2. E2E Tests: Playwright with Encore api-events on :4000 (not the retired Go API on :8080)
3. Deploy (push to dev, qa, main): one image build, pushed to the Azure registry and to the environment’s Artifact Registry <env>-web repository. It deploys to Azure App Service, then to Cloud Run <env>-web by digest, then invalidates Cloud CDN on <env>-web-url-map. The Cloud Run steps need GCP_WEB_WIF_PROVIDER_<ENV> and GCP_WEB_DEPLOYER_SA_<ENV>; while both are unset the job deploys to Azure only. See infra-gcp/modules/web/README.md (CI deploys, Cutover runbook).

​

Additive nightly coverage

​

Required secrets

​

API Events CI/CD

​

Triggers

​

Pipeline (summary)

1. Set environment: Resolve dev / qa / prod from branch or dispatch input
2. Checks: Lint, format, SQL scanner steps, typecheck, unit tests
3. Parallel gates: schema-gates (Prisma validate, layout, drift, ephemeral migrate, status, replay), openapi-parity (emitted spec, path templates, breaking changes, MCP coverage), integration-tests, encore-e2e, pr-build (Docker validation, PR only)
4. Deploy: Roll out new revision (shared env DB migrations apply on boot)

​

Concurrency

concurrency:
  group: backend-deploy-${{ env_name }}
  cancel-in-progress: false

​

Deploy chain

​

MCP CI/CD

​

Database schema CI

​

Steps

1. prisma validate and migrations layout check
2. migrate diff drift check (schema.prisma vs migrations) against a shadow Postgres
3. npm run db:migrate against fresh GHA Postgres on :5434 (soundcheck_test), then db:migrate:status
4. npm run test:migration-idempotence replays required migrations

​

Shared-env apply (not in this job)

​

Linear Releases

​

Pipelines

​

Ticket states

• Fully promoted means every merged GitHub PR attached to the issue shipped in every component it touches. A PR’s files are matched against each deploy workflow’s on.push.paths at the deployed revision (web, api-events, mobile, the same lists sync uses). A renamed file counts for both its old and new path. For each touched component, the newest qualifying run on the branch must contain the PR’s merge_commit_sha (git merge-base --is-ancestor, full-history checkout); a commit missing from the checkout counts as not shipped. A qualifying run follows the same rules as resolve: its deploy job succeeded. Web and api-events runs, push or workflow_dispatch (such as the Terraform-dispatched API deploy), must also own a successful Deployment to qa or prod. Mobile counts push runs only, and also needs the publishing step: Submit to App Store Connect on qa, Publish OTA update on main. An OTA skipped for a native change therefore does not count, and mobile tickets wait for the next published update. A PR that touches no deploy path only needs to be in the deployed head. Closed, unmerged PRs and linked promotion PRs (dev→qa, qa→main) are ignored.
• Fails closed. The issue is left alone, so a partial ship never moves, when it has any of these: a linked PR that is still open, a merged PR not yet shipped, a PR from another repository or one that cannot be read, no PR attachment, or more than one page of labels, attachments or hold comments. Only the newest 30 successful runs per component are searched.
• No stale writes. Each move or comment re-reads the issue first and is dropped if the issue left the source state or the hold comment already exists. A qa pass planned before a main pass therefore never moves a Done ticket back to QA.
• Hold comments are deduped per branch by their text, so repeated deploys of qa comment once and the first main deploy comments once more.
• The decision rules live in scripts/linear-ticket-states/decide.mjs, covered by decide.test.mjs (run in Lint Workflows).

• No LINEAR_API_KEY environment secret: the job prints a notice and exits 0. The key is a Linear personal API key with write access, stored on the qa and prod environments (pipeline access keys cannot change issue state).
• Automatic runs are dry until the repository variable LINEAR_TICKET_STATES is live. A dry run writes the planned moves to the job summary and changes nothing.
• workflow_dispatch uses the same branch and dry_run inputs. Dispatch also runs sync with the same dry_run, so keep dry_run=true unless both should go live.

# Preview the ticket moves the current qa head would make
gh workflow run linear-release.yml --ref dev -f component=mobile -f branch=qa -f dry_run=true

​

What counts as a deploy

• The source run must conclude success, and the component’s deploy job (deploy, api-events / build-and-deploy, OTA Update, or Smart build - iOS) must itself have succeeded. Runs where every job was skipped still report success and are ignored.
• The mobile version-bump commit (chore: update version to X.Y.Z [skip mobile build]) is rejected before a runner starts.
• Web and api-events deploy jobs declare environment:, so the workflow also requires a GitHub Deployment for head_sha in qa or prod. An api-events dispatch that targets a different environment than its branch is skipped.
• Web and api-events additionally require that the Deployment’s success status was logged by this run, so an older deployment of the same sha does not count.
• Manual CI / CD dispatches (smart_build, build_submit, promote) are not recorded automatically; their submit behaviour depends on inputs the run payload does not expose. Record those with a dispatch of this workflow.
• Release version is the short SHA, so re-running a deploy re-syncs the same release instead of creating a second one.
• Any GitHub API failure in the gate fails the run; nothing is synced on uncertainty.
• Every checkout pins workflow_run.head_sha. github.sha under workflow_run is the default-branch tip and must not be used.

​

Dry run and backfill

# Preview what API Production would record for the last week of main
gh workflow run linear-release.yml --ref dev -f component=api -f branch=main -f dry_run=true -f base_ref=<sha>

# One-time backfill so the first real release is not a single commit
gh workflow run linear-release.yml --ref dev -f component=api -f branch=main -f dry_run=false -f base_ref=<sha>

​

Infrastructure (Terraform)

​

Azure

​

Triggers

​

Pipeline steps

1. Init: Initialize Terraform with backend
2. Security Scan: Run tfsec
3. Plan: Generate execution plan
4. Apply: Apply changes (merge only)

​

Required approvals

• Dev/QA: Automatic on merge
• Prod: Requires manual approval from infrastructure leads

​

GCP

​

Branch strategy

​

PR flow

feature → dev → qa → main