> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soundchecklive.io/llms.txt
> Use this file to discover all available pages before exploring further.

# api-events operations

> Provider secrets, env inventory, and operational bindings for the api-events Encore.ts service

Provider and authorization configuration for `api-events/`. Values are read in
`api-events/src/config/config.ts` (Encore `secret(...)` with env fallbacks). Use
this page for smoke tests and incident diagnosis.

## Local provider emulators

Every outbound provider has a local emulator built on
[emulate](https://emulate.dev/docs/custom-emulators) (`emulate@0.12.0`, pinned
under the npm release-age policy). Custom providers are `defineEmulator`
definitions with JSON-only state:

* `tools/emulators/` — Stripe, SendGrid, Resend, Brevo, QuickBooks,
  Dropbox, Google Calendar, Expo push, Supermemory, context.dev, n8n, PostHog,
  Luma, Apple Music, the web-only Google Maps and Spotify, plus emulate's built-in
  Slack and Google (Drive). Ports `4101-4125`; the
  Compose service is `emulators`. See `tools/emulators/README.md` for the port
  and env table.
* `tools/twilio-emulator/` (`:4013`) and `tools/cleo-pay-emulator/` (`:4092`) —
  same framework, own packages and existing contracts.
* `tools/clerk-emulator/` (`:4711`) — still the vendored upstream emulator.
  Switching to emulate's Clerk must preserve our persona manifest, persistent
  signing keys, and Svix-signed webhooks.

Overrides added for the emulators (`STRIPE_API_BASE_URL`,
`SENDGRID_API_BASE_URL`, `QUICKBOOKS_*_BASE_URL`, `DROPBOX_API_BASE_URL`,
`GOOGLE_DRIVE_API_BASE_URL`, `GOOGLE_CALENDAR_API_BASE_URL`,
`EXPO_PUSH_API_BASE_URL`, `SLACK_API_BASE_URL`, `HUBSPOT_API_BASE_URL`,
`FLEX_API_BASE_URL`) go through
`localEmulatorBaseUrl()` in `config.ts`: loopback or the `emulators` Compose
host only, ignored in QA and production. The older provider URL settings the
emulators also use (`RESEND_BASE_URL`, `BREVO_BASE_URL`,
`SUPERMEMORY_BASE_URL`, `PRICING_CONTEXTDEV_BASE_URL`, `N8N_*_WEBHOOK_URL`,
`POSTHOG_HOST`, `POSTHOG_QUERY_HOST`) are honored in every environment, so
never set them to an emulator outside local dev and CI. Stub flags still win
when set. The emulators are opt-in for `dev:emulator`; set the env listed in
the README to route a provider to them.

Contract tests (`*.emulator.test.ts` beside each client) run the real client
against its emulator in-process and are part of the normal unit suite. The
`checks` job also type-checks `tools/emulators` and boots every provider
(`npm run smoke`).

Known gaps: Twilio's emulator does not cover Conversations SDK WebSockets;
built-in Slack has no `apps.uninstall`; built-in Google accepts any bearer and
has no Docs export. HubSpot and Flex have no API emulator because api-events
only builds links to them (Flex quotes go through the n8n webhook).

## Organization sending domains (SendGrid)

Pro-and-above orgs authenticate their own domain at SendGrid and send from
`local_part@their-domain` instead of the platform address. The system lives in
`api-events/src/systems/sending-domains/`, provider HTTP in
`api-events/src/clients/sendGridDomainAuthClient.ts`, persistence in
`api-events/src/db/organizationSendingDomains.ts`.

| Endpoint | Purpose |
| - | - |
| `GET /api/v1/organizations/{orgId}/sending-domains` | Live domains plus the platform From address. Excludes `DISCONNECTED`. |
| `POST /api/v1/organizations/{orgId}/sending-domains` | Claims the domain locally, then provisions it at SendGrid and stores the CNAME records. |
| `POST /api/v1/organizations/{orgId}/sending-domains/{domainId}/verify` | Re-runs SendGrid validation and merges the per-record verdicts. |
| `PUT /api/v1/organizations/{orgId}/sending-domains/{domainId}/default-from` | Sets the default From local part and display name. `VERIFIED` only. |
| `POST /api/v1/organizations/{orgId}/sending-domains/{domainId}/test-email` | Sends a test message to the calling admin's own address. |
| `DELETE /api/v1/organizations/{orgId}/sending-domains/{domainId}` | Marks the row `DISCONNECTED` and queues the SendGrid delete, in one transaction. |

Statuses are `PENDING` → `VERIFIED` | `FAILED`, plus `DISCONNECTED` once
removed. Every check stores a human summary of the invalid records in
`last_error` and stamps `last_checked_at`.

A failing check does **not** immediately mean `FAILED`. SendGrid returns the
same `valid: false` for "the records are wrong" and "the records are published
but have not propagated yet", so a domain stays `PENDING` for 48 hours after it
was added and only then starts reporting `FAILED` — an admin who re-checks a
minute after editing their zone file sees "waiting", not "action needed". The
two exceptions are a domain that was already `VERIFIED` (a regression is never
propagation lag, so it fails at once) and a row whose `created_at` cannot be
parsed (fails closed). The audit `outcome` metadata distinguishes all three:
`pending` | `failed` | `verified`.

If SendGrid answers the validate call with a 404 — the authenticated domain was
deleted out from under us — the check does not report a provider outage. It
marks the row `FAILED` with `last_error` "This domain no longer exists at the
email provider. Remove it and add it again." and returns 200, so the admin gets
an instruction instead of a retry loop. There is no scheduled re-check in V1 —
verification is the manual button only.

Every endpoint requires org `ADMIN`/`OWNER`. Add, verify, default-from and
test-email additionally require the `sending-domains` entitlement (Pro and above) and answer
`403 PLAN_UPGRADE_REQUIRED` (not a 404) without it, so the settings page can
show an upgrade CTA. **The list and remove endpoints deliberately do not check
the entitlement**: an org that drops below Pro while holding a domain must still
be able to see it and disconnect it, otherwise the row keeps the global domain
claim and the SendGrid resource forever. The list response carries
`entitled: boolean` so the page can render a downgraded org in remove-only
mode. Error codes: `INVALID_DOMAIN`,
`DOMAIN_ALREADY_ADDED`, `DOMAIN_CLAIMED`, `DOMAIN_NOT_VERIFIED`,
`DEFAULT_FROM_REQUIRED`, `CALLER_EMAIL_REQUIRED`, `PROVIDER_ERROR`,
`NOT_FOUND`.

**The add flow is row-first, and the row is never thrown away.** The row is
inserted `PENDING` with a null `provider_domain_id` *before* SendGrid is called,
so the claim on the domain is taken under the unique index rather than by a
provider round trip, and the `409` conflicts are decided with no provider call
at all. A provider-less row means "needs provisioning", and
`provisionPlaceholder` is the single path that fills it, from both add and
re-check: look the domain up by name at SendGrid
(`GET /v3/whitelabel/domains?domain=`), **adopt** it if it is already there,
create it only if it is genuinely absent, then attach the id and records under a
guarded write (`provider_domain_id IS NULL AND status IN ('PENDING','FAILED')`,
which also resets the row to `PENDING` and clears `last_error`).

The lookup-before-create is what makes the ambiguous case safe: a request that
timed out *after* SendGrid accepted the create leaves a real authenticated
domain behind, and a naive retry would create a second one and orphan the first.

**Adoption requires proof of ownership.** SendGrid authenticated domains live in
one shared account, so an exact name match proves nothing — a Pro org could
otherwise adopt a domain someone else authenticated (manually, or through
another integration) and reach `VERIFIED` without ever publishing DNS. Every
object we create therefore carries a per-row identity derived from the row uuid:
`subdomain` = `sc` + the first 10 hex digits, and `custom_dkim_selector` = 3
alphanumerics from the same digits, so two rows on one root domain never collide
on `s1._domainkey.<domain>`. Nothing is stored for this; it is recomputed from
`row.id`. The lookup returns **every** exact-name match and we adopt only the one
whose `subdomain` equals ours. Other matches are left strictly alone — never
adopted, never deleted.

That identity also removes the deleted-org handover race: an object queued for
release belonged to a different row and therefore carries a different subdomain,
so it can never be adopted by whoever claims the domain next. Before creating we
do eagerly release any queue entry for the same domain, so a rejected create
(SendGrid refusing a duplicate) does not leave the next org waiting on an
unrelated drain.

The drain carries the mirror-image guard: before deleting at SendGrid it
re-reads `organization_sending_domains` for an active row referencing that
provider id, and if one exists it drops the queue entry untouched and counts it
`superseded`. That is defensive — guard one should make it unreachable — but it
also covers handles attached by any other route, including manual repair.

A provider failure during provisioning **never deletes the row**. The row stays
`PENDING` with `last_error` "Provider setup did not complete. Re-check to retry."
and a fresh `last_checked_at`, and the endpoint returns 502. That keeps the
domain claimed by the org that asked for it — releasing the claim on a transient
outage would let another organization take the domain out from under them — and
makes re-check the retry button. Re-adding the same domain in the same org
reuses the stalled row; another org still gets `DOMAIN_CLAIMED`. `sending_domain_added`
is audited only on the first successful attach; a re-provision from verify
audits `sending_domain_checked`.

**Organization deletion.** The `organization_id` cascade would destroy the only
copy of `provider_domain_id`, so deletion enqueues the cleanup instead of
performing it. Inside the delete transaction — in `deleteOrganizationOnClient`,
so both the org-settings endpoint and the Clerk `organization.deleted` webhook
get it — one `sending_domain_release_queue` row is written per live domain that
holds a provider handle, immediately before the cascade. That table carries **no
foreign key to `Organization`** for the same reason
the `twilio_conversation_deletion_v2` effect in `effect_outbox` does not: the
queue has to outlive the organization it describes. A deletion that never commits enqueues nothing.

Removal works the same way round. `DELETE` disconnects the row **and** enqueues
its provider id in one transaction, then returns 200 — it never calls SendGrid
on the request path, so a provider outage can no longer leave an active row
pointing at a deleted resource, and the response does not depend on the provider
being reachable. A single targeted release runs eagerly afterwards as a latency
optimisation only; the queue is what guarantees the delete.

Draining is a bounded pass (`drainSendingDomainReleaseQueue`): a SendGrid delete
(404 counts as success) removes the queue row, an entry whose provider id a live
row has since claimed is dropped without calling SendGrid and counted
`superseded`, and any failure increments `attempts`, records `last_error` and
leaves the row for the next pass. It never throws into its caller.

The scheduled job `sendingDomainReleaseDrain` runs it every 15 minutes (limit
25, 60s time budget) under the shared job lock, with the usual private
`expose: false` manual endpoint for ops. Request paths do **not** await a drain:
organization deletion (both the settings endpoint and the Clerk webhook) and the
add / verify operations publish to the drain topic after commit, the same shape
as `publishTwilioConversationDeletePrompts`. The only eager drains left are the
two single-entry, specifically-targeted ones above (pre-create and post-remove),
which stay best effort.

Watch the depth of `sending_domain_release_queue` and its `attempts` column —
broker backlog is not the same as unfinished work. PRO-449 now covers only the
verification re-check sweep.

**Clerk verified-domain mirror (PRO-450).** When a check takes a domain to
`VERIFIED`, the same request best-effort mirrors it onto the org's Clerk
Organization Domains via `ensureVerifiedOrganizationDomain`: an existing entry
with that name (case-insensitively) is flipped `verified: true` keeping its
enrollment mode, and a missing one is created `automatic_suggestion` +
`verified: true`. Success stores `clerk_domain_id` and `clerk_domain_synced_at`
on the row under a `status = 'VERIFIED'` guard; failure is reported through
`captureServiceError(…, 'clerk', 'EnsureVerifiedOrganizationDomain', …)` and the
verify **still returns 200 VERIFIED** — DNS proof is the product promise, the
Clerk mirror is a convenience. The `sending_domain_verified` audit metadata
carries `clerk_domain: created | verified | already_verified | failed`.

A re-check also runs the mirror when a row is already `VERIFIED` but
`clerk_domain_synced_at` is null, so domains verified before this shipped — or
during a Clerk outage — catch up on the next re-check rather than needing a
backfill. `clerk_domain_synced_at` therefore has **no column default**: stamping
every existing row would claim a sync that never happened and permanently
disable that catch-up.

`SENDING_DOMAIN_CLERK_SYNC=false` switches the mirror off for a Clerk instance
that has Verified Domains disabled; `SENDING_DOMAIN_CLERK_SYNC_USE_STUB`, and
the notification dry run, stub it.

**Removal deliberately does not touch Clerk.** Disconnecting a sending domain
says nothing about whether people with an address there should still be able to
join the org — that is enrollment, and it belongs to the Staff page's Team
Domain panel. Revoking it here would silently change who can sign up.

**Audit triggers.** `organization_sending_domains` and
`sending_domain_release_queue` are classified in `audit.table_config` with
`enabled = true` and carry **no capture trigger**, which is the intended state
rather than an oversight: per `20260825210000_audit_trigger_floor`, `enabled`
means "MAY be audited", attachment is a deliberate named wave, and 102 of the
107 classified tables sit exactly here. Only wave 5a (`bank_accounts`,
`OrgMember`, `Organization`, `Subscription`, `white_label_config`) has ever
attached. Every mutation on a sending domain already writes a semantic
`recordAudit` row, which is the customer-facing surface; adding these tables to
the trigger floor is a separate write-volume decision for whoever runs the next
wave.

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `SENDGRID_API_KEY` / Encore `SendGridAPIKey` | `sendGridConfig()` | Reused as-is — no new secret. **The key needs the "Sender Authentication" (whitelabel) scope with read *and* write**, which the mail-send-only key does not have. Without it every add/verify/remove returns `502 PROVIDER_ERROR` and the SendGrid call logs a 403. |
| `SENDGRID_USE_STUB_EMAIL` | `sendGridConfig()` | Stub mode. The domain-auth client returns synthetic CNAME records, validate reports every record valid, and delete is a no-op, so local dev and tests never call SendGrid. |
| `NOTIFICATIONS_DRY_RUN` | `dryRun()` / `sendGridConfig()` | Also forces the stub unless it is exactly `false`. |

The API key itself is never logged; provider errors carry the domain, the
provider domain id, and the HTTP status only. `provider_domain_id` is not
exposed on the wire and is deny-listed in `auditClassification.ts`.

Org correspondence sends from the org's verified identity. Each org-context
sender wraps its params in `withOrgSender(params, orgId)`
(`systems/sending-domains/orgEmail.ts`). It sets `from` and `replyTo` to
`resolveOrgSendingIdentity(orgId)` and leaves the platform sender in place when
the resolver returns `null`. The resolver returns `null` when any of these
fails: the domain is `VERIFIED`, it has a From address, the org has a Pro+
entitlement, and the `org-email-domains` PostHog flag is on for the org. It
caches per org for 60 s (per instance), and the verify / From / remove
operations invalidate that entry.

Wired senders:

* availability requests
* crew offers (including reschedule `.ics`)
* gig invitations (including `.ics`)
* event cancellation
* direct notifications (bulk, broadcast and template)
* the lead-request confirmation (reply-to prefers the org contact email)
* Clerk org team invites

These stay on the platform address:

* emails to the org's own staff (crew responses, new-lead alerts)
* Clerk-sent auth mail
* CheckAI

The follow-up work is a scheduled re-check sweep and inbound parse (MAR-39).

## Cleo Pay

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `CLEO_ENABLED` | `cleoPayConfig()` | Literal `false` disables Cleo. Unset means on |
| `CLEO_WEBHOOK_SECRET` | `cleoPayConfig()` | Verifies `Cleo-Signature` on `POST /webhooks/cleo`. Empty secret → 500 |
| Encore `CleoWebhookSecret` | `cleoPayConfig()` | Same value when the env var is unset |

`CLEO_ENABLED=false` blanks the API key and webhook secret. Register and payment sweeps then return without listing the queue or capturing a missing-credentials error, and the publish paths stay quiet. Messages already on the register and payment subscriptions ack without calling Cleo or marking the row failed, so a rollout does not retry them into the DLQ. Those rows stay pending for the sweep once the flag is turned back on. Dev, QA, and production set that value until real credentials are in place. Missing secrets with the flag unset still alarm when queued work exists.

GCP Secret Manager provisions `CleoWebhookSecret` (mounted as `CLEO_WEBHOOK_SECRET`). It is not copied from Azure Key Vault — after Terraform apply, replace the `PLACEHOLDER` version with `gcloud secrets versions add`.

Cleo-lane Pub/Sub DLQs (`cleo_payment`, `cleo_bank_register`, `cleo_contact_register`, `cleo_webhook_received`), Brevo lane DLQs (`brevo_signup_contact`, `brevo_tracked_event`), scheduled-maintenance DLQs, and the crew-notification fan-out DLQs (availability request/SMS/email/push/push-target/request-response, invitation send/response, crew-offer send/response) page Slack via Cloud Monitoring through the same channel list. qa/prod must pass `cleo_dlq_alert_notification_channel_ids` — the same OOB Slack channel resource names operators attach to platform-signals pipeline alerts. Dev may stay empty. Do not invent channel IDs; copy them from Cloud Monitoring after the Slack OAuth channel exists. Terraform apply remains Cloud-Agent-only. An empty channel list still creates the policies; they just do not page. Replay is a manual pull from `<env>-<subscription>-dlq` and republish to the source topic — there is no Slack consumer on DLQ topics and no auto-replay.

`POST /webhooks/cleo` verifies `Cleo-Signature`, publishes to `cleo-webhook-received`, then returns 200. Only the api-events service account can publish that topic. Bank-status apply runs eagerly after the ack and again on subscription `apply-cleo-webhook` (apply failures are captured, then Pub/Sub retries). Signed bank events (`bank_account.created` / `verified` / `verification_failed` / `removed`) lock the `bank_accounts` row and update `cleo_bank_account_status` when `cleo_funding_source_id` still matches and the event is newer than the stored Cleo cursor (`cleo_last_event_sequence`, then `cleo_last_event_at`). Replacing a funding-source id clears that cursor. Subscription ordering uses Cleo `createdAt`/`sequence` only, not receipt time. Stale or equal events are acknowledged without rewriting status. HTTP create/verify must not overwrite that webhook cursor or regress a terminal status. Payable events (`payable.paid` / `payable.payment.failed` / `partially_paid` / `reversed`) always `GET /v1/payables/{id}` before a terminal write: fully `settled` succeeds the locked pair, unpaid `failed` fails it, in-flight or split-fail stays `PROCESSING`, and clawback after `PAID` inserts a `REVERSED` ACH row. Member contacts have no Cleo webhook in staging — the contact-register cron GETs registered contacts and remaps `archived` / `revoked` to `REMOVED`. Bind topic `dev|qa|prod.integration.cleo_webhook_received.v1` before relying on the async path in a deployed environment.

A local public probe (Cleo dashboard → laptop) is Encore on `:4000` plus an ngrok tunnel. Use the ngrok **agent authtoken** (`NGROK_AUTHTOKEN`), not the ngrok API key — leftover files named `ngrok.authtoken` are often the API key and fail with `ERR_NGROK_107`. See `.agents/skills/ngrok-tunnels/SKILL.md` and [pitfall 17](https://docs.soundchecklive.io/engineering/reference/pitfalls.md#17-ngrok-agent-authtoken-is-not-the-api-key). Sign a fresh body that includes `id`; Cleo signatures expire after five minutes and cover `{t}.{rawBody}` only.

## Client forms

Client forms are native: questions, rendering and responses live in Soundcheck
(`form_definitions`, `form_revisions`, `form_assignments`, `form_submissions`),
so they need no provider keys, webhooks or secrets. Each assignment's direct
link carries its `form_assignments.share_token` and is answered through
`/api/v1/public/form-links/*`. The token is minted on first use, by the link
email or by one of the two share-link routes (staff and holder); see
[Forms as a channel](https://docs.soundchecklive.io/engineering/backend/forms-channel.md).

## n8n

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `N8N_CRM_SYNC_SECRET` | `n8nConfig()` | CRM sync + flex-quote-sync HMAC (non-prod defaults) |
| `N8N_CRM_SYNC_ORGANIZATION_ID` | `n8nConfig()` | Deployment org lock |
| `N8N_TRIGGER_SECRET` | `n8nConfig()` | Outbound trigger auth |
| `N8N_FLEX_QUOTE_WEBHOOK_URL` | `n8nConfig()` | Outbound flex-quote sync trigger |
| `N8N_SETTLEMENT_WEBHOOK_URL` | `n8nConfig()` | Post-settle flex-settle trigger |
| `N8N_INVENTORY_WEBHOOK_URL` | `n8nConfig()` | Inventory sync trigger |
| `N8N_VENUES_WEBHOOK_URL` | `n8nConfig()` | Venues sync trigger |
| `N8N_HUBSPOT_WEBHOOK_URL` | `n8nConfig()` | HubSpot sync trigger |

Integrations deep links (same as historical Go): `HUBSPOT_APP_BASE_URL`,
`HUBSPOT_PORTAL_ID`, `FLEX_BASE_URL` via `integrationsDeepLinkConfig()`.

## App file storage

api-events uses GCS when `GCS_BUCKET` is set and Azure Blob Storage otherwise ([App file storage](https://docs.soundchecklive.io/engineering/backend/cloud-storage#app-file-storage)).

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `GCS_BUCKET` | `gcsConfig()` | App file bucket (Terraform `app_storage_bucket`) |
| `GCS_EMULATOR_HOST` | `gcsConfig()` | fake-gcs-server base URL for local dev and CI |

### Azure Storage (until decommission)

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `AZURE_STORAGE_CONNECTION_STRING` | `azureConfig()` | Blob connection (or account name + key) |
| `AZURE_STORAGE_ACCOUNT_NAME` / `KEY` | `azureConfig()` | Alternate credential shape |
| `AZURE_STORAGE_CONTAINER` / `_NAME` | `azureConfig()` | Upload container |
| `AZURE_STORAGE_USE_EMULATOR` | `azureConfig()` | Azurite for local / e2e |
| `AZURE_STORAGE_PUBLIC_ENDPOINT` | `azureConfig()` | Optional public endpoint override |

## AI Gateway and event ingestion

Model calls (ingest analysis, multi-file merge, import column mapping) go through
[Vercel AI Gateway](https://vercel.com/docs/ai-gateway) using its Anthropic-compatible
Messages endpoint. Model ids use the gateway namespace (`anthropic/claude-sonnet-4.6`);
legacy direct ids (`claude-sonnet-4-6`) are rewritten on the way out.

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `EVENT_INGESTION_ENABLED` | `eventIngestionConfig()` | Gate (`1` / `true` / `yes` / `on`) |
| `INGEST_MERGE_ENABLED` | `eventIngestionConfig()` | Merge path gate |
| `AI_GATEWAY_API_KEY` | `eventIngestionConfig()` | AI analysis; sets `aiGatewayConfigured` (`PLACEHOLDER` = unset) |
| `AI_GATEWAY_BASE_URL` | `aiGateway.ts` | Optional host override (default `https://ai-gateway.vercel.sh`) |
| `INGEST_MAX_FILES_PER_BATCH` | `eventIngestionConfig()` | Batch size (default 20, max 100) |
| `EVENT_PLANS_ENABLED` | `eventPlansConfig()` | Gate for brief-to-plan (`1` / `true` / `yes` / `on`) |
| `EVENT_PLAN_EXTRACTOR` | `eventPlansConfig()` | `fixture` runs a deterministic offline extractor (tests/local dev, no gateway key needed); anything else calls the model |

AI-merge dependency notes live in
`api-events/src/systems/event-ingestions/AI_MERGE_DEPS.md`.

## Mobile update-policy store verification (ENG-57)

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `GOOGLE_PLAY_PACKAGE_NAME` | `googlePlayStoreConfig()` | Play package name gate for version-availability verification |
| `GOOGLE_PLAY_SERVICE_ACCOUNT_JSON` | `googlePlayStoreConfig()` | Play Developer API service-account JSON |
| `APP_STORE_CONNECT_KEY_ID` | `appStoreConnectConfig()` | App Store Connect API key id |
| `APP_STORE_CONNECT_ISSUER_ID` | `appStoreConnectConfig()` | App Store Connect API issuer id |
| `APP_STORE_CONNECT_PRIVATE_KEY` | `appStoreConnectConfig()` | App Store Connect API `.p8` private key |
| `APP_STORE_CONNECT_APP_ID` | `appStoreConnectConfig()` | App Store Connect app id |

Each env name is also an Encore `secret(...)` (`GooglePlayPackageName`,
`GooglePlayServiceAccountJson`, `AppStoreConnectKeyId`,
`AppStoreConnectIssuerId`, `AppStoreConnectPrivateKey`, `AppStoreConnectAppId`)
used when the env var is unset; `PLACEHOLDER`/empty reads as unconfigured, not
a boot failure. GCP Secret Manager provisions them only when the environment
sets `enable_google_play_store_secrets` / `enable_app_store_connect_secrets`
(`infra-gcp` events module). Neither has an Azure Key Vault source — after
apply, populate with `gcloud secrets versions add` (Play Console service
account + package name; App Store Connect API key, issuer, and app id), then
roll the Cloud Run revision.

### Operator-triggered publication (bootstrap)

Publication is intentionally manual: `POST /internal/mobile-update-policy/publish`
publishes store-verified update rules. Scoped disable and the global service-console
switch can advance pointers to empty snapshots; reading a new scope while the
switch is OFF initializes an empty snapshot at revision 1. Scheduled reconciliation
only refreshes releases already referenced by a published policy. Keep enforcement
OFF until acceptance is complete, then use the authorized service-console workflow
to permit fresh publication.

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `MOBILE_UPDATE_PUBLISH_CALLER_EMAIL` | `mobileUpdatePublishOidcConfig()` | Google OIDC `email` allowed to publish. Terraform sets the scheduler HTTP service account. Empty = endpoint rejects every call. |
| `MOBILE_UPDATE_PUBLISH_OIDC_AUDIENCE` | `mobileUpdatePublishOidcConfig()` | Required token `aud` (`soundcheck.api-events/mobile-update-policy/publish`). Distinct from the scheduled-jobs audience, so a Cloud Scheduler token can never publish. |

Runbook (`scripts/publish-mobile-update-policy.mjs`, dry-run unless `--confirm`):

First read the current scope through the authenticated
`GET /api/v2/organizations/{organizationId}/mobile/update-policy` endpoint using
the same platform/channel and installed version/build parameters. Set
`CURRENT_SNAPSHOT_REVISION` to the returned `data.snapshotRevision`. An empty
policy does not imply revision 0: an OFF read starts at revision 1. Revision 0
is valid only if the scope has never initialized a snapshot. Inspect current
state after a conflict before preparing a new command; do not automatically
replace a stale expected revision.

```bash theme={null}
API_EVENTS_BASE_URL="https://<api host>" \
MOBILE_UPDATE_PUBLISH_CALLER_SA="<scheduler-http SA email>" \
node scripts/publish-mobile-update-policy.mjs \
  --org <orgId> --platform ios --channel production \
  --policy-id ios-min --intent initial \
  --minimum-supported 1.4.0 --version 1.6.0 --build 2031 \
  --territories US,CA --expected-revision "$CURRENT_SNAPSHOT_REVISION" \
  --idempotency-key publish-ios-1.6.0-2031 --confirm
```

The script mints a token with
`gcloud auth print-identity-token --impersonate-service-account=$MOBILE_UPDATE_PUBLISH_CALLER_SA --audiences=<audience> --include-email`
(no secret on argv; a pre-minted token may be supplied via
`MOBILE_UPDATE_PUBLISH_ID_TOKEN`). The operator principal needs
`roles/iam.serviceAccountTokenCreator` on the caller service account. The
trusted actor is the verified caller email — the body cannot assert an actor.

Failure/optimistic-concurrency behavior (exit codes in the script header):
`conflict` (409) means the expected revision is stale or the idempotency key
was reused with a different payload — re-read current scope state and retry
with a fresh key for a changed command, the SAME key to replay. `not-verified`
(422) means no fresh AVAILABLE store observation covers the release/territory
audience — run `evidenceMode=verify-fresh` (the default) so the platform store
adapter is checked at publish time, or wait for a reconciliation observation
younger than 60 minutes and use `evidenceMode=persisted`. `unconfigured` (503)
means the store credentials above are not deployed for that platform.

## Resend contact sync

Hourly full recompute of global Resend Contacts (`0 * * * *` in
`api-events/src/systems/resend-contacts/`). The job is read-only against
Postgres and writes outbound HTTP only. Merge does not turn it on.

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `RESEND_CONTACT_SYNC_ENABLED` | `resendConfig()` | Job gate. Default off. |
| `RESEND_API_KEY` / Encore `ResendAPIKey` | `resendConfig()` | Global Contacts API. Required when the gate is on and not stubbing. |
| `RESEND_BASE_URL` | `resendConfig()` | Optional; defaults to `https://api.resend.com` |
| `RESEND_TIMEOUT_MS` | `resendConfig()` | Per-request timeout. Must be finite and `> 0`; default 15000. |
| `RESEND_MAX_REQUESTS_PER_SECOND` | `resendConfig()` | Client throttle. Default 2 (Resend's current allowance). |
| `RESEND_CONTACT_SYNC_TIME_BUDGET_MS` | `resendConfig()` | Wall-clock budget for one run. Default 240000. Must stay under the Cloud Run request timeout. |
| `RESEND_USE_STUB` | `resendConfig()` | Force stub writes even when dry-run is off. |
| `NOTIFICATIONS_DRY_RUN` | `dryRun()` / `resendConfig()` | Writes stub unless this is exactly `false`. |
| `POSTHOG_QUERY_API_KEY` / Encore `PostHogQueryAPIKey` | `postHogQueryConfig()` | HogQL read key: dedicated PostHog user `phx_` (or later `phs_` if query scopes ship). Not the ingest token (`phc_` / `POSTHOG_API_KEY`). Not a human’s MCP key. |
| `POSTHOG_PROJECT_ID` | `postHogQueryConfig()` | Numeric project id. Both this and the query key are required to query. |
| `POSTHOG_QUERY_TIMEOUT_MS` | `postHogQueryConfig()` | HogQL timeout. Must be finite and `> 0`; default 30000. |
| `POSTHOG_HOST` | `analyticsConfig()` / `postHogQueryConfig()` | Ingest host. Query host is this with `.i.posthog.com` mapped to `.posthog.com` unless `POSTHOG_QUERY_HOST` is set. |
| `POSTHOG_QUERY_HOST` | `postHogQueryConfig()` | Optional HogQL API host. Default derived from `POSTHOG_HOST` (`https://us.posthog.com`). |

Enablement after merge:

1. Apply `infra-gcp` so Cloud Run mounts `ResendAPIKey` and `PostHogQueryAPIKey` (PLACEHOLDER until populated) and so `POSTHOG_PROJECT_ID=276289` is set. This apply creates `{env}-PostHogQueryAPIKey` and removes `{env}-PostHogPersonalAPIKey`.
2. Create matching Contact Property keys in the Resend dashboard. A rename here without a dashboard key is a silent no-op.
3. Populate `ResendAPIKey` and `{env}-PostHogQueryAPIKey` with `gcloud secrets versions add` (not Azure KV, not `POSTHOG_API_KEY`). Use a dedicated PostHog user scoped to query/read on project `276289`.
4. Stub run: `RESEND_CONTACT_SYNC_ENABLED=true` with dry-run still on. Confirm phase timings and the 28 property keys.
5. Small live slice, then prod.

`is_internal` prefers PostHog cohort `410365`. If that HogQL cohort query fails, the job logs a warning and falls back to `@soundchecklive.io` plus `INTERNAL_EMAIL_ALLOWLIST` in `posthogQueryClient.ts`. PostHog query timeout, a missing `results` array, or an empty person-property result set omits PostHog-derived keys for that run so a transient or empty HogQL response cannot overwrite existing Resend values; Clerk and Postgres writes still proceed. Lifecycle does not mark `dormant` from Clerk sign-in alone when PostHog is unavailable, so a transient query failure cannot overwrite an active contact as dormant. A failed Postgres fact query aborts the run before any Resend writes so a transient DB error cannot empty org, plan, or lifecycle properties.

`INCOMPLETE` subscriptions are excluded from the live-subscription join, so those orgs get empty plan, status, and seats. Capacity assumes a worst-case two Resend requests per contact (`PATCH` then `POST`). Consecutive hourly runs advance the roster by that capacity so slices do not overlap. Raising `RESEND_MAX_REQUESTS_PER_SECOND` covers the full roster in one run without a code change.

## Brevo signals

Marketing lifecycle email runs out of Brevo. api-events owns two write paths
into it: a contact upsert at signup and nine tracked events that enter and exit
Brevo automations (`api-events/src/systems/brevo-signals/`). Both go through
`api-events/src/clients/brevoClient.ts`. Tracked events have their own switch
(`BREVO_TRACKED_EVENTS_ENABLED`), so they can be turned off without stopping the
signup sync. Prod sets both switches `true` and sends tracked events to the live
Brevo account. Dev and qa set both `true` with `BREVO_USE_STUB`, so stub writes
are logged and nothing reaches Brevo.

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `BREVO_SIGNALS_ENABLED` | `brevoConfig()` | Kill switch for every Brevo write. Default off. |
| `BREVO_TRACKED_EVENTS_ENABLED` | `brevoConfig().trackedEventsEnabled` | Separate switch for the nine tracked events; effective only when `BREVO_SIGNALS_ENABLED` is also on. Default off; set `true` in dev, qa and prod Terraform. While off, nothing is recorded (request paths, Stripe webhook, reconcile, lifecycle job), the relay publishes nothing, and a message already in flight is handed back to the relay. |
| `ORG_KNOWLEDGE_ENABLED` | `orgKnowledgeConfig()` | Server switch for the org knowledge base. Unset defaults on in local `development` and off in deployed environments; set `true` per environment to allow opt-ins and syncing. While off, opt-in returns 503, reads report `not_configured`, uploads stop, and opt-out purges still run. `ORG_KNOWLEDGE_WRITER=fixture` swaps the page writer for a deterministic offline one. |
| `SUPERMEMORY_API_KEY` / Encore `SupermemoryAPIKey` | `supermemoryConfig()` | Org knowledge base (PRO-416). Optional: unset or `PLACEHOLDER` makes the knowledge tools report `not_configured` and the `org-knowledge-sync` job a no-op. Use a key scoped to `soundcheck-org-{env}-*` containers. `SUPERMEMORY_CONTAINER_ENV` overrides the `dev`/`qa`/`prod` container prefix. |
| `BREVO_API_KEY` / Encore `BrevoAPIKey` | `brevoConfig()` | Contacts + Events API. Required when the gate is on and not stubbing. Use the non-expiring `soundcheck-app-prod` key, never the PostHog warehouse key (that one expires and returns a 401 that looks like an IP block). |
| `BREVO_BASE_URL` | `brevoConfig()` | Optional; defaults to `https://api.brevo.com/v3` |
| `BREVO_TIMEOUT_MS` | `brevoConfig()` | Per-request timeout. Must be finite and `> 0`; default 15000. |
| `BREVO_MAX_REQUESTS_PER_SECOND` | `brevoConfig()` | Client throttle. Default 5 (Brevo allows 10/s on `/v3/events`). |
| `BREVO_MAX_RETRIES` | `brevoConfig()` | Retries with `Retry-After` or exponential backoff: 429 on every call, 5xx only for contact upserts. Default 3. |
| `BREVO_USE_STUB` | `brevoConfig()` | Force stub writes even when dry-run is off. |
| `NOTIFICATIONS_DRY_RUN` | `dryRun()` / `brevoConfig()` | Writes stub unless this is exactly `false`. |

### Signup contact

The Clerk `user.created` webhook publishes a `BrevoSignupContactRequested`
message to Pub/Sub (`brevo-signup-contact`) after the durable user steps
succeed; a publish failure fails the webhook so Clerk redelivers. The
`upsert-brevo-signup-contact` subscription owns the Brevo call: it upserts
keyed on `EXT_ID` (Clerk user id) with `updateEnabled`, so a redelivery updates
rather than duplicates, and it throws on a Brevo failure so Pub/Sub retries
(5 attempts, then the DLQ). Attribute names are the uppercase Brevo contract
in `brevoAttributes.ts`; Brevo silently drops unknown keys, so a typo produces
an empty field rather than an error. Booleans are JSON `true` / `false`: the
Brevo UI shows them as a Yes/No dropdown, but the API silently discards the
strings `"Yes"` / `"No"` (every Sep 23 sign-up landed without `IS_PAYING` until
this was fixed). Dates are `YYYY-MM-DD` in UTC.

The webhook reads the web sign-up wizard's answers from Clerk
`unsafe_metadata.onboarding`. When that object exists (the buyer path), the
upsert adds `USER_TYPE=buyer`, plus `TEAM_SIZE` and `GIG_FREQUENCY` when their
values are in the Brevo category enums (any other value is dropped, because
Brevo rejects it silently). A sign-up without the wizard (mobile crew, invite
acceptance) sends no `USER_TYPE`. The crew events set it later.

Consent lists 7, 8 and 9 (`BREVO_SIGNUP_CONSENT_LIST_IDS`) and `IS_PAYING=false`
go only on a **create**. The client first posts `/contacts` without
`updateEnabled`, with `listIds`. If Brevo answers `400 duplicate_parameter` (the
contact already exists: marketing import, earlier `crew_invited`, or a retry),
it falls back to the plain `updateEnabled` upsert without lists or
`IS_PAYING`. An existing contact is never re-subscribed, and never marked
unpaid by a sign-up. Crew-invite contacts never get lists. A person invited
before they sign up therefore stays off the lists until marketing adds them.

### Tracked events

| Event | Kind | Recipient (`identifiers.email_id`) | Recorded from | Dedupe key (arming period) |
| - | - | - | - | - |
| `org_setup_completed` | entry | org primary admin (OWNER, then ADMIN, earliest joined) | Clerk `organizationMembership.created` for an owner/admin, plus hourly reconcile: org created with an addressable owner/admin | `org_setup_completed:<orgId>` (once per org, ever) |
| `event_created` | exit | org primary admin | `createEventOperation`, plus reconcile for every other creation path; `TEMPLATE` rows (saved gig templates) never count, here or in the gig facts | `event_created:<eventId>` (every gig) |
| `crew_invited` | entry | invitee (user email, else invitation email) | gig invitation: bulk send after dispatch marks it `SENT`, plus reconcile (`invite_source=gig_invite`). Org invitation: Clerk `organizationInvitation.created` for a non-owner/admin role (`invite_source=org_invite`) | `crew_invited:<lowercased email>` (once per person, ever, across both sources) |
| `crew_invite_accepted` | gate | the person who accepted | Clerk `organizationInvitation.accepted` for a non-owner/admin role. Owners creating their own org and memberships added without an invitation emit nothing | `crew_invite_accepted:<Clerk invitation id>` (once per invitation) |
| `crew_profile_completed` | exit | crew member | onboarding runtime flipping `member_onboardings` to `COMPLETED`, plus reconcile (also admin `onboarded_at` overrides) | `crew_profile_completed:<orgId>:<userId>` |
| `subscription_started` | exit | org primary admin | Stripe `customer.subscription.created/updated`, plus reconcile | `subscription_started:<subscriptionId>` |
| `subscription_cancelled` | entry | org primary admin | Stripe `customer.subscription.updated` (cancel at period end) / `.deleted`, plus reconcile | `subscription_cancelled:<subscriptionId>` |
| `buyer_stalled` | entry | org primary admin | daily job: `org_setup_completed` 14+ days ago and zero gigs ever | `buyer_stalled:<orgId>` (once per org, ever) |
| `buyer_upgrade_candidate` | entry | org primary admin | daily job: 1+ gig ever, not paying, not comped | `buyer_upgrade_candidate:<orgId>:<latest subscription_cancelled id or initial>` (re-arms after a subscription starts and later cancels) |
| `buyer_went_quiet` | entry | org primary admin | daily job: 1+ booked gig, `LAST_GIG_DATE` 30+ days ago, not in active win-back | `buyer_went_quiet:<orgId>:<LAST_GIG_DATE>` (re-arms after a new gig goes quiet again) |

Contact properties ride along through `contact_properties`: `EXT_ID`,
`ORG_ID`, `ORG_NAME`, `USER_TYPE=buyer` on every event addressed to the org
primary admin; `EVENTS_BOOKED_COUNT`, `FIRST_GIG_DATE`, `LAST_GIG_DATE` on
`event_created` and the computed events; `IS_PAYING` (active or past-due
subscription not cancelling at period end) on every org-addressed event,
computed from the live subscription state when the event is sent (not when it
is recorded), so a late delivery never writes a stale value; `IS_PAYING=true`, `PLAN_TIER`, `MRR` on
`subscription_started`; `IS_PAYING` on `subscription_cancelled` (`false` unless
another live paid plan remains). The crew events carry `USER_TYPE=crew`,
`FIRSTNAME` / `LASTNAME` (the roster name in that org, then the name on the
invitation) and `EXT_ID` when known. `USER_TYPE=crew` is skipped for anyone
who is an active OWNER or ADMIN of any org (matched by user id or email), so a
buyer is never downgraded. `crew_invited` and `crew_profile_completed` never
carry `SIGNUP_DATE`: an invite is not a sign-up. `crew_invited` also carries
`INVITER_NAME` (the gig creator's or Clerk inviter's roster name, then
username, then org name). `crew_invite_accepted` fills in the contact the
invite created: `EXT_ID`, `FIRSTNAME` / `LASTNAME` and `SIGNUP_DATE` from the
Clerk account (names fall back to the roster), plus `ORG_ID`, `ORG_NAME` and
`LIFECYCLE_STAGE=crew_active` unless the person is already a buyer, whose own
org and lifecycle stay untouched. Its event properties carry `invitation_id`,
`invite_source` and `inviter_user_id` when Clerk supplies one.

The two org-invitation signals have no database fact to reconcile from, so the
Clerk webhook is their durable trigger: a ledger write failure returns 500 and
Svix redelivers. `organizationInvitation.accepted` looks the person up in Clerk
by email (it can arrive before `user.created`) and answers 503 until Clerk
returns them. Brevo's "wait until event" step misses an event that fires before
the contact reaches the wait, which is why acceptance also sets
`LIFECYCLE_STAGE=crew_active`: an attribute the automation can check at any
time. `MRR` is the plan price × seats per month
(annual ÷ 12), computed with `decimal.js`. `LAST_GIG_DATE` is the latest
`eventDate` of a non-cancelled gig. Active win-back means the org's latest
`subscription_cancelled` is newer than its latest `subscription_started` and
it is not paying now. The computed events only consider orgs that already
have a ledger row, so pre-launch orgs join the lifecycle when they next create
a gig rather than all at once.

**Durable owner.** Every signal is a row in `brevo_signal_emissions` (the
ledger and the outbox). `dedupe_key` is unique and encodes the arming period,
so an entry event is recorded at most once per period and an exit event once
per occurrence; `id` is the UUID v5 of the key and is the Pub/Sub event id and
inbox key. Producers insert with `ON CONFLICT DO NOTHING`.

* Request paths (gig create, invitation send, onboarding completion) record
  best-effort after their own write commits: a ledger failure is captured to
  Sentry and never fails the user's request. The hourly `brevo-signal-reconcile`
  job is the recovery owner: it rescans the committed domain rows from the last
  three days (never before `BREVO_SIGNALS_EPOCH_ISO`, 2026-09-23) and records
  anything a request path missed, including creation paths with no hook.
* The Stripe webhook records synchronously and fails closed (500, Stripe
  redelivers) if the ledger write fails, like the Clerk signup publish.
* `brevo-signal-relay` (every 5 min) publishes unpublished rows to
  `brevo-tracked-event` and stamps `published_at` only after Pub/Sub accepts.
* `track-brevo-event` checks the `brevo_tracked_event` inbox, then takes a 90s
  lease (`brevo_tracked_event_claim`) before loading the row. Brevo events have
  no idempotency key, so a delivery that finds the lease held throws and
  retries later instead of posting a second copy. If Brevo accepted the event but
  the terminal write failed, the lease is sealed, and the next delivery finishes
  the write without posting again. Each send is aborted after 60s, so the
  lease only lapses once the previous attempt is over (finished, aborted, or
  its worker died). A delivery that reclaims a lapsed lease sends again, which
  errs toward a duplicate over a lost event. The processor loads the row,
  posts `/v3/events`, then stamps `delivered_at` and seals the inbox in one
  transaction. A Brevo 400/404/422 is a permanent rejection (`rejected_at`,
  sealed, captured). Anything else throws so Pub/Sub retries 5 times, then
  dead-letters. If the kill switch is off when a message arrives, the row's
  `published_at` is cleared so the relay resends it once the switch is back on.
* `brevo-lifecycle-signals` runs daily at 15:00 UTC and records the computed
  events.

Turning `BREVO_TRACKED_EVENTS_ENABLED` on after a period off does not replay
the gap: the reconcile window is `max(BREVO_SIGNALS_EPOCH_ISO, now − 3 days)`,
so the first run records direct signals for domain rows from the last three
days only (gigs created, invitations sent, onboardings completed, org setups,
subscription changes). `subscription_started` additionally needs the
subscription itself to be created on or after the epoch. The computed buyer
events start from orgs that have a ledger row, so they begin with those
recent orgs. Anything older than three days at enable time is not sent.

Brevo has no idempotency key on `/v3/events`, so the inbox narrows but cannot
close the duplicate window: a crash after Brevo accepts and before the inbox
commit, or two overlapping deliveries of one message, can repeat an event.
The client does not retry a 5xx on `/events` for that reason.

Useful checks: unfinished work is
`SELECT signal, count(*) FROM brevo_signal_emissions WHERE delivered_at IS NULL AND rejected_at IS NULL GROUP BY 1`;
a relay backlog is rows with `published_at IS NULL` older than ten minutes.

**Entry/exit coverage audit.** `org_setup_completed` (entry, once per org)
and `event_created` (exit, every gig) gate the buyer sequences. Run
`api-events/scripts/brevo/audit-signal-coverage.sql`. It is read-only and wrapped
in `BEGIN TRANSACTION READ ONLY … ROLLBACK`:

```bash theme={null}
psql "$DATABASE_URL" -v since="'2026-09-23T00:00:00Z'" -f api-events/scripts/brevo/audit-signal-coverage.sql
```

It classifies every org and non-template gig created since `since` as
`recorded_once`, `DUPLICATE`, `missing_no_owner_or_admin` (expected: nobody
to address), `pending_next_reconcile` (under 65 minutes old),
`MISSING_outside_reconcile_lookback`, or `MISSING`. It lists the unrecorded
ids and reports ledger delivery state per signal. A healthy result has no
`DUPLICATE`, `MISSING`, or stuck rows. The real-Postgres suite
`brevoSignalReliability.integration.test.ts` pins both guarantees. It checks
re-saves, overlapping sweeps, and a late admin for `org_setup_completed`, and
450 gigs across batches plus the inline path for `event_created`.

### DLQ replay

Every lane's DLQ topic has a Terraform-managed pull subscription named
`<env>-<subscription>-dlq` (7-day retention, never expires, no dead-letter
policy of its own, output `dlq_subscription_names`). It only holds messages
dead-lettered after it was created; older ones are reachable through the DLQ
topic's 14-day retention with a temporary subscription and `seek`.

| Lane | DLQ subscription | Republish to |
| - | - | - |
| Signup contact | `<env>-upsert-brevo-signup-contact-dlq` | `<env>.marketing.brevo_signup_contact.v1` |
| Tracked event | `<env>-track-brevo-event-dlq` | `<env>.marketing.brevo_tracked_event.v1` |

A dead-lettered signup message is the only durable copy of that signal. A
dead-lettered tracked event still has its ledger row (`published_at` set,
`delivered_at` and `rejected_at` NULL). On a page, fix the cause first (wrong
or expired key, Authorized IPs, an attribute name that no longer exists),
then:

```bash theme={null}
ENV=prod PROJECT=soundcheck-live   # dev-soundcheck-live / qa-soundcheck-live
SUB="$ENV-track-brevo-event-dlq"    # or $ENV-upsert-brevo-signup-contact-dlq
TOPIC="$ENV.marketing.brevo_tracked_event.v1"  # or $ENV.marketing.brevo_signup_contact.v1

gcloud pubsub subscriptions pull "$SUB" --project="$PROJECT" --limit=50 --format=json > dlq.json
jq -r '.[].message.data' dlq.json | while read -r data; do
  gcloud pubsub topics publish "$TOPIC" --project="$PROJECT" \
    --message="$(printf '%s' "$data" | base64 --decode)"
done
gcloud pubsub subscriptions ack "$SUB" --project="$PROJECT" \
  --ack-ids="$(jq -r '[.[].ackId] | join(",")' dlq.json)"
```

Republish only the message data. Drop the `CloudPubSubDeadLetter*`
attributes Pub/Sub adds on dead-lettering; the processors read nothing but
the JSON body. Pull without `--auto-ack` and ack only after every publish
succeeds; unacked messages reappear after the 60-second ack deadline. Replay
is safe: both processors dedupe on `event_id`, the signup upsert is
idempotent on `EXT_ID`, and a tracked event already delivered is sealed
without calling Brevo. For a tracked event whose DLQ copy has aged out,
`UPDATE brevo_signal_emissions SET published_at = NULL WHERE id = '<id>' AND delivered_at IS NULL AND rejected_at IS NULL`
hands it back to the relay.

The client retries a 429 on every call. It retries a 5xx only for contact
upserts: a tracked event has no idempotency key, and a duplicate re-enters a
Brevo automation, so an ambiguous 5xx on `/events` surfaces to the caller.

Enablement after merge:

1. Apply `infra-gcp` so Cloud Run mounts `BrevoAPIKey` (PLACEHOLDER until populated) and the `marketing.brevo_signup_contact.v1` and `marketing.brevo_tracked_event.v1` topics, subscriptions, DLQs, DLQ pull subscriptions, and the three `maintenance.brevo-*` scheduled jobs exist. The `brevo_signal_emissions` migration applies on the next api-events boot.
2. Populate `{env}-BrevoAPIKey` with `gcloud secrets versions add` using the `soundcheck-app-prod` key from the Brevo account.
3. Brevo's Authorized IPs: either keep the feature off, or allowlist the environment's Cloud NAT egress address. Prod egress is pinned through Cloud NAT; read it from the Terraform output `cloud_run_nat_ip` (`terraform -chdir=infra-gcp/environments/prod output cloud_run_nat_ip`). A blocked IP surfaces as a 401.
4. Stub run: `BREVO_SIGNALS_ENABLED=true` and `BREVO_TRACKED_EVENTS_ENABLED=true` (set it in `extra_cloud_run_env` beside `BREVO_SIGNALS_ENABLED`) with dry-run still on. Confirm the `STUB BREVO CONTACT UPSERT` log line on a test signup and `STUB BREVO EVENT` lines after creating a gig (within five minutes, once the relay runs).
5. Live on dev with one test signup and one gig. Check the contact in Brevo carries `EXT_ID`, `SIGNUP_DATE`, and names, and that `org_setup_completed` / `event_created` appear on its activity; then qa, then prod.

## Resend webhook

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `RESEND_WEBHOOK_SECRET` | `resendWebhookConfig()` | Optional at startup. Svix verification for `POST /webhooks/resend`; empty or placeholder → 503 (route inert). |

Subscribes to Resend email lifecycle events (`delivered`, `clicked`, `bounced`,
`complained`, `suppressed`, `failed`). The handler verifies the Svix signature,
publishes a `ResendEmailLifecycleEvent` to Pub/Sub (`resend-email-lifecycle`),
and marks ingress dedupe (`resend_webhook`) only after publish succeeds. A
subscription processor (`capture-resend-email-analytics`) claims with a 20s
lease (not the default 5m) so a crashed in-flight owner is reclaimable before
five Pub/Sub deliveries exhaust, then awaits PostHog capture and marks
`resend_email_analytics` on success. Capture is subscription-only so a
killed request cannot hold the inbox claim after 200. Programmatic sends
should attach identity tags (`clerk_user_id`, `org_id`) so captures can resolve
`distinct_id`. Missing identity tags still ack 200 but skip analytics.

GCP Secret Manager provisions `ResendWebhookSecret` (mounted as
`RESEND_WEBHOOK_SECRET`). It is not copied from Azure Key Vault — after
Terraform apply, replace the `PLACEHOLDER` version with `gcloud secrets versions add`.
Cutover (register the Resend webhook URL and event types) is tracked in ENG-570.

## Stripe webhook

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `STRIPE_WEBHOOK_SECRET` | `stripeConfig()` | Verifies `POST /webhooks/stripe` |
| `STRIPE_*` (API keys) | `stripeConfig()` | Connect, checkout, payouts |

## Clerk

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `CLERK_SECRET_KEY` | `clerkConfig()` | JWT verification |
| `CLERK_WEBHOOK_SECRET` | `clerkWebhookConfig()` | Svix verification for Clerk webhooks |
| `CLERK_API_BASE_URL` | `clerkConfig()` | Local emulator only (`:4711`); rejected in QA / production |

### Portal Clerk `public_metadata` binding

Portal invitations bind the contact and org via Clerk
`public_metadata`:

* `portal_contact_id`
* `portal_organization_id`

Provision middleware skips `/api/v1/portal/**`. Portal scope is resolved
server-side from that metadata — never from the request body or query.

## Caller auth

Clerk session JWTs are the only caller auth. `ENABLE_BASIC_AUTH=true` fails
closed. E2e fixtures mint emulator session tokens so JWT `sub` matches
Postgres `User.id`.

`/api/v1/admin/**` is parked and 404s every authenticated request. Optional
`WHITE_LABEL_ENFORCE_ENTITLEMENT=true` still surfaces on entitlement bundles
if that console is revived later.

## Plan entitlements

`entitlements/registry.json` says which plan tier each paid feature needs;
`src/generated/entitlements.ts` is generated from it. Check a feature with
`requireEntitlement(orgId, key)` from `src/systems/entitlements/entitlements.ts`.
It resolves the org's tier from the newest active, trialing or past-due
subscription, then a live comp, and caches it for 10 seconds (at most 5,000 orgs).

* A key marked `enforced` answers `403 PLAN_UPGRADE_REQUIRED` below its tier.
* A key marked `shadow` lets the call through. It logs `entitlement: would deny`
  and sends `entitlement_rejected` to PostHog at most once an hour per org and key.
* `ENTITLEMENTS_ENFORCE=off` turns every key into shadow mode.
* `GET /api/v1/organizations/:orgId/entitlements` returns the tier, seat limits,
  and each key's `allowed` and `enforced` state to any org member.

Org-scoped routes are checked by `entitlementMiddleware`, from the route table in
`src/systems/entitlements/entitlementRoutes.ts` (first match wins). Every route
starts in shadow mode, whatever its key's `enforcement` says, because web, mobile
and MCP surfaces that were never plan-gated call the same routes. A route blocks
only once its rule sets `enforce: true`, and then only for org members: a
non-member falls through to the handler's own 403 so the org's plan is not
revealed. Crew self-service routes (offers, availability requests, own
availability, `/onboarding/me`), Assets, data exports, disconnects, OAuth
callbacks and status reads are deliberately left out; `entitlementRoutes.test.ts`
lists them. Each member MCP tool call checks `mcp-access` for the org its input
names, falling back to the token's active org. Tools that take only a resource id
(such as an event id) are attributed to the token's org, so move the check into the
operation-level org authorization before enforcing `mcp-access`.

A row in `organization_entitlement_grants` gives one org an entitlement outside
its tier (grandfathering, a sales deal, a support exception); plan checks and
the entitlements endpoint count it. Staff manage grants with
`scripts/entitlements/grants.ts`. Before switching a key with a
`grandfatherFromTier` to `enforced`, run `--grandfather <key> --before <ISO>` (dry run) and
then add `--apply`. `--before` is when the key moved up a tier. Only orgs created before it,
that held `grandfatherFromTier` or above at `--before` (per the audited subscription and
comp history) and are still on it, get a grant. A cutoff older than the audit history is
refused.

White-label and the dedicated e-sign identity still pass unless
`WHITE_LABEL_ENFORCE_ENTITLEMENT=true`.

## Twilio and event chats

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` | `twilioConfig()` | Programmable Messaging + Conversations webhook signatures |
| `TWILIO_FROM_NUMBER` | `twilioConfig()` | Invitation / reminder SMS sender — **not** a chat proxy |
| `TWILIO_USE_STUB_SMS` | `twilioConfig()` | Stub invitation SMS locally |
| `TWILIO_API_KEY_SID` / `TWILIO_API_KEY_SECRET` | `eventChatsConfig()` | Conversations REST + AccessToken JWTs (`SK…`) |
| `TWILIO_CONVERSATIONS_SERVICE_SID` | `eventChatsConfig()` | Dedicated Conversations Service (`IS…`) |
| `TWILIO_GROUPCHAT_NUMBERS` | `eventChatsConfig()` | SMS proxy pool (2–4 E.164s; must not overlap `TWILIO_FROM_NUMBER`) |
| `TWILIO_WEBHOOK_URL` | `twilioConfig()` | Programmable Messaging webhook URL used to validate inbound invitation/reminder SMS callbacks; distinct from the Conversations webhook URL |
| `TWILIO_CONVERSATIONS_WEBHOOK_URL` | env | Exact URL Twilio signed for `POST /webhooks/twilio/conversations` |
| `TWILIO_SMS_STATUS_WEBHOOK_URL` | `twilioConfig()` | `StatusCallback` sent with each outbound SMS, and the URL Twilio signed for `POST /webhooks/twilio/sms-status`. Unset: no callbacks |
| `TWILIO_CONVERSATIONS_ENDPOINT` | `eventChatsConfig()` | Emulator base URL only — unset in deployed environments |

Launch steps: [event chats runbook](./event-chats).

### Platform admin (service console)

Soundcheck staff identity is a Clerk user `public_metadata` flag, not an org
role and not the parked `X-Staff-Actor` allowlist:

* Grant or revoke: set `public_metadata.platformAdmin` to `true` or `false` on the
  Clerk user. Users cannot write this themselves. A platform admin can do it from
  `/service-console` for one existing Clerk user whose email domain is exactly
  `soundchecklive.io`. The Dashboard and Backend API remain available.
* Session token: include user public metadata on the JWT. Clerk's default
  claim name is `metadata`; the web middleware also accepts `public_metadata`.
* Web: `/service-console` is gated by that flag. Org `admin` / `owner` is not
  enough. Local emulator: Avery Owner is seeded with the flag.
* API: authorize with `isPlatformAdminFromClaims` on Clerk-verified
  session claims (`api-events/src/auth/platformAdmin.ts`).
  `isPlatformAdminFromJwt` only decodes an already-verified token and is
  not an authentication gate. Do not reintroduce Basic Auth.

## Public URLs

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `API_BASE_URL` | `publicUrlConfig()` | Public API origin (default `http://localhost:4000`) |
| `WEB_APP_URL` | `publicUrlConfig()` | Portal invite links |
| `EVENT_DETAILS_URL` | `publicUrlConfig()` | Event deep links |

## Bank-account encryption

| Env / secret | Config accessor | Purpose |
| - | - | - |
| `CREDENTIAL_ENCRYPTION_KEY` | `bankAccountConfig()` | Base64 AES-256 key shared with QuickBooks token crypto |
| `CREDENTIAL_ENCRYPTION_DISABLED` | `bankAccountConfig()` | Opt-out (forbidden in deployed environments when storage needs a key) |
| `BANK_ACCOUNT_STORAGE_DISABLED` | `bankAccountConfig()` | Disables org/member bank-account HTTP; admin verification routes ignore this flag |

Admin bank-account routes stay available even when storage is disabled.
Org funding for ACH transfer paths expects a Cleo-VERIFIED funding account
with a Cleo funding-source id. The member must have a Cleo-ready contact.
Queueing a direct deposit persists an immutable ACH instruction on the
transaction (source, destination, Cleo contact, and funding-source ids) and
publishes `cleo.payment.requested` as a wake-up. The processor claims a 60s
submit lease and creates the Cleo payment from those stored fields only.
Cleo sweeps require the frozen instruction columns so they cannot fail
legacy/manual ACH. A failed publish fails the PENDING/PROCESSING pair so the
payable is not left in flight. Settlement to PAID is still webhook-driven and
GET-authoritative; a terminal webhook whose GET is still processing is retried.
Bank-account delete returns 409 while any in-flight ACH references the row,
then marks `REMOVAL_PENDING` and revokes the Cleo resource. Owner lists exclude
removed/removal rows.

## Feature-flag gated routes (server-side PostHog evaluation)

api-events evaluates PostHog feature flags server-side (`src/systems/shared/featureFlags.ts`,
`posthog-node`, remote evaluation, 60 s cache, fail-closed). A gated route or job uses the same
flag key the web uses, so rollout lives in one place. Do not add per-org env allowlists.

| Env / flag | Where | Purpose |
| - | - | - |
| `POSTHOG_API_KEY`, `POSTHOG_HOST` | Cloud Run (`enable_posthog`) | Project token + host used for evaluation; without them every gated route fails closed (404) and gated jobs stay off |
| `FEATURE_FLAG_OVERRIDES` | local dev / test only | Comma list of `key` or `key=true\|false`; ignored when deployed (`K_SERVICE`) or outside development/test |
| `POSTHOG_FLAGS_LOCAL_EVALUATION` | optional | `true` switches to local evaluation using the query key |
| `crew-offer-unclaimed-dispatch` | PostHog flag (per org) | Allows crew-offer and availability-request **email** to imported ("unclaimed") roster members who never signed up |

`crew-offer-unclaimed-dispatch` replaces the retired `CREW_OFFER_UNCLAIMED_DISPATCH_ORGS`
allowlist and is evaluated per organization from the dispatchers (no caller). It is off everywhere
by default: an org can import every freelancer it ever hired, and those people never opted in, so
one "send to all" would become hundreds of first-contact emails from a domain that never mailed
them. Target an org only once its first-contact template and unsubscribe are in place. SMS to
unclaimed recipients stays suppressed regardless of the flag (TCPA exposure is worse than CAN-SPAM).

## Gig workbook export (West Coast Music)

`GET /api/v1/organizations/:orgId/events/:eventId/wcm-workbook.xlsx` fills the
customer's own Excel binder (13 tabs, curated template at
`api-events/src/systems/gig-workbook/templates/`) from a gig. It is a
per-customer surface, gated by one PostHog flag on both sides:

| Env / flag | Where | Purpose |
| - | - | - |
| `wcm-gig-workbook` | PostHog flag (web + api-events) | Shows the "Event workbook (Excel)" row in the gig Export hub, and gates the endpoint itself — api-events evaluates the same flag server-side for the caller (`distinctId` = Clerk user id, person properties `email` / `active_organization_id` / `organization_ids`). Flag off, PostHog unreachable, or PostHog unconfigured ⇒ 404, so the surface never leaks |
| `FEATURE_FLAG_OVERRIDES` | api-events env | Local/e2e escape hatch: comma list of `key` or `key=true\|false`, honoured only when `ENVIRONMENT` is `development` (non-Cloud-Run) or `test`. `dev-emulator.mjs` and the encore-e2e job set `wcm-gig-workbook` |

Callers need the gig-manager role; prices, deposits, overtime rate and payroll
cells are written only for callers with financial clearance. The template
recomputes its contract, invoice and details tabs on open (`fullCalcOnLoad`);
re-curate it with `scripts/gig-workbook/curateWcmTemplate.ts` when the
customer sends a new master copy.

## Apple Music song import

Mobile and web search the Apple Music catalog, or paste a song, album or playlist link, to create library songs. The system lives in `api-events/src/systems/music/`; provider HTTP and developer-token signing are in `api-events/src/clients/appleMusicClient.ts`. Imports are metadata only: Apple Music audio is DRM-protected and is never downloaded or stored. Responses carry title, artist, album, duration, ISRC, artwork and the catalog link for the picker; a song created from a track keeps its title, artist and duration, with the catalog link as a media link. ISRC, artwork and ids are display and duplicate-check data and are not persisted.

| Endpoint | Purpose |
| - | - |
| `GET /api/v1/organizations/{orgId}/music/search?q=` | Song search in one catalog. `provider` is `apple_music` (default), `spotify` or `soundcloud`; `limit` 1-25 (default 10), `storefront` two-letter country (default `us`). Clients search every catalog from `/providers` in parallel. |
| `POST /api/v1/organizations/{orgId}/music/match` | Matches a pasted set list (`lines`, up to 50) across every available catalog. Each line returns the title and artist read from it, `status` (`matched`, `check`, `none`), up to 3 candidates merged across catalogs (same ISRC, or same title and artist) with a link per catalog, and `library_song_id` when the library already has the song. Lines run four at a time; a line fails only when every catalog fails, and the request fails only when every line does. |
| `GET /api/v1/organizations/{orgId}/music/providers` | The catalogs this organization can search now, in display order (`{ search: [{ id, label }] }`). |
| `GET /api/v1/organizations/{orgId}/music/resolve?url=` | Resolves an Apple Music song, album or playlist link to its tracks (at most 300; `truncated` says when there were more). |

Both routes need the caller to manage gigs in the organization (`gig.manage`), the same bar as creating songs. Neither writes anything: the client creates songs through the existing `POST /api/v1/songs`. Links are parsed, never fetched; only `https://music.apple.com` and `https://embed.music.apple.com` paths are accepted and only the storefront and catalog id are used.

Configuration is three secrets, all from one Apple Developer MusicKit key: `APPLE_MUSIC_TEAM_ID`, `APPLE_MUSIC_KEY_ID` and `APPLE_MUSIC_PRIVATE_KEY` (the full `.p8` file; literal `\n` sequences are accepted). Terraform: `enable_apple_music_secrets` creates the three PLACEHOLDER-seeded secrets (`infra-gcp/README.md`). There is no rollout flag: with any of the three unset or still `PLACEHOLDER`, both routes answer `503 MUSIC_PROVIDER_UNAVAILABLE` and nothing leaves the service. The client signs an ES256 developer token (raw 64-byte signature, one hour lifetime) and reuses it until five minutes before expiry. A rejected token or an Apple outage also answers 503 and is reported to Sentry as `music.search` or `music.resolve`; Apple's 429 answers `503 MUSIC_PROVIDER_BUSY` without an alert. `APPLE_MUSIC_API_BASE_URL` points the client at the emulator (loopback or `emulators` only, ignored in QA and production).

### Spotify and SoundCloud search

Each extra catalog has its own client in `api-events/src/clients/` (`spotifyClient.ts`, `soundCloudClient.ts`, sharing `musicProviderHttp.ts`) and joins `/providers` only once its secrets hold real values. Terraform: `enable_music_search_secrets` creates `SPOTIFY_CLIENT_ID`, `SPOTIFY_CLIENT_SECRET`, `SOUNDCLOUD_CLIENT_ID` and `SOUNDCLOUD_CLIENT_SECRET`, PLACEHOLDER-seeded. A catalog without credentials answers `503 MUSIC_PROVIDER_UNAVAILABLE` and nothing leaves the service.

* **Spotify**: client-credentials token (cached until a minute before expiry, refreshed once on a 401), `/v1/search?type=track`. Pasted Spotify track and album links (and `spotify:` URIs) resolve through the same client.
* **SoundCloud**: OAuth 2.1 client-credentials token from `secure.soundcloud.com`, `/tracks?q=` limited to playable and preview tracks. Needs an approved SoundCloud API app.

A provider 429 answers `503 MUSIC_PROVIDER_BUSY` without an alert; other provider failures answer 503 and report to Sentry with the provider name. `SPOTIFY_API_BASE_URL`, `SPOTIFY_ACCOUNTS_BASE_URL`, `SOUNDCLOUD_API_BASE_URL` and `SOUNDCLOUD_AUTH_BASE_URL` point a client at an emulator locally (ignored in QA and production).

Search, resolve and match results carry `library_song_id` when the organization's song library already has the song. The check compares title and artist ignoring case, accents and punctuation, so a live or remastered version is a different song; a library song with no artist matches any artist. The library's song titles (at most 10,000) are read once per request.

Smoke test after populating the secrets: `GET /api/v1/organizations/{orgId}/music/search?q=dreams` as a gig manager returns tracks with `duration_seconds`; the same call as a crew member returns 403.

## Forwarded confirmation intake (MAR-39)

The signed Inbound Parse endpoint is `POST /webhooks/forwarded-email`. It is separate from outbound organization sending identities and the conversational CheckAI email channel. Sending-domain verification CNAMEs and Clerk team-domain enrollment do not provision this receiving host or authenticate a forwarded message.

Configure `FORWARDED_EMAIL_ENABLED=true`, `FORWARDED_EMAIL_DOMAIN` (a dedicated inbound subdomain with SendGrid MX/Parse routing), and `INBOUND_EMAIL_VERIFICATION_KEY` or Encore secret `InboundEmailVerificationKey` (the Parse signature public key). Enable signed parsed multipart delivery, not raw MIME mode. The settings address is hidden until all three configuration values are present. Keep the existing ingestion-recovery scheduled job and structured AI provider configured; it owns retry and source cleanup. The feature defaults off. The self-hosted dev, qa, and prod infra files declare an empty `InboundEmailVerificationKey` fallback so disabled deployments can build without provisioning a receiving key. Set `INBOUND_EMAIL_VERIFICATION_KEY` at runtime before enabling forwarding; the environment value takes precedence over that empty fallback.

The recipient alias selects the organization. Aligned sender authentication and exactly one active member are required for extraction; a verified organizational sending domain never bypasses either check. Disconnecting or changing the organization's From domain leaves the inbound alias unchanged. No sender replies are generated by this workflow.

Acknowledgement follows durable receipt storage. Sources expire after 30 days; imported/discarded history and reviewed proposals remain. General audit snapshots exclude the source ledger to avoid duplicating private email content. Deployment verification must include one signed member forward, one held unknown sender, an identical redelivery, manager approval, and inspection of the resulting native gig records.

Source cleanup continues while forwarding is disabled. Review responses include source text and attachment filenames; attachment bytes remain available only to extraction. Each extraction has an eight-minute deadline within a ten-minute claim lease, and a stale completion cannot publish a proposal.

A later confirmation updates matching flight legs. A new route for an existing passenger booking requires resolving the itinerary on the gig before import; a fresh multi-leg booking can be approved together. Contract corrections replace their marked section in admin notes and preserve surrounding notes. If that section has been removed or its markers changed, approval stops for manual resolution.

Non-flight confirmations match by organization, record type, and case-insensitive booking reference. Changing a title, passenger detail, record order, or selected gig cannot create a second mapping for that reference; selecting another gig requires resolving the existing booking first. Records without a confirmation reference remain specific to their receipt and row, since there is no reliable identity for matching separate emails. Repeated references of the same type within one review are rejected as ambiguous.

CSV extraction uses a preview of the header and at most 200 data rows, 40 columns, and 256 characters per cell. CSV attachments share a 64,000-byte preview budget plus filename and truncation notices. Reviewers should consult the original attachment when a preview omits information.

CSV parsing scans at most the first 128 KiB and stops after 201 complete records, retaining quoted multiline cells while discarding an incomplete trailing record at the byte limit. PDF text conversion is attempted before binary analysis; binary PDFs are limited to 20 MiB and 100 pages across the request, images to 5 MiB each, and combined binary attachments to 20 MiB. Unsupported sizes stop with an actionable review error without automatic model retries.

Flight legs are shared by airline, flight number, route, and departure instant. Passenger confirmation codes are stored on the passenger association, so separate bookings for the same scheduled flight do not create duplicate legs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.