api-events operations

Provider and authorization configuration for api-events/. Values are read in api-events/src/config/config.ts (Encore secret(...) with env fallbacks). Use this page for smoke tests and incident diagnosis.

Local provider emulators

Every outbound provider has a local emulator built on emulate (emulate@0.12.0, pinned under the npm release-age policy). Custom providers are defineEmulator definitions with JSON-only state:

tools/emulators/ — Stripe, SendGrid, Resend, Brevo, QuickBooks, Dropbox, Google Calendar, Expo push, Supermemory, context.dev, n8n, PostHog, Luma, Apple Music, the web-only Google Maps and Spotify, plus emulate’s built-in Slack and Google (Drive). Ports 4101-4125; the Compose service is emulators. See tools/emulators/README.md for the port and env table.
tools/twilio-emulator/ (:4013) and tools/cleo-pay-emulator/ (:4092) — same framework, own packages and existing contracts.
tools/clerk-emulator/ (:4711) — still the vendored upstream emulator. Switching to emulate’s Clerk must preserve our persona manifest, persistent signing keys, and Svix-signed webhooks.

Overrides added for the emulators (STRIPE_API_BASE_URL, SENDGRID_API_BASE_URL, QUICKBOOKS_*_BASE_URL, DROPBOX_API_BASE_URL, GOOGLE_DRIVE_API_BASE_URL, GOOGLE_CALENDAR_API_BASE_URL, EXPO_PUSH_API_BASE_URL, SLACK_API_BASE_URL, HUBSPOT_API_BASE_URL, FLEX_API_BASE_URL) go through localEmulatorBaseUrl() in config.ts: loopback or the emulators Compose host only, ignored in QA and production. The older provider URL settings the emulators also use (RESEND_BASE_URL, BREVO_BASE_URL, SUPERMEMORY_BASE_URL, PRICING_CONTEXTDEV_BASE_URL, N8N_*_WEBHOOK_URL, POSTHOG_HOST, POSTHOG_QUERY_HOST) are honored in every environment, so never set them to an emulator outside local dev and CI. Stub flags still win when set. The emulators are opt-in for dev:emulator; set the env listed in the README to route a provider to them. Contract tests (*.emulator.test.ts beside each client) run the real client against its emulator in-process and are part of the normal unit suite. The checks job also type-checks tools/emulators and boots every provider (npm run smoke). Known gaps: Twilio’s emulator does not cover Conversations SDK WebSockets; built-in Slack has no apps.uninstall; built-in Google accepts any bearer and has no Docs export. HubSpot and Flex have no API emulator because api-events only builds links to them (Flex quotes go through the n8n webhook).

Organization sending domains (SendGrid)

Pro-and-above orgs authenticate their own domain at SendGrid and send from local_part@their-domain instead of the platform address. The system lives in api-events/src/systems/sending-domains/, provider HTTP in api-events/src/clients/sendGridDomainAuthClient.ts, persistence in api-events/src/db/organizationSendingDomains.ts.

Endpoint	Purpose
`GET /api/v1/organizations/{orgId}/sending-domains`	Live domains plus the platform From address. Excludes `DISCONNECTED`.
`POST /api/v1/organizations/{orgId}/sending-domains`	Claims the domain locally, then provisions it at SendGrid and stores the CNAME records.
`POST /api/v1/organizations/{orgId}/sending-domains/{domainId}/verify`	Re-runs SendGrid validation and merges the per-record verdicts.
`PUT /api/v1/organizations/{orgId}/sending-domains/{domainId}/default-from`	Sets the default From local part and display name. `VERIFIED` only.
`POST /api/v1/organizations/{orgId}/sending-domains/{domainId}/test-email`	Sends a test message to the calling admin’s own address.
`DELETE /api/v1/organizations/{orgId}/sending-domains/{domainId}`	Marks the row `DISCONNECTED` and queues the SendGrid delete, in one transaction.

Statuses are PENDING → VERIFIED | FAILED, plus DISCONNECTED once removed. Every check stores a human summary of the invalid records in last_error and stamps last_checked_at. A failing check does not immediately mean FAILED. SendGrid returns the same valid: false for “the records are wrong” and “the records are published but have not propagated yet”, so a domain stays PENDING for 48 hours after it was added and only then starts reporting FAILED — an admin who re-checks a minute after editing their zone file sees “waiting”, not “action needed”. The two exceptions are a domain that was already VERIFIED (a regression is never propagation lag, so it fails at once) and a row whose created_at cannot be parsed (fails closed). The audit outcome metadata distinguishes all three: pending | failed | verified. If SendGrid answers the validate call with a 404 — the authenticated domain was deleted out from under us — the check does not report a provider outage. It marks the row FAILED with last_error “This domain no longer exists at the email provider. Remove it and add it again.” and returns 200, so the admin gets an instruction instead of a retry loop. There is no scheduled re-check in V1 — verification is the manual button only. Every endpoint requires org ADMIN/OWNER. Add, verify, default-from and test-email additionally require the sending-domains entitlement (Pro and above) and answer 403 PLAN_UPGRADE_REQUIRED (not a 404) without it, so the settings page can show an upgrade CTA. The list and remove endpoints deliberately do not check the entitlement: an org that drops below Pro while holding a domain must still be able to see it and disconnect it, otherwise the row keeps the global domain claim and the SendGrid resource forever. The list response carries entitled: boolean so the page can render a downgraded org in remove-only mode. Error codes: INVALID_DOMAIN, DOMAIN_ALREADY_ADDED, DOMAIN_CLAIMED, DOMAIN_NOT_VERIFIED, DEFAULT_FROM_REQUIRED, CALLER_EMAIL_REQUIRED, PROVIDER_ERROR, NOT_FOUND. The add flow is row-first, and the row is never thrown away. The row is inserted PENDING with a null provider_domain_id before SendGrid is called, so the claim on the domain is taken under the unique index rather than by a provider round trip, and the 409 conflicts are decided with no provider call at all. A provider-less row means “needs provisioning”, and provisionPlaceholder is the single path that fills it, from both add and re-check: look the domain up by name at SendGrid (GET /v3/whitelabel/domains?domain=), adopt it if it is already there, create it only if it is genuinely absent, then attach the id and records under a guarded write (provider_domain_id IS NULL AND status IN ('PENDING','FAILED'), which also resets the row to PENDING and clears last_error). The lookup-before-create is what makes the ambiguous case safe: a request that timed out after SendGrid accepted the create leaves a real authenticated domain behind, and a naive retry would create a second one and orphan the first. Adoption requires proof of ownership. SendGrid authenticated domains live in one shared account, so an exact name match proves nothing — a Pro org could otherwise adopt a domain someone else authenticated (manually, or through another integration) and reach VERIFIED without ever publishing DNS. Every object we create therefore carries a per-row identity derived from the row uuid: subdomain = sc + the first 10 hex digits, and custom_dkim_selector = 3 alphanumerics from the same digits, so two rows on one root domain never collide on s1._domainkey.<domain>. Nothing is stored for this; it is recomputed from row.id. The lookup returns every exact-name match and we adopt only the one whose subdomain equals ours. Other matches are left strictly alone — never adopted, never deleted. That identity also removes the deleted-org handover race: an object queued for release belonged to a different row and therefore carries a different subdomain, so it can never be adopted by whoever claims the domain next. Before creating we do eagerly release any queue entry for the same domain, so a rejected create (SendGrid refusing a duplicate) does not leave the next org waiting on an unrelated drain. The drain carries the mirror-image guard: before deleting at SendGrid it re-reads organization_sending_domains for an active row referencing that provider id, and if one exists it drops the queue entry untouched and counts it superseded. That is defensive — guard one should make it unreachable — but it also covers handles attached by any other route, including manual repair. A provider failure during provisioning never deletes the row. The row stays PENDING with last_error “Provider setup did not complete. Re-check to retry.” and a fresh last_checked_at, and the endpoint returns 502. That keeps the domain claimed by the org that asked for it — releasing the claim on a transient outage would let another organization take the domain out from under them — and makes re-check the retry button. Re-adding the same domain in the same org reuses the stalled row; another org still gets DOMAIN_CLAIMED. sending_domain_added is audited only on the first successful attach; a re-provision from verify audits sending_domain_checked. Organization deletion. The organization_id cascade would destroy the only copy of provider_domain_id, so deletion enqueues the cleanup instead of performing it. Inside the delete transaction — in deleteOrganizationOnClient, so both the org-settings endpoint and the Clerk organization.deleted webhook get it — one sending_domain_release_queue row is written per live domain that holds a provider handle, immediately before the cascade. That table carries no foreign key to Organization for the same reason the twilio_conversation_deletion_v2 effect in effect_outbox does not: the queue has to outlive the organization it describes. A deletion that never commits enqueues nothing. Removal works the same way round. DELETE disconnects the row and enqueues its provider id in one transaction, then returns 200 — it never calls SendGrid on the request path, so a provider outage can no longer leave an active row pointing at a deleted resource, and the response does not depend on the provider being reachable. A single targeted release runs eagerly afterwards as a latency optimisation only; the queue is what guarantees the delete. Draining is a bounded pass (drainSendingDomainReleaseQueue): a SendGrid delete (404 counts as success) removes the queue row, an entry whose provider id a live row has since claimed is dropped without calling SendGrid and counted superseded, and any failure increments attempts, records last_error and leaves the row for the next pass. It never throws into its caller. The scheduled job sendingDomainReleaseDrain runs it every 15 minutes (limit 25, 60s time budget) under the shared job lock, with the usual private expose: false manual endpoint for ops. Request paths do not await a drain: organization deletion (both the settings endpoint and the Clerk webhook) and the add / verify operations publish to the drain topic after commit, the same shape as publishTwilioConversationDeletePrompts. The only eager drains left are the two single-entry, specifically-targeted ones above (pre-create and post-remove), which stay best effort. Watch the depth of sending_domain_release_queue and its attempts column — broker backlog is not the same as unfinished work. PRO-449 now covers only the verification re-check sweep. Clerk verified-domain mirror (PRO-450). When a check takes a domain to VERIFIED, the same request best-effort mirrors it onto the org’s Clerk Organization Domains via ensureVerifiedOrganizationDomain: an existing entry with that name (case-insensitively) is flipped verified: true keeping its enrollment mode, and a missing one is created automatic_suggestion + verified: true. Success stores clerk_domain_id and clerk_domain_synced_at on the row under a status = 'VERIFIED' guard; failure is reported through captureServiceError(…, 'clerk', 'EnsureVerifiedOrganizationDomain', …) and the verify still returns 200 VERIFIED — DNS proof is the product promise, the Clerk mirror is a convenience. The sending_domain_verified audit metadata carries clerk_domain: created | verified | already_verified | failed. A re-check also runs the mirror when a row is already VERIFIED but clerk_domain_synced_at is null, so domains verified before this shipped — or during a Clerk outage — catch up on the next re-check rather than needing a backfill. clerk_domain_synced_at therefore has no column default: stamping every existing row would claim a sync that never happened and permanently disable that catch-up. SENDING_DOMAIN_CLERK_SYNC=false switches the mirror off for a Clerk instance that has Verified Domains disabled; SENDING_DOMAIN_CLERK_SYNC_USE_STUB, and the notification dry run, stub it. Removal deliberately does not touch Clerk. Disconnecting a sending domain says nothing about whether people with an address there should still be able to join the org — that is enrollment, and it belongs to the Staff page’s Team Domain panel. Revoking it here would silently change who can sign up. Audit triggers. organization_sending_domains and sending_domain_release_queue are classified in audit.table_config with enabled = true and carry no capture trigger, which is the intended state rather than an oversight: per 20260825210000_audit_trigger_floor, enabled means “MAY be audited”, attachment is a deliberate named wave, and 102 of the 107 classified tables sit exactly here. Only wave 5a (bank_accounts, OrgMember, Organization, Subscription, white_label_config) has ever attached. Every mutation on a sending domain already writes a semantic recordAudit row, which is the customer-facing surface; adding these tables to the trigger floor is a separate write-volume decision for whoever runs the next wave.

Env / secret	Config accessor	Purpose
`SENDGRID_API_KEY` / Encore `SendGridAPIKey`	`sendGridConfig()`	Reused as-is — no new secret. *The key needs the “Sender Authentication” (whitelabel) scope with read and* write**, which the mail-send-only key does not have. Without it every add/verify/remove returns `502 PROVIDER_ERROR` and the SendGrid call logs a 403.
`SENDGRID_USE_STUB_EMAIL`	`sendGridConfig()`	Stub mode. The domain-auth client returns synthetic CNAME records, validate reports every record valid, and delete is a no-op, so local dev and tests never call SendGrid.
`NOTIFICATIONS_DRY_RUN`	`dryRun()` / `sendGridConfig()`	Also forces the stub unless it is exactly `false`.

The API key itself is never logged; provider errors carry the domain, the provider domain id, and the HTTP status only. provider_domain_id is not exposed on the wire and is deny-listed in auditClassification.ts. Org correspondence sends from the org’s verified identity. Each org-context sender wraps its params in withOrgSender(params, orgId) (systems/sending-domains/orgEmail.ts). It sets from and replyTo to resolveOrgSendingIdentity(orgId) and leaves the platform sender in place when the resolver returns null. The resolver returns null when any of these fails: the domain is VERIFIED, it has a From address, the org has a Pro+ entitlement, and the org-email-domains PostHog flag is on for the org. It caches per org for 60 s (per instance), and the verify / From / remove operations invalidate that entry. Wired senders:

availability requests
crew offers (including reschedule .ics)
gig invitations (including .ics)
event cancellation
direct notifications (bulk, broadcast and template)
the lead-request confirmation (reply-to prefers the org contact email)
Clerk org team invites

These stay on the platform address:

emails to the org’s own staff (crew responses, new-lead alerts)
Clerk-sent auth mail
CheckAI

The follow-up work is a scheduled re-check sweep and inbound parse (MAR-39).

Cleo Pay

Env / secret	Config accessor	Purpose
`CLEO_ENABLED`	`cleoPayConfig()`	Literal `false` disables Cleo. Unset means on
`CLEO_WEBHOOK_SECRET`	`cleoPayConfig()`	Verifies `Cleo-Signature` on `POST /webhooks/cleo`. Empty secret → 500
Encore `CleoWebhookSecret`	`cleoPayConfig()`	Same value when the env var is unset

CLEO_ENABLED=false blanks the API key and webhook secret. Register and payment sweeps then return without listing the queue or capturing a missing-credentials error, and the publish paths stay quiet. Messages already on the register and payment subscriptions ack without calling Cleo or marking the row failed, so a rollout does not retry them into the DLQ. Those rows stay pending for the sweep once the flag is turned back on. Dev, QA, and production set that value until real credentials are in place. Missing secrets with the flag unset still alarm when queued work exists. GCP Secret Manager provisions CleoWebhookSecret (mounted as CLEO_WEBHOOK_SECRET). It is not copied from Azure Key Vault — after Terraform apply, replace the PLACEHOLDER version with gcloud secrets versions add. Cleo-lane Pub/Sub DLQs (cleo_payment, cleo_bank_register, cleo_contact_register, cleo_webhook_received), Brevo lane DLQs (brevo_signup_contact, brevo_tracked_event), scheduled-maintenance DLQs, and the crew-notification fan-out DLQs (availability request/SMS/email/push/push-target/request-response, invitation send/response, crew-offer send/response) page Slack via Cloud Monitoring through the same channel list. qa/prod must pass cleo_dlq_alert_notification_channel_ids — the same OOB Slack channel resource names operators attach to platform-signals pipeline alerts. Dev may stay empty. Do not invent channel IDs; copy them from Cloud Monitoring after the Slack OAuth channel exists. Terraform apply remains Cloud-Agent-only. An empty channel list still creates the policies; they just do not page. Replay is a manual pull from <env>-<subscription>-dlq and republish to the source topic — there is no Slack consumer on DLQ topics and no auto-replay. POST /webhooks/cleo verifies Cleo-Signature, publishes to cleo-webhook-received, then returns 200. Only the api-events service account can publish that topic. Bank-status apply runs eagerly after the ack and again on subscription apply-cleo-webhook (apply failures are captured, then Pub/Sub retries). Signed bank events (bank_account.created / verified / verification_failed / removed) lock the bank_accounts row and update cleo_bank_account_status when cleo_funding_source_id still matches and the event is newer than the stored Cleo cursor (cleo_last_event_sequence, then cleo_last_event_at). Replacing a funding-source id clears that cursor. Subscription ordering uses Cleo createdAt/sequence only, not receipt time. Stale or equal events are acknowledged without rewriting status. HTTP create/verify must not overwrite that webhook cursor or regress a terminal status. Payable events (payable.paid / payable.payment.failed / partially_paid / reversed) always GET /v1/payables/{id} before a terminal write: fully settled succeeds the locked pair, unpaid failed fails it, in-flight or split-fail stays PROCESSING, and clawback after PAID inserts a REVERSED ACH row. Member contacts have no Cleo webhook in staging — the contact-register cron GETs registered contacts and remaps archived / revoked to REMOVED. Bind topic dev|qa|prod.integration.cleo_webhook_received.v1 before relying on the async path in a deployed environment. A local public probe (Cleo dashboard → laptop) is Encore on :4000 plus an ngrok tunnel. Use the ngrok agent authtoken (NGROK_AUTHTOKEN), not the ngrok API key — leftover files named ngrok.authtoken are often the API key and fail with ERR_NGROK_107. See .agents/skills/ngrok-tunnels/SKILL.md and pitfall 17. Sign a fresh body that includes id; Cleo signatures expire after five minutes and cover {t}.{rawBody} only.

Client forms

Client forms are native: questions, rendering and responses live in Soundcheck (form_definitions, form_revisions, form_assignments, form_submissions), so they need no provider keys, webhooks or secrets. Each assignment’s direct link carries its form_assignments.share_token and is answered through /api/v1/public/form-links/*. The token is minted on first use, by the link email or by one of the two share-link routes (staff and holder); see Forms as a channel.

n8n

Env / secret	Config accessor	Purpose
`N8N_CRM_SYNC_SECRET`	`n8nConfig()`	CRM sync + flex-quote-sync HMAC (non-prod defaults)
`N8N_CRM_SYNC_ORGANIZATION_ID`	`n8nConfig()`	Deployment org lock
`N8N_TRIGGER_SECRET`	`n8nConfig()`	Outbound trigger auth
`N8N_FLEX_QUOTE_WEBHOOK_URL`	`n8nConfig()`	Outbound flex-quote sync trigger
`N8N_SETTLEMENT_WEBHOOK_URL`	`n8nConfig()`	Post-settle flex-settle trigger
`N8N_INVENTORY_WEBHOOK_URL`	`n8nConfig()`	Inventory sync trigger
`N8N_VENUES_WEBHOOK_URL`	`n8nConfig()`	Venues sync trigger
`N8N_HUBSPOT_WEBHOOK_URL`	`n8nConfig()`	HubSpot sync trigger

Integrations deep links (same as historical Go): HUBSPOT_APP_BASE_URL, HUBSPOT_PORTAL_ID, FLEX_BASE_URL via integrationsDeepLinkConfig().

App file storage

api-events uses GCS when GCS_BUCKET is set and Azure Blob Storage otherwise (App file storage).

Env / secret	Config accessor	Purpose
`GCS_BUCKET`	`gcsConfig()`	App file bucket (Terraform `app_storage_bucket`)
`GCS_EMULATOR_HOST`	`gcsConfig()`	fake-gcs-server base URL for local dev and CI

Azure Storage (until decommission)

Env / secret	Config accessor	Purpose
`AZURE_STORAGE_CONNECTION_STRING`	`azureConfig()`	Blob connection (or account name + key)
`AZURE_STORAGE_ACCOUNT_NAME` / `KEY`	`azureConfig()`	Alternate credential shape
`AZURE_STORAGE_CONTAINER` / `_NAME`	`azureConfig()`	Upload container
`AZURE_STORAGE_USE_EMULATOR`	`azureConfig()`	Azurite for local / e2e
`AZURE_STORAGE_PUBLIC_ENDPOINT`	`azureConfig()`	Optional public endpoint override

AI Gateway and event ingestion

Model calls (ingest analysis, multi-file merge, import column mapping) go through Vercel AI Gateway using its Anthropic-compatible Messages endpoint. Model ids use the gateway namespace (anthropic/claude-sonnet-4.6); legacy direct ids (claude-sonnet-4-6) are rewritten on the way out.

Env / secret	Config accessor	Purpose
`EVENT_INGESTION_ENABLED`	`eventIngestionConfig()`	Gate (`1` / `true` / `yes` / `on`)
`INGEST_MERGE_ENABLED`	`eventIngestionConfig()`	Merge path gate
`AI_GATEWAY_API_KEY`	`eventIngestionConfig()`	AI analysis; sets `aiGatewayConfigured` (`PLACEHOLDER` = unset)
`AI_GATEWAY_BASE_URL`	`aiGateway.ts`	Optional host override (default `https://ai-gateway.vercel.sh`)
`INGEST_MAX_FILES_PER_BATCH`	`eventIngestionConfig()`	Batch size (default 20, max 100)
`EVENT_PLANS_ENABLED`	`eventPlansConfig()`	Gate for brief-to-plan (`1` / `true` / `yes` / `on`)
`EVENT_PLAN_EXTRACTOR`	`eventPlansConfig()`	`fixture` runs a deterministic offline extractor (tests/local dev, no gateway key needed); anything else calls the model

AI-merge dependency notes live in api-events/src/systems/event-ingestions/AI_MERGE_DEPS.md.

Mobile update-policy store verification (ENG-57)

Env / secret	Config accessor	Purpose
`GOOGLE_PLAY_PACKAGE_NAME`	`googlePlayStoreConfig()`	Play package name gate for version-availability verification
`GOOGLE_PLAY_SERVICE_ACCOUNT_JSON`	`googlePlayStoreConfig()`	Play Developer API service-account JSON
`APP_STORE_CONNECT_KEY_ID`	`appStoreConnectConfig()`	App Store Connect API key id
`APP_STORE_CONNECT_ISSUER_ID`	`appStoreConnectConfig()`	App Store Connect API issuer id
`APP_STORE_CONNECT_PRIVATE_KEY`	`appStoreConnectConfig()`	App Store Connect API `.p8` private key
`APP_STORE_CONNECT_APP_ID`	`appStoreConnectConfig()`	App Store Connect app id

Each env name is also an Encore secret(...) (GooglePlayPackageName, GooglePlayServiceAccountJson, AppStoreConnectKeyId, AppStoreConnectIssuerId, AppStoreConnectPrivateKey, AppStoreConnectAppId) used when the env var is unset; PLACEHOLDER/empty reads as unconfigured, not a boot failure. GCP Secret Manager provisions them only when the environment sets enable_google_play_store_secrets / enable_app_store_connect_secrets (infra-gcp events module). Neither has an Azure Key Vault source — after apply, populate with gcloud secrets versions add (Play Console service account + package name; App Store Connect API key, issuer, and app id), then roll the Cloud Run revision.

Operator-triggered publication (bootstrap)

Publication is intentionally manual: POST /internal/mobile-update-policy/publish publishes store-verified update rules. Scoped disable and the global service-console switch can advance pointers to empty snapshots; reading a new scope while the switch is OFF initializes an empty snapshot at revision 1. Scheduled reconciliation only refreshes releases already referenced by a published policy. Keep enforcement OFF until acceptance is complete, then use the authorized service-console workflow to permit fresh publication.

Env / secret	Config accessor	Purpose
`MOBILE_UPDATE_PUBLISH_CALLER_EMAIL`	`mobileUpdatePublishOidcConfig()`	Google OIDC `email` allowed to publish. Terraform sets the scheduler HTTP service account. Empty = endpoint rejects every call.
`MOBILE_UPDATE_PUBLISH_OIDC_AUDIENCE`	`mobileUpdatePublishOidcConfig()`	Required token `aud` (`soundcheck.api-events/mobile-update-policy/publish`). Distinct from the scheduled-jobs audience, so a Cloud Scheduler token can never publish.

Runbook (scripts/publish-mobile-update-policy.mjs, dry-run unless --confirm): First read the current scope through the authenticated GET /api/v2/organizations/{organizationId}/mobile/update-policy endpoint using the same platform/channel and installed version/build parameters. Set CURRENT_SNAPSHOT_REVISION to the returned data.snapshotRevision. An empty policy does not imply revision 0: an OFF read starts at revision 1. Revision 0 is valid only if the scope has never initialized a snapshot. Inspect current state after a conflict before preparing a new command; do not automatically replace a stale expected revision.

API_EVENTS_BASE_URL="https://<api host>" \
MOBILE_UPDATE_PUBLISH_CALLER_SA="<scheduler-http SA email>" \
node scripts/publish-mobile-update-policy.mjs \
  --org <orgId> --platform ios --channel production \
  --policy-id ios-min --intent initial \
  --minimum-supported 1.4.0 --version 1.6.0 --build 2031 \
  --territories US,CA --expected-revision "$CURRENT_SNAPSHOT_REVISION" \
  --idempotency-key publish-ios-1.6.0-2031 --confirm

The script mints a token with

gcloud auth print-identity-token --impersonate-service-account=$MOBILE_UPDATE_PUBLISH_CALLER_SA --audiences=<audience> --include-email

(no secret on argv; a pre-minted token may be supplied via MOBILE_UPDATE_PUBLISH_ID_TOKEN). The operator principal needs roles/iam.serviceAccountTokenCreator on the caller service account. The trusted actor is the verified caller email — the body cannot assert an actor. Failure/optimistic-concurrency behavior (exit codes in the script header): conflict (409) means the expected revision is stale or the idempotency key was reused with a different payload — re-read current scope state and retry with a fresh key for a changed command, the SAME key to replay. not-verified (422) means no fresh AVAILABLE store observation covers the release/territory audience — run evidenceMode=verify-fresh (the default) so the platform store adapter is checked at publish time, or wait for a reconciliation observation younger than 60 minutes and use evidenceMode=persisted. unconfigured (503) means the store credentials above are not deployed for that platform.

Resend contact sync

Hourly full recompute of global Resend Contacts (0 * * * * in api-events/src/systems/resend-contacts/). The job is read-only against Postgres and writes outbound HTTP only. Merge does not turn it on.

Env / secret	Config accessor	Purpose
`RESEND_CONTACT_SYNC_ENABLED`	`resendConfig()`	Job gate. Default off.
`RESEND_API_KEY` / Encore `ResendAPIKey`	`resendConfig()`	Global Contacts API. Required when the gate is on and not stubbing.
`RESEND_BASE_URL`	`resendConfig()`	Optional; defaults to `https://api.resend.com`
`RESEND_TIMEOUT_MS`	`resendConfig()`	Per-request timeout. Must be finite and `> 0`; default 15000.
`RESEND_MAX_REQUESTS_PER_SECOND`	`resendConfig()`	Client throttle. Default 2 (Resend’s current allowance).
`RESEND_CONTACT_SYNC_TIME_BUDGET_MS`	`resendConfig()`	Wall-clock budget for one run. Default 240000. Must stay under the Cloud Run request timeout.
`RESEND_USE_STUB`	`resendConfig()`	Force stub writes even when dry-run is off.
`NOTIFICATIONS_DRY_RUN`	`dryRun()` / `resendConfig()`	Writes stub unless this is exactly `false`.
`POSTHOG_QUERY_API_KEY` / Encore `PostHogQueryAPIKey`	`postHogQueryConfig()`	HogQL read key: dedicated PostHog user `phx_` (or later `phs_` if query scopes ship). Not the ingest token (`phc_` / `POSTHOG_API_KEY`). Not a human’s MCP key.
`POSTHOG_PROJECT_ID`	`postHogQueryConfig()`	Numeric project id. Both this and the query key are required to query.
`POSTHOG_QUERY_TIMEOUT_MS`	`postHogQueryConfig()`	HogQL timeout. Must be finite and `> 0`; default 30000.
`POSTHOG_HOST`	`analyticsConfig()` / `postHogQueryConfig()`	Ingest host. Query host is this with `.i.posthog.com` mapped to `.posthog.com` unless `POSTHOG_QUERY_HOST` is set.
`POSTHOG_QUERY_HOST`	`postHogQueryConfig()`	Optional HogQL API host. Default derived from `POSTHOG_HOST` (`https://us.posthog.com`).

Enablement after merge:

Apply infra-gcp so Cloud Run mounts ResendAPIKey and PostHogQueryAPIKey (PLACEHOLDER until populated) and so POSTHOG_PROJECT_ID=276289 is set. This apply creates {env}-PostHogQueryAPIKey and removes {env}-PostHogPersonalAPIKey.
Create matching Contact Property keys in the Resend dashboard. A rename here without a dashboard key is a silent no-op.
Populate ResendAPIKey and {env}-PostHogQueryAPIKey with gcloud secrets versions add (not Azure KV, not POSTHOG_API_KEY). Use a dedicated PostHog user scoped to query/read on project 276289.
Stub run: RESEND_CONTACT_SYNC_ENABLED=true with dry-run still on. Confirm phase timings and the 28 property keys.
Small live slice, then prod.

is_internal prefers PostHog cohort 410365. If that HogQL cohort query fails, the job logs a warning and falls back to @soundchecklive.io plus INTERNAL_EMAIL_ALLOWLIST in posthogQueryClient.ts. PostHog query timeout, a missing results array, or an empty person-property result set omits PostHog-derived keys for that run so a transient or empty HogQL response cannot overwrite existing Resend values; Clerk and Postgres writes still proceed. Lifecycle does not mark dormant from Clerk sign-in alone when PostHog is unavailable, so a transient query failure cannot overwrite an active contact as dormant. A failed Postgres fact query aborts the run before any Resend writes so a transient DB error cannot empty org, plan, or lifecycle properties. INCOMPLETE subscriptions are excluded from the live-subscription join, so those orgs get empty plan, status, and seats. Capacity assumes a worst-case two Resend requests per contact (PATCH then POST). Consecutive hourly runs advance the roster by that capacity so slices do not overlap. Raising RESEND_MAX_REQUESTS_PER_SECOND covers the full roster in one run without a code change.

Brevo signals

Marketing lifecycle email runs out of Brevo. api-events owns two write paths into it: a contact upsert at signup and nine tracked events that enter and exit Brevo automations (api-events/src/systems/brevo-signals/). Both go through api-events/src/clients/brevoClient.ts. Tracked events have their own switch (BREVO_TRACKED_EVENTS_ENABLED), so they can be turned off without stopping the signup sync. Prod sets both switches true and sends tracked events to the live Brevo account. Dev and qa set both true with BREVO_USE_STUB, so stub writes are logged and nothing reaches Brevo.

Env / secret	Config accessor	Purpose
`BREVO_SIGNALS_ENABLED`	`brevoConfig()`	Kill switch for every Brevo write. Default off.
`BREVO_TRACKED_EVENTS_ENABLED`	`brevoConfig().trackedEventsEnabled`	Separate switch for the nine tracked events; effective only when `BREVO_SIGNALS_ENABLED` is also on. Default off; set `true` in dev, qa and prod Terraform. While off, nothing is recorded (request paths, Stripe webhook, reconcile, lifecycle job), the relay publishes nothing, and a message already in flight is handed back to the relay.
`ORG_KNOWLEDGE_ENABLED`	`orgKnowledgeConfig()`	Server switch for the org knowledge base. Unset defaults on in local `development` and off in deployed environments; set `true` per environment to allow opt-ins and syncing. While off, opt-in returns 503, reads report `not_configured`, uploads stop, and opt-out purges still run. `ORG_KNOWLEDGE_WRITER=fixture` swaps the page writer for a deterministic offline one.
`SUPERMEMORY_API_KEY` / Encore `SupermemoryAPIKey`	`supermemoryConfig()`	Org knowledge base (PRO-416). Optional: unset or `PLACEHOLDER` makes the knowledge tools report `not_configured` and the `org-knowledge-sync` job a no-op. Use a key scoped to `soundcheck-org-{env}-*` containers. `SUPERMEMORY_CONTAINER_ENV` overrides the `dev`/`qa`/`prod` container prefix.
`BREVO_API_KEY` / Encore `BrevoAPIKey`	`brevoConfig()`	Contacts + Events API. Required when the gate is on and not stubbing. Use the non-expiring `soundcheck-app-prod` key, never the PostHog warehouse key (that one expires and returns a 401 that looks like an IP block).
`BREVO_BASE_URL`	`brevoConfig()`	Optional; defaults to `https://api.brevo.com/v3`
`BREVO_TIMEOUT_MS`	`brevoConfig()`	Per-request timeout. Must be finite and `> 0`; default 15000.
`BREVO_MAX_REQUESTS_PER_SECOND`	`brevoConfig()`	Client throttle. Default 5 (Brevo allows 10/s on `/v3/events`).
`BREVO_MAX_RETRIES`	`brevoConfig()`	Retries with `Retry-After` or exponential backoff: 429 on every call, 5xx only for contact upserts. Default 3.
`BREVO_USE_STUB`	`brevoConfig()`	Force stub writes even when dry-run is off.
`NOTIFICATIONS_DRY_RUN`	`dryRun()` / `brevoConfig()`	Writes stub unless this is exactly `false`.

The Clerk user.created webhook publishes a BrevoSignupContactRequested message to Pub/Sub (brevo-signup-contact) after the durable user steps succeed; a publish failure fails the webhook so Clerk redelivers. The upsert-brevo-signup-contact subscription owns the Brevo call: it upserts keyed on EXT_ID (Clerk user id) with updateEnabled, so a redelivery updates rather than duplicates, and it throws on a Brevo failure so Pub/Sub retries (5 attempts, then the DLQ). Attribute names are the uppercase Brevo contract in brevoAttributes.ts; Brevo silently drops unknown keys, so a typo produces an empty field rather than an error. Booleans are JSON true / false: the Brevo UI shows them as a Yes/No dropdown, but the API silently discards the strings "Yes" / "No" (every Sep 23 sign-up landed without IS_PAYING until this was fixed). Dates are YYYY-MM-DD in UTC. The webhook reads the web sign-up wizard’s answers from Clerk unsafe_metadata.onboarding. When that object exists (the buyer path), the upsert adds USER_TYPE=buyer, plus TEAM_SIZE and GIG_FREQUENCY when their values are in the Brevo category enums (any other value is dropped, because Brevo rejects it silently). A sign-up without the wizard (mobile crew, invite acceptance) sends no USER_TYPE. The crew events set it later. Consent lists 7, 8 and 9 (BREVO_SIGNUP_CONSENT_LIST_IDS) and IS_PAYING=false go only on a create. The client first posts /contacts without updateEnabled, with listIds. If Brevo answers 400 duplicate_parameter (the contact already exists: marketing import, earlier crew_invited, or a retry), it falls back to the plain updateEnabled upsert without lists or IS_PAYING. An existing contact is never re-subscribed, and never marked unpaid by a sign-up. Crew-invite contacts never get lists. A person invited before they sign up therefore stays off the lists until marketing adds them.

Tracked events

Event	Kind	Recipient (`identifiers.email_id`)	Recorded from	Dedupe key (arming period)
`org_setup_completed`	entry	org primary admin (OWNER, then ADMIN, earliest joined)	Clerk `organizationMembership.created` for an owner/admin, plus hourly reconcile: org created with an addressable owner/admin	`org_setup_completed:<orgId>` (once per org, ever)
`event_created`	exit	org primary admin	`createEventOperation`, plus reconcile for every other creation path; `TEMPLATE` rows (saved gig templates) never count, here or in the gig facts	`event_created:<eventId>` (every gig)
`crew_invited`	entry	invitee (user email, else invitation email)	gig invitation: bulk send after dispatch marks it `SENT`, plus reconcile (`invite_source=gig_invite`). Org invitation: Clerk `organizationInvitation.created` for a non-owner/admin role (`invite_source=org_invite`)	`crew_invited:<lowercased email>` (once per person, ever, across both sources)
`crew_invite_accepted`	gate	the person who accepted	Clerk `organizationInvitation.accepted` for a non-owner/admin role. Owners creating their own org and memberships added without an invitation emit nothing	`crew_invite_accepted:<Clerk invitation id>` (once per invitation)
`crew_profile_completed`	exit	crew member	onboarding runtime flipping `member_onboardings` to `COMPLETED`, plus reconcile (also admin `onboarded_at` overrides)	`crew_profile_completed:<orgId>:<userId>`
`subscription_started`	exit	org primary admin	Stripe `customer.subscription.created/updated`, plus reconcile	`subscription_started:<subscriptionId>`
`subscription_cancelled`	entry	org primary admin	Stripe `customer.subscription.updated` (cancel at period end) / `.deleted`, plus reconcile	`subscription_cancelled:<subscriptionId>`
`buyer_stalled`	entry	org primary admin	daily job: `org_setup_completed` 14+ days ago and zero gigs ever	`buyer_stalled:<orgId>` (once per org, ever)
`buyer_upgrade_candidate`	entry	org primary admin	daily job: 1+ gig ever, not paying, not comped	`buyer_upgrade_candidate:<orgId>:<latest subscription_cancelled id or initial>` (re-arms after a subscription starts and later cancels)
`buyer_went_quiet`	entry	org primary admin	daily job: 1+ booked gig, `LAST_GIG_DATE` 30+ days ago, not in active win-back	`buyer_went_quiet:<orgId>:<LAST_GIG_DATE>` (re-arms after a new gig goes quiet again)

Contact properties ride along through contact_properties: EXT_ID, ORG_ID, ORG_NAME, USER_TYPE=buyer on every event addressed to the org primary admin; EVENTS_BOOKED_COUNT, FIRST_GIG_DATE, LAST_GIG_DATE on event_created and the computed events; IS_PAYING (active or past-due subscription not cancelling at period end) on every org-addressed event, computed from the live subscription state when the event is sent (not when it is recorded), so a late delivery never writes a stale value; IS_PAYING=true, PLAN_TIER, MRR on subscription_started; IS_PAYING on subscription_cancelled (false unless another live paid plan remains). The crew events carry USER_TYPE=crew, FIRSTNAME / LASTNAME (the roster name in that org, then the name on the invitation) and EXT_ID when known. USER_TYPE=crew is skipped for anyone who is an active OWNER or ADMIN of any org (matched by user id or email), so a buyer is never downgraded. crew_invited and crew_profile_completed never carry SIGNUP_DATE: an invite is not a sign-up. crew_invited also carries INVITER_NAME (the gig creator’s or Clerk inviter’s roster name, then username, then org name). crew_invite_accepted fills in the contact the invite created: EXT_ID, FIRSTNAME / LASTNAME and SIGNUP_DATE from the Clerk account (names fall back to the roster), plus ORG_ID, ORG_NAME and LIFECYCLE_STAGE=crew_active unless the person is already a buyer, whose own org and lifecycle stay untouched. Its event properties carry invitation_id, invite_source and inviter_user_id when Clerk supplies one. The two org-invitation signals have no database fact to reconcile from, so the Clerk webhook is their durable trigger: a ledger write failure returns 500 and Svix redelivers. organizationInvitation.accepted looks the person up in Clerk by email (it can arrive before user.created) and answers 503 until Clerk returns them. Brevo’s “wait until event” step misses an event that fires before the contact reaches the wait, which is why acceptance also sets LIFECYCLE_STAGE=crew_active: an attribute the automation can check at any time. MRR is the plan price × seats per month (annual ÷ 12), computed with decimal.js. LAST_GIG_DATE is the latest eventDate of a non-cancelled gig. Active win-back means the org’s latest subscription_cancelled is newer than its latest subscription_started and it is not paying now. The computed events only consider orgs that already have a ledger row, so pre-launch orgs join the lifecycle when they next create a gig rather than all at once. Durable owner. Every signal is a row in brevo_signal_emissions (the ledger and the outbox). dedupe_key is unique and encodes the arming period, so an entry event is recorded at most once per period and an exit event once per occurrence; id is the UUID v5 of the key and is the Pub/Sub event id and inbox key. Producers insert with ON CONFLICT DO NOTHING.

Request paths (gig create, invitation send, onboarding completion) record best-effort after their own write commits: a ledger failure is captured to Sentry and never fails the user’s request. The hourly brevo-signal-reconcile job is the recovery owner: it rescans the committed domain rows from the last three days (never before BREVO_SIGNALS_EPOCH_ISO, 2026-09-23) and records anything a request path missed, including creation paths with no hook.
The Stripe webhook records synchronously and fails closed (500, Stripe redelivers) if the ledger write fails, like the Clerk signup publish.
brevo-signal-relay (every 5 min) publishes unpublished rows to brevo-tracked-event and stamps published_at only after Pub/Sub accepts.
track-brevo-event checks the brevo_tracked_event inbox, then takes a 90s lease (brevo_tracked_event_claim) before loading the row. Brevo events have no idempotency key, so a delivery that finds the lease held throws and retries later instead of posting a second copy. If Brevo accepted the event but the terminal write failed, the lease is sealed, and the next delivery finishes the write without posting again. Each send is aborted after 60s, so the lease only lapses once the previous attempt is over (finished, aborted, or its worker died). A delivery that reclaims a lapsed lease sends again, which errs toward a duplicate over a lost event. The processor loads the row, posts /v3/events, then stamps delivered_at and seals the inbox in one transaction. A Brevo 400/404/422 is a permanent rejection (rejected_at, sealed, captured). Anything else throws so Pub/Sub retries 5 times, then dead-letters. If the kill switch is off when a message arrives, the row’s published_at is cleared so the relay resends it once the switch is back on.
brevo-lifecycle-signals runs daily at 15:00 UTC and records the computed events.

Turning BREVO_TRACKED_EVENTS_ENABLED on after a period off does not replay the gap: the reconcile window is max(BREVO_SIGNALS_EPOCH_ISO, now − 3 days), so the first run records direct signals for domain rows from the last three days only (gigs created, invitations sent, onboardings completed, org setups, subscription changes). subscription_started additionally needs the subscription itself to be created on or after the epoch. The computed buyer events start from orgs that have a ledger row, so they begin with those recent orgs. Anything older than three days at enable time is not sent. Brevo has no idempotency key on /v3/events, so the inbox narrows but cannot close the duplicate window: a crash after Brevo accepts and before the inbox commit, or two overlapping deliveries of one message, can repeat an event. The client does not retry a 5xx on /events for that reason. Useful checks: unfinished work is SELECT signal, count(*) FROM brevo_signal_emissions WHERE delivered_at IS NULL AND rejected_at IS NULL GROUP BY 1; a relay backlog is rows with published_at IS NULL older than ten minutes. Entry/exit coverage audit. org_setup_completed (entry, once per org) and event_created (exit, every gig) gate the buyer sequences. Run api-events/scripts/brevo/audit-signal-coverage.sql. It is read-only and wrapped in BEGIN TRANSACTION READ ONLY … ROLLBACK:

psql "$DATABASE_URL" -v since="'2026-09-23T00:00:00Z'" -f api-events/scripts/brevo/audit-signal-coverage.sql

It classifies every org and non-template gig created since since as recorded_once, DUPLICATE, missing_no_owner_or_admin (expected: nobody to address), pending_next_reconcile (under 65 minutes old), MISSING_outside_reconcile_lookback, or MISSING. It lists the unrecorded ids and reports ledger delivery state per signal. A healthy result has no DUPLICATE, MISSING, or stuck rows. The real-Postgres suite brevoSignalReliability.integration.test.ts pins both guarantees. It checks re-saves, overlapping sweeps, and a late admin for org_setup_completed, and 450 gigs across batches plus the inline path for event_created.

DLQ replay

Every lane’s DLQ topic has a Terraform-managed pull subscription named <env>-<subscription>-dlq (7-day retention, never expires, no dead-letter policy of its own, output dlq_subscription_names). It only holds messages dead-lettered after it was created; older ones are reachable through the DLQ topic’s 14-day retention with a temporary subscription and seek.

Lane	DLQ subscription	Republish to
Signup contact	`<env>-upsert-brevo-signup-contact-dlq`	`<env>.marketing.brevo_signup_contact.v1`
Tracked event	`<env>-track-brevo-event-dlq`	`<env>.marketing.brevo_tracked_event.v1`

A dead-lettered signup message is the only durable copy of that signal. A dead-lettered tracked event still has its ledger row (published_at set, delivered_at and rejected_at NULL). On a page, fix the cause first (wrong or expired key, Authorized IPs, an attribute name that no longer exists), then:

ENV=prod PROJECT=soundcheck-live   # dev-soundcheck-live / qa-soundcheck-live
SUB="$ENV-track-brevo-event-dlq"    # or $ENV-upsert-brevo-signup-contact-dlq
TOPIC="$ENV.marketing.brevo_tracked_event.v1"  # or $ENV.marketing.brevo_signup_contact.v1

gcloud pubsub subscriptions pull "$SUB" --project="$PROJECT" --limit=50 --format=json > dlq.json
jq -r '.[].message.data' dlq.json | while read -r data; do
  gcloud pubsub topics publish "$TOPIC" --project="$PROJECT" \
    --message="$(printf '%s' "$data" | base64 --decode)"
done
gcloud pubsub subscriptions ack "$SUB" --project="$PROJECT" \
  --ack-ids="$(jq -r '[.[].ackId] | join(",")' dlq.json)"

Republish only the message data. Drop the CloudPubSubDeadLetter* attributes Pub/Sub adds on dead-lettering; the processors read nothing but the JSON body. Pull without --auto-ack and ack only after every publish succeeds; unacked messages reappear after the 60-second ack deadline. Replay is safe: both processors dedupe on event_id, the signup upsert is idempotent on EXT_ID, and a tracked event already delivered is sealed without calling Brevo. For a tracked event whose DLQ copy has aged out, UPDATE brevo_signal_emissions SET published_at = NULL WHERE id = '<id>' AND delivered_at IS NULL AND rejected_at IS NULL hands it back to the relay. The client retries a 429 on every call. It retries a 5xx only for contact upserts: a tracked event has no idempotency key, and a duplicate re-enters a Brevo automation, so an ambiguous 5xx on /events surfaces to the caller. Enablement after merge:

Apply infra-gcp so Cloud Run mounts BrevoAPIKey (PLACEHOLDER until populated) and the marketing.brevo_signup_contact.v1 and marketing.brevo_tracked_event.v1 topics, subscriptions, DLQs, DLQ pull subscriptions, and the three maintenance.brevo-* scheduled jobs exist. The brevo_signal_emissions migration applies on the next api-events boot.
Populate {env}-BrevoAPIKey with gcloud secrets versions add using the soundcheck-app-prod key from the Brevo account.
Brevo’s Authorized IPs: either keep the feature off, or allowlist the environment’s Cloud NAT egress address. Prod egress is pinned through Cloud NAT; read it from the Terraform output cloud_run_nat_ip (terraform -chdir=infra-gcp/environments/prod output cloud_run_nat_ip). A blocked IP surfaces as a 401.
Stub run: BREVO_SIGNALS_ENABLED=true and BREVO_TRACKED_EVENTS_ENABLED=true (set it in extra_cloud_run_env beside BREVO_SIGNALS_ENABLED) with dry-run still on. Confirm the STUB BREVO CONTACT UPSERT log line on a test signup and STUB BREVO EVENT lines after creating a gig (within five minutes, once the relay runs).
Live on dev with one test signup and one gig. Check the contact in Brevo carries EXT_ID, SIGNUP_DATE, and names, and that org_setup_completed / event_created appear on its activity; then qa, then prod.

Resend webhook

Env / secret	Config accessor	Purpose
`RESEND_WEBHOOK_SECRET`	`resendWebhookConfig()`	Optional at startup. Svix verification for `POST /webhooks/resend`; empty or placeholder → 503 (route inert).

Subscribes to Resend email lifecycle events (delivered, clicked, bounced, complained, suppressed, failed). The handler verifies the Svix signature, publishes a ResendEmailLifecycleEvent to Pub/Sub (resend-email-lifecycle), and marks ingress dedupe (resend_webhook) only after publish succeeds. A subscription processor (capture-resend-email-analytics) claims with a 20s lease (not the default 5m) so a crashed in-flight owner is reclaimable before five Pub/Sub deliveries exhaust, then awaits PostHog capture and marks resend_email_analytics on success. Capture is subscription-only so a killed request cannot hold the inbox claim after 200. Programmatic sends should attach identity tags (clerk_user_id, org_id) so captures can resolve distinct_id. Missing identity tags still ack 200 but skip analytics. GCP Secret Manager provisions ResendWebhookSecret (mounted as RESEND_WEBHOOK_SECRET). It is not copied from Azure Key Vault — after Terraform apply, replace the PLACEHOLDER version with gcloud secrets versions add. Cutover (register the Resend webhook URL and event types) is tracked in ENG-570.

Stripe webhook

Env / secret	Config accessor	Purpose
`STRIPE_WEBHOOK_SECRET`	`stripeConfig()`	Verifies `POST /webhooks/stripe`
`STRIPE_*` (API keys)	`stripeConfig()`	Connect, checkout, payouts

Clerk

Env / secret	Config accessor	Purpose
`CLERK_SECRET_KEY`	`clerkConfig()`	JWT verification
`CLERK_WEBHOOK_SECRET`	`clerkWebhookConfig()`	Svix verification for Clerk webhooks
`CLERK_API_BASE_URL`	`clerkConfig()`	Local emulator only (`:4711`); rejected in QA / production

Portal Clerk `public_metadata` binding

Portal invitations bind the contact and org via Clerk public_metadata:

portal_contact_id
portal_organization_id

Provision middleware skips /api/v1/portal/**. Portal scope is resolved server-side from that metadata — never from the request body or query.

Caller auth

Clerk session JWTs are the only caller auth. ENABLE_BASIC_AUTH=true fails closed. E2e fixtures mint emulator session tokens so JWT sub matches Postgres User.id. /api/v1/admin/** is parked and 404s every authenticated request. Optional WHITE_LABEL_ENFORCE_ENTITLEMENT=true still surfaces on entitlement bundles if that console is revived later.

Plan entitlements

entitlements/registry.json says which plan tier each paid feature needs; src/generated/entitlements.ts is generated from it. Check a feature with requireEntitlement(orgId, key) from src/systems/entitlements/entitlements.ts. It resolves the org’s tier from the newest active, trialing or past-due subscription, then a live comp, and caches it for 10 seconds (at most 5,000 orgs).

A key marked enforced answers 403 PLAN_UPGRADE_REQUIRED below its tier.
A key marked shadow lets the call through. It logs entitlement: would deny and sends entitlement_rejected to PostHog at most once an hour per org and key.
ENTITLEMENTS_ENFORCE=off turns every key into shadow mode.
GET /api/v1/organizations/:orgId/entitlements returns the tier, seat limits, and each key’s allowed and enforced state to any org member.

Org-scoped routes are checked by entitlementMiddleware, from the route table in src/systems/entitlements/entitlementRoutes.ts (first match wins). Every route starts in shadow mode, whatever its key’s enforcement says, because web, mobile and MCP surfaces that were never plan-gated call the same routes. A route blocks only once its rule sets enforce: true, and then only for org members: a non-member falls through to the handler’s own 403 so the org’s plan is not revealed. Crew self-service routes (offers, availability requests, own availability, /onboarding/me), Assets, data exports, disconnects, OAuth callbacks and status reads are deliberately left out; entitlementRoutes.test.ts lists them. Each member MCP tool call checks mcp-access for the org its input names, falling back to the token’s active org. Tools that take only a resource id (such as an event id) are attributed to the token’s org, so move the check into the operation-level org authorization before enforcing mcp-access. A row in organization_entitlement_grants gives one org an entitlement outside its tier (grandfathering, a sales deal, a support exception); plan checks and the entitlements endpoint count it. Staff manage grants with scripts/entitlements/grants.ts. Before switching a key with a grandfatherFromTier to enforced, run --grandfather <key> --before <ISO> (dry run) and then add --apply. --before is when the key moved up a tier. Only orgs created before it, that held grandfatherFromTier or above at --before (per the audited subscription and comp history) and are still on it, get a grant. A cutoff older than the audit history is refused. White-label and the dedicated e-sign identity still pass unless WHITE_LABEL_ENFORCE_ENTITLEMENT=true.

Twilio and event chats

Env / secret	Config accessor	Purpose
`TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN`	`twilioConfig()`	Programmable Messaging + Conversations webhook signatures
`TWILIO_FROM_NUMBER`	`twilioConfig()`	Invitation / reminder SMS sender — not a chat proxy
`TWILIO_USE_STUB_SMS`	`twilioConfig()`	Stub invitation SMS locally
`TWILIO_API_KEY_SID` / `TWILIO_API_KEY_SECRET`	`eventChatsConfig()`	Conversations REST + AccessToken JWTs (`SK…`)
`TWILIO_CONVERSATIONS_SERVICE_SID`	`eventChatsConfig()`	Dedicated Conversations Service (`IS…`)
`TWILIO_GROUPCHAT_NUMBERS`	`eventChatsConfig()`	SMS proxy pool (2–4 E.164s; must not overlap `TWILIO_FROM_NUMBER`)
`TWILIO_WEBHOOK_URL`	`twilioConfig()`	Programmable Messaging webhook URL used to validate inbound invitation/reminder SMS callbacks; distinct from the Conversations webhook URL
`TWILIO_CONVERSATIONS_WEBHOOK_URL`	env	Exact URL Twilio signed for `POST /webhooks/twilio/conversations`
`TWILIO_SMS_STATUS_WEBHOOK_URL`	`twilioConfig()`	`StatusCallback` sent with each outbound SMS, and the URL Twilio signed for `POST /webhooks/twilio/sms-status`. Unset: no callbacks
`TWILIO_CONVERSATIONS_ENDPOINT`	`eventChatsConfig()`	Emulator base URL only — unset in deployed environments

Launch steps: event chats runbook.

Platform admin (service console)

Soundcheck staff identity is a Clerk user public_metadata flag, not an org role and not the parked X-Staff-Actor allowlist:

Grant or revoke: set public_metadata.platformAdmin to true or false on the Clerk user. Users cannot write this themselves. A platform admin can do it from /service-console for one existing Clerk user whose email domain is exactly soundchecklive.io. The Dashboard and Backend API remain available.
Session token: include user public metadata on the JWT. Clerk’s default claim name is metadata; the web middleware also accepts public_metadata.
Web: /service-console is gated by that flag. Org admin / owner is not enough. Local emulator: Avery Owner is seeded with the flag.
API: authorize with isPlatformAdminFromClaims on Clerk-verified session claims (api-events/src/auth/platformAdmin.ts). isPlatformAdminFromJwt only decodes an already-verified token and is not an authentication gate. Do not reintroduce Basic Auth.

Public URLs

Env / secret	Config accessor	Purpose
`API_BASE_URL`	`publicUrlConfig()`	Public API origin (default `http://localhost:4000`)
`WEB_APP_URL`	`publicUrlConfig()`	Portal invite links
`EVENT_DETAILS_URL`	`publicUrlConfig()`	Event deep links

Bank-account encryption

Env / secret	Config accessor	Purpose
`CREDENTIAL_ENCRYPTION_KEY`	`bankAccountConfig()`	Base64 AES-256 key shared with QuickBooks token crypto
`CREDENTIAL_ENCRYPTION_DISABLED`	`bankAccountConfig()`	Opt-out (forbidden in deployed environments when storage needs a key)
`BANK_ACCOUNT_STORAGE_DISABLED`	`bankAccountConfig()`	Disables org/member bank-account HTTP; admin verification routes ignore this flag

Admin bank-account routes stay available even when storage is disabled. Org funding for ACH transfer paths expects a Cleo-VERIFIED funding account with a Cleo funding-source id. The member must have a Cleo-ready contact. Queueing a direct deposit persists an immutable ACH instruction on the transaction (source, destination, Cleo contact, and funding-source ids) and publishes cleo.payment.requested as a wake-up. The processor claims a 60s submit lease and creates the Cleo payment from those stored fields only. Cleo sweeps require the frozen instruction columns so they cannot fail legacy/manual ACH. A failed publish fails the PENDING/PROCESSING pair so the payable is not left in flight. Settlement to PAID is still webhook-driven and GET-authoritative; a terminal webhook whose GET is still processing is retried. Bank-account delete returns 409 while any in-flight ACH references the row, then marks REMOVAL_PENDING and revokes the Cleo resource. Owner lists exclude removed/removal rows.

Feature-flag gated routes (server-side PostHog evaluation)

api-events evaluates PostHog feature flags server-side (src/systems/shared/featureFlags.ts, posthog-node, remote evaluation, 60 s cache, fail-closed). A gated route or job uses the same flag key the web uses, so rollout lives in one place. Do not add per-org env allowlists.

Env / flag	Where	Purpose
`POSTHOG_API_KEY`, `POSTHOG_HOST`	Cloud Run (`enable_posthog`)	Project token + host used for evaluation; without them every gated route fails closed (404) and gated jobs stay off
`FEATURE_FLAG_OVERRIDES`	local dev / test only	Comma list of `key` or `key=true\|false`; ignored when deployed (`K_SERVICE`) or outside development/test
`POSTHOG_FLAGS_LOCAL_EVALUATION`	optional	`true` switches to local evaluation using the query key
`crew-offer-unclaimed-dispatch`	PostHog flag (per org)	Allows crew-offer and availability-request email to imported (“unclaimed”) roster members who never signed up

crew-offer-unclaimed-dispatch replaces the retired CREW_OFFER_UNCLAIMED_DISPATCH_ORGS allowlist and is evaluated per organization from the dispatchers (no caller). It is off everywhere by default: an org can import every freelancer it ever hired, and those people never opted in, so one “send to all” would become hundreds of first-contact emails from a domain that never mailed them. Target an org only once its first-contact template and unsubscribe are in place. SMS to unclaimed recipients stays suppressed regardless of the flag (TCPA exposure is worse than CAN-SPAM).

Gig workbook export (West Coast Music)

GET /api/v1/organizations/:orgId/events/:eventId/wcm-workbook.xlsx fills the customer’s own Excel binder (13 tabs, curated template at api-events/src/systems/gig-workbook/templates/) from a gig. It is a per-customer surface, gated by one PostHog flag on both sides:

Env / flag	Where	Purpose
`wcm-gig-workbook`	PostHog flag (web + api-events)	Shows the “Event workbook (Excel)” row in the gig Export hub, and gates the endpoint itself — api-events evaluates the same flag server-side for the caller (`distinctId` = Clerk user id, person properties `email` / `active_organization_id` / `organization_ids`). Flag off, PostHog unreachable, or PostHog unconfigured ⇒ 404, so the surface never leaks
`FEATURE_FLAG_OVERRIDES`	api-events env	Local/e2e escape hatch: comma list of `key` or `key=true\|false`, honoured only when `ENVIRONMENT` is `development` (non-Cloud-Run) or `test`. `dev-emulator.mjs` and the encore-e2e job set `wcm-gig-workbook`

Callers need the gig-manager role; prices, deposits, overtime rate and payroll cells are written only for callers with financial clearance. The template recomputes its contract, invoice and details tabs on open (fullCalcOnLoad); re-curate it with scripts/gig-workbook/curateWcmTemplate.ts when the customer sends a new master copy.

Apple Music song import

Mobile and web search the Apple Music catalog, or paste a song, album or playlist link, to create library songs. The system lives in api-events/src/systems/music/; provider HTTP and developer-token signing are in api-events/src/clients/appleMusicClient.ts. Imports are metadata only: Apple Music audio is DRM-protected and is never downloaded or stored. Responses carry title, artist, album, duration, ISRC, artwork and the catalog link for the picker; a song created from a track keeps its title, artist and duration, with the catalog link as a media link. ISRC, artwork and ids are display and duplicate-check data and are not persisted.

Endpoint	Purpose
`GET /api/v1/organizations/{orgId}/music/search?q=`	Song search in one catalog. `provider` is `apple_music` (default), `spotify` or `soundcloud`; `limit` 1-25 (default 10), `storefront` two-letter country (default `us`). Clients search every catalog from `/providers` in parallel.
`POST /api/v1/organizations/{orgId}/music/match`	Matches a pasted set list (`lines`, up to 50) across every available catalog. Each line returns the title and artist read from it, `status` (`matched`, `check`, `none`), up to 3 candidates merged across catalogs (same ISRC, or same title and artist) with a link per catalog, and `library_song_id` when the library already has the song. Lines run four at a time; a line fails only when every catalog fails, and the request fails only when every line does.
`GET /api/v1/organizations/{orgId}/music/providers`	The catalogs this organization can search now, in display order (`{ search: [{ id, label }] }`).
`GET /api/v1/organizations/{orgId}/music/resolve?url=`	Resolves an Apple Music song, album or playlist link to its tracks (at most 300; `truncated` says when there were more).

Both routes need the caller to manage gigs in the organization (gig.manage), the same bar as creating songs. Neither writes anything: the client creates songs through the existing POST /api/v1/songs. Links are parsed, never fetched; only https://music.apple.com and https://embed.music.apple.com paths are accepted and only the storefront and catalog id are used. Configuration is three secrets, all from one Apple Developer MusicKit key: APPLE_MUSIC_TEAM_ID, APPLE_MUSIC_KEY_ID and APPLE_MUSIC_PRIVATE_KEY (the full .p8 file; literal \n sequences are accepted). Terraform: enable_apple_music_secrets creates the three PLACEHOLDER-seeded secrets (infra-gcp/README.md). There is no rollout flag: with any of the three unset or still PLACEHOLDER, both routes answer 503 MUSIC_PROVIDER_UNAVAILABLE and nothing leaves the service. The client signs an ES256 developer token (raw 64-byte signature, one hour lifetime) and reuses it until five minutes before expiry. A rejected token or an Apple outage also answers 503 and is reported to Sentry as music.search or music.resolve; Apple’s 429 answers 503 MUSIC_PROVIDER_BUSY without an alert. APPLE_MUSIC_API_BASE_URL points the client at the emulator (loopback or emulators only, ignored in QA and production).

Spotify and SoundCloud search

Each extra catalog has its own client in api-events/src/clients/ (spotifyClient.ts, soundCloudClient.ts, sharing musicProviderHttp.ts) and joins /providers only once its secrets hold real values. Terraform: enable_music_search_secrets creates SPOTIFY_CLIENT_ID, SPOTIFY_CLIENT_SECRET, SOUNDCLOUD_CLIENT_ID and SOUNDCLOUD_CLIENT_SECRET, PLACEHOLDER-seeded. A catalog without credentials answers 503 MUSIC_PROVIDER_UNAVAILABLE and nothing leaves the service.

Spotify: client-credentials token (cached until a minute before expiry, refreshed once on a 401), /v1/search?type=track. Pasted Spotify track and album links (and spotify: URIs) resolve through the same client.
SoundCloud: OAuth 2.1 client-credentials token from secure.soundcloud.com, /tracks?q= limited to playable and preview tracks. Needs an approved SoundCloud API app.

A provider 429 answers 503 MUSIC_PROVIDER_BUSY without an alert; other provider failures answer 503 and report to Sentry with the provider name. SPOTIFY_API_BASE_URL, SPOTIFY_ACCOUNTS_BASE_URL, SOUNDCLOUD_API_BASE_URL and SOUNDCLOUD_AUTH_BASE_URL point a client at an emulator locally (ignored in QA and production). Search, resolve and match results carry library_song_id when the organization’s song library already has the song. The check compares title and artist ignoring case, accents and punctuation, so a live or remastered version is a different song; a library song with no artist matches any artist. The library’s song titles (at most 10,000) are read once per request. Smoke test after populating the secrets: GET /api/v1/organizations/{orgId}/music/search?q=dreams as a gig manager returns tracks with duration_seconds; the same call as a crew member returns 403.

Forwarded confirmation intake (MAR-39)

The signed Inbound Parse endpoint is POST /webhooks/forwarded-email. It is separate from outbound organization sending identities and the conversational CheckAI email channel. Sending-domain verification CNAMEs and Clerk team-domain enrollment do not provision this receiving host or authenticate a forwarded message. Configure FORWARDED_EMAIL_ENABLED=true, FORWARDED_EMAIL_DOMAIN (a dedicated inbound subdomain with SendGrid MX/Parse routing), and INBOUND_EMAIL_VERIFICATION_KEY or Encore secret InboundEmailVerificationKey (the Parse signature public key). Enable signed parsed multipart delivery, not raw MIME mode. The settings address is hidden until all three configuration values are present. Keep the existing ingestion-recovery scheduled job and structured AI provider configured; it owns retry and source cleanup. The feature defaults off. The self-hosted dev, qa, and prod infra files declare an empty InboundEmailVerificationKey fallback so disabled deployments can build without provisioning a receiving key. Set INBOUND_EMAIL_VERIFICATION_KEY at runtime before enabling forwarding; the environment value takes precedence over that empty fallback. The recipient alias selects the organization. Aligned sender authentication and exactly one active member are required for extraction; a verified organizational sending domain never bypasses either check. Disconnecting or changing the organization’s From domain leaves the inbound alias unchanged. No sender replies are generated by this workflow. Acknowledgement follows durable receipt storage. Sources expire after 30 days; imported/discarded history and reviewed proposals remain. General audit snapshots exclude the source ledger to avoid duplicating private email content. Deployment verification must include one signed member forward, one held unknown sender, an identical redelivery, manager approval, and inspection of the resulting native gig records. Source cleanup continues while forwarding is disabled. Review responses include source text and attachment filenames; attachment bytes remain available only to extraction. Each extraction has an eight-minute deadline within a ten-minute claim lease, and a stale completion cannot publish a proposal. A later confirmation updates matching flight legs. A new route for an existing passenger booking requires resolving the itinerary on the gig before import; a fresh multi-leg booking can be approved together. Contract corrections replace their marked section in admin notes and preserve surrounding notes. If that section has been removed or its markers changed, approval stops for manual resolution. Non-flight confirmations match by organization, record type, and case-insensitive booking reference. Changing a title, passenger detail, record order, or selected gig cannot create a second mapping for that reference; selecting another gig requires resolving the existing booking first. Records without a confirmation reference remain specific to their receipt and row, since there is no reliable identity for matching separate emails. Repeated references of the same type within one review are rejected as ambiguous. CSV extraction uses a preview of the header and at most 200 data rows, 40 columns, and 256 characters per cell. CSV attachments share a 64,000-byte preview budget plus filename and truncation notices. Reviewers should consult the original attachment when a preview omits information. CSV parsing scans at most the first 128 KiB and stops after 201 complete records, retaining quoted multiline cells while discarding an incomplete trailing record at the byte limit. PDF text conversion is attempted before binary analysis; binary PDFs are limited to 20 MiB and 100 pages across the request, images to 5 MiB each, and combined binary attachments to 20 MiB. Unsupported sizes stop with an actionable review error without automatic model retries. Flight legs are shared by airline, flight number, route, and departure instant. Passenger confirmation codes are stored on the passenger association, so separate bookings for the same scheduled flight do not create duplicate legs.

​Local provider emulators

​Organization sending domains (SendGrid)

​Cleo Pay

​Client forms

​n8n

​App file storage

​Azure Storage (until decommission)

​AI Gateway and event ingestion

​Mobile update-policy store verification (ENG-57)

​Operator-triggered publication (bootstrap)

​Resend contact sync

​Brevo signals

​Signup contact

​Tracked events

​DLQ replay

​Resend webhook

​Stripe webhook

​Clerk

​Portal Clerk public_metadata binding

​Caller auth

​Plan entitlements

​Twilio and event chats

​Platform admin (service console)

​Public URLs

​Bank-account encryption

​Feature-flag gated routes (server-side PostHog evaluation)

​Gig workbook export (West Coast Music)

​Apple Music song import

​Spotify and SoundCloud search

​Forwarded confirmation intake (MAR-39)