> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soundchecklive.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Event chats

> Production runbook for Twilio Conversations event-group chats and the SMS proxy pool

Operational runbook for **event group chats** in `api-events`. Twilio Conversations remains the live transport and SDK history source; Postgres holds chat metadata, membership, and a webhook-fed message mirror used by authorized REST and MCP reads. In-app members join with a Conversations AccessToken (`user_<id>`); phone-only crew are bridged over SMS through a dedicated proxy-number pool.

This page does **not** provision Twilio, A2P, or cloud resources. Configure those in the Twilio Console and secret stores, then point `api-events` at the existing SIDs.

Product behavior for end users: [Chat](https://docs.soundchecklive.io/features/chat/overview.md). Env inventory: [Integrations](https://docs.soundchecklive.io/engineering/backend/integrations.md) and [api-events operations](https://docs.soundchecklive.io/engineering/backend/api-events-operations.md). Event-chat e2e: [`e2e/README.md`](https://github.com/soundchecklive/soundcheck-mobile-app/blob/main/e2e/README.md#event-chats-twilio-emulator).

## What is shipped

| Surface | Behavior |
| - | - |
| Scope | Named group chats **on an event**. Managers create them; there is no auto-thread per gig. |
| In-app | Conversations SDK on web and mobile, token from `GET /api/v1/chat/token`. |
| SMS bridge | Unclaimed roster members with a usable E.164 join as SMS participants and get one welcome text from their proxy number. Replies reach the whole group. |
| Rollout | The `event-chats` PostHog flag (`flags/registry.json`). Authenticated routes evaluate it for the caller (email, memberships, active org), like web and mobile; background paths (webhook persistence, push, SMS welcomes) evaluate it for the chat's org, so the flag needs an org-property release condition for them to match. The server also stays closed until Twilio Conversations is configured (API Key SID/secret, `IS…` service, proxy pool, webhook URL; Terraform `PLACEHOLDER` counts as unset). Off for an org → its authenticated chat routes `404`; signed Conversations webhooks still persist provider state while push delivery and SMS welcomes are suppressed. |
| Token and membership | `GET /api/v1/chat/token` mints `user_<id>` when the flag is on for the caller in at least one org where they have an active chat. The Conversations grant covers the whole service, so access is limited by participant set instead: every minute the participant-sync job re-evaluates the flag (caller context) for the (org, user) pairs not checked in the last five minutes, 50 per page, for up to 20 seconds, records it in `event_chat_flag_gates`, and on a flip enqueues a participant sync for that user's chats in that org. `checked_at` moves only when a decision is written. An `unavailable` evaluation or a failed pair instead sets `retry_after` one minute out, so it is skipped (not blocking later pairs) and retried on the next run with its old `checked_at`; a run that hits the 20-second budget with pairs still due logs `event chat flag sweep ran out of time`. The reconciler then removes the in-app participant while the gate is suppressed and re-adds it when it clears; the membership row, SMS participants and history are untouched. Every other path that creates an in-app participant follows the same gate (`eventChatInAppGate.ts`): creating a chat, adding members, a manager joining, a claimed SMS member switching to in-app, and restoring access after a failed removal. Before taking the participant lock, an add re-evaluates the flag for that person (caller context) and records the result. If the evaluation is `unavailable` or fails and no gate row exists yet, the add records the pair as undecided: suppressed, with `checked_at` null, and queues participant syncs for that user's live chats in the org. The sweep then decides it first; on, it flips and re-adds; off, it stays as is. An existing row is kept as it is. Only add paths create undecided rows. For the reconciler and the sweep, a missing row still means allowed, so members from before #2492, who have no rows, are not mass-removed on deploy. Inside the lock it reads the stored gate. While suppressed, the membership row is created (or restored, or left on SMS) with no Twilio participant and is reported as added, so turning the flag back on re-adds it through the normal flip. After saving, every path that writes a `user_id` onto a CHAT row (a new member, or an SMS → CHAT claim) rechecks the gate in one transaction. It creates the row as undecided (suppressed) if missing, takes a `FOR SHARE` lock, and queues a participant sync if the row no longer matches what the add decided. So a flip in the window between the check and the save cannot miss the row, and a pair with no decision converges to removed until the sweep decides it. A flag change therefore reaches Twilio within about six minutes while each run finishes inside its budget; if that warning repeats, the population has outgrown the budget and the delay grows with it. An `unavailable` evaluation (PostHog down) changes no decision, so while PostHog stays down no removal or re-add can happen; it resumes on the first run after evaluations succeed. |
| Client flag | PostHog `event-chats` hides the UI. The API reads the same flag; there is no env toggle. |
| CheckAI | Identity `checkai` is added to every conversation but does not post in v1. In-thread AI is not shipped. |
| Mentions | Clients send `mentions: [{org_member_id, name}]` and `mentions_everyone: true` in the Twilio message attributes, next to readable `@Name` text in the body. The pre-send `onMessageAdd` hook validates them against the chat (current members only, never the author, `@everyone` only from `canManageGigOperations`) and returns an `attributes` override that rewrites only those two keys; it runs before the message exists, so it never races reactions or edits. A validation timeout fails closed on `@everyone` only. `onMessageAdded` re-validates and stores `mentioned_org_member_ids`, `mentions_everyone` and the Twilio `provider_index` on `event_chat_messages`. Push fan-out sets `isMention` per recipient from those columns and uses mention copy ("Maya mentioned you: …", "Maya to everyone: …"). `EventChat.my_last_mention_index` is the newest mention of the caller by someone else; clients compare it with their Twilio read horizon for the `@` badge. Edits never change mentions. |
| Per-chat notifications | Each member sets a level (All messages / Mentions only / Nothing) and an optional pause (one hour, or until the gig ends) per chat, stored in `event_chat_notification_prefs` keyed by (chat, org member). `GET/PUT /api/v1/chats/{chat_id}/notifications`, and every `EventChat` response carries the caller's `my_notifications`. A one-hour pause stores its instant; "until the gig ends" stores `GIG_END` and resolves at read and push time with `eventActionableUntil` in the gig's zone, so a reschedule moves it. `is_muted` is computed on the server and clients only read it. Push fan-out drops a recipient when `chatPushAllowed` says so: Nothing never pushes; Mentions only and a pause push only mentions. The org-level "Event chat messages" toggle is checked first, unchanged. SMS members have no per-chat mute yet. |

REST: `GET/POST /api/v1/events/{event_id}/chats`, `PATCH /api/v1/chats/{chat_id}`, `POST …/archive`, `POST …/join`, `POST …/leave`, `POST …/members`, `DELETE …/members/{org_member_id}`, `GET …/messages`, `GET …/messages/{message_id}`, `GET /api/v1/chat/token`. Listing is read-only (no Twilio, no membership insert). Managers join via `POST …/join` before minting a token. In-app members leave themselves via `POST …/leave`. Revocation removes the stored participant and checks the claimed user’s Twilio identity as well, so an interrupted SMS-to-app migration cannot leave an unrecorded participant behind. A failed identity lookup keeps the local membership available for retry. Webhook: `POST /webhooks/twilio/conversations`.

## Dedicated Conversations Service

Create a **Conversations Service** used only by Soundcheck event chats (`IS…`). Put that SID in `TWILIO_CONVERSATIONS_SERVICE_SID`.

Do **not**:

* Reuse the default / shared Conversations Service that other products already write to.
* Point this SID at a Messaging Service. Conversations and Programmable Messaging are different products.
* Set `TWILIO_CONVERSATIONS_ENDPOINT` in a deployed environment. That override is emulator-only (`http://localhost:4013/conversations` for `tools/twilio-emulator`).

Every REST call is scoped to `/v1/Services/{TWILIO_CONVERSATIONS_SERVICE_SID}/…`. AccessToken ChatGrants must use the same SID or the SDK cannot open the conversation.

## Canonical pre-event and post-event webhooks

Configure **service-level** webhooks on that Conversations Service (Twilio Console → Conversations → Services → the dedicated `IS…` → Webhooks). Do not configure these on individual conversations, on phone numbers, or on a Messaging Service.

| Twilio field | Canonical value |
| - | - |
| Pre-event webhook URL | `{API_BASE_URL}/webhooks/twilio/conversations` |
| Pre-event method | `POST` |
| Pre-event filters | `onMessageAdd` (optional: `onParticipantAdd`) |
| Post-event webhook URL | `{API_BASE_URL}/webhooks/twilio/conversations` |
| Post-event method | `POST` |
| Post-event filters | `onMessageAdded`, `onMessageUpdated`, `onMessageRemoved` |

`api-events` verifies `X-Twilio-Signature` with the **account auth token** against `TWILIO_CONVERSATIONS_WEBHOOK_URL` (not `TWILIO_WEBHOOK_URL`, which is the invitation SMS route and will always fail signature checks here). Set `TWILIO_CONVERSATIONS_WEBHOOK_URL` to the exact public URL Twilio POSTs, including scheme and path. A mismatch is `403` / `INVALID_SIGNATURE`.

The handler prefixes SMS bodies on `EventType=onMessageAdd` with Twilio's JSON `{"body":…}` override. Post-event `onMessageAdded`, `onMessageUpdated`, and `onMessageRemoved` persistence and topic publication are awaited before HTTP `200`; a failure returns non-2xx so Twilio retries. Creates also bump `last_message_at` and publish `EventChatMessageTopic` with a deterministic event ID derived from message ID, action, and citation version. Created events always use citation version 1 so a late `onMessageAdded` replay after edits keeps the same ProcessedEvent id; updated and deleted ids still include the current citation version. A remove that arrives before add writes a `DELETED` tombstone keyed by `MessageSid`, so the delayed add cannot resurrect it. `eventChatMessageProcessor` consumes created messages with a ProcessedEvent inbox, suppresses CheckAI and the author, honors the `event_chat` push preference, and batches Expo delivery. Update and remove events are terminally consumed without push. Other event types are acknowledged `200` and ignored.

If an SMS author posts a STOP-family control word (`STOP`, `STOPALL`, `UNSUBSCRIBE`, `CANCEL`, `END`, `QUIT`, including the `Name: STOP` prefix rewrite), the pre-action hook removes that Twilio participant and the local membership, then returns HTTP 403 so the control message is rejected. If the pre-action hook times out and Twilio accepts the message, the post-action hook attempts membership revocation first, then provider-message deletion. A delete failure does not skip opt-out; either failure returns non-2xx so Twilio retries. Retries stay safe once membership is already gone (`already_gone`). Neither path persists or pushes the control message. Carrier-level suppression can still swallow some STOP texts before they reach this webhook — treat this as a defensive complement, not a guarantee that every opt-out is visible to `api-events`.

If an SMS author sends exactly `HELP` (trimmed, any case), the pre-action hook queues a help reply and returns HTTP 403, so the group never sees the message. The reply goes only to that phone, from its bound proxy number, through Programmable Messaging:

```text theme={null}
Soundcheck gig chat help: info@soundchecklive.io. Reply STOP to leave. Msg & data rates may apply.
```

The reply is a durable SMS notice (see below). If queueing it fails, the pre-action hook returns non-2xx instead of 403, so Twilio accepts the message and the post-action fallback handles it. If the pre-action hook times out or fails and Twilio accepts the message, the post-action hook queues the reply keyed by `MessageSid`, then deletes the message from the conversation, and skips persist and push. Either failure returns non-2xx so Twilio retries; the `MessageSid` key keeps retries to one reply. Neither path sends the SMS before answering Twilio: the queued reply is attempted only after the webhook response is written, and the minute maintenance job sends it if that attempt does not run or fails.

Pre-action events carry no `MessageSid`, so one text can reach both paths under different keys: the pre-action hook queued a reply but answered Twilio too late, or the two ran concurrently. Both paths take a per-member transaction advisory lock (`event-chat-sms-help:{member}`) around check-and-insert, and treat a pre-action row and an accepted row as the same text when the pre-action row's `created_at` is within `SMS_HELP_SAME_MESSAGE_WINDOW_SECONDS` (15 s) of the accepted message's own `DateCreated`. Whichever path commits second queues nothing. The window covers Twilio's pre-action timeout plus clock skew between Twilio and the database; separate HELP texts further apart each get a reply. Two HELP texts from one member within 15 s of each other, where the first was rejected and the second accepted, get one reply. HELP handling does not depend on the `event-chats` flag or on the chat being active.

Twilio may also answer HELP itself (default keyword handling or Advanced Opt-Out on the Messaging Service). Turn that off or set it to the same text on the chat-only Messaging Service, or the sender gets two replies.

## SMS notices: welcome and HELP

Carriers require the first text a recipient gets on an A2P campaign to identify the sender and program and to give opt-out instructions. When a member is added on the SMS channel, `api-events` sends that phone one welcome from its bound proxy number:

```text theme={null}
Soundcheck: {org} added you to the "{chat}" chat for {event}. Replies go to the group. Reply STOP to leave, HELP for help.
```

Names are reduced to the GSM-7 basic character set. Extension-table characters (`^ { } [ ] ~ | \ €`), the backtick, curly quotes, and dashes are mapped to basic equivalents; accents outside GSM-7 are stripped; anything else is dropped. Each name is then truncated to 40 septets. The fixed text is 103 septets, so the welcome is at most 223 septets: always GSM-7, one segment for typical names, never more than two (306). A blank organization, chat, or event name falls back to generic wording. The HELP reply and the disclosure suffix below are fixed GSM-7 text.

Welcomes and HELP replies go through Programmable Messaging (`sendSMS` with an explicit `From`), not a Conversations message, so in-app members never see them. They are only ever sent from a number in `TWILIO_GROUPCHAT_NUMBERS` that is not `TWILIO_FROM_NUMBER`; any other stored proxy abandons the notice instead of falling back to the invitation sender.

### Welcome before the first group message

* Each SMS member's welcome is attempted as soon as that member's add releases the chat locks, before the next member in the same request is added.
* While an SMS member is joining or their welcome is unsent, the pre-action `onMessageAdd` hook appends ` (Soundcheck gig chat. Reply STOP to leave, HELP for help)` to every message posted in that chat, whatever the author. A media-only message with no text gets the disclosure alone, without the leading space. It uses the same `{"body":…}` override as the SMS author prefix. So a group message that beats the welcome still carries the program name and STOP/HELP instructions.
* "Joining" is an `event_chat_sms_join_intents` row keyed by chat and org member. It is committed before the Twilio participant is bound, so it also covers the window between the bind and the membership commit.
  * The membership commit (or the orphan-recovery commit) deletes it in the same transaction that queues the welcome, and the welcome effect takes over.
  * A definitive bind failure, an exhausted pool, or an activation failure whose Twilio participant rollback succeeded deletes it. If the intent can't be written, the member add fails before any Twilio call.
  * An ambiguous bind, or an add that ends in queued removal, leaves it in place: the participant may still exist.
  * Backstop: an intent older than `SMS_JOIN_INTENT_TTL_MINUTES` (10) no longer counts, and the minute maintenance job deletes it.
* The check is one `EXISTS … OR EXISTS …` query per inbound message, keyed by conversation SID through the unique index, and it runs in parallel with the prefix lookup under the same 100 ms budget. It fails closed: if the lookup errors or times out, the suffix is appended anyway. Dead-lettered or settled welcomes stop the suffix.

### Durability

* Both kinds are `event_chat_sms_notice` rows in `effect_outbox`, one per notice. The unique idempotency key is the notice key: `welcome:{membership id}` for welcomes, `help:{MessageSid}` for accepted HELPs, and a fresh key for pre-action HELPs. The payload (`dedupeKey`, `kind`, `memberId`, plus `messageSid` and `messageCreatedAtIso` for an accepted HELP) is the source of truth; no other table holds notice state.
* The member insert and its welcome effect commit in one transaction. So does the orphan-recovery transaction that re-activates an SMS membership. A failed enqueue rolls the membership and Twilio participant back like any other activation failure. The recovery path that only retains the row for queued removal writes no welcome.
* The request makes one bounded eager attempt per notice. It shares the effect's two-minute claim with the scheduled runner.
* The effect sweep and the participant-sync maintenance job (every minute) claim due notices through the effect worker. While Conversations is not configured, maintenance runs only HELP effects; a welcome that is claimed anyway backs off by its own age (a minute at first, doubling in effect up to an hour) without spending an attempt. A welcome for an org with the `event-chats` flag off is deferred an hour and retried, and one whose flag lookup is unavailable is deferred five minutes.
* Sending and recording are not atomic, and Programmable Messaging has no idempotency key or custom metadata to look up. So immediately before its first provider send, fenced on the claim token it holds, a notice records the exact text it is about to send (`sentBody`) and when that attempt started (`firstAttemptedAtIso`) in its payload. Later attempts keep both values and resend `sentBody` unchanged, even if the chat or event was renamed in between. If the claim was lost before the record write, nothing is sent (`outcome=lease_lost`).
* A null `firstAttemptedAtIso` proves no provider call happened, because the record write always precedes the send, so that notice is sent without any lookup.
* When `firstAttemptedAtIso` is set, the notice looks up the earlier send first. The lookup pages the Messages list from that pool number to that phone, newest first (50 per page, at most 5 pages), until it reaches a message sent before the window start. A message created inside the window, with a status other than `failed`/`undelivered`/`canceled` and a body exactly equal to `sentBody`, counts as the earlier send: the effect settles without sending (`outcome=reconciled`). The window starts 2 s before `firstAttemptedAtIso`, which covers Twilio's whole-second `date_created` and clock drift between Postgres and Twilio. Another chat's welcome on a reused pool number was sent before this notice's first attempt, so it never matches.
* HELP replies are serialized per member: a HELP effect waits (retries in 30 s without spending an attempt) while an earlier-queued HELP effect for the same member, ordered by `created_at` then `id`, is still `PENDING`. Every settled send stores the provider message SID in the payload (`providerSid`), in a claim-fenced write: the SID the send returned, or the SID a reconciliation matched. The lookup skips any message whose SID is stored on another notice for the same member, so a later HELP can only reconcile to its own earlier send. Remaining edge: an earlier HELP dead-lettered after an ambiguous last send has no SID, so a later HELP for that member can match that reply, and the member gets one reply for two requests. A lookup that keeps failing retries every five minutes without spending an attempt and holds later HELPs for that member (Sentry `sms_notice_lookup` each time).
* If the lookup fails or hits the page cap without reaching the window's start, the notice is deferred five minutes (`outcome=lookup_deferred`) without counting a send attempt, so lookup failures alone never dead-letter it.
* After a send Twilio accepted, the `providerSid` write is retried up to three times. If it still fails, or the claim was lost, the outcome is `sent_unrecorded` and the effect retries in a minute without being settled. The next claim reconciles it through the lookup above and records the SID, so a later HELP can never match an uncredited reply. These retries count against the effect's attempt limit and dead-letter at ten.
* A failed send backs off exponentially (1 minute doubling to 1 hour). After six failed attempts, or when the stored proxy is not a pool number, the effect goes `DEAD`; `requeueDeadEffects` re-queues it by hand. A recipient on the undeliverable list settles `DONE` (`outcome=undeliverable`).
* Settled effects are kept for the standard 28 days and then deleted by the effect cleanup sweep. Removing a member does not delete its effects; the next claim finds no deliverable target and settles without sending (`outcome=skipped`), and a re-add gets a fresh membership id and so a fresh welcome. A replayed add of an existing member queues nothing new.
* A welcome is not sent for an archived chat or a member that is no longer SMS. The worker reads the membership and chat again after the organization and event name lookups and immediately before sending. It does not hold the archive lock across the Twilio call; that lock serializes manager archive requests, which would otherwise fail as busy for the length of the provider call. An archive that lands after that final read can still let one welcome out.
* A welcome failure never fails the member add.
* Programmable Messaging has no idempotency key. If the process dies after Twilio accepts the send but before the effect settles, the next claim looks the send up by its recorded text and window, and sends again only when it is not found. Duplicates are limited to a send whose attempt record was lost.
* Members added before this queue existed get no retroactive welcome.

When event chats are off for a chat's org (flag off, or Conversations not configured), signed Conversations webhooks keep processing adds, edits, deletes, and SMS STOP cleanup so provider and local state stay aligned. Added messages atomically record the existing push consumer's terminal marker, and queued delivery observes the same marker before each provider batch. That marker expires with the 30-day ProcessedEvent retention window: a replay of a message first seen while disabled stays silent during that window, while a new message first received after re-enable can notify normally.

## Proxy pool (2–4 numbers)

`TWILIO_GROUPCHAT_NUMBERS` is a comma-separated E.164 list used as `MessagingBinding.ProxyAddress` for SMS participants.

| Rule | Why |
| - | - |
| **2–4 numbers** | Twilio enforces uniqueness of `(address, proxy_address)` **across conversations**. One person in two active chats needs two distinct proxies. Two is the minimum that allows overlapping membership; four is enough for dogfood without a large A2P footprint. |
| **No overlap with `TWILIO_FROM_NUMBER`** | The invitation sender owns inbound reply-code routing on `POST /webhooks/twilio`. Sharing it swallows Conversations SMS or invitation replies. |
| **Chat-only Messaging Service** | Pool numbers sit in a dedicated Messaging Service tied to the A2P Proxy campaign, never the invitation Service. See [Messaging Service for the pool](#messaging-service-for-the-pool). |

When every pool number is already bound to that phone, add-member returns a per-member failure with reason `sms group-chat number pool exhausted` (Twilio `50416` is walked internally; exhaustion is not a 500).

Buy and register the pool numbers in Twilio **before** enabling the feature. This runbook does not purchase numbers.

## Auto-creation disabled

On the dedicated Conversations Service, **disable conversation auto-creation** (Address Configuration / Autocreation for the pool numbers).

Soundcheck creates conversations only through `createConversation` when a manager creates an event chat. If auto-creation is on, an inbound SMS to a pool number can spawn an orphan conversation that:

* has no Postgres `event_chats` row
* never receives the `checkai` participant
* never fans out push
* burns a proxy binding

Leave auto-creation off in every environment that uses real Twilio, including dogfood.

## Messaging Service for the pool

US 10DLC numbers are registered to a campaign through a Messaging Service, so the pool numbers **do** belong to a Messaging Service. Use a dedicated, chat-only Messaging Service tied to the A2P "Proxy" campaign. Never use the Messaging Service, sender number, or campaign that sends invitations, reminders, or other Programmable Messaging traffic.

| Path | Number | Messaging Service | Inbound route |
| - | - | - | - |
| Invitation / reminder SMS | `TWILIO_FROM_NUMBER` | Invitation Service and campaign | `POST /webhooks/twilio` (`TWILIO_WEBHOOK_URL`) |
| Event-chat SMS bridge | `TWILIO_GROUPCHAT_NUMBERS` | Chat-only Service on the A2P Proxy campaign | Conversations service webhooks → `POST /webhooks/twilio/conversations` |

Set the chat-only Service's incoming-message handling to route to Conversations. Do not point its "A message comes in" webhook at an API route, and keep conversation auto-creation off (see [Auto-creation disabled](#auto-creation-disabled)). A Service that handles inbound SMS itself takes replies away from Conversations, so they never become `onMessageAdded` on the conversation.

Conversations binds `MessagingBinding.Address` + `ProxyAddress` directly. The welcome and HELP replies use Programmable Messaging with an explicit `From` of the member's pool number. They do not pass a Messaging Service SID and never use the invitation sender.

## A2P 10DLC

US carriers filter unregistered application-to-person SMS.

* **Local 10DLC pool numbers** need a registered **brand** and a **campaign** whose use case matches group-chat / customer-care traffic to crew phones. Approval is a Twilio / TCR process — do it in the Console; do not invent a campaign from this repo.
* Register the pool in the chat-only Messaging Service on its own "Proxy" campaign. The first text each phone gets is the welcome, which carries the sender, program, and STOP/HELP disclosure that campaign review expects.
* **Do not** hang the chat pool off the invitation sender's campaign or (if used) toll-free verification. Invitation traffic and group-chat traffic are different use cases and different numbers.
* Toll-free invitation senders (see `handoffs/eng-418/`) do **not** substitute for 10DLC on local proxy numbers.
* Until the campaign is `APPROVED`, treat SMS bridge as blocked even if in-app chat works. Dogfood only after at least one US handset sends and receives a group-chat SMS through a pool number.

`STOP` / carrier opt-out is primarily handled by Twilio and the carrier on the proxy number. When Conversations delivers a STOP-family keyword, `api-events` rejects it in the pre-action webhook and removes the SMS member (Twilio first, then Postgres). The post-action fallback attempts member cleanup first and provider-message deletion independently and skips persist/push; either failure is retried, including after membership is already gone. Carrier-level suppression may prevent the webhook from seeing every STOP.

## API keys and secrets

Conversations REST and AccessToken JWTs use an **API Key pair**, not the account auth token.

| Name | Shape | Used for |
| - | - | - |
| `TWILIO_ACCOUNT_SID` | `AC…` | AccessToken issuer; shared with Programmable Messaging |
| `TWILIO_AUTH_TOKEN` | secret | `X-Twilio-Signature` on **both** webhook routes |
| `TWILIO_API_KEY_SID` | `SK…` | Conversations REST Basic auth + JWT signing |
| `TWILIO_API_KEY_SECRET` | secret | Pair for the SID above |
| `TWILIO_CONVERSATIONS_SERVICE_SID` | `IS…` | Service scope |
| `TWILIO_GROUPCHAT_NUMBERS` | E.164 list | SMS proxy pool |
| `TWILIO_CONVERSATIONS_WEBHOOK_URL` | `https://…/webhooks/twilio/conversations` | Signature URL |
| `TWILIO_CONVERSATIONS_ENDPOINT` | emulator URL | **Local / e2e only** |

The auth token cannot sign Conversations JWTs. Missing `SK` / secret / `IS` throws `twilio conversations client is not configured` on create/token.

Store SIDs and secrets in the environment secret store (Encore secrets / GCP Secret Manager — see `infra-gcp/modules/events`). Do not commit values. Rotate the API Key by creating a new `SK…`, deploying it, then deleting the old key.

## Feature rollout

Order matters. Do not load the Conversations secrets until the Twilio side is live: once they are valid, the server treats event chats as configured, and the `event-chats` flag decides which orgs see them.

1. Dedicated `IS…` service exists; auto-creation **off**.
2. Canonical pre/post webhooks point at `/webhooks/twilio/conversations`.
3. 2–4 pool numbers purchased, in a dedicated chat-only Messaging Service (**not** the invitation Service), **not** equal to `TWILIO_FROM_NUMBER`.
4. A2P 10DLC brand/campaign approved for those numbers (US).
5. Secrets populated (`SK` / secret / `IS` / pool / conversations webhook URL). Confirm `TWILIO_CONVERSATIONS_ENDPOINT` is unset in the deploy.
6. Roll a revision so it picks up the secrets. Hit `POST /webhooks/twilio/conversations` with a bad signature and expect `403`; for an org with the flag off, a valid signed callback still persists provider state but terminally suppresses notification delivery.
7. Roll PostHog `event-chats` to dogfood orgs, then wider. Web, mobile and api-events read the same flag, so there is no second server toggle to keep in step.

## Monitoring

| Signal | Where | Meaning |
| - | - | - |
| `service: twilio_conversations` | Sentry (`captureServiceError`) | REST create/add/close/delete/send failures (except expected `50416`) |
| `twilio_conversations` / `sms_proxy_pool_exhausted` plus `event_chats.sms_proxy_pool_exhausted` | Sentry + `incrementCounter` | Pool exhausted for one member; identifiers are chat/event/org/member ids only. The request still returns a per-member failure. |
| `twilio_conversations` / `signature_verification` | Sentry (`captureWebhookError`) | Bad or missing signature |
| `twilio_conversations` / `message_persist` | Sentry | Post-event message persist, publication, or `last_message_at` work threw; the request returns non-2xx for Twilio retry |
| `notification.event_chat_message` (`outcome` = `pushed` / `skipped` / `replayed`) | Counter | Processor fan-out. Tags are low-cardinality only — never chat, user, or message ids. |
| Warn logs `sms group-chat number pool exhausted` / failed member reasons | App logs | Pool too small or leaked bindings |
| Warn `failed to persist… rolling back twilio participant` | App logs | Postgres/Twilio membership drift averted or rollback failed |
| `event_chats.twilio_lifecycle` (`operation` = `close`/`delete`, `outcome` = `failed`, `source` = `archive`/`event_lifecycle`/`event_purge`/`org_purge`) | Counter | Twilio close/delete recovery failed; inspect the retained pending job and its next scheduled retry |
| `event_chats.sms_opt_out` (`outcome` = `removed`/`already_gone`/`failed`) | Counter | Conversations webhook handled or retried a STOP-family SMS |
| `event_chats.sms_notice` (`kind` = `WELCOME`/`HELP`, `outcome` = `sent`/`skipped`/`deferred`/`failed`/`abandoned`/`lease_lost`/`reconciled`/`sent_unrecorded`/`lookup_deferred`) | Counter | Welcome and HELP delivery. For `abandoned`, inspect `effect_outbox` rows of kind `event_chat_sms_notice` in state `DEAD` |
| `twilio_conversations` / `sms_notice` and `deferred_sms_notice` | Sentry | A notice send failed or its worker threw; tags are kind, chat/member ids, and attempt count only |
| `event_chats.sms_help` (`outcome` = `queued`/`duplicate`/`skipped`) plus `twilio_conversations` / `sms_help` | Counter + Sentry | HELP intake. `sms_help` in Sentry means queueing or message deletion failed and Twilio will retry |
| Warn `sms disclosure lookup failed closed` | App logs | The pending-welcome check timed out or failed, so the disclosure suffix was appended without knowing |
| Warn `failed to close/delete twilio conversation` | App logs | Provider recovery failed; the worker throws and retains the durable job for retry |
| Expo push `data.type = event_chat_message` | Device / push delivery logs | In-app fan-out via the message processor (SMS members do not get this push; muted `event_chat` push prefs skip) |

Pre-event `onMessageAdd` stays fast and synchronous for prefixing only. Post-event persistence and topic publication run on the request thread so Twilio can retry failures. Push fan-out remains on `EventChatMessageTopic` → `eventChatMessageProcessor`, not in the webhook handler.

## Rollback

1. Turn PostHog `event-chats` off (for the affected orgs, or everyone). Routes disappear for clients; Twilio keeps POSTing signed callbacks, which continue persistence and cleanup while push delivery is suppressed. Within about six minutes the flag sweep removes affected users' in-app participants, so an already-minted token stops reaching those conversations.
2. Leave the Conversations Service, webhooks, provider credentials, scheduler, reconciliation workers, and pool numbers in place. Deleting the service destroys message history and orphan-binds phones; queued cleanup and reconciliation still protect existing conversations while the flag is off.
3. If SMS must stop, managers archive chats before disabling because authenticated chat routes are then unavailable. Already-queued and event-lifecycle archive recovery continue while disabled; their durable close jobs retry provider uncertainty asynchronously.
4. Do not point pool numbers at `/webhooks/twilio` as a "safe" fallback — that mixes invitation reply codes with group SMS.

Re-enable by turning the PostHog flag back on; the sweep re-adds in-app participants, and existing `event_chats` rows and `CH…` SIDs remain valid if the same `IS…` is still configured. Pushes already accepted by Expo cannot be recalled.

## Reconciliation

| Concern | Source of truth | Repair |
| - | - | - |
| Chat name, archive, membership rows | Postgres | Rename commits DB first; Twilio `FriendlyName` is best-effort. Archive intent and its durable close job commit before provider work, with no transaction held across HTTP. A per-chat session lock on one leased database connection serializes manual and lifecycle requests; contention returns a retryable failure. The worker retains failed or uncertain close work for asynchronous retry, and `404` completes cleanup idempotently. |
| Message bodies | Twilio Conversations for live SDK history; Postgres webhook mirror for REST/MCP | `onMessageAdded` upserts by Twilio message SID; content changes and deletes advance citation version and status. A newer same-body update advances only the provider ordering timestamp, preventing a delayed older edit from overwriting the latest body. Replays publish deterministic event IDs for ProcessedEvent dedup. Remove-before-add creates a tombstone. Reconcile a missing mirror row by replaying the signed post-event webhook or backfilling from Twilio. |
| SMS join intents | `event_chat_sms_join_intents` | One row per SMS member mid-join. Cleared by the membership commit, a definitive bind failure, or a rolled-back activation; ignored after 10 minutes and swept by the minute job. Chat delete cascades. |
| SMS welcome and HELP notices | `effect_outbox` (`event_chat_sms_notice`) | One effect per welcome (per membership) or HELP reply (per message). The eager attempt and the minute worker share its claim; failed sends back off, and six failures set `DEAD`. `requeueDeadEffects` re-queues it. |
| In-app / SMS participants | Postgres membership plus revocation snapshot | `persistMember` deletes the Twilio participant if the DB insert fails. If rollback also fails, a last-resort upsert preserves the participant SID or the request fails loudly. Claimed SMS members reconcile to CHAT on join. Removal commits the revocation snapshot and idempotent participant-sync job together before provider revocation; effective access excludes the snapshot until membership is restored. The worker retries provider reconciliation, and `404` is already gone. |
| Create with zero successful members | Neither | Rolls back the Twilio conversation first, then the chat row. A failed Twilio delete keeps the local row and SID for retry. |
| Event / organization delete | Postgres transaction and `effect_outbox` (`twilio_conversation_deletion_v2`) | Conversation SIDs are queued in the same transaction as the cascade. The maintenance worker deletes Twilio conversations after commit and retains failed work for retry; `404` completes cleanup idempotently. |
| Event archive / cancel | `closeChatsForEvent` | Each chat commits archive intent and a durable close job under the same session lock as manual archive, then publishes a prompt. Queue-write failures propagate; provider close failures remain in the background job for retry. Already archived conversations retain their marker and can re-queue close work. |
| Organization member leave | `revokeChatAccessForOrgUser` | Provider revocation must succeed before local access is removed; failures remain retryable. |
| Last invitation deletion | Postgres transaction after provider revocation | Invitation mutations use per-item savepoints; last-active roster/chat cleanup uses the final batch state in the enclosing transaction. Payable voids and deletions commit with cleanup. Cleanup failure rolls back the batch and retains invitations for retry; prepared provider access is restored best-effort. Unprepared last-active cleanup fails instead of being skipped. |
| In-app self-leave | `POST /api/v1/chats/{chat_id}/leave` | Removes the caller's Twilio participant first (`404` = already gone), then the local row. Removing anyone else stays manager-only `DELETE …/members/{org_member_id}`. |
| Proxy bindings | Twilio uniqueness + `activeProxyAddressesForPhone` | Walk the pool; treat `50416` as "try next". If Twilio and Postgres disagree, prefer Twilio: a leaked binding burns a slot until the conversation is closed or the participant is removed. |

An ambiguous close failure reads back Twilio state: closed or missing is success, while any other or unavailable state retains the archive marker and durable close job and returns a retryable failure. Even an active read cannot exclude the original timed-out request completing later. [Twilio error 50377](https://www.twilio.com/docs/api/errors/50377) also confirms the final closed state. Successful close requests make no extra read; each close attempt uses the existing bounded GET retry budget; a later queued retry can reassert the close POST.

The archive marker and close intent commit together before provider HTTP. The archive-close worker runs every minute, claims up to five jobs for a five-minute lease, and rechecks each pending intent under the same per-chat session lock as manual archive. Successful closure acknowledges the job; provider or acknowledgement failure retains it for retry, even when the chat feature is disabled. Definitive provider failures also retain the archive marker and close job for retry. Already-archived retries ensure intent exists before HTTP, and the migration seeds existing archived chats. Deleting the chat cascades its close job; the separate deletion queue then owns provider cleanup. Recovery failures use `event_chats.twilio_lifecycle` with `source=archive_recovery` and Sentry operation `deferred_conversation_close`.

Participant removal records durable intent in `event_chat_member_revocations` before provider revocation. The saved transport identities survive local membership deletion so queued retries can finish cleanup. Revocation, restoration, re-addition, and participant reconciliation share a per-member session lock; roster removals acquire every affected participant key in sorted order on one lock-pool connection and retain them through preparation, the short roster transaction, and compensation. A retained membership row does not cancel removal intent. Prepare and org-leave set `retain_local_snapshot` so the reconcile worker never deletes that row — it converges provider absence and leaves row and marker for the business removal, which deletes the exact snapshot and enqueues a fresh generation in one transaction; a later pass retires the marker once the row is gone. Direct remove, leave, and STOP record the exact snapshot identity with retain false; if provider revocation succeeded but the local snapshot delete did not, reconciliation CAS-deletes that leftover and ACKs. Explicit restoration first verifies that the retained row is still the exact prepared snapshot, then clears the marker and bumps the generation in one transaction before provider work, so a failed eager restore leaves queued PRESENT recovery that restores the persisted membership channel. SMS recovery also removes a stray CHAT identity left by an interrupted explicit join. A membership update newer than the revocation also supersedes it. Archived chats still reconcile effective revocations using the saved transport identities; only effective PRESENT work is skipped. Each participant-sync enqueue advances a generation, so an older worker cannot acknowledge newer work. Prompt delivery accelerates processing, and the minute-based worker retries durable pending jobs after lease expiry.

The deletion queue retries committed cleanup automatically. If initial conversation creation succeeds but neither the chat row nor a deletion job can be persisted, the request attempts direct provider cleanup and reports any cleanup failure. There is no general nightly scan for other divergence. After an incident, list conversations on the dedicated `IS…` and compare `CH…` SIDs to `event_chats.twilio_conversation_sid`. Orphans (Twilio only) should be closed or deleted by hand. DB rows whose conversation is gone should be archived.

An advisory-lock connection failure aborts the protected work's signal and makes its ownership checks fail. Provider calls receive that signal, and mutation boundaries assert ownership before continuing; the failed lock connection is destroyed instead of returning to the pool. The wrapper propagates failures before, during, and after protected work without terminating unrelated requests. Already-issued provider calls remain subject to reconciliation. The separate four-connection lock pool and lock ordering are unchanged.

## CI and emulator limits

The `encore-e2e` job in `.github/workflows/api-events-deploy.yml` starts
`tools/twilio-emulator` on `:4013`, boots api-events with
the Conversations test credentials pointed at that emulator and
`FEATURE_FLAG_OVERRIDES` including `event-chats`, and runs
`e2e/tests/event-chats/crud.test.ts` with `E2E_EVENT_CHATS=1`. Locally the
CRUD file is still skipped unless that flag is set. See `e2e/README.md` for
the exact env (seeded `AC000…0` / `SK000…0` / `IS000…0`, and
`TWILIO_GROUPCHAT_NUMBERS`).

`e2e/tests/event-chats/routes-probe.test.ts` always runs: unauthenticated
probes get `401` before the kill switch, which registers route-coverage hits
in both flag states.

Known emulator limits:

* Not production Twilio (no A2P, no carrier `STOP`, no real SMS, no AccessToken validation).
* Conversations webhooks are not fired by the emulator; e2e posts signed
  `onMessageAdd` / `onMessageAdded` / `onMessageUpdated` / `onMessageRemoved`
  bodies to `/webhooks/twilio/conversations`.
* `DELETE …/Participants/{sid}` **is** implemented so member remove and
  `user.deleted` revoke can be asserted.
* Signature checks still use the account auth token over the exact webhook URL
  (`E2E_CONVERSATIONS_WEBHOOK_URL` or `${E2E_API_URL}/webhooks/twilio/conversations`).

Do not point CI at a real Twilio account. Deployed environments must leave
`TWILIO_CONVERSATIONS_ENDPOINT` unset.

## Dogfood checklist

Do this against the **target** Twilio account after secrets are set. Do not buy numbers or register A2P from the agent.

* [ ] Dedicated `IS…`; auto-creation **disabled**.
* [ ] Pre-event and post-event webhooks both `POST` `{API}/webhooks/twilio/conversations`.
* [ ] Pool is 2–4 E.164s, disjoint from `TWILIO_FROM_NUMBER`, in a chat-only Messaging Service (never the invitation Service) whose inbound handling routes to Conversations.
* [ ] US 10DLC Proxy campaign **approved** for that chat-only Messaging Service (or dogfood is non-US / in-app only and that limit is written down).
* [ ] `TWILIO_API_KEY_SID` / `SECRET` mint a token (`GET /api/v1/chat/token` as a signed-in user with the flag on).
* [ ] Conversations secrets loaded (not `PLACEHOLDER`); PostHog `event-chats` on for the dogfood org.
* [ ] Manager creates a named event chat with at least one **in-app** member and one **SMS** member.
* [ ] Adding the SMS member sends one welcome text from that member's **pool** number (not the invitation sender) before any group message. In-app members do not see it, and repeating the add does not send a second one.
* [ ] A message posted while a welcome is still pending carries the ` (Soundcheck gig chat. Reply STOP to leave, HELP for help)` suffix; once the welcome is sent, new messages do not.
* [ ] After a manager posts an introduction, the SMS arrives from the same pool number.
* [ ] Texting `HELP` from the handset returns exactly one help reply from the pool number (check that Twilio's own HELP auto-reply is off or matches), and the web/mobile thread does not show `HELP`.
* [ ] In-app message appears on the handset; handset reply appears in the web/mobile thread and pushes other in-app members (`event_chat_message`).
* [ ] Second concurrent chat for the same SMS phone uses a **different** pool number; a third overlapping chat (if pool size is 2) fails that member with pool exhausted.
* [ ] Rename updates the UI; Twilio friendly name is best-effort.
* [ ] Archive stops SMS; further posts do not fan out.
* [ ] Archive/cancel the **event**; remaining chats close.
* [ ] Forged conversations signature → `403` / `INVALID_SIGNATURE`.
* [ ] Turn `event-chats` off for the dogfood org: UI/API hide (`404`); valid signed callbacks still persist, but no new push is sent.
* [ ] Sentry `twilio_conversations` clean for the dogfood window.

## Related

* [Integrations](https://docs.soundchecklive.io/engineering/backend/integrations.md) — Twilio SMS + chat env block
* [api-events operations](https://docs.soundchecklive.io/engineering/backend/api-events-operations.md) — secret inventory
* [Environment variables](https://docs.soundchecklive.io/engineering/reference/environment-variables.md)
* [API testing](https://docs.soundchecklive.io/engineering/backend/testing.md) — Vitest + e2e gates
* [Chat (product)](https://docs.soundchecklive.io/features/chat/overview.md)

### Stable message pagination

For stored history, pass `before` as the last message's `id` from the previous page to `list_event_chat_messages` or `GET /api/v1/chats/{chatId}/messages`. This anchors the next page to immutable creation time and id, so newer arrivals do not repeat or skip older messages. Do not combine `before` with the legacy `offset` parameter.

Organization deletion queues Twilio conversation SIDs in the deletion transaction. The internal cleanup worker runs every minute and retries failed deletions after a five-minute lease. A database rollback therefore preserves provider conversations and history; a provider outage leaves durable work for later retry.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.