Event chats

Operational runbook for event group chats in api-events. Twilio Conversations remains the live transport and SDK history source; Postgres holds chat metadata, membership, and a webhook-fed message mirror used by authorized REST and MCP reads. In-app members join with a Conversations AccessToken (user_<id>); phone-only crew are bridged over SMS through a dedicated proxy-number pool. This page does not provision Twilio, A2P, or cloud resources. Configure those in the Twilio Console and secret stores, then point api-events at the existing SIDs. Product behavior for end users: Chat. Env inventory: Integrations and api-events operations. Event-chat e2e: e2e/README.md.

What is shipped

Surface	Behavior
Scope	Named group chats on an event. Managers create them; there is no auto-thread per gig.
In-app	Conversations SDK on web and mobile, token from `GET /api/v1/chat/token`.
SMS bridge	Unclaimed roster members with a usable E.164 join as SMS participants and get one welcome text from their proxy number. Replies reach the whole group.
Rollout	The `event-chats` PostHog flag (`flags/registry.json`). Authenticated routes evaluate it for the caller (email, memberships, active org), like web and mobile; background paths (webhook persistence, push, SMS welcomes) evaluate it for the chat’s org, so the flag needs an org-property release condition for them to match. The server also stays closed until Twilio Conversations is configured (API Key SID/secret, `IS…` service, proxy pool, webhook URL; Terraform `PLACEHOLDER` counts as unset). Off for an org → its authenticated chat routes `404`; signed Conversations webhooks still persist provider state while push delivery and SMS welcomes are suppressed.
Token and membership	`GET /api/v1/chat/token` mints `user_<id>` when the flag is on for the caller in at least one org where they have an active chat. The Conversations grant covers the whole service, so access is limited by participant set instead: every minute the participant-sync job re-evaluates the flag (caller context) for the (org, user) pairs not checked in the last five minutes, 50 per page, for up to 20 seconds, records it in `event_chat_flag_gates`, and on a flip enqueues a participant sync for that user’s chats in that org. `checked_at` moves only when a decision is written. An `unavailable` evaluation or a failed pair instead sets `retry_after` one minute out, so it is skipped (not blocking later pairs) and retried on the next run with its old `checked_at`; a run that hits the 20-second budget with pairs still due logs `event chat flag sweep ran out of time`. The reconciler then removes the in-app participant while the gate is suppressed and re-adds it when it clears; the membership row, SMS participants and history are untouched. Every other path that creates an in-app participant follows the same gate (`eventChatInAppGate.ts`): creating a chat, adding members, a manager joining, a claimed SMS member switching to in-app, and restoring access after a failed removal. Before taking the participant lock, an add re-evaluates the flag for that person (caller context) and records the result. If the evaluation is `unavailable` or fails and no gate row exists yet, the add records the pair as undecided: suppressed, with `checked_at` null, and queues participant syncs for that user’s live chats in the org. The sweep then decides it first; on, it flips and re-adds; off, it stays as is. An existing row is kept as it is. Only add paths create undecided rows. For the reconciler and the sweep, a missing row still means allowed, so members from before #2492, who have no rows, are not mass-removed on deploy. Inside the lock it reads the stored gate. While suppressed, the membership row is created (or restored, or left on SMS) with no Twilio participant and is reported as added, so turning the flag back on re-adds it through the normal flip. After saving, every path that writes a `user_id` onto a CHAT row (a new member, or an SMS → CHAT claim) rechecks the gate in one transaction. It creates the row as undecided (suppressed) if missing, takes a `FOR SHARE` lock, and queues a participant sync if the row no longer matches what the add decided. So a flip in the window between the check and the save cannot miss the row, and a pair with no decision converges to removed until the sweep decides it. A flag change therefore reaches Twilio within about six minutes while each run finishes inside its budget; if that warning repeats, the population has outgrown the budget and the delay grows with it. An `unavailable` evaluation (PostHog down) changes no decision, so while PostHog stays down no removal or re-add can happen; it resumes on the first run after evaluations succeed.
Client flag	PostHog `event-chats` hides the UI. The API reads the same flag; there is no env toggle.
CheckAI	Identity `checkai` is added to every conversation but does not post in v1. In-thread AI is not shipped.
Mentions	Clients send `mentions: [{org_member_id, name}]` and `mentions_everyone: true` in the Twilio message attributes, next to readable `@Name` text in the body. The pre-send `onMessageAdd` hook validates them against the chat (current members only, never the author, `@everyone` only from `canManageGigOperations`) and returns an `attributes` override that rewrites only those two keys; it runs before the message exists, so it never races reactions or edits. A validation timeout fails closed on `@everyone` only. `onMessageAdded` re-validates and stores `mentioned_org_member_ids`, `mentions_everyone` and the Twilio `provider_index` on `event_chat_messages`. Push fan-out sets `isMention` per recipient from those columns and uses mention copy (“Maya mentioned you: …”, “Maya to everyone: …”). `EventChat.my_last_mention_index` is the newest mention of the caller by someone else; clients compare it with their Twilio read horizon for the `@` badge. Edits never change mentions.
Per-chat notifications	Each member sets a level (All messages / Mentions only / Nothing) and an optional pause (one hour, or until the gig ends) per chat, stored in `event_chat_notification_prefs` keyed by (chat, org member). `GET/PUT /api/v1/chats/{chat_id}/notifications`, and every `EventChat` response carries the caller’s `my_notifications`. A one-hour pause stores its instant; “until the gig ends” stores `GIG_END` and resolves at read and push time with `eventActionableUntil` in the gig’s zone, so a reschedule moves it. `is_muted` is computed on the server and clients only read it. Push fan-out drops a recipient when `chatPushAllowed` says so: Nothing never pushes; Mentions only and a pause push only mentions. The org-level “Event chat messages” toggle is checked first, unchanged. SMS members have no per-chat mute yet.

REST: GET/POST /api/v1/events/{event_id}/chats, PATCH /api/v1/chats/{chat_id}, POST …/archive, POST …/join, POST …/leave, POST …/members, DELETE …/members/{org_member_id}, GET …/messages, GET …/messages/{message_id}, GET /api/v1/chat/token. Listing is read-only (no Twilio, no membership insert). Managers join via POST …/join before minting a token. In-app members leave themselves via POST …/leave. Revocation removes the stored participant and checks the claimed user’s Twilio identity as well, so an interrupted SMS-to-app migration cannot leave an unrecorded participant behind. A failed identity lookup keeps the local membership available for retry. Webhook: POST /webhooks/twilio/conversations.

Dedicated Conversations Service

Create a Conversations Service used only by Soundcheck event chats (IS…). Put that SID in TWILIO_CONVERSATIONS_SERVICE_SID. Do not:

Reuse the default / shared Conversations Service that other products already write to.
Point this SID at a Messaging Service. Conversations and Programmable Messaging are different products.
Set TWILIO_CONVERSATIONS_ENDPOINT in a deployed environment. That override is emulator-only (http://localhost:4013/conversations for tools/twilio-emulator).

Every REST call is scoped to /v1/Services/{TWILIO_CONVERSATIONS_SERVICE_SID}/…. AccessToken ChatGrants must use the same SID or the SDK cannot open the conversation.

Canonical pre-event and post-event webhooks

Configure service-level webhooks on that Conversations Service (Twilio Console → Conversations → Services → the dedicated IS… → Webhooks). Do not configure these on individual conversations, on phone numbers, or on a Messaging Service.

Twilio field	Canonical value
Pre-event webhook URL	`{API_BASE_URL}/webhooks/twilio/conversations`
Pre-event method	`POST`
Pre-event filters	`onMessageAdd` (optional: `onParticipantAdd`)
Post-event webhook URL	`{API_BASE_URL}/webhooks/twilio/conversations`
Post-event method	`POST`
Post-event filters	`onMessageAdded`, `onMessageUpdated`, `onMessageRemoved`

api-events verifies X-Twilio-Signature with the account auth token against TWILIO_CONVERSATIONS_WEBHOOK_URL (not TWILIO_WEBHOOK_URL, which is the invitation SMS route and will always fail signature checks here). Set TWILIO_CONVERSATIONS_WEBHOOK_URL to the exact public URL Twilio POSTs, including scheme and path. A mismatch is 403 / INVALID_SIGNATURE. The handler prefixes SMS bodies on EventType=onMessageAdd with Twilio’s JSON {"body":…} override. Post-event onMessageAdded, onMessageUpdated, and onMessageRemoved persistence and topic publication are awaited before HTTP 200; a failure returns non-2xx so Twilio retries. Creates also bump last_message_at and publish EventChatMessageTopic with a deterministic event ID derived from message ID, action, and citation version. Created events always use citation version 1 so a late onMessageAdded replay after edits keeps the same ProcessedEvent id; updated and deleted ids still include the current citation version. A remove that arrives before add writes a DELETED tombstone keyed by MessageSid, so the delayed add cannot resurrect it. eventChatMessageProcessor consumes created messages with a ProcessedEvent inbox, suppresses CheckAI and the author, honors the event_chat push preference, and batches Expo delivery. Update and remove events are terminally consumed without push. Other event types are acknowledged 200 and ignored. If an SMS author posts a STOP-family control word (STOP, STOPALL, UNSUBSCRIBE, CANCEL, END, QUIT, including the Name: STOP prefix rewrite), the pre-action hook removes that Twilio participant and the local membership, then returns HTTP 403 so the control message is rejected. If the pre-action hook times out and Twilio accepts the message, the post-action hook attempts membership revocation first, then provider-message deletion. A delete failure does not skip opt-out; either failure returns non-2xx so Twilio retries. Retries stay safe once membership is already gone (already_gone). Neither path persists or pushes the control message. Carrier-level suppression can still swallow some STOP texts before they reach this webhook — treat this as a defensive complement, not a guarantee that every opt-out is visible to api-events. If an SMS author sends exactly HELP (trimmed, any case), the pre-action hook queues a help reply and returns HTTP 403, so the group never sees the message. The reply goes only to that phone, from its bound proxy number, through Programmable Messaging:

Soundcheck gig chat help: info@soundchecklive.io. Reply STOP to leave. Msg & data rates may apply.

The reply is a durable SMS notice (see below). If queueing it fails, the pre-action hook returns non-2xx instead of 403, so Twilio accepts the message and the post-action fallback handles it. If the pre-action hook times out or fails and Twilio accepts the message, the post-action hook queues the reply keyed by MessageSid, then deletes the message from the conversation, and skips persist and push. Either failure returns non-2xx so Twilio retries; the MessageSid key keeps retries to one reply. Neither path sends the SMS before answering Twilio: the queued reply is attempted only after the webhook response is written, and the minute maintenance job sends it if that attempt does not run or fails. Pre-action events carry no MessageSid, so one text can reach both paths under different keys: the pre-action hook queued a reply but answered Twilio too late, or the two ran concurrently. Both paths take a per-member transaction advisory lock (event-chat-sms-help:{member}) around check-and-insert, and treat a pre-action row and an accepted row as the same text when the pre-action row’s created_at is within SMS_HELP_SAME_MESSAGE_WINDOW_SECONDS (15 s) of the accepted message’s own DateCreated. Whichever path commits second queues nothing. The window covers Twilio’s pre-action timeout plus clock skew between Twilio and the database; separate HELP texts further apart each get a reply. Two HELP texts from one member within 15 s of each other, where the first was rejected and the second accepted, get one reply. HELP handling does not depend on the event-chats flag or on the chat being active. Twilio may also answer HELP itself (default keyword handling or Advanced Opt-Out on the Messaging Service). Turn that off or set it to the same text on the chat-only Messaging Service, or the sender gets two replies.

SMS notices: welcome and HELP

Carriers require the first text a recipient gets on an A2P campaign to identify the sender and program and to give opt-out instructions. When a member is added on the SMS channel, api-events sends that phone one welcome from its bound proxy number:

Soundcheck: {org} added you to the "{chat}" chat for {event}. Replies go to the group. Reply STOP to leave, HELP for help.

Names are reduced to the GSM-7 basic character set. Extension-table characters (^ { } [ ] ~ | \ €), the backtick, curly quotes, and dashes are mapped to basic equivalents; accents outside GSM-7 are stripped; anything else is dropped. Each name is then truncated to 40 septets. The fixed text is 103 septets, so the welcome is at most 223 septets: always GSM-7, one segment for typical names, never more than two (306). A blank organization, chat, or event name falls back to generic wording. The HELP reply and the disclosure suffix below are fixed GSM-7 text. Welcomes and HELP replies go through Programmable Messaging (sendSMS with an explicit From), not a Conversations message, so in-app members never see them. They are only ever sent from a number in TWILIO_GROUPCHAT_NUMBERS that is not TWILIO_FROM_NUMBER; any other stored proxy abandons the notice instead of falling back to the invitation sender.

Welcome before the first group message

Each SMS member’s welcome is attempted as soon as that member’s add releases the chat locks, before the next member in the same request is added.
While an SMS member is joining or their welcome is unsent, the pre-action onMessageAdd hook appends (Soundcheck gig chat. Reply STOP to leave, HELP for help) to every message posted in that chat, whatever the author. A media-only message with no text gets the disclosure alone, without the leading space. It uses the same {"body":…} override as the SMS author prefix. So a group message that beats the welcome still carries the program name and STOP/HELP instructions.
“Joining” is an event_chat_sms_join_intents row keyed by chat and org member. It is committed before the Twilio participant is bound, so it also covers the window between the bind and the membership commit.
- The membership commit (or the orphan-recovery commit) deletes it in the same transaction that queues the welcome, and the welcome effect takes over.
- A definitive bind failure, an exhausted pool, or an activation failure whose Twilio participant rollback succeeded deletes it. If the intent can’t be written, the member add fails before any Twilio call.
- An ambiguous bind, or an add that ends in queued removal, leaves it in place: the participant may still exist.
- Backstop: an intent older than SMS_JOIN_INTENT_TTL_MINUTES (10) no longer counts, and the minute maintenance job deletes it.
The check is one EXISTS … OR EXISTS … query per inbound message, keyed by conversation SID through the unique index, and it runs in parallel with the prefix lookup under the same 100 ms budget. It fails closed: if the lookup errors or times out, the suffix is appended anyway. Dead-lettered or settled welcomes stop the suffix.

Durability

Both kinds are event_chat_sms_notice rows in effect_outbox, one per notice. The unique idempotency key is the notice key: welcome:{membership id} for welcomes, help:{MessageSid} for accepted HELPs, and a fresh key for pre-action HELPs. The payload (dedupeKey, kind, memberId, plus messageSid and messageCreatedAtIso for an accepted HELP) is the source of truth; no other table holds notice state.
The member insert and its welcome effect commit in one transaction. So does the orphan-recovery transaction that re-activates an SMS membership. A failed enqueue rolls the membership and Twilio participant back like any other activation failure. The recovery path that only retains the row for queued removal writes no welcome.
The request makes one bounded eager attempt per notice. It shares the effect’s two-minute claim with the scheduled runner.
The effect sweep and the participant-sync maintenance job (every minute) claim due notices through the effect worker. While Conversations is not configured, maintenance runs only HELP effects; a welcome that is claimed anyway backs off by its own age (a minute at first, doubling in effect up to an hour) without spending an attempt. A welcome for an org with the event-chats flag off is deferred an hour and retried, and one whose flag lookup is unavailable is deferred five minutes.
Sending and recording are not atomic, and Programmable Messaging has no idempotency key or custom metadata to look up. So immediately before its first provider send, fenced on the claim token it holds, a notice records the exact text it is about to send (sentBody) and when that attempt started (firstAttemptedAtIso) in its payload. Later attempts keep both values and resend sentBody unchanged, even if the chat or event was renamed in between. If the claim was lost before the record write, nothing is sent (outcome=lease_lost).
A null firstAttemptedAtIso proves no provider call happened, because the record write always precedes the send, so that notice is sent without any lookup.
When firstAttemptedAtIso is set, the notice looks up the earlier send first. The lookup pages the Messages list from that pool number to that phone, newest first (50 per page, at most 5 pages), until it reaches a message sent before the window start. A message created inside the window, with a status other than failed/undelivered/canceled and a body exactly equal to sentBody, counts as the earlier send: the effect settles without sending (outcome=reconciled). The window starts 2 s before firstAttemptedAtIso, which covers Twilio’s whole-second date_created and clock drift between Postgres and Twilio. Another chat’s welcome on a reused pool number was sent before this notice’s first attempt, so it never matches.
HELP replies are serialized per member: a HELP effect waits (retries in 30 s without spending an attempt) while an earlier-queued HELP effect for the same member, ordered by created_at then id, is still PENDING. Every settled send stores the provider message SID in the payload (providerSid), in a claim-fenced write: the SID the send returned, or the SID a reconciliation matched. The lookup skips any message whose SID is stored on another notice for the same member, so a later HELP can only reconcile to its own earlier send. Remaining edge: an earlier HELP dead-lettered after an ambiguous last send has no SID, so a later HELP for that member can match that reply, and the member gets one reply for two requests. A lookup that keeps failing retries every five minutes without spending an attempt and holds later HELPs for that member (Sentry sms_notice_lookup each time).
If the lookup fails or hits the page cap without reaching the window’s start, the notice is deferred five minutes (outcome=lookup_deferred) without counting a send attempt, so lookup failures alone never dead-letter it.
After a send Twilio accepted, the providerSid write is retried up to three times. If it still fails, or the claim was lost, the outcome is sent_unrecorded and the effect retries in a minute without being settled. The next claim reconciles it through the lookup above and records the SID, so a later HELP can never match an uncredited reply. These retries count against the effect’s attempt limit and dead-letter at ten.
A failed send backs off exponentially (1 minute doubling to 1 hour). After six failed attempts, or when the stored proxy is not a pool number, the effect goes DEAD; requeueDeadEffects re-queues it by hand. A recipient on the undeliverable list settles DONE (outcome=undeliverable).
Settled effects are kept for the standard 28 days and then deleted by the effect cleanup sweep. Removing a member does not delete its effects; the next claim finds no deliverable target and settles without sending (outcome=skipped), and a re-add gets a fresh membership id and so a fresh welcome. A replayed add of an existing member queues nothing new.
A welcome is not sent for an archived chat or a member that is no longer SMS. The worker reads the membership and chat again after the organization and event name lookups and immediately before sending. It does not hold the archive lock across the Twilio call; that lock serializes manager archive requests, which would otherwise fail as busy for the length of the provider call. An archive that lands after that final read can still let one welcome out.
A welcome failure never fails the member add.
Programmable Messaging has no idempotency key. If the process dies after Twilio accepts the send but before the effect settles, the next claim looks the send up by its recorded text and window, and sends again only when it is not found. Duplicates are limited to a send whose attempt record was lost.
Members added before this queue existed get no retroactive welcome.

When event chats are off for a chat’s org (flag off, or Conversations not configured), signed Conversations webhooks keep processing adds, edits, deletes, and SMS STOP cleanup so provider and local state stay aligned. Added messages atomically record the existing push consumer’s terminal marker, and queued delivery observes the same marker before each provider batch. That marker expires with the 30-day ProcessedEvent retention window: a replay of a message first seen while disabled stays silent during that window, while a new message first received after re-enable can notify normally.

Proxy pool (2–4 numbers)

TWILIO_GROUPCHAT_NUMBERS is a comma-separated E.164 list used as MessagingBinding.ProxyAddress for SMS participants.

Rule	Why
2–4 numbers	Twilio enforces uniqueness of `(address, proxy_address)` across conversations. One person in two active chats needs two distinct proxies. Two is the minimum that allows overlapping membership; four is enough for dogfood without a large A2P footprint.
No overlap with `TWILIO_FROM_NUMBER`	The invitation sender owns inbound reply-code routing on `POST /webhooks/twilio`. Sharing it swallows Conversations SMS or invitation replies.
Chat-only Messaging Service	Pool numbers sit in a dedicated Messaging Service tied to the A2P Proxy campaign, never the invitation Service. See Messaging Service for the pool.

When every pool number is already bound to that phone, add-member returns a per-member failure with reason sms group-chat number pool exhausted (Twilio 50416 is walked internally; exhaustion is not a 500). Buy and register the pool numbers in Twilio before enabling the feature. This runbook does not purchase numbers.

Auto-creation disabled

On the dedicated Conversations Service, disable conversation auto-creation (Address Configuration / Autocreation for the pool numbers). Soundcheck creates conversations only through createConversation when a manager creates an event chat. If auto-creation is on, an inbound SMS to a pool number can spawn an orphan conversation that:

has no Postgres event_chats row
never receives the checkai participant
never fans out push
burns a proxy binding

Leave auto-creation off in every environment that uses real Twilio, including dogfood.

Messaging Service for the pool

US 10DLC numbers are registered to a campaign through a Messaging Service, so the pool numbers do belong to a Messaging Service. Use a dedicated, chat-only Messaging Service tied to the A2P “Proxy” campaign. Never use the Messaging Service, sender number, or campaign that sends invitations, reminders, or other Programmable Messaging traffic.

Path	Number	Messaging Service	Inbound route
Invitation / reminder SMS	`TWILIO_FROM_NUMBER`	Invitation Service and campaign	`POST /webhooks/twilio` (`TWILIO_WEBHOOK_URL`)
Event-chat SMS bridge	`TWILIO_GROUPCHAT_NUMBERS`	Chat-only Service on the A2P Proxy campaign	Conversations service webhooks → `POST /webhooks/twilio/conversations`

Set the chat-only Service’s incoming-message handling to route to Conversations. Do not point its “A message comes in” webhook at an API route, and keep conversation auto-creation off (see Auto-creation disabled). A Service that handles inbound SMS itself takes replies away from Conversations, so they never become onMessageAdded on the conversation. Conversations binds MessagingBinding.Address + ProxyAddress directly. The welcome and HELP replies use Programmable Messaging with an explicit From of the member’s pool number. They do not pass a Messaging Service SID and never use the invitation sender.

A2P 10DLC

US carriers filter unregistered application-to-person SMS.

Local 10DLC pool numbers need a registered brand and a campaign whose use case matches group-chat / customer-care traffic to crew phones. Approval is a Twilio / TCR process — do it in the Console; do not invent a campaign from this repo.
Register the pool in the chat-only Messaging Service on its own “Proxy” campaign. The first text each phone gets is the welcome, which carries the sender, program, and STOP/HELP disclosure that campaign review expects.
Do not hang the chat pool off the invitation sender’s campaign or (if used) toll-free verification. Invitation traffic and group-chat traffic are different use cases and different numbers.
Toll-free invitation senders (see handoffs/eng-418/) do not substitute for 10DLC on local proxy numbers.
Until the campaign is APPROVED, treat SMS bridge as blocked even if in-app chat works. Dogfood only after at least one US handset sends and receives a group-chat SMS through a pool number.

STOP / carrier opt-out is primarily handled by Twilio and the carrier on the proxy number. When Conversations delivers a STOP-family keyword, api-events rejects it in the pre-action webhook and removes the SMS member (Twilio first, then Postgres). The post-action fallback attempts member cleanup first and provider-message deletion independently and skips persist/push; either failure is retried, including after membership is already gone. Carrier-level suppression may prevent the webhook from seeing every STOP.

API keys and secrets

Conversations REST and AccessToken JWTs use an API Key pair, not the account auth token.

Name	Shape	Used for
`TWILIO_ACCOUNT_SID`	`AC…`	AccessToken issuer; shared with Programmable Messaging
`TWILIO_AUTH_TOKEN`	secret	`X-Twilio-Signature` on both webhook routes
`TWILIO_API_KEY_SID`	`SK…`	Conversations REST Basic auth + JWT signing
`TWILIO_API_KEY_SECRET`	secret	Pair for the SID above
`TWILIO_CONVERSATIONS_SERVICE_SID`	`IS…`	Service scope
`TWILIO_GROUPCHAT_NUMBERS`	E.164 list	SMS proxy pool
`TWILIO_CONVERSATIONS_WEBHOOK_URL`	`https://…/webhooks/twilio/conversations`	Signature URL
`TWILIO_CONVERSATIONS_ENDPOINT`	emulator URL	Local / e2e only

The auth token cannot sign Conversations JWTs. Missing SK / secret / IS throws twilio conversations client is not configured on create/token. Store SIDs and secrets in the environment secret store (Encore secrets / GCP Secret Manager — see infra-gcp/modules/events). Do not commit values. Rotate the API Key by creating a new SK…, deploying it, then deleting the old key.

Feature rollout

Order matters. Do not load the Conversations secrets until the Twilio side is live: once they are valid, the server treats event chats as configured, and the event-chats flag decides which orgs see them.

Dedicated IS… service exists; auto-creation off.
Canonical pre/post webhooks point at /webhooks/twilio/conversations.
2–4 pool numbers purchased, in a dedicated chat-only Messaging Service (not the invitation Service), not equal to TWILIO_FROM_NUMBER.
A2P 10DLC brand/campaign approved for those numbers (US).
Secrets populated (SK / secret / IS / pool / conversations webhook URL). Confirm TWILIO_CONVERSATIONS_ENDPOINT is unset in the deploy.
Roll a revision so it picks up the secrets. Hit POST /webhooks/twilio/conversations with a bad signature and expect 403; for an org with the flag off, a valid signed callback still persists provider state but terminally suppresses notification delivery.
Roll PostHog event-chats to dogfood orgs, then wider. Web, mobile and api-events read the same flag, so there is no second server toggle to keep in step.

Monitoring

Signal	Where	Meaning
`service: twilio_conversations`	Sentry (`captureServiceError`)	REST create/add/close/delete/send failures (except expected `50416`)
`twilio_conversations` / `sms_proxy_pool_exhausted` plus `event_chats.sms_proxy_pool_exhausted`	Sentry + `incrementCounter`	Pool exhausted for one member; identifiers are chat/event/org/member ids only. The request still returns a per-member failure.
`twilio_conversations` / `signature_verification`	Sentry (`captureWebhookError`)	Bad or missing signature
`twilio_conversations` / `message_persist`	Sentry	Post-event message persist, publication, or `last_message_at` work threw; the request returns non-2xx for Twilio retry
`notification.event_chat_message` (`outcome` = `pushed` / `skipped` / `replayed`)	Counter	Processor fan-out. Tags are low-cardinality only — never chat, user, or message ids.
Warn logs `sms group-chat number pool exhausted` / failed member reasons	App logs	Pool too small or leaked bindings
Warn `failed to persist… rolling back twilio participant`	App logs	Postgres/Twilio membership drift averted or rollback failed
`event_chats.twilio_lifecycle` (`operation` = `close`/`delete`, `outcome` = `failed`, `source` = `archive`/`event_lifecycle`/`event_purge`/`org_purge`)	Counter	Twilio close/delete recovery failed; inspect the retained pending job and its next scheduled retry
`event_chats.sms_opt_out` (`outcome` = `removed`/`already_gone`/`failed`)	Counter	Conversations webhook handled or retried a STOP-family SMS
`event_chats.sms_notice` (`kind` = `WELCOME`/`HELP`, `outcome` = `sent`/`skipped`/`deferred`/`failed`/`abandoned`/`lease_lost`/`reconciled`/`sent_unrecorded`/`lookup_deferred`)	Counter	Welcome and HELP delivery. For `abandoned`, inspect `effect_outbox` rows of kind `event_chat_sms_notice` in state `DEAD`
`twilio_conversations` / `sms_notice` and `deferred_sms_notice`	Sentry	A notice send failed or its worker threw; tags are kind, chat/member ids, and attempt count only
`event_chats.sms_help` (`outcome` = `queued`/`duplicate`/`skipped`) plus `twilio_conversations` / `sms_help`	Counter + Sentry	HELP intake. `sms_help` in Sentry means queueing or message deletion failed and Twilio will retry
Warn `sms disclosure lookup failed closed`	App logs	The pending-welcome check timed out or failed, so the disclosure suffix was appended without knowing
Warn `failed to close/delete twilio conversation`	App logs	Provider recovery failed; the worker throws and retains the durable job for retry
Expo push `data.type = event_chat_message`	Device / push delivery logs	In-app fan-out via the message processor (SMS members do not get this push; muted `event_chat` push prefs skip)

Pre-event onMessageAdd stays fast and synchronous for prefixing only. Post-event persistence and topic publication run on the request thread so Twilio can retry failures. Push fan-out remains on EventChatMessageTopic → eventChatMessageProcessor, not in the webhook handler.

Rollback

Turn PostHog event-chats off (for the affected orgs, or everyone). Routes disappear for clients; Twilio keeps POSTing signed callbacks, which continue persistence and cleanup while push delivery is suppressed. Within about six minutes the flag sweep removes affected users’ in-app participants, so an already-minted token stops reaching those conversations.
Leave the Conversations Service, webhooks, provider credentials, scheduler, reconciliation workers, and pool numbers in place. Deleting the service destroys message history and orphan-binds phones; queued cleanup and reconciliation still protect existing conversations while the flag is off.
If SMS must stop, managers archive chats before disabling because authenticated chat routes are then unavailable. Already-queued and event-lifecycle archive recovery continue while disabled; their durable close jobs retry provider uncertainty asynchronously.
Do not point pool numbers at /webhooks/twilio as a “safe” fallback — that mixes invitation reply codes with group SMS.

Re-enable by turning the PostHog flag back on; the sweep re-adds in-app participants, and existing event_chats rows and CH… SIDs remain valid if the same IS… is still configured. Pushes already accepted by Expo cannot be recalled.

Reconciliation

Concern	Source of truth	Repair
Chat name, archive, membership rows	Postgres	Rename commits DB first; Twilio `FriendlyName` is best-effort. Archive intent and its durable close job commit before provider work, with no transaction held across HTTP. A per-chat session lock on one leased database connection serializes manual and lifecycle requests; contention returns a retryable failure. The worker retains failed or uncertain close work for asynchronous retry, and `404` completes cleanup idempotently.
Message bodies	Twilio Conversations for live SDK history; Postgres webhook mirror for REST/MCP	`onMessageAdded` upserts by Twilio message SID; content changes and deletes advance citation version and status. A newer same-body update advances only the provider ordering timestamp, preventing a delayed older edit from overwriting the latest body. Replays publish deterministic event IDs for ProcessedEvent dedup. Remove-before-add creates a tombstone. Reconcile a missing mirror row by replaying the signed post-event webhook or backfilling from Twilio.
SMS join intents	`event_chat_sms_join_intents`	One row per SMS member mid-join. Cleared by the membership commit, a definitive bind failure, or a rolled-back activation; ignored after 10 minutes and swept by the minute job. Chat delete cascades.
SMS welcome and HELP notices	`effect_outbox` (`event_chat_sms_notice`)	One effect per welcome (per membership) or HELP reply (per message). The eager attempt and the minute worker share its claim; failed sends back off, and six failures set `DEAD`. `requeueDeadEffects` re-queues it.
In-app / SMS participants	Postgres membership plus revocation snapshot	`persistMember` deletes the Twilio participant if the DB insert fails. If rollback also fails, a last-resort upsert preserves the participant SID or the request fails loudly. Claimed SMS members reconcile to CHAT on join. Removal commits the revocation snapshot and idempotent participant-sync job together before provider revocation; effective access excludes the snapshot until membership is restored. The worker retries provider reconciliation, and `404` is already gone.
Create with zero successful members	Neither	Rolls back the Twilio conversation first, then the chat row. A failed Twilio delete keeps the local row and SID for retry.
Event / organization delete	Postgres transaction and `effect_outbox` (`twilio_conversation_deletion_v2`)	Conversation SIDs are queued in the same transaction as the cascade. The maintenance worker deletes Twilio conversations after commit and retains failed work for retry; `404` completes cleanup idempotently.
Event archive / cancel	`closeChatsForEvent`	Each chat commits archive intent and a durable close job under the same session lock as manual archive, then publishes a prompt. Queue-write failures propagate; provider close failures remain in the background job for retry. Already archived conversations retain their marker and can re-queue close work.
Organization member leave	`revokeChatAccessForOrgUser`	Provider revocation must succeed before local access is removed; failures remain retryable.
Last invitation deletion	Postgres transaction after provider revocation	Invitation mutations use per-item savepoints; last-active roster/chat cleanup uses the final batch state in the enclosing transaction. Payable voids and deletions commit with cleanup. Cleanup failure rolls back the batch and retains invitations for retry; prepared provider access is restored best-effort. Unprepared last-active cleanup fails instead of being skipped.
In-app self-leave	`POST /api/v1/chats/{chat_id}/leave`	Removes the caller’s Twilio participant first (`404` = already gone), then the local row. Removing anyone else stays manager-only `DELETE …/members/{org_member_id}`.
Proxy bindings	Twilio uniqueness + `activeProxyAddressesForPhone`	Walk the pool; treat `50416` as “try next”. If Twilio and Postgres disagree, prefer Twilio: a leaked binding burns a slot until the conversation is closed or the participant is removed.

An ambiguous close failure reads back Twilio state: closed or missing is success, while any other or unavailable state retains the archive marker and durable close job and returns a retryable failure. Even an active read cannot exclude the original timed-out request completing later. Twilio error 50377 also confirms the final closed state. Successful close requests make no extra read; each close attempt uses the existing bounded GET retry budget; a later queued retry can reassert the close POST. The archive marker and close intent commit together before provider HTTP. The archive-close worker runs every minute, claims up to five jobs for a five-minute lease, and rechecks each pending intent under the same per-chat session lock as manual archive. Successful closure acknowledges the job; provider or acknowledgement failure retains it for retry, even when the chat feature is disabled. Definitive provider failures also retain the archive marker and close job for retry. Already-archived retries ensure intent exists before HTTP, and the migration seeds existing archived chats. Deleting the chat cascades its close job; the separate deletion queue then owns provider cleanup. Recovery failures use event_chats.twilio_lifecycle with source=archive_recovery and Sentry operation deferred_conversation_close. Participant removal records durable intent in event_chat_member_revocations before provider revocation. The saved transport identities survive local membership deletion so queued retries can finish cleanup. Revocation, restoration, re-addition, and participant reconciliation share a per-member session lock; roster removals acquire every affected participant key in sorted order on one lock-pool connection and retain them through preparation, the short roster transaction, and compensation. A retained membership row does not cancel removal intent. Prepare and org-leave set retain_local_snapshot so the reconcile worker never deletes that row — it converges provider absence and leaves row and marker for the business removal, which deletes the exact snapshot and enqueues a fresh generation in one transaction; a later pass retires the marker once the row is gone. Direct remove, leave, and STOP record the exact snapshot identity with retain false; if provider revocation succeeded but the local snapshot delete did not, reconciliation CAS-deletes that leftover and ACKs. Explicit restoration first verifies that the retained row is still the exact prepared snapshot, then clears the marker and bumps the generation in one transaction before provider work, so a failed eager restore leaves queued PRESENT recovery that restores the persisted membership channel. SMS recovery also removes a stray CHAT identity left by an interrupted explicit join. A membership update newer than the revocation also supersedes it. Archived chats still reconcile effective revocations using the saved transport identities; only effective PRESENT work is skipped. Each participant-sync enqueue advances a generation, so an older worker cannot acknowledge newer work. Prompt delivery accelerates processing, and the minute-based worker retries durable pending jobs after lease expiry. The deletion queue retries committed cleanup automatically. If initial conversation creation succeeds but neither the chat row nor a deletion job can be persisted, the request attempts direct provider cleanup and reports any cleanup failure. There is no general nightly scan for other divergence. After an incident, list conversations on the dedicated IS… and compare CH… SIDs to event_chats.twilio_conversation_sid. Orphans (Twilio only) should be closed or deleted by hand. DB rows whose conversation is gone should be archived. An advisory-lock connection failure aborts the protected work’s signal and makes its ownership checks fail. Provider calls receive that signal, and mutation boundaries assert ownership before continuing; the failed lock connection is destroyed instead of returning to the pool. The wrapper propagates failures before, during, and after protected work without terminating unrelated requests. Already-issued provider calls remain subject to reconciliation. The separate four-connection lock pool and lock ordering are unchanged.

CI and emulator limits

The encore-e2e job in .github/workflows/api-events-deploy.yml starts tools/twilio-emulator on :4013, boots api-events with the Conversations test credentials pointed at that emulator and FEATURE_FLAG_OVERRIDES including event-chats, and runs e2e/tests/event-chats/crud.test.ts with E2E_EVENT_CHATS=1. Locally the CRUD file is still skipped unless that flag is set. See e2e/README.md for the exact env (seeded AC000…0 / SK000…0 / IS000…0, and TWILIO_GROUPCHAT_NUMBERS). e2e/tests/event-chats/routes-probe.test.ts always runs: unauthenticated probes get 401 before the kill switch, which registers route-coverage hits in both flag states. Known emulator limits:

Not production Twilio (no A2P, no carrier STOP, no real SMS, no AccessToken validation).
Conversations webhooks are not fired by the emulator; e2e posts signed onMessageAdd / onMessageAdded / onMessageUpdated / onMessageRemoved bodies to /webhooks/twilio/conversations.
DELETE …/Participants/{sid} is implemented so member remove and user.deleted revoke can be asserted.
Signature checks still use the account auth token over the exact webhook URL (E2E_CONVERSATIONS_WEBHOOK_URL or ${E2E_API_URL}/webhooks/twilio/conversations).

Do not point CI at a real Twilio account. Deployed environments must leave TWILIO_CONVERSATIONS_ENDPOINT unset.

Dogfood checklist

Do this against the target Twilio account after secrets are set. Do not buy numbers or register A2P from the agent.

Integrations — Twilio SMS + chat env block
api-events operations — secret inventory
Environment variables
API testing — Vitest + e2e gates
Chat (product)

Stable message pagination

For stored history, pass before as the last message’s id from the previous page to list_event_chat_messages or GET /api/v1/chats/{chatId}/messages. This anchors the next page to immutable creation time and id, so newer arrivals do not repeat or skip older messages. Do not combine before with the legacy offset parameter. Organization deletion queues Twilio conversation SIDs in the deletion transaction. The internal cleanup worker runs every minute and retries failed deletions after a five-minute lease. A database rollback therefore preserves provider conversations and history; a provider outage leaves durable work for later retry.

Overview

Backend

Backend integrations

API reference

Frontend

Mobile

Infrastructure

CI/CD

Development

Guides

Reference

What is shipped

Dedicated Conversations Service

Canonical pre-event and post-event webhooks

SMS notices: welcome and HELP

Welcome before the first group message

Durability

Proxy pool (2–4 numbers)

Auto-creation disabled

Messaging Service for the pool

A2P 10DLC

API keys and secrets

Feature rollout

Monitoring

Rollback

Reconciliation

CI and emulator limits

Dogfood checklist

​What is shipped

​Dedicated Conversations Service

​Canonical pre-event and post-event webhooks

​SMS notices: welcome and HELP

​Welcome before the first group message

​Durability

​Proxy pool (2–4 numbers)

​Auto-creation disabled

​Messaging Service for the pool

​A2P 10DLC

​API keys and secrets

​Feature rollout

​Monitoring

​Rollback

​Reconciliation

​CI and emulator limits

​Dogfood checklist

​Related

​Stable message pagination

What is shipped

Dedicated Conversations Service

Canonical pre-event and post-event webhooks

SMS notices: welcome and HELP

Welcome before the first group message

Durability

Proxy pool (2–4 numbers)

Auto-creation disabled

Messaging Service for the pool

A2P 10DLC

API keys and secrets

Feature rollout

Monitoring

Rollback

Reconciliation

CI and emulator limits

Dogfood checklist

Related

Stable message pagination