> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soundchecklive.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Durable asynchronous workflows

> Architecture decisions and invariants for publication, reconciliation, commands, webhooks, and long-running work.

This is the target architecture and review contract for reliable asynchronous side effects in Soundcheck. It does not assert that an outbox or every invariant below is already implemented. The [current feature assessment](./async-workflow-assessment) records the inspected baseline, gaps, and migration order. Use the repository's `reliable-async-workflows` skill when designing or changing one of these workflows.

## Decision and boundaries

For business changes that require reliable asynchronous consequences, prefer **business state plus outbox in one transaction → reliable relay → Pub/Sub → independent retry-safe workers**. PostgreSQL remains authoritative for local business state. Pub/Sub carries work to consumers; it is not a business ledger or a distributed transaction with PostgreSQL or a provider.

Keep a database-backed work queue when it directly fits a bounded job or desired-state reconciliation. It is a valid architecture, not technical debt merely because it polls. Introduce a broker when independent consumers, workload isolation, fan-out, or dispatch latency justify it. Adding Pub/Sub to an existing queue is optional, not a prerequisite for an outbox migration.

This decision does not require microservices, event sourcing, a workflow-engine purchase, or moving every API operation to a queue. Keep authorization, validation, ordinary reads, local transactional invariants, and response-critical provider interactions synchronous unless an explicit product/API change permits asynchronous acceptance.

## Select the durable fact first

| Workflow | Durable fact and execution rule | Completion means |
| - | - | - |
| Local request/transaction | An authorized business mutation; return its actual result | The local transaction committed |
| Immutable event and notification fan-out | A versioned fact or frozen delivery decision, with a stable event ID; preserve event-time content and deterministic child identities | Each required consumer has completed or durably handed off its own work |
| External command | A specific authorized instruction with immutable inputs and an operation/idempotency key, such as paying a particular amount or sending a contract version | Provider outcome is durably resolved; acceptance and settlement may be different states |
| Desired-state reconciliation | An entity's latest desired state plus a durable dirty/version marker; reread it and converge the provider | The claimed generation has converged, with later generations still discoverable |
| Webhook ingestion | Authenticated provider event identity and durably accepted content or reference | Receipt is stored/queued; downstream business completion is separate |
| Multi-step workflow/saga | Persisted execution, step identities, outcomes, timers, and explicit compensation states | All required steps or a recorded terminal recovery outcome are resolved |
| Scheduled work or drift repair | Due time/watermark, eligibility, and a resumable bounded cursor where necessary | The due work is durably dispatched or the scan checkpoint advances safely |
| Optional telemetry | Explicitly accepted loss/duplication policy and bounded resource use | Best effort, with no business correctness dependency |

An event, a command, and a reconciliation hint need different semantics even if they use the same relay and broker. A payment must not change amount on replay because a row changed. A membership hint usually must not restore an obsolete membership snapshot. Freeze notification content where replay must reproduce the original decision; explicitly define whether current consent, membership, or cancellation suppresses an old delivery.

## Target flow and ownership

```mermaid theme={null}
flowchart TD
    request[Authorized request] --> txn[One DB transaction: business state and outbox]
    txn --> response[Return committed result or accepted operation]
    txn --> relay[Relay claims unpublished events]
    relay --> broker[Pub/Sub confirms publication]
    broker --> worker[Worker claims event or operation]
    worker --> provider[External side effect]
    provider --> result[Persist outcome, child outbox work, and completion]
    result --> ack[Acknowledge delivery]
    worker --> retry[Retryable failure: throw or negative acknowledgement]
    retry --> broker
    broker --> dlq[Exhausted delivery: dead-letter recovery]
```

| Record/mechanism | Owns | Does not establish |
| - | - | - |
| Outbox | Reliable handoff from a committed local transaction to the broker | Provider execution or consumer completion |
| Broker subscription | Delivery/redelivery within configured retention and retry policy | An event that was never published, unlimited retries, or atomic provider effects |
| Receipt/ingress inbox | Durable acceptance and identity of an incoming message | Successful downstream processing |
| `ProcessedEvent` completion inbox | Terminal completion for one `(event_id, consumer_name)` | A desired-state queue or exactly-once provider calls |
| Pending operation / reconciliation state | Unresolved work, generation, ownership, and recovery | Proof that a broker event is complete |
| `NotificationDelivery` | Notification attempt/outcome audit | Generic participant sync, payment, or job scheduling |

These distinctions prevent a common failure: treating the presence of a receipt, an audit row, or a broker message as proof that the entire business operation succeeded.

## Required invariants

### Durable acceptance and publication

1. **Atomic intent.** If a committed mutation requires asynchronous work, commit its outbox event or pending operation in the same database transaction. A post-commit `publish()` is a second write even when awaited. A catch-and-log or fire-and-forget publish is not a durable handoff.
2. **Honest HTTP results.** Do not report durable asynchronous acceptance if storing that intent failed. Preserve the endpoint's existing success/error contract; switching to `202` or returning an operation ID is a product/API decision. A provider failure may be decoupled from the response only after recoverable work exists.
3. **Confirmed relay progress.** Claim in bounded batches, publish outside the business transaction, and mark the claimed outbox item published only after the broker confirms. Crashing after publication but before recording it causes duplicate publication; retain the same application event ID. The relay must run in a lifecycle with guaranteed execution, not unawaited request code.
4. **Durable every hop.** A consumer that updates local state and creates downstream work commits that state, child outbox events, and its own completion atomically. For an existing direct-publish fan-out, await all required child publications before sealing the parent, use deterministic child IDs, and safely retry partial fan-out. Do not acknowledge the parent while a required child exists only in memory.
5. **Minimal versioned envelopes.** Define event ID, type/schema version, tenant/entity identity, operation or aggregate version where needed, and correlation/causation identifiers. Carry immutable command inputs or a durable reference when needed; a lookup of mutable state is not a substitute for a historical payload. Avoid provider credentials and unnecessary personal data in retained events.

The database/broker gap and duplicate-publication consequence are the reasons for the [transactional outbox pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html). Existing direct webhook-to-broker ingestion can also be durable without a database outbox when there is no accompanying business transaction: authenticate, await confirmed publication, and return failure on publication failure so the producer can retry. Document the provider's actual retry window.

### Consumption, concurrency, and provider outcomes

6. **Receipt is not completion.** Use separate states for accepted, running, and terminal. Seal `ProcessedEvent` only after successful effects, a documented terminal skip, or a durable handoff that has transferred responsibility. Retryable failures must propagate. Persist terminal rejection or manual recovery separately so “processed” is not mislabeled “delivered.”
7. **Two idempotency boundaries.** Application event IDs deduplicate deliveries; business operation keys protect external actions. A read-before-send inbox check alone does not prevent concurrent sends or a repeat after provider success followed by a crash. Use provider idempotency, provider lookup/reconciliation, or explicitly documented duplicate/loss tolerance. Never claim an inbox makes an external call exactly once.
8. **Fenced ownership.** Coordinate all eager, subscriber, and scheduled runners through the same operation claims. Lease expiry can leave an old runner alive: completion must check the claim token/generation, and newer work must not be deleted by an old attempt. Locks and leases have different purposes; Pub/Sub acknowledgement deadlines, DB claim leases, request timeouts, and retry delays are not interchangeable. Bound calls or renew leases, and account for providers that cannot accept fencing tokens.
9. **No lost re-enqueue.** When coalescing pending work, update a desired generation or preserve a dirty marker without stealing the current lease. A worker completes only its claimed generation. `ON CONFLICT DO NOTHING` plus unconditional deletion is insufficient if a later mutation can arrive while the row is in flight. Preserve serialization where provider operations cannot safely overlap; ordering keys alone do not serialize database writes or side effects.
10. **Preserve cleanup identity.** Deleting a business row must not erase the only provider ID or cleanup instruction. Store required provider references in the cleanup operation/tombstone before deletion; a cascade is safe only when that work is truly obsolete. Treat provider “already absent” as success only for operations whose desired state is absence.
11. **Explicit failure classes.** Retry transient failures with bounded concurrency, backoff and jitter, honoring provider throttling. Distinguish permanent rejection, obsolete work, and ambiguous external success. Quarantine poison messages and record a recoverable terminal/manual state rather than retrying invalid or unsafe instructions forever. “Retry until converged” is conditional on valid desired state and eventual dependency recovery.

For PostgreSQL queues, use short claim transactions and `FOR UPDATE SKIP LOCKED`, an eligibility index, and the repository's SQL timezone conventions. Do not hold ordinary business transactions across slow provider calls. Existing domain locks that protect an external ordering invariant must be preserved until a reviewed state/version design replaces them; extract shared coordination rather than silently dropping locks in a new subscriber.

### Recovery and operability

12. **A dead-letter topic is not recovery.** Assign an owner, alerts, retention sufficient for the response window, and a tested inspect/replay procedure. Replay preserves the original event/operation identity; a broker redrive or new message ID must not bypass deduplication. A corrected business instruction gets a new explicitly authorized operation/version. Do not clear completion inboxes indiscriminately to force replay.
13. **Retention covers replay.** Choose inbox/receipt retention from the full source, retry, dead-letter, manual-replay, and archival windows. Preserve financial operation identities for their domain lifetime where required. Define what happens when a replay is older than the deduplication window. Outbox cleanup removes only confirmed published records under the documented replay/audit policy.
14. **Measure unfinished business work.** Track oldest unpublished intent, dispatch latency, oldest unfinished operation, completion latency, retries by class, lease expiry, provider throttling, and dead-letter growth/age. An empty broker backlog does not prove reconciliation succeeded if messages were acknowledged, expired, or quarantined. Correlate request → event → operation → provider outcome without logging secrets or unnecessary payloads.
15. **State assumptions.** Application-level durability assumes the authoritative database, configured broker retention, and recovery processes remain available or recoverable. Set DB backup/restore and disaster-recovery objectives separately. An outbox cannot promise zero data loss from an unrecoverable database failure.

Pub/Sub's [dead-letter forwarding](https://docs.cloud.google.com/pubsub/docs/dead-letter-topics) uses approximate attempt counts and needs appropriate IAM and a recovery consumer. Verify the effective Encore retry policy, subscription policy, acknowledgement behavior, retention, and DLQ together in a deployed test environment. Do not infer the full retry lifecycle from either the SDK or Terraform alone.

## Domain recipes

### Notifications and fan-out

Keep independent channel/recipient delivery units and deterministic per-target child IDs. A successful SMS must not be repeated because email failed. Completion of the orchestrator means its required children are durably accepted, not that they all arrived. Snapshot recipients when every original recipient must receive a historical notification; use paginated/checkpointed fan-out for large audiences. Document invalid tokens, opt-outs, stale invitations, and expired notifications as explicit terminal decisions. Loss-tolerant push suppression is not a reusable payment or access-convergence policy.

### Twilio conversations and membership

Use a shared execution boundary for close, delete, archive-close, and participant reconciliation while retaining operation-specific preconditions. Apply the current committed membership or terminal conversation state; do not reopen an archived conversation from an old message. Preserve the archive/conversation ordering lock where that implementation requires it. Missing membership can make a restore intent obsolete, but it does not by itself prove that a required provider removal occurred; represent removal separately when needed. A delete must retain the conversation SID after its local chat row disappears.

Invitation and identity handlers should own their local mutation and durable intent, while Twilio workers own provider convergence. Required synchronous revocation or authorization guarantees remain explicit exceptions until the product accepts eventual enforcement. Do not use a notification audit table or a synthetic completed inbox event as the pending participant queue.

### Contracts, payments, and other irreversible commands

Persist an authorized, immutable instruction and provider operation identity before execution. A contract send pins its document/recipient version; a payment pins amount, currency, payee, account identity, and authorization. Treat timeout-after-submit as unknown, not as permission to submit a new instruction. Separate submission retries from provider status polling and settlement reconciliation. Compensation is a new governed domain action, not automatic rollback of an external transaction. Do not introduce generic automatic payment retries that bypass existing uncertain-outcome guards.

### Identity and provider webhooks

Verify authenticity and tenant scope before durable acceptance. Deduplicate by provider event identity, handle reordering/deletion/tombstones, and durably store an ingress receipt or await confirmed broker acceptance before early acknowledgement when deferring processing. Keep request-time authorization checks and required provisioning invariants; an asynchronous projection must not grant stale access. Never return `2xx` for required processing that failed before any durable owner accepted it.

### Storage lifecycle and generated artifacts

Object storage and PostgreSQL do not commit atomically. Before deleting/replacing metadata, retain object identity and durable cleanup intent; an uploaded object whose metadata write fails needs compensating cleanup or orphan discovery. Cleanup must verify ownership/references and use conditional deletion where supported so it cannot remove a newly replaced object. Keep orphan sweeps where they detect failures outside known queued work; a dispatch outbox does not make them redundant.

Large export/PDF generation can use a durable artifact job with source revision or as-of/filter semantics, checkpoints and output metadata. Recheck download authorization, expire artifacts, and include tenant/role-sensitive content in cache boundaries. Ordinary bounded downloads, draft AI previews and aggregate reads can remain synchronous; add background jobs or materialized views only for a stated latency, freshness or capacity need.

### Imports, CRM synchronization, and agent workflows

Long imports and synchronization runs need a persisted job, immutable source/version, bounded batches, checkpoints, per-record outcomes, and cancellation semantics. Replaying a job must not duplicate already-applied records or CRM actions. Human review/approval and long waits fit a persisted workflow/state machine with deadlines and compensations; a chain of topics alone is not a workflow engine.

For CheckAI, distinguish conversation identity, inbound-message identity, workflow execution ID, approval ID, and tool-action idempotency key. Session history and tool annotations do not prove durable action execution. Persist resumable workflow state and bind approvals to the exact operation; recheck validity before delayed execution. Keep API/MCP reads and immediate validation synchronous, and expose job progress explicitly if a tool becomes asynchronous.

## Scalability end-state

Share the envelope, relay mechanics, observability and replay tooling while keeping domain handlers and business ledgers distinct. Start with bounded polling of an indexed outbox and a small worker deployment; adopt CDC only when measured latency/DB load and operational capacity justify its lifecycle and replication overhead.

Partition execution by provider/workload and tenant fairness. Tune outstanding messages/bytes, in-flight operations, database connections, provider quotas, and maximum worker instances together. More subscribers cannot overcome a provider quota or an exhausted database pool. Use independent subscriptions for independent consumers; use competing workers on one subscription to distribute one consumer's work. Avoid topics or services per entity.

Keep HTTP capacity isolated from slow integrations when evidence warrants separate worker deployment. For pull consumers, verify how the chosen runtime actually scales; queue depth alone is not automatically a scaling signal. Use measured oldest-work age and throughput to size/scale workers, and apply [subscriber flow control](https://docs.cloud.google.com/pubsub/docs/pull#flow_control) rather than unlimited parallelism.

Before migrating each workload, agree numerical objectives for accepted-to-start and accepted-to-complete latency, outage/recovery window, tolerated duplicates/loss, and resource budgets. Test those objectives under realistic bursts and dependency outages; this document does not claim current production throughput or select unmeasured capacity targets.

## Migration and the cron decision

| Stage | Durable owner | Cron/scheduler role |
| - | - | - |
| Database work queue | Pending operation committed with domain state | Required dispatcher/retry runner; an eager attempt is optional |
| Pending queue plus best-effort Pub/Sub | Same pending operation; subscriber is a prompt to run it | Still required for lost publication and unfinished work |
| Outbox plus broker workers | Outbox until confirmed publication; worker/broker or durable operation thereafter | Domain retry cron can retire after recovery proves equivalent; relay may poll |
| Scheduled business work / provider settlement / drift repair | Persisted due work, unsettled operation, or scan cursor | Still needed; these jobs serve different purposes from dispatch retries |

Move one representative workflow at a time. Build relay and workers without prematurely removing an existing recovery mechanism. Subscriber and cron must share claims during overlap. Before disabling a domain retry cron, prove atomic intent, restart-safe relay delivery, retry-safe provider execution, dead-letter alert/replay, generation safety, and handling of every existing pending row. Define rollback that resumes draining without replaying successful irreversible actions. Do not require every workflow to migrate in the same PR.

A bounded post-commit eager attempt is permitted only after durable recording and with the same coordination. It is optional latency optimization. Awaiting it still adds response latency; scheduling it in memory after the response does not guarantee execution. Prefer worker-only provider execution unless the measured user benefit justifies another runner.

## Required evidence for an implementation PR

Use focused tests appropriate to the selected pattern, rather than matching documentation wording:

| Failure scenario | Required observable outcome |
| - | - |
| Domain transaction rolls back | Neither state nor required outbox/pending work survives |
| Crash after commit, before publish | Relay/queue runner eventually discovers the work |
| Publish succeeds, relay crashes before recording it | Duplicate delivery retains identity and honors the domain's required idempotency or explicitly accepted duplicate policy |
| Provider succeeds, worker crashes before recording it | Idempotent retry or reconciliation resolves the original operation safely |
| Two runners overlap or a lease expires | Ownership checks and provider coordination protect completion |
| Desired state changes during an attempt | Old completion cannot discard the newer generation |
| Partial fan-out or a failed downstream publish | Missing children remain recoverable; durably completed children are skipped and ambiguous outcomes follow the provider-specific duplicate policy |
| Provider outage, throttling, or poison event | Bounded resources, correct failure classification, alertable recovery |
| Dead-letter replay or deployment rollback | Original identities, compatibility and audit history remain intact |
| Business row is deleted before cleanup | Necessary provider reference and cleanup intent remain available |

Unit and real-database concurrency tests establish application behavior; deployed broker/provider-emulator failure drills establish the actual handoff and recovery contracts. Approval to implement a narrow feature does not imply approval to replay production side effects or deploy infrastructure.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.