Async workflow assessment and roadmap

Assessment baseline: dev commit fde0a61e1de4f5b31ddb4d7c44dbb4a6b1e3c51a, inspected September 12, 2026. Source links below pin that revision. This is a representative assessment across current feature families, not a production configuration, load, or disaster-recovery certification. Runtime/provider behavior and capacity remain to be verified in deployment. The durable workflow contract defines the target invariants; this page distinguishes current evidence from recommendations.

Findings that change the architecture decision

There is no shared transactional outbox in this baseline. The API README explicitly says so. Producers include both detached and awaited publications; neither makes a separate business DB commit atomic with Pub/Sub. The old Go outbox service-account names in Terraform do not establish a running TypeScript outbox.
Pub/Sub and some useful consumer patterns already exist. Availability fan-out awaits child publication, uses stable per-device identities, and records terminal outcomes. This is a foundation to preserve, not a reason to replace the broker.
Consumer guarantees differ. Invitation and crew-offer consumers can seal completion after caught delivery failures. Checklist push deliberately favors avoiding duplicates after an expired claim, accepting possible loss. An outbox alone does not change those policies.
The earlier Twilio PR #1703 discussion described pending close/delete/participant queues and chat-push subscribers. Those artifacts are absent from this baseline. This assessment evaluates current direct Twilio operations, not that proposed branch. Reassess the relevant row when that work lands.
Some durable domain state is already mature: payment instructions and provider keys, prerequisite webhook queues, import previews/jobs, CRM ingestion claims, and targeted reconciliation sweeps. Reuse them rather than flattening every workflow into ProcessedEvent.

Messaging, staffing, and chat

Feature and evidence	Current behavior and implication	Recommended end-state / constraint
Availability requests, responses and channel/device fan-out: producer, orchestrator, SMS leaf, push fan-out	Initial publication can be detached after domain writes. Parents await child publications; device IDs are deterministic; leaves persist audit plus terminal inbox and throw on failure. The provider call still precedes completion recording.	A contained first outbox migration: atomically record the initial notification decision; preserve existing leaf isolation and child identities. Test partial fan-out and provider-success/crash duplicates. Add claims/provider-specific duplicate policy; inbox presence checks alone are insufficient.
Invitation notifications and explicit send: dispatcher, consumer, send operation	Automatic publication can fail after membership/status changes. The consumer can catch channel failures and mark the parent processed. Explicit send calls the processor directly and reports channel attempt results in HTTP.	Outbox for automatic notifications plus retryable per-channel work. Define stale/cancelled notification policy. Preserve explicit send’s actual-result response or deliberately redesign it to accepted/pending with UI progress; worker-only execution is not contract-preserving by default.
Crew-offer responses and notifications: response side effects, side-effect implementation, notification consumer	Detached response work includes event membership, availability, payable and calendar changes. Notification helpers swallow send and persistence failures before parent completion.	High business-impact recovery priority. First classify which same-DB staffing/accounting invariants must commit with the response. Persist permitted asynchronous follow-on work, then split retryable delivery units. Do not make local ledger/capacity decisions eventually consistent just to add a bus.
Checklist assignment push: notifier, consumer	Publish is awaited with a detached direct-send fallback. Per-token claims/receipts exist; expired/reclaimed claims are deliberately treated as sent to limit duplicate noise.	Outbox can close dispatch loss, while the current at-most-once preference requires an explicit product decision. Keep it local to optional push; never reuse this suppression policy for access changes, money, or required cleanup.
Twilio archive/close: archive, event close	DB archive precedes provider close; provider errors are caught. Already-archived conversations can be retried when another caller invokes lifecycle close. There is no autonomous pending queue here.	Persist desired terminal-state work and dispatch independently. Preserve DB-first ordering and whatever archive/lifecycle coordination the implementation requires. A durable queue with a cron is a valid immediate fix; an outbox subscriber is the shared target.
Twilio delete/cleanup: purge, create rollback	Some paths log provider delete failure and then delete local chat state. Organization purge has different retry/failure behavior and uses pre-delete snapshots.	Persist provider SID/tombstone and cleanup intent before losing the business row. Cleanup must survive deletion and distinguish provider absence from a transient failure. A worker that only reloads the now-deleted chat cannot repair this.
Twilio access and membership: add/persist, remove member, user revocation	Provider add can precede local membership insert with best-effort compensation. Removal/revocation requires provider success or absence before local deletion. The SMS-to-chat claim implementation from the earlier plan is absent.	Use durable reconciliation/compensation for recoverable membership transitions. Preserve current fail-closed access removal until an explicit authorization/enforcement contract permits asynchronous revocation. Do not apply notification-style swallow-and-200 behavior universally.
Twilio message push ingress: webhook, push processing	The signature-verified webhook returns `200` before detached processing; processing calls Expo directly and logs failures. Current path lacks the later plan’s durable push-event identity and receipts.	Verify → durably accept provider event identity/content → acknowledge → fan out to device workers. Keep business message acceptance distinct from optional push-delivery guarantees and preserve recipient/tenant eligibility.

External integrations and business workflows

Feature and evidence	Current behavior and implication	Recommended end-state / constraint
Cleo ACH submit: instruction creation, payment processor, provider submission	Persists a pending instruction with frozen bank/contact/funding IDs, claims submission, and uses transaction ID as provider idempotency key. Ambiguous publish acknowledgement retains pending work.	Preserve the domain state machine and frozen amount/currency/destination. An outbox can reliably wake the existing processor; it must not substitute current default bank details or mint a new provider key on retry.
Cleo recovery and incoming events: sweep, webhook, ingress handoff	Sweep both republishes unsubmitted instructions and polls submitted payables. HMAC-verified ingress awaits broker publication before `200`, then may eagerly apply; a distinct ingress marker records acceptance.	An outbox can replace publication recovery, not provider settlement polling. Direct awaited webhook-to-broker handoff can remain valid when no business DB transaction must commit alongside it; acceptance dedup and final execution dedup remain separate.
Stripe charge/SCA and webhook projection: charge, webhook, prerequisite queue	Provider idempotency is tied to the transaction. Uncertain outcomes intentionally remain pending/processing. Single-item SCA can require a provider result in the normal HTTP response. Incoming events have classified retry behavior and deferral when the organization is missing.	Bounded synchronous initiation where required, durable financial operation state, and asynchronous final reconciliation. Audit webhook event classes before changing acknowledgement. Preserve SCA, financial guards and pending prerequisite replay; no generic retry may turn uncertainty into a second charge.
Clerk projection and downstream profile sync: webhook projection, Stripe profile sync	Synchronous verified webhook applies identity, replays pending relationships and provider events, and marks completion. Some downstream customer synchronization errors are logged and swallowed.	Keep authoritative identity/auth mutation and required revocation guarantees. Atomically record downstream facts/intents with projection; independent consumers own CRM/customer convergence and notifications. Preserve out-of-order parent handling and deletion tombstones.
HubSpot/Flex/n8n ingest: CRM ingress, claims, Flex stages	Stable external IDs, failed/stale claim recovery and transactional core upserts exist. Staffing/link work follows the core transaction; cancellation has explicit retry gates. A changed payload can reopen the same input ID.	Durable ingest job or staged state machine, with source revision/hash and field ownership. Outbox records work between stages. Distinguish immutable revisions from intentionally coalesced latest-state sync; preserve cancellation/accounting dependencies.
Outbound CRM settlement and manual sync: settlement notification, manual trigger	Automatic n8n settlement notification catches failures. Manual sync returns the provider trigger result; it does not establish a local independently drained job.	Automatic outbound updates are a useful outbox/desired-state pilot. For manual long syncs, introduce durable job acceptance and progress only with an explicit API/UI contract. A provider’s `202` is not proof of completed synchronization.
Large imports and approved previews: commit, retry state	Preview/job state, conditional committing status, batched row outcomes, and retry of committing/failed work exist, but execution is awaited by the request.	Strong scalability candidate: durable dispatch of an immutable approved preview revision to bounded resumable workers. Preserve per-record identities/results and authorization; outbox delivery alone is not the import orchestrator.

Specific provider policies continue to govern. Stripe’s normal SCA response and Cleo’s fail-closed org/user revocation sequence are intentional constraints, not incidental synchronous code to delete. Preserve financial instruction and provider identity guards; provider acceptance, execution, and settlement are different milestones.

CheckAI, MCP, and time-driven work

Feature and evidence	Current behavior and implication	Recommended end-state / constraint
Slack/SMS agent turns: Slack, SMS	Adapters delegate dispatch and replies to eve. Source wrappers alone do not prove the deployed runtime’s durable acceptance, message deduplication or crash resume.	Verify eve’s actual runtime/backend before adding another queue. Persist accepted runs under one owner and serialize conflicting session turns where needed. Reuse a proven workflow runtime rather than duplicating it.
Email agent turns and replies: email channel	Root message ID routes the conversation; outbound send failures can become a false result without a delivery job. Thread continuity does not prove individual inbound delivery or tool-action deduplication.	Separate conversation, provider-message, turn/run and outbound-action identities. If replies are required, persist delivery work independently of agent-turn completion.
Agent actions and MCP: HTTP backend, confirmation gate, tool audit	APIs/MCP provide direct authenticated calls. Confirmation token consumption uses an in-memory map; tool idempotency hints are annotations. Invocation reporting is awaited.	Approval authorizes an exact action; it does not make that action retry-safe. Consequential operations need server-owned durable action keys/results. Keep ordinary reads/validation synchronous; return operation status for intentionally asynchronous tools. Replace awaited audit only with a durable handoff.
Scheduled attention digests: org sweep	Weekday scheduler with opt-in dispatch and autonomous-write restrictions.	Keep scheduling. Add stable occurrence identity and durable run acceptance if completion is required; events may trigger/coalesce additional work. Human approvals and autonomous-write restrictions remain binding.
Structured ingestion compute: ingest endpoint	HMAC-authenticated, synchronous, zero-session bounded compute called by another service.	It can remain a bounded RPC inside an api-events-owned durable ingestion workflow. Persist source/proposal/progress and approved commits in the system of record; no topic is needed for every function boundary.

Ordinary CRUD/read paths, validation, auth, capacity decisions and transactional local invariants do not benefit from a broker simply because it is available. Optional telemetry/demo delivery may retain explicit best-effort semantics. Long-running approvals, waits and cancellation need persisted workflow checkpoints; a series of subscriptions is not itself resumable execution.

Lifecycle, intake, storage, and scheduled consequences

These producers and workflows need distinct recovery contracts even where they share notification, import or payment infrastructure.

Feature and evidence	Current behavior and implication	Recommended end-state / constraint
Event rescheduling, range reduction, cancellation and deletion: post-commit effects, range snapshots, cancellation, deletion	Updates order detached tails through a process-local map. Range reduction needs snapshots of removed resources that cannot be reconstructed later. Cancellation has useful state-based retries and final release verification; deletion performs several provider cleanups/notices after local deletion.	High-priority persisted per-event workflow, with versions, durable removed-resource/recipient snapshots and step outcomes. Preserve date/capacity/financial gates and cancellation-before-reschedule ordering. “Event status cancelled” and “crew release completed” remain distinct; in-memory ordering is not cross-instance recovery.
Google Calendar topology, attendees and cleanup: sync, provider insert	Real provider events are created before mappings are persisted. Some delete failures are caught before removing local mappings. Provider inserts can send attendee invitations.	Desired-state reconciliation by event/day with generation protection and durable provider cleanup identity. Resolve ambiguous creation before retrying so a crash does not produce duplicate calendar events/invitations. Preserve attendee/RSVP behavior and kill switches. Authenticated calendar feeds are a separate read path, not pushed synchronization.
QuickBooks accounting documents and voids: ledger side effects, provider sync	Sync/void work is detached. A provider document can be created before external-ID persistence fails and only logs; the create fast path depends on having that ID. The production client is implemented, with a separate explicit stub mode.	Durable accounting operations with ordered create/void instructions and reconciliation of provider-success/local-write ambiguity. Adding blind retries can create duplicate accounting documents. Keep OAuth and required chart-of-accounts validation synchronous; do not substitute fallback accounts after a connected-provider failure.
Event-file ingestion and AI merge: upload acceptance, unstarted/lease query, pipeline, recovery, merge coordination	Upload persists `UPLOADED`, starts detached work and returns `202`; a process-local three-slot limiter precedes the active lease. Recovery scans expired active stages, excluding accepted unstarted `UPLOADED` files, and marks stalled files failed for manual retry. Merge dispatch also uses local timers alongside persisted merge state.	High-priority durable acceptance/dispatch, distinct from CSV/import commit. Discover every accepted unstarted item; persist responsibility between analysis/merge stages and fence stale completions. Reuse file/batch/proposal records and approved versions. Coordinate AI concurrency across instances; distinguish manual retry from automatic resume.
Attachments, org/onboarding documents and orphan cleanup: attachment upload/delete, org-document deletion, contract sweep	Object-store upload and metadata writes are separate, with best-effort compensation. Deletion/replacement can lose references while blob cleanup fails. Contract PDF cleanup already pages with a persisted cursor, minimum age, reference checks and ETag-conditional deletion.	Persist cleanup identity/intent before deleting or replacing metadata; use upload operation state and orphan detection for object-first failures. Keep the contract sweep as orphan detection even after outbox adoption. Do not copy its contract-specific reference/age predicates blindly to other document types.
Automated closeout tasks, offer expiry and reminders: closeout producer, offer expiry, manual reminders	Closeout tasks have a unique source key, but later sweeps skip notification when the task already exists. Offer expiry changes status before dispatch, so a missed notice is no longer selected as overdue/open. Reminder endpoints create distinct user-requested sends.	Keep schedulers for elapsed-time discovery. Atomically couple required task/expiry notices to their state transitions, with bounded scans. Each new reminder occurrence needs a new logical identity, while retries of that occurrence keep it. Preserve optional-push policy where intentional.
Direct bulk/team notification sends: direct sends	Actual SMS/email sends and event-template fan-out run in the request and report results. Pending notification audit can fail to persist without preventing the provider call; it is not an independently drained queue.	Immutable batch/send command plus per-recipient/channel jobs can improve burst handling. Preserve actual-result HTTP behavior or explicitly introduce accepted/pending batch status. Do not assume a `PENDING` audit row implies retry ownership.
User/account deletion and organization provisioning: user deletion, org lifecycle	User deletion revokes Cleo, removes local state and then calls Clerk; Clerk failure returns an explicit error because provisioning could recreate the row. When Clerk is configured, org creation calls Clerk and returns accepted while awaiting webhook projection; the alternate path creates locally.	Persist unresolved deletion/provisioning lifecycle and tombstones if stronger recovery is required. Prevent reprovisioning during pending deletion; keep financial/revocation gates and honest response semantics. Verify the existing webhook handoff before adding redundant acceptance layers.
Resend contacts and email lifecycle analytics: contact sweep, analytics ingress, analytics consumer	Contacts use bounded hourly rotation and enrichment. Delivery analytics already awaits broker acceptance and uses a leased, retrying consumer; it is not a missing event bus.	Incremental dirty-contact reconciliation can improve freshness and reduce repeated scans, while a periodic sweep still covers time-derived/enrichment changes. Preserve rate limits and safe failed-fact handling. Keep lifecycle analytics’ required acceptance semantics and verify capture-success/before-seal duplicate policy.

Assessed boundaries that do not automatically need a broker

Calendar feeds and entitlements: feed rendering validates current token/access; subscription sessions must return usable Checkout/portal URLs, and entitlement enforcement belongs on the request path. Asynchronous billing/profile consequences must not delay revocation or authorize stale access. Some subscription-webhook helper exports lack a production caller in this baseline; routed behavior remains the Stripe webhook assessed above.
Availability autoclose: the existing sweep makes guarded local state transitions. Keep the timer and local transaction; it has no required provider worker merely because other expiry flows send notifications.
Workspace/audit exports and generated PDFs: workspace ZIP export, streaming audit export, and call-sheet PDF are implemented synchronous paths; workspace and audit exports enforce explicit limits. Larger/bulk/repeated jobs can benefit from durable artifact generation, checkpoints and shared admission control. Preserve snapshot/filter, tenant/role and cancellation semantics, and define expiry/download authorization for stored artifacts; an interrupted download is not a lost business mutation.
AI normalization and Wrapped statistics: normalization returns a review-required draft without committing business data; Wrapped computes DB aggregates. Keep bounded synchronous behavior until measured latency or read load justifies job UX or a materialized projection with an explicit freshness/rebuild contract.
Team notification inbox: the inbox is an authenticated read/enrichment surface. No api-events runtime writer was found for its message table in this baseline. A desired send-to-inbox history feature needs a defined producer; do not claim current direct sends already populate it durably.

High-priority recovery work includes event lifecycle orchestration, accepted-but-unstarted AI ingestion, calendar/accounting provider identity recovery, and form intake stages. Contact freshness, materialized statistics and bulk artifact generation are performance/product choices rather than evidence every current synchronous path is defective.

Broker and runtime evidence

The Terraform events module configures subscription backoff of 10 seconds to 10 minutes, a default approximately five delivery attempts before dead lettering, seven-day source retention and fourteen-day DLQ topic retention in defaults. Alert configuration pages Slack on Cleo, Brevo, scheduled-maintenance, availability, invitation, and crew-offer DLQs. Remaining fan-out keys (checklist assignment, event assignment, event chat, Twilio reconcile, Resend lifecycle, Slack channel sync, lead discovery) still have no equivalent policy. Topic retention is not a staffed replay process. Encore handlers also declare retry and acknowledgement settings. The effective SDK/provider behavior, permission grants, subscriptions, retention and replay must be tested together. Configuration committed to Git is not proof it is applied in production; the attempt limit is approximate by Pub/Sub design. The Cloud Run configuration keeps CPU allocated for background pull loops and one or two warm instances according to HA configuration. Its default maximum is five instances. HTTP and consumers share the api-events deployment. This supplies a running consumer but does not prove backlog-aware autoscaling, sufficient provider/DB capacity, per-tenant fairness, or isolation from long integration work. No throughput or latency measurements were taken for this assessment.

Recommended end-goal

Keep PostgreSQL, Encore and Pub/Sub. Establish one reusable, tested durable handoff and recovery capability, with domain-specific semantics:

Transaction owners record required work. An indexed outbox stores immutable versioned events beside the state change. Existing operation ledgers remain authoritative for payments, imports and desired-state work. Verified webhook ingress may publish directly if confirmed broker acceptance is the only handoff required.
Relays and consumers have explicit ownership. Stable event IDs, bounded claims, generation-safe completion, repeatable partial fan-out, and measured dispatch/completion age. Consumption that emits more required work preserves durability at that boundary too.
Workers isolate resource use. Separate execution by provider/workload when needed, with global/provider limits, tenant fairness and bounded DB connections. Scale using measured throughput and oldest-work age. Start with a small deployment and indexed polling; CDC or more services require evidence, not a branding preference.
Domains preserve their meaning. Notifications replay a delivery decision; reconciliation follows current desired state; financial/document commands execute an immutable authorized instruction; long jobs/sagas persist steps and outcomes. Authorization and approval survive delayed execution.
Recovery is a product of operations. Every required workflow has an owner, alertable unfinished state, retention policy, replay/runbook, and a tested unknown-outcome path. Scheduled reminders, settlement polling and drift repair remain where they add independent coverage.

The desired guarantee is: a successfully accepted required operation has a durable owner until completion, an explicit terminal outcome, or visible manual recovery. It is not an unqualified exactly-once or infinite-retention guarantee. Agree per-workload completion SLOs, outage/replay windows and duplicate tolerance before implementation; database disaster recovery is a separate obligation.

Sequenced implementation roadmap

Step	Scope and outcome	Exit evidence
1. Adopt this contract and inventory	Correct existing outbox claims; classify required effects and synchronous contracts. Prioritize known lost staffing/access/cleanup work on business impact even before the shared platform is ready.	Each selected workflow has a durable fact, failure policy, owner, and explicit HTTP/approval contract.
2. Build one vertical outbox slice	Use availability automatic notifications as the first representative path, or a narrower automatic CRM convergence path if scope demands. Include relay, safe worker execution, metrics and DLQ recovery—not just a table.	Commit/publish crash tests, duplicate publication and provider-outcome recovery, deployed redelivery/DLQ replay, bounded resource use.
3. Migrate other required delivery and lifecycle paths	Fix invitation/crew consumer swallowing before promising delivery retries. Persist event lifecycle, Twilio/calendar and blob cleanup intent while preserving revocation contracts. Couple form, expiry and closeout consequences to their durable state. Database queue fixes can ship independently.	Per-domain concurrency, partial fan-out, stale-generation and cleanup-after-delete tests; existing pending work drains.
4. Move long work to resumable execution	AI file ingestion, imports, CRM stages and verified CheckAI workflows gain durable acceptance, operation status and checkpoints. Bulk exports/artifacts join only where measured demand warrants it.	Accepted-but-unstarted recovery, restart/resume, cancellation, approval, ordering and per-record idempotency tests; measured API latency/resource improvement.
5. Consolidate financial publication conservatively	Reuse Cleo/Stripe ledgers and provider keys; replace publication plumbing only after recovery is equivalent. Give QuickBooks create/void work durable ownership and provider-identity recovery before enabling retries.	Ambiguous submit, SCA, settlement and fail-closed deletion behavior remain intact; replay cannot create a second instruction or accounting document.
6. Retire redundant retry scans selectively	Remove a domain dispatch/retry cron only after outbox, worker and DLQ recovery cover its responsibility. Keep schedules, settlement polling and useful drift repair.	Empty/migrated pending inventory, failure drills, agreed alert ownership, rollback that resumes without repeating completed irreversible actions.

This PR records architecture and guidance only. Runtime outbox tables, relays, topic/subscription changes and feature migrations are separate implementation work. B-for-all-four Twilio operations is a possible intermediate layout, not a required milestone; keeping both a full database queue and a broker should have a stated operational benefit.

Evidence index

Links identify representative decision points rather than asserting every caller in a feature behaves identically. Re-read the implementation and applicable provider skills when making a change; update this assessment when ownership or guarantees change.

Overview

Backend

Backend integrations

API reference

Frontend

Mobile

Infrastructure

CI/CD

Development

Guides

Reference

Async workflow assessment and roadmap

Findings that change the architecture decision

Messaging, staffing, and chat

External integrations and business workflows

CheckAI, MCP, and time-driven work

Lifecycle, intake, storage, and scheduled consequences

Assessed boundaries that do not automatically need a broker

Broker and runtime evidence

Recommended end-goal

Sequenced implementation roadmap

Evidence index

​Findings that change the architecture decision

​Messaging, staffing, and chat

​External integrations and business workflows

​CheckAI, MCP, and time-driven work

​Lifecycle, intake, storage, and scheduled consequences

​Assessed boundaries that do not automatically need a broker

​Broker and runtime evidence

​Recommended end-goal

​Sequenced implementation roadmap

​Evidence index

Findings that change the architecture decision

Messaging, staffing, and chat

External integrations and business workflows

CheckAI, MCP, and time-driven work

Lifecycle, intake, storage, and scheduled consequences

Assessed boundaries that do not automatically need a broker

Broker and runtime evidence

Recommended end-goal

Sequenced implementation roadmap

Evidence index