ADR-008: Agent Process Architecture

• Status: Accepted
• Date: 2026-03-24
• Refined by: ADR-0010 (lifecycle module as sole status writer), ADR-0014 (walker step-module loop; plan step dropped)

Context

The Agent Pipeline processes inbound messages through five steps: security scan, context retrieval, classification, draft generation, and routing. These are the members of the AgentStepKind union in apps/api/convex/agent/steps/types.ts (security_scan, context_retrieval, classify, draft, route). A standalone "action planning" step existed in an early draft but was dropped pre-prod with ADR-0014 — the classifier never transitioned into it and the drafter wrote a plan row whose payload was just literal JSON construction.

The simplest implementation is a single function that runs all five steps in sequence. But this creates problems:

1. No partial retry — if draft generation fails (LLM timeout, rate limit), the entire pipeline restarts from the security scan, wasting LLM calls.
2. No concurrency — one long-running pipeline blocks the next message from starting. A 10-second draft generation delays all subsequent messages.
3. No per-step state — security results, classification, and the draft each need to be persisted at their own boundary so a crash mid-pipeline doesn't lose finished work.
4. No observability — a monolithic function provides one timing metric, not a per-step breakdown.

Convex's serverless model naturally supports independent function execution — an internalAction can be scheduled, can re-schedule itself, and runs in its own transaction-free context, while mutations give atomic writes. The architecture splits along exactly that line: the LLM/compute work lives in actions, and every database write that drives the state machine lives in one mutation.

Decision

Process inbound messages with one self-scheduling step walker (an internalAction) plus one lifecycle coordinator (an internalMutation). The walker runs the per-step compute; the coordinator owns every write to the state machine.

How a message flows

The walker carries the next step's input forward directly — there is no shared accumulator (ADR-0014, choice D1). The lifecycle is the only module that touches processingStatus, so a crash between the status patch and the per-step write can no longer leave a row half-updated.

Why two modules instead of one big action

Some steps run inside a single processingStatus rather than at a state boundary — for example context_retrieval and classify both run while the message is classifying. Splitting recordStepBegin / recordStepEnd / recordStepFail out of transition lets each step atomically mark its own agentAction row without the lifecycle needing to know which step is in flight, while still keeping a single owner for processingStatus itself.

Retry semantics

Retry is uniform across all step kinds, not differentiated per step. There is no exponential backoff and no per-type retry-count.

• On any exception in a step's execute or route, the walker transitions the message to failed and marks the in-flight agentAction as failed with errorMessage (apps/api/convex/agent/walker.ts).
• A cron — retry failed agent actions, every 5 minutes (apps/api/convex/crons.ts) — calls internal.inbox.processingLifecycle.retryFailedActions. That mutation takes failed agentActions whose retryCount < maxRetries (where maxRetries = 3 for every step kind), resets each one to pending, and brings its message back to received so the walker re-runs the pipeline from security_scan.
• Once retryCount reaches 3, the action is left failed and the message stays in its failed state for a human to inspect.

Multi-intent branching

Consequences

Enables:

• Independent step retry — a failed step is reset and re-run by the cron; finished steps before it keep their persisted output.
• Concurrent processing — each message's walker schedules independently, so multiple messages process at once across the deployment.
• Per-step observability — every step writes an agentActions row with status, durationMs, modelUsed, and tokenUsage.
• Crash-safe state — because the lifecycle coordinator is the sole writer of processingStatus and patches companion fields in the same mutation, a crash can't leave a status/data mismatch.

Trade-offs:

• State lives across two tables — the message-level state machine is inboundMessages.processingStatus (see apps/api/convex/schema/inbox.ts), and per-step tracking is in agentActions rows (status, retryCount, durationMs, modelUsed, tokenUsage). There is no single stepTimings blob — per-step timing is read by joining agentActions for a message.
• Coordinator logic adds complexity versus a simple sequential function — the legal-edge graph and effect reducer need careful testing to avoid stuck messages.
• Slightly higher latency than a single function call due to scheduling overhead between steps (each runStep re-enqueues the next).

Related