Providers

Wherever Owlat talks to an external system — an LLM, an email delivery backend, a notification transport, a vector store, an analytics sink — it goes through a provider abstraction. Most are factories that read a single env var to pick an implementation, cache it per-process, and expose a small, stable interface to the rest of the codebase.

The point: self-hosters swap providers without code changes, and we can add new ones without touching every call site.

Two pluggable provider abstractions ship: the LLM provider and the send (email) provider, both fully consumed by the backend (agent steps, knowledge extraction, translation, every send path). The LLM provider reads one env var (LLM_PROVIDER) and caches the resolved client per-process. The send (email) provider is the exception: it uses a static registry keyed by provider kind (passed in per call) instead of reading one env var and caching the resolved instance. That difference is called out below.

Factory layout

The two shipping abstractions live under apps/api/convex/lib/:

lib/
├── llmProvider.ts          ← LLM_PROVIDER — single module
│                              exports getLLMProvider(task) / getEmbeddingModel() / getLLMConfig()
├── sendProviders/          ← registry keyed by SendProviderKind (not one env var)
│   ├── types.ts            ← SendProviderModule<K>, EmailSendAttempt, EmailErrorCode
│   ├── index.ts            ← SEND_PROVIDERS registry + providerFor(kind)
│   ├── dispatch.ts         ← sendProviderDispatch() — owns the retry loop
│   ├── transports.ts       ← transport registry: id → configured instance
│   ├── transportEnv.ts     ← per-instance configuration reads
│   ├── routing.ts          ← resolveRoute() — per-org route selection
│   ├── health.ts           ← providerHealth recording + reads
│   ├── capability.ts       ← isSendProviderReady() — required env present?
│   ├── fallbackEligibility.ts ← may this kind be the deliverability-fallback relay?
│   ├── mta/index.ts        ← built-in MTA adapter
│   ├── ses/index.ts        ← Amazon SES adapter
│   ├── smtp/index.ts       ← generic SMTP-relay adapter (SMTP_RELAY_*)
│   ├── resend/index.ts     ← Resend adapter
│   ├── mandrill/index.ts   ← Mailchimp Transactional adapter (messages/send-raw)
│   └── strategies/         ← single / priority_failover / workload_split / adaptive_mix
└── emailProviders/         ← identity & domain verification only (NOT the send factory)
    ├── domainVerification.ts
    ├── mtaIdentity.ts
    └── sesIdentity.ts

Sending-domain identity (SPF/DKIM registration and verification at the provider) is a separate registry under apps/api/convex/domains/providers/, keyed by the same kind: mta/, ses/, mandrill/, plus plugin/ — one host-owned adapter serving every bundled plugin transport that contributes a domainIdentity. A core kind only appears there if its catalog entry declares domainVerification: 'api', and a mapped-type guard fails the build if one declares it without registering a provider; for a plugin transport that word is derived from the contribution itself, so the promise and the code that keeps it are one declaration.

That folder answers two questions with two registries. SENDING_DOMAIN_PROVIDERS / providerFor(kind) is the PRIMARY one — the value a domains row records in providerType, whose adapter owns registration, the DNS bundle, the sibling identity row and the return path — and it stays a closed core union. relayIdentityProviderFor(kind) answers the smaller RELAY question ("can this kind prove a domain it does not own the lifecycle of?"), asked by the routing gate, the identity backfill, the alignment pre-flight and the due-check sweep; it is composed at build time from every core adapter implementing all three relay seams plus one entry per bundled plugin identity. Widening the primary one would run a domain's whole lifecycle through code this repository does not contain.

LLM provider

Env var: LLM_PROVIDER (default: openai)

Supported values: openai (default), openrouter, ollama

All three speak the OpenAI Chat Completions shape (so anything OpenAI-compatible plugs in — a self-hosted vLLM or LM Studio, for example). Every client is built with createOpenAI from the Vercel AI SDK.

The whole abstraction is a single module, apps/api/convex/lib/llmProvider.ts, exposing standalone functions:

// apps/api/convex/lib/llmProvider.ts
export function getLLMProvider(task: LLMTask): LanguageModel; // returns AI SDK LanguageModel
export function getEmbeddingModel(): EmbeddingModel;
export function getLLMConfig(): {
    provider: string;
    modelFast: string;
    modelCapable: string;
    embeddingModel: string;
    baseURL: string | undefined;
    hasApiKey: boolean; // snapshot — never the key itself — safe to log
};

// task tiers — classify/extract/guard/summarize → fast model
//              draft/plan                       → capable model

To run Claude models, point the OpenAI client at an OpenAI-compatible endpoint — there is no native Anthropic provider, and anthropic is not a recognized LLM_PROVIDER value:

LLM_PROVIDER=openai
LLM_BASE_URL=https://api.anthropic.com/v1/   # or an OpenAI-compat shim
LLM_MODEL_CAPABLE=claude-sonnet-4-6
LLM_MODEL_FAST=claude-haiku-4-5-20251001

Required env vars depend on which provider is selected:

All OpenAI-compatible providers accept any of LLM_API_KEY, OPENROUTER_API_KEY, or OPENAI_API_KEY (first set wins — see resolveApiKey() in apps/api/convex/lib/llmProvider.ts). The ollama default base URL is the Docker service hostname http://ollama:11434/v1, resolved in resolveBaseURL() in the same file.

The ai feature flag requires LLM_PROVIDER and LLM_API_KEY to be set. The admin UI blocks toggling ai on until they exist.

Send (email) provider

The send-side abstraction lives under apps/api/convex/lib/sendProviders/ (per ADR-0020). Unlike the other factories it is not selected by reading one env var into a cached singleton — instead it is a static registry keyed by provider kind, and the kind is resolved per send (from the org's route config, falling back to the EMAIL_PROVIDER env var, then to unconfigured when neither names a provider). Resolution is fail-closed: there is no implicit MTA default, so a send that reaches dispatch without a configured provider is refused rather than dispatched to a phantom MTA. The send entry points gate on an isDeliveryConfigured capability check first (so a campaign or transactional send is rejected before any row is written), making this routing fallback defence-in-depth.

Supported kinds: mta, ses, resend, smtp, mandrill, emailit — in catalog order. SEND_TRANSPORT_KINDS, setup surfaces, runtime composition, and conformance suites all derive from that declaration. Curated picker copy may order familiar providers first; an entry without custom copy is appended in catalog order.

mta is the built-in sender that ships with self-host (apps/mta). The alternatives are native Amazon SES, generic SMTP submission, Resend, Mailchimp Transactional, and Emailit. Each is the same composed bundle shape: transport plus optional feedback, domain identity, setup, and platform hooks.

Declared capabilities

Every catalog entry answers thirteen questions about the provider, and the rest of the system asks the catalog rather than naming a kind:

Every cell above is the value declared in packages/shared/src/sendProviderCatalog.ts — the table restates the catalog, it never defines it. apps/docs/__tests__/providerCapabilityDocs.test.ts reads both and fails if they drift, so a cell that disagrees with the code is a test failure rather than something a reader has to catch. It also fails on a capability field the catalog declares and this table omits, so the next one added cannot be invisible here either.

Five consequences worth naming, because they used to be hard-coded identity checks:

• Fallback eligibility (fallbackEligibility.ts) — a kind may be the deliverability-fallback relay iff it is configured and is not mta. The MTA is the arm a fallback moves traffic away from, so routing it to itself would relieve a reputation problem through the transport that has it. Nothing names SES.
• Relay domain verification (relayDomainVerification.ts) — a relay's From domain counts as verified iff a registered sending-domain provider for that kind says so. Kinds with no such provider (resend, smtp) keep an honest "unverifiable" posture and fail closed.
• Governed dispatch (delivery/governedDispatch.ts) — the pre-dispatch identity binding, the message-id substitution, the acceptedForDelivery verdict and the replay of an ambiguous acceptance all read acceptanceSemantics and messageIdSource. That file no longer compares a provider kind to a literal at all; a test asserts it.
• Redacted complaints (webhooks/complaintDispatch.ts) — an RFC 5965 §3.2 complaint carries only the recipient, so the blocklist decision rests on whether the report is about real production mail. A source that tags its feedback must show deliveryDomain: 'production'; one that does not tag it has no tag to show and is suppressed on sight. It used to read providerType === 'ses', which dropped the byte-identical complaint from every other ESP and left the complainer mailable.
• System and auth mail (lib/systemMailOutcome.ts) — whether an ambiguous send (a password reset whose response was lost) may be retried reads deduplicatesOnIdempotencyKey. It used to be the list provider === 'mta' || provider === 'resend', spelled in one file and restated in another.

Mailchimp Transactional (Mandrill)

The migration arm. A team arriving from Mailchimp pastes their existing API key and keeps sending on the reputation they already have; the shipped ramp controller then grows the own-MTA share cell by cell under its gates, and backs off automatically on bounce, complaint, engagement or seed-placement signals. There is no flag day and no manual traffic shifting — see Deliverability Infrastructure for the machinery.

Setup

1. API key. MANDRILL_API_KEY from Mailchimp Transactional → Settings → API keys. Set EMAIL_PROVIDER=mandrill to route everything through it, or add it to a providerRoutes row to run it beside the built-in MTA. Named instances (mandrill#eu) work exactly like every other kind, via SEND_TRANSPORT_INSTANCES and MANDRILL_API_KEY__EU.
2. Webhook. Create a webhook at Settings → Webhooks pointing at <CONVEX_SITE_URL>/webhooks/mandrill, and enable exactly send, deferral, hard_bounce, soft_bounce, spam, unsub, reject. Leave open and click off: Owlat tracks opens and clicks first-party on every transport, and the engagement ramp gate can only compare the two arms because both are measured on the same instrument. Copy the signing key Mandrill shows once into MANDRILL_WEBHOOK_KEY. Signature verification is HMAC-SHA1 over the exact webhook URL plus the alphabetically-sorted POST params, so a redirect or a trailing-slash difference fails it.
3. Domain verification. Mandrill signs every account's mail with one shared, account-independent key under the mandrill selector, so the DNS is derived from the domain name rather than minted per identity: an SPF record including spf.mandrillapp.com, and a TXT record at mandrill._domainkey. A third step is easy to miss — Mandrill also wants proof of ownership (a mandrill_verify.<key> TXT record, or a confirmation in its own dashboard for accounts that offer no token). A domain with flawless SPF and DKIM but no ownership proof is one Mandrill still rejects with reject_reason: unsigned, so Owlat does not treat it as verified. Delivery → Sending domains renders all three with Mandrill's own verdict text.

Capabilities, honestly

• No custom return path. Mandrill accepts a per-message return_path_domain, but it mints its own bounce local part, so Owlat's VERP envelope sender does not survive. The catalog declares supportsCustomReturnPath: 'probe' for uniformity and the probe machinery settles it at unsupported with reason no_envelope_control. The practical consequence is narrow and worth stating plainly: bounces that arrive only at the SMTP envelope cannot be attributed to a specific send by us — Mandrill's webhook feedback covers the rest — and the cell's measurement is marked degraded rather than blocked. Nothing stops sending.
• Provider feedback: yes. Every event in the list above moves the matching emailSends / transactionalSends row through providerMessageId (Mandrill's _id), and reject events mirror Mandrill's own blacklist into blockedEmails so the own arm never mails an address the reference arm was silently spared. That mirror is what keeps a measured migration comparing the two arms on the same population.
• Domain verification: api. Registered through senders/add-domain, re-checked hourly through senders/check-domain, stored in the generic sendingDomainRelayIdentities table. The proof expires after 7 days (vs. SES's 30): the check is one cheap HTTP call, so a proof that cheap has no excuse to be a month old before routing stops trusting it. A verified-but-stale identity reads "re-checking", never "verified".
• No idempotency key. Mandrill's API has none, so a timed-out request may or may not have been accepted. The adapter returns AMBIGUOUS_TIMEOUT with acceptanceUnknown rather than retrying; governed dispatch's re-entry snapshot and the workpool's single-attempt discipline are what prevent a double delivery.

Registry & lookup

providers/composition.ts composes every first-party and bundled provider. sendProviders/index.ts exposes the compatibility registry and lookup helpers from that one view:

export const SEND_PROVIDER_BUNDLES = composeProviderBundles(assigned);
export const SEND_PROVIDERS = Object.fromEntries(
    SEND_PROVIDER_BUNDLES.filter(({ descriptor }) => isCoreSendProviderKind(descriptor.kind))
);

export function providerFor<K extends SendProviderKind>(kind: K): SendProviderModule<K>;
export function isSendProviderKind(kind: string | undefined | null): kind is SendProviderKind;

A compile-time satisfies-style mapped-type check pins each registry value to SendProviderModule<thatKind>, so a missing method fails the build.

Adapter interface

Each adapter implements SendProviderModule<K> from sendProviders/types.ts:

export interface SendProviderModule<K extends SendProviderKind> {
    readonly kind: K;
    readonly retryDelays: readonly number[]; // backoff schedule; dispatch owns the loop
    sendEmail(params: EmailSendParams, extras?: ExtrasFor<K>): Promise<EmailSendAttempt>;
    categorizeError(message: string, httpStatus?: number): EmailErrorCode;
}

// Single-attempt result — no internal retry:
export type EmailSendAttempt =
    | { success: true; id: string }
    | { success: false; errorMessage: string; errorCode: EmailErrorCode };

sendEmail performs exactly one attempt and never retries internally. extras is a typed per-provider second arg (MtaExtras carries pool/signing metadata plus the governed routing context and lease, or the explicit system intake path; Resend carries an idempotency key; SES takes none). EmailErrorCode is an enum (RATE_LIMIT, SERVER_ERROR, ROUTING_DEFERRED, ROUTING_LEASE_UNREADABLE, INVALID_RECIPIENT, INVALID_SENDER, AUTH_FAILED, CONTENT_REJECTED, AMBIGUOUS_TIMEOUT, SMTPUTF8_UNSUPPORTED, UNKNOWN); routing deferrals return to bounded fresh-route scheduling, while only RATE_LIMIT and SERVER_ERROR use provider-attempt retries. The two routing codes reschedule identically and differ only in what they claim: ROUTING_DEFERRED is the MTA declining this sending identity (counted against the ramp's deferral gate), ROUTING_LEASE_UNREADABLE is the MTA failing to read back a lease record it wrote (our own storage, deferred as local and not counted).

Dispatch

The single public entry point is sendProviderDispatch() in sendProviders/dispatch.ts:

sendProviderDispatch(ctx, transportId, params, extras?): Promise<DispatchResult>

The dispatch unit is a transport id, not a bare provider kind. Every kind has one default instance whose id IS the kind (mta, ses, resend, smtp), reading the unsuffixed variables. Extra instances of the same kind are declared in SEND_TRANSPORT_INSTANCES as <kind>#<instanceKey> and read the same variables under an __<INSTANCEKEY> suffix — smtp#backup reads SMTP_RELAY_HOST__BACKUP — so a deployment can keep a warm fallback relay while trialling a second one. Each adapter resolves its own configuration from the resolved transport record; the record itself carries only variable NAMES, never values.

Resolution fails closed. An id that is malformed, names an unknown kind, names an instance nothing declared, or names a declared instance whose configuration was removed throws SendTransportResolutionError before any attempt, any authorization call and any health write — it never falls back to another transport. Named instances follow the CONFIGURATION, not the tier: a kind can have them when it declares variables of its own that an instance suffix reaches. Core kinds always do. A plugin-contributed kind (plugin.<pluginId>.<localId>) does when its sendTransports contribution declares requiredEnvVars / optionalEnvVars — the host then resolves them per instance and hands the module exactly those values. A plugin transport that declares none is configured from the plugin's own deployment-wide environment, which the __<INSTANCEKEY> suffix does not reach, so plugin.…#alt is rejected with instances_unsupported rather than sending with the default plugin instance's credentials.

listSendTransports() enumerates every transport this deployment can dispatch through, and only those: a declared named instance that is unconfigured, or belongs to a kind that cannot have instances, is omitted rather than listed, so enumerate-then-dispatch can never pick an id that resolution would reject.

The helper owns the retry loop (driven by the module's retryDelays + categorizeError), and after every terminal outcome — success or exhausted retries — it records provider health by scheduling internal.lib.sendProviders.health.recordSendResult. Health stays keyed by provider KIND (providerHealth holds one row per kind), so instances of a kind share a health row. DispatchResult carries the final EmailSendAttempt, the providerType used, the transportId it went through, total latencyMs, and the number of attempts.

Routing & health

Routes with the built-in MTA and an enabled relay can opt into a deliverability escape hatch. Authenticated MTA snapshots are applied per receiver provider: a Gmail-only breaker or persistent-defer signal moves only Gmail traffic to the relay. A pool-wide FCrDNS quarantine or critical DNSBL exhaustion applies to every provider slice. Failback requires 15 healthy minutes and a 30-minute minimum fallback period, preventing flapping.

Relay credentials are not DNS proof. The From-domain must be verified for the exact relay path. Enabling SES fallback provisions a sibling SES identity for every existing verified MTA domain by cursor, and the sending-domain lifecycle provisions future domains when they become verified. Provider Routing displays the exact sibling DNS plan and its live DNS/SES status. The plan contains one merged apex SPF record (owned MTA plus SES), SES DKIM, and a dedicated ses-mail MAIL FROM pair; the primary domain's DMARC remains authoritative. Resend and generic SMTP relay targets fail closed until they expose a provider-specific verification adapter. Campaign routes may also opt into sending overflow above the synced owned-IP warming cap through the verified relay. Relay sends never mutate the owned-IP counters or cap.

resolveRoute() in sendProviders/routing.ts picks the provider kind for a send from an org's providerRoutes config, dispatching to a strategy module under sendProviders/strategies/ — single, priority_failover, or workload_split. It falls back to the EMAIL_PROVIDER env var, then returns null (unconfigured) — never a phantom mta — when there is no config, no enabled providers, the strategy returns null, and EMAIL_PROVIDER names no provider.

Admins configure these routes per message type (transactional, campaigns, automations) under Settings → Technical → Provider Routing in the dashboard — pick the strategy, order the providers, set workload-split weights, and optionally pin an IP pool. With no route configured, every send uses the EMAIL_PROVIDER value; with neither a route nor EMAIL_PROVIDER, the send is refused (no delivery provider configured). See Operating Modes for which deployment shapes need a provider.

sendProviders/health.ts records rolling success/failure counts, average latency, and a healthy/degraded/down status per provider into the providerHealth table. Dispatch is the only writer; resolveRoute is the only reader of the all-providers snapshot.

Identity & domain verification

The older lib/emailProviders/ directory still exists, but it now holds only identity and domain-verification helpers — domainVerification.ts, mtaIdentity.ts, sesIdentity.ts — not the send factory.

See Environment Variables → Email Sending for the per-provider env vars.

Adding a new provider

LLM: because every supported client speaks the OpenAI shape, most "new providers" need no code at all — set LLM_PROVIDER=openai and point LLM_BASE_URL at the OpenAI-compatible endpoint. For a genuinely non-OpenAI-compatible backend you would edit the single apps/api/convex/lib/llmProvider.ts module directly (e.g. add a base-URL case in resolveBaseURL() and, if it needs a different SDK client, branch in getClient()). If the provider needs new env vars, declare them in FEATURE_FLAGS[<flag>].requiredEnvVars in packages/shared/src/featureFlags.ts so the wizard prompts for them, and document them in Environment Variables.

Send (email) — the provider-N+1 checklist

Adding a sender used to mean a twelve-file hunt, two of which contained hard-coded 'ses'-only gates. It no longer does: it is one catalog entry plus the modules that entry promises. Everything below is additive, and nothing in routing, ramp control, measurement, or governed dispatch changes — those all read declared capabilities (ADR-0055).

The list is the same at both tiers, artifact for artifact, which is the whole claim of the seam: a provider is a bundle, and where the bundle lives is a packaging decision rather than a difference in what it has to provide. A core kind is an in-repo bundle (mta, ses, resend, smtp, mandrill, emailit); a plugin kind is a bundled package contributing the same bundle through sendTransports, and since parity landed it is the default for provider N+1 — reach for the core tier only when the provider needs something the plugin tier cannot declare, which today is exactly three things: an envelope sender we sign (supportsCustomReturnPath other than no, and with it per-message bounce attribution over VERP), custody semantics (acceptanceSemantics, and messageIdSource: 'idempotency-key' with it), and a host-side setupProbe. Every other field is either declarable or derived from a half the bundle already ships. (tagsFeedbackProvenance is undeclarable at that tier too, but it is only ever true of our own MTA, so it is never the reason a third-party provider needs the core tier.) The plugin column here is the summary; Send Providers (plugins) is the guide that tier is written against.

There is no "declare the kind" step. The core kind union is (typeof CORE_SEND_PROVIDER_CATALOG)[number]['kind'] (and the backend's SendProviderKind is that union plus the bundled plugin kinds), so step 1 is the declaration: SEND_TRANSPORT_KINDS (packages/shared/src/transportAlignment.ts) re-exports it, DELIVERY_PROVIDER_KINDS and getSendPathRequiredEnv (featureFlags.ts) read it, and PROVIDER_ENV_KEYS (setupSendingPresets.ts) is derived from the entry's credentialFields. Those four used to be independent literals a sixth provider had to be remembered in; adding one to any of them now would be re-introducing the duplication the seam removed.

Five of those are compile-time enforced, which is the point of the seam:

• _typecheck: { [K in CoreSendProviderKind]: SendProviderModule<K> } in sendProviders/index.ts breaks the build if a declared kind has no adapter (step 3).
• _typecheck: { [K in FeedbackReportingSendProviderKind]: AnyInboundAdapter<K> } in webhooks/adapters/index.ts breaks the build if a kind declares hasProviderFeedback: true with no registered feedback adapter — and, because the mapped type pins each key to an adapter whose source is that key, on an adapter filed under the wrong kind. _RegisteredFeedbackAdaptersAreDeclared in the same file breaks it in the converse direction, on an adapter registered for a kind the catalog says reports nothing (step 4).
• _ApiVerifiedKindsHaveDomainProviders in domains/providers/index.ts breaks the build if a kind declares domainVerification: 'api' with no registered provider (step 5).
• CoreSendProviderCatalogEntry — the core entry type, in packages/shared/src/sendProviderCatalogTypes.ts and re-exported from both catalog modules — is a union, not a bag of independent fields: it breaks the build on a core entry that omits domainVerification / acceptanceSemantics / messageIdSource / deduplicatesOnIdempotencyKey, or that splits the custody pair in either direction — acceptanceSemantics: 'accepted' with an id the provider mints (which would answer an ambiguous send with an idempotency key the provider never saw), or messageIdSource: 'idempotency-key' without the custody that makes pre-binding that id worth its prerequisites (step 1). The same union carries the feedback pair: hasProviderFeedback: true requires a providerFeedback channel and a channel requires the boolean, so you cannot declare a webhook the measurement plane grades as "reports nothing", nor a kind with feedback and no panel, no endpoint and no route (steps 1 and 4).
• bun run lint:env and apps/api/scripts/check-env-docs.sh fail on an EnvKey that is read outside lib/env.ts or left undocumented (step 2).

Step 7 is deliberately NOT enforced. The three web tables used to be exhaustive Record<DeliveryProviderKind, …> maps, so a sixth kind was a compile error in apps/web — a file the provider's own bundle has no business touching, which is the opposite of "adding a provider is additive". They are Partial overrides now: a kind with no row is rendered from its catalog entry (transportKindLabel falls back to entry.label, transportDnsGuidance to the tier / domainVerification paragraph, the picker to the entry's label and a neutral icon). The generated web catalog gives plugin kinds the same fallbacks. Nothing breaks, and nothing is blank — so write a core override when this surface should word that provider differently, not because the build made you.

The tier difference at step 4 is who verifies, and the answer is the same at both: the host does. A plugin's webhook export is parse-only and never sees an unverified byte, a contribution whose webhook carries no signature contract or no replay provisions fails manifest validation rather than shipping an open endpoint, and the feedback lands on one route — POST /webhooks/plugin/<pluginId>, behind the hosted-contribution authorization seam — where a core kind keeps a static /webhooks/<kind> path of its own. Two routing shapes, one security floor. See Plugin Contributions → Feedback webhook.

A bundled plugin transport is generated rather than written, so it never reaches that union — for it, the one declaration whose prerequisites live outside the catalog is refused at composition time instead: catalog.ts throws while building the catalog if a plugin entry declares acceptanceSemantics: 'accepted' or messageIdSource: 'idempotency-key', naming the entry and pointing at the PREREQUISITES note. The two fail-closed defaults and that guard have their own suites (undeclaredSemanticsFailClosed, pluginCustodyGuard), which mock a generated catalog because the shipped one cannot produce either shape.

The rest is runtime-enforced by the shared conformance suites in lib/sendProviders/__tests__/ (registry, providers, selection, dispatch.integration, transportIdDispatch, twoTransportsSameKind, unknownTransportFailsClosed, transportSecrets, dispatchExtras, fallbackEligibility, systemMailExtras, feedbackRoutes), all of which iterate every catalog kind — so a new kind joins them by existing, not by anyone remembering to add a case. feedbackRoutes is the one that crosses packages: it walks the real httpRouter and fails if a declared providerFeedback.webhookPath is not a route http.ts serves, or if a /webhooks/<kind> route exists for a kind whose entry declares no channel — which is the pairing an operator's pasted endpoint depends on (step 4). Its companion is webhooks/__tests__/adapterRegistry.test.ts, which fetches each declared path through the real router and proves the answer came from that kind's registered adapter. That is the half neither the mapped type nor feedbackRoutes can see: the guards prove the registry is well-keyed and the route exists, but http.route({ path: '/webhooks/ses', handler: providerFeedbackWebhook('resend') }) satisfies both and typechecks.

Design notes

• Caching: the LLM client is cached at module level (cachedClient in lib/llmProvider.ts) so each process resolves it once. There is no exported clear*Cache() — tests reset the module state with vi.resetModules(). The send provider has no cache at all: it uses the static SEND_PROVIDERS registry with the kind passed per call.
• Throwing on unknown: providerFor throws a descriptive Unknown send provider: <kind> error, so a typo fails loudly instead of silently falling back. (An unrecognized LLM_PROVIDER value — e.g. anthropic — silently falls through to the default OpenAI client with no base URL.)
• No secrets in getLLMConfig(): getLLMConfig() returns a snapshot of resolved settings with hasApiKey: boolean rather than the key itself, safe to log. (The send provider has no config getter — SendProviderModule exposes only kind, retryDelays, sendEmail, categorizeError, and the optional buildDispatchExtras, which is env-free by contract.)
• Embeddings: the embedding plane is resolved INDEPENDENTLY of the language plane by resolveEmbeddingModel(ctx) (through the same resolveAiConfig(ctx) point) and is LOCAL BY DEFAULT — a local OpenAI-compatible /embeddings sidecar (LOCAL_EMBEDDING_BASE_URL / LOCAL_EMBEDDING_MODEL) so retrieval works under any language provider (incl. Anthropic, which has no embeddings API). Optional hosted embedders (openai / google) are overrides with their own encrypted key; env LLM_EMBEDDING_MODEL remains the deployment fallback. A misconfigured hosted embedder throws an actionable error at resolve time (never a silent empty vector); a known-dimension mismatch against EMBEDDING_DIMENSIONS throws at resolve time and assertEmbeddingDimension enforces the width at write time.