ADR-010: Listing Engine

• Status: Accepted
• Date: 2026-05-26

Context

The Convex backend already had deep write-side modules — lifecycles, intake, dispatch, find-or-create — each owning how one entity changes. The read side never got the same treatment. "List a page of <entity>" was open-coded across ~80 list* query endpoints, and the duplication was not the worst of it: no two of them agreed on their own contract.

There were four mutually incompatible "give me a filtered, paginated page" shapes:

Four list queries, four return contracts, none canonical. A single piece of pagination UI physically could not consume all of them. Four concrete problems compounded the divergence:

1. The cursor was a lie on the search path. Contact search returned continueCursor: 'search' (cast as unknown as string) and always re-read from the top, so asking for page 2 of a search re-served page 1. Search results were silently single-page.
2. The index-vs-collect decision was made per file, often wrongly. Contacts used a real searchIndex; campaigns collected a whole status bucket and filtered search in memory; email templates .collect()-ed the entire table for every list, filter, and sort — a scaling cliff with no index in sight.
3. Counts were their own zoo. countByStatus, countByType, and count each used a different strategy (collect-and-group, collect-and-group again, a denormalized cached counter), with no shared surface — even though the dashboards that render a list need their facet counts alongside the page.
4. Enrichment was duplicated and N+1. The topic contactCount enrichment was inlined separately in both list and get, and the per-row count was a full pagination scan whenever the cached field was absent.

This is the read-side counterpart to the write-side lifecycle modules, and it follows the codebase's own resolved precedent: a thin generic dispatcher (the Walker) over per-type data (the Block modules).

Decision

Make resource listing one seam: a generic Listing engine dispatching over per-entity Listing descriptors, returning one Convex-native contract. The engine lives in apps/api/convex/lib/listing.ts; each entity owns a descriptor at apps/api/convex/<entity>/listing.ts.

A descriptor declares the entity's read surface as data — its search index, browse index, legal sorts and filters, soft-delete policy, per-row enrichment, and named facet counts:

// apps/api/convex/contacts/listing.ts — the cleanest case
export const contactListing: ListingDescriptor<'contacts'> = {
  table: 'contacts',
  search: { index: 'search_contacts', field: 'searchableText', filterFields: ['deletedAt'] },
  browse: { index: 'by_deleted_at_and_created_at', order: 'desc' },
  softDelete: true,
  facets: { total: { kind: 'cachedCounter', table: 'instanceSettings', field: 'contactCount' } },
};

The session-auth shell (contacts/contacts.ts:list) and the API-key shell (<entity>/organization.ts) keep their own auth posture and call into the engine:

// apps/api/convex/contacts/contacts.ts (shape)
const page  = await listResources(ctx.db, contactListing, args);
const total = await countFacet(ctx.db, contactListing, 'total');

What the engine guarantees

• A real Convex cursor on both paths. Search uses .withSearchIndex(...).paginate() (a real, opaque cursor — the 'search' sentinel is gone); browse uses .withIndex(...).order().paginate(). Search is genuinely multi-page the moment a query routes through the engine.
• Search means relevance order. Search results are relevance-ordered, so sortKeys apply to the browse path only. Passing search ignores sort. This is part of the interface, not a hidden detail — see listResources in apps/api/convex/lib/listing.ts.
• Soft-delete rides the index, never thins the page. When softDelete is true, deletedAt === undefined is fixed inside the index range on both paths. The browse index must therefore lead with deletedAt (contacts use by_deleted_at_and_created_at). Ordinary equality filters may thin a page; only deletedAt is barred from that fate.
• Index-native filters where a compound index exists. A single equality filter with a dedicated compound index (e.g. status → by_status_and_updated_at) is served index-natively and ordered. A filter with no such index falls back to a post-index .filter().
• The descriptor owns enrichment cost. The engine runs enrich over the page (and the entity's get reuses it), and does not hide whether each call is O(1) cached or a scan — the cost is stated in the descriptor. Topics enrich contactCount from the denormalized cachedMemberCount (with a bounded membership scan fallback); the automation descriptor declares no per-row enrichment.

Facets are closed to three strategies

The count zoo collapses into exactly three Facet kinds — anything richer is rejected at the interface, and a one-off count is written as a plain query outside the seam:

countFacet(db, descriptor, name) resolves the strategy; groupBy returns per-bucket counts whose total is their sum. The implementation lives in countFacet in apps/api/convex/lib/listing.ts and reuses the countWithPagination primitive in apps/api/convex/lib/pagination.ts.

Decisions resolved during design

1. Walker + descriptors hybrid, not a generic config-bag and not a per-entity module family — a single generic engine over thin per-type data, the codebase's already-accepted dispatcher-over-data shape.
2. Convex-native cursor only, accepting the schema bill: campaigns and email templates gained search_campaigns / search_templates search indexes with filterFields, and deletedAt joined the relevant browse index. The in-memory paginateArray offset fallback was rejected as the source of both the offset cursor and the table-scan-on-search.
3. Scope is page + enrichment + facet counts. The descriptor owns the entity's whole read surface, so the count queries and the topic enrichment all collapse in. A page-only seam would have left the count zoo and the N+1 standing.
4. The descriptor owns enrichment cost rather than the engine refusing non-O(1) enrichment, which would have forced premature denormalization.
5. Facets are closed to three strategies, exactly what existed in the wild.
6. Auth stays in the shells. The engine takes a DatabaseReader, never a session — it reads, it does not authenticate or scope.

Enforcement

A lint:listing guard (apps/api/scripts/check-listing.sh, run as part of bun run lint, sibling to lint:env and lint:errors) keeps the seam from eroding. It bans, in query files outside the engine and the descriptors:

• paginateArray( — the removed stringified-integer offset helper.
• Manual numItems + 1 "take n+1 to compute hasMore" pagination — the hand-rolled pattern that produced the 'search'-sentinel fake cursor.

The third open-coded shape (.collect()-then-filter-then-paginate) is held down by the existing .collect() baseline check; porting a list query to the engine only ever lowers it.

Consequences

Enables:

• One read contract — every list endpoint returns the same { page, isDone, continueCursor } with a real Convex cursor plus its facets, so one pagination UI works against all of them.
• The contact-search cursor bug dies on cutover; search becomes genuinely multi-page.
• The index-vs-collect decision is made once, in one tested place. The full-table .collect() that email-template listing got wrong is gone; a new listable entity is a ~6-line descriptor instead of 60 lines of re-derived query. Six entities now use the engine — contacts, campaigns, topics, segments, automations, and email templates.
• The count zoo and the N+1 collapse: countByStatus / countByType / count become declared facets, and per-row enrichment is declared once and shared by list and get.

Trade-offs:

• An additive schema bill — search_campaigns / search_templates search indexes, by_status_and_updated_at / by_type_and_updated_at compound indexes, and deletedAt folded into the soft-deletable browse indexes.
• An intentional, atomic break: the email-template HTTP list changed from a bare array to a paginated { page, … }, in the same clean-break spirit as the SDK changes (no two-phase migration).
• Convex's withIndex / withSearchIndex want per-table string-literal index names that a function generic over TableNames cannot satisfy, so the engine casts its query builders internally — the same trade countWithPagination already makes, confined to one module.

This decision is recorded as ADR-0037 in the engineering ADR set (docs/adr/0037-resource-listing-engine.md). It builds on ADR-002: Convex as backend and mirrors the dispatcher-over-data pattern; for the broader backend layout see the Convex Backend reference.