Platform Operations

This page documents the operational surface that sits behind the product: the abuse-status lifecycle that gates sending, the platform-admin role and its queries, the content review queue, the organization-deletion walker, system health and in-app self-update, and the dev-mode endpoints, crons, and migrations that keep a deployment maintainable. It is written for operators and contributors working on a self-hosted Owlat instance.

Abuse status lifecycle and the sending gate

Every deployment carries a single abuse status on instanceSettings.abuseStatus. It drives whether the instance is allowed to send. The four statuses, defined in apps/api/convex/organizations/abuseStatus.ts, form a severity ladder:

The read and write halves are deliberately split into sibling modules (ADR-0011, docs/adr/0011-abuse-status-modules.md):

• Abuse gate (apps/api/convex/organizations/abuseGate.ts) is the home of the sending predicate. It exports two surfaces: isSendingAllowed(status), the pure predicate that the live send paths call (the transactional dispatch path in transactional/dispatch.ts), and requireSendingAllowed(ctx), a mutation-context helper that fetches instanceSettings and throws on suspended/banned. The campaign send path enforces the same predicate inline through its pre-flight query (campaigns/preflight.ts, which checks abuseStatus against suspended/banned directly) rather than calling the helper. Either way the gate always runs — it is not behind a feature flag.
• Abuse status (apps/api/convex/organizations/abuseStatus.ts) is the only writer of abuseStatus and its companion fields (abuseStatusReason, abuseStatusChangedAt, abuseStatusChangedBy).

Transition rules

There are two write entry points. The internal-writer path enforces severity rules; the admin path bypasses them.

Both paths write an abuse_status_changed audit-log row on every call, including same-state no-ops (so "circuit breaker tripped again while already warned" is observable). A missing instanceSettings row returns { ok: false, reason: 'no_settings_row' } rather than throwing — an early-deployment edge case.

What flips the status automatically

Reputation is derived on read (a rolling 30-day window — see Deliverability Infrastructure), but auto-enforcement is deliberately kept off the read hot path: an hourly evaluate reputation auto-enforce cron (apps/api/convex/crons.ts) calls evaluateAutoEnforce, which summarizes the org window and schedules autoEnforceReputation when risk is high/critical. A separate hourly cleanup sending reputation cron (recalculateAll) ages out >60-day buckets. Because severity downgrades are refused, a critical → warned transition never silently relaxes an existing suspension.

Platform admin role, roster, and operational queries

Platform admins are rows in the platformAdmins table, keyed by BetterAuth user id. Two roles exist: admin and superadmin. The first admin is created with seedPlatformAdmin (an internal mutation that only succeeds while the table is empty and always seeds a superadmin); after that, superadmins manage the roster.

Gating is centralized in apps/api/convex/platformAdmin/platformAdmin.ts:

The web app gates the single admin page with the platform-admin route middleware (apps/web/app/middleware/platform-admin.ts), which calls isPlatformAdmin and redirects non-admins to /dashboard.

Operational queries

All of these live in apps/api/convex/platformAdmin/queries.ts and start with requirePlatformAdmin. They read the singleton instanceSettings row plus derived reputation/stats:

Roster + status mutations

These live in apps/api/convex/platformAdmin/mutations.ts:

Content review queue (approve/reject pending content)

Campaigns and transactional emails can land in a pending_review status (for example when content scanning flags them — see Email Security). The review surface is platform-admin only.

getContentReviewQueue (in platformAdmin/queries.ts) returns the pending items joined with their latest content-scan result, plus the recently-reviewed log:

{
  pending: [{ type: 'campaign' | 'transactional', id, name, subject, scan: { score, level } | null, ... }],
  pendingCount: number,
  recentlyReviewed: [{ action, details, userId, createdAt }],
}

The three review mutations (in platformAdmin/mutations.ts) all require pending_review as the current status and write a platform_admin.content_approved or platform_admin.content_rejected audit row:

Both approval paths and the rejection path move the resource out of the queue; there is no separate "blocked" terminal state for reviewed content — rejection reverts to draft with the reason captured in the audit log.

Organization deletion walker

Deleting the organization is a hard, irreversible wipe of the entire data plane. It is owner-only: organizationSettings.remove (in apps/api/convex/organizations/settings.ts) checks session.role === 'owner' and schedules internal.organizations.deletion.walker.start. The walker lives in apps/api/convex/organizations/deletion/walker.ts and implements ADR-0025 (docs/adr/0025-organization-deletion-module-family.md).

How it works:

• Ordered cascade. STEPS is an ordered list of the full per-organization table set, ~90 tables. Children are deleted before parents; storage-bearing tables (e.g. mediaAssets, semanticFiles, mailMessages) purge their blobs before the row delete; auditLogs is second-to-last (it keeps accumulating from delegated lifecycle calls during the wipe); instanceSettings is the terminal step.
• One module per table. Each entry has a sibling step module under deletion/steps/<table>.ts implementing the deleteBatch(ctx) contract from steps/_common.ts (default batch size 100). A few tables delegate — for example the contacts step sweeps five child tables (contactTopics, contactPropertyValues, contactActivities, contactIdentities, contactRelationships) that are not standalone steps.
• Self-scheduled hop. runStep runs one batch, re-fires itself while hasMore is true, advances to nextTable when a step drains, and terminates when there is no next step. The table argument is validated against the literal union in _common.ts, so a typo is a compile-time and boot-time error rather than a silent no-op.

Any new per-organization table added to the schema must also add a literal to OrganizationDeletionTable and a sibling step module, or the deletion cascade will leave it orphaned.

System health and in-app self-update (updater sidecar)

The System & Updates page (/dashboard/settings/system, platform-admin only) surfaces the current version, container health, the GitHub release check, the update flow, and update history.

Update check

apps/api/convex/systemUpdates.ts polls the GitHub Releases API (https://api.github.com/repos/wolvesdotink/owlat/releases/latest), caches the result in a singleton systemUpdates row (kind: 'latestCheck') with a 1-hour TTL, and computes updateAvailable by comparing the cached latest version to the running OWLAT_VERSION.

A local build reporting OWLAT_VERSION=dev (or a non-semver value) is always treated as "no update available", because Owlat cannot tell whether a dev build is ahead of or behind the release tag.

Applying an update

Clicking Update now posts to the Nitro route apps/web/server/api/system/update.post.ts, which:

The updater sidecar

The updater (apps/updater/src/index.ts, image ghcr.io/wolvesdotink/updater, listening on port 3200, exposed only on an internal docker network (no host port mapping) so only the web container can reach it via http://updater:3200) is the only component that touches Docker. It authenticates every request with a timing-safe X-Instance-Secret compare and reaches Docker through a least-privilege socket proxy (it can pull/recreate/list containers but not exec, build, or touch volumes).

The System & Updates page reads /health (proxied by apps/web/server/api/internal/updater-health.get.ts) to render the container-health table. The convex-deploy-before-restart ordering is the safety property: an incompatible schema fails the deploy step and the running Web/MTA containers are never restarted against a half-deployed backend.

For the operator-facing update playbook (rollback, recovery, automation), see Maintenance & Updates.

Dev-mode endpoints, crons, and migrations

Dev-mode endpoints

A small set of destructive shortcuts exist for local development. They are fail-closed: the guard in apps/api/convex/devShortcuts/_guard.ts treats the deployment as production unless the operator explicitly sets OWLAT_DEV_MODE=true in the Convex backend runtime env (npx convex env set OWLAT_DEV_MODE true). The CLI-side CONVEX_DEPLOYMENT env var is not propagated into the function runtime, so it cannot be used as a security boundary.

Crons

Scheduled jobs are registered in apps/api/convex/crons.ts. Operationally relevant ones include:

Counter-reconciliation crons (reconcile contact counts, reconcile topic member counts, refresh segment counts, reconcile transactional send counts) keep denormalized counters honest against any drift from partial failures.

Migrations

Schema/data migrations are one-shot internal mutations under apps/api/convex/migrations/, each exporting a run mutation. They follow the pre-prod "atomic breaking change" pattern: each migration is idempotent (re-running is a no-op once every row is rewritten) and, because deployments are single-org and resettable, most are written to run synchronously against .collect().

You invoke one by name through the Convex CLI, for example:

npx convex run migrations/0033_campaign_audience:run

Related pages