74 — Lifecycle management and data administration

Status: proposal, 2026-09-25. Nothing here is shipped. This is the companion to docs/design/73-data-architecture-and-lifecycle.md (the data model) and to docs/plans/data-architecture-plan.md (milestones), and it answers review findings R2, R8, R26, R28, R33 and R34 in docs/plans/data-architecture-review.md. It defines how every kind of Vak data is born, changes state, and ends, which policies govern that, how people see and act on it, and which screens, APIs and commands that takes.

1. Rules

1. People ask; the reconciler acts. No screen, API or command deletes bytes directly. They create or approve a request (trash, erase, release, expire). One level-triggered reconciler performs every destructive transition, so policy, holds, leases and recording are applied in exactly one place.
2. Every transition is a record in audit/lifecycle: subject, from, to, cause (policy, request, hold, quota), actor, time and TraceKey. "Why is this gone?" always has an answer that contains no content.
3. Preview before commit. Every destructive action shows its impact first, computed from catalog lineage: counts per class, derived copies, bytes, and processors outside our reach. The commit carries the preview's digest; if the impact changed in between, the commit is refused and re-previewed.
4. Holds beat everything: user delete, retention expiry, quota pressure, Agent or space erasure, tenant offboarding.
5. Records expire per conversation, never per entry. Partial deletion would break derive_messages() and the hash chain. Non-content shared ledgers (cost, routing, activity) expire by sealed segment.
6. Recoverable before irreversible. Trash is hidden everywhere and restorable. Erasure destroys keys and cannot be undone. The gap between them is policy (default 30 days).
7. Receipts are honest. An erasure receipt lists what was destroyed and what Vak cannot reach: messages already sent to Telegram, Slack or Discord, prompts already processed by an LLM provider, and an external NATS deployment.
8. Screens show measured evidence only (invariant 26) and filter by audience before anything else (invariant 37). Operations URLs keep workspace and time context, and every drill-down ends at raw evidence.

2. Lifecycles

Notation: a state list, the transitions (trigger → actor), the data effect, and whether the step is reversible. Every transition writes a lifecycle record (rule 2), so that is not repeated per row.

2.1 Tenant

provisioning → active → suspended → offboarding → shredded

2.2 Space

bound → active → detached → archived → erased

• Detached: every machine binding is removed. Data is kept, and the space can be re-bound. This replaces today's workspaces/forget, which forgets a path and leaves data scattered.
• Archived: read-only. Its schedules are disabled and its retention clocks start.
• Erased is an erasure request with scope space (§3.4). Files in the person's own working tree are never touched: Vak erases what it stored, never the directory a person owns.

2.3 Agent

active → paused → archived → revoked → erased

AgentLifecycle (crates/vak-server/src/agents.rs:15) gains Revoked. erased is not a state on the definition; the definition itself becomes a tombstone.

2.4 Conversation and session

open → idle → sealed → (archived) → trashed → erased, and separately expired by policy.

This replaces today's archive.json/deleted.json sidecars (crates/vak-server/src/lib.rs:9056-9210), whose "delete" only hides from two lists (review R8).

2.5 Run and schedule

• Run: claimed → running → settled{completed | failed | cancelled | abandoned | skipped}. The claim is a CAS on (schedule, slot). A lease that expires while running becomes abandoned. The Run record stays with its conversation's retention; a Run with no conversation (a skip, a script watchdog) follows the space's runs label.
• Schedule: draft → enabled ⇄ paused → disabled → deleted.
  • Deleting a schedule tombstones the definition; its Runs are kept.
  • Pausing records every slot that passes as skipped{paused}, so the history has no silent gap.
  • on_crash = skip | retry_once decides what an abandoned slot does.

2.6 Execution and environment

• Execution: prepared → running → settled → scrubbed. Scrubbing removes runtime/executions/<exe> once the execution settles (plus a grace period). Its stdout, stderr and produced files are objects that follow the owning conversation.
• Environment: reuses vak_sandbox::EnvironmentState: planned → preparing → ready → running → stopped → expired → removed.
  • Expiry comes after the candidate is promoted or rejected, or after the idle TTL, whichever comes first.
  • A worktree environment also removes its branch.
  • An environment with an unreviewed candidate is never removed without the candidate first moving to rejected{expired}, which is recorded and notified.

2.7 Artifact and version

draft → candidate → verified → promoted | shared | published | rejected per version; the artifact itself is active → superseded → expired → erased.

• A version is immutable. Promotion writes into a space's working tree through the existing Review path (invariant 39) and records the target digest.
• Sharing is a grant (§6, screen A6) and never copies data.
• Unpublishing revokes grants and links; external copies are named in the receipt.

2.8 Documents (memory, entities, skills, prompt layers, presentation packs)

active → amended (new version) → forgotten (tombstone).

• Every save keeps the previous content as a version, under the Document's history retention (default 90 days for memory and entities).
• Forget tombstones the current value and schedules history erasure once the history window ends.
• A Document whose derived_from names an erased conversation is erased with it (§4). A person may choose to keep a memory note by re-affirming it, which writes a new version with provenance person.

2.9 Checkpoints

captured → retained → pruned.

• Policy keeps the session baseline, the last N (default 20) while the session is open, and baseline plus final after sealing, for delete_after (default 30 days).
• Checkpoints are space-owned: erasing a conversation removes its manifests' turn linkage and labels, never the space files a person kept.

2.10 Delivery jobs and inbox

• Delivery job: queued → attempting → delivered | dead-lettered → sealed. When a job settles its record moves into the deliveries ledger and the job file is removed. Today delivered jobs are never removed.
• Inbox entry: unread → acked → expired (default 90 days). Entry bodies are content keyed to their conversation (§4).

2.11 Endpoints and allowlist

pending → allowed | denied → revoked. Pending and denied entries hold personal data (sender ids, chat ids) and expire by policy (default 90 days). A denied entry that expires is replaced by a hashed sticky-deny fingerprint, so "a denied entry never re-prompts" (invariant 15) survives without keeping the identifier.

2.12 Incidents, actions and audit

Incidents follow open → resolved → reopened (doc 28) and expire by policy. Lifecycle, erasure, hold, grant and promotion records are audit class: they are retained for the audit label's duration (default 7 years) and contain no content, only ids, digests and reasons. That is what lets them outlive the erasures they record.

2.13 Objects and keys

• Object: live (≥1 key grant or ref) → orphaned → grace → collected.
• Conversation key: active → destroyed. Destruction is erasure.
• Tenant KEK: active → rotating → active. Rotation re-wraps the keys; it never re-encrypts content.

2.14 Telemetry, caches, runtime

These are ephemeral or telemetry classes with no per-item lifecycle, only size and age limits and a boot sweep of runtime/. They never hold content (plan M5), so erasure never needs them.

3. Policies

3.1 Retention labels

RetentionLabel {
    id, name, description,
    rules: [{ class, retain_for?: Duration, delete_after?: Duration,
              on_expiry: erase | archive | review }],
}

• Attached to a tenant, space, Agent or conversation. It inherits downward, and a node may break inheritance and attach its own label, as with SharePoint labels.
• Resolution per class:
  • the minimum keep is the maximum retain_for along the chain;
  • the maximum keep is the minimum delete_after, but never below the minimum keep;
  • holds suspend expiry entirely. A person can make data live longer by labelling. Nobody can make a hold shorter.
• on_expiry = review queues the item for a data steward instead of acting, which is useful for regulated spaces.
• Defaults (tenant label default):

3.2 Legal hold

Hold { id, scope: Tenant | Space | Agent | Conversation | Audience(person) | Query(catalog filter),
       reason, created_by, created_at, released_by?, released_at? }

A hold blocks every destructive transition on anything in scope, including items that enter the scope later (a Query hold is re-evaluated each tick). Creating and releasing a hold are audit records, and releasing needs a reason. A person whose trash is held sees "retained by your organisation's policy" when the tenant allows that disclosure, and otherwise the item simply stays in trash.

3.3 Quotas

• Soft quota: a notification and a Data-posture warning.
• Hard quota on Ephemeral or Derived data: evict caches and derived stores, which are rebuildable.
• Hard quota on Record, Object or Document data: new work is refused at admission with a clear error. It fails closed and never silently evicts records.
• Quotas attach at the tenant, space or Agent.

3.4 Erasure scopes and approval

A request moves previewed → approved → executing → completed | blocked{hold} | failed{reason}. Execution is resumable: the reconciler checkpoints its progress in the request record.

3.5 Export and portability

Export can be scoped by conversation, person, Agent, space or tenant. The bundle contains plaintext records (JSONL), objects, Documents, the manifest and signatures, and is itself a Run with a receipt. It is what offboarding, portability requests and vak data export all produce.

3.6 Roles

Today the admin console has a single operator credential. A customer space needs the following, which M7 introduces:

Full RBAC for the rest of the product is a separate design. Until it lands, a single-operator install holds Owner, and the two-person rules degrade to a typed confirmation plus a delay (default 24 h, cancellable).

4. Erasure mechanics

Erasure is only complete if every copy of the content is keyed or linked. Two rules make that true:

• Content is keyed to its conversation wherever it is written. A content field in a shared ledger (inbox body, delivery text, outbox job payload, commitment statement, turn-card text) is field-encrypted under the conversation key. Non-content fields (ids, timestamps, states) stay readable for audit and scheduling.
• Derived writes record derived_from. Memory notes, entities, skill proposals, FTS rows and embeddings name their source conversation and turn (plan M1).

An erasure of scope S then runs these steps:

1. Resolve S to its conversations through the catalog (for a person: every conversation whose audience includes them).
2. Check holds. Any hold in scope blocks the whole request, recorded as blocked{hold} with the hold ids.
3. Walk lineage outward from those conversations: Documents with derived_from, artifacts produced and not promoted elsewhere, catalog rows, embeddings, key grants, run records, delivery jobs, inbox entries.
4. Destroy conversation keys. Records, field-encrypted content and key grants become unreadable.
5. Remove derived plaintext:
   • catalog rows, with secure_delete and a WAL checkpoint;
   • embeddings;
   • Document versions derived only from the erased scope (a version with another live source is re-derived or kept, and the receipt says which);
   • checkpoint labels and turn links.
6. GC objects whose last key grant was destroyed.
7. Write the receipt:
   • counts per class, and a digest of the erased identifiers;
   • the processors outside reach, with timestamps: channel deliveries from deliveries, provider dispatches from WorkReceipt, the bus if configured;
   • backups that still hold ciphertext, and when they expire. The receipt is an audit-class record, signed, and downloadable.
8. Restore guard. A backup restored later re-applies every erasure tombstone from the audit record before any restored data becomes readable (plan M7).

Headless honesty (doc 73 §7.3): where the KEK sits in the encrypted-file fallback, whose key lives beside it, at-rest encryption is nominal. Crypto-shred still works because it destroys the conversation key.

5. The reconciler

loop every tick (default 10 min; hints wake it early; one pass at boot):
  observe   catalog + refs + policy + holds + quotas + leases   (read-only)
  plan      desired transitions, ordered: runtime scrub → env expiry →
            checkpoint prune → trash/expiry erasure → document history →
            segment seal/tier → object GC → quota eviction
  guard     drop any action whose subject is held, leased by a live turn,
            or whose preview digest changed; record why
  stage     quarantine: move to lifecycle/quarantine/<action>/ (same fs)
  commit    make the change permanent after the stage grace (0 for ephemeral,
            1 tick for everything else)
  record    one lifecycle record per transition; update metrics

• Idempotent and crash-safe. Every action is keyed; a crash between stage and commit resumes; nothing is committed that was not staged.
• Paced. An IO budget per tick (bytes and files) and per-tenant fairness. It never runs a heavy phase while a turn in the same space is mid-dispatch.
• Dry-run is the same plan, printed. vak data gc --dry-run, the admin Lifecycle screen, and doctor all show the identical plan object.
• Undeclared paths found under a Vak-owned root are quarantined for one cycle and listed as incidents. That is a code defect to fix, so the quarantine exists to make it visible, not to hide it.
• Metrics: last tick, duration, actions by kind, bytes reclaimed, blocked by hold, errors. These are shown as measured values only.

6. Admin console

6.1 Navigation

The existing groups stay (crates/vak-admin-ui/src/App.tsx:6862-6945). Changes:

#/operations/work/runs/<session_id> is replaced by #/runs/<run_id> (review R13). Scheduled-task definitions stay editable under Operate › Schedules; there is one place for schedules, not two.

6.2 Screens

Every screen follows rule 8: measured evidence only, audience-filtered, URL carries workspace and time, drill-downs end at raw records, objects or receipts. "Guard" names the confirmation a destructive action needs.

6.3 Client app (people, not operators)

7. API

All routes sit under the existing authentication. Mutations follow invariant 34 (cookie mutations need a recognised Origin; headerless clients need a real token). Destructive routes also require a confirm_digest, the preview digest the caller just displayed, so a stale preview can never authorise a different impact.

GET    /data/status                        classes, quotas, reconciler, integrity summary
GET    /data/usage?scope=&class=&since=    measured usage and growth
GET    /data/lifecycle/plan                the dry-run plan
POST   /data/lifecycle/tick                run now (operator)
GET    /data/lifecycle/transitions?…       lifecycle records
GET    /data/integrity   POST /data/integrity/verify   POST /data/catalog/rebuild
GET|POST|PUT  /data/labels[/{id}]          labels; POST /data/labels/{id}/attach, /detach
GET    /data/labels/effective?node=        resolved policy + chain
GET|POST      /data/holds  POST /data/holds/{id}/release
POST   /data/erasure/preview               → { preview, confirm_digest }
POST   /data/erasure                       { scope, confirm_digest } → request
POST   /data/erasure/{id}/approve | /cancel   GET /data/erasure/{id}[/receipt]
POST   /data/export                        { scope } → Run
GET|POST      /data/keys  POST /data/keys/rotate  POST /data/keys/escrow
GET|POST      /data/backups  POST /data/backups/{id}/restore/preview | /restore
POST   /conversations/{id}/trash | /restore | /archive | /unarchive
GET    /runs  /runs/{id}  /runs/{id}/spans   POST /runs/{id}/cancel | /rerun
GET    /schedules  /schedules/{id}/slots   POST /schedules/{id}/pause | /resume | /run | /backfill
GET    /search  /lineage/{node}  /nodes/{id}
GET    /library …  (M8)
GET    /data/sync …  (M9)

These replace, in the same change (invariant 30):

• POST /sessions/{id}/archive, DELETE /sessions/{id}, DELETE /sessions/archived, POST /sessions/{id}/restore, GET /sessions?trash=true (M0's trash, vak_core::trash)
• POST /memory/cleanup
• POST /workspaces/forget
• POST /backup/export|import
• /agents/{id}/schedule|runs (already deleted in M0)
• the session-keyed Operations "runs"

8. CLI parity

vak data status | usage | plan | gc [--dry-run] | verify | rebuild-catalog | labels … | hold … | erase --scope … [--preview] | export --scope … | keys … | backup … | cat | grep, plus vak runs … and vak schedules ….

Every destructive command prints the preview and requires --confirm <digest> (or an interactive typed confirmation). vak doctor --repair may invoke only reconciler actions classed as mechanical (invariant 19).

9. Acceptance

• Per screen: a browser acceptance run (AGENTS.md acceptance contract) on a /tmp workspace shows real data, and each drill-down reaches raw evidence. A test asserts no screen renders a value that has no API source (invariant 26).
• End-to-end scenarios:
  • person trashes → restores → deletes permanently, and search, recall and prompts never show it again;
  • a steward erases a person across three chats and two Agents, and the receipt lists the Telegram deliveries and provider dispatches it could not reach;
  • a hold placed mid-erasure blocks it;
  • label shortening previews its impact and needs confirmation;
  • a restore re-applies erasures;
  • an Agent revoke cuts its endpoints within one tick while its data stays;
  • a paused schedule records every skipped slot.