76 — Intake and knowledge: one path for data in, one place to use it

Status: proposal, 2026-09-27. Nothing here is shipped. It supersedes the feed system's target model in docs/design/51-feed-system.md (which stays the accurate description of the shipped 5.x Python pipeline until this lands) and is the intake-and-retrieval facet of the data architecture in docs/design/73-data-architecture-and-lifecycle.md and docs/design/74-lifecycle-and-data-administration.md. It depends on that work: M1 (trace key), M2 (object store), M6 (catalog). §8 places it in the plan. §9 is a security fix that lands independently and now.

0. The thesis, and why today misses it

The intended model: one place any data comes in, and once in, it is usable for everything — the agent answering a question, a person searching, an alert firing, a digest, a report. What ships instead is a narrow poller with a human-only reader:

• The agent cannot use feeds. feed_mcp.py is invoked only by the Rust HTTP handlers (crates/vak-server/src/feeds.rs); it is never registered as an MCP server the model can reach, and there is no feed_* tool in the agent's toolset. Ingested content is a dead end for the assistant, which answers from webfetch/browse instead.
• "Any data" is one shape. Only RSS/Atom, YouTube, HN/Reddit/Lobsters and custom HTTP. Channel attachments and dropped files take a separate path (save_to_inbox → inbox/, crates/vak-server/src/inbox.rs). Nothing else enters.
• Four disconnected stores. Feeds (DuckDB), memory notes (MEMORY.md), entities (ENTITIES.jsonl) and transcripts (SQLite FTS store.db) are four schemas and three search implementations. session_search merges memory + entities + transcripts but not feeds; the feed's own search_index table is written on every item and read by nothing.
• No lifecycle. Items, the search index, the alert log and the run log are never pruned; max_items_per_feed is loaded and never enforced; a matched alert with no channel target is a silent run error rather than a durable signal.

This document makes the thesis real: intake is one contract feeding the catalog, retrieval is one query the agent and every surface share, and lifecycle is the shared reconciler — not a subsystem beside the platform.

1. Vocabulary

• Source — a durable declaration that produces items over time by pulling (an RSS feed, a channel, a mailbox, a drive folder). Desired-class state owned by an Agent (docs/design/73 §5), with a src_ id.
• Intake — the one pipeline every item travels, whatever its origin.
• Item — one observation from a source or a push. It is an Object (content-addressed body) plus a catalog node with a TraceKey, derived_from (its source and the run that fetched it), and a security disposition. It is never a bespoke row in a private DuckDB file.
• Connector — host code (Rust, or a pinned worker-shipped script) that knows how to fetch and normalise one source kind. Connectors are a closed, host-owned set, never resolved from a workspace (§4).
• Push intake — an item that arrives without a source: a channel attachment, a file dropped in a client, a URL or note a person hands in. save_to_inbox becomes one push connector, not a parallel path.

2. The one pipeline

Source (pull)  ┐
Push (attach,  ├─▶ Run (cause: Schedule|Channel|User)  docs/design/73 §4,§8
 drop, "save") ┘        │
                        ▼  in the broker worker, sandboxed, SSRF-checked (§4)
                 Fetch ─▶ Normalise ─▶ Detect (label, never drop; §5)
                        │
                        ▼
     Object store (content-addressed, deduped)   +   Catalog rows
        body, dedup by object id                     nodes: item, TraceKey,
                                                      derived_from, disposition
                                                      text: FTS projection
                                                      edges: produced_by,
                                                             references
                        │
        ┌───────────────┼───────────────────────────┐
        ▼               ▼                             ▼
  Agent tool      UI reader / search            Alert evaluation
  (§6, ACL)       (§6, ACL)                     → Inbox + optional channel (§7)
                        │
                        ▼
              Lifecycle reconciler (§8): retention, dedup window,
              max-items, seal, erase — the shared one, not a feed cron

Every stage is a Run record (docs/design/73 §8): a poll that fetched nothing, a fetch that failed, a source not due, a blocked SSRF target — each is a typed decision, never eprintln! + return. This fixes today's "a failed fetch is recorded as empty" and "a source that never succeeds has no row" directly, because the record exists whatever the outcome.

3. Identity and classes

An item maps onto the existing data classes (docs/design/73 §5); intake introduces no new storage model.

Consequences that fix today's leaks by construction:

• Scope is the space/audience, not a cwd hash. An item is visible through the catalog's ACL filter (docs/design/73 §9, invariant 37), so the cross-workspace leak in the current reader cannot recur — there is one filtered query, not a scoped path and an unscoped one.
• Quarantine and removal are node state, applied before ranking in the one query, so no surface can forget them (today get_latest does).
• Every item has a stable catalog id the UI can open — the current reader returns rows with no id.

4. Security — the boundary is the worker, not the server

Today the feed scripts run in the server process: feeds.rs spawns python3 with .envs(...) added to the inherited environment (no env_clear), and feeds_dir() prefers the workspace's own scripts/feeds/feed_ingest.py. That is arbitrary code execution with the server's secrets — see §9, reproduced live.

The redesign closes it structurally:

• Connectors are host-owned and closed. Fetch and parse run in the broker worker (invariants 14, 39), under a network-scoped, env_cleared sandbox with only the operational allowlist — never the server's VAK_GATEWAY_TOKEN or provider keys (invariant 12). A workspace can never contribute a connector, and no path is ever resolved from cwd.
• Untrusted parsing runs where hostile bytes belong — the same worker, network-denied for the parse step, bounded in size and time — because feeds are external content exactly like an Office file (invariant 39).
• Enabling intake is privileged. feeds.enabled / intake config is stripped from untrusted project layers, like hooks/mcp/gateway (docs/design/73 and the demotion block in crates/vak-config/src/lib.rs).
• SSRF and redirect policy are the host's, applied to every source URL and every redirect target, once, in the worker — not re-implemented per driver.

5. Detection labels, never drops

Today the injection detector runs inside every driver and discards any item that matches, and the false-positive rate makes it unusable: ordinary tech headlines ("a tiny eval() sandbox", "system: notes", "pretend you are a compiler", "Assistant: we shipped it", "<s>old</s>") are dropped, and the recorded evidence is an empty string.

The contract (already stated in docs/design/51 and 73/74, now enforced):

• Detection labels, it does not drop. An item gets a disposition (accepted, quarantined, blocked) and retained evidence; quarantined content is excluded from agent retrieval and alerts until an operator releases it, but it is never silently lost and its evidence is never empty.
• Instruction-like content from a source is data and cannot change the agent's authority, tools, prompt layers or permissions — enforced by the boundary (§4), not by pattern-matching prose.
• The detector is tuned against a real headline corpus and its false-positive rate is a tracked metric, because a filter that quarantines a third of a normal feed trains people to bulk-release.

6. Retrieval — one query, and the agent can finally use it

• One catalog search (docs/design/73 §9: FTS5 + optional vector) replaces the feed's unread search_index, store.db, and the ad-hoc scans. It ranks by relevance first (freshness/trust only re-rank a match, never create one), filters by the caller's audience and grants before ranking, and returns only what the caller could open directly.
• The agent gets a tool. Intake content becomes reachable in a turn through one retrieval tool over the catalog. The cleanest shape is to widen the existing recall rather than add a fourth search: session_search (crates/vak-core/src/session_search.rs) already merges transcripts, memory and entities — it gains catalog items as another source, or is renamed to a single recall/knowledge_search that the model reaches for by default (open decision D1, §10). Either way, "usable for various things" means the assistant, not only a person in a tab.
• The UI reader and alerts call the same query, so there is one relevance model and one ACL, not three.

7. Alerts reach the durable attention layer

An alert match becomes a durable Inbox entry (a new vak_core::inbox::Kind, e.g. FeedMatch), so nothing is silently dropped when no channel is configured — every other signal already lands there (docs/design/29). Channel delivery stays optional and goes through the one host delivery runtime (vak-delivery); intake code never writes an outbox record itself. Evaluation is idempotent per (alert, item) with a real cooldown, fixing today's re-fire-every-tick and unbounded alert_log.

8. Lifecycle and sequencing

• Retention is the shared reconciler (docs/design/74 §5), not a feed cron. Intake objects and records carry retention labels; the reconciler seals, tiers, prunes and erases them with everything else. max_items per source and the dedup window are enforced there, and a run row is written only when work happened (not once per idle 20-second tick, as today).

• Erasure reaches intake because items are objects with derived_from; erasing a conversation or a source erases its derived catalog rows and embeddings and GCs orphaned objects (docs/design/74 §4).

• Sequencing. Intake is a consumer of the catalog, so it builds on:

  • M1 TraceKey — every item carries one;
  • M2 object store — item bodies are objects;
  • M6 catalog — nodes/edges/text are where items live and are searched.

  It is therefore an M6-or-later deliverable (proposed M6.5, before or with M8 artifacts). It does not start before M6, and it ships whole (invariant 30): the Python pipeline, the DuckDB store, feed_mcp.py, feed_search.py's BM25, the search_index table and the four dead helpers are deleted in the same change that lands catalog intake. There is no compatibility path and no migration (0 users; 5.x refuses pre-baseline state, invariant 29).

9. The one fix that lands now, independently

The code-execution hole (§4) is live in 5.x and must not wait for M6. Verified on a running server against an untrusted /tmp workspace (trusted: false): a .vak/config.toml with [feeds] enabled = true plus a workspace scripts/feeds/feed_ingest.py caused the scheduler to run that script every 20 seconds unattended, and the script read VAK_GATEWAY_TOKEN from the inherited environment. A feed script has network access by design.

A small change on the current pipeline, independent of the redesign:

1. Strip feeds.enabled (and any intake config) from untrusted project layers in crates/vak-config/src/lib.rs, beside hooks/mcp/gateway. An untrusted repo cannot enable an unattended surface (invariant 15).
2. env_clear the feed subprocess in crates/vak-server/src/feeds.rs and pass only the operational allowlist plus the VAK_FEEDS_* paths — the parent environment, gateway token and provider keys never reach it (invariant 12).
3. Resolve connectors from the install, never cwd — drop the cwd/scripts/feeds preference in feeds_dir() so a workspace cannot supply the script the server runs.

These three are the interim boundary until §4 makes it structural. They are a separate, small change from the redesign and carry their own regression tests (an untrusted workspace's feeds.enabled is ignored; the subprocess sees no parent secret; a workspace scripts/feeds is never executed).

10. Open decisions (need the maintainer)

• D1 — one recall tool or a named intake tool? Widen session_search into a single recall over transcripts + memory + entities + intake, or add a distinct knowledge_search. Recommendation: one recall — fewer tools measure better on small models, and "everything I know" is one question.
• D2 — connectors in Rust or worker-Python? Rewrite the drivers in Rust (one language, no Python runtime dependency, easiest to sandbox), or keep the parsers as pinned worker-shipped Python behind the broker. Recommendation: Rust for RSS/HTTP/YouTube-XML (small), reassess for anything needing a large parser.
• D3 — is intake M6.5 (before M8) or folded into M8 artifacts? Recommendation: M6.5, so retrieval is unified as soon as the catalog exists.
• D4 — do we ship §9 now as a standalone 5.x change ahead of the redesign? Recommendation: yes — it is a live token-exfil hole.