SLW Data Engine — spine build brief (workstream: engine)

You are the engine workstream session. You build the spine of the SLW Data Engine. Work autonomously; this brief defines scope, rules, checkers, and stop points. When in doubt, the rules win over speed.

The decision this executes (Shawn, 2026-08-07)

Webster and Dewey merge as a concept, not as code: one product — the SLW Data Engine — with three internal organs kept as separate services:

• substrate (Dewey) — documents in, one embedding space, cited search
• registry (Webster) — entities: people & firms, resolution, links
• spine (this build) — what connects them: mentions, typed tags, relationship edges, entity creation

Apps talk to "the engine." Nobody downstream needs the internal names.

Read first (source of truth, in order)

1. /root/slw-system-design/architecture/future-ai-plan.md — the plan this brief is Phase 1 of
2. /root/FUTURE-AI-AUDIT-2026-08-04.md — the code audit (what exists, what is missing, where each matcher lives)
3. /root/slw-webster/README.md, BRIEF.md, migrations/ — the registry you are extending; its grain and conventions are yours to follow
4. /root/slw-ford/docs/decisions.md, entry 2026-08-07 — ford.intake_links, the working precedent you are generalizing
5. /root/slw-dewey/DESIGN.md — the substrate. Read-only. You never modify Dewey.

What you build (in order)

1. Spine schema design doc — mention store (doc X mentions entity Y, with source + as-of + passage provenance), typed tag vocabulary (sectors/topics/themes as a controlled table, not strings), relationship edges (investor-in / partner-at / led-round, typed + dated), and an entity-create path (propose-then-approve; the registry still never merges). Include: how Ford's intake_links maps onto it, and how each of the three existing app matchers would call it. STOP POINT 1 before any migration.
2. Migrations — additive only, new tables in the Webster database, numbered in slw-webster/migrations/. Backup immediately before each apply.
3. Spine API in the Webster service — mention write/read, tag vocabulary read, edge write/read, create-entity proposals. Same auth and conventions as the existing resolve/decide API.
4. Match assessment layer (AI on the queue, never past it) — a model pass over ambiguous suggestions (the 0.55–0.75 confidence band): each gets a verdict (same / different / unsure), a one-line rationale, and a model confidence, stored on the suggestion with the model + prompt version so it is re-runnable. The review queue sorts and groups by verdict for bulk-accept. The model never confirms a link itself — confirm stays a human decide() with actor + audit trail, unchanged. Fold Terminal's Haiku-tiebreaker pattern (fail closed) into the engine so there is one assessment brain, not one per app.
5. Matching policy as config — the tier weights, thresholds, ambiguous band, and auto-link bar move from hardcoded literals into a versioned matching_policy table, editable from the admin queue UI. Every stored assessment and auto-link records the policy version that produced it, so a policy change is a setting plus an optional re-run — not a deploy.
6. Backfills, one at a time — Ford intake_links → spine; Podcast's private entity table → registry proposals; Terminal literature mentions → spine. Each: idempotent, resumable, counts logged, nothing silently dropped. Sample of 50 first — STOP POINT 2 before full volume.
7. Freshness — nightly delta enrollment — the registry's knowledge of LP Flow and DealFlow is the 2026-08-03 seed snapshot. Build the delta job: nightly snapshot diff enrolls new/changed CRM records (exact keys auto-link as today; everything else becomes a suggestion). Reuse the existing snapshot machinery (scripts/snapshot.sh, source_snapshots); runs on the slwdev cron pattern the fleet already uses.
8. Registry dedup proposals — entity creation makes in-registry duplicates possible. Detect probable duplicates (shared keys, near-name within same kind) and surface merge proposals to the admin queue — propose-then-approve, never auto-merge, full audit like every decide().
9. Coverage scoreboard — one metrics page on the engine: per-app linked coverage %, queue depth, auto-link rate, assessment precision over time. Reads from the tables that already exist; no new instrumentation service.
10. Eval gate — a golden set (~50 known mention/link cases, including known-ambiguous pairs the assessment layer must call correctly) with a scored precision threshold. No consumer flag flips until it passes; the score lives in STATUS.md.
11. The engine gets its face — one page in slw-system-design (site section or architecture doc) presenting the Data Engine as one thing with three organs; retire "Webster vs Dewey" from any doc that confuses them.

Phases 2–4 of the plan (extractors onto the spine, Ford adapters, cards) are not this session's scope. Land the spine, prove it, stop.

Rules (the safeguards — non-negotiable)

• Additive-only database changes. New tables and views only; never ALTER or DROP an existing table. pg_dump (or Supabase backup) before every migration apply; note the backup path in STATUS.md.
• Branch → PR → green CI → merge, every repo, including slw-webster. Never commit to any main directly.
• The registry links, never merges — unchanged. Nothing writes into LP Flow, DealFlow, Terminal, or Podcast databases. Dewey code and worker: untouched.
• Provenance on every row: source + as-of, enforced at the schema level.
• Model-proposed links below the exact-key bar go to the review queue — never auto-committed. The assessment layer advises the queue; only a human decide() (actor + reason + audit event) confirms. Derived flags computed by code, not asserted by a model.
• Every model verdict and auto-link is reproducible: it records model, prompt version, and matching-policy version. An assessment that cannot say what produced it does not get stored.
• Everything consumer-visible ships behind a flag defaulting off.
• Spend discipline: for any model-assisted backfill, run the 50-item sample, report cost and accuracy, then wait (STOP POINT 2).
• Do not touch: Terminal/Podcast ingestion pipelines, the CRM books, Dewey, dev-portal, DNS/email routing. No credentials hunting — if a key is not on this server, that is a stop point, not a scavenger hunt.

Checkers (how work proves itself)

• Webster service tests green on every PR; new spine endpoints ship with tests in the same suite.
• Migration verify: after each apply, a read-back script confirms the new tables exist and existing Webster tables are byte-identical in row counts.
• Backfill ledger: every run logs attempted / written / held / skipped; a nonzero silent-drop is a bug, stop and report.
• Eval gate score printed in STATUS.md before any flag discussion.
• Watch the post-merge CI run on every merge; a red run is a blocker, report it immediately.

Stop points (pause, update STATUS.md, report to Shawn)

1. Spine schema design doc — before any migration is applied.
2. Each backfill's 50-item sample — accuracy + cost, before full volume.
3. Before merging any PR into a consumer app (anything outside slw-webster and slw-system-design).
4. Any missing credential or permission — report, work another thread.

Cadence

Keep initiatives/data-engine/STATUS.md (this folder) current — done / in flight / next / blocked — and commit it with every milestone. Blocked on Shawn → note it there and work another thread. End every session with a handoff summary in STATUS.md.