Hybrid Wiki + Embedding Retrieval — Architecture

This design document describes an embedding-retrieval layer intended to sit alongside the curated wiki and serve as the substrate for an eventual AI-policy chatbot. It sets out the problem the layer addresses, a three-layer architecture, the query pipeline, the interchangeable technology choices (embedding model, vector store, chunking, refresh cadence), an implementation stub, and two integration paths for the chatbot, together with the design's stated recommendations and its open decisions.

Problem statement

At the time of writing the vault contains 1,003 pages and 497 raw sources. At that scale, a query that re-reads candidate wiki pages costs 30K–80K tokens per answer. The compact Wiki/retrieval-index.md (Tier 2 #6) reduces which pages to read, narrowing a full scan; it does not reduce the per-page read cost when several candidates must be consumed in full.

The stated chatbot goal — an "expert in AI policy that is updated in real time" — exposes the gap, since a chatbot answering hundreds of questions per day cannot read 5–10 full wiki pages per question. The design's response is to treat the wiki as the curated synthesis layer, build an embedding index over Raw Sources/ and Wiki/ as a long-tail retrieval layer, and have the chatbot prefer wiki pages when available and fall back to embedding search over raw sources for the long tail. The document frames this as an added answering layer rather than a substitute for the wiki.

Three-layer architecture

The design separates curated synthesis, raw provenance, and disposable embeddings, with the chatbot consuming candidates from the embedding layer.

┌─────────────────────────────────────────────────────────────┐
│ LAYER 1 — CURATED SYNTHESIS                                  │
│ Wiki/{concepts,companies,entities,...}/*.md                  │
│ Hand-shaped, high-confidence, cross-linked.                  │
│ Source of truth. Compounds.                                  │
└─────────────────────────────────────────────────────────────┘
                          ▲
                          │ folds-into
                          │
┌─────────────────────────┴───────────────────────────────────┐
│ LAYER 2 — RAW PROVENANCE                                     │
│ Raw Sources/*.md  +  New Developments Log/*.md               │
│ Immutable evidence. The fold-source for every wiki claim.    │
└─────────────────────────────────────────────────────────────┘
                          ▲
                          │ embeds
                          │
┌─────────────────────────┴───────────────────────────────────┐
│ LAYER 3 — EMBEDDING RETRIEVAL  (THIS DESIGN ADDS)           │
│ embeddings/*.lance  (vectors over Layer 1 + Layer 2)         │
│ Disposable, regenerated nightly. Cheap to query.             │
│ Returns top-k candidates with citations.                     │
└─────────────────────────────────────────────────────────────┘
                          ▲
                          │ candidates
                          │
                  ┌───────┴────────┐
                  │  CHATBOT       │
                  │  pipeline      │
                  └────────────────┘

Layer 1 — curated synthesis is the hand-shaped, high-confidence, cross-linked wiki content under Wiki/{concepts,companies,entities,...}/*.md, treated as the source of truth. Layer 2 — raw provenance is the immutable evidence in Raw Sources/*.md and New Developments Log/*.md that every wiki claim folds from. Layer 3 — embedding retrieval, the layer this design adds, holds vectors over Layers 1 and 2 in embeddings/*.lance; it is disposable, regenerated nightly, cheap to query, and returns top-k candidates with citations.

Query pipeline

For each user question, the design specifies a seven-step pipeline:

1. Embed the question — a single call to the embedding API, with millisecond latency.
2. Vector search over the embedding index, retrieving top-k=20 candidate chunks (a mix of wiki pages and raw-source chunks).
3. Re-rank to prefer wiki pages over raw sources when both cover the same content, on the grounds that wiki pages are higher-confidence, hand-shaped, and citation-stable while raw sources are the long-tail fallback.
4. Read the top 3–5 candidates in full, where most of the answer-token cost lies; wiki pages are typically short enough to read entirely, while raw sources may need chunked retrieval.
5. Synthesize with Claude, citing each candidate — wiki citations as [[page]], raw-source citations as (Source: Raw Sources/filename.md).
6. Falsify-flag any answer whose top-k contains conflicting candidates, mirroring the contradiction-check in ingest but applied at query time.
7. Log the question, top-k, and answer to a Wiki/queue/processed-queries/ log so future tuning can review which questions worked and which surfaced gaps.

The document characterizes this as roughly the standard RAG-over-curated-knowledge pattern, with two non-standard moves: the wiki layer is preferred over raw sources at re-rank time (which not all RAG systems do), and failed queries surface as wiki gaps to be filled by the next ingest cycle.

Technology decisions

The design states that the technology choices below are not load-bearing for the architecture and are interchangeable. Each carries the document's stated recommendation.

Embedding model

Vector store

Chunking

Wiki pages are typically 500–3,000 words and are chunked at H2 boundaries (one chunk per ## section plus page metadata), producing 3–8 chunks each. Raw sources range from 500 words (news articles) to 15,000+ words (papers) and use a recursive splitter at 1,500 tokens per chunk with 200-token overlap, producing 5–30 chunks each. Each chunk carries its source path, page/section title, frontmatter (status, confidence, last_updated, tags), and the chunk text; the frontmatter serves as the cheap re-rank signal.

Refresh cadence

A nightly delta re-embeds every page modified in the last 24 hours and removes deleted pages from the index, running as part of the scheduled-ingest routine. A weekly full rebuild at Sunday 04:00 regenerates every embedding from scratch to catch drift. At Voyage AI prices the nightly delta is sub-cent and the weekly rebuild is ~$0.50–2.

Implementation

A starter bin/build-embedding-index.py is provided as a deliberate stub: the user fills in two interchangeable functions, embed_chunks and vector_store_upsert, with the chosen embedding model and vector store. The script walks Wiki/, Raw Sources/, and New Developments Log/; chunks by the rules above; maintains a checksum file so unchanged content is not re-embedded; and provides a query helper for testing, for example python3 bin/build-embedding-index.py --query "what's the status of the trump pre-release vetting EO?". The actual code lives in bin/build-embedding-index.py.

The document lists the sequence to bring the layer online: pick the embedding model and vector store from the tables above; run pip install lancedb voyageai (or the chosen pair); open bin/build-embedding-index.py and fill in embed_chunks and vector_store_upsert; run python3 bin/build-embedding-index.py --rebuild for the first full pass; test with python3 bin/build-embedding-index.py --query "..." and verify the top-k looks sensible; add the embedding-rebuild routine to the scheduled-ingest cron (see .claude/skills/scheduled-ingest/SKILL.md); and only then start the chatbot work.

Chatbot integration

The design treats the chatbot as a separate concern for which this architecture provides only the retrieval substrate, and offers two paths. The first is an Anthropic API direct route: a small Python service (FastAPI, Flask, or aiohttp) exposing /chat, where each request embeds the question, runs vector search, re-ranks, reads selected pages, calls the Claude API with a system prompt plus retrieved context, and returns citations, estimated at ~150 lines of Python; the wiki, the embedding index, and this service together constitute the chatbot. The second is the Claude Agent SDK route: the chatbot is built as a Claude agent with a custom retrieval tool wrapping the embedding index, where the agent decides when to retrieve, when to read pages in full, and when to crystallize an answer back as a policy-brief; the document describes this as having more moving parts but better long-term flexibility for "expert" behavior, since the agent can call firecrawl-search for fresh data when the wiki is stale and can offer to ingest a new source. In either case the chatbot calls into this architecture and the architecture does not call out to the chatbot.

Scope and limitations

The document states the design keeps the wiki as the source of truth, makes the embedding layer disposable and rebuildable, has the chatbot prefer curated content, turns failed queries into wiki gaps, and scales the answering layer without changing the wiki's discipline.

It also lists what the design does not address:

• Real-time data — the chatbot still needs firecrawl-search for anything that broke after the last ingest cycle.
• Multi-modal — the wiki is text-only, with no images or audio. The A Taxonomy of Systemic Risks and AI 2027 raw sources have image folders, but the embedding pass treats them as text-only.
• Ranking trustworthiness across sources — the embedding store does not know that an Anthropic system card is more reliable than a Stratechery essay; the wiki's confidence and status fields encode that and the re-rank step uses them, but the result is heuristic.
• The expert-judgment question — a retrieval system plus Claude synthesis is not the same as an expert. The document argues that the wiki's analysis/, briefings/, policy-briefs/, track-record.md, and open-questions.md are what make the chatbot expert-shaped rather than only RAG-shaped, and that this architecture is the substrate rather than the substance.

Audit log

Embedding rebuilds are logged in Wiki/log.md under Operation: Index-Rebuild, the same Operation type used for the retrieval-index, since the audit log treats all index rebuilds as one class.