From 7ad97cc0f8ae14f10eb8ae5a4c759027595ab3f6 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 8 Aug 2026 18:25:53 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20cross-model=20doc-review=20fixes=20?= =?UTF-8?q?=E2=80=94=20cache-key=20phrasing,=20counter=20scope,=20slug-fen?= =?UTF-8?q?ce=20allow-list,=20entry=20date?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - KEY_FILES.md: the per-call reranker-override cache-skip rationale now says the rrm= key part is bundle/config-resolved (a reranker field IS in the key; it just never reflects a per-call override); cross-source drop prose reworded to current-state (clears the doc guard's history-marker warn). - README.md: scope the skipped-candidate counting to the DB extract paths (--source db, --stale) — the default fs walk can't see other sources. - brains-and-sources.md: the slug-fence bullet now matches CLIENT_FENCED_WRITE_OPS — a fenced allow-list of write ops is available to slug-bound clients and everything else non-read is refused (fail-closed), instead of the self-contradictory "non-read operations are refused". - CHANGELOG.md: 0.42.78.0 entry date 2026-08-09 → 2026-08-08 (matches the release commit and adjacent entries; header only, content untouched). - llms.txt / llms-full.txt regenerated. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- README.md | 2 +- docs/architecture/KEY_FILES.md | 4 ++-- docs/architecture/brains-and-sources.md | 5 ++++- llms-full.txt | 2 +- 5 files changed, 9 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e2784aa1..5ae2f4ac0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ All notable changes to GBrain will be documented in this file. -## [0.42.78.0] - 2026-08-09 +## [0.42.78.0] - 2026-08-08 **Search stops collapsing rich result pools on weak matches, cross-source wikilinks stop vanishing silently, and the remote-access rules are written down in one place.** diff --git a/README.md b/README.md index b237a9a6f..051a369e6 100644 --- a/README.md +++ b/README.md @@ -264,7 +264,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Multi-source brains:** a wikilink whose target page lives in a *different source* creates an edge once you opt in with `gbrain config set link_resolution.cross_source true` (then run `gbrain extract links --source db`); with it off (the default), the extract summary counts the skipped candidates — cross-source links never vanish silently. +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Multi-source brains:** a wikilink whose target page lives in a *different source* creates an edge once you opt in with `gbrain config set link_resolution.cross_source true` (then run `gbrain extract links --source db`); with it off (the default), the DB extract paths (`extract links --source db`, `extract --stale`) count the skipped candidates in their summaries instead of dropping them silently. **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 0bc1a34bf..2ea113e6d 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -128,7 +128,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/embedding-column.ts` — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a frozen `ResolvedColumn` descriptor (`{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default; throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default so a repointed `embedding` builtin doesn't serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch`, `gateway.embedQuery(text, {embeddingModel, dimensions})`, `cosineReScore`, and the `query` MCP op (per-call `embedding_column` param). Pinned by `test/search/embedding-column.test.ts` (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison). - `src/core/search/rerank.ts` — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, appends the un-reranked tail unchanged (recall protection). Fail-open on every `RerankError.reason`: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` ("fall through to mode bundle"). Also exports `rerankerEmitsNormalizedScores(model)` — does this reranker emit [0,1]-calibrated relevance? `rerank_score` is taken verbatim from the provider (no sigmoid), and the raw-logit local rerankers (`llama-server-reranker`/`qwen3-reranker` name substrings) return unbounded cross-encoder logits; conservative default true for unrecognized models. Consumed by `hybrid.ts`'s autocut weak-top-floor gate (see the `autocut.ts` entry); a `scoreScale` recipe property is the filed successor seam (TODOS.md). Test seam: `opts.rerankerFn` stubs `gateway.rerank` without the network (it does NOT redefine the effective reranker model — only `opts.reranker.model` does). - `src/core/search/return-policy.ts` (default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of returning the full top-K. `entity` intent gets a tight cap; `temporal`/`event`/`general` get a recall-preserving cap. A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench instrumentation (gbrain-evals) measured the rank1→rank2 RRF gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial | undefined`), `adaptiveReturnFromConfig(cfg)`, `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (cache-skip gate check), `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped ≥1). Wired into `hybridSearch` AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa). `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. Agent-facing: the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached` — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract). -- `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). Weak-top floor (`minTopScore`, default 0.5): the cliff test is normalized by the top score, so a weak top (e.g. 0.317 on a rare-term cross-source query) rescales ordinary decay into a confident-looking cliff and collapses a rich pool to 1 — when the top score is below the floor, autocut no-ops and preserves recall. Range [0,1]: 0 disables, 1 pins at the ceiling; out-of-range config values are IGNORED (fall through to bundle/module default — nothing clamps); `applyAutocut` applies `?? 0.5` so a partial-knobs literal caller computes the same trimmed set the cache key claims. The floor is only meaningful on [0,1]-normalized reranker scores: `hybrid.ts` gates it on the EFFECTIVE reranker model via `rerank.ts:rerankerEmitsNormalizedScores` (a per-call `SearchOpts.reranker.model` override wins over the bundle default) and passes 0 (disabled) for raw-logit local rerankers (llama-server/Qwen3, unbounded scores); the same per-call model override also skips the query cache in `hybridSearchCached` — no reranker field is in the cache key, so a row written under one reranker must never serve a caller overriding another. WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1, minTopScore=0.5`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump`/`autocut_min_top_score` (conservative false, balanced/tokenmax true@0.20; floor 0.5 in all three bundles) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). Autocut folds into `knobsHash` as its own parts entries, incl. the floor as `acmts=` at 4 decimals — nearby floors can flip trim-vs-no-op and must not collide on the cache key (`mode.ts:KNOBS_HASH_VERSION` is the single source of truth for the current hash version; every bump is a one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`, `search.autocut_min_top_score`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn, incl. the floor cases), `test/search/autocut-floor-knobs.test.ts` (config→bundle resolution ladder for the floor), `test/search/autocut-floor-gate.serial.test.ts` (raw-logit gate + per-call reranker-override cache-skip seam, real PGLite), `test/search/knobs-hash-reranker.test.ts` (`acmts=` cache-key pins), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the knobsHash assertions in `test/search-mode.test.ts`. +- `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). Weak-top floor (`minTopScore`, default 0.5): the cliff test is normalized by the top score, so a weak top (e.g. 0.317 on a rare-term cross-source query) rescales ordinary decay into a confident-looking cliff and collapses a rich pool to 1 — when the top score is below the floor, autocut no-ops and preserves recall. Range [0,1]: 0 disables, 1 pins at the ceiling; out-of-range config values are IGNORED (fall through to bundle/module default — nothing clamps); `applyAutocut` applies `?? 0.5` so a partial-knobs literal caller computes the same trimmed set the cache key claims. The floor is only meaningful on [0,1]-normalized reranker scores: `hybrid.ts` gates it on the EFFECTIVE reranker model via `rerank.ts:rerankerEmitsNormalizedScores` (a per-call `SearchOpts.reranker.model` override wins over the bundle default) and passes 0 (disabled) for raw-logit local rerankers (llama-server/Qwen3, unbounded scores); the same per-call model override also skips the query cache in `hybridSearchCached` — the cache key's `rrm=` part is bundle/config-resolved and never reflects a per-call override, so a row written under one reranker must never serve a caller overriding another. WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1, minTopScore=0.5`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump`/`autocut_min_top_score` (conservative false, balanced/tokenmax true@0.20; floor 0.5 in all three bundles) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). Autocut folds into `knobsHash` as its own parts entries, incl. the floor as `acmts=` at 4 decimals — nearby floors can flip trim-vs-no-op and must not collide on the cache key (`mode.ts:KNOBS_HASH_VERSION` is the single source of truth for the current hash version; every bump is a one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`, `search.autocut_min_top_score`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn, incl. the floor cases), `test/search/autocut-floor-knobs.test.ts` (config→bundle resolution ladder for the floor), `test/search/autocut-floor-gate.serial.test.ts` (raw-logit gate + per-call reranker-override cache-skip seam, real PGLite), `test/search/knobs-hash-reranker.test.ts` (`acmts=` cache-key pins), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the knobsHash assertions in `test/search-mode.test.ts`. - `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. Declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio), avoiding the backfill loop where tiktoken-grounded budgeting undercounted Voyage's actual token usage. Declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. Recipe docstring at `:7-16` names the seven hosted flexible-dim models that accept `output_dimension` (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and notes `voyage-4-nano` is the open-weight variant fixed at 1024-dim that does NOT accept the parameter (negative regression assertion in `test/ai/gateway.test.ts`: `dimsProviderOptions` returns `undefined` for `voyage-4-nano`). `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`); discoverability surfaces: decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`. - `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). Canonical id is `claude-sonnet-4-6` (no date suffix); a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize`). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts`. - `src/core/model-pricing.ts` — single source of truth for paid-cloud CHAT/completion model pricing (USD per 1M tokens, input | output). `CANONICAL_PRICING` is a `provider:model`-keyed table (Anthropic Opus 5/4.8/4.7/4.6 `$5/$25`, Sonnet 4.6 `$3/$15`, Haiku 4.5 `$1/$5` both dateless + dated, plus OpenAI / Google / Together / DeepSeek panel models). `canonicalLookup(modelId)` resolves bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and slash (`anthropic/...`) forms — bare ids default to the `anthropic:` provider; nested OpenRouter ids (`openrouter:anthropic/...`) intentionally MISS so OpenRouter markup isn't repriced as the inner vendor. Every other chat-pricing table is a DERIVED view of this one (NOT a hand-copied duplicate), so cross-table price drift is structurally impossible. Embeddings live separately in `embedding-pricing.ts` (different unit). Pinned by `test/model-pricing.test.ts` whose drift guard asserts each derived view equals canonical and that the cross-modal panel models are all present. @@ -206,7 +206,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/conversation-parser/` — 17-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (17 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md). - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. -- `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. #2576: markdown links, bare-slug prose refs, and slash-shaped wikilinks match ANY dir-shaped path (`ANY_DIR_SEGMENT`), not a directory whitelist — nonexistent targets are dropped by the persist paths' page-existence checks (`resolveCandidateSources`, put_page's allSlugs filter, `addLinksBatch` INNER JOINs) and counted as `skippedMissingTarget` in the extract summaries; the `DIR_PATTERN` whitelist survives only as the typed fast-path for pass-2b wikilinks (non-whitelisted `[[dir/...]]` get an equivalent direct typed candidate in pass 2c, plus the flag-gated suffix rescue for non-exact matches). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). **Opt-in cross-source wikilink edges** (issue #2589, default off): `isCrossSourceLinksEnabled(engine)` (env `GBRAIN_LINK_RESOLUTION_CROSS_SOURCE` → DB config `link_resolution.cross_source` → default false). On, a wikilink whose target slug exists ONLY in other sources (neither the origin page's source nor `'default'`) still produces an edge, with `to_source_id` picked deterministically — the lexicographically smallest matching source, so repeated extracts and both engines converge on the same edge under the `(source_id, slug)` composite key. Off, those candidates are counted (`skippedCrossSource`) and surfaced in the extract summaries with a paste-ready enable hint — never silently dropped (pre-fix they were indistinguishable from missing endpoints and multi-source graphs went sparse with `dead_links` stuck at 0). SCOPE: the DB extract paths only (`extract links --source db`, `extract --stale`) — the FS-walk paths build their slug set from one source's walked files, so cross-source targets aren't resolvable there (FS-walk parity is a filed follow-up in TODOS.md). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. +- `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. #2576: markdown links, bare-slug prose refs, and slash-shaped wikilinks match ANY dir-shaped path (`ANY_DIR_SEGMENT`), not a directory whitelist — nonexistent targets are dropped by the persist paths' page-existence checks (`resolveCandidateSources`, put_page's allSlugs filter, `addLinksBatch` INNER JOINs) and counted as `skippedMissingTarget` in the extract summaries; the `DIR_PATTERN` whitelist survives only as the typed fast-path for pass-2b wikilinks (non-whitelisted `[[dir/...]]` get an equivalent direct typed candidate in pass 2c, plus the flag-gated suffix rescue for non-exact matches). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). **Opt-in cross-source wikilink edges** (issue #2589, default off): `isCrossSourceLinksEnabled(engine)` (env `GBRAIN_LINK_RESOLUTION_CROSS_SOURCE` → DB config `link_resolution.cross_source` → default false). On, a wikilink whose target slug exists ONLY in other sources (neither the origin page's source nor `'default'`) still produces an edge, with `to_source_id` picked deterministically — the lexicographically smallest matching source, so repeated extracts and both engines converge on the same edge under the `(source_id, slug)` composite key. Off, those candidates are counted (`skippedCrossSource`) and surfaced in the extract summaries with a paste-ready enable hint — never silently dropped (a silent drop is indistinguishable from a missing endpoint, the failure mode that leaves multi-source graphs sparse with `dead_links` stuck at 0). SCOPE: the DB extract paths only (`extract links --source db`, `extract --stale`) — the FS-walk paths build their slug set from one source's walked files, so cross-source targets aren't resolvable there (FS-walk parity is a filed follow-up in TODOS.md). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). No in-memory dedup pre-load — candidates buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, `created` counter returns real rows inserted. `resolveCandidateSources` (pure helper shared by `extractLinksFromDB` + `extractStaleFromDB` so multi-source resolution can't drift) validates both endpoints exist, then picks from/to source ids — origin page's source first, `'default'` fallback; with `link_resolution.cross_source` on (see `link-extraction.ts:isCrossSourceLinksEnabled`) a target that exists only in OTHER sources resolves to the lexicographically smallest matching source, and with it off the helper returns the distinguishable `'cross-source-only'` sentinel so callers COUNT the drop instead of burying it. Both DB paths track `skippedMissingTarget` + `skippedCrossSource`, print summary lines with the enable hint, and ride the counters on their return values; the `--json` surfaces carry `skipped_missing_target`/`skipped_cross_source` (additive on `runExtract`'s links-path result; on the `extract_stale_done` event for `--stale`). The `--stale` hint says to re-run `extract links --source db` after enabling — a `--stale` re-run will NOT revisit already-stamped pages. Pinned by `test/extract-cross-source-links.test.ts` (pure resolution + counters) and `test/extract-cross-source-links-db.serial.test.ts` (real-PGLite config ladder + edge writes). `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id ` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id ] [--catch-up] [--dry-run] [--json]` branch (`extractStaleFromDB`) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three new `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush: link/timeline flush throws propagate and abort the batch; stamp LAST so a crash leaves pages unstamped and they re-extract idempotently (`addLinksBatch` ON CONFLICT DO NOTHING + timeline dedup). Race fix: `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Schema parity in schema.sql + pglite-schema.ts + schema-embedded.ts + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` gains `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` for `pagesAffected` at the inline-extract call site, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` adds `'extract.stale'` to `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` adds `links_extraction_lag` to `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp regression + crash-contract), `test/sync-inline-extract-stamps.serial.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite regression in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. - Extract CLI help — `EXTRACT_HELP` in `src/commands/extract.ts` is the canonical detailed usage shared by `--help` and invalid-subcommand errors. `src/cli.ts` routes `extract --help` before engine connection so help works on unconfigured installs; the top-level TOOLS block advertises every mode-specific flag. Pinned by `test/cli-help-discoverability.test.ts`. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` — unified extract operator surface. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts`. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. `extract_health` doctor check reads last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok`. CLI: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted halt_rate desc + cost desc, top-5 + "more rows" hint, stable `schema_version: 1`); `gbrain extract --explain ` (resolution chain pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)`, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; ships as a stub-reporter). `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` — parses but refuses at runtime); `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Pinned by `test/extractable-spec-widening.test.ts` (22), `test/extract/receipt-writer.test.ts` (12, canonical PGLite block R3+R4), `test/extract/benchmark.test.ts` (17), `test/extract/status.test.ts` (15), `test/schema-pack/scaffold-extractable.test.ts` (15, privacy guards), `test/doctor-extract-health.test.ts` (8). diff --git a/docs/architecture/brains-and-sources.md b/docs/architecture/brains-and-sources.md index 028040b00..86763c8f3 100644 --- a/docs/architecture/brains-and-sources.md +++ b/docs/architecture/brains-and-sources.md @@ -248,7 +248,10 @@ fail-closed and tested: - **Takes holders** — per-token allow-lists (`gbrain auth permissions set-takes-holders ...`) scope which held takes a remote caller sees. - **Write-side slug fences** — a client bound to slug prefixes can only write - under them, and non-read operations are refused for slug-bound clients. + under them, and only a small fenced allow-list of write operations + (`put_page`, `add_link`, `add_timeline_entry`, …) is available to + slug-bound clients; every other non-read operation is refused (fail-closed: + a write op added later is denied until it is fenced and allow-listed). One known soft edge: the backlink-count ranking boost counts referrers without source filtering, so the *existence* of out-of-grant referrers can diff --git a/llms-full.txt b/llms-full.txt index 13522ca50..6a9dd5d61 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1778,7 +1778,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Multi-source brains:** a wikilink whose target page lives in a *different source* creates an edge once you opt in with `gbrain config set link_resolution.cross_source true` (then run `gbrain extract links --source db`); with it off (the default), the extract summary counts the skipped candidates — cross-source links never vanish silently. +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Multi-source brains:** a wikilink whose target page lives in a *different source* creates an edge once you opt in with `gbrain config set link_resolution.cross_source true` (then run `gbrain extract links --source db`); with it off (the default), the DB extract paths (`extract links --source db`, `extract --stale`) count the skipped candidates in their summaries instead of dropping them silently. **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.