docs: cross-model doc-review fixes — four dim-pinned columns in the migration guide, exact signal spelling, honest skip-reporting claim

Doc-review pass findings applied:
- docs/guides/embedding-migration.md: schema transition rebuilds four
  dim-pinned text-embedding columns (takes.embedding joined the set)
- docs/architecture/KEY_FILES.md: TAKE_ANCHOR_NOT_FOUND signal carries a
  space after the colon (prefix-matchers need the exact form); takes-lane
  skip reporting only prints when the batch also had embed failures
- docs/architecture/thin-client.md: think op trust-boundary pointer
  updated to the fail-closed ctx.remote !== false gate (line-number-free)
- TODOS.md: next-take-row allocator follow-up names the real consolidate
  path (src/core/cycle/phases/consolidate.ts)
- src/core/think/index.ts: doc-comment matches the emitted signal form

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-08 20:51:56 -07:00
co-authored by Claude Fable 5
parent 1dbd8665fc
commit a88cf5e0e3
5 changed files with 19 additions and 14 deletions
+1 -1
View File
@@ -4895,7 +4895,7 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
**What:** the `MAX(row_num)+1` next-row allocation for takes is hand-rolled in
four places with inconsistent locking: `src/core/takes-append.ts` (DB-only
fallback, under withPageLock), both engines' `supersedeTake` (in-SQL), and
`src/core/consolidate.ts`. Extract one shared helper with a consistent locking
`src/core/cycle/phases/consolidate.ts`. Extract one shared helper with a consistent locking
story so a fifth writer can't invent a fifth allocation.
**Why:** every divergent copy is a future duplicate-row or clobber bug of the
+2 -2
View File
@@ -106,7 +106,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/trajectory-format.ts` — shared `formatTrajectoryBlock(points, entitySlug, opts)` consumed by both `gbrain think` (production) and the LongMemEval harness (benchmark). Groups by `(metric ?? event_type)`, per-metric cap 20, total cap 100, knowledge_update intent annotates value-change rows with `(superseded prior)`. Emits a `<trajectory entity="...">` XML envelope — `INJECTION_PATTERNS` in `src/core/think/sanitize.ts` escapes `</trajectory>`, `<trajectory ...>` open tags, and attribute injection so adversarial fact text can't break out. Pinned by `test/trajectory-format.test.ts`.
- `src/core/think/intent.ts` + `src/core/think/entity-extract.ts` — pure `classifyIntent(question)` returns `'temporal' | 'knowledge_update' | 'other'` (regex-first, no LLM, `'other'` fast path short-circuits with zero SQL). `extractCandidateEntities(question, retrievedSlugs)` pulls high-precision candidates from retrieved entity-prefix slugs (`people/`, `companies/`, `organizations/`) and medium-precision noun phrases. Stop-word boundaries + leading-verb stripper handle "When did I last meet Marco" → `marco`. Both consumed by `runThink` and the LongMemEval harness so the two paths cannot drift. Pinned by `test/think-intent.test.ts` and `test/think-entity-extract.test.ts`.
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts``gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned; UTF-8-safe truncation; confidence-floor double-enforcement; resolution_kind output drives paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — avoids bias from silent skip), trend writes to `eval_contradictions_runs`, source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker for stable cache hit-rate). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (batched), `writeContradictionsRun` + `loadContradictionsTrend`, `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache`. Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). Doctor check surfaces high-severity findings with paste-ready resolution commands; synthesize phase pre-fetches the latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. Architecture doc: `docs/contradictions.md`.
- `src/core/think/index.ts``runThink` builds its internal `LLMClient` via a small adapter wrapping `gateway.chat()` from `src/core/ai/gateway.ts` (not `new Anthropic()` directly) so stdio MCP launches (Claude Desktop, Cursor) that don't inherit shell env still find a key set via `gbrain config set anthropic_api_key` (the gateway reads `~/.gbrain/config.json` AND env). Test seam: `opts.client?: ThinkLLMClient` injection works (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`); `opts.stubResponse` short-circuits before any LLM call. When neither key nor client is available, the "no LLM available" stub fires with `NO_ANTHROPIC_API_KEY`. Trajectory injection (default ON): `runThink` orchestrates `classifyIntent(question)``extractCandidateEntities(question, retrievedSlugs)``findTrajectory` (5s `Promise.race` timeout per candidate, concurrency cap 3) → `formatTrajectoryBlock`. `buildThinkUserMessage` (in `src/core/think/prompt.ts`) has a `trajectory?: ThinkTrajectoryBlockOpts` slot honoring BOTH prompt orderings (calibration mode: retrieval → calibration → trajectory → question; default mode: question → retrieval → trajectory → instruction). The MCP `think` op handler maps `sourceScopeOpts(ctx)` onto `RunThinkOpts` via `thinkSourceScopeOpts(ctx)` (operations.ts), and `runThink` threads the scope into `runGather` (`src/core/think/gather.ts`) — so every gather stream (hybrid retrieval, takes keyword + vector via the engines' scoped `searchTakes`/`searchTakesVector`, graph walk via `traversePaths`) AND trajectory resolution stay within the caller's source grant (federated `sourceIds[]` wins over scalar `sourceId`); pinned by `test/e2e/think-source-isolation-pglite.test.ts`. Config key `think.trajectory_enabled` (default `true`). Any error in the trajectory path degrades to "no block injected" + `TRAJECTORY_INJECTION_FAILED` warning — the think call never crashes from trajectory. Production path skips `fallback_slugify` resolutions (avoid querying invented slugs); the LongMemEval harness accepts them. Pinned by `test/think-trajectory-injection.test.ts`. Debug: `GBRAIN_THINK_DEBUG=1 gbrain think "..."` prints the spliced prompt to stderr. `persistThinkTake(engine, result, {anchor, sourceId?, sourceIds?})` (#2556) persists `--take` as the next append-only take row on the anchor page: claim = the synthesis answer flattened to one line and capped at `THINK_TAKE_CLAIM_MAX_CHARS` (2000, `TAKE_CLAIM_TRUNCATED` warning when it fires), holder `'brain'`, kind `'take'`, source `'gbrain think'`, written via the shared dual-plane `appendTake` helper (`src/core/takes-append.ts`) — the SAME path `gbrain takes add` uses, so fence-allocated and DB row numbers cannot collide. Never throws for expected shapes (mirroring `persistSynthesis`): signals `TAKE_REQUIRES_ANCHOR`, `TAKE_EMPTY_NOT_PERSISTED`, `TAKE_ANCHOR_NOT_FOUND:<slug>`, `TAKE_FILE_PLANE_UNAVAILABLE` (DB-only fallback, row still written). The CLI path resolves the caller's ambient source via `resolveSourceWithTier` (`seed_default` and `__all__` stay unscoped) and exits 1 when `--take` writes nothing; the op handler gates BOTH `--save` and `--take` on `ctx.remote !== false` fail-closed (remote callers get `remote_persisted_blocked: true`) and maps `thinkScope.allowedSources``sourceIds` explicitly (federated array > scalar) for the anchor lookup — spreading `thinkScope` would silently drop the federated array. Both surfaces return `take_row` + `take_inserted`. Pinned by `test/think-take-cli.serial.test.ts`, `test/think-take-op-federated.test.ts`, `test/think-take-concurrency.serial.test.ts`, `test/think-pipeline.serial.test.ts`.
- `src/core/think/index.ts``runThink` builds its internal `LLMClient` via a small adapter wrapping `gateway.chat()` from `src/core/ai/gateway.ts` (not `new Anthropic()` directly) so stdio MCP launches (Claude Desktop, Cursor) that don't inherit shell env still find a key set via `gbrain config set anthropic_api_key` (the gateway reads `~/.gbrain/config.json` AND env). Test seam: `opts.client?: ThinkLLMClient` injection works (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`); `opts.stubResponse` short-circuits before any LLM call. When neither key nor client is available, the "no LLM available" stub fires with `NO_ANTHROPIC_API_KEY`. Trajectory injection (default ON): `runThink` orchestrates `classifyIntent(question)``extractCandidateEntities(question, retrievedSlugs)``findTrajectory` (5s `Promise.race` timeout per candidate, concurrency cap 3) → `formatTrajectoryBlock`. `buildThinkUserMessage` (in `src/core/think/prompt.ts`) has a `trajectory?: ThinkTrajectoryBlockOpts` slot honoring BOTH prompt orderings (calibration mode: retrieval → calibration → trajectory → question; default mode: question → retrieval → trajectory → instruction). The MCP `think` op handler maps `sourceScopeOpts(ctx)` onto `RunThinkOpts` via `thinkSourceScopeOpts(ctx)` (operations.ts), and `runThink` threads the scope into `runGather` (`src/core/think/gather.ts`) — so every gather stream (hybrid retrieval, takes keyword + vector via the engines' scoped `searchTakes`/`searchTakesVector`, graph walk via `traversePaths`) AND trajectory resolution stay within the caller's source grant (federated `sourceIds[]` wins over scalar `sourceId`); pinned by `test/e2e/think-source-isolation-pglite.test.ts`. Config key `think.trajectory_enabled` (default `true`). Any error in the trajectory path degrades to "no block injected" + `TRAJECTORY_INJECTION_FAILED` warning — the think call never crashes from trajectory. Production path skips `fallback_slugify` resolutions (avoid querying invented slugs); the LongMemEval harness accepts them. Pinned by `test/think-trajectory-injection.test.ts`. Debug: `GBRAIN_THINK_DEBUG=1 gbrain think "..."` prints the spliced prompt to stderr. `persistThinkTake(engine, result, {anchor, sourceId?, sourceIds?})` (#2556) persists `--take` as the next append-only take row on the anchor page: claim = the synthesis answer flattened to one line and capped at `THINK_TAKE_CLAIM_MAX_CHARS` (2000, `TAKE_CLAIM_TRUNCATED` warning when it fires), holder `'brain'`, kind `'take'`, source `'gbrain think'`, written via the shared dual-plane `appendTake` helper (`src/core/takes-append.ts`) — the SAME path `gbrain takes add` uses, so fence-allocated and DB row numbers cannot collide. Never throws for expected shapes (mirroring `persistSynthesis`): signals `TAKE_REQUIRES_ANCHOR`, `TAKE_EMPTY_NOT_PERSISTED`, `TAKE_ANCHOR_NOT_FOUND: <slug>` (note the space after the colon), `TAKE_FILE_PLANE_UNAVAILABLE` (DB-only fallback, row still written). The CLI path resolves the caller's ambient source via `resolveSourceWithTier` (`seed_default` and `__all__` stay unscoped) and exits 1 when `--take` writes nothing; the op handler gates BOTH `--save` and `--take` on `ctx.remote !== false` fail-closed (remote callers get `remote_persisted_blocked: true`) and maps `thinkScope.allowedSources``sourceIds` explicitly (federated array > scalar) for the anchor lookup — spreading `thinkScope` would silently drop the federated array. Both surfaces return `take_row` + `take_inserted`. Pinned by `test/think-take-cli.serial.test.ts`, `test/think-take-op-federated.test.ts`, `test/think-take-concurrency.serial.test.ts`, `test/think-pipeline.serial.test.ts`.
- `src/core/operations.ts` extension (orphans fix) — `findOrphanPages` (both engines) filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side, so soft-deleted pages don't appear as orphans AND links from soft-deleted source pages don't suppress live pages from orphan results. Pinned by `test/orphans.test.ts`'s soft-delete cases.
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts``gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. One in-memory PGLite per run via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` (schema-migration-safe); infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) preserved. `cli.ts` pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults OFF (deterministic, no per-query Haiku); pass `--expansion` to opt in. Default model via `resolveModel()` 6-tier chain with `models.eval.longmemeval` config key. Sanitization parity: `harness.ts` reuses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` so adding a pattern covers takes AND benchmarks. Retrieved chat content wrapped in `<chat_session id="..." date="...">`; the answer-gen system prompt declares content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client without an API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (`test/eval-longmemeval.test.ts` perf gate). Hand the JSONL to LongMemEval's `evaluate_qa.py` to score (not bundled — needs OpenAI gpt-4o). Per-question JSONL row carries `question: string` (additive; `evaluate_qa.py` ignores unknown fields) so `gbrain eval cross-modal --batch` has the `task` text without joining; also `question_type: string` and `recall_hit?: boolean` so a `--resume-from` run rebuilds cumulative `recallByType` from the file alone. `--by-type` flag emits a `{schema_version:1, kind:"by_type_summary", recall_by_type:{...}, aggregate:{...}}` line as the FINAL line; resume-replace strips any prior summary at the tail so 5 resumed runs produce 1 summary. Empty-bucket guard: `aggregate.rate` is `null` (not NaN) when no questions had ground truth. Optional `--by-type-floor F` (0..1) exits non-zero with a stderr line per breached `question_type` (default informational). Pure `buildByTypeSummary(buckets)` + `emitByTypeSummary(path, summary)` + `seedRecallByTypeFromFile(path, bucket)` exported for unit tests. Inline Haiku extractor + trajectory routing (methodology change): `src/eval/longmemeval/extract.ts` runs `extractAndInsertClaims()` over each haystack session before retrieval, populating the benchmark brain's `facts` table inline at import. Single Haiku call per session with content-hash cache (cuts a 3-iteration run from $1.50 to $0.50 when sessions repeat). Per-question alias map (fresh per question, never leaks) collapses `"Marco"` + `"Marco Smith"` + `"marco"` to one canonical slug via first-mention-wins. Fail-open on every error path (malformed JSON, Haiku throw, insert collision, empty array → `inserted: 0`). `getCacheStats()` writes empirical hit rate to stderr. `src/eval/longmemeval/intent.ts` prefers the dataset's `question_type` label before falling back to the SHARED regex set from `src/core/think/intent.ts` — single source of truth means think and longmemeval cannot drift. `runOneQuestion` routes temporal/knowledge_update intents through shared `extractCandidateEntities``findTrajectory` → splice into the answer-gen prompt before the retrieved-sessions block. `--no-trajectory` bypasses BOTH extractor and intent routing (baseline default-on vs no-trajectory across 3 seeds with paired-bootstrap CI). JSON envelope adds 5 per-question fields when trajectory routing is on: `intent`, `trajectory_points`, `entity_resolved`, `resolution_source`, `methodology_note`. The `methodology_note` writes to stderr at run completion (`extractor=haiku-preprocess-full-haystack-v1`) — honest disclosure that the published number is "gbrain + Haiku-preprocess pipeline" vs "gbrain alone", NOT directly comparable to baseline LongMemEval scores without that note. Pinned by `test/longmemeval-extract.test.ts`, `test/longmemeval-intent.test.ts`, `test/longmemeval-trajectory-routing.test.ts` (end-to-end through `runEvalLongMemEval` with both clients stubbed).
- `docs/eval-bench.md` — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
@@ -199,7 +199,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/extraction-review.ts` — Extraction quarantine lane markers (issue #160), sibling of `src/core/quarantine.ts` / `embed-skip.ts` (frontmatter-key pattern, no schema migration). Auto-extracted stubs from untrusted input carry the PAIR `provenance: 'auto-extracted'` + `status: 'unverified'` (both required — user pages with their own `status`/`provenance` never match). Exports `quarantineMarkers()`, `isUnverifiedExtraction()` (JS predicate) and `unverifiedExtractionFragment(alias)` — the single SQL source of truth consumed by `buildSourceFactorCase` (namespace source-boost guard), both engines' `getUnverifiedExtractionPageIds`, the `extraction_pending` op, and the `unverified_extractions` doctor check, so filter and marker keys can never drift. Consequences: unverified stubs are excluded from the compiled-truth fusion boost + the `people/`/`companies/` source-boost (rank as ordinary content), stamped `unverified: true` in search results (`stampUnverifiedExtractions`, hybrid.ts), listed by `extraction_pending`, promoted (status → `verified`, provenance kept for audit) or rejected (soft-delete) by the owner-only `extraction_review` op. Pinned by `test/extraction-review.test.ts` (PGLite) + `test/e2e/extraction-review-postgres.test.ts` (live Postgres parity).
- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts``gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich:<src>:<slug>')``getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1``runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `<context>…</context>` data-envelope delimiters (injection escape, mirrors the `</trajectory>` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity).
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping.
- `src/commands/embed.ts``gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp per page when every chunk embedded cleanly. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. Embed failures are never silent (#3037): all three page paths embed via `embedPageTexts`, which tries the page's chunks in one batch and, on a PERMANENT request-shaped failure (non-429, non-`AITransientError`, non-auth), retries once per chunk so one bad chunk costs one chunk instead of darkening the whole page (failed chunks stay `embedding IS NULL` for the next `--stale` pass; a partially-failed page is never signature-stamped). Rate-limit/outage/auth failures do NOT fan out (cost bounding — `embedBatchWithBackoff` already owns 429 backoff). Failed chunk counts land on `EmbedResult.failures` + capped `failure_samples`, and `src/cli.ts`'s embed case sets a non-zero exit verdict on `failures > 0` (mirror of the `import` errors>0 guard). Pinned by `test/embed-partial-failure-3037.serial.test.ts` + `test/embed-exit-code-3037.serial.test.ts` (real spawned CLI). `--all`/`--stale` runs end with the takes lane (#2089, `embedStaleTakes`): after the pages/chunks lane completes un-aborted (a wall-clock-budget hit or caller abort skips the lane; the next run picks it up via the NULL-embedding predicate), it counts + lists stale take claims (`countStaleTakes`/`listStaleTakes`, both `sourceId`-scoped through the page join so a `--source X` run only touches X's takes — the same source(s) whose single-flight locks the run holds), embeds them through `embedPageTexts` (the SAME gateway helper the chunks lane feeds: 429 backoff, spend gates, per-item failure isolation, abort signals) in `TAKES_EMBED_BATCH_SIZE` (64) batches, and writes via `engine.updateTakeEmbeddingsBatch` under pacer observation with `pacer.pace()` between batches. Progress phase `embed.takes` (one tick per batch); results land on `EmbedResult.takes_embedded` / `takes_would_embed` (dry-run is count-only — no gateway calls, no writes); a failed stale-count probe skips the lane, never the run; writer-skipped rows (superseded mid-flight) are reported separately from embed failures so the arithmetic stays honest. Pinned by `test/embed-takes-lane.serial.test.ts`.
- `src/commands/embed.ts``gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp per page when every chunk embedded cleanly. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. Embed failures are never silent (#3037): all three page paths embed via `embedPageTexts`, which tries the page's chunks in one batch and, on a PERMANENT request-shaped failure (non-429, non-`AITransientError`, non-auth), retries once per chunk so one bad chunk costs one chunk instead of darkening the whole page (failed chunks stay `embedding IS NULL` for the next `--stale` pass; a partially-failed page is never signature-stamped). Rate-limit/outage/auth failures do NOT fan out (cost bounding — `embedBatchWithBackoff` already owns 429 backoff). Failed chunk counts land on `EmbedResult.failures` + capped `failure_samples`, and `src/cli.ts`'s embed case sets a non-zero exit verdict on `failures > 0` (mirror of the `import` errors>0 guard). Pinned by `test/embed-partial-failure-3037.serial.test.ts` + `test/embed-exit-code-3037.serial.test.ts` (real spawned CLI). `--all`/`--stale` runs end with the takes lane (#2089, `embedStaleTakes`): after the pages/chunks lane completes un-aborted (a wall-clock-budget hit or caller abort skips the lane; the next run picks it up via the NULL-embedding predicate), it counts + lists stale take claims (`countStaleTakes`/`listStaleTakes`, both `sourceId`-scoped through the page join so a `--source X` run only touches X's takes — the same source(s) whose single-flight locks the run holds), embeds them through `embedPageTexts` (the SAME gateway helper the chunks lane feeds: 429 backoff, spend gates, per-item failure isolation, abort signals) in `TAKES_EMBED_BATCH_SIZE` (64) batches, and writes via `engine.updateTakeEmbeddingsBatch` under pacer observation with `pacer.pace()` between batches. Progress phase `embed.takes` (one tick per batch); results land on `EmbedResult.takes_embedded` / `takes_would_embed` (dry-run is count-only — no gateway calls, no writes); a failed stale-count probe skips the lane, never the run; when a batch has embed failures, the stderr line reports writer-skipped rows (superseded mid-flight) separately from the failures; a clean batch with skips stays silent (counts only). Pinned by `test/embed-takes-lane.serial.test.ts`.
- `src/core/retrieval-upgrade-planner.ts``runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL FOUR dim-pinned text-embedding-space columns at `targetDim``content_chunks.embedding`, `query_cache.embedding`, `facts.embedding`, `takes.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache, facts, and takes are created at brain-birth width by `migrate.ts` (v55 / v42 / v37+v126) and no LATER migration ALTERs them, so omitting any of them leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine), and a narrow `takes.embedding` fails every `embed --stale` takes-lane write forever while still paying the gateway each run (takes' HNSW recreation keeps the partial `WHERE active AND embedding IS NOT NULL` predicate). Dropped vectors re-fill on their own: query_cache on the next query, facts on the next write / extract pass, takes via the `embed --stale` takes lane (its staleness predicate is `embedding IS NULL`, which the drop satisfies). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all four widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`.
- `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state``runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions``persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})``SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector).
- `src/commands/migrate-embeddings.ts``gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite).
+4 -2
View File
@@ -65,6 +65,8 @@ Key files:
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
to op params. `think` is a special case: the server's `think` op
intentionally disables `--save`/`--take` for remote callers
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
intentionally disables `--save`/`--take` for remote callers (the think
handler's fail-closed trust-boundary gate in operations.ts — persistence
requires `ctx.remote !== false`, and blocked callers get
`remote_persisted_blocked: true`); thin-client `think` warns
loudly when those flags are set.
+10 -8
View File
@@ -51,14 +51,16 @@ llama-server, and other bring-your-own-model providers).
people running deliberate experiments.
5. **Apply.** When the target width differs from the actual column width,
runs the same atomic schema transition `ze-switch` uses, in one
transaction. It rebuilds **all three dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
`facts.embedding` — at the new width, preserving each column's type
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
three leaves it silently broken: a narrow `query_cache.embedding` makes
every cache write and read fail *by design* (the cache swallows errors so
it can never break search) for a permanent 0% hit rate, and a narrow
`facts.embedding` fails every per-fact embed write. The image/multimodal
transaction. It rebuilds **all four dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`,
`facts.embedding`, and `takes.embedding` — at the new width, preserving
each column's type (`vector` vs `halfvec`) and recreating its HNSW index.
Missing any of the four leaves it silently broken: a narrow
`query_cache.embedding` makes every cache write and read fail *by design*
(the cache swallows errors so it can never break search) for a permanent
0% hit rate, a narrow `facts.embedding` fails every per-fact embed write,
and a narrow `takes.embedding` fails every `embed --stale` takes-lane
write while still paying the embedding gateway each run. The image/multimodal
columns ARE deliberately untouched — they use separate models whose
dimensions are independent of the text embedding model.
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
+2 -1
View File
@@ -701,7 +701,8 @@ export const THINK_TAKE_CLAIM_MAX_CHARS = 2000;
* Signals (never throws for expected shapes, mirroring persistSynthesis):
* - TAKE_REQUIRES_ANCHOR — no anchor given.
* - TAKE_EMPTY_NOT_PERSISTED — synthesis failed or empty answer.
* - TAKE_ANCHOR_NOT_FOUND:<s> — anchor page absent in the caller's scope
* - `TAKE_ANCHOR_NOT_FOUND: <s>` (space after the colon) — anchor page
* absent in the caller's scope
* (scope resolved via the sourceIds > sourceId precedence, matching
* sourceScopeOpts). Remote callers never reach here — the op handler's
* fail-closed gate zeroes `take` for them.