Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 113b4507f1 docs(changelog): account for all 18 audit items; mention the takes-add ordering fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 20:53:06 -07:00
Garry TanandClaude Fable 5 a88cf5e0e3 docs: cross-model doc-review fixes — four dim-pinned columns in the migration guide, exact signal spelling, honest skip-reporting claim
Doc-review pass findings applied:
- docs/guides/embedding-migration.md: schema transition rebuilds four
  dim-pinned text-embedding columns (takes.embedding joined the set)
- docs/architecture/KEY_FILES.md: TAKE_ANCHOR_NOT_FOUND signal carries a
  space after the colon (prefix-matchers need the exact form); takes-lane
  skip reporting only prints when the batch also had embed failures
- docs/architecture/thin-client.md: think op trust-boundary pointer
  updated to the fail-closed ctx.remote !== false gate (line-number-free)
- TODOS.md: next-take-row allocator follow-up names the real consolidate
  path (src/core/cycle/phases/consolidate.ts)
- src/core/think/index.ts: doc-comment matches the emitted signal form

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 20:51:56 -07:00
Garry TanandClaude Fable 5 1dbd8665fc docs(key-files): document think --take dual-plane persistence, takes embedding pipeline, and the v126 dims rebuild
Post-ship documentation pass for v0.42.79.0 (#2556, #2089):
- new entry for src/core/takes-append.ts (shared dual-plane appendTake,
  fence-first row allocation, TAKE_FILE_PLANE_UNAVAILABLE posture)
- think/index.ts entry: persistThinkTake signals, claim cap, ambient-source
  CLI resolution, fail-closed remote gate, take_row/take_inserted outputs
- embed.ts entry: the embed.takes lane (batching, source scoping, dry-run,
  abort/budget skip semantics)
- engine.ts + both engine entries: updateTakeEmbeddingsBatch,
  scoped countStaleTakes/listStaleTakes, invalidate covers takes
- retrieval-upgrade-planner.ts entry: takes joins the dim-transition set
  (three columns -> four)
- migrate.ts entry: v126 takes_embedding_active_dims

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 20:40:12 -07:00
Garry TanandClaude Fable 5 9a68e43e9d v0.42.79.0 fix(think,takes): think --take persists dual-plane; takes embeddings get a writer, dims, and an embed lane
Version bump + CHANGELOG for the Wave 4 think-ops release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:53:35 -07:00
Garry TanandClaude Fable 5 d42b25bee0 fix(review): wave-4 batch — takes join the dim-transition set, dual-plane think-take append, source-scoped embed lane, claim bounds
Two criticals:

1. Dim-transition hole: `takes` was missing from
   TEXT_EMBEDDING_DIM_PINNED_TABLES, so migrating embeddings to a
   different dim left takes.embedding at the old width — the embed takes
   lane then failed every write forever while still paying the gateway
   each run. takes now transitions with query_cache + facts (partial
   HNSW index recreated, #1734 dim-cap gate honored), and
   invalidateStaleSignatureEmbeddings re-stales takes on drifted pages
   in BOTH engines (same page-signature predicate + source scope as the
   chunks sweep) so a same-dim provider swap can't score old-space take
   vectors against new-space queries.

2. Fence-plane clobber: persistThinkTake wrote DB-only rows while
   `takes add` allocates row numbers from the markdown fence — the next
   fence-allocated row at the same number deterministically OVERWROTE
   the think-take via ON CONFLICT (page_id, row_num) DO UPDATE. Both
   writers now share one dual-plane helper (src/core/takes-append.ts):
   withPageLock → scoped page lookup → fence allocates the row number
   (file-first) → DB mirror. Headless/DB-only brains fall back to
   MAX(row_num)+1 with the TAKE_FILE_PLANE_UNAVAILABLE warning.

Also in this batch: countStaleTakes/listStaleTakes accept a sourceId
scope (page-join filter) and the embed takes lane threads it so
`embed --stale --source X` touches only X's takes; think-take claims
are flattened to one line and bounded at THINK_TAKE_CLAIM_MAX_CHARS
(2000) with TAKE_CLAIM_TRUNCATED; updateTakeEmbeddingsBatch routes
through batchRetry with its own BATCH_AUDIT_SITES entry (both engines);
the think op's remote gate uses the fail-closed `ctx.remote !== false`
pattern; the takes-lane failure line reports written vs skipped
(row retired mid-flight) honestly; and `gbrain think --take` resolves
the caller's ambient source (env/dotfile/local_path/brain-default
tiers) so duplicate anchor slugs land on the caller's-context page.

Includes the 5 wave-4 coverage-audit test files, new pins for the dim
transition + takes invalidation + lane scoping + claim bound, and the
"Think-ops follow-ups (Wave 4, 2026-08)" TODO section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:51:51 -07:00
Garry TanandClaude Fable 5 b9782dc743 docs(takes): extraction model doc-comment tells the truth — getChatModel, not facts.extraction_model (#1857 residual)
The ThinkOps audit's last live residual: the opts.model doc claimed the
default is facts.extraction_model, but #2997 deliberately routed to the
configured chat model (a tier fallback would reintroduce the hardcoded-
Haiku failure on OAuth/local-only installs). Recording the decision at
the seam instead of adding a new config surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:09:37 -07:00
Garry TanandClaude Fable 5 79fdd3c33b fix(takes): embeddings get a writer — dims-correct column, batch write path, embed --stale lane (#2089)
takes.embedding never had a writer: no `UPDATE takes SET embedding` existed
anywhere in src, so searchTakesVector — and think's takes_vec recall arm —
was structurally dead on every brain since the column shipped. The column
was also the wrong width on non-1536 brains: migration v37 hardcodes
VECTOR(1536), and since `takes` is migration-created (absent from
schema.sql) the init-time embedding-dim templating never reached it.

Three pieces close the loop:

- Migration v126 (takes_embedding_active_dims) rebuilds the column at the
  brain's configured dim (config.embedding_dimensions, v40 resolution
  pattern: default 1536, clamp 1..4096) and recreates the v37 HNSW partial
  index only when the dim is within the #1734 cap (hnswIndexExpected).
  Provably lossless: with no writer there are no real embeddings to lose —
  the defensive NULLing only clears hand-written strays that are unusable
  at the corrected width anyway. Idempotent: no-ops when atttypmod already
  matches the target dim. Identical DDL on both engines.

- New engine method updateTakeEmbeddingsBatch (BrainEngine + both engines,
  lockstep): RETURNING-based actually-updated count, active-rows-only so a
  claim superseded mid-embed (new stale row) can't stamp the retired row,
  empty input returns 0 without a query.

- runEmbedCore grows a takes lane AFTER the pages/chunks lane (--all and
  --stale runs): batches of 64 claims through the SAME embedPageTexts
  gateway helper the chunks lane uses, so rate-limit backoff, per-item
  failure isolation, spend gates, and abort signals apply automatically —
  no second gateway path to drift. DB writes route through the pace-mode
  pacer (observed + pace between batches); a wall-clock budget hit or
  caller abort in the pages lane skips takes gracefully (embedAll /
  embedAllStale now report cut-short so the lane never starts new work
  after a timeout). Dry-run mirrors the pages lane (count-only). New
  `embed.takes` progress phase (docs/progress-events.md); takes_embedded /
  takes_would_embed counts on EmbedResult; failures feed the existing
  failures>0 non-zero exit verdict.

Tests: writer round-trip + embedded_at stamp + stale-count decrement +
inactive-row guard on PGLite (takes-engine); write→read-back→vector-search
parity on both engines (engine-parity, verified against Postgres 17 +
pgvector 0.8.1); migration test at a non-default dim covering the rebuild,
HNSW recreation, idempotent re-run, and the writer end-to-end at the new
width (migrations-takes-embedding-dims.serial). takes.embedding added to
COLUMN_EXEMPTIONS (migration-created table, no schema-blob forward ref).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:07:37 -07:00
Garry TanandClaude Fable 5 547b2a9e0b fix(think): --take actually persists — append the synthesis as an anchor take row (#2556)
ThinkOpts.take was declared but nothing consumed it: the CLI validated
--take (requires --anchor) and then persisted nothing; the op handler
computed safeTake, forwarded it, and had no persist branch — a silent
no-op on both surfaces. persistThinkTake appends the synthesis answer as
the next append-only take row on the anchor page (claim=answer,
kind=take, holder=brain, source='gbrain think'), with the row-number
computation + insert serialized under withPageLock so it can't race
`takes add` into a duplicate row, the anchor lookup scoped through the
caller's source grant (federated array > scalar, mapped explicitly from
thinkSourceScopeOpts' dialect), and the same honesty contract as --save:
empty synthesis or missing anchor exits non-zero instead of pretending.
Remote callers stay blocked by the existing fail-closed gate. Also flips
the runThink `remote:` forward from `ctx.remote === true` to
`ctx.remote !== false` (the fail-closed invariant; undefined must not
read as trusted).

Design from community PR #2618 (successor #3164 declined for bundled
feature scope; maintainer confirmed this half is legitimate) — credit
the PR author; page-lock serialization added on top. Closes #2556.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:42:55 -07:00
36 changed files with 1926 additions and 94 deletions
+21
View File
@@ -2,6 +2,27 @@
All notable changes to GBrain will be documented in this file.
## [0.42.79.0] - 2026-08-09
**`think --take` actually persists, and takes finally get embeddings — vector search over your takes goes from structurally impossible to working.**
**`gbrain think --take` writes the take it promised.** The flag was accepted, validated (`--take` requires `--anchor`), and then consumed by nothing — a silent no-op on both the CLI and MCP paths. Now the synthesis answer is appended as the next take row on the anchor page (holder `brain`, source `gbrain think`), written to both planes the way `takes add` writes: the page's takes fence in your brain repo allocates the row number and the database mirrors it, so a later `takes add` can't collide with it. Claims are bounded (2,000 chars, flattened to one line) with a loud warning when truncation fires; an empty synthesis or missing anchor exits non-zero instead of pretending. Remote callers remain blocked from persistence, and the trust gate now treats an *absent* caller identity as untrusted on every internal forward. When no brain repo is reachable (headless installs), the take lands database-only with an explicit warning naming the posture. A related `takes add` fix rides along: the page lookup now precedes the fence write, so adding a take to a missing page fails cleanly instead of leaving a half-written fence. Design from a community pull request whose fix half was confirmed legitimate but died carrying unrelated feature scope — credited.
**Takes embeddings exist now.** The takes table has had an embedding column since it shipped — with no writer anywhere in the codebase, and a hardcoded width that only matched the default provider. Vector search over takes (including think's takes-recall arm) was structurally dead on every brain. This release: a migration rebuilds the column at your brain's actual embedding width (provably lossless — there was never a writer, so there is nothing to lose), `gbrain embed --stale` gains a takes lane that backfills every active claim through the same batching, spend gates, pacing, and abort handling the pages lane uses (`--source` scoping honored; dry-run counts via `takes_would_embed`), and takes join the dimension-transition set so future provider migrations rebuild them alongside chunks instead of leaving a permanently failing, permanently re-billed backfill. Provider swaps at the same width now re-stale take embeddings too, so old-space vectors are never scored against new-space queries.
### To take advantage of v0.42.79.0
```bash
gbrain upgrade
gbrain embed --stale # one-time backfill: your takes get embeddings
```
Then `gbrain think` starts drawing on vector-matched takes automatically. Try `gbrain think "question" --anchor <page> --take` to persist the synthesis as a take on that page.
### For contributors
The think-ops audit re-verified 18 planned items against master first: 10 already fixed, 3 obsolete, 2 folded into recorded decisions, leaving `#2556`, `#2089`, and a doc-truth residual. Review army + red team caught two release-blocking issues in the new code before ship — the missing dimension-transition registration (which would have turned every future provider migration into a paid retry loop) and the fence-plane row-number collision (which let one ordinary `takes add` silently overwrite a think-take) — both fixed in-wave with concurrency, federated-scope, CLI-honesty, and migration-rerun-preservation tests added. Follow-ups filed in TODOS.md ("Think-ops follow-ups"). Credit to the #2618 author for the take-persistence design.
## [0.42.75.0] - 2026-08-08
**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.**
+54
View File
@@ -4886,3 +4886,57 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
+ the models-doctor tests.
**Depends on:** nothing.
## Think-ops follow-ups (Wave 4, 2026-08)
### Next-take-row allocation helper (unify 4 hand-rolled copies)
**Priority:** P3
**What:** the `MAX(row_num)+1` next-row allocation for takes is hand-rolled in
four places with inconsistent locking: `src/core/takes-append.ts` (DB-only
fallback, under withPageLock), both engines' `supersedeTake` (in-SQL), and
`src/core/cycle/phases/consolidate.ts`. Extract one shared helper with a consistent locking
story so a fifth writer can't invent a fifth allocation.
**Why:** every divergent copy is a future duplicate-row or clobber bug of the
exact class Wave 4 fixed (think-take rows overwritten by fence-allocated rows).
**How to start:** put the allocator next to `appendTake` in
`src/core/takes-append.ts`; migrate call sites one at a time, pinning each with
the existing takes-engine / consolidate tests.
### Takes-lane terminal-state warning parity + failure-path tests
**Priority:** P3
**What:** the chunks lane warns loudly when a catch-up run finishes with
persistently-failing chunks (embed.ts's post-loop `countStaleChunks` probe);
the takes lane has no equivalent — the same takes can fail every run silently
(deferral note sits at the takes-lane batch loop in `src/commands/embed.ts`).
Add the mirror warning plus failure-path tests: batch-throw (embedPageTexts
rejects) and probe-throw (countStaleTakes rejects) paths.
### CLI take success-path output test
**Priority:** P3
**What:** `test/think-take-cli.serial.test.ts` covers the two exit(1) branches
but not the success path (`Take: appended row #N ...` line + exit 0). The
review provided a chat-transport seam stub (see
`test/think-take-op-federated.test.ts`'s `__setChatTransportForTests` canned
envelope) — drive `runThinkCli` with it and assert the success output.
### v126 HNSW dim-cap branch test (3072d) + real-PG run
**Priority:** P3
**What:** migration v126's `hnswIndexExpected` gate (skip HNSW above the #1734
cap) has no test at a >cap dim (e.g. 3072d: column rebuilt, index deliberately
absent, exact scan still works). Add the PGLite test plus a DATABASE_URL-gated
real-PG run — PGLite cannot surface real pgvector index-build failures.
### Federated think --take write-policy note
**Priority:** P4
**What:** a take synthesized under a federated grant [A,B] persists into the
ANCHOR page's source — documented behavior (the anchor lookup is
grant-confined; the write inherits the anchor's source). Revisit if per-source
take provenance ever lands; a grant-scoped caller writing brain-holder takes
into a mounted source may want an explicit provenance column instead.
+1 -1
View File
@@ -1 +1 @@
0.42.75.0
0.42.79.0
File diff suppressed because one or more lines are too long
+4 -2
View File
@@ -65,6 +65,8 @@ Key files:
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
to op params. `think` is a special case: the server's `think` op
intentionally disables `--save`/`--take` for remote callers
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
intentionally disables `--save`/`--take` for remote callers (the think
handler's fail-closed trust-boundary gate in operations.ts — persistence
requires `ctx.remote !== false`, and blocked callers get
`remote_persisted_blocked: true`); thin-client `think` warns
loudly when those flags are set.
+10 -8
View File
@@ -51,14 +51,16 @@ llama-server, and other bring-your-own-model providers).
people running deliberate experiments.
5. **Apply.** When the target width differs from the actual column width,
runs the same atomic schema transition `ze-switch` uses, in one
transaction. It rebuilds **all three dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
`facts.embedding` — at the new width, preserving each column's type
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
three leaves it silently broken: a narrow `query_cache.embedding` makes
every cache write and read fail *by design* (the cache swallows errors so
it can never break search) for a permanent 0% hit rate, and a narrow
`facts.embedding` fails every per-fact embed write. The image/multimodal
transaction. It rebuilds **all four dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`,
`facts.embedding`, and `takes.embedding` — at the new width, preserving
each column's type (`vector` vs `halfvec`) and recreating its HNSW index.
Missing any of the four leaves it silently broken: a narrow
`query_cache.embedding` makes every cache write and read fail *by design*
(the cache swallows errors so it can never break search) for a permanent
0% hit rate, a narrow `facts.embedding` fails every per-fact embed write,
and a narrow `takes.embedding` fails every `embed --stale` takes-lane
write while still paying the embedding gateway each run. The image/multimodal
columns ARE deliberately untouched — they use separate models whose
dimensions are independent of the text embedding model.
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
+2
View File
@@ -136,6 +136,8 @@ Stable phase names shipped in v0.15.2:
- `doctor.db_checks` (umbrella for all DB-side doctor checks)
- `orphans.scan`
- `embed.pages`
- `embed.takes` (the takes lane of `gbrain embed --all/--stale`; one tick per
64-claim batch, total is the stale-take batch count at lane start)
- `extract.links_fs`, `extract.timeline_fs`, `extract.links_db`, `extract.timeline_db`
- `import.files`
- `sync.deletes`, `sync.renames`, `sync.imports`
+1 -1
View File
@@ -148,7 +148,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.75.0",
"version": "0.42.79.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.5",
+149 -5
View File
@@ -196,6 +196,17 @@ export interface EmbedResult {
* corpus-wide outage doesn't bloat structured output. Additive field.
*/
failure_samples: string[];
/**
* #2089: take claims embedded by the takes lane (`--all`/`--stale` runs
* only; per-slug runs skip the lane). Present only when the lane ran.
* 0-or-absent in dryRun (see `takes_would_embed`).
*/
takes_embedded?: number;
/**
* #2089: take claims that WOULD be embedded if not for dryRun. Mirrors
* `would_embed` naming. Present only when the lane ran in dryRun mode.
*/
takes_would_embed?: number;
/** True if this run was a dry-run. */
dryRun: boolean;
/**
@@ -416,7 +427,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
}
}
try {
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
const laneAborted = await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
batchSize: opts.batchSize,
priority: opts.priority,
catchUp: opts.catchUp,
@@ -425,6 +436,24 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
quiet: opts.quiet,
includeNullSignature: opts.includeNullSignature,
}, opts.signal);
// #2089: takes lane. Runs AFTER the pages/chunks lane so a wall-clock
// budget hit (or caller abort) inside the pages lane skips takes
// gracefully — the next run picks them up via the NULL-embedding
// predicate. Inside this try so the single-flight locks are still held
// — and the lane threads opts.sourceId into its enumerators, so a
// `--source X` run touches ONLY X's takes: the work is confined to the
// same source(s) whose locks this run actually holds (per-source
// mutual exclusion holds for the takes lane too). Pacer telemetry in
// the finally covers takes DB writes as well.
if (!laneAborted && !isAborted(opts.signal)) {
await embedStaleTakes(engine, result, {
dryRun: !!opts.dryRun,
pacer,
signal: opts.signal,
quiet: opts.quiet,
sourceId: opts.sourceId,
});
}
} finally {
// E1: surface pacing telemetry (human + structured) when pacing was on.
const snap = pacer.snapshot();
@@ -750,6 +779,11 @@ function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
};
}
/**
* Returns true when the run was cut short (caller abort or, on the stale
* path, the internal wall-clock budget) so runEmbedCore can skip the takes
* lane instead of starting new work after a timeout (#2089).
*/
async function embedAll(
engine: BrainEngine,
staleOnly: boolean,
@@ -771,7 +805,7 @@ async function embedAll(
includeNullSignature?: boolean;
},
signal?: AbortSignal,
) {
): Promise<boolean> {
// v0.41.31: current embedding provenance signature. Stamped onto pages
// when their chunks are (re)embedded so a later model/dimension swap is
// detectable as stale.
@@ -947,6 +981,8 @@ async function embedAll(
slog(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
}
}
// #2089: --all has no wall-clock budget; only a caller abort cuts it short.
return isAborted(signal);
}
/**
@@ -988,7 +1024,7 @@ async function embedAllStale(
},
signature?: string,
externalSignal?: AbortSignal,
) {
): Promise<boolean> {
// D7: thread sourceId so source-scoped runs only count + visit
// that source's NULL embeddings.
const sourceOpt = sourceId ? { sourceId } : undefined;
@@ -1051,7 +1087,7 @@ async function embedAllStale(
slog('Embedded 0 chunks (0 stale found)');
}
}
return;
return isAborted(externalSignal);
}
if (dryRun) {
@@ -1059,7 +1095,7 @@ async function embedAllStale(
result.total_chunks += staleCount;
if (onProgress) onProgress(1, 1, 0);
if (!staleOpts?.quiet) slog(`[dry-run] Would embed ${staleCount} stale chunks`);
return;
return isAborted(externalSignal);
}
// v0.33.3: cursor-paginated stale loading. Instead of pulling all 48K+
@@ -1344,6 +1380,114 @@ async function embedAllStale(
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${result.failures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
}
}
// #2089: report whether the budget/caller abort cut the run short so the
// takes lane can be skipped gracefully (same next-run pickup semantics).
return effectiveSignal.aborted;
}
/** #2089: claims per gateway batch in the takes lane. */
export const TAKES_EMBED_BATCH_SIZE = 64;
/**
* #2089: the takes lane. Backfills `takes.embedding` for every stale claim
* (active AND embedding IS NULL — supersede creates a new row, so edited
* claims re-stale automatically). Runs after the pages/chunks lane in
* `--all`/`--stale` runs.
*
* Reuses `embedPageTexts` — the SAME helper the chunks lane feeds — so
* rate-limit backoff, per-item failure isolation, spend gates, and abort
* signals apply identically; no second gateway path exists to drift.
* DB writes route through `observed(pacer, ...)` + `pacer.pace()` between
* batches, mirroring the pages lane's DB-contention pacing.
*
* Dry-run mirrors the pages lane: count-only (`takes_would_embed`), no
* gateway calls, no writes.
*/
async function embedStaleTakes(
engine: BrainEngine,
result: EmbedResult,
opts: { dryRun: boolean; pacer: DbPacer; signal?: AbortSignal; quiet?: boolean; sourceId?: string },
): Promise<void> {
// Source scope: thread the run's --source through both enumerators so a
// scoped run never enumerates (or pays to embed) another source's takes.
const scope = opts.sourceId !== undefined ? { sourceId: opts.sourceId } : undefined;
let staleCount: number;
try {
// Defensive Number(): int8 counts can arrive as BigInt/string from a
// driver; a null/undefined (partial engine double in tests) reads as 0.
staleCount = Number(await engine.countStaleTakes(scope)) || 0;
} catch (e: unknown) {
// Pre-v37 brains (or a wedged probe) must not fail the whole embed run
// over the optional takes lane.
serr(` [takes] stale-count probe failed; skipping takes lane: ${e instanceof Error ? e.message : e}`);
return;
}
if (staleCount <= 0) return;
if (opts.dryRun) {
result.takes_would_embed = (result.takes_would_embed ?? 0) + staleCount;
if (!opts.quiet) slog(`[dry-run] Would embed ${staleCount} stale take claim(s)`);
return;
}
const staleTakes = (await engine.listStaleTakes(scope)) ?? [];
if (staleTakes.length === 0) return;
result.takes_embedded ??= 0;
// Same createProgress reporter pattern as 'embed.pages' (runEmbed): one
// tick per batch, stderr-only, non-TTY plain lines unless --progress-json.
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start('embed.takes', Math.ceil(staleTakes.length / TAKES_EMBED_BATCH_SIZE));
try {
for (let i = 0; i < staleTakes.length; i += TAKES_EMBED_BATCH_SIZE) {
if (isAborted(opts.signal)) break; // bail between batches (pages-loop contract)
const batch = staleTakes.slice(i, i + TAKES_EMBED_BATCH_SIZE);
try {
const { embeddings, failed, firstError } = await embedPageTexts(
batch.map((t) => t.claim),
opts.signal ? { abortSignal: opts.signal } : {},
);
const writes: Array<{ take_id: number; embedding: Float32Array }> = [];
for (let j = 0; j < batch.length; j++) {
const emb = embeddings[j];
// int8 ids may arrive as BigInt from the driver — coerce.
if (emb) writes.push({ take_id: Number(batch[j].take_id), embedding: emb });
}
const updated = await observed(opts.pacer, () => engine.updateTakeEmbeddingsBatch(writes));
result.takes_embedded += updated;
if (failed > 0) {
recordFailure(result, failed, `takes:${batch[0]?.page_slug ?? '?'}`, firstError);
// `updated` is the RETURNING-based count from the writer; rows the
// writer skipped (superseded mid-flight, `AND active` guard) are
// called out separately so the arithmetic is honest.
const skipped = writes.length - updated;
serr(
` [takes] ${failed} claim(s) failed to embed in batch; ${updated} written` +
(skipped > 0 ? `, ${skipped} skipped (row retired mid-flight)` : ''),
);
}
// NOTE: unlike the chunks lane's catch-up terminal-state warning
// (see the countStaleChunks probe after the pages loop), persistently
// failing takes are not yet flagged as stuck — deferred, see
// TODOS.md "Think-ops follow-ups (Wave 4, 2026-08)".
} catch (e: unknown) {
if (isAborted(opts.signal)) break; // shutdown, not a failure
recordFailure(result, batch.length, `takes:${batch[0]?.page_slug ?? '?'}`, e);
serr(` [takes] Error embedding claim batch: ${e instanceof Error ? e.message : e}`);
}
progress.tick(1);
// Cooperative DB-contention pace between batches (no-op when unpaced).
try {
await opts.pacer.pace(opts.signal);
} catch (e) {
if (!(e instanceof AbortError)) throw e;
}
}
} finally {
progress.finish();
}
if (!opts.quiet) slog(`Embedded ${result.takes_embedded} take claim(s)`);
}
/**
+34 -40
View File
@@ -19,16 +19,22 @@
* 6. releases the lock (auto via withPageLock)
*/
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { existsSync } from 'node:fs';
import type { BrainEngine, TakeKind } from '../core/engine.ts';
import {
parseTakesFence,
upsertTakeRow,
supersedeRow,
type ParsedTake,
} from '../core/takes-fence.ts';
import { withPageLock } from '../core/page-lock.ts';
import {
appendTake,
TakePageNotFoundError,
pageFilePath,
readBodyOrEmpty,
writeBody,
resolveBrainDirOrNull,
} from '../core/takes-append.ts';
import { resolveSourceId } from '../core/source-resolver.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
@@ -44,26 +50,20 @@ function flagPresent(args: string[], name: string): boolean {
return args.includes(name);
}
// CLI-strict wrapper around the shared resolveBrainDirOrNull (takes-append.ts):
// unresolvable → loud error + exit(1). Core callers (persistThinkTake) use the
// null-returning form and fall back to DB-only allocation instead.
async function resolveBrainDir(engine: BrainEngine | null, explicitDir: string | null): Promise<string> {
if (explicitDir) {
if (!existsSync(explicitDir)) {
console.error(`--dir path does not exist: ${explicitDir}`);
process.exit(1);
}
return explicitDir;
}
if (engine) {
const configured = await engine.getConfig('sync.repo_path');
if (configured && existsSync(configured)) return configured;
if (explicitDir && !existsSync(explicitDir)) {
console.error(`--dir path does not exist: ${explicitDir}`);
process.exit(1);
}
const dir = engine ? await resolveBrainDirOrNull(engine, explicitDir) : (explicitDir ?? null);
if (dir) return dir;
console.error('No brain directory configured. Pass --dir <path> or run `gbrain init` first.');
process.exit(1);
}
function pageFilePath(brainDir: string, slug: string): string {
return join(brainDir, `${slug}.md`);
}
function ensureKind(raw: string | undefined): TakeKind {
if (!raw) {
console.error('Missing --kind. Expected one of: fact, take, bet, hunch.');
@@ -117,15 +117,8 @@ async function resolveTakesSourceId(engine: BrainEngine): Promise<string> {
return resolveSourceId(engine, null);
}
function readBodyOrEmpty(path: string): string {
if (!existsSync(path)) return '';
return readFileSync(path, 'utf-8');
}
function writeBody(path: string, body: string): void {
mkdirSync(dirname(path), { recursive: true });
writeFileSync(path, body, 'utf-8');
}
// readBodyOrEmpty / writeBody / pageFilePath now live in
// src/core/takes-append.ts (shared with the dual-plane appendTake helper).
// --- Subcommands ---
@@ -209,22 +202,23 @@ async function cmdAdd(engine: BrainEngine, args: string[], sourceId?: string): P
const dirArg = flagValue(args, '--dir');
const brainDir = await resolveBrainDir(engine, dirArg ?? null);
await withPageLock(slug, async () => {
const path = pageFilePath(brainDir, slug);
const body = readBodyOrEmpty(path);
const { body: nextBody, rowNum } = upsertTakeRow(body, {
claim, kind, holder, weight, source, sinceDate: since, active: true,
// Dual-plane append via the shared core helper (takes-append.ts): page
// lock → scoped page lookup → fence allocates the row number (file-first)
// → DB mirror. Same path persistThinkTake uses, so the planes can't drift.
try {
const { rowNum } = await appendTake(engine, {
slug, sourceId, claim, kind, holder, weight,
source, sinceDate: since, brainDir,
});
writeBody(path, nextBody);
// Mirror to DB. Page may not be in DB yet if not synced — caller must run sync first.
const pageId = await getPageId(engine, slug, sourceId);
await engine.addTakesBatch([{
page_id: pageId, row_num: rowNum, claim, kind, holder, weight,
since_date: since, source, active: true, superseded_by: null,
}]);
console.log(`Added take #${rowNum} to ${slug}.`);
});
} catch (e) {
if (e instanceof TakePageNotFoundError) {
// Page may not be in DB yet if not synced — caller must run sync first.
console.error(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}. Run \`gbrain sync\` first.`);
process.exit(1);
}
throw e;
}
}
async function cmdUpdate(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
+44 -1
View File
@@ -6,10 +6,11 @@
* degrades to gather-only output with a warning if missing.
*/
import type { BrainEngine } from '../core/engine.ts';
import { runThink, persistSynthesis, stripGapsSection } from '../core/think/index.ts';
import { runThink, persistSynthesis, persistThinkTake, stripGapsSection } from '../core/think/index.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
import { canonicalLookup } from '../core/model-pricing.ts';
import { resolveSourceWithTier, ALL_SOURCES } from '../core/source-resolver.ts';
function flagValue(args: string[], name: string): string | undefined {
const i = args.indexOf(name);
@@ -114,6 +115,8 @@ prints what would have been the input (exit 0).
let result: any;
let savedSlug: string | undefined;
let evidenceInserted = 0;
let takeRow: number | null = null;
let takeInserted = 0;
const cfg = loadConfig();
if (isThinClient(cfg)) {
if (save || take) {
@@ -160,6 +163,41 @@ prints what would have been the input (exit 0).
process.exit(1);
}
}
// #2556: --take was accepted-and-ignored (validated at parse time, then
// nothing consumed it). Same honesty contract as --save: persist or
// exit non-zero — an explicit --take that writes nothing must be loud.
if (take) {
// Wave-4: resolve the caller's ambient source context (GBRAIN_SOURCE /
// .gbrain-source dotfile / local_path / brain default — the same
// resolveSourceId chain every other CLI surface uses) so a duplicate
// anchor slug across sources lands on the CALLER's-context page, not
// a planner-chosen one. The seed_default tier means NOTHING was
// configured anywhere — keep the historical unscoped posture there
// (undefined → unscoped getPage), and __all__ has span-everything
// semantics, which for a single-page lookup is also unscoped.
// Resolution errors (e.g. GBRAIN_SOURCE naming an unregistered
// source) propagate to the catch below — fail-closed, matching the
// `gbrain takes` posture.
const resolvedSource = await resolveSourceWithTier(engine, undefined);
const takeSourceId =
resolvedSource.tier === 'seed_default' || resolvedSource.source_id === ALL_SOURCES
? undefined
: resolvedSource.source_id;
const persistedTake = await persistThinkTake(engine, result, {
anchor,
...(takeSourceId !== undefined ? { sourceId: takeSourceId } : {}),
});
takeRow = persistedTake.rowNum;
takeInserted = persistedTake.inserted;
for (const w of persistedTake.warnings) result.warnings.push(w);
if (!persistedTake.rowNum) {
console.error(
'think: --take requested but no take row was written (empty synthesis ' +
'or anchor page not found) — nothing persisted.',
);
process.exit(1);
}
}
} catch (e) {
// #1698: an unresolvable explicit --model throws here. Clean non-zero exit
// with the actionable message, not a stack trace.
@@ -179,6 +217,8 @@ prints what would have been the input (exit 0).
cost_usd: costUsd ?? null,
saved_slug: savedSlug ?? null,
evidence_inserted: evidenceInserted,
take_row: takeRow,
take_inserted: takeInserted,
}, null, 2));
return;
}
@@ -198,6 +238,9 @@ prints what would have been the input (exit 0).
if (savedSlug) {
console.log(`Saved: ${savedSlug} (${evidenceInserted} evidence rows)`);
}
if (takeRow) {
console.log(`Take: appended row #${takeRow} to ${anchor} (holder=brain)`);
}
if (result.warnings.length > 0) {
console.error(`Warnings: ${result.warnings.join(', ')}`);
}
+31 -4
View File
@@ -1050,6 +1050,11 @@ export interface BrainEngine {
* default grandfather clause would silently keep them mixed into the new
* index. `gbrain migrate embeddings` and `embed --stale
* --include-null-signature` set this.
*
* ALSO re-stales `takes` rows (embedding + embedded_at NULL) on the same
* pages, under the SAME signature predicate + source scope takes carry
* text-embedding-space vectors too, and a same-dim provider swap has no
* schema transition to drop them. The returned count remains chunks-only.
*/
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number>;
/**
@@ -1552,11 +1557,33 @@ export interface BrainEngine {
/** Look up embeddings by take id (mirrors getEmbeddingsByChunkIds). */
getTakeEmbeddings(ids: number[]): Promise<Map<number, Float32Array>>;
/** Pre-flight count for `gbrain embed --stale`. WHERE active AND embedding IS NULL. */
countStaleTakes(): Promise<number>;
/**
* Pre-flight count for `gbrain embed --stale`. WHERE active AND embedding
* IS NULL. `opts.sourceId` scopes via the page join (p.source_id) so a
* `--source X` run only counts X's takes.
*/
countStaleTakes(opts?: { sourceId?: string }): Promise<number>;
/** List stale takes (no embedding column in payload — same pattern as listStaleChunks). */
listStaleTakes(): Promise<StaleTakeRow[]>;
/**
* List stale takes (no embedding column in payload same pattern as
* listStaleChunks). `opts.sourceId` scopes via the page join.
*/
listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]>;
/**
* #2089: batch-write take embeddings the writer half of the takes
* embedding pipeline (`countStaleTakes`/`listStaleTakes` enumerate,
* this persists). Sets `embedding` + `embedded_at = now()` on each
* ACTIVE row only: a claim superseded mid-embed produces a new stale
* row via supersedeTake, and the retired row must not receive the
* vector. Returns the number of rows actually updated (superseded /
* deleted ids are silently skipped). Empty input 0 without a query.
*
* Wrapped in `batchRetry` like the other batch primitives (addTakesBatch
* precedent), so `opts` (auditSite, AbortSignal) is honored; same
* no-double-wrap contract as `BatchOpts`.
*/
updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number>;
/**
* Update a take's mutable fields. May NOT change claim/kind/holder per the
+6 -1
View File
@@ -55,7 +55,12 @@ export interface ExtractTakesFromPagesOpts {
includeCovered?: boolean;
/** Owner identifier for the inserted takes. Default 'system'. */
holder?: string;
/** Model override; defaults to facts.extraction_model. */
/**
* Model override; defaults to the configured chat model via getChatModel()
* (#2997 NOT facts.extraction_model, and deliberately not a tier default:
* OAuth/local-only installs have exactly one working model, and a tier
* fallback would reintroduce the hardcoded-Haiku failure #2997 fixed).
*/
model?: string;
/** Progress hook called per page. */
onProgress?: (done: number, total: number, claims: number) => void;
+81 -1
View File
@@ -1,7 +1,7 @@
import type { BrainEngine } from './engine.ts';
import { slugifyPath } from './sync.ts';
import { getFtsLanguage } from './fts-language.ts';
import { hnswMaxDimsForType } from './vector-index.ts';
import { hnswIndexExpected, hnswMaxDimsForType } from './vector-index.ts';
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
@@ -5618,6 +5618,86 @@ export const MIGRATIONS: Migration[] = [
ON take_proposals (source_id, page_slug, content_hash, prompt_version, md5(claim_text));
`,
},
{
version: 126,
name: 'takes_embedding_active_dims',
// #2089: takes.embedding shipped in v37 hardcoded to VECTOR(1536). The
// takes table is created by migration, NOT by schema.sql, so the
// init-time embedding-dim templating never reached it — brains embedded
// at any other width (Voyage 1024, ZE 1280, ...) got a column whose
// dimension can never match their vectors. This went unnoticed because
// the column also had NO WRITER anywhere in the codebase: nothing ever
// ran `UPDATE takes SET embedding = ...`, so `searchTakesVector` (and
// think's takes_vec arm) was structurally dead on every brain.
//
// Rebuild the column at the brain's configured dim. Provably lossless:
// since no writer ever existed, there are no real embeddings to lose —
// the defensive NULLing below only clears hand-written strays, which are
// unusable at the corrected width anyway. The same-release embed lane
// (`gbrain embed --stale`) backfills every active claim afterwards.
//
// Idempotent: when the column already has the target dim, this no-ops.
// Runs identically on both engines (v37's PGLite variant diverges only
// in the RLS DO-block; the column + index DDL is byte-identical).
idempotent: true,
sql: '',
handler: async (engine: BrainEngine) => {
// Step 1: resolve the target dim from config (exact v40 pattern):
// default 1536, accept only finite integers in 1..4096.
let embeddingDim = 1536;
try {
const dimRows = await engine.executeRaw<{ value: string }>(
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
);
if (dimRows.length > 0) {
const parsed = parseInt(dimRows[0].value, 10);
if (Number.isFinite(parsed) && parsed > 0 && parsed <= 4096) {
embeddingDim = parsed;
}
}
} catch {
// No config row yet — fall back to default (v40 precedent).
}
// Step 2: read the CURRENT dim. For pgvector columns atttypmod IS the
// dimension (no varlena header offset like varchar). -1 / missing /
// non-positive → unknown → proceed with the rebuild.
let currentDim: number | null = null;
try {
const rows = await engine.executeRaw<{ atttypmod: number }>(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
);
const raw = rows.length > 0 ? Number(rows[0].atttypmod) : NaN;
currentDim = Number.isFinite(raw) && raw > 0 ? raw : null;
} catch {
currentDim = null; // probe failed → unknown → rebuild
}
if (currentDim === embeddingDim) return; // already correct — no-op
// Step 3: rebuild. embeddingDim is a validated integer (never
// string-interpolate unvalidated input into DDL). HNSW recreation is
// gated by the #1734 dim cap; index DDL mirrors v37 verbatim,
// including the partial WHERE clause.
const recreateIndexSql = hnswIndexExpected('vector', embeddingDim)
? `CREATE INDEX IF NOT EXISTS idx_takes_embedding_hnsw ON takes
USING hnsw (embedding vector_cosine_ops)
WHERE active AND embedding IS NOT NULL;`
: `-- idx_takes_embedding_hnsw skipped: pgvector HNSW vector indexes support
-- at most ${hnswMaxDimsForType('vector')} dimensions; exact vector scans remain available.`;
await engine.runMigration(126, `
UPDATE takes SET embedding = NULL, embedded_at = NULL WHERE embedding IS NOT NULL;
DROP INDEX IF EXISTS idx_takes_embedding_hnsw;
ALTER TABLE takes DROP COLUMN embedding;
ALTER TABLE takes ADD COLUMN embedding VECTOR(${embeddingDim});
${recreateIndexSql}
`);
process.stderr.write(
` v126: takes.embedding rebuilt ${currentDim === null ? '(unknown)' : `VECTOR(${currentDim})`} → VECTOR(${embeddingDim}) (#2089; column had no writer, rebuild is lossless)\n`,
);
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+27 -3
View File
@@ -2287,7 +2287,7 @@ const think: Operation = {
},
mutating: true,
handler: async (ctx, p) => {
const remote = ctx.remote ?? true;
const remote = ctx.remote !== false; // fail-closed: anything not strictly false is untrusted (CLAUDE.md invariant)
// Codex P1 #7 + privacy: remote callers cannot persist via MCP.
const safeSave = remote ? false : Boolean(p.save);
const safeTake = remote ? false : Boolean(p.take);
@@ -2297,7 +2297,7 @@ const think: Operation = {
// forwards to findTrajectory. CLI callers don't go through this op
// and get default scope + remote=false from runThink's CLI path.
const thinkScope = thinkSourceScopeOpts(ctx);
const { runThink, persistSynthesis } = await import('./think/index.ts');
const { runThink, persistSynthesis, persistThinkTake } = await import('./think/index.ts');
const result = await runThink(ctx.engine, {
question: String(p.question),
anchor: p.anchor ? String(p.anchor) : undefined,
@@ -2314,18 +2314,40 @@ const think: Operation = {
until: p.until ? String(p.until) : undefined,
takesHoldersAllowList: ctx.takesHoldersAllowList,
...thinkScope,
remote: ctx.remote === true,
remote: ctx.remote !== false, // fail-closed: anything not strictly false is untrusted (CLAUDE.md invariant)
});
// Persist if --save was passed locally
let savedSlug: string | undefined;
let evidenceInserted = 0;
let takeRow: number | null = null;
let takeInserted = 0;
if (safeSave) {
const persisted = await persistSynthesis(ctx.engine, result);
savedSlug = persisted.slug;
evidenceInserted = persisted.evidenceInserted;
for (const w of persisted.warnings) result.warnings.push(w);
}
// #2556: --take was declared but nothing consumed it — a silent no-op on
// both CLI and MCP. Local callers now get the take row appended to the
// anchor page; remote callers stay blocked via the safeTake gate above.
if (safeTake) {
// thinkSourceScopeOpts speaks runThink's dialect (allowedSources);
// persistThinkTake's getPage speaks the engine's (sourceIds). Map
// explicitly — spreading thinkScope would silently drop the federated
// array and unscope the anchor lookup.
const persistedTake = await persistThinkTake(ctx.engine, result, {
anchor: p.anchor ? String(p.anchor) : undefined,
...(thinkScope.allowedSources !== undefined
? { sourceIds: thinkScope.allowedSources }
: thinkScope.sourceId !== undefined
? { sourceId: thinkScope.sourceId }
: {}),
});
takeRow = persistedTake.rowNum;
takeInserted = persistedTake.inserted;
for (const w of persistedTake.warnings) result.warnings.push(w);
}
return {
...result,
@@ -2333,6 +2355,8 @@ const think: Operation = {
// falsy) to null so callers never see an empty-string "slug".
saved_slug: savedSlug || null,
evidence_inserted: evidenceInserted,
take_row: takeRow,
take_inserted: takeInserted,
remote_persisted_blocked: remote && (Boolean(p.save) || Boolean(p.take)),
};
},
+60 -5
View File
@@ -2767,6 +2767,23 @@ export class PGLiteEngine implements BrainEngine {
RETURNING cc.page_id`,
params,
);
// Takes carry TEXT-embedding-space vectors keyed off the same pages, so a
// same-dim provider swap (no schema transition to drop the column) must
// re-stale them too — old-space take vectors must never score against
// new-space queries. Scoped by the SAME page-signature predicate as the
// chunks UPDATE above: routine `embed --stale` runs (where nothing
// drifted) touch zero takes, so there is no re-embed churn. The `embed
// --stale` takes lane picks the NULLed rows up via `embedding IS NULL`.
// Return value stays the CHUNK count (existing contract).
await this.db.query(
`UPDATE takes t
SET embedding = NULL, embedded_at = NULL
FROM pages p
WHERE t.page_id = p.id
AND t.embedding IS NOT NULL
AND ${sigClause}${srcClause}`,
params,
);
return (rows as unknown[]).length;
}
@@ -5225,25 +5242,63 @@ export class PGLiteEngine implements BrainEngine {
return out;
}
async countStaleTakes(): Promise<number> {
async countStaleTakes(opts?: { sourceId?: string }): Promise<number> {
// Source scope goes through the page join (takes has no source_id of its
// own) so `embed --stale --source X` only counts X's takes.
const srcClause = opts?.sourceId !== undefined ? ` AND p.source_id = $1` : '';
const params = opts?.sourceId !== undefined ? [opts.sourceId] : [];
const { rows } = await this.db.query(
`SELECT count(*)::int AS count FROM takes WHERE active AND embedding IS NULL`
`SELECT count(*)::int AS count
FROM takes t
JOIN pages p ON p.id = t.page_id
WHERE t.active AND t.embedding IS NULL${srcClause}`,
params,
);
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
}
async listStaleTakes(): Promise<StaleTakeRow[]> {
async listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]> {
const srcClause = opts?.sourceId !== undefined ? ` AND p.source_id = $1` : '';
const params = opts?.sourceId !== undefined ? [opts.sourceId] : [];
const { rows } = await this.db.query(
`SELECT t.id AS take_id, p.slug AS page_slug, t.row_num, t.claim
FROM takes t
JOIN pages p ON p.id = t.page_id
WHERE t.active AND t.embedding IS NULL
WHERE t.active AND t.embedding IS NULL${srcClause}
ORDER BY t.id
LIMIT 100000`
LIMIT 100000`,
params,
);
return rows as unknown as StaleTakeRow[];
}
async updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number> {
if (rows.length === 0) return 0;
// Wave-4: batch primitive → batchRetry, matching the addTakesBatch
// precedent above so a connection blip mid-backfill retries + audits
// instead of failing the whole takes lane batch.
return this.batchRetry(opts?.auditSite ?? 'updateTakeEmbeddingsBatch', opts?.signal, () => this._updateTakeEmbeddingsBatchOnce(rows), rows.length);
}
private async _updateTakeEmbeddingsBatchOnce(rows: Array<{ take_id: number; embedding: Float32Array }>): Promise<number> {
let updated = 0;
for (const r of rows) {
// pgvector text literal (searchTakesVector precedent). `AND active`
// guards the supersede race: a claim edited mid-embed retires this row
// and re-stales as a NEW row — the old row must not get the vector.
const vec = `[${Array.from(r.embedding).join(',')}]`;
const { rows: ret } = await this.db.query(
`UPDATE takes
SET embedding = $1::vector, embedded_at = now()
WHERE id = $2 AND active
RETURNING id`,
[vec, r.take_id],
);
updated += (ret as unknown[]).length;
}
return updated;
}
async updateTake(
pageId: number,
rowNum: number,
+54 -3
View File
@@ -2719,6 +2719,23 @@ export class PostgresEngine implements BrainEngine {
RETURNING cc.page_id`,
params as Parameters<typeof this.sql.unsafe>[1],
);
// Takes carry TEXT-embedding-space vectors keyed off the same pages, so a
// same-dim provider swap (no schema transition to drop the column) must
// re-stale them too — old-space take vectors must never score against
// new-space queries. Scoped by the SAME page-signature predicate as the
// chunks UPDATE above: routine `embed --stale` runs (where nothing
// drifted) touch zero takes, so there is no re-embed churn. The `embed
// --stale` takes lane picks the NULLed rows up via `embedding IS NULL`.
// Return value stays the CHUNK count (existing contract).
await this.sql.unsafe(
`UPDATE takes t
SET embedding = NULL, embedded_at = NULL
FROM pages p
WHERE t.page_id = p.id
AND t.embedding IS NOT NULL
AND ${sigClause}${srcClause}`,
params as Parameters<typeof this.sql.unsafe>[1],
);
return (rows as unknown[]).length;
}
@@ -5155,27 +5172,61 @@ export class PostgresEngine implements BrainEngine {
return out;
}
async countStaleTakes(): Promise<number> {
async countStaleTakes(opts?: { sourceId?: string }): Promise<number> {
// Source scope goes through the page join (takes has no source_id of its
// own) so `embed --stale --source X` only counts X's takes.
const sql = this.sql;
const [row] = await sql`
SELECT count(*)::int AS count FROM takes WHERE active AND embedding IS NULL
SELECT count(*)::int AS count
FROM takes t
JOIN pages p ON p.id = t.page_id
WHERE t.active AND t.embedding IS NULL
${opts?.sourceId !== undefined ? sql`AND p.source_id = ${opts.sourceId}` : sql``}
`;
return Number((row as { count?: number } | undefined)?.count ?? 0);
}
async listStaleTakes(): Promise<StaleTakeRow[]> {
async listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]> {
const sql = this.sql;
const rows = await sql`
SELECT t.id AS take_id, p.slug AS page_slug, t.row_num, t.claim
FROM takes t
JOIN pages p ON p.id = t.page_id
WHERE t.active AND t.embedding IS NULL
${opts?.sourceId !== undefined ? sql`AND p.source_id = ${opts.sourceId}` : sql``}
ORDER BY t.id
LIMIT 100000
`;
return rows as unknown as StaleTakeRow[];
}
async updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number> {
if (rows.length === 0) return 0;
// Wave-4: batch primitive → batchRetry, matching the addTakesBatch
// precedent so a connection blip mid-backfill retries + audits
// instead of failing the whole takes lane batch.
return this.batchRetry(opts?.auditSite ?? 'updateTakeEmbeddingsBatch', opts?.signal, () => this._updateTakeEmbeddingsBatchOnce(rows), rows.length);
}
private async _updateTakeEmbeddingsBatchOnce(rows: Array<{ take_id: number; embedding: Float32Array }>): Promise<number> {
const sql = this.sql;
let updated = 0;
for (const r of rows) {
// pgvector text literal (searchTakesVector precedent). `AND active`
// guards the supersede race: a claim edited mid-embed retires this row
// and re-stales as a NEW row — the old row must not get the vector.
const vec = `[${Array.from(r.embedding).join(',')}]`;
const res = await sql`
UPDATE takes
SET embedding = ${vec}::vector, embedded_at = now()
WHERE id = ${r.take_id} AND active
RETURNING id
`;
updated += (res as unknown[]).length;
}
return updated;
}
async updateTake(
pageId: number,
rowNum: number,
+23 -6
View File
@@ -601,17 +601,22 @@ export async function runSchemaTransition(engine: BrainEngine, targetDim: number
);
}
// #3390: the OTHER two dim-pinned columns that carry TEXT-embedding-space
// vectors. Both are created at brain-birth width (migrate.ts v55 for
// query_cache, v42 for facts) and NO migration ever ALTERs them, so before
// this fix a dimension change left them at the old width:
// #3390: the OTHER dim-pinned columns that carry TEXT-embedding-space
// vectors. All are created at brain-birth width (migrate.ts v55 for
// query_cache, v42 for facts, v37/v126 for takes) and no LATER migration
// ever ALTERs them, so before this fix a dimension change left them at
// the old width:
// - query_cache.embedding stayed narrow → every store() AND lookup()
// silently swallowed the width error (by design, so the cache can
// never break search), i.e. a PERMANENT 0% hit rate.
// - facts.embedding stayed narrow → every per-fact embed write failed
// ($N::vector into the old width), and the doctor check that would
// warn is skipped on PGLite (the DEFAULT engine).
// Both are text-embedding-space columns, so they MUST move with
// - takes.embedding stayed narrow → the #2089 embed takes lane failed
// every write forever while still paying the gateway each run
// (migration v126 only fixes the dim at migration-run time; a LATER
// `gbrain migrate embeddings --to <dim>` has to move it here).
// All are text-embedding-space columns, so they MUST move with
// content_chunks.embedding. The image/multimodal columns above are the
// deliberate exception (separate models, independent dims).
for (const t of TEXT_EMBEDDING_DIM_PINNED_TABLES) {
@@ -647,6 +652,17 @@ const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
ON facts USING hnsw (embedding ${opclass})
WHERE embedding IS NOT NULL AND expired_at IS NULL`,
},
{
// Partial predicate mirrors migrate.ts v37 (and the v126 rebuild)
// verbatim. hnswIndexExpected in transitionDimPinnedColumn carries the
// #1734 dim-cap gate, matching v126's recreateIndexSql behavior.
table: 'takes',
index: 'idx_takes_embedding_hnsw',
indexSql: (opclass) =>
`CREATE INDEX IF NOT EXISTS idx_takes_embedding_hnsw
ON takes USING hnsw (embedding ${opclass})
WHERE active AND embedding IS NOT NULL`,
},
];
/**
@@ -658,7 +674,8 @@ const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
* Dropping the column discards the stored vectors, which is correct: they are
* in the OLD embedding space and unusable after the swap. query_cache is a
* cache (refills on the next query); facts re-embed on their next write /
* `gbrain extract` pass.
* `gbrain extract` pass; takes re-embed via the `gbrain embed --stale` takes
* lane (staleness predicate is `embedding IS NULL`, which the drop satisfies).
*/
async function transitionDimPinnedColumn(
tx: { executeRaw: <T = unknown>(sql: string, params?: unknown[]) => Promise<T[]> },
+1
View File
@@ -77,6 +77,7 @@ export const BATCH_AUDIT_SITES = [
'addLinksBatch',
'addTimelineEntriesBatch',
'addTakesBatch',
'updateTakeEmbeddingsBatch',
'upsertChunks',
// extract.ts per-site labels.
'extract.links_inc',
+175
View File
@@ -0,0 +1,175 @@
/**
* Shared dual-plane take append (Wave-4 review fix).
*
* Markdown is the source of truth for takes: `gbrain takes add` allocates
* row numbers from the FILE fence (upsertTakeRow max fence rowNum + 1),
* then mirrors to the DB via addTakesBatch's ON CONFLICT (page_id, row_num)
* DO UPDATE. Any writer that appends DB-only rows (the original
* persistThinkTake did) allocates from DB MAX(row_num) instead and the
* next fence-allocated row at the same number deterministically OVERWRITES
* it. This module is the single append path both surfaces call, so the two
* planes cannot drift:
*
* withPageLock(slug)
* resolve the page (scoped: sourceIds > sourceId > unscoped)
* resolve the brain dir (explicit override > sync.repo_path config)
* file plane: upsertTakeRow allocates the row number (file-first)
* writeBody
* DB plane: addTakesBatch mirror at the fence-allocated row number
*
* Fallback posture (headless / DB-only brains): when no brain dir resolves,
* the row is allocated DB-only (MAX(row_num)+1) and the
* 'TAKE_FILE_PLANE_UNAVAILABLE' warning is returned documented posture:
* DB-only rows can be renumbered/overwritten by a later fence reconcile.
*/
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import type { BrainEngine, TakeKind } from './engine.ts';
import { upsertTakeRow } from './takes-fence.ts';
import { withPageLock } from './page-lock.ts';
/** Warning code returned when the append fell back to DB-only allocation. */
export const TAKE_FILE_PLANE_UNAVAILABLE = 'TAKE_FILE_PLANE_UNAVAILABLE';
/** Thrown when the (scoped) page lookup finds nothing. Callers map this to
* their own surface signal (CLI error+exit, TAKE_ANCHOR_NOT_FOUND, ...). */
export class TakePageNotFoundError extends Error {
readonly slug: string;
readonly sourceId?: string;
constructor(slug: string, sourceId?: string) {
super(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}`);
this.name = 'TakePageNotFoundError';
this.slug = slug;
this.sourceId = sourceId;
}
}
export function pageFilePath(brainDir: string, slug: string): string {
return join(brainDir, `${slug}.md`);
}
export function readBodyOrEmpty(path: string): string {
if (!existsSync(path)) return '';
return readFileSync(path, 'utf-8');
}
export function writeBody(path: string, body: string): void {
mkdirSync(dirname(path), { recursive: true });
writeFileSync(path, body, 'utf-8');
}
/**
* Resolve the on-disk brain dir for the file plane, or null when none is
* available (headless install, unset/dangling sync.repo_path). An explicit
* dir that does not exist ALSO returns null the CLI wraps this with its
* own loud error+exit; core callers fall back to DB-only allocation.
*/
export async function resolveBrainDirOrNull(
engine: BrainEngine,
explicitDir?: string | null,
): Promise<string | null> {
if (explicitDir) return existsSync(explicitDir) ? explicitDir : null;
try {
const configured = await engine.getConfig('sync.repo_path');
if (configured && existsSync(configured)) return configured;
} catch {
// Config read failure → treat as no file plane (fallback posture).
}
return null;
}
export interface AppendTakeInput {
slug: string;
/** Scalar source scope (sourceIds wins when both set — sourceScopeOpts precedence). */
sourceId?: string;
/** Federated source scope for the page lookup. */
sourceIds?: string[];
claim: string;
kind: TakeKind;
holder: string;
weight?: number;
source?: string;
sinceDate?: string;
/**
* Pre-resolved brain dir (CLI --dir, or a caller that already validated
* sync.repo_path). When omitted, resolves via resolveBrainDirOrNull; when
* that yields nothing, the append falls back to DB-only allocation with
* the TAKE_FILE_PLANE_UNAVAILABLE warning.
*/
brainDir?: string;
}
export interface AppendTakeResult {
rowNum: number;
inserted: number;
warnings: string[];
}
/**
* Append one take row to a page on BOTH planes (file fence + DB), serialized
* under withPageLock(slug) so concurrent appends can't race the allocation.
*
* The page lookup runs FIRST (inside the lock, before any file write) so a
* missing page throws {@link TakePageNotFoundError} without leaving a
* half-written fence behind.
*/
export async function appendTake(
engine: BrainEngine,
input: AppendTakeInput,
): Promise<AppendTakeResult> {
const warnings: string[] = [];
return withPageLock(input.slug, async () => {
// Scoped page lookup: federated array > scalar > unscoped.
const pageOpts: { sourceId?: string; sourceIds?: string[] } = {};
if (input.sourceIds !== undefined) pageOpts.sourceIds = input.sourceIds;
else if (input.sourceId !== undefined) pageOpts.sourceId = input.sourceId;
const page = await engine.getPage(input.slug, pageOpts);
if (!page) throw new TakePageNotFoundError(input.slug, input.sourceId);
const brainDir = input.brainDir ?? await resolveBrainDirOrNull(engine);
let rowNum: number;
if (brainDir) {
// File plane first: the fence allocates the row number.
const path = pageFilePath(brainDir, input.slug);
const body = readBodyOrEmpty(path);
const up = upsertTakeRow(body, {
claim: input.claim,
kind: input.kind,
holder: input.holder,
weight: input.weight ?? 0.5,
source: input.source,
sinceDate: input.sinceDate,
active: true,
});
writeBody(path, up.body);
rowNum = up.rowNum;
} else {
// DB-only fallback (no file plane). MAX(row_num)+1 under the same page
// lock. A later fence reconcile may renumber/overwrite these rows —
// documented posture for headless brains.
warnings.push(TAKE_FILE_PLANE_UNAVAILABLE);
const rows = await engine.executeRaw<{ next: number | string }>(
`SELECT (COALESCE(MAX(row_num), 0) + 1)::int AS next FROM takes WHERE page_id = $1`,
[page.id],
);
rowNum = Number(rows[0]?.next ?? 1);
}
const inserted = await engine.addTakesBatch([{
page_id: page.id,
row_num: rowNum,
claim: input.claim,
kind: input.kind,
holder: input.holder,
weight: input.weight ?? 0.5,
since_date: input.sinceDate,
source: input.source,
active: true,
superseded_by: null,
}]);
return { rowNum, inserted, warnings };
});
}
+100
View File
@@ -657,6 +657,106 @@ export async function persistSynthesis(
return { slug, evidenceInserted: persisted.inserted, warnings: persisted.warnings };
}
export interface PersistThinkTakeOpts {
anchor?: string;
sourceId?: string;
sourceIds?: string[];
}
export interface PersistThinkTakeResult {
rowNum: number | null;
inserted: number;
warnings: string[];
}
/**
* Cap on the persisted think-take claim. Rationale: the prompt renderer
* (sanitize.ts:sanitizeTakeForPrompt) truncates claims at 500 chars anyway,
* and fence cells are single-line a multi-page synthesis answer stored
* verbatim as one claim bloats the fence + DB for content nothing ever
* reads past. 2000 keeps ample headroom over the 500-char read while
* bounding the cell.
*/
export const THINK_TAKE_CLAIM_MAX_CHARS = 2000;
/**
* Persist `gbrain think --take` as the next append-only take row on the
* anchor page (#2556 the flag was declared but nothing consumed it, so
* `think --take` silently wrote nothing on both the CLI and MCP paths).
*
* The synthesis answer is the claim; holder='brain' because this is
* gbrain's own analysis; kind='take' (KIND_VALUES member). Dual-plane
* write via the shared appendTake helper (takes-append.ts) the SAME path
* `gbrain takes add` uses: withPageLock(anchor slug) fence allocates the
* row number from the FILE (markdown is the source of truth) writeBody
* addTakesBatch mirror. When no brain dir resolves (headless / DB-only
* brains), appendTake falls back to DB-only MAX(row_num)+1 allocation and
* surfaces TAKE_FILE_PLANE_UNAVAILABLE (documented posture: DB-only rows
* can be renumbered/overwritten by a later fence reconcile).
*
* The claim is flattened to a single line (fence cells are single-line;
* newlines ' ') and truncated at THINK_TAKE_CLAIM_MAX_CHARS with the
* TAKE_CLAIM_TRUNCATED warning.
*
* Signals (never throws for expected shapes, mirroring persistSynthesis):
* - TAKE_REQUIRES_ANCHOR no anchor given.
* - TAKE_EMPTY_NOT_PERSISTED synthesis failed or empty answer.
* - `TAKE_ANCHOR_NOT_FOUND: <s>` (space after the colon) anchor page
* absent in the caller's scope
* (scope resolved via the sourceIds > sourceId precedence, matching
* sourceScopeOpts). Remote callers never reach here the op handler's
* fail-closed gate zeroes `take` for them.
* - TAKE_CLAIM_TRUNCATED claim exceeded the cap (row still written).
* - TAKE_FILE_PLANE_UNAVAILABLE DB-only fallback fired (row still written).
*
* Design provenance: community PR #2618 (its successor #3164 was declined
* for bundled feature scope; the maintainer confirmed this half is a
* legitimate fix). Re-implemented with the page-lock serialization added;
* Wave-4 review moved the write onto the dual-plane appendTake path.
*/
export async function persistThinkTake(
engine: BrainEngine,
result: ThinkResult,
opts: PersistThinkTakeOpts,
): Promise<PersistThinkTakeResult> {
const anchor = opts.anchor?.trim();
if (!anchor) {
return { rowNum: null, inserted: 0, warnings: ['TAKE_REQUIRES_ANCHOR'] };
}
if (result.synthesisOk === false || result.answer.trim().length === 0) {
return { rowNum: null, inserted: 0, warnings: ['TAKE_EMPTY_NOT_PERSISTED'] };
}
const warnings: string[] = [];
// Fence cells are single-line: flatten first, then bound the length.
let claim = result.answer.trim().replace(/\s*\r?\n\s*/g, ' ');
if (claim.length > THINK_TAKE_CLAIM_MAX_CHARS) {
claim = claim.slice(0, THINK_TAKE_CLAIM_MAX_CHARS);
warnings.push('TAKE_CLAIM_TRUNCATED');
}
const { appendTake, TakePageNotFoundError } = await import('../takes-append.ts');
try {
const appended = await appendTake(engine, {
slug: anchor,
...(opts.sourceIds !== undefined ? { sourceIds: opts.sourceIds } : {}),
...(opts.sourceId !== undefined ? { sourceId: opts.sourceId } : {}),
claim,
kind: 'take',
holder: 'brain',
weight: 0.5,
source: 'gbrain think',
});
warnings.push(...appended.warnings);
return { rowNum: appended.rowNum, inserted: appended.inserted, warnings };
} catch (e) {
if (e instanceof TakePageNotFoundError) {
return { rowNum: null, inserted: 0, warnings: [`TAKE_ANCHOR_NOT_FOUND: ${anchor}`] };
}
throw e;
}
}
// ─────────────────────────────────────────────────────────────────
// Gateway adapter for #952 (think over MCP returns "no LLM available").
// ─────────────────────────────────────────────────────────────────
+1 -1
View File
@@ -339,7 +339,7 @@ describe('BATCH_AUDIT_SITES typed enum + isBatchAuditSite guard (D10c codex)', (
// Pin the set so a future "cleanup" PR can't silently drop a site and
// break audit-attribution for the corresponding caller.
const expected = new Set([
'addLinksBatch', 'addTimelineEntriesBatch', 'addTakesBatch', 'upsertChunks',
'addLinksBatch', 'addTimelineEntriesBatch', 'addTakesBatch', 'updateTakeEmbeddingsBatch', 'upsertChunks',
'extract.links_inc', 'extract.timeline_inc',
'extract.links_fs', 'extract.timeline_fs',
'extract.links_db', 'extract.timeline_db',
+41
View File
@@ -732,6 +732,47 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
expect(slugs.indexOf('ep/ec-bob')).toBeLessThan(slugs.indexOf('companies/ec-widget'));
expect(pg.find((r) => r.slug === 'ep/ec-alice')!.inbound_count).toBe(2);
});
// #2089: updateTakeEmbeddingsBatch — the takes-embedding writer. Both
// engines must (a) return 0 on empty input, (b) report the same
// actually-updated count, (c) round-trip through getTakeEmbeddings, and
// (d) surface the row via searchTakesVector afterwards. A drift here
// means `gbrain embed --stale` silently under/over-counts on one engine.
test('updateTakeEmbeddingsBatch parity: write → read-back → vector-search on both engines', async () => {
const slug = 'concepts/fat-code-thin-harness';
const CLAIM_A = 'takes-parity claim alpha';
const CLAIM_B = 'takes-parity claim beta';
for (const eng of [pgEngine, pgliteEngine]) {
// Empty input → 0 without touching the DB.
expect(await eng.updateTakeEmbeddingsBatch([])).toBe(0);
const page = await eng.getPage(slug);
expect(page).not.toBeNull();
await eng.addTakesBatch([
{ page_id: page!.id, row_num: 90, claim: CLAIM_A, kind: 'take', holder: 'world', weight: 0.5 },
{ page_id: page!.id, row_num: 91, claim: CLAIM_B, kind: 'take', holder: 'world', weight: 0.5 },
]);
const stale = (await eng.listStaleTakes()).filter(t => t.claim.startsWith('takes-parity claim'));
expect(stale.length).toBe(2);
const updated = await eng.updateTakeEmbeddingsBatch(
stale.map((t, i) => ({ take_id: Number(t.take_id), embedding: basisEmbedding(i + 200) })),
);
expect(updated).toBe(2);
const back = await eng.getTakeEmbeddings(stale.map(t => Number(t.take_id)));
expect(back.size).toBe(2);
// Vector search finds the exact-match claim (cosine similarity 1).
const hits = await eng.searchTakesVector(basisEmbedding(200), { limit: 5 });
expect(hits.map(h => h.claim)).toContain(stale[0].claim);
// The written rows left the stale pool on this engine.
const staleAfter = (await eng.listStaleTakes()).filter(t => t.claim.startsWith('takes-parity claim'));
expect(staleAfter.length).toBe(0);
}
});
});
// ── relationalFanout parity (v0.43) ─────────────────────────────────────
+6 -4
View File
@@ -174,12 +174,14 @@ d('embedding migration (live Postgres + pgvector)', () => {
expect(persisted).toEqual([[toModel, targetDims]]);
expect(await columnDims()).toBe(targetDims);
// All THREE dim-pinned text-embedding-space columns move together on real
// pgvector (query_cache + facts are created at brain-birth width and no
// migration ever ALTERs them — before the fix they stayed narrow, which
// silently killed the query cache and every per-fact embed write).
// ALL dim-pinned text-embedding-space columns move together on real
// pgvector (query_cache + facts + takes are created at brain-birth width
// and no later migration ALTERs them — before the fix they stayed
// narrow, which silently killed the query cache, every per-fact embed
// write, and every takes-lane embed write).
expect(await embeddingColWidth('query_cache')).toBe(targetDims);
expect(await embeddingColWidth('facts')).toBe(targetDims);
expect(await embeddingColWidth('takes')).toBe(targetDims);
// And the rebuilt columns actually ACCEPT a vector at the new width.
const nv = `[${new Array(targetDims).fill(0.01).join(',')}]`;
await engine.executeRaw(
+170
View File
@@ -0,0 +1,170 @@
/**
* #2089 the takes lane of `gbrain embed --stale` (embedStaleTakes via
* runEmbedCore). The wave shipped the lane with ZERO tests driving it: the
* engine writer (updateTakeEmbeddingsBatch) is covered, but the lane's
* control flow dry-run counting, id coercion into the writer, and the
* budget/abort cutShort skip was untested.
*
* Hermetic: mock.module on core/embedding.ts (same pattern as
* embed.serial.test.ts) + the gateway embed-transport seam for the creds
* preflight. Serial: mock.module + env mutation are process-global.
*/
import { test, expect, mock, beforeEach, afterEach } from 'bun:test';
import type { BrainEngine } from '../src/core/engine.ts';
let totalEmbedCalls = 0;
mock.module('../src/core/embedding.ts', () => ({
embedBatch: async (texts: string[]) => {
totalEmbedCalls++;
await new Promise(r => setTimeout(r, 5));
return texts.map(() => new Float32Array(1536));
},
currentEmbeddingSignature: () => 'test:model:1536',
}));
// Import AFTER mocking.
const { runEmbedCore } = await import('../src/commands/embed.ts');
const { __setEmbedTransportForTests } = await import('../src/core/ai/gateway.ts');
__setEmbedTransportForTests(async () => ({ embeddings: [], usage: { tokens: 0 } } as any));
/** Proxy mock engine (embed.serial.test.ts pattern) with call tracking. */
function mockEngine(overrides: Partial<Record<string, any>> = {}): BrainEngine {
const calls: { method: string; args: any[] }[] = [];
const track = (method: string) => (...args: any[]) => {
calls.push({ method, args });
if (overrides[method]) return overrides[method](...args);
return Promise.resolve(null);
};
const engine = new Proxy({} as any, {
get(_, prop: string) {
if (prop === '_calls') return calls;
if (overrides[prop]) {
return (...args: any[]) => {
calls.push({ method: prop, args });
return overrides[prop](...args);
};
}
return track(prop);
},
});
return engine;
}
function callCount(engine: BrainEngine, method: string): number {
return (engine as any)._calls.filter((c: any) => c.method === method).length;
}
beforeEach(() => {
totalEmbedCalls = 0;
});
afterEach(() => {
delete process.env.GBRAIN_EMBED_TIME_BUDGET_MS;
});
test('#2089 --stale takes lane: dry-run counts only; real run embeds and batch-writes with coerced numeric ids', async () => {
// Driver-shaped ids on purpose: string + bigint must arrive at the writer
// as plain numbers (the lane's Number() coercion).
const staleTakes = [
{ take_id: '11' as any, page_slug: 'p/a', row_num: 1, claim: 'claim one' },
{ take_id: 12n as any, page_slug: 'p/a', row_num: 2, claim: 'claim two' },
{ take_id: 13, page_slug: 'p/b', row_num: 1, claim: 'claim three' },
];
let written: Array<{ take_id: number; embedding: Float32Array }> = [];
const makeEngine = () => mockEngine({
countStaleChunks: async () => 0, // pages lane: nothing stale
invalidateStaleSignatureEmbeddings: async () => 0,
countStaleTakes: async () => staleTakes.length,
listStaleTakes: async () => staleTakes,
updateTakeEmbeddingsBatch: async (rows: any[]) => { written = rows; return rows.length; },
});
// Dry-run: count-only — no gateway call, no listStaleTakes, no writes.
const dryEngine = makeEngine();
const dry = await runEmbedCore(dryEngine, { stale: true, dryRun: true, quiet: true });
expect(dry.takes_would_embed).toBe(3);
expect(dry.takes_embedded ?? 0).toBe(0);
expect(totalEmbedCalls).toBe(0);
expect(callCount(dryEngine, 'listStaleTakes')).toBe(0);
expect(callCount(dryEngine, 'updateTakeEmbeddingsBatch')).toBe(0);
expect(written).toHaveLength(0);
// Real run: 3 claims fit one 64-cap batch → one gateway call, 3 writes.
const res = await runEmbedCore(makeEngine(), { stale: true, quiet: true });
expect(res.takes_embedded).toBe(3);
expect(res.failures).toBe(0);
expect(written).toHaveLength(3);
expect(written.map(w => w.take_id)).toEqual([11, 12, 13]);
for (const w of written) {
expect(typeof w.take_id).toBe('number');
expect(w.embedding).toBeInstanceOf(Float32Array);
expect(w.embedding.length).toBe(1536);
}
}, 20_000);
test('Wave-4 source scoping: `--stale --source X` threads sourceId into BOTH takes enumerators', async () => {
const enumeratorArgs: Array<{ method: string; opts: unknown }> = [];
let written: Array<{ take_id: number; embedding: Float32Array }> = [];
const engine = mockEngine({
countStaleChunks: async () => 0, // pages lane: nothing stale
invalidateStaleSignatureEmbeddings: async () => 0,
countStaleTakes: async (opts: unknown) => {
enumeratorArgs.push({ method: 'countStaleTakes', opts });
return 1;
},
listStaleTakes: async (opts: unknown) => {
enumeratorArgs.push({ method: 'listStaleTakes', opts });
return [{ take_id: 21, page_slug: 'p/scoped', row_num: 1, claim: 'scoped claim' }];
},
updateTakeEmbeddingsBatch: async (rows: any[]) => { written = rows; return rows.length; },
});
const res = await runEmbedCore(engine, { stale: true, quiet: true, sourceId: 'tenant-a' });
// Both enumerators saw the run's source scope — a `--source X` run must
// never enumerate (or pay to embed) another source's takes.
expect(enumeratorArgs).toEqual([
{ method: 'countStaleTakes', opts: { sourceId: 'tenant-a' } },
{ method: 'listStaleTakes', opts: { sourceId: 'tenant-a' } },
]);
expect(res.takes_embedded).toBe(1);
expect(written.map(w => w.take_id)).toEqual([21]);
// Unscoped run: enumerators get NO scope object (all-source semantics).
enumeratorArgs.length = 0;
const engine2 = mockEngine({
countStaleChunks: async () => 0,
invalidateStaleSignatureEmbeddings: async () => 0,
countStaleTakes: async (opts: unknown) => {
enumeratorArgs.push({ method: 'countStaleTakes', opts });
return 0;
},
});
await runEmbedCore(engine2, { stale: true, quiet: true });
expect(enumeratorArgs).toEqual([{ method: 'countStaleTakes', opts: undefined }]);
}, 20_000);
test('#2089 cutShort: a wall-clock-budget abort in the pages lane SKIPS the takes lane (next run picks them up)', async () => {
// 1ms budget; the pages lane's first page-load sleeps 30ms so the budget
// signal is guaranteed to have fired by the time the lane reports back.
process.env.GBRAIN_EMBED_TIME_BUDGET_MS = '1';
const engine = mockEngine({
countStaleChunks: async () => 5, // pass the 0-stale early return
invalidateStaleSignatureEmbeddings: async () => 0,
listStaleChunks: async () => {
await new Promise(r => setTimeout(r, 30));
return [];
},
countStaleTakes: async () => 99, // must never be reached
});
const res = await runEmbedCore(engine, { stale: true, quiet: true });
// The takes lane never even probed the stale count — a timed-out run must
// not start NEW work (#2089 contract: graceful skip, not extra load).
expect(callCount(engine, 'countStaleTakes')).toBe(0);
expect(res.takes_embedded).toBeUndefined();
expect(res.takes_would_embed).toBeUndefined();
expect(totalEmbedCalls).toBe(0);
}, 20_000);
+48
View File
@@ -165,6 +165,54 @@ describe('#3391 includeNullSignature widening', () => {
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(0);
expect(await engine.countStaleChunks({ signature: 'new:model:1' })).toBe(0);
});
// Wave-4 review: invalidation also re-stales TAKES on the same pages under
// the SAME signature predicate — a same-dim provider swap has no schema
// transition to drop takes.embedding, and old-space take vectors must not
// score against new-space queries. Routine runs (nothing drifted) touch
// zero takes, so there is no re-embed churn.
test('invalidation re-stales takes on drifted pages; fresh kept; grandfather lifts with the flag', async () => {
await seedEmbedded('legacy', 'abcde', null);
await seedEmbedded('drifted', 'fghij', 'old:model:1');
await seedEmbedded('fresh', 'klmno', 'new:model:1');
// One EMBEDDED take per page (text-embedding space, same as the chunks).
for (const slug of ['legacy', 'drifted', 'fresh']) {
const page = await engine.getPage(slug);
await engine.addTakesBatch([{
page_id: page!.id, row_num: 1, claim: `take on ${slug}`,
kind: 'fact', holder: 'world', weight: 0.9,
}]);
}
await engine.executeRaw(
`UPDATE takes
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector,
embedded_at = now()`,
[colDim],
);
expect(await engine.countStaleTakes()).toBe(0);
// Default (grandfather): only the drifted page's take re-stales; the
// returned count remains CHUNKS-only (existing contract).
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(1);
let staleTakes = await engine.listStaleTakes();
expect(staleTakes.map(t => t.page_slug)).toEqual(['drifted']);
const cleared = await engine.executeRaw<{ n: number }>(
`SELECT count(*)::int AS n FROM takes WHERE embedding IS NULL AND embedded_at IS NULL`,
);
expect(Number(cleared[0]?.n)).toBe(1); // embedded_at cleared alongside
// Widened (#3391): the legacy (NULL-sig) take re-stales too; fresh kept.
await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1', includeNullSignature: true });
staleTakes = await engine.listStaleTakes();
expect(staleTakes.map(t => t.page_slug).sort()).toEqual(['drifted', 'legacy']);
const keptTake = await engine.executeRaw<{ n: number }>(
`SELECT count(*)::int AS n FROM takes t
JOIN pages p ON p.id = t.page_id
WHERE p.slug = 'fresh' AND t.embedding IS NOT NULL`,
);
expect(Number(keptTake[0]?.n)).toBe(1);
});
});
describe('planEmbeddingMigration', () => {
@@ -0,0 +1,128 @@
/**
* Migration v126 (#2089) takes_embedding_active_dims.
*
* takes.embedding shipped in v37 hardcoded to VECTOR(1536); the takes table
* is migration-created (absent from schema.sql), so init-time dim templating
* never reached it. On a brain configured for any other width the column
* could never hold a real vector. v126 rebuilds it at the active dim.
*
* Pins (on a PGLite brain init'd at a NON-default dim):
* - after migrations, atttypmod (which IS the pgvector dimension) shows
* the active dim, not 1536
* - the HNSW partial index is recreated (dim under the #1734 cap)
* - idempotent: forcing a re-run of v126 no-ops (column shape unchanged)
* - the writer works end-to-end at the new width:
* updateTakeEmbeddingsBatch persists + searchTakesVector returns the row
*
* Serial: configureGateway mutates process-global gateway state (the file
* needs a non-default dim for its whole lifetime).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import { runMigrations } from '../src/core/migrate.ts';
const DIMS = 1024; // non-default (legacy test baseline is 1536; prod default 1280)
let engine: PGLiteEngine;
let pageId: number;
async function takesEmbeddingDim(): Promise<number> {
const rows = await engine.executeRaw<{ atttypmod: number }>(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
);
expect(rows.length).toBe(1);
return Number(rows[0].atttypmod);
}
beforeAll(async () => {
configureGateway({
embedding_model: 'litellm:custom-1024d',
embedding_dimensions: DIMS,
env: { ...process.env },
});
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
const p = await engine.putPage('companies/acme-example', {
title: 'Acme Example',
type: 'company' as const,
compiled_truth: '## Takes\n\nAcme is a widget company.\n',
});
pageId = p.id;
await engine.addTakesBatch([
{ page_id: pageId, row_num: 1, claim: 'dims-test claim at 1024', kind: 'fact', holder: 'world', weight: 0.9 },
]);
}, 60_000);
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
describe('migration v126 takes.embedding dim rebuild', () => {
test('column rebuilt at the active dim, not v37 hardcoded 1536', async () => {
// Sanity: the brain really is configured at the non-default dim.
const cfg = await engine.executeRaw<{ value: string }>(
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
);
expect(parseInt(cfg[0]?.value ?? '0', 10)).toBe(DIMS);
// For pgvector columns atttypmod IS the dimension.
expect(await takesEmbeddingDim()).toBe(DIMS);
});
test('HNSW partial index recreated (dim within the #1734 cap)', async () => {
const rows = await engine.executeRaw<{ indexdef: string }>(
`SELECT indexdef FROM pg_indexes
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
);
expect(rows.length).toBe(1);
expect(rows[0].indexdef).toContain('hnsw');
expect(rows[0].indexdef).toContain('vector_cosine_ops');
// v37's partial predicate preserved.
expect(rows[0].indexdef).toMatch(/WHERE\s+\(?active/i);
});
test('idempotent: forced re-run of v126 no-ops on a correct column', async () => {
await engine.setConfig('version', '125');
const { applied } = await runMigrations(engine);
expect(applied).toBeGreaterThanOrEqual(1); // v126 re-stamped
expect(await takesEmbeddingDim()).toBe(DIMS);
const idx = await engine.executeRaw<{ exists: boolean }>(
`SELECT EXISTS (
SELECT 1 FROM pg_indexes
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'
) AS exists`,
);
expect(idx[0]?.exists).toBe(true);
});
test('writer works end-to-end at the new width: batch write + vector search', async () => {
const stale = await engine.listStaleTakes();
const target = stale.find(t => t.claim === 'dims-test claim at 1024');
expect(target).toBeDefined();
const emb = new Float32Array(DIMS);
emb[7] = 1;
const updated = await engine.updateTakeEmbeddingsBatch([
{ take_id: Number(target!.take_id), embedding: emb },
]);
expect(updated).toBe(1);
// Round-trip through the reader at the new width.
const map = await engine.getTakeEmbeddings([Number(target!.take_id)]);
expect(map.get(Number(target!.take_id))?.length).toBe(DIMS);
// And through vector search (exact match → similarity ~1, top hit).
const hits = await engine.searchTakesVector(emb, { limit: 5 });
expect(hits.map(h => h.claim)).toContain('dims-test claim at 1024');
expect(hits[0].score).toBeCloseTo(1, 3);
// The written row left the stale pool.
const staleAfter = await engine.listStaleTakes();
expect(staleAfter.some(t => Number(t.take_id) === Number(target!.take_id))).toBe(false);
});
});
@@ -0,0 +1,82 @@
/**
* Migration v126 (#2089) the DEFAULT-dims no-op path, and the guard that
* protects backfilled vectors.
*
* The shipped serial suite (migrations-takes-embedding-dims.serial.test.ts)
* pins the 1024 REBUILD path only, and its idempotency re-run happens before
* any embedding exists so the most consequential property of the
* `currentDim === embeddingDim` early return was unpinned: a forced re-run
* of v126 on a correct column must NOT wipe already-backfilled take
* embeddings (the rebuild branch NULLs + drops the column). This test runs
* on a fresh default-dims (1536) brain the overwhelmingly common install
* asserts v126 no-opped at init, backfills a vector via the new writer, then
* forces a re-run and asserts the vector SURVIVES.
*/
import { test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runMigrations } from '../src/core/migrate.ts';
const DEFAULT_DIMS = 1536;
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
async function takesEmbeddingDim(): Promise<number> {
const rows = await engine.executeRaw<{ atttypmod: number }>(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
);
expect(rows.length).toBe(1);
return Number(rows[0].atttypmod);
}
test('v126 no-ops on a default-dims brain, and a forced re-run PRESERVES backfilled take embeddings', async () => {
// Fresh default brain: column already VECTOR(1536) from v37 → v126 no-op.
expect(await takesEmbeddingDim()).toBe(DEFAULT_DIMS);
const idx = await engine.executeRaw<{ indexdef: string }>(
`SELECT indexdef FROM pg_indexes
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
);
expect(idx.length).toBe(1); // v37's HNSW index intact
// Backfill one claim through the #2089 writer.
const page = await engine.putPage('companies/rerun-example', {
title: 'Rerun Example', type: 'company' as const,
compiled_truth: '## Takes\n\nA placeholder company page.\n',
});
await engine.addTakesBatch([
{ page_id: page.id, row_num: 1, claim: 'rerun-preservation claim', kind: 'fact', holder: 'world', weight: 0.9 },
]);
const stale = (await engine.listStaleTakes()).filter(t => t.claim === 'rerun-preservation claim');
expect(stale).toHaveLength(1);
const emb = new Float32Array(DEFAULT_DIMS);
emb[3] = 1;
expect(await engine.updateTakeEmbeddingsBatch([
{ take_id: Number(stale[0].take_id), embedding: emb },
])).toBe(1);
// Force v126 to run again (version rollback → re-migrate).
await engine.setConfig('version', '125');
const { applied } = await runMigrations(engine);
expect(applied).toBeGreaterThanOrEqual(1);
// The no-op guard held: same dims, index intact, and — the load-bearing
// bit — the backfilled embedding was NOT nulled/dropped by a re-rebuild.
expect(await takesEmbeddingDim()).toBe(DEFAULT_DIMS);
const kept = await engine.executeRaw<{ has_embedding: boolean }>(
`SELECT (embedding IS NOT NULL) AS has_embedding FROM takes WHERE id = $1`,
[Number(stale[0].take_id)],
);
expect(kept[0]?.has_embedding).toBe(true);
const hits = await engine.searchTakesVector(emb, { limit: 5 });
expect(hits.map(h => h.claim)).toContain('rerun-preservation claim');
}, 30_000);
+65
View File
@@ -288,6 +288,71 @@ describe('applyRetrievalUpgrade — state machine + atomicity (D12, D18)', () =>
expect(byName.get('embedding_multimodal')).toBe('vector(1024)');
});
// Wave-4 review: takes joined TEXT_EMBEDDING_DIM_PINNED_TABLES. Before the
// fix, a dim-change transition left takes.embedding at the old width and
// the #2089 embed takes lane failed every write forever (while still
// paying the gateway each run).
test('runSchemaTransition rebuilds takes.embedding at the target dim + recreates the partial HNSW index', async () => {
await setLegacyDefaultConfig();
await seedPages(150);
// Seed one take and give it a vector at the OLD width (1536).
const page = await engine.putPage('takes/transition-example', {
title: 'Takes transition example', type: 'note',
compiled_truth: 'A safe placeholder page for the takes dim transition.',
});
await engine.addTakesBatch([{
page_id: page.id, row_num: 1, claim: 'pre-transition claim',
kind: 'fact', holder: 'world', weight: 0.9,
}]);
const stale = (await engine.listStaleTakes()).filter(t => t.claim === 'pre-transition claim');
expect(stale).toHaveLength(1);
const takeId = Number(stale[0].take_id);
// Probe the CURRENT column width for the pre-transition write — the
// suite shares one engine, and an earlier apply test may already have
// transitioned the takes column (schema persists across resetPgliteState).
const cur = await engine.executeRaw<{ atttypmod: number }>(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'
AND attnum > 0 AND NOT attisdropped`,
);
const curDim = Number(cur[0]?.atttypmod);
expect(curDim).toBeGreaterThan(0);
const oldVec = new Float32Array(curDim);
oldVec[0] = 1;
expect(await engine.updateTakeEmbeddingsBatch([{ take_id: takeId, embedding: oldVec }])).toBe(1);
const plan = await planRetrievalUpgrade(engine);
await applyRetrievalUpgrade(engine, plan);
// Column rebuilt at the target dim (atttypmod IS the pgvector dimension).
const dim = await engine.executeRaw<{ atttypmod: number }>(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'
AND attnum > 0 AND NOT attisdropped`,
);
expect(Number(dim[0]?.atttypmod)).toBe(ZE_TARGET_EMBEDDING_DIM);
// Partial HNSW index recreated (dim under the #1734 cap), v37 predicate.
const idx = await engine.executeRaw<{ indexdef: string }>(
`SELECT indexdef FROM pg_indexes
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
);
expect(idx).toHaveLength(1);
expect(idx[0].indexdef).toContain('hnsw');
expect(idx[0].indexdef).toMatch(/WHERE\s+\(?active/i);
// Old-space vector discarded — the take is back in the stale pool.
const restaled = (await engine.listStaleTakes()).filter(t => Number(t.take_id) === takeId);
expect(restaled).toHaveLength(1);
// And the rebuilt column ACCEPTS a vector at the new width end-to-end.
const newVec = new Float32Array(ZE_TARGET_EMBEDDING_DIM);
newVec[3] = 1;
expect(await engine.updateTakeEmbeddingsBatch([{ take_id: takeId, embedding: newVec }])).toBe(1);
expect((await engine.getTakeEmbeddings([takeId])).get(takeId)?.length).toBe(ZE_TARGET_EMBEDDING_DIM);
});
test('runSchemaTransition restores partial WHERE on idx_chunks_embedding_image', async () => {
await setLegacyDefaultConfig();
await seedPages(150);
+7
View File
@@ -782,6 +782,13 @@ const COLUMN_EXEMPTIONS = new Set<string>([
'minion_jobs.budget_remaining_cents',
'minion_jobs.budget_owner_job_id',
'minion_jobs.budget_root_owner_id',
// v126 (#2089) — takes_embedding_active_dims rebuilds takes.embedding at
// the brain's configured dim (DROP COLUMN + ADD COLUMN inside the handler).
// Same precedent as facts.* / query_cache.*: the `takes` table is
// migration-created (v37), absent from PGLITE_SCHEMA_SQL, so no schema-blob
// forward reference can exist; the HNSW index recreation lives INSIDE the
// same v126 migration. Column-only rebuild, no bootstrap probe needed.
'takes.embedding',
]);
test('every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by applyForwardReferenceBootstrap (column-only class)', async () => {
+10
View File
@@ -19,6 +19,16 @@ function makeEngine(opts: { knownSources?: string[] } = {}) {
const pageLookups: unknown[][] = [];
const engine = {
getConfig: async () => null,
// Wave-4: cmdAdd routes through the shared appendTake helper
// (takes-append.ts), whose scoped page lookup is engine.getPage —
// the same seam persistThinkTake uses. Record [slug, scope] so the
// scoping assertions below stay shape-identical to the old raw-SQL pin.
getPage: async (slug: string, pageOpts?: { sourceId?: string; sourceIds?: string[] }) => {
pageLookups.push([slug, pageOpts?.sourceIds ?? pageOpts?.sourceId]);
if (slug === 'shared/page' && pageOpts?.sourceId === 'dept') return { id: 22, slug };
if (slug === 'shared/page' && (pageOpts?.sourceId === 'default' || pageOpts?.sourceId === undefined)) return { id: 11, slug };
return null;
},
executeRaw: async (sql: string, params: unknown[] = []) => {
if (sql.includes('FROM sources WHERE id = $1')) {
// Default (no `knownSources` override): every id "exists", matching
+73
View File
@@ -392,3 +392,76 @@ describe('countStaleTakes + listStaleTakes', () => {
expect(stale[0]).toHaveProperty('claim');
});
});
// #2089: the writer half of the takes embedding pipeline.
describe('updateTakeEmbeddingsBatch', () => {
test('empty input returns 0 without a query', async () => {
expect(await engine.updateTakeEmbeddingsBatch([])).toBe(0);
});
test('round-trip: writes 2 embeddings, stamps embedded_at, decrements stale count', async () => {
// Vector width must match the brain's active dim (config row), not a
// hardcoded literal — v126 rebuilds takes.embedding at this dim.
const dimRows = await engine.executeRaw<{ value: string }>(
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
);
const dims = parseInt(dimRows[0]?.value ?? '1536', 10);
const staleBefore = await engine.countStaleTakes();
expect(staleBefore).toBeGreaterThanOrEqual(2);
const stale = await engine.listStaleTakes();
const [a, b] = stale;
const embA = new Float32Array(dims);
embA[0] = 1;
const embB = new Float32Array(dims);
embB[1] = 1;
const updated = await engine.updateTakeEmbeddingsBatch([
{ take_id: Number(a.take_id), embedding: embA },
{ take_id: Number(b.take_id), embedding: embB },
]);
expect(updated).toBe(2);
// Read back via getTakeEmbeddings (the paired reader).
const map = await engine.getTakeEmbeddings([Number(a.take_id), Number(b.take_id)]);
expect(map.size).toBe(2);
expect(map.get(Number(a.take_id))!.length).toBe(dims);
expect(map.get(Number(a.take_id))![0]).toBeCloseTo(1, 5);
expect(map.get(Number(b.take_id))![1]).toBeCloseTo(1, 5);
// embedded_at stamped on both rows.
const rows = await engine.executeRaw<{ embedded_at: string | null }>(
`SELECT embedded_at FROM takes WHERE id = ANY($1::bigint[])`,
[[Number(a.take_id), Number(b.take_id)]],
);
expect(rows.length).toBe(2);
for (const r of rows) expect(r.embedded_at).not.toBeNull();
// Stale pool shrank by exactly the two written rows.
expect(await engine.countStaleTakes()).toBe(staleBefore - 2);
});
test('inactive (superseded) rows are skipped — count reflects actual updates', async () => {
// The supersedeTake suite above retired (alicePageId, row 3). Its row is
// active=false with a NULL embedding; the writer must not touch it.
const inactiveRows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM takes WHERE active = FALSE LIMIT 1`,
);
expect(inactiveRows.length).toBe(1);
const dims = parseInt(
(await engine.executeRaw<{ value: string }>(
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
))[0]?.value ?? '1536',
10,
);
const updated = await engine.updateTakeEmbeddingsBatch([
{ take_id: Number(inactiveRows[0].id), embedding: new Float32Array(dims) },
]);
expect(updated).toBe(0);
const check = await engine.executeRaw<{ embedding: unknown }>(
`SELECT embedding FROM takes WHERE id = $1`,
[Number(inactiveRows[0].id)],
);
expect(check[0]?.embedding).toBeNull();
});
});
+116 -1
View File
@@ -1,7 +1,7 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { operationsByName } from '../src/core/operations.ts';
import { runThink, persistSynthesis, type ThinkLLMClient } from '../src/core/think/index.ts';
import { runThink, persistSynthesis, persistThinkTake, THINK_TAKE_CLAIM_MAX_CHARS, type ThinkLLMClient } from '../src/core/think/index.ts';
import { sanitizeTakeForPrompt, renderTakesBlock } from '../src/core/think/sanitize.ts';
import { resolveCitations, parseInlineCitations, normalizeStructuredCitations } from '../src/core/think/cite-render.ts';
import { runGather } from '../src/core/think/gather.ts';
@@ -470,3 +470,118 @@ describe('think MCP op — #1698 C3 + #10', () => {
expect(res.warnings).toContain('SYNTHESIS_EMPTY_NOT_PERSISTED');
});
});
describe('persistThinkTake — #2556 think --take actually persists', () => {
function synthResult(answer: string, ok = true): any {
return {
question: 'what should this page remember?',
answer, citations: [], gaps: [], pagesGathered: 0, takesGathered: 0,
graphHits: 0, modelUsed: 'stub', rounds: 1, warnings: [], synthesisOk: ok,
diagnostics: { pagesFromHybrid: 0, takesFromKeyword: 0, takesFromVector: 0, graphHits: 0 },
};
}
test('appends the synthesis answer as the next anchor take row', async () => {
const target = await engine.putPage('notes/think-take-target-example', {
title: 'Think take target', type: 'note',
compiled_truth: 'A safe placeholder page for think take persistence.',
});
const persisted = await persistThinkTake(
engine, synthResult('This page should remember the synthesized placeholder insight.'),
{ anchor: 'notes/think-take-target-example' },
);
// Wave-4 dual-plane contract: this hermetic engine has no brain repo
// (sync.repo_path unset), so appendTake falls back to DB-only allocation
// and surfaces TAKE_FILE_PLANE_UNAVAILABLE. The row still lands.
expect(persisted).toMatchObject({ rowNum: 1, inserted: 1 });
expect(persisted.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
const takes = await engine.listTakes({ page_id: target.id });
expect(takes).toHaveLength(1);
expect(takes[0]).toMatchObject({
row_num: 1,
claim: 'This page should remember the synthesized placeholder insight.',
kind: 'take', holder: 'brain', source: 'gbrain think',
});
// Second take lands on the NEXT row (MAX+1 under the page lock).
const second = await persistThinkTake(
engine, synthResult('A second synthesized insight.'),
{ anchor: 'notes/think-take-target-example' },
);
expect(second.rowNum).toBe(2);
});
test('bounds the claim: 2500-char multi-line answer → single-line 2000-char claim + TAKE_CLAIM_TRUNCATED', async () => {
const target = await engine.putPage('notes/think-take-truncate-example', {
title: 'Think take truncate target', type: 'note',
compiled_truth: 'A safe placeholder page for claim-bound persistence.',
});
// Multi-line answer well past the cap: fence cells are single-line, and
// sanitizeTakeForPrompt only ever reads the first 500 chars anyway.
const line = 'a'.repeat(99);
const answer = Array.from({ length: 25 }, () => line).join('\n'); // 25*99 + 24 = 2499 chars
expect(answer.length).toBeGreaterThan(THINK_TAKE_CLAIM_MAX_CHARS);
const persisted = await persistThinkTake(engine, synthResult(answer), {
anchor: 'notes/think-take-truncate-example',
});
expect(persisted.rowNum).toBe(1);
expect(persisted.inserted).toBe(1);
expect(persisted.warnings).toContain('TAKE_CLAIM_TRUNCATED');
const takes = await engine.listTakes({ page_id: target.id });
expect(takes).toHaveLength(1);
expect(takes[0].claim.length).toBe(THINK_TAKE_CLAIM_MAX_CHARS);
expect(takes[0].claim).not.toContain('\n'); // flattened before bounding
});
test('refuses empty/no-LLM synthesis instead of writing a blank take', async () => {
const target = await engine.putPage('notes/think-take-empty-example', {
title: 'Think take empty target', type: 'note',
compiled_truth: 'A safe placeholder page for empty think take persistence.',
});
const persisted = await persistThinkTake(
engine, synthResult('(no LLM available)', false),
{ anchor: 'notes/think-take-empty-example' },
);
expect(persisted).toEqual({ rowNum: null, inserted: 0, warnings: ['TAKE_EMPTY_NOT_PERSISTED'] });
expect(await engine.listTakes({ page_id: target.id })).toHaveLength(0);
});
test('missing anchor and out-of-scope anchor fail with distinct loud signals', async () => {
expect(await persistThinkTake(engine, synthResult('x'), {})).toEqual(
{ rowNum: null, inserted: 0, warnings: ['TAKE_REQUIRES_ANCHOR'] });
const missing = await persistThinkTake(engine, synthResult('x'), { anchor: 'notes/no-such-page' });
expect(missing.rowNum).toBeNull();
expect(missing.warnings[0]).toContain('TAKE_ANCHOR_NOT_FOUND');
// Scope confinement: the anchor exists in the default source but the
// caller's federated grant excludes it -> not found (fail-closed).
const scoped = await persistThinkTake(engine, synthResult('x'), {
anchor: 'notes/think-take-target-example', sourceIds: ['some-other-source'],
});
expect(scoped.rowNum).toBeNull();
expect(scoped.warnings[0]).toContain('TAKE_ANCHOR_NOT_FOUND');
});
test('op handler: local take persists; remote take stays blocked (#2556 + trust gate)', async () => {
const op = operationsByName['think'];
await engine.putPage('notes/think-take-op-example', {
title: 'Think take op target', type: 'note',
compiled_truth: 'Safe placeholder for the op-layer take test.',
});
const baseCtx = (remote: boolean) => ({
engine, config: {} as any, dryRun: false, remote,
logger: { info() {}, warn() {}, error() {}, debug() {} } as any,
});
// Remote: take must be zeroed by the trust gate; nothing written.
const remoteRes: any = await withoutAnthropicKey(() =>
op.handler(baseCtx(true) as any, { question: 'q', anchor: 'notes/think-take-op-example', take: true }));
expect(remoteRes.take_row).toBeNull();
expect(remoteRes.remote_persisted_blocked).toBe(true);
// Local with no LLM: synthesis empty -> take refused with the loud warning
// (proves the local path CONSUMES the flag instead of ignoring it).
const localRes: any = await withoutAnthropicKey(() =>
op.handler(baseCtx(false) as any, { question: 'q', anchor: 'notes/think-take-op-example', take: true }));
expect(localRes.take_row).toBeNull();
expect(localRes.warnings).toContain('TAKE_EMPTY_NOT_PERSISTED');
});
});
+81
View File
@@ -0,0 +1,81 @@
/**
* #2556 `gbrain think --take` CLI exit-honesty pins.
*
* The shipped tests cover persistThinkTake (the core) and the MCP op
* handler, but nothing drives src/commands/think.ts's take path the two
* exit(1) branches ("--take requires --anchor" and "--take requested but no
* take row was written") were untested. An explicit --take that writes
* nothing MUST be loud (same honesty contract as --save / #1698 F2).
*
* Serial: spies on process.exit + console.error (process-global), and the
* no-LLM path mutates ANTHROPIC_API_KEY/GBRAIN_HOME via withoutAnthropicKey.
*/
import { test, expect, beforeAll, afterAll, spyOn } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runThinkCli } from '../src/commands/think.ts';
import { withoutAnthropicKey } from './helpers/no-anthropic-key.ts';
let engine: PGLiteEngine;
let anchorPageId: number;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
const p = await engine.putPage('notes/cli-take-anchor-example', {
title: 'CLI take anchor', type: 'note',
compiled_truth: 'A safe placeholder page for the CLI take exit tests.',
});
anchorPageId = p.id;
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
/**
* Run runThinkCli with process.exit + console.error captured. The exit spy
* throws a sentinel so control flow actually stops at the exit site (a
* non-throwing mock would let the command keep running past exit(1)).
* Note: an exit(1) inside the runThink try-block is caught by the #1698
* catch, which console.errors the sentinel message and exits again the
* FIRST captured code is the one asserted.
*/
async function runExpectingExit(args: string[]): Promise<{ code: number | null; stderr: string }> {
const errors: string[] = [];
let code: number | null = null;
const errSpy = spyOn(console, 'error').mockImplementation((...a: unknown[]) => {
errors.push(a.map(String).join(' '));
});
const exitSpy = spyOn(process, 'exit').mockImplementation(((c?: number) => {
if (code === null) code = c ?? 0;
throw new Error(`EXIT:${c}`);
}) as never);
try {
await runThinkCli(engine, args);
} catch (e) {
if (!(e instanceof Error) || !e.message.startsWith('EXIT:')) throw e;
} finally {
exitSpy.mockRestore();
errSpy.mockRestore();
}
return { code, stderr: errors.join('\n') };
}
test('--take without --anchor exits 1 with the actionable message (parse-time guard)', async () => {
const { code, stderr } = await runExpectingExit(['what do we know', '--take']);
expect(code).toBe(1);
expect(stderr).toContain('--take requires --anchor');
});
test('--take with no LLM (empty synthesis) exits 1 loudly and persists NOTHING', async () => {
const { code, stderr } = await withoutAnthropicKey(() =>
runExpectingExit(['what do we know about this page', '--take', '--anchor', 'notes/cli-take-anchor-example']),
);
// #2556 honesty contract: explicit --take that wrote nothing → exit 1
// with the take-specific message, not a silent 0.
expect(code).toBe(1);
expect(stderr).toContain('--take requested but no take row was written');
// And the DB really is untouched — no blank/stub take row.
expect(await engine.listTakes({ page_id: anchorPageId })).toHaveLength(0);
}, 30_000);
@@ -0,0 +1,83 @@
/**
* #2556 persistThinkTake concurrency pin.
*
* The function's doc-comment CLAIMS the MAX(row_num)+1 computation + insert
* are serialized under withPageLock(anchor) so two concurrent `think --take`
* calls (or a takes add racing a think --take) can't produce duplicate
* row_nums. The shipped tests only exercise SEQUENTIAL calls (row 1 then
* row 2), which passes even with no lock at all. This test drives the two
* calls CONCURRENTLY via Promise.all and asserts they land on distinct
* consecutive rows the behavioral proof the lock is actually keyed and
* actually serializes.
*
* Serial: the page lock is a real file under $GBRAIN_HOME/.gbrain/page-locks
* (process-global fs + env), and the loser of the race polls at 200ms.
*/
import { test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { persistThinkTake } from '../src/core/think/index.ts';
import { withEnv, emptyHome } from './helpers/with-env.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
function synthResult(answer: string): any {
return {
question: 'what should this page remember?',
answer, citations: [], gaps: [], pagesGathered: 0, takesGathered: 0,
graphHits: 0, modelUsed: 'stub', rounds: 1, warnings: [], synthesisOk: true,
diagnostics: { pagesFromHybrid: 0, takesFromKeyword: 0, takesFromVector: 0, graphHits: 0 },
};
}
test('two CONCURRENT persistThinkTake calls on the same anchor serialize to rows 1 and 2 (no duplicate row_num)', async () => {
const page = await engine.putPage('notes/think-take-race-example', {
title: 'Think take race target', type: 'note',
compiled_truth: 'A safe placeholder page for concurrent think take persistence.',
});
// GBRAIN_HOME → fresh temp dir so the page-lock files land in a hermetic
// location (gbrainPath honors GBRAIN_HOME) instead of the user's ~/.gbrain.
await withEnv({ GBRAIN_HOME: emptyHome() }, async () => {
const [a, b] = await Promise.all([
persistThinkTake(engine, synthResult('first concurrent synthesized insight'), {
anchor: 'notes/think-take-race-example',
}),
persistThinkTake(engine, synthResult('second concurrent synthesized insight'), {
anchor: 'notes/think-take-race-example',
}),
]);
// Both persisted and the row numbers are EXACTLY {1, 2} — a lost lock
// would yield {1, 1} (duplicate) or a failed insert. Wave-4 dual-plane
// contract: no brain repo is configured on this hermetic engine, so both
// appends fall back to DB-only allocation and surface
// TAKE_FILE_PLANE_UNAVAILABLE (still serialized under the page lock).
expect(a.inserted).toBe(1);
expect(b.inserted).toBe(1);
expect(a.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
expect(b.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
expect([a.rowNum, b.rowNum].sort()).toEqual([1, 2]);
});
// DB ground truth: two rows, distinct row_nums, both holder=brain.
const takes = await engine.listTakes({ page_id: page.id });
expect(takes).toHaveLength(2);
expect(new Set(takes.map(t => t.row_num)).size).toBe(2);
const dupCheck = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM takes
WHERE page_id = $1
GROUP BY row_num HAVING COUNT(*) > 1`,
[page.id],
);
expect(dupCheck).toHaveLength(0);
}, 15_000);
+129
View File
@@ -0,0 +1,129 @@
/**
* #2556 think op handler: the scope-dialect mapping under a FEDERATED ctx.
*
* The op handler maps thinkSourceScopeOpts's `allowedSources` (runThink's
* dialect) onto persistThinkTake's `sourceIds` (the engine's getPage
* dialect). The inline comment warns that spreading thinkScope would
* "silently drop the federated array and unscope the anchor lookup" but
* no shipped test constructs a ctx with `auth.allowedSources`, so that exact
* regression (a cross-source take write) would land green.
*
* These tests drive the FULL op handler to a successful synthesis using the
* gateway chat-transport test seam (no real LLM call), with the anchor page
* living in tenant-a:
* - grant includes tenant-a take row lands (mapping threads the array)
* - grant is tenant-b only TAKE_ANCHOR_NOT_FOUND, nothing written
* (fail-closed; an unscoped getPage would wrongly find the page)
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { operationsByName } from '../src/core/operations.ts';
import {
__setChatTransportForTests,
__setEmbedTransportForTests,
} from '../src/core/ai/gateway.ts';
import { withEnv, emptyHome } from './helpers/with-env.ts';
let engine: PGLiteEngine;
let anchorPageId: number;
const ANSWER = 'Federated synthesis insight for the anchor page.';
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
for (const id of ['tenant-a', 'tenant-b']) {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ($1, $1, NULL, '{}'::jsonb, NOW()) ON CONFLICT (id) DO NOTHING`,
[id],
);
}
const p = await engine.putPage('people/fed-anchor-example', {
title: 'Federated anchor', type: 'person',
compiled_truth: 'A safe placeholder page living in tenant-a.',
}, { sourceId: 'tenant-a' });
anchorPageId = p.id;
// Canned successful synthesis — chat() calls the transport directly,
// skipping provider resolution/SDK; think parses the JSON envelope.
__setChatTransportForTests(async () => ({
text: JSON.stringify({ answer: ANSWER, citations: [], gaps: [] }),
blocks: [],
stopReason: 'end' as const,
usage: { input_tokens: 10, output_tokens: 5, cache_read_tokens: 0, cache_creation_tokens: 0 },
model: 'anthropic:claude-opus-4-7',
providerId: 'anthropic',
}) as any);
// Neutralize any query-embedding attempt during gather (fake key below
// must never reach the network).
__setEmbedTransportForTests((async (args: any) => ({
embeddings: ((args?.values ?? []) as string[]).map(() => {
const v = new Array(1536).fill(0);
v[0] = 1;
return v;
}),
usage: { tokens: 1 },
})) as any);
}, 60_000);
afterAll(async () => {
__setChatTransportForTests(null);
__setEmbedTransportForTests(null);
await engine.disconnect();
});
function fedCtx(allowedSources: string[]): any {
return {
engine,
config: {} as any,
dryRun: false,
remote: false, // trusted local — safeTake honored; scope still MUST confine
auth: { allowedSources },
logger: { info() {}, warn() {}, error() {}, debug() {} } as any,
};
}
async function runThinkOpTake(allowedSources: string[]): Promise<any> {
const op = operationsByName['think'];
// Fake key so tryBuildGatewayClient builds a client (transport is stubbed);
// empty GBRAIN_HOME so the real user config can't leak into model routing
// and the page-lock files land in a hermetic temp dir.
return withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake', GBRAIN_HOME: emptyHome() }, () =>
op.handler(fedCtx(allowedSources), {
question: 'what do we know about the federated anchor?',
anchor: 'people/fed-anchor-example',
take: true,
}),
);
}
describe('think op — federated allowedSources → sourceIds mapping (#2556)', () => {
test('in-grant federated ctx: take row lands on the anchor in the granted source', async () => {
const res = await runThinkOpTake(['tenant-a']);
expect(res.take_row).toBe(1);
expect(res.take_inserted).toBe(1);
expect(res.remote_persisted_blocked).toBe(false);
const takes = await engine.listTakes({ page_id: anchorPageId });
expect(takes).toHaveLength(1);
expect(takes[0]).toMatchObject({
row_num: 1, claim: ANSWER, kind: 'take', holder: 'brain', source: 'gbrain think',
});
}, 30_000);
test('out-of-grant federated ctx: anchor lookup fail-closes — no cross-source take write', async () => {
const res = await runThinkOpTake(['tenant-b']);
// If the handler spread thinkScope (dropping the federated array), the
// getPage lookup would be UNSCOPED, find the tenant-a page, and write a
// cross-source row — this pins the mapping instead.
expect(res.take_row).toBeNull();
expect(res.take_inserted).toBe(0);
expect((res.warnings as string[]).some(w => w.startsWith('TAKE_ANCHOR_NOT_FOUND'))).toBe(true);
// Still exactly the one row from the in-grant test above.
expect(await engine.listTakes({ page_id: anchorPageId })).toHaveLength(1);
}, 30_000);
});