mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
113b4507f1 | ||
|
|
a88cf5e0e3 | ||
|
|
1dbd8665fc | ||
|
|
9a68e43e9d | ||
|
|
d42b25bee0 | ||
|
|
b9782dc743 | ||
|
|
79fdd3c33b | ||
|
|
547b2a9e0b |
@@ -2,6 +2,27 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.79.0] - 2026-08-09
|
||||
|
||||
**`think --take` actually persists, and takes finally get embeddings — vector search over your takes goes from structurally impossible to working.**
|
||||
|
||||
**`gbrain think --take` writes the take it promised.** The flag was accepted, validated (`--take` requires `--anchor`), and then consumed by nothing — a silent no-op on both the CLI and MCP paths. Now the synthesis answer is appended as the next take row on the anchor page (holder `brain`, source `gbrain think`), written to both planes the way `takes add` writes: the page's takes fence in your brain repo allocates the row number and the database mirrors it, so a later `takes add` can't collide with it. Claims are bounded (2,000 chars, flattened to one line) with a loud warning when truncation fires; an empty synthesis or missing anchor exits non-zero instead of pretending. Remote callers remain blocked from persistence, and the trust gate now treats an *absent* caller identity as untrusted on every internal forward. When no brain repo is reachable (headless installs), the take lands database-only with an explicit warning naming the posture. A related `takes add` fix rides along: the page lookup now precedes the fence write, so adding a take to a missing page fails cleanly instead of leaving a half-written fence. Design from a community pull request whose fix half was confirmed legitimate but died carrying unrelated feature scope — credited.
|
||||
|
||||
**Takes embeddings exist now.** The takes table has had an embedding column since it shipped — with no writer anywhere in the codebase, and a hardcoded width that only matched the default provider. Vector search over takes (including think's takes-recall arm) was structurally dead on every brain. This release: a migration rebuilds the column at your brain's actual embedding width (provably lossless — there was never a writer, so there is nothing to lose), `gbrain embed --stale` gains a takes lane that backfills every active claim through the same batching, spend gates, pacing, and abort handling the pages lane uses (`--source` scoping honored; dry-run counts via `takes_would_embed`), and takes join the dimension-transition set so future provider migrations rebuild them alongside chunks instead of leaving a permanently failing, permanently re-billed backfill. Provider swaps at the same width now re-stale take embeddings too, so old-space vectors are never scored against new-space queries.
|
||||
|
||||
### To take advantage of v0.42.79.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain embed --stale # one-time backfill: your takes get embeddings
|
||||
```
|
||||
|
||||
Then `gbrain think` starts drawing on vector-matched takes automatically. Try `gbrain think "question" --anchor <page> --take` to persist the synthesis as a take on that page.
|
||||
|
||||
### For contributors
|
||||
|
||||
The think-ops audit re-verified 18 planned items against master first: 10 already fixed, 3 obsolete, 2 folded into recorded decisions, leaving `#2556`, `#2089`, and a doc-truth residual. Review army + red team caught two release-blocking issues in the new code before ship — the missing dimension-transition registration (which would have turned every future provider migration into a paid retry loop) and the fence-plane row-number collision (which let one ordinary `takes add` silently overwrite a think-take) — both fixed in-wave with concurrency, federated-scope, CLI-honesty, and migration-rerun-preservation tests added. Follow-ups filed in TODOS.md ("Think-ops follow-ups"). Credit to the #2618 author for the take-persistence design.
|
||||
|
||||
## [0.42.75.0] - 2026-08-08
|
||||
|
||||
**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.**
|
||||
|
||||
@@ -4886,3 +4886,57 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
+ the models-doctor tests.
|
||||
|
||||
**Depends on:** nothing.
|
||||
|
||||
## Think-ops follow-ups (Wave 4, 2026-08)
|
||||
|
||||
### Next-take-row allocation helper (unify 4 hand-rolled copies)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** the `MAX(row_num)+1` next-row allocation for takes is hand-rolled in
|
||||
four places with inconsistent locking: `src/core/takes-append.ts` (DB-only
|
||||
fallback, under withPageLock), both engines' `supersedeTake` (in-SQL), and
|
||||
`src/core/cycle/phases/consolidate.ts`. Extract one shared helper with a consistent locking
|
||||
story so a fifth writer can't invent a fifth allocation.
|
||||
|
||||
**Why:** every divergent copy is a future duplicate-row or clobber bug of the
|
||||
exact class Wave 4 fixed (think-take rows overwritten by fence-allocated rows).
|
||||
|
||||
**How to start:** put the allocator next to `appendTake` in
|
||||
`src/core/takes-append.ts`; migrate call sites one at a time, pinning each with
|
||||
the existing takes-engine / consolidate tests.
|
||||
|
||||
### Takes-lane terminal-state warning parity + failure-path tests
|
||||
**Priority:** P3
|
||||
|
||||
**What:** the chunks lane warns loudly when a catch-up run finishes with
|
||||
persistently-failing chunks (embed.ts's post-loop `countStaleChunks` probe);
|
||||
the takes lane has no equivalent — the same takes can fail every run silently
|
||||
(deferral note sits at the takes-lane batch loop in `src/commands/embed.ts`).
|
||||
Add the mirror warning plus failure-path tests: batch-throw (embedPageTexts
|
||||
rejects) and probe-throw (countStaleTakes rejects) paths.
|
||||
|
||||
### CLI take success-path output test
|
||||
**Priority:** P3
|
||||
|
||||
**What:** `test/think-take-cli.serial.test.ts` covers the two exit(1) branches
|
||||
but not the success path (`Take: appended row #N ...` line + exit 0). The
|
||||
review provided a chat-transport seam stub (see
|
||||
`test/think-take-op-federated.test.ts`'s `__setChatTransportForTests` canned
|
||||
envelope) — drive `runThinkCli` with it and assert the success output.
|
||||
|
||||
### v126 HNSW dim-cap branch test (3072d) + real-PG run
|
||||
**Priority:** P3
|
||||
|
||||
**What:** migration v126's `hnswIndexExpected` gate (skip HNSW above the #1734
|
||||
cap) has no test at a >cap dim (e.g. 3072d: column rebuilt, index deliberately
|
||||
absent, exact scan still works). Add the PGLite test plus a DATABASE_URL-gated
|
||||
real-PG run — PGLite cannot surface real pgvector index-build failures.
|
||||
|
||||
### Federated think --take write-policy note
|
||||
**Priority:** P4
|
||||
|
||||
**What:** a take synthesized under a federated grant [A,B] persists into the
|
||||
ANCHOR page's source — documented behavior (the anchor lookup is
|
||||
grant-confined; the write inherits the anchor's source). Revisit if per-source
|
||||
take provenance ever lands; a grant-scoped caller writing brain-holder takes
|
||||
into a mounted source may want an explicit provenance column instead.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -65,6 +65,8 @@ Key files:
|
||||
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
|
||||
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
|
||||
to op params. `think` is a special case: the server's `think` op
|
||||
intentionally disables `--save`/`--take` for remote callers
|
||||
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
|
||||
intentionally disables `--save`/`--take` for remote callers (the think
|
||||
handler's fail-closed trust-boundary gate in operations.ts — persistence
|
||||
requires `ctx.remote !== false`, and blocked callers get
|
||||
`remote_persisted_blocked: true`); thin-client `think` warns
|
||||
loudly when those flags are set.
|
||||
|
||||
@@ -51,14 +51,16 @@ llama-server, and other bring-your-own-model providers).
|
||||
people running deliberate experiments.
|
||||
5. **Apply.** When the target width differs from the actual column width,
|
||||
runs the same atomic schema transition `ze-switch` uses, in one
|
||||
transaction. It rebuilds **all three dim-pinned text-embedding-space
|
||||
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
|
||||
`facts.embedding` — at the new width, preserving each column's type
|
||||
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
|
||||
three leaves it silently broken: a narrow `query_cache.embedding` makes
|
||||
every cache write and read fail *by design* (the cache swallows errors so
|
||||
it can never break search) for a permanent 0% hit rate, and a narrow
|
||||
`facts.embedding` fails every per-fact embed write. The image/multimodal
|
||||
transaction. It rebuilds **all four dim-pinned text-embedding-space
|
||||
columns** — `content_chunks.embedding`, `query_cache.embedding`,
|
||||
`facts.embedding`, and `takes.embedding` — at the new width, preserving
|
||||
each column's type (`vector` vs `halfvec`) and recreating its HNSW index.
|
||||
Missing any of the four leaves it silently broken: a narrow
|
||||
`query_cache.embedding` makes every cache write and read fail *by design*
|
||||
(the cache swallows errors so it can never break search) for a permanent
|
||||
0% hit rate, a narrow `facts.embedding` fails every per-fact embed write,
|
||||
and a narrow `takes.embedding` fails every `embed --stale` takes-lane
|
||||
write while still paying the embedding gateway each run. The image/multimodal
|
||||
columns ARE deliberately untouched — they use separate models whose
|
||||
dimensions are independent of the text embedding model.
|
||||
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
|
||||
|
||||
@@ -136,6 +136,8 @@ Stable phase names shipped in v0.15.2:
|
||||
- `doctor.db_checks` (umbrella for all DB-side doctor checks)
|
||||
- `orphans.scan`
|
||||
- `embed.pages`
|
||||
- `embed.takes` (the takes lane of `gbrain embed --all/--stale`; one tick per
|
||||
64-claim batch, total is the stale-take batch count at lane start)
|
||||
- `extract.links_fs`, `extract.timeline_fs`, `extract.links_db`, `extract.timeline_db`
|
||||
- `import.files`
|
||||
- `sync.deletes`, `sync.renames`, `sync.imports`
|
||||
|
||||
+1
-1
@@ -148,7 +148,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.75.0",
|
||||
"version": "0.42.79.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.5",
|
||||
|
||||
+149
-5
@@ -196,6 +196,17 @@ export interface EmbedResult {
|
||||
* corpus-wide outage doesn't bloat structured output. Additive field.
|
||||
*/
|
||||
failure_samples: string[];
|
||||
/**
|
||||
* #2089: take claims embedded by the takes lane (`--all`/`--stale` runs
|
||||
* only; per-slug runs skip the lane). Present only when the lane ran.
|
||||
* 0-or-absent in dryRun (see `takes_would_embed`).
|
||||
*/
|
||||
takes_embedded?: number;
|
||||
/**
|
||||
* #2089: take claims that WOULD be embedded if not for dryRun. Mirrors
|
||||
* `would_embed` naming. Present only when the lane ran in dryRun mode.
|
||||
*/
|
||||
takes_would_embed?: number;
|
||||
/** True if this run was a dry-run. */
|
||||
dryRun: boolean;
|
||||
/**
|
||||
@@ -416,7 +427,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
}
|
||||
}
|
||||
try {
|
||||
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
|
||||
const laneAborted = await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
|
||||
batchSize: opts.batchSize,
|
||||
priority: opts.priority,
|
||||
catchUp: opts.catchUp,
|
||||
@@ -425,6 +436,24 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
quiet: opts.quiet,
|
||||
includeNullSignature: opts.includeNullSignature,
|
||||
}, opts.signal);
|
||||
// #2089: takes lane. Runs AFTER the pages/chunks lane so a wall-clock
|
||||
// budget hit (or caller abort) inside the pages lane skips takes
|
||||
// gracefully — the next run picks them up via the NULL-embedding
|
||||
// predicate. Inside this try so the single-flight locks are still held
|
||||
// — and the lane threads opts.sourceId into its enumerators, so a
|
||||
// `--source X` run touches ONLY X's takes: the work is confined to the
|
||||
// same source(s) whose locks this run actually holds (per-source
|
||||
// mutual exclusion holds for the takes lane too). Pacer telemetry in
|
||||
// the finally covers takes DB writes as well.
|
||||
if (!laneAborted && !isAborted(opts.signal)) {
|
||||
await embedStaleTakes(engine, result, {
|
||||
dryRun: !!opts.dryRun,
|
||||
pacer,
|
||||
signal: opts.signal,
|
||||
quiet: opts.quiet,
|
||||
sourceId: opts.sourceId,
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
// E1: surface pacing telemetry (human + structured) when pacing was on.
|
||||
const snap = pacer.snapshot();
|
||||
@@ -750,6 +779,11 @@ function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true when the run was cut short (caller abort or, on the stale
|
||||
* path, the internal wall-clock budget) so runEmbedCore can skip the takes
|
||||
* lane instead of starting new work after a timeout (#2089).
|
||||
*/
|
||||
async function embedAll(
|
||||
engine: BrainEngine,
|
||||
staleOnly: boolean,
|
||||
@@ -771,7 +805,7 @@ async function embedAll(
|
||||
includeNullSignature?: boolean;
|
||||
},
|
||||
signal?: AbortSignal,
|
||||
) {
|
||||
): Promise<boolean> {
|
||||
// v0.41.31: current embedding provenance signature. Stamped onto pages
|
||||
// when their chunks are (re)embedded so a later model/dimension swap is
|
||||
// detectable as stale.
|
||||
@@ -947,6 +981,8 @@ async function embedAll(
|
||||
slog(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
|
||||
}
|
||||
}
|
||||
// #2089: --all has no wall-clock budget; only a caller abort cuts it short.
|
||||
return isAborted(signal);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -988,7 +1024,7 @@ async function embedAllStale(
|
||||
},
|
||||
signature?: string,
|
||||
externalSignal?: AbortSignal,
|
||||
) {
|
||||
): Promise<boolean> {
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
const sourceOpt = sourceId ? { sourceId } : undefined;
|
||||
@@ -1051,7 +1087,7 @@ async function embedAllStale(
|
||||
slog('Embedded 0 chunks (0 stale found)');
|
||||
}
|
||||
}
|
||||
return;
|
||||
return isAborted(externalSignal);
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
@@ -1059,7 +1095,7 @@ async function embedAllStale(
|
||||
result.total_chunks += staleCount;
|
||||
if (onProgress) onProgress(1, 1, 0);
|
||||
if (!staleOpts?.quiet) slog(`[dry-run] Would embed ${staleCount} stale chunks`);
|
||||
return;
|
||||
return isAborted(externalSignal);
|
||||
}
|
||||
|
||||
// v0.33.3: cursor-paginated stale loading. Instead of pulling all 48K+
|
||||
@@ -1344,6 +1380,114 @@ async function embedAllStale(
|
||||
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${result.failures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
|
||||
}
|
||||
}
|
||||
|
||||
// #2089: report whether the budget/caller abort cut the run short so the
|
||||
// takes lane can be skipped gracefully (same next-run pickup semantics).
|
||||
return effectiveSignal.aborted;
|
||||
}
|
||||
|
||||
/** #2089: claims per gateway batch in the takes lane. */
|
||||
export const TAKES_EMBED_BATCH_SIZE = 64;
|
||||
|
||||
/**
|
||||
* #2089: the takes lane. Backfills `takes.embedding` for every stale claim
|
||||
* (active AND embedding IS NULL — supersede creates a new row, so edited
|
||||
* claims re-stale automatically). Runs after the pages/chunks lane in
|
||||
* `--all`/`--stale` runs.
|
||||
*
|
||||
* Reuses `embedPageTexts` — the SAME helper the chunks lane feeds — so
|
||||
* rate-limit backoff, per-item failure isolation, spend gates, and abort
|
||||
* signals apply identically; no second gateway path exists to drift.
|
||||
* DB writes route through `observed(pacer, ...)` + `pacer.pace()` between
|
||||
* batches, mirroring the pages lane's DB-contention pacing.
|
||||
*
|
||||
* Dry-run mirrors the pages lane: count-only (`takes_would_embed`), no
|
||||
* gateway calls, no writes.
|
||||
*/
|
||||
async function embedStaleTakes(
|
||||
engine: BrainEngine,
|
||||
result: EmbedResult,
|
||||
opts: { dryRun: boolean; pacer: DbPacer; signal?: AbortSignal; quiet?: boolean; sourceId?: string },
|
||||
): Promise<void> {
|
||||
// Source scope: thread the run's --source through both enumerators so a
|
||||
// scoped run never enumerates (or pays to embed) another source's takes.
|
||||
const scope = opts.sourceId !== undefined ? { sourceId: opts.sourceId } : undefined;
|
||||
let staleCount: number;
|
||||
try {
|
||||
// Defensive Number(): int8 counts can arrive as BigInt/string from a
|
||||
// driver; a null/undefined (partial engine double in tests) reads as 0.
|
||||
staleCount = Number(await engine.countStaleTakes(scope)) || 0;
|
||||
} catch (e: unknown) {
|
||||
// Pre-v37 brains (or a wedged probe) must not fail the whole embed run
|
||||
// over the optional takes lane.
|
||||
serr(` [takes] stale-count probe failed; skipping takes lane: ${e instanceof Error ? e.message : e}`);
|
||||
return;
|
||||
}
|
||||
if (staleCount <= 0) return;
|
||||
|
||||
if (opts.dryRun) {
|
||||
result.takes_would_embed = (result.takes_would_embed ?? 0) + staleCount;
|
||||
if (!opts.quiet) slog(`[dry-run] Would embed ${staleCount} stale take claim(s)`);
|
||||
return;
|
||||
}
|
||||
|
||||
const staleTakes = (await engine.listStaleTakes(scope)) ?? [];
|
||||
if (staleTakes.length === 0) return;
|
||||
result.takes_embedded ??= 0;
|
||||
|
||||
// Same createProgress reporter pattern as 'embed.pages' (runEmbed): one
|
||||
// tick per batch, stderr-only, non-TTY plain lines unless --progress-json.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('embed.takes', Math.ceil(staleTakes.length / TAKES_EMBED_BATCH_SIZE));
|
||||
try {
|
||||
for (let i = 0; i < staleTakes.length; i += TAKES_EMBED_BATCH_SIZE) {
|
||||
if (isAborted(opts.signal)) break; // bail between batches (pages-loop contract)
|
||||
const batch = staleTakes.slice(i, i + TAKES_EMBED_BATCH_SIZE);
|
||||
try {
|
||||
const { embeddings, failed, firstError } = await embedPageTexts(
|
||||
batch.map((t) => t.claim),
|
||||
opts.signal ? { abortSignal: opts.signal } : {},
|
||||
);
|
||||
const writes: Array<{ take_id: number; embedding: Float32Array }> = [];
|
||||
for (let j = 0; j < batch.length; j++) {
|
||||
const emb = embeddings[j];
|
||||
// int8 ids may arrive as BigInt from the driver — coerce.
|
||||
if (emb) writes.push({ take_id: Number(batch[j].take_id), embedding: emb });
|
||||
}
|
||||
const updated = await observed(opts.pacer, () => engine.updateTakeEmbeddingsBatch(writes));
|
||||
result.takes_embedded += updated;
|
||||
if (failed > 0) {
|
||||
recordFailure(result, failed, `takes:${batch[0]?.page_slug ?? '?'}`, firstError);
|
||||
// `updated` is the RETURNING-based count from the writer; rows the
|
||||
// writer skipped (superseded mid-flight, `AND active` guard) are
|
||||
// called out separately so the arithmetic is honest.
|
||||
const skipped = writes.length - updated;
|
||||
serr(
|
||||
` [takes] ${failed} claim(s) failed to embed in batch; ${updated} written` +
|
||||
(skipped > 0 ? `, ${skipped} skipped (row retired mid-flight)` : ''),
|
||||
);
|
||||
}
|
||||
// NOTE: unlike the chunks lane's catch-up terminal-state warning
|
||||
// (see the countStaleChunks probe after the pages loop), persistently
|
||||
// failing takes are not yet flagged as stuck — deferred, see
|
||||
// TODOS.md "Think-ops follow-ups (Wave 4, 2026-08)".
|
||||
} catch (e: unknown) {
|
||||
if (isAborted(opts.signal)) break; // shutdown, not a failure
|
||||
recordFailure(result, batch.length, `takes:${batch[0]?.page_slug ?? '?'}`, e);
|
||||
serr(` [takes] Error embedding claim batch: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
progress.tick(1);
|
||||
// Cooperative DB-contention pace between batches (no-op when unpaced).
|
||||
try {
|
||||
await opts.pacer.pace(opts.signal);
|
||||
} catch (e) {
|
||||
if (!(e instanceof AbortError)) throw e;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
progress.finish();
|
||||
}
|
||||
if (!opts.quiet) slog(`Embedded ${result.takes_embedded} take claim(s)`);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+34
-40
@@ -19,16 +19,22 @@
|
||||
* 6. releases the lock (auto via withPageLock)
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { existsSync } from 'node:fs';
|
||||
import type { BrainEngine, TakeKind } from '../core/engine.ts';
|
||||
import {
|
||||
parseTakesFence,
|
||||
upsertTakeRow,
|
||||
supersedeRow,
|
||||
type ParsedTake,
|
||||
} from '../core/takes-fence.ts';
|
||||
import { withPageLock } from '../core/page-lock.ts';
|
||||
import {
|
||||
appendTake,
|
||||
TakePageNotFoundError,
|
||||
pageFilePath,
|
||||
readBodyOrEmpty,
|
||||
writeBody,
|
||||
resolveBrainDirOrNull,
|
||||
} from '../core/takes-append.ts';
|
||||
import { resolveSourceId } from '../core/source-resolver.ts';
|
||||
import { resolveOwnerHolder } from '../core/owner-holder.ts';
|
||||
|
||||
@@ -44,26 +50,20 @@ function flagPresent(args: string[], name: string): boolean {
|
||||
return args.includes(name);
|
||||
}
|
||||
|
||||
// CLI-strict wrapper around the shared resolveBrainDirOrNull (takes-append.ts):
|
||||
// unresolvable → loud error + exit(1). Core callers (persistThinkTake) use the
|
||||
// null-returning form and fall back to DB-only allocation instead.
|
||||
async function resolveBrainDir(engine: BrainEngine | null, explicitDir: string | null): Promise<string> {
|
||||
if (explicitDir) {
|
||||
if (!existsSync(explicitDir)) {
|
||||
console.error(`--dir path does not exist: ${explicitDir}`);
|
||||
process.exit(1);
|
||||
}
|
||||
return explicitDir;
|
||||
}
|
||||
if (engine) {
|
||||
const configured = await engine.getConfig('sync.repo_path');
|
||||
if (configured && existsSync(configured)) return configured;
|
||||
if (explicitDir && !existsSync(explicitDir)) {
|
||||
console.error(`--dir path does not exist: ${explicitDir}`);
|
||||
process.exit(1);
|
||||
}
|
||||
const dir = engine ? await resolveBrainDirOrNull(engine, explicitDir) : (explicitDir ?? null);
|
||||
if (dir) return dir;
|
||||
console.error('No brain directory configured. Pass --dir <path> or run `gbrain init` first.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function pageFilePath(brainDir: string, slug: string): string {
|
||||
return join(brainDir, `${slug}.md`);
|
||||
}
|
||||
|
||||
function ensureKind(raw: string | undefined): TakeKind {
|
||||
if (!raw) {
|
||||
console.error('Missing --kind. Expected one of: fact, take, bet, hunch.');
|
||||
@@ -117,15 +117,8 @@ async function resolveTakesSourceId(engine: BrainEngine): Promise<string> {
|
||||
return resolveSourceId(engine, null);
|
||||
}
|
||||
|
||||
function readBodyOrEmpty(path: string): string {
|
||||
if (!existsSync(path)) return '';
|
||||
return readFileSync(path, 'utf-8');
|
||||
}
|
||||
|
||||
function writeBody(path: string, body: string): void {
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
writeFileSync(path, body, 'utf-8');
|
||||
}
|
||||
// readBodyOrEmpty / writeBody / pageFilePath now live in
|
||||
// src/core/takes-append.ts (shared with the dual-plane appendTake helper).
|
||||
|
||||
// --- Subcommands ---
|
||||
|
||||
@@ -209,22 +202,23 @@ async function cmdAdd(engine: BrainEngine, args: string[], sourceId?: string): P
|
||||
const dirArg = flagValue(args, '--dir');
|
||||
const brainDir = await resolveBrainDir(engine, dirArg ?? null);
|
||||
|
||||
await withPageLock(slug, async () => {
|
||||
const path = pageFilePath(brainDir, slug);
|
||||
const body = readBodyOrEmpty(path);
|
||||
const { body: nextBody, rowNum } = upsertTakeRow(body, {
|
||||
claim, kind, holder, weight, source, sinceDate: since, active: true,
|
||||
// Dual-plane append via the shared core helper (takes-append.ts): page
|
||||
// lock → scoped page lookup → fence allocates the row number (file-first)
|
||||
// → DB mirror. Same path persistThinkTake uses, so the planes can't drift.
|
||||
try {
|
||||
const { rowNum } = await appendTake(engine, {
|
||||
slug, sourceId, claim, kind, holder, weight,
|
||||
source, sinceDate: since, brainDir,
|
||||
});
|
||||
writeBody(path, nextBody);
|
||||
|
||||
// Mirror to DB. Page may not be in DB yet if not synced — caller must run sync first.
|
||||
const pageId = await getPageId(engine, slug, sourceId);
|
||||
await engine.addTakesBatch([{
|
||||
page_id: pageId, row_num: rowNum, claim, kind, holder, weight,
|
||||
since_date: since, source, active: true, superseded_by: null,
|
||||
}]);
|
||||
console.log(`Added take #${rowNum} to ${slug}.`);
|
||||
});
|
||||
} catch (e) {
|
||||
if (e instanceof TakePageNotFoundError) {
|
||||
// Page may not be in DB yet if not synced — caller must run sync first.
|
||||
console.error(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}. Run \`gbrain sync\` first.`);
|
||||
process.exit(1);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
async function cmdUpdate(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
|
||||
|
||||
+44
-1
@@ -6,10 +6,11 @@
|
||||
* degrades to gather-only output with a warning if missing.
|
||||
*/
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { runThink, persistSynthesis, stripGapsSection } from '../core/think/index.ts';
|
||||
import { runThink, persistSynthesis, persistThinkTake, stripGapsSection } from '../core/think/index.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
import { canonicalLookup } from '../core/model-pricing.ts';
|
||||
import { resolveSourceWithTier, ALL_SOURCES } from '../core/source-resolver.ts';
|
||||
|
||||
function flagValue(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
@@ -114,6 +115,8 @@ prints what would have been the input (exit 0).
|
||||
let result: any;
|
||||
let savedSlug: string | undefined;
|
||||
let evidenceInserted = 0;
|
||||
let takeRow: number | null = null;
|
||||
let takeInserted = 0;
|
||||
const cfg = loadConfig();
|
||||
if (isThinClient(cfg)) {
|
||||
if (save || take) {
|
||||
@@ -160,6 +163,41 @@ prints what would have been the input (exit 0).
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
// #2556: --take was accepted-and-ignored (validated at parse time, then
|
||||
// nothing consumed it). Same honesty contract as --save: persist or
|
||||
// exit non-zero — an explicit --take that writes nothing must be loud.
|
||||
if (take) {
|
||||
// Wave-4: resolve the caller's ambient source context (GBRAIN_SOURCE /
|
||||
// .gbrain-source dotfile / local_path / brain default — the same
|
||||
// resolveSourceId chain every other CLI surface uses) so a duplicate
|
||||
// anchor slug across sources lands on the CALLER's-context page, not
|
||||
// a planner-chosen one. The seed_default tier means NOTHING was
|
||||
// configured anywhere — keep the historical unscoped posture there
|
||||
// (undefined → unscoped getPage), and __all__ has span-everything
|
||||
// semantics, which for a single-page lookup is also unscoped.
|
||||
// Resolution errors (e.g. GBRAIN_SOURCE naming an unregistered
|
||||
// source) propagate to the catch below — fail-closed, matching the
|
||||
// `gbrain takes` posture.
|
||||
const resolvedSource = await resolveSourceWithTier(engine, undefined);
|
||||
const takeSourceId =
|
||||
resolvedSource.tier === 'seed_default' || resolvedSource.source_id === ALL_SOURCES
|
||||
? undefined
|
||||
: resolvedSource.source_id;
|
||||
const persistedTake = await persistThinkTake(engine, result, {
|
||||
anchor,
|
||||
...(takeSourceId !== undefined ? { sourceId: takeSourceId } : {}),
|
||||
});
|
||||
takeRow = persistedTake.rowNum;
|
||||
takeInserted = persistedTake.inserted;
|
||||
for (const w of persistedTake.warnings) result.warnings.push(w);
|
||||
if (!persistedTake.rowNum) {
|
||||
console.error(
|
||||
'think: --take requested but no take row was written (empty synthesis ' +
|
||||
'or anchor page not found) — nothing persisted.',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
// #1698: an unresolvable explicit --model throws here. Clean non-zero exit
|
||||
// with the actionable message, not a stack trace.
|
||||
@@ -179,6 +217,8 @@ prints what would have been the input (exit 0).
|
||||
cost_usd: costUsd ?? null,
|
||||
saved_slug: savedSlug ?? null,
|
||||
evidence_inserted: evidenceInserted,
|
||||
take_row: takeRow,
|
||||
take_inserted: takeInserted,
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
@@ -198,6 +238,9 @@ prints what would have been the input (exit 0).
|
||||
if (savedSlug) {
|
||||
console.log(`Saved: ${savedSlug} (${evidenceInserted} evidence rows)`);
|
||||
}
|
||||
if (takeRow) {
|
||||
console.log(`Take: appended row #${takeRow} to ${anchor} (holder=brain)`);
|
||||
}
|
||||
if (result.warnings.length > 0) {
|
||||
console.error(`Warnings: ${result.warnings.join(', ')}`);
|
||||
}
|
||||
|
||||
+31
-4
@@ -1050,6 +1050,11 @@ export interface BrainEngine {
|
||||
* default grandfather clause would silently keep them mixed into the new
|
||||
* index. `gbrain migrate embeddings` and `embed --stale
|
||||
* --include-null-signature` set this.
|
||||
*
|
||||
* ALSO re-stales `takes` rows (embedding + embedded_at → NULL) on the same
|
||||
* pages, under the SAME signature predicate + source scope — takes carry
|
||||
* text-embedding-space vectors too, and a same-dim provider swap has no
|
||||
* schema transition to drop them. The returned count remains chunks-only.
|
||||
*/
|
||||
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number>;
|
||||
/**
|
||||
@@ -1552,11 +1557,33 @@ export interface BrainEngine {
|
||||
/** Look up embeddings by take id (mirrors getEmbeddingsByChunkIds). */
|
||||
getTakeEmbeddings(ids: number[]): Promise<Map<number, Float32Array>>;
|
||||
|
||||
/** Pre-flight count for `gbrain embed --stale`. WHERE active AND embedding IS NULL. */
|
||||
countStaleTakes(): Promise<number>;
|
||||
/**
|
||||
* Pre-flight count for `gbrain embed --stale`. WHERE active AND embedding
|
||||
* IS NULL. `opts.sourceId` scopes via the page join (p.source_id) so a
|
||||
* `--source X` run only counts X's takes.
|
||||
*/
|
||||
countStaleTakes(opts?: { sourceId?: string }): Promise<number>;
|
||||
|
||||
/** List stale takes (no embedding column in payload — same pattern as listStaleChunks). */
|
||||
listStaleTakes(): Promise<StaleTakeRow[]>;
|
||||
/**
|
||||
* List stale takes (no embedding column in payload — same pattern as
|
||||
* listStaleChunks). `opts.sourceId` scopes via the page join.
|
||||
*/
|
||||
listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]>;
|
||||
|
||||
/**
|
||||
* #2089: batch-write take embeddings — the writer half of the takes
|
||||
* embedding pipeline (`countStaleTakes`/`listStaleTakes` enumerate,
|
||||
* this persists). Sets `embedding` + `embedded_at = now()` on each
|
||||
* ACTIVE row only: a claim superseded mid-embed produces a new stale
|
||||
* row via supersedeTake, and the retired row must not receive the
|
||||
* vector. Returns the number of rows actually updated (superseded /
|
||||
* deleted ids are silently skipped). Empty input → 0 without a query.
|
||||
*
|
||||
* Wrapped in `batchRetry` like the other batch primitives (addTakesBatch
|
||||
* precedent), so `opts` (auditSite, AbortSignal) is honored; same
|
||||
* no-double-wrap contract as `BatchOpts`.
|
||||
*/
|
||||
updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number>;
|
||||
|
||||
/**
|
||||
* Update a take's mutable fields. May NOT change claim/kind/holder per the
|
||||
|
||||
@@ -55,7 +55,12 @@ export interface ExtractTakesFromPagesOpts {
|
||||
includeCovered?: boolean;
|
||||
/** Owner identifier for the inserted takes. Default 'system'. */
|
||||
holder?: string;
|
||||
/** Model override; defaults to facts.extraction_model. */
|
||||
/**
|
||||
* Model override; defaults to the configured chat model via getChatModel()
|
||||
* (#2997 — NOT facts.extraction_model, and deliberately not a tier default:
|
||||
* OAuth/local-only installs have exactly one working model, and a tier
|
||||
* fallback would reintroduce the hardcoded-Haiku failure #2997 fixed).
|
||||
*/
|
||||
model?: string;
|
||||
/** Progress hook called per page. */
|
||||
onProgress?: (done: number, total: number, claims: number) => void;
|
||||
|
||||
+81
-1
@@ -1,7 +1,7 @@
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import { hnswMaxDimsForType } from './vector-index.ts';
|
||||
import { hnswIndexExpected, hnswMaxDimsForType } from './vector-index.ts';
|
||||
// runMigrations executes while an initialized engine is live. Keep its helper
|
||||
// modules in the static graph rather than importing them from async handlers.
|
||||
import {
|
||||
@@ -5618,6 +5618,86 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON take_proposals (source_id, page_slug, content_hash, prompt_version, md5(claim_text));
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 126,
|
||||
name: 'takes_embedding_active_dims',
|
||||
// #2089: takes.embedding shipped in v37 hardcoded to VECTOR(1536). The
|
||||
// takes table is created by migration, NOT by schema.sql, so the
|
||||
// init-time embedding-dim templating never reached it — brains embedded
|
||||
// at any other width (Voyage 1024, ZE 1280, ...) got a column whose
|
||||
// dimension can never match their vectors. This went unnoticed because
|
||||
// the column also had NO WRITER anywhere in the codebase: nothing ever
|
||||
// ran `UPDATE takes SET embedding = ...`, so `searchTakesVector` (and
|
||||
// think's takes_vec arm) was structurally dead on every brain.
|
||||
//
|
||||
// Rebuild the column at the brain's configured dim. Provably lossless:
|
||||
// since no writer ever existed, there are no real embeddings to lose —
|
||||
// the defensive NULLing below only clears hand-written strays, which are
|
||||
// unusable at the corrected width anyway. The same-release embed lane
|
||||
// (`gbrain embed --stale`) backfills every active claim afterwards.
|
||||
//
|
||||
// Idempotent: when the column already has the target dim, this no-ops.
|
||||
// Runs identically on both engines (v37's PGLite variant diverges only
|
||||
// in the RLS DO-block; the column + index DDL is byte-identical).
|
||||
idempotent: true,
|
||||
sql: '',
|
||||
handler: async (engine: BrainEngine) => {
|
||||
// Step 1: resolve the target dim from config (exact v40 pattern):
|
||||
// default 1536, accept only finite integers in 1..4096.
|
||||
let embeddingDim = 1536;
|
||||
try {
|
||||
const dimRows = await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
);
|
||||
if (dimRows.length > 0) {
|
||||
const parsed = parseInt(dimRows[0].value, 10);
|
||||
if (Number.isFinite(parsed) && parsed > 0 && parsed <= 4096) {
|
||||
embeddingDim = parsed;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// No config row yet — fall back to default (v40 precedent).
|
||||
}
|
||||
|
||||
// Step 2: read the CURRENT dim. For pgvector columns atttypmod IS the
|
||||
// dimension (no varlena header offset like varchar). -1 / missing /
|
||||
// non-positive → unknown → proceed with the rebuild.
|
||||
let currentDim: number | null = null;
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ atttypmod: number }>(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
|
||||
);
|
||||
const raw = rows.length > 0 ? Number(rows[0].atttypmod) : NaN;
|
||||
currentDim = Number.isFinite(raw) && raw > 0 ? raw : null;
|
||||
} catch {
|
||||
currentDim = null; // probe failed → unknown → rebuild
|
||||
}
|
||||
|
||||
if (currentDim === embeddingDim) return; // already correct — no-op
|
||||
|
||||
// Step 3: rebuild. embeddingDim is a validated integer (never
|
||||
// string-interpolate unvalidated input into DDL). HNSW recreation is
|
||||
// gated by the #1734 dim cap; index DDL mirrors v37 verbatim,
|
||||
// including the partial WHERE clause.
|
||||
const recreateIndexSql = hnswIndexExpected('vector', embeddingDim)
|
||||
? `CREATE INDEX IF NOT EXISTS idx_takes_embedding_hnsw ON takes
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WHERE active AND embedding IS NOT NULL;`
|
||||
: `-- idx_takes_embedding_hnsw skipped: pgvector HNSW vector indexes support
|
||||
-- at most ${hnswMaxDimsForType('vector')} dimensions; exact vector scans remain available.`;
|
||||
await engine.runMigration(126, `
|
||||
UPDATE takes SET embedding = NULL, embedded_at = NULL WHERE embedding IS NOT NULL;
|
||||
DROP INDEX IF EXISTS idx_takes_embedding_hnsw;
|
||||
ALTER TABLE takes DROP COLUMN embedding;
|
||||
ALTER TABLE takes ADD COLUMN embedding VECTOR(${embeddingDim});
|
||||
${recreateIndexSql}
|
||||
`);
|
||||
process.stderr.write(
|
||||
` v126: takes.embedding rebuilt ${currentDim === null ? '(unknown)' : `VECTOR(${currentDim})`} → VECTOR(${embeddingDim}) (#2089; column had no writer, rebuild is lossless)\n`,
|
||||
);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+27
-3
@@ -2287,7 +2287,7 @@ const think: Operation = {
|
||||
},
|
||||
mutating: true,
|
||||
handler: async (ctx, p) => {
|
||||
const remote = ctx.remote ?? true;
|
||||
const remote = ctx.remote !== false; // fail-closed: anything not strictly false is untrusted (CLAUDE.md invariant)
|
||||
// Codex P1 #7 + privacy: remote callers cannot persist via MCP.
|
||||
const safeSave = remote ? false : Boolean(p.save);
|
||||
const safeTake = remote ? false : Boolean(p.take);
|
||||
@@ -2297,7 +2297,7 @@ const think: Operation = {
|
||||
// forwards to findTrajectory. CLI callers don't go through this op
|
||||
// and get default scope + remote=false from runThink's CLI path.
|
||||
const thinkScope = thinkSourceScopeOpts(ctx);
|
||||
const { runThink, persistSynthesis } = await import('./think/index.ts');
|
||||
const { runThink, persistSynthesis, persistThinkTake } = await import('./think/index.ts');
|
||||
const result = await runThink(ctx.engine, {
|
||||
question: String(p.question),
|
||||
anchor: p.anchor ? String(p.anchor) : undefined,
|
||||
@@ -2314,18 +2314,40 @@ const think: Operation = {
|
||||
until: p.until ? String(p.until) : undefined,
|
||||
takesHoldersAllowList: ctx.takesHoldersAllowList,
|
||||
...thinkScope,
|
||||
remote: ctx.remote === true,
|
||||
remote: ctx.remote !== false, // fail-closed: anything not strictly false is untrusted (CLAUDE.md invariant)
|
||||
});
|
||||
|
||||
// Persist if --save was passed locally
|
||||
let savedSlug: string | undefined;
|
||||
let evidenceInserted = 0;
|
||||
let takeRow: number | null = null;
|
||||
let takeInserted = 0;
|
||||
if (safeSave) {
|
||||
const persisted = await persistSynthesis(ctx.engine, result);
|
||||
savedSlug = persisted.slug;
|
||||
evidenceInserted = persisted.evidenceInserted;
|
||||
for (const w of persisted.warnings) result.warnings.push(w);
|
||||
}
|
||||
// #2556: --take was declared but nothing consumed it — a silent no-op on
|
||||
// both CLI and MCP. Local callers now get the take row appended to the
|
||||
// anchor page; remote callers stay blocked via the safeTake gate above.
|
||||
if (safeTake) {
|
||||
// thinkSourceScopeOpts speaks runThink's dialect (allowedSources);
|
||||
// persistThinkTake's getPage speaks the engine's (sourceIds). Map
|
||||
// explicitly — spreading thinkScope would silently drop the federated
|
||||
// array and unscope the anchor lookup.
|
||||
const persistedTake = await persistThinkTake(ctx.engine, result, {
|
||||
anchor: p.anchor ? String(p.anchor) : undefined,
|
||||
...(thinkScope.allowedSources !== undefined
|
||||
? { sourceIds: thinkScope.allowedSources }
|
||||
: thinkScope.sourceId !== undefined
|
||||
? { sourceId: thinkScope.sourceId }
|
||||
: {}),
|
||||
});
|
||||
takeRow = persistedTake.rowNum;
|
||||
takeInserted = persistedTake.inserted;
|
||||
for (const w of persistedTake.warnings) result.warnings.push(w);
|
||||
}
|
||||
|
||||
return {
|
||||
...result,
|
||||
@@ -2333,6 +2355,8 @@ const think: Operation = {
|
||||
// falsy) to null so callers never see an empty-string "slug".
|
||||
saved_slug: savedSlug || null,
|
||||
evidence_inserted: evidenceInserted,
|
||||
take_row: takeRow,
|
||||
take_inserted: takeInserted,
|
||||
remote_persisted_blocked: remote && (Boolean(p.save) || Boolean(p.take)),
|
||||
};
|
||||
},
|
||||
|
||||
@@ -2767,6 +2767,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
RETURNING cc.page_id`,
|
||||
params,
|
||||
);
|
||||
// Takes carry TEXT-embedding-space vectors keyed off the same pages, so a
|
||||
// same-dim provider swap (no schema transition to drop the column) must
|
||||
// re-stale them too — old-space take vectors must never score against
|
||||
// new-space queries. Scoped by the SAME page-signature predicate as the
|
||||
// chunks UPDATE above: routine `embed --stale` runs (where nothing
|
||||
// drifted) touch zero takes, so there is no re-embed churn. The `embed
|
||||
// --stale` takes lane picks the NULLed rows up via `embedding IS NULL`.
|
||||
// Return value stays the CHUNK count (existing contract).
|
||||
await this.db.query(
|
||||
`UPDATE takes t
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE t.page_id = p.id
|
||||
AND t.embedding IS NOT NULL
|
||||
AND ${sigClause}${srcClause}`,
|
||||
params,
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
@@ -5225,25 +5242,63 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return out;
|
||||
}
|
||||
|
||||
async countStaleTakes(): Promise<number> {
|
||||
async countStaleTakes(opts?: { sourceId?: string }): Promise<number> {
|
||||
// Source scope goes through the page join (takes has no source_id of its
|
||||
// own) so `embed --stale --source X` only counts X's takes.
|
||||
const srcClause = opts?.sourceId !== undefined ? ` AND p.source_id = $1` : '';
|
||||
const params = opts?.sourceId !== undefined ? [opts.sourceId] : [];
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count FROM takes WHERE active AND embedding IS NULL`
|
||||
`SELECT count(*)::int AS count
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.active AND t.embedding IS NULL${srcClause}`,
|
||||
params,
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async listStaleTakes(): Promise<StaleTakeRow[]> {
|
||||
async listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]> {
|
||||
const srcClause = opts?.sourceId !== undefined ? ` AND p.source_id = $1` : '';
|
||||
const params = opts?.sourceId !== undefined ? [opts.sourceId] : [];
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT t.id AS take_id, p.slug AS page_slug, t.row_num, t.claim
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.active AND t.embedding IS NULL
|
||||
WHERE t.active AND t.embedding IS NULL${srcClause}
|
||||
ORDER BY t.id
|
||||
LIMIT 100000`
|
||||
LIMIT 100000`,
|
||||
params,
|
||||
);
|
||||
return rows as unknown as StaleTakeRow[];
|
||||
}
|
||||
|
||||
async updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number> {
|
||||
if (rows.length === 0) return 0;
|
||||
// Wave-4: batch primitive → batchRetry, matching the addTakesBatch
|
||||
// precedent above so a connection blip mid-backfill retries + audits
|
||||
// instead of failing the whole takes lane batch.
|
||||
return this.batchRetry(opts?.auditSite ?? 'updateTakeEmbeddingsBatch', opts?.signal, () => this._updateTakeEmbeddingsBatchOnce(rows), rows.length);
|
||||
}
|
||||
|
||||
private async _updateTakeEmbeddingsBatchOnce(rows: Array<{ take_id: number; embedding: Float32Array }>): Promise<number> {
|
||||
let updated = 0;
|
||||
for (const r of rows) {
|
||||
// pgvector text literal (searchTakesVector precedent). `AND active`
|
||||
// guards the supersede race: a claim edited mid-embed retires this row
|
||||
// and re-stales as a NEW row — the old row must not get the vector.
|
||||
const vec = `[${Array.from(r.embedding).join(',')}]`;
|
||||
const { rows: ret } = await this.db.query(
|
||||
`UPDATE takes
|
||||
SET embedding = $1::vector, embedded_at = now()
|
||||
WHERE id = $2 AND active
|
||||
RETURNING id`,
|
||||
[vec, r.take_id],
|
||||
);
|
||||
updated += (ret as unknown[]).length;
|
||||
}
|
||||
return updated;
|
||||
}
|
||||
|
||||
async updateTake(
|
||||
pageId: number,
|
||||
rowNum: number,
|
||||
|
||||
@@ -2719,6 +2719,23 @@ export class PostgresEngine implements BrainEngine {
|
||||
RETURNING cc.page_id`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
// Takes carry TEXT-embedding-space vectors keyed off the same pages, so a
|
||||
// same-dim provider swap (no schema transition to drop the column) must
|
||||
// re-stale them too — old-space take vectors must never score against
|
||||
// new-space queries. Scoped by the SAME page-signature predicate as the
|
||||
// chunks UPDATE above: routine `embed --stale` runs (where nothing
|
||||
// drifted) touch zero takes, so there is no re-embed churn. The `embed
|
||||
// --stale` takes lane picks the NULLed rows up via `embedding IS NULL`.
|
||||
// Return value stays the CHUNK count (existing contract).
|
||||
await this.sql.unsafe(
|
||||
`UPDATE takes t
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE t.page_id = p.id
|
||||
AND t.embedding IS NOT NULL
|
||||
AND ${sigClause}${srcClause}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
@@ -5155,27 +5172,61 @@ export class PostgresEngine implements BrainEngine {
|
||||
return out;
|
||||
}
|
||||
|
||||
async countStaleTakes(): Promise<number> {
|
||||
async countStaleTakes(opts?: { sourceId?: string }): Promise<number> {
|
||||
// Source scope goes through the page join (takes has no source_id of its
|
||||
// own) so `embed --stale --source X` only counts X's takes.
|
||||
const sql = this.sql;
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count FROM takes WHERE active AND embedding IS NULL
|
||||
SELECT count(*)::int AS count
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.active AND t.embedding IS NULL
|
||||
${opts?.sourceId !== undefined ? sql`AND p.source_id = ${opts.sourceId}` : sql``}
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async listStaleTakes(): Promise<StaleTakeRow[]> {
|
||||
async listStaleTakes(opts?: { sourceId?: string }): Promise<StaleTakeRow[]> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT t.id AS take_id, p.slug AS page_slug, t.row_num, t.claim
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.active AND t.embedding IS NULL
|
||||
${opts?.sourceId !== undefined ? sql`AND p.source_id = ${opts.sourceId}` : sql``}
|
||||
ORDER BY t.id
|
||||
LIMIT 100000
|
||||
`;
|
||||
return rows as unknown as StaleTakeRow[];
|
||||
}
|
||||
|
||||
async updateTakeEmbeddingsBatch(rows: Array<{ take_id: number; embedding: Float32Array }>, opts?: BatchOpts): Promise<number> {
|
||||
if (rows.length === 0) return 0;
|
||||
// Wave-4: batch primitive → batchRetry, matching the addTakesBatch
|
||||
// precedent so a connection blip mid-backfill retries + audits
|
||||
// instead of failing the whole takes lane batch.
|
||||
return this.batchRetry(opts?.auditSite ?? 'updateTakeEmbeddingsBatch', opts?.signal, () => this._updateTakeEmbeddingsBatchOnce(rows), rows.length);
|
||||
}
|
||||
|
||||
private async _updateTakeEmbeddingsBatchOnce(rows: Array<{ take_id: number; embedding: Float32Array }>): Promise<number> {
|
||||
const sql = this.sql;
|
||||
let updated = 0;
|
||||
for (const r of rows) {
|
||||
// pgvector text literal (searchTakesVector precedent). `AND active`
|
||||
// guards the supersede race: a claim edited mid-embed retires this row
|
||||
// and re-stales as a NEW row — the old row must not get the vector.
|
||||
const vec = `[${Array.from(r.embedding).join(',')}]`;
|
||||
const res = await sql`
|
||||
UPDATE takes
|
||||
SET embedding = ${vec}::vector, embedded_at = now()
|
||||
WHERE id = ${r.take_id} AND active
|
||||
RETURNING id
|
||||
`;
|
||||
updated += (res as unknown[]).length;
|
||||
}
|
||||
return updated;
|
||||
}
|
||||
|
||||
async updateTake(
|
||||
pageId: number,
|
||||
rowNum: number,
|
||||
|
||||
@@ -601,17 +601,22 @@ export async function runSchemaTransition(engine: BrainEngine, targetDim: number
|
||||
);
|
||||
}
|
||||
|
||||
// #3390: the OTHER two dim-pinned columns that carry TEXT-embedding-space
|
||||
// vectors. Both are created at brain-birth width (migrate.ts v55 for
|
||||
// query_cache, v42 for facts) and NO migration ever ALTERs them, so before
|
||||
// this fix a dimension change left them at the old width:
|
||||
// #3390: the OTHER dim-pinned columns that carry TEXT-embedding-space
|
||||
// vectors. All are created at brain-birth width (migrate.ts v55 for
|
||||
// query_cache, v42 for facts, v37/v126 for takes) and no LATER migration
|
||||
// ever ALTERs them, so before this fix a dimension change left them at
|
||||
// the old width:
|
||||
// - query_cache.embedding stayed narrow → every store() AND lookup()
|
||||
// silently swallowed the width error (by design, so the cache can
|
||||
// never break search), i.e. a PERMANENT 0% hit rate.
|
||||
// - facts.embedding stayed narrow → every per-fact embed write failed
|
||||
// ($N::vector into the old width), and the doctor check that would
|
||||
// warn is skipped on PGLite (the DEFAULT engine).
|
||||
// Both are text-embedding-space columns, so they MUST move with
|
||||
// - takes.embedding stayed narrow → the #2089 embed takes lane failed
|
||||
// every write forever while still paying the gateway each run
|
||||
// (migration v126 only fixes the dim at migration-run time; a LATER
|
||||
// `gbrain migrate embeddings --to <dim>` has to move it here).
|
||||
// All are text-embedding-space columns, so they MUST move with
|
||||
// content_chunks.embedding. The image/multimodal columns above are the
|
||||
// deliberate exception (separate models, independent dims).
|
||||
for (const t of TEXT_EMBEDDING_DIM_PINNED_TABLES) {
|
||||
@@ -647,6 +652,17 @@ const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
|
||||
ON facts USING hnsw (embedding ${opclass})
|
||||
WHERE embedding IS NOT NULL AND expired_at IS NULL`,
|
||||
},
|
||||
{
|
||||
// Partial predicate mirrors migrate.ts v37 (and the v126 rebuild)
|
||||
// verbatim. hnswIndexExpected in transitionDimPinnedColumn carries the
|
||||
// #1734 dim-cap gate, matching v126's recreateIndexSql behavior.
|
||||
table: 'takes',
|
||||
index: 'idx_takes_embedding_hnsw',
|
||||
indexSql: (opclass) =>
|
||||
`CREATE INDEX IF NOT EXISTS idx_takes_embedding_hnsw
|
||||
ON takes USING hnsw (embedding ${opclass})
|
||||
WHERE active AND embedding IS NOT NULL`,
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
@@ -658,7 +674,8 @@ const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
|
||||
* Dropping the column discards the stored vectors, which is correct: they are
|
||||
* in the OLD embedding space and unusable after the swap. query_cache is a
|
||||
* cache (refills on the next query); facts re-embed on their next write /
|
||||
* `gbrain extract` pass.
|
||||
* `gbrain extract` pass; takes re-embed via the `gbrain embed --stale` takes
|
||||
* lane (staleness predicate is `embedding IS NULL`, which the drop satisfies).
|
||||
*/
|
||||
async function transitionDimPinnedColumn(
|
||||
tx: { executeRaw: <T = unknown>(sql: string, params?: unknown[]) => Promise<T[]> },
|
||||
|
||||
@@ -77,6 +77,7 @@ export const BATCH_AUDIT_SITES = [
|
||||
'addLinksBatch',
|
||||
'addTimelineEntriesBatch',
|
||||
'addTakesBatch',
|
||||
'updateTakeEmbeddingsBatch',
|
||||
'upsertChunks',
|
||||
// extract.ts per-site labels.
|
||||
'extract.links_inc',
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
/**
|
||||
* Shared dual-plane take append (Wave-4 review fix).
|
||||
*
|
||||
* Markdown is the source of truth for takes: `gbrain takes add` allocates
|
||||
* row numbers from the FILE fence (upsertTakeRow → max fence rowNum + 1),
|
||||
* then mirrors to the DB via addTakesBatch's ON CONFLICT (page_id, row_num)
|
||||
* DO UPDATE. Any writer that appends DB-only rows (the original
|
||||
* persistThinkTake did) allocates from DB MAX(row_num) instead — and the
|
||||
* next fence-allocated row at the same number deterministically OVERWRITES
|
||||
* it. This module is the single append path both surfaces call, so the two
|
||||
* planes cannot drift:
|
||||
*
|
||||
* withPageLock(slug)
|
||||
* → resolve the page (scoped: sourceIds > sourceId > unscoped)
|
||||
* → resolve the brain dir (explicit override > sync.repo_path config)
|
||||
* → file plane: upsertTakeRow allocates the row number (file-first)
|
||||
* → writeBody
|
||||
* → DB plane: addTakesBatch mirror at the fence-allocated row number
|
||||
*
|
||||
* Fallback posture (headless / DB-only brains): when no brain dir resolves,
|
||||
* the row is allocated DB-only (MAX(row_num)+1) and the
|
||||
* 'TAKE_FILE_PLANE_UNAVAILABLE' warning is returned — documented posture:
|
||||
* DB-only rows can be renumbered/overwritten by a later fence reconcile.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import type { BrainEngine, TakeKind } from './engine.ts';
|
||||
import { upsertTakeRow } from './takes-fence.ts';
|
||||
import { withPageLock } from './page-lock.ts';
|
||||
|
||||
/** Warning code returned when the append fell back to DB-only allocation. */
|
||||
export const TAKE_FILE_PLANE_UNAVAILABLE = 'TAKE_FILE_PLANE_UNAVAILABLE';
|
||||
|
||||
/** Thrown when the (scoped) page lookup finds nothing. Callers map this to
|
||||
* their own surface signal (CLI error+exit, TAKE_ANCHOR_NOT_FOUND, ...). */
|
||||
export class TakePageNotFoundError extends Error {
|
||||
readonly slug: string;
|
||||
readonly sourceId?: string;
|
||||
constructor(slug: string, sourceId?: string) {
|
||||
super(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}`);
|
||||
this.name = 'TakePageNotFoundError';
|
||||
this.slug = slug;
|
||||
this.sourceId = sourceId;
|
||||
}
|
||||
}
|
||||
|
||||
export function pageFilePath(brainDir: string, slug: string): string {
|
||||
return join(brainDir, `${slug}.md`);
|
||||
}
|
||||
|
||||
export function readBodyOrEmpty(path: string): string {
|
||||
if (!existsSync(path)) return '';
|
||||
return readFileSync(path, 'utf-8');
|
||||
}
|
||||
|
||||
export function writeBody(path: string, body: string): void {
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
writeFileSync(path, body, 'utf-8');
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the on-disk brain dir for the file plane, or null when none is
|
||||
* available (headless install, unset/dangling sync.repo_path). An explicit
|
||||
* dir that does not exist ALSO returns null — the CLI wraps this with its
|
||||
* own loud error+exit; core callers fall back to DB-only allocation.
|
||||
*/
|
||||
export async function resolveBrainDirOrNull(
|
||||
engine: BrainEngine,
|
||||
explicitDir?: string | null,
|
||||
): Promise<string | null> {
|
||||
if (explicitDir) return existsSync(explicitDir) ? explicitDir : null;
|
||||
try {
|
||||
const configured = await engine.getConfig('sync.repo_path');
|
||||
if (configured && existsSync(configured)) return configured;
|
||||
} catch {
|
||||
// Config read failure → treat as no file plane (fallback posture).
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export interface AppendTakeInput {
|
||||
slug: string;
|
||||
/** Scalar source scope (sourceIds wins when both set — sourceScopeOpts precedence). */
|
||||
sourceId?: string;
|
||||
/** Federated source scope for the page lookup. */
|
||||
sourceIds?: string[];
|
||||
claim: string;
|
||||
kind: TakeKind;
|
||||
holder: string;
|
||||
weight?: number;
|
||||
source?: string;
|
||||
sinceDate?: string;
|
||||
/**
|
||||
* Pre-resolved brain dir (CLI --dir, or a caller that already validated
|
||||
* sync.repo_path). When omitted, resolves via resolveBrainDirOrNull; when
|
||||
* that yields nothing, the append falls back to DB-only allocation with
|
||||
* the TAKE_FILE_PLANE_UNAVAILABLE warning.
|
||||
*/
|
||||
brainDir?: string;
|
||||
}
|
||||
|
||||
export interface AppendTakeResult {
|
||||
rowNum: number;
|
||||
inserted: number;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Append one take row to a page on BOTH planes (file fence + DB), serialized
|
||||
* under withPageLock(slug) so concurrent appends can't race the allocation.
|
||||
*
|
||||
* The page lookup runs FIRST (inside the lock, before any file write) so a
|
||||
* missing page throws {@link TakePageNotFoundError} without leaving a
|
||||
* half-written fence behind.
|
||||
*/
|
||||
export async function appendTake(
|
||||
engine: BrainEngine,
|
||||
input: AppendTakeInput,
|
||||
): Promise<AppendTakeResult> {
|
||||
const warnings: string[] = [];
|
||||
return withPageLock(input.slug, async () => {
|
||||
// Scoped page lookup: federated array > scalar > unscoped.
|
||||
const pageOpts: { sourceId?: string; sourceIds?: string[] } = {};
|
||||
if (input.sourceIds !== undefined) pageOpts.sourceIds = input.sourceIds;
|
||||
else if (input.sourceId !== undefined) pageOpts.sourceId = input.sourceId;
|
||||
const page = await engine.getPage(input.slug, pageOpts);
|
||||
if (!page) throw new TakePageNotFoundError(input.slug, input.sourceId);
|
||||
|
||||
const brainDir = input.brainDir ?? await resolveBrainDirOrNull(engine);
|
||||
|
||||
let rowNum: number;
|
||||
if (brainDir) {
|
||||
// File plane first: the fence allocates the row number.
|
||||
const path = pageFilePath(brainDir, input.slug);
|
||||
const body = readBodyOrEmpty(path);
|
||||
const up = upsertTakeRow(body, {
|
||||
claim: input.claim,
|
||||
kind: input.kind,
|
||||
holder: input.holder,
|
||||
weight: input.weight ?? 0.5,
|
||||
source: input.source,
|
||||
sinceDate: input.sinceDate,
|
||||
active: true,
|
||||
});
|
||||
writeBody(path, up.body);
|
||||
rowNum = up.rowNum;
|
||||
} else {
|
||||
// DB-only fallback (no file plane). MAX(row_num)+1 under the same page
|
||||
// lock. A later fence reconcile may renumber/overwrite these rows —
|
||||
// documented posture for headless brains.
|
||||
warnings.push(TAKE_FILE_PLANE_UNAVAILABLE);
|
||||
const rows = await engine.executeRaw<{ next: number | string }>(
|
||||
`SELECT (COALESCE(MAX(row_num), 0) + 1)::int AS next FROM takes WHERE page_id = $1`,
|
||||
[page.id],
|
||||
);
|
||||
rowNum = Number(rows[0]?.next ?? 1);
|
||||
}
|
||||
|
||||
const inserted = await engine.addTakesBatch([{
|
||||
page_id: page.id,
|
||||
row_num: rowNum,
|
||||
claim: input.claim,
|
||||
kind: input.kind,
|
||||
holder: input.holder,
|
||||
weight: input.weight ?? 0.5,
|
||||
since_date: input.sinceDate,
|
||||
source: input.source,
|
||||
active: true,
|
||||
superseded_by: null,
|
||||
}]);
|
||||
|
||||
return { rowNum, inserted, warnings };
|
||||
});
|
||||
}
|
||||
@@ -657,6 +657,106 @@ export async function persistSynthesis(
|
||||
return { slug, evidenceInserted: persisted.inserted, warnings: persisted.warnings };
|
||||
}
|
||||
|
||||
export interface PersistThinkTakeOpts {
|
||||
anchor?: string;
|
||||
sourceId?: string;
|
||||
sourceIds?: string[];
|
||||
}
|
||||
|
||||
export interface PersistThinkTakeResult {
|
||||
rowNum: number | null;
|
||||
inserted: number;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Cap on the persisted think-take claim. Rationale: the prompt renderer
|
||||
* (sanitize.ts:sanitizeTakeForPrompt) truncates claims at 500 chars anyway,
|
||||
* and fence cells are single-line — a multi-page synthesis answer stored
|
||||
* verbatim as one claim bloats the fence + DB for content nothing ever
|
||||
* reads past. 2000 keeps ample headroom over the 500-char read while
|
||||
* bounding the cell.
|
||||
*/
|
||||
export const THINK_TAKE_CLAIM_MAX_CHARS = 2000;
|
||||
|
||||
/**
|
||||
* Persist `gbrain think --take` as the next append-only take row on the
|
||||
* anchor page (#2556 — the flag was declared but nothing consumed it, so
|
||||
* `think --take` silently wrote nothing on both the CLI and MCP paths).
|
||||
*
|
||||
* The synthesis answer is the claim; holder='brain' because this is
|
||||
* gbrain's own analysis; kind='take' (KIND_VALUES member). Dual-plane
|
||||
* write via the shared appendTake helper (takes-append.ts) — the SAME path
|
||||
* `gbrain takes add` uses: withPageLock(anchor slug) → fence allocates the
|
||||
* row number from the FILE (markdown is the source of truth) → writeBody →
|
||||
* addTakesBatch mirror. When no brain dir resolves (headless / DB-only
|
||||
* brains), appendTake falls back to DB-only MAX(row_num)+1 allocation and
|
||||
* surfaces TAKE_FILE_PLANE_UNAVAILABLE (documented posture: DB-only rows
|
||||
* can be renumbered/overwritten by a later fence reconcile).
|
||||
*
|
||||
* The claim is flattened to a single line (fence cells are single-line;
|
||||
* newlines → ' ') and truncated at THINK_TAKE_CLAIM_MAX_CHARS with the
|
||||
* TAKE_CLAIM_TRUNCATED warning.
|
||||
*
|
||||
* Signals (never throws for expected shapes, mirroring persistSynthesis):
|
||||
* - TAKE_REQUIRES_ANCHOR — no anchor given.
|
||||
* - TAKE_EMPTY_NOT_PERSISTED — synthesis failed or empty answer.
|
||||
* - `TAKE_ANCHOR_NOT_FOUND: <s>` (space after the colon) — anchor page
|
||||
* absent in the caller's scope
|
||||
* (scope resolved via the sourceIds > sourceId precedence, matching
|
||||
* sourceScopeOpts). Remote callers never reach here — the op handler's
|
||||
* fail-closed gate zeroes `take` for them.
|
||||
* - TAKE_CLAIM_TRUNCATED — claim exceeded the cap (row still written).
|
||||
* - TAKE_FILE_PLANE_UNAVAILABLE — DB-only fallback fired (row still written).
|
||||
*
|
||||
* Design provenance: community PR #2618 (its successor #3164 was declined
|
||||
* for bundled feature scope; the maintainer confirmed this half is a
|
||||
* legitimate fix). Re-implemented with the page-lock serialization added;
|
||||
* Wave-4 review moved the write onto the dual-plane appendTake path.
|
||||
*/
|
||||
export async function persistThinkTake(
|
||||
engine: BrainEngine,
|
||||
result: ThinkResult,
|
||||
opts: PersistThinkTakeOpts,
|
||||
): Promise<PersistThinkTakeResult> {
|
||||
const anchor = opts.anchor?.trim();
|
||||
if (!anchor) {
|
||||
return { rowNum: null, inserted: 0, warnings: ['TAKE_REQUIRES_ANCHOR'] };
|
||||
}
|
||||
if (result.synthesisOk === false || result.answer.trim().length === 0) {
|
||||
return { rowNum: null, inserted: 0, warnings: ['TAKE_EMPTY_NOT_PERSISTED'] };
|
||||
}
|
||||
|
||||
const warnings: string[] = [];
|
||||
// Fence cells are single-line: flatten first, then bound the length.
|
||||
let claim = result.answer.trim().replace(/\s*\r?\n\s*/g, ' ');
|
||||
if (claim.length > THINK_TAKE_CLAIM_MAX_CHARS) {
|
||||
claim = claim.slice(0, THINK_TAKE_CLAIM_MAX_CHARS);
|
||||
warnings.push('TAKE_CLAIM_TRUNCATED');
|
||||
}
|
||||
|
||||
const { appendTake, TakePageNotFoundError } = await import('../takes-append.ts');
|
||||
try {
|
||||
const appended = await appendTake(engine, {
|
||||
slug: anchor,
|
||||
...(opts.sourceIds !== undefined ? { sourceIds: opts.sourceIds } : {}),
|
||||
...(opts.sourceId !== undefined ? { sourceId: opts.sourceId } : {}),
|
||||
claim,
|
||||
kind: 'take',
|
||||
holder: 'brain',
|
||||
weight: 0.5,
|
||||
source: 'gbrain think',
|
||||
});
|
||||
warnings.push(...appended.warnings);
|
||||
return { rowNum: appended.rowNum, inserted: appended.inserted, warnings };
|
||||
} catch (e) {
|
||||
if (e instanceof TakePageNotFoundError) {
|
||||
return { rowNum: null, inserted: 0, warnings: [`TAKE_ANCHOR_NOT_FOUND: ${anchor}`] };
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Gateway adapter for #952 (think over MCP returns "no LLM available").
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -339,7 +339,7 @@ describe('BATCH_AUDIT_SITES typed enum + isBatchAuditSite guard (D10c codex)', (
|
||||
// Pin the set so a future "cleanup" PR can't silently drop a site and
|
||||
// break audit-attribution for the corresponding caller.
|
||||
const expected = new Set([
|
||||
'addLinksBatch', 'addTimelineEntriesBatch', 'addTakesBatch', 'upsertChunks',
|
||||
'addLinksBatch', 'addTimelineEntriesBatch', 'addTakesBatch', 'updateTakeEmbeddingsBatch', 'upsertChunks',
|
||||
'extract.links_inc', 'extract.timeline_inc',
|
||||
'extract.links_fs', 'extract.timeline_fs',
|
||||
'extract.links_db', 'extract.timeline_db',
|
||||
|
||||
@@ -732,6 +732,47 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
expect(slugs.indexOf('ep/ec-bob')).toBeLessThan(slugs.indexOf('companies/ec-widget'));
|
||||
expect(pg.find((r) => r.slug === 'ep/ec-alice')!.inbound_count).toBe(2);
|
||||
});
|
||||
|
||||
// #2089: updateTakeEmbeddingsBatch — the takes-embedding writer. Both
|
||||
// engines must (a) return 0 on empty input, (b) report the same
|
||||
// actually-updated count, (c) round-trip through getTakeEmbeddings, and
|
||||
// (d) surface the row via searchTakesVector afterwards. A drift here
|
||||
// means `gbrain embed --stale` silently under/over-counts on one engine.
|
||||
test('updateTakeEmbeddingsBatch parity: write → read-back → vector-search on both engines', async () => {
|
||||
const slug = 'concepts/fat-code-thin-harness';
|
||||
const CLAIM_A = 'takes-parity claim alpha';
|
||||
const CLAIM_B = 'takes-parity claim beta';
|
||||
for (const eng of [pgEngine, pgliteEngine]) {
|
||||
// Empty input → 0 without touching the DB.
|
||||
expect(await eng.updateTakeEmbeddingsBatch([])).toBe(0);
|
||||
|
||||
const page = await eng.getPage(slug);
|
||||
expect(page).not.toBeNull();
|
||||
await eng.addTakesBatch([
|
||||
{ page_id: page!.id, row_num: 90, claim: CLAIM_A, kind: 'take', holder: 'world', weight: 0.5 },
|
||||
{ page_id: page!.id, row_num: 91, claim: CLAIM_B, kind: 'take', holder: 'world', weight: 0.5 },
|
||||
]);
|
||||
|
||||
const stale = (await eng.listStaleTakes()).filter(t => t.claim.startsWith('takes-parity claim'));
|
||||
expect(stale.length).toBe(2);
|
||||
|
||||
const updated = await eng.updateTakeEmbeddingsBatch(
|
||||
stale.map((t, i) => ({ take_id: Number(t.take_id), embedding: basisEmbedding(i + 200) })),
|
||||
);
|
||||
expect(updated).toBe(2);
|
||||
|
||||
const back = await eng.getTakeEmbeddings(stale.map(t => Number(t.take_id)));
|
||||
expect(back.size).toBe(2);
|
||||
|
||||
// Vector search finds the exact-match claim (cosine similarity 1).
|
||||
const hits = await eng.searchTakesVector(basisEmbedding(200), { limit: 5 });
|
||||
expect(hits.map(h => h.claim)).toContain(stale[0].claim);
|
||||
|
||||
// The written rows left the stale pool on this engine.
|
||||
const staleAfter = (await eng.listStaleTakes()).filter(t => t.claim.startsWith('takes-parity claim'));
|
||||
expect(staleAfter.length).toBe(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── relationalFanout parity (v0.43) ─────────────────────────────────────
|
||||
|
||||
@@ -174,12 +174,14 @@ d('embedding migration (live Postgres + pgvector)', () => {
|
||||
expect(persisted).toEqual([[toModel, targetDims]]);
|
||||
expect(await columnDims()).toBe(targetDims);
|
||||
|
||||
// All THREE dim-pinned text-embedding-space columns move together on real
|
||||
// pgvector (query_cache + facts are created at brain-birth width and no
|
||||
// migration ever ALTERs them — before the fix they stayed narrow, which
|
||||
// silently killed the query cache and every per-fact embed write).
|
||||
// ALL dim-pinned text-embedding-space columns move together on real
|
||||
// pgvector (query_cache + facts + takes are created at brain-birth width
|
||||
// and no later migration ALTERs them — before the fix they stayed
|
||||
// narrow, which silently killed the query cache, every per-fact embed
|
||||
// write, and every takes-lane embed write).
|
||||
expect(await embeddingColWidth('query_cache')).toBe(targetDims);
|
||||
expect(await embeddingColWidth('facts')).toBe(targetDims);
|
||||
expect(await embeddingColWidth('takes')).toBe(targetDims);
|
||||
// And the rebuilt columns actually ACCEPT a vector at the new width.
|
||||
const nv = `[${new Array(targetDims).fill(0.01).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* #2089 — the takes lane of `gbrain embed --stale` (embedStaleTakes via
|
||||
* runEmbedCore). The wave shipped the lane with ZERO tests driving it: the
|
||||
* engine writer (updateTakeEmbeddingsBatch) is covered, but the lane's
|
||||
* control flow — dry-run counting, id coercion into the writer, and the
|
||||
* budget/abort cutShort skip — was untested.
|
||||
*
|
||||
* Hermetic: mock.module on core/embedding.ts (same pattern as
|
||||
* embed.serial.test.ts) + the gateway embed-transport seam for the creds
|
||||
* preflight. Serial: mock.module + env mutation are process-global.
|
||||
*/
|
||||
import { test, expect, mock, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
let totalEmbedCalls = 0;
|
||||
|
||||
mock.module('../src/core/embedding.ts', () => ({
|
||||
embedBatch: async (texts: string[]) => {
|
||||
totalEmbedCalls++;
|
||||
await new Promise(r => setTimeout(r, 5));
|
||||
return texts.map(() => new Float32Array(1536));
|
||||
},
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
// Import AFTER mocking.
|
||||
const { runEmbedCore } = await import('../src/commands/embed.ts');
|
||||
const { __setEmbedTransportForTests } = await import('../src/core/ai/gateway.ts');
|
||||
__setEmbedTransportForTests(async () => ({ embeddings: [], usage: { tokens: 0 } } as any));
|
||||
|
||||
/** Proxy mock engine (embed.serial.test.ts pattern) with call tracking. */
|
||||
function mockEngine(overrides: Partial<Record<string, any>> = {}): BrainEngine {
|
||||
const calls: { method: string; args: any[] }[] = [];
|
||||
const track = (method: string) => (...args: any[]) => {
|
||||
calls.push({ method, args });
|
||||
if (overrides[method]) return overrides[method](...args);
|
||||
return Promise.resolve(null);
|
||||
};
|
||||
const engine = new Proxy({} as any, {
|
||||
get(_, prop: string) {
|
||||
if (prop === '_calls') return calls;
|
||||
if (overrides[prop]) {
|
||||
return (...args: any[]) => {
|
||||
calls.push({ method: prop, args });
|
||||
return overrides[prop](...args);
|
||||
};
|
||||
}
|
||||
return track(prop);
|
||||
},
|
||||
});
|
||||
return engine;
|
||||
}
|
||||
|
||||
function callCount(engine: BrainEngine, method: string): number {
|
||||
return (engine as any)._calls.filter((c: any) => c.method === method).length;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
totalEmbedCalls = 0;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.GBRAIN_EMBED_TIME_BUDGET_MS;
|
||||
});
|
||||
|
||||
test('#2089 --stale takes lane: dry-run counts only; real run embeds and batch-writes with coerced numeric ids', async () => {
|
||||
// Driver-shaped ids on purpose: string + bigint must arrive at the writer
|
||||
// as plain numbers (the lane's Number() coercion).
|
||||
const staleTakes = [
|
||||
{ take_id: '11' as any, page_slug: 'p/a', row_num: 1, claim: 'claim one' },
|
||||
{ take_id: 12n as any, page_slug: 'p/a', row_num: 2, claim: 'claim two' },
|
||||
{ take_id: 13, page_slug: 'p/b', row_num: 1, claim: 'claim three' },
|
||||
];
|
||||
let written: Array<{ take_id: number; embedding: Float32Array }> = [];
|
||||
const makeEngine = () => mockEngine({
|
||||
countStaleChunks: async () => 0, // pages lane: nothing stale
|
||||
invalidateStaleSignatureEmbeddings: async () => 0,
|
||||
countStaleTakes: async () => staleTakes.length,
|
||||
listStaleTakes: async () => staleTakes,
|
||||
updateTakeEmbeddingsBatch: async (rows: any[]) => { written = rows; return rows.length; },
|
||||
});
|
||||
|
||||
// Dry-run: count-only — no gateway call, no listStaleTakes, no writes.
|
||||
const dryEngine = makeEngine();
|
||||
const dry = await runEmbedCore(dryEngine, { stale: true, dryRun: true, quiet: true });
|
||||
expect(dry.takes_would_embed).toBe(3);
|
||||
expect(dry.takes_embedded ?? 0).toBe(0);
|
||||
expect(totalEmbedCalls).toBe(0);
|
||||
expect(callCount(dryEngine, 'listStaleTakes')).toBe(0);
|
||||
expect(callCount(dryEngine, 'updateTakeEmbeddingsBatch')).toBe(0);
|
||||
expect(written).toHaveLength(0);
|
||||
|
||||
// Real run: 3 claims fit one 64-cap batch → one gateway call, 3 writes.
|
||||
const res = await runEmbedCore(makeEngine(), { stale: true, quiet: true });
|
||||
expect(res.takes_embedded).toBe(3);
|
||||
expect(res.failures).toBe(0);
|
||||
expect(written).toHaveLength(3);
|
||||
expect(written.map(w => w.take_id)).toEqual([11, 12, 13]);
|
||||
for (const w of written) {
|
||||
expect(typeof w.take_id).toBe('number');
|
||||
expect(w.embedding).toBeInstanceOf(Float32Array);
|
||||
expect(w.embedding.length).toBe(1536);
|
||||
}
|
||||
}, 20_000);
|
||||
|
||||
test('Wave-4 source scoping: `--stale --source X` threads sourceId into BOTH takes enumerators', async () => {
|
||||
const enumeratorArgs: Array<{ method: string; opts: unknown }> = [];
|
||||
let written: Array<{ take_id: number; embedding: Float32Array }> = [];
|
||||
const engine = mockEngine({
|
||||
countStaleChunks: async () => 0, // pages lane: nothing stale
|
||||
invalidateStaleSignatureEmbeddings: async () => 0,
|
||||
countStaleTakes: async (opts: unknown) => {
|
||||
enumeratorArgs.push({ method: 'countStaleTakes', opts });
|
||||
return 1;
|
||||
},
|
||||
listStaleTakes: async (opts: unknown) => {
|
||||
enumeratorArgs.push({ method: 'listStaleTakes', opts });
|
||||
return [{ take_id: 21, page_slug: 'p/scoped', row_num: 1, claim: 'scoped claim' }];
|
||||
},
|
||||
updateTakeEmbeddingsBatch: async (rows: any[]) => { written = rows; return rows.length; },
|
||||
});
|
||||
|
||||
const res = await runEmbedCore(engine, { stale: true, quiet: true, sourceId: 'tenant-a' });
|
||||
|
||||
// Both enumerators saw the run's source scope — a `--source X` run must
|
||||
// never enumerate (or pay to embed) another source's takes.
|
||||
expect(enumeratorArgs).toEqual([
|
||||
{ method: 'countStaleTakes', opts: { sourceId: 'tenant-a' } },
|
||||
{ method: 'listStaleTakes', opts: { sourceId: 'tenant-a' } },
|
||||
]);
|
||||
expect(res.takes_embedded).toBe(1);
|
||||
expect(written.map(w => w.take_id)).toEqual([21]);
|
||||
|
||||
// Unscoped run: enumerators get NO scope object (all-source semantics).
|
||||
enumeratorArgs.length = 0;
|
||||
const engine2 = mockEngine({
|
||||
countStaleChunks: async () => 0,
|
||||
invalidateStaleSignatureEmbeddings: async () => 0,
|
||||
countStaleTakes: async (opts: unknown) => {
|
||||
enumeratorArgs.push({ method: 'countStaleTakes', opts });
|
||||
return 0;
|
||||
},
|
||||
});
|
||||
await runEmbedCore(engine2, { stale: true, quiet: true });
|
||||
expect(enumeratorArgs).toEqual([{ method: 'countStaleTakes', opts: undefined }]);
|
||||
}, 20_000);
|
||||
|
||||
test('#2089 cutShort: a wall-clock-budget abort in the pages lane SKIPS the takes lane (next run picks them up)', async () => {
|
||||
// 1ms budget; the pages lane's first page-load sleeps 30ms so the budget
|
||||
// signal is guaranteed to have fired by the time the lane reports back.
|
||||
process.env.GBRAIN_EMBED_TIME_BUDGET_MS = '1';
|
||||
const engine = mockEngine({
|
||||
countStaleChunks: async () => 5, // pass the 0-stale early return
|
||||
invalidateStaleSignatureEmbeddings: async () => 0,
|
||||
listStaleChunks: async () => {
|
||||
await new Promise(r => setTimeout(r, 30));
|
||||
return [];
|
||||
},
|
||||
countStaleTakes: async () => 99, // must never be reached
|
||||
});
|
||||
|
||||
const res = await runEmbedCore(engine, { stale: true, quiet: true });
|
||||
|
||||
// The takes lane never even probed the stale count — a timed-out run must
|
||||
// not start NEW work (#2089 contract: graceful skip, not extra load).
|
||||
expect(callCount(engine, 'countStaleTakes')).toBe(0);
|
||||
expect(res.takes_embedded).toBeUndefined();
|
||||
expect(res.takes_would_embed).toBeUndefined();
|
||||
expect(totalEmbedCalls).toBe(0);
|
||||
}, 20_000);
|
||||
@@ -165,6 +165,54 @@ describe('#3391 includeNullSignature widening', () => {
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(0);
|
||||
expect(await engine.countStaleChunks({ signature: 'new:model:1' })).toBe(0);
|
||||
});
|
||||
|
||||
// Wave-4 review: invalidation also re-stales TAKES on the same pages under
|
||||
// the SAME signature predicate — a same-dim provider swap has no schema
|
||||
// transition to drop takes.embedding, and old-space take vectors must not
|
||||
// score against new-space queries. Routine runs (nothing drifted) touch
|
||||
// zero takes, so there is no re-embed churn.
|
||||
test('invalidation re-stales takes on drifted pages; fresh kept; grandfather lifts with the flag', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null);
|
||||
await seedEmbedded('drifted', 'fghij', 'old:model:1');
|
||||
await seedEmbedded('fresh', 'klmno', 'new:model:1');
|
||||
|
||||
// One EMBEDDED take per page (text-embedding space, same as the chunks).
|
||||
for (const slug of ['legacy', 'drifted', 'fresh']) {
|
||||
const page = await engine.getPage(slug);
|
||||
await engine.addTakesBatch([{
|
||||
page_id: page!.id, row_num: 1, claim: `take on ${slug}`,
|
||||
kind: 'fact', holder: 'world', weight: 0.9,
|
||||
}]);
|
||||
}
|
||||
await engine.executeRaw(
|
||||
`UPDATE takes
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector,
|
||||
embedded_at = now()`,
|
||||
[colDim],
|
||||
);
|
||||
expect(await engine.countStaleTakes()).toBe(0);
|
||||
|
||||
// Default (grandfather): only the drifted page's take re-stales; the
|
||||
// returned count remains CHUNKS-only (existing contract).
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(1);
|
||||
let staleTakes = await engine.listStaleTakes();
|
||||
expect(staleTakes.map(t => t.page_slug)).toEqual(['drifted']);
|
||||
const cleared = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM takes WHERE embedding IS NULL AND embedded_at IS NULL`,
|
||||
);
|
||||
expect(Number(cleared[0]?.n)).toBe(1); // embedded_at cleared alongside
|
||||
|
||||
// Widened (#3391): the legacy (NULL-sig) take re-stales too; fresh kept.
|
||||
await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1', includeNullSignature: true });
|
||||
staleTakes = await engine.listStaleTakes();
|
||||
expect(staleTakes.map(t => t.page_slug).sort()).toEqual(['drifted', 'legacy']);
|
||||
const keptTake = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE p.slug = 'fresh' AND t.embedding IS NOT NULL`,
|
||||
);
|
||||
expect(Number(keptTake[0]?.n)).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('planEmbeddingMigration', () => {
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* Migration v126 (#2089) — takes_embedding_active_dims.
|
||||
*
|
||||
* takes.embedding shipped in v37 hardcoded to VECTOR(1536); the takes table
|
||||
* is migration-created (absent from schema.sql), so init-time dim templating
|
||||
* never reached it. On a brain configured for any other width the column
|
||||
* could never hold a real vector. v126 rebuilds it at the active dim.
|
||||
*
|
||||
* Pins (on a PGLite brain init'd at a NON-default dim):
|
||||
* - after migrations, atttypmod (which IS the pgvector dimension) shows
|
||||
* the active dim, not 1536
|
||||
* - the HNSW partial index is recreated (dim under the #1734 cap)
|
||||
* - idempotent: forcing a re-run of v126 no-ops (column shape unchanged)
|
||||
* - the writer works end-to-end at the new width:
|
||||
* updateTakeEmbeddingsBatch persists + searchTakesVector returns the row
|
||||
*
|
||||
* Serial: configureGateway mutates process-global gateway state (the file
|
||||
* needs a non-default dim for its whole lifetime).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { runMigrations } from '../src/core/migrate.ts';
|
||||
|
||||
const DIMS = 1024; // non-default (legacy test baseline is 1536; prod default 1280)
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let pageId: number;
|
||||
|
||||
async function takesEmbeddingDim(): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ atttypmod: number }>(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
return Number(rows[0].atttypmod);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'litellm:custom-1024d',
|
||||
embedding_dimensions: DIMS,
|
||||
env: { ...process.env },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
const p = await engine.putPage('companies/acme-example', {
|
||||
title: 'Acme Example',
|
||||
type: 'company' as const,
|
||||
compiled_truth: '## Takes\n\nAcme is a widget company.\n',
|
||||
});
|
||||
pageId = p.id;
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: pageId, row_num: 1, claim: 'dims-test claim at 1024', kind: 'fact', holder: 'world', weight: 0.9 },
|
||||
]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('migration v126 takes.embedding dim rebuild', () => {
|
||||
test('column rebuilt at the active dim, not v37 hardcoded 1536', async () => {
|
||||
// Sanity: the brain really is configured at the non-default dim.
|
||||
const cfg = await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
);
|
||||
expect(parseInt(cfg[0]?.value ?? '0', 10)).toBe(DIMS);
|
||||
// For pgvector columns atttypmod IS the dimension.
|
||||
expect(await takesEmbeddingDim()).toBe(DIMS);
|
||||
});
|
||||
|
||||
test('HNSW partial index recreated (dim within the #1734 cap)', async () => {
|
||||
const rows = await engine.executeRaw<{ indexdef: string }>(
|
||||
`SELECT indexdef FROM pg_indexes
|
||||
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].indexdef).toContain('hnsw');
|
||||
expect(rows[0].indexdef).toContain('vector_cosine_ops');
|
||||
// v37's partial predicate preserved.
|
||||
expect(rows[0].indexdef).toMatch(/WHERE\s+\(?active/i);
|
||||
});
|
||||
|
||||
test('idempotent: forced re-run of v126 no-ops on a correct column', async () => {
|
||||
await engine.setConfig('version', '125');
|
||||
const { applied } = await runMigrations(engine);
|
||||
expect(applied).toBeGreaterThanOrEqual(1); // v126 re-stamped
|
||||
expect(await takesEmbeddingDim()).toBe(DIMS);
|
||||
const idx = await engine.executeRaw<{ exists: boolean }>(
|
||||
`SELECT EXISTS (
|
||||
SELECT 1 FROM pg_indexes
|
||||
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'
|
||||
) AS exists`,
|
||||
);
|
||||
expect(idx[0]?.exists).toBe(true);
|
||||
});
|
||||
|
||||
test('writer works end-to-end at the new width: batch write + vector search', async () => {
|
||||
const stale = await engine.listStaleTakes();
|
||||
const target = stale.find(t => t.claim === 'dims-test claim at 1024');
|
||||
expect(target).toBeDefined();
|
||||
|
||||
const emb = new Float32Array(DIMS);
|
||||
emb[7] = 1;
|
||||
const updated = await engine.updateTakeEmbeddingsBatch([
|
||||
{ take_id: Number(target!.take_id), embedding: emb },
|
||||
]);
|
||||
expect(updated).toBe(1);
|
||||
|
||||
// Round-trip through the reader at the new width.
|
||||
const map = await engine.getTakeEmbeddings([Number(target!.take_id)]);
|
||||
expect(map.get(Number(target!.take_id))?.length).toBe(DIMS);
|
||||
|
||||
// And through vector search (exact match → similarity ~1, top hit).
|
||||
const hits = await engine.searchTakesVector(emb, { limit: 5 });
|
||||
expect(hits.map(h => h.claim)).toContain('dims-test claim at 1024');
|
||||
expect(hits[0].score).toBeCloseTo(1, 3);
|
||||
|
||||
// The written row left the stale pool.
|
||||
const staleAfter = await engine.listStaleTakes();
|
||||
expect(staleAfter.some(t => Number(t.take_id) === Number(target!.take_id))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* Migration v126 (#2089) — the DEFAULT-dims no-op path, and the guard that
|
||||
* protects backfilled vectors.
|
||||
*
|
||||
* The shipped serial suite (migrations-takes-embedding-dims.serial.test.ts)
|
||||
* pins the 1024 REBUILD path only, and its idempotency re-run happens before
|
||||
* any embedding exists — so the most consequential property of the
|
||||
* `currentDim === embeddingDim` early return was unpinned: a forced re-run
|
||||
* of v126 on a correct column must NOT wipe already-backfilled take
|
||||
* embeddings (the rebuild branch NULLs + drops the column). This test runs
|
||||
* on a fresh default-dims (1536) brain — the overwhelmingly common install —
|
||||
* asserts v126 no-opped at init, backfills a vector via the new writer, then
|
||||
* forces a re-run and asserts the vector SURVIVES.
|
||||
*/
|
||||
import { test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runMigrations } from '../src/core/migrate.ts';
|
||||
|
||||
const DEFAULT_DIMS = 1536;
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function takesEmbeddingDim(): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ atttypmod: number }>(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
return Number(rows[0].atttypmod);
|
||||
}
|
||||
|
||||
test('v126 no-ops on a default-dims brain, and a forced re-run PRESERVES backfilled take embeddings', async () => {
|
||||
// Fresh default brain: column already VECTOR(1536) from v37 → v126 no-op.
|
||||
expect(await takesEmbeddingDim()).toBe(DEFAULT_DIMS);
|
||||
const idx = await engine.executeRaw<{ indexdef: string }>(
|
||||
`SELECT indexdef FROM pg_indexes
|
||||
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
|
||||
);
|
||||
expect(idx.length).toBe(1); // v37's HNSW index intact
|
||||
|
||||
// Backfill one claim through the #2089 writer.
|
||||
const page = await engine.putPage('companies/rerun-example', {
|
||||
title: 'Rerun Example', type: 'company' as const,
|
||||
compiled_truth: '## Takes\n\nA placeholder company page.\n',
|
||||
});
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: page.id, row_num: 1, claim: 'rerun-preservation claim', kind: 'fact', holder: 'world', weight: 0.9 },
|
||||
]);
|
||||
const stale = (await engine.listStaleTakes()).filter(t => t.claim === 'rerun-preservation claim');
|
||||
expect(stale).toHaveLength(1);
|
||||
const emb = new Float32Array(DEFAULT_DIMS);
|
||||
emb[3] = 1;
|
||||
expect(await engine.updateTakeEmbeddingsBatch([
|
||||
{ take_id: Number(stale[0].take_id), embedding: emb },
|
||||
])).toBe(1);
|
||||
|
||||
// Force v126 to run again (version rollback → re-migrate).
|
||||
await engine.setConfig('version', '125');
|
||||
const { applied } = await runMigrations(engine);
|
||||
expect(applied).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// The no-op guard held: same dims, index intact, and — the load-bearing
|
||||
// bit — the backfilled embedding was NOT nulled/dropped by a re-rebuild.
|
||||
expect(await takesEmbeddingDim()).toBe(DEFAULT_DIMS);
|
||||
const kept = await engine.executeRaw<{ has_embedding: boolean }>(
|
||||
`SELECT (embedding IS NOT NULL) AS has_embedding FROM takes WHERE id = $1`,
|
||||
[Number(stale[0].take_id)],
|
||||
);
|
||||
expect(kept[0]?.has_embedding).toBe(true);
|
||||
const hits = await engine.searchTakesVector(emb, { limit: 5 });
|
||||
expect(hits.map(h => h.claim)).toContain('rerun-preservation claim');
|
||||
}, 30_000);
|
||||
@@ -288,6 +288,71 @@ describe('applyRetrievalUpgrade — state machine + atomicity (D12, D18)', () =>
|
||||
expect(byName.get('embedding_multimodal')).toBe('vector(1024)');
|
||||
});
|
||||
|
||||
// Wave-4 review: takes joined TEXT_EMBEDDING_DIM_PINNED_TABLES. Before the
|
||||
// fix, a dim-change transition left takes.embedding at the old width and
|
||||
// the #2089 embed takes lane failed every write forever (while still
|
||||
// paying the gateway each run).
|
||||
test('runSchemaTransition rebuilds takes.embedding at the target dim + recreates the partial HNSW index', async () => {
|
||||
await setLegacyDefaultConfig();
|
||||
await seedPages(150);
|
||||
|
||||
// Seed one take and give it a vector at the OLD width (1536).
|
||||
const page = await engine.putPage('takes/transition-example', {
|
||||
title: 'Takes transition example', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for the takes dim transition.',
|
||||
});
|
||||
await engine.addTakesBatch([{
|
||||
page_id: page.id, row_num: 1, claim: 'pre-transition claim',
|
||||
kind: 'fact', holder: 'world', weight: 0.9,
|
||||
}]);
|
||||
const stale = (await engine.listStaleTakes()).filter(t => t.claim === 'pre-transition claim');
|
||||
expect(stale).toHaveLength(1);
|
||||
const takeId = Number(stale[0].take_id);
|
||||
// Probe the CURRENT column width for the pre-transition write — the
|
||||
// suite shares one engine, and an earlier apply test may already have
|
||||
// transitioned the takes column (schema persists across resetPgliteState).
|
||||
const cur = await engine.executeRaw<{ atttypmod: number }>(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
);
|
||||
const curDim = Number(cur[0]?.atttypmod);
|
||||
expect(curDim).toBeGreaterThan(0);
|
||||
const oldVec = new Float32Array(curDim);
|
||||
oldVec[0] = 1;
|
||||
expect(await engine.updateTakeEmbeddingsBatch([{ take_id: takeId, embedding: oldVec }])).toBe(1);
|
||||
|
||||
const plan = await planRetrievalUpgrade(engine);
|
||||
await applyRetrievalUpgrade(engine, plan);
|
||||
|
||||
// Column rebuilt at the target dim (atttypmod IS the pgvector dimension).
|
||||
const dim = await engine.executeRaw<{ atttypmod: number }>(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'takes'::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
);
|
||||
expect(Number(dim[0]?.atttypmod)).toBe(ZE_TARGET_EMBEDDING_DIM);
|
||||
|
||||
// Partial HNSW index recreated (dim under the #1734 cap), v37 predicate.
|
||||
const idx = await engine.executeRaw<{ indexdef: string }>(
|
||||
`SELECT indexdef FROM pg_indexes
|
||||
WHERE tablename = 'takes' AND indexname = 'idx_takes_embedding_hnsw'`,
|
||||
);
|
||||
expect(idx).toHaveLength(1);
|
||||
expect(idx[0].indexdef).toContain('hnsw');
|
||||
expect(idx[0].indexdef).toMatch(/WHERE\s+\(?active/i);
|
||||
|
||||
// Old-space vector discarded — the take is back in the stale pool.
|
||||
const restaled = (await engine.listStaleTakes()).filter(t => Number(t.take_id) === takeId);
|
||||
expect(restaled).toHaveLength(1);
|
||||
|
||||
// And the rebuilt column ACCEPTS a vector at the new width end-to-end.
|
||||
const newVec = new Float32Array(ZE_TARGET_EMBEDDING_DIM);
|
||||
newVec[3] = 1;
|
||||
expect(await engine.updateTakeEmbeddingsBatch([{ take_id: takeId, embedding: newVec }])).toBe(1);
|
||||
expect((await engine.getTakeEmbeddings([takeId])).get(takeId)?.length).toBe(ZE_TARGET_EMBEDDING_DIM);
|
||||
});
|
||||
|
||||
test('runSchemaTransition restores partial WHERE on idx_chunks_embedding_image', async () => {
|
||||
await setLegacyDefaultConfig();
|
||||
await seedPages(150);
|
||||
|
||||
@@ -782,6 +782,13 @@ const COLUMN_EXEMPTIONS = new Set<string>([
|
||||
'minion_jobs.budget_remaining_cents',
|
||||
'minion_jobs.budget_owner_job_id',
|
||||
'minion_jobs.budget_root_owner_id',
|
||||
// v126 (#2089) — takes_embedding_active_dims rebuilds takes.embedding at
|
||||
// the brain's configured dim (DROP COLUMN + ADD COLUMN inside the handler).
|
||||
// Same precedent as facts.* / query_cache.*: the `takes` table is
|
||||
// migration-created (v37), absent from PGLITE_SCHEMA_SQL, so no schema-blob
|
||||
// forward reference can exist; the HNSW index recreation lives INSIDE the
|
||||
// same v126 migration. Column-only rebuild, no bootstrap probe needed.
|
||||
'takes.embedding',
|
||||
]);
|
||||
|
||||
test('every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by applyForwardReferenceBootstrap (column-only class)', async () => {
|
||||
|
||||
@@ -19,6 +19,16 @@ function makeEngine(opts: { knownSources?: string[] } = {}) {
|
||||
const pageLookups: unknown[][] = [];
|
||||
const engine = {
|
||||
getConfig: async () => null,
|
||||
// Wave-4: cmdAdd routes through the shared appendTake helper
|
||||
// (takes-append.ts), whose scoped page lookup is engine.getPage —
|
||||
// the same seam persistThinkTake uses. Record [slug, scope] so the
|
||||
// scoping assertions below stay shape-identical to the old raw-SQL pin.
|
||||
getPage: async (slug: string, pageOpts?: { sourceId?: string; sourceIds?: string[] }) => {
|
||||
pageLookups.push([slug, pageOpts?.sourceIds ?? pageOpts?.sourceId]);
|
||||
if (slug === 'shared/page' && pageOpts?.sourceId === 'dept') return { id: 22, slug };
|
||||
if (slug === 'shared/page' && (pageOpts?.sourceId === 'default' || pageOpts?.sourceId === undefined)) return { id: 11, slug };
|
||||
return null;
|
||||
},
|
||||
executeRaw: async (sql: string, params: unknown[] = []) => {
|
||||
if (sql.includes('FROM sources WHERE id = $1')) {
|
||||
// Default (no `knownSources` override): every id "exists", matching
|
||||
|
||||
@@ -392,3 +392,76 @@ describe('countStaleTakes + listStaleTakes', () => {
|
||||
expect(stale[0]).toHaveProperty('claim');
|
||||
});
|
||||
});
|
||||
|
||||
// #2089: the writer half of the takes embedding pipeline.
|
||||
describe('updateTakeEmbeddingsBatch', () => {
|
||||
test('empty input returns 0 without a query', async () => {
|
||||
expect(await engine.updateTakeEmbeddingsBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('round-trip: writes 2 embeddings, stamps embedded_at, decrements stale count', async () => {
|
||||
// Vector width must match the brain's active dim (config row), not a
|
||||
// hardcoded literal — v126 rebuilds takes.embedding at this dim.
|
||||
const dimRows = await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
);
|
||||
const dims = parseInt(dimRows[0]?.value ?? '1536', 10);
|
||||
|
||||
const staleBefore = await engine.countStaleTakes();
|
||||
expect(staleBefore).toBeGreaterThanOrEqual(2);
|
||||
const stale = await engine.listStaleTakes();
|
||||
const [a, b] = stale;
|
||||
const embA = new Float32Array(dims);
|
||||
embA[0] = 1;
|
||||
const embB = new Float32Array(dims);
|
||||
embB[1] = 1;
|
||||
|
||||
const updated = await engine.updateTakeEmbeddingsBatch([
|
||||
{ take_id: Number(a.take_id), embedding: embA },
|
||||
{ take_id: Number(b.take_id), embedding: embB },
|
||||
]);
|
||||
expect(updated).toBe(2);
|
||||
|
||||
// Read back via getTakeEmbeddings (the paired reader).
|
||||
const map = await engine.getTakeEmbeddings([Number(a.take_id), Number(b.take_id)]);
|
||||
expect(map.size).toBe(2);
|
||||
expect(map.get(Number(a.take_id))!.length).toBe(dims);
|
||||
expect(map.get(Number(a.take_id))![0]).toBeCloseTo(1, 5);
|
||||
expect(map.get(Number(b.take_id))![1]).toBeCloseTo(1, 5);
|
||||
|
||||
// embedded_at stamped on both rows.
|
||||
const rows = await engine.executeRaw<{ embedded_at: string | null }>(
|
||||
`SELECT embedded_at FROM takes WHERE id = ANY($1::bigint[])`,
|
||||
[[Number(a.take_id), Number(b.take_id)]],
|
||||
);
|
||||
expect(rows.length).toBe(2);
|
||||
for (const r of rows) expect(r.embedded_at).not.toBeNull();
|
||||
|
||||
// Stale pool shrank by exactly the two written rows.
|
||||
expect(await engine.countStaleTakes()).toBe(staleBefore - 2);
|
||||
});
|
||||
|
||||
test('inactive (superseded) rows are skipped — count reflects actual updates', async () => {
|
||||
// The supersedeTake suite above retired (alicePageId, row 3). Its row is
|
||||
// active=false with a NULL embedding; the writer must not touch it.
|
||||
const inactiveRows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM takes WHERE active = FALSE LIMIT 1`,
|
||||
);
|
||||
expect(inactiveRows.length).toBe(1);
|
||||
const dims = parseInt(
|
||||
(await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
))[0]?.value ?? '1536',
|
||||
10,
|
||||
);
|
||||
const updated = await engine.updateTakeEmbeddingsBatch([
|
||||
{ take_id: Number(inactiveRows[0].id), embedding: new Float32Array(dims) },
|
||||
]);
|
||||
expect(updated).toBe(0);
|
||||
const check = await engine.executeRaw<{ embedding: unknown }>(
|
||||
`SELECT embedding FROM takes WHERE id = $1`,
|
||||
[Number(inactiveRows[0].id)],
|
||||
);
|
||||
expect(check[0]?.embedding).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { operationsByName } from '../src/core/operations.ts';
|
||||
import { runThink, persistSynthesis, type ThinkLLMClient } from '../src/core/think/index.ts';
|
||||
import { runThink, persistSynthesis, persistThinkTake, THINK_TAKE_CLAIM_MAX_CHARS, type ThinkLLMClient } from '../src/core/think/index.ts';
|
||||
import { sanitizeTakeForPrompt, renderTakesBlock } from '../src/core/think/sanitize.ts';
|
||||
import { resolveCitations, parseInlineCitations, normalizeStructuredCitations } from '../src/core/think/cite-render.ts';
|
||||
import { runGather } from '../src/core/think/gather.ts';
|
||||
@@ -470,3 +470,118 @@ describe('think MCP op — #1698 C3 + #10', () => {
|
||||
expect(res.warnings).toContain('SYNTHESIS_EMPTY_NOT_PERSISTED');
|
||||
});
|
||||
});
|
||||
|
||||
describe('persistThinkTake — #2556 think --take actually persists', () => {
|
||||
function synthResult(answer: string, ok = true): any {
|
||||
return {
|
||||
question: 'what should this page remember?',
|
||||
answer, citations: [], gaps: [], pagesGathered: 0, takesGathered: 0,
|
||||
graphHits: 0, modelUsed: 'stub', rounds: 1, warnings: [], synthesisOk: ok,
|
||||
diagnostics: { pagesFromHybrid: 0, takesFromKeyword: 0, takesFromVector: 0, graphHits: 0 },
|
||||
};
|
||||
}
|
||||
|
||||
test('appends the synthesis answer as the next anchor take row', async () => {
|
||||
const target = await engine.putPage('notes/think-take-target-example', {
|
||||
title: 'Think take target', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for think take persistence.',
|
||||
});
|
||||
const persisted = await persistThinkTake(
|
||||
engine, synthResult('This page should remember the synthesized placeholder insight.'),
|
||||
{ anchor: 'notes/think-take-target-example' },
|
||||
);
|
||||
// Wave-4 dual-plane contract: this hermetic engine has no brain repo
|
||||
// (sync.repo_path unset), so appendTake falls back to DB-only allocation
|
||||
// and surfaces TAKE_FILE_PLANE_UNAVAILABLE. The row still lands.
|
||||
expect(persisted).toMatchObject({ rowNum: 1, inserted: 1 });
|
||||
expect(persisted.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
|
||||
const takes = await engine.listTakes({ page_id: target.id });
|
||||
expect(takes).toHaveLength(1);
|
||||
expect(takes[0]).toMatchObject({
|
||||
row_num: 1,
|
||||
claim: 'This page should remember the synthesized placeholder insight.',
|
||||
kind: 'take', holder: 'brain', source: 'gbrain think',
|
||||
});
|
||||
// Second take lands on the NEXT row (MAX+1 under the page lock).
|
||||
const second = await persistThinkTake(
|
||||
engine, synthResult('A second synthesized insight.'),
|
||||
{ anchor: 'notes/think-take-target-example' },
|
||||
);
|
||||
expect(second.rowNum).toBe(2);
|
||||
});
|
||||
|
||||
test('bounds the claim: 2500-char multi-line answer → single-line 2000-char claim + TAKE_CLAIM_TRUNCATED', async () => {
|
||||
const target = await engine.putPage('notes/think-take-truncate-example', {
|
||||
title: 'Think take truncate target', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for claim-bound persistence.',
|
||||
});
|
||||
// Multi-line answer well past the cap: fence cells are single-line, and
|
||||
// sanitizeTakeForPrompt only ever reads the first 500 chars anyway.
|
||||
const line = 'a'.repeat(99);
|
||||
const answer = Array.from({ length: 25 }, () => line).join('\n'); // 25*99 + 24 = 2499 chars
|
||||
expect(answer.length).toBeGreaterThan(THINK_TAKE_CLAIM_MAX_CHARS);
|
||||
|
||||
const persisted = await persistThinkTake(engine, synthResult(answer), {
|
||||
anchor: 'notes/think-take-truncate-example',
|
||||
});
|
||||
expect(persisted.rowNum).toBe(1);
|
||||
expect(persisted.inserted).toBe(1);
|
||||
expect(persisted.warnings).toContain('TAKE_CLAIM_TRUNCATED');
|
||||
|
||||
const takes = await engine.listTakes({ page_id: target.id });
|
||||
expect(takes).toHaveLength(1);
|
||||
expect(takes[0].claim.length).toBe(THINK_TAKE_CLAIM_MAX_CHARS);
|
||||
expect(takes[0].claim).not.toContain('\n'); // flattened before bounding
|
||||
});
|
||||
|
||||
test('refuses empty/no-LLM synthesis instead of writing a blank take', async () => {
|
||||
const target = await engine.putPage('notes/think-take-empty-example', {
|
||||
title: 'Think take empty target', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for empty think take persistence.',
|
||||
});
|
||||
const persisted = await persistThinkTake(
|
||||
engine, synthResult('(no LLM available)', false),
|
||||
{ anchor: 'notes/think-take-empty-example' },
|
||||
);
|
||||
expect(persisted).toEqual({ rowNum: null, inserted: 0, warnings: ['TAKE_EMPTY_NOT_PERSISTED'] });
|
||||
expect(await engine.listTakes({ page_id: target.id })).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('missing anchor and out-of-scope anchor fail with distinct loud signals', async () => {
|
||||
expect(await persistThinkTake(engine, synthResult('x'), {})).toEqual(
|
||||
{ rowNum: null, inserted: 0, warnings: ['TAKE_REQUIRES_ANCHOR'] });
|
||||
const missing = await persistThinkTake(engine, synthResult('x'), { anchor: 'notes/no-such-page' });
|
||||
expect(missing.rowNum).toBeNull();
|
||||
expect(missing.warnings[0]).toContain('TAKE_ANCHOR_NOT_FOUND');
|
||||
// Scope confinement: the anchor exists in the default source but the
|
||||
// caller's federated grant excludes it -> not found (fail-closed).
|
||||
const scoped = await persistThinkTake(engine, synthResult('x'), {
|
||||
anchor: 'notes/think-take-target-example', sourceIds: ['some-other-source'],
|
||||
});
|
||||
expect(scoped.rowNum).toBeNull();
|
||||
expect(scoped.warnings[0]).toContain('TAKE_ANCHOR_NOT_FOUND');
|
||||
});
|
||||
|
||||
test('op handler: local take persists; remote take stays blocked (#2556 + trust gate)', async () => {
|
||||
const op = operationsByName['think'];
|
||||
await engine.putPage('notes/think-take-op-example', {
|
||||
title: 'Think take op target', type: 'note',
|
||||
compiled_truth: 'Safe placeholder for the op-layer take test.',
|
||||
});
|
||||
const baseCtx = (remote: boolean) => ({
|
||||
engine, config: {} as any, dryRun: false, remote,
|
||||
logger: { info() {}, warn() {}, error() {}, debug() {} } as any,
|
||||
});
|
||||
// Remote: take must be zeroed by the trust gate; nothing written.
|
||||
const remoteRes: any = await withoutAnthropicKey(() =>
|
||||
op.handler(baseCtx(true) as any, { question: 'q', anchor: 'notes/think-take-op-example', take: true }));
|
||||
expect(remoteRes.take_row).toBeNull();
|
||||
expect(remoteRes.remote_persisted_blocked).toBe(true);
|
||||
// Local with no LLM: synthesis empty -> take refused with the loud warning
|
||||
// (proves the local path CONSUMES the flag instead of ignoring it).
|
||||
const localRes: any = await withoutAnthropicKey(() =>
|
||||
op.handler(baseCtx(false) as any, { question: 'q', anchor: 'notes/think-take-op-example', take: true }));
|
||||
expect(localRes.take_row).toBeNull();
|
||||
expect(localRes.warnings).toContain('TAKE_EMPTY_NOT_PERSISTED');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
/**
|
||||
* #2556 — `gbrain think --take` CLI exit-honesty pins.
|
||||
*
|
||||
* The shipped tests cover persistThinkTake (the core) and the MCP op
|
||||
* handler, but nothing drives src/commands/think.ts's take path — the two
|
||||
* exit(1) branches ("--take requires --anchor" and "--take requested but no
|
||||
* take row was written") were untested. An explicit --take that writes
|
||||
* nothing MUST be loud (same honesty contract as --save / #1698 F2).
|
||||
*
|
||||
* Serial: spies on process.exit + console.error (process-global), and the
|
||||
* no-LLM path mutates ANTHROPIC_API_KEY/GBRAIN_HOME via withoutAnthropicKey.
|
||||
*/
|
||||
import { test, expect, beforeAll, afterAll, spyOn } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runThinkCli } from '../src/commands/think.ts';
|
||||
import { withoutAnthropicKey } from './helpers/no-anthropic-key.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let anchorPageId: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const p = await engine.putPage('notes/cli-take-anchor-example', {
|
||||
title: 'CLI take anchor', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for the CLI take exit tests.',
|
||||
});
|
||||
anchorPageId = p.id;
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
/**
|
||||
* Run runThinkCli with process.exit + console.error captured. The exit spy
|
||||
* throws a sentinel so control flow actually stops at the exit site (a
|
||||
* non-throwing mock would let the command keep running past exit(1)).
|
||||
* Note: an exit(1) inside the runThink try-block is caught by the #1698
|
||||
* catch, which console.errors the sentinel message and exits again — the
|
||||
* FIRST captured code is the one asserted.
|
||||
*/
|
||||
async function runExpectingExit(args: string[]): Promise<{ code: number | null; stderr: string }> {
|
||||
const errors: string[] = [];
|
||||
let code: number | null = null;
|
||||
const errSpy = spyOn(console, 'error').mockImplementation((...a: unknown[]) => {
|
||||
errors.push(a.map(String).join(' '));
|
||||
});
|
||||
const exitSpy = spyOn(process, 'exit').mockImplementation(((c?: number) => {
|
||||
if (code === null) code = c ?? 0;
|
||||
throw new Error(`EXIT:${c}`);
|
||||
}) as never);
|
||||
try {
|
||||
await runThinkCli(engine, args);
|
||||
} catch (e) {
|
||||
if (!(e instanceof Error) || !e.message.startsWith('EXIT:')) throw e;
|
||||
} finally {
|
||||
exitSpy.mockRestore();
|
||||
errSpy.mockRestore();
|
||||
}
|
||||
return { code, stderr: errors.join('\n') };
|
||||
}
|
||||
|
||||
test('--take without --anchor exits 1 with the actionable message (parse-time guard)', async () => {
|
||||
const { code, stderr } = await runExpectingExit(['what do we know', '--take']);
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain('--take requires --anchor');
|
||||
});
|
||||
|
||||
test('--take with no LLM (empty synthesis) exits 1 loudly and persists NOTHING', async () => {
|
||||
const { code, stderr } = await withoutAnthropicKey(() =>
|
||||
runExpectingExit(['what do we know about this page', '--take', '--anchor', 'notes/cli-take-anchor-example']),
|
||||
);
|
||||
// #2556 honesty contract: explicit --take that wrote nothing → exit 1
|
||||
// with the take-specific message, not a silent 0.
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain('--take requested but no take row was written');
|
||||
// And the DB really is untouched — no blank/stub take row.
|
||||
expect(await engine.listTakes({ page_id: anchorPageId })).toHaveLength(0);
|
||||
}, 30_000);
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* #2556 — persistThinkTake concurrency pin.
|
||||
*
|
||||
* The function's doc-comment CLAIMS the MAX(row_num)+1 computation + insert
|
||||
* are serialized under withPageLock(anchor) so two concurrent `think --take`
|
||||
* calls (or a takes add racing a think --take) can't produce duplicate
|
||||
* row_nums. The shipped tests only exercise SEQUENTIAL calls (row 1 then
|
||||
* row 2), which passes even with no lock at all. This test drives the two
|
||||
* calls CONCURRENTLY via Promise.all and asserts they land on distinct
|
||||
* consecutive rows — the behavioral proof the lock is actually keyed and
|
||||
* actually serializes.
|
||||
*
|
||||
* Serial: the page lock is a real file under $GBRAIN_HOME/.gbrain/page-locks
|
||||
* (process-global fs + env), and the loser of the race polls at 200ms.
|
||||
*/
|
||||
import { test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { persistThinkTake } from '../src/core/think/index.ts';
|
||||
import { withEnv, emptyHome } from './helpers/with-env.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
function synthResult(answer: string): any {
|
||||
return {
|
||||
question: 'what should this page remember?',
|
||||
answer, citations: [], gaps: [], pagesGathered: 0, takesGathered: 0,
|
||||
graphHits: 0, modelUsed: 'stub', rounds: 1, warnings: [], synthesisOk: true,
|
||||
diagnostics: { pagesFromHybrid: 0, takesFromKeyword: 0, takesFromVector: 0, graphHits: 0 },
|
||||
};
|
||||
}
|
||||
|
||||
test('two CONCURRENT persistThinkTake calls on the same anchor serialize to rows 1 and 2 (no duplicate row_num)', async () => {
|
||||
const page = await engine.putPage('notes/think-take-race-example', {
|
||||
title: 'Think take race target', type: 'note',
|
||||
compiled_truth: 'A safe placeholder page for concurrent think take persistence.',
|
||||
});
|
||||
|
||||
// GBRAIN_HOME → fresh temp dir so the page-lock files land in a hermetic
|
||||
// location (gbrainPath honors GBRAIN_HOME) instead of the user's ~/.gbrain.
|
||||
await withEnv({ GBRAIN_HOME: emptyHome() }, async () => {
|
||||
const [a, b] = await Promise.all([
|
||||
persistThinkTake(engine, synthResult('first concurrent synthesized insight'), {
|
||||
anchor: 'notes/think-take-race-example',
|
||||
}),
|
||||
persistThinkTake(engine, synthResult('second concurrent synthesized insight'), {
|
||||
anchor: 'notes/think-take-race-example',
|
||||
}),
|
||||
]);
|
||||
|
||||
// Both persisted and the row numbers are EXACTLY {1, 2} — a lost lock
|
||||
// would yield {1, 1} (duplicate) or a failed insert. Wave-4 dual-plane
|
||||
// contract: no brain repo is configured on this hermetic engine, so both
|
||||
// appends fall back to DB-only allocation and surface
|
||||
// TAKE_FILE_PLANE_UNAVAILABLE (still serialized under the page lock).
|
||||
expect(a.inserted).toBe(1);
|
||||
expect(b.inserted).toBe(1);
|
||||
expect(a.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
|
||||
expect(b.warnings).toEqual(['TAKE_FILE_PLANE_UNAVAILABLE']);
|
||||
expect([a.rowNum, b.rowNum].sort()).toEqual([1, 2]);
|
||||
});
|
||||
|
||||
// DB ground truth: two rows, distinct row_nums, both holder=brain.
|
||||
const takes = await engine.listTakes({ page_id: page.id });
|
||||
expect(takes).toHaveLength(2);
|
||||
expect(new Set(takes.map(t => t.row_num)).size).toBe(2);
|
||||
const dupCheck = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM takes
|
||||
WHERE page_id = $1
|
||||
GROUP BY row_num HAVING COUNT(*) > 1`,
|
||||
[page.id],
|
||||
);
|
||||
expect(dupCheck).toHaveLength(0);
|
||||
}, 15_000);
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* #2556 — think op handler: the scope-dialect mapping under a FEDERATED ctx.
|
||||
*
|
||||
* The op handler maps thinkSourceScopeOpts's `allowedSources` (runThink's
|
||||
* dialect) onto persistThinkTake's `sourceIds` (the engine's getPage
|
||||
* dialect). The inline comment warns that spreading thinkScope would
|
||||
* "silently drop the federated array and unscope the anchor lookup" — but
|
||||
* no shipped test constructs a ctx with `auth.allowedSources`, so that exact
|
||||
* regression (a cross-source take write) would land green.
|
||||
*
|
||||
* These tests drive the FULL op handler to a successful synthesis using the
|
||||
* gateway chat-transport test seam (no real LLM call), with the anchor page
|
||||
* living in tenant-a:
|
||||
* - grant includes tenant-a → take row lands (mapping threads the array)
|
||||
* - grant is tenant-b only → TAKE_ANCHOR_NOT_FOUND, nothing written
|
||||
* (fail-closed; an unscoped getPage would wrongly find the page)
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { operationsByName } from '../src/core/operations.ts';
|
||||
import {
|
||||
__setChatTransportForTests,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { withEnv, emptyHome } from './helpers/with-env.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let anchorPageId: number;
|
||||
|
||||
const ANSWER = 'Federated synthesis insight for the anchor page.';
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
for (const id of ['tenant-a', 'tenant-b']) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config, created_at)
|
||||
VALUES ($1, $1, NULL, '{}'::jsonb, NOW()) ON CONFLICT (id) DO NOTHING`,
|
||||
[id],
|
||||
);
|
||||
}
|
||||
const p = await engine.putPage('people/fed-anchor-example', {
|
||||
title: 'Federated anchor', type: 'person',
|
||||
compiled_truth: 'A safe placeholder page living in tenant-a.',
|
||||
}, { sourceId: 'tenant-a' });
|
||||
anchorPageId = p.id;
|
||||
|
||||
// Canned successful synthesis — chat() calls the transport directly,
|
||||
// skipping provider resolution/SDK; think parses the JSON envelope.
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: JSON.stringify({ answer: ANSWER, citations: [], gaps: [] }),
|
||||
blocks: [],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 10, output_tokens: 5, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-opus-4-7',
|
||||
providerId: 'anthropic',
|
||||
}) as any);
|
||||
// Neutralize any query-embedding attempt during gather (fake key below
|
||||
// must never reach the network).
|
||||
__setEmbedTransportForTests((async (args: any) => ({
|
||||
embeddings: ((args?.values ?? []) as string[]).map(() => {
|
||||
const v = new Array(1536).fill(0);
|
||||
v[0] = 1;
|
||||
return v;
|
||||
}),
|
||||
usage: { tokens: 1 },
|
||||
})) as any);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
__setChatTransportForTests(null);
|
||||
__setEmbedTransportForTests(null);
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
function fedCtx(allowedSources: string[]): any {
|
||||
return {
|
||||
engine,
|
||||
config: {} as any,
|
||||
dryRun: false,
|
||||
remote: false, // trusted local — safeTake honored; scope still MUST confine
|
||||
auth: { allowedSources },
|
||||
logger: { info() {}, warn() {}, error() {}, debug() {} } as any,
|
||||
};
|
||||
}
|
||||
|
||||
async function runThinkOpTake(allowedSources: string[]): Promise<any> {
|
||||
const op = operationsByName['think'];
|
||||
// Fake key so tryBuildGatewayClient builds a client (transport is stubbed);
|
||||
// empty GBRAIN_HOME so the real user config can't leak into model routing
|
||||
// and the page-lock files land in a hermetic temp dir.
|
||||
return withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake', GBRAIN_HOME: emptyHome() }, () =>
|
||||
op.handler(fedCtx(allowedSources), {
|
||||
question: 'what do we know about the federated anchor?',
|
||||
anchor: 'people/fed-anchor-example',
|
||||
take: true,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
describe('think op — federated allowedSources → sourceIds mapping (#2556)', () => {
|
||||
test('in-grant federated ctx: take row lands on the anchor in the granted source', async () => {
|
||||
const res = await runThinkOpTake(['tenant-a']);
|
||||
expect(res.take_row).toBe(1);
|
||||
expect(res.take_inserted).toBe(1);
|
||||
expect(res.remote_persisted_blocked).toBe(false);
|
||||
|
||||
const takes = await engine.listTakes({ page_id: anchorPageId });
|
||||
expect(takes).toHaveLength(1);
|
||||
expect(takes[0]).toMatchObject({
|
||||
row_num: 1, claim: ANSWER, kind: 'take', holder: 'brain', source: 'gbrain think',
|
||||
});
|
||||
}, 30_000);
|
||||
|
||||
test('out-of-grant federated ctx: anchor lookup fail-closes — no cross-source take write', async () => {
|
||||
const res = await runThinkOpTake(['tenant-b']);
|
||||
// If the handler spread thinkScope (dropping the federated array), the
|
||||
// getPage lookup would be UNSCOPED, find the tenant-a page, and write a
|
||||
// cross-source row — this pins the mapping instead.
|
||||
expect(res.take_row).toBeNull();
|
||||
expect(res.take_inserted).toBe(0);
|
||||
expect((res.warnings as string[]).some(w => w.startsWith('TAKE_ANCHOR_NOT_FOUND'))).toBe(true);
|
||||
|
||||
// Still exactly the one row from the in-grant test above.
|
||||
expect(await engine.listTakes({ page_id: anchorPageId })).toHaveLength(1);
|
||||
}, 30_000);
|
||||
});
|
||||
Reference in New Issue
Block a user