mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-16 09:52:22 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b196665fc6 | ||
|
|
e72d93fdb5 | ||
|
|
85286a556c | ||
|
|
a8a3b6df9f |
@@ -304,7 +304,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape.
|
||||
- `src/core/console-prefix.ts` — `AsyncLocalStorage<string>`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as active prefix; nested wraps replace then restore), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log`/`console.error`). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog`/`serr` fall through to bare `console.log`/`console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form) to defeat log-injection through newline/control-character names. Coverage: `src/commands/sync.ts` performSync + callees, `src/commands/embed.ts` runEmbedCore + helpers, `src/core/progress.ts` emitHumanLine.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` / `--brain <id>` stripped. `--brain` is the brain-axis (which database) selector: exact-match only (`--brain-*` per-command flags pass through), value validated against the mount-id regex at parse time, missing/malformed value THROWS — never a silent host fallback. `connectEngine` in `src/cli.ts` feeds it (plus the ambient `GBRAIN_BRAIN_ID` / `.gbrain-mount` / mount-path tiers) through `resolveBrainId` → `BrainRegistry.getBrain`, which throws `UnknownBrainError` for an unregistered id; mounts get no auto-migrations and keep the host-config AI gateway. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators (propagates `--brain=<id>` so children stay on the parent's brain). `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the `gbrain_cycle_locks` table. Parameterized lock id so scopes nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID`) for `performSync`'s narrower writer window. UPSERT-with-TTL semantics survive PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. It also does automatic same-host dead-pid takeover: when the upsert finds a held, NOT-TTL-expired lock whose holder is on this host and provably dead, it reclaims via a guarded `DELETE WHERE id=$1 AND holder_pid=$2` + one normal-upsert retry returning the standard handle (refresh/release intact). The liveness check is the exported `classifyHolderLiveness(pid, host, ageMs, opts?)` / `isHolderDeadLocally(...)` (injectable `process.kill` seam; `HOLDER_TAKEOVER_GRACE_MS = 60_000` PID-reuse guard; EPERM classified as `alive` so a live process you don't own is never stolen). TTL-expired locks stay the upsert's job; cross-host stays TTL-only. `runBreakLock` (`src/commands/sync.ts`) consumes the same predicate. Background reaper (#1972): `reapDeadHolderLocks(engine)` is the periodic sweep the contention path lacked — it deletes locks whose holder is `isHolderDeadLocally`, scoped to the `gbrain-sync:*` / `gbrain-cycle`/`gbrain-cycle:*` namespaces ONLY (election/supervisor/reindex locks keep TTL-only behavior, untouched), via `deleteLockRowExact(engine, id, pid, acquiredAt)` — a snapshot-matched delete (`date_trunc('milliseconds', acquired_at) = $3`, so the ms a JS Date keeps survives) that's TOCTOU-safe against a reused PID taking the lock between SELECT and DELETE. `cycle.ts` runs it at cycle start (before the sync phase); `gbrain doctor --fix` runs it for no-autopilot brains. `selectLockRows(engine, opts?)` + a shared row→`LockSnapshot` mapper are the single canonical reader now backing `inspectLock` + `listStaleLocks` + the reaper (was triplicated). `isLockHolderLive(snap, ttlMinutes)` (#2227) is the observability liveness predicate — freshness-keyed (`ttl_expired` plus the heartbeat steal-grace), never `process.kill`, so `gbrain jobs supervisor status` / `gbrain doctor` can report a live supervisor via its queue lock without a PID-reuse false-positive. Pinned by `test/db-lock-auto-takeover.test.ts` + `test/db-lock-reap.test.ts`.
|
||||
- `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. `resolveMaxConnections()` (reads `GBRAIN_MAX_CONNECTIONS`, undefined when unset) + `clampWorkersForConnectionBudget(workers, perWorkerPool, maxConnections, parentPool)` back the opt-in single-sync connection-footprint clamp so a big sync stays under a low pooler cap (`parent_pool + workers×perWorkerPool ≤ budget`); `gbrain doctor`'s `pool_budget` check (`computePoolBudgetCheck` / `checkPoolBudget` in `src/commands/doctor.ts`) warns when the budget leaves no room for a worker, pointing at `GBRAIN_POOL_SIZE=2`. Pinned by `test/pglite-workers-clamp.test.ts`.
|
||||
- `src/core/worker-pool.ts` — Canonical sliding-pool + bounded-semaphore primitive (extracted from `src/commands/embed.ts` sliding-pool sites and `src/commands/eval-cross-modal.ts` `runWithLimit` semaphore). Two exports: `runSlidingPool<T>({items, workers, onItem, signal?, onError?, failureLabel?, onProgress?})` + `runWithLimit<TIn, TOut>({items, limit, fn, signal?})`. Atomicity invariant: `const idx = nextIdx++` is one synchronous JS statement (no `await` between read and write — guaranteed by the single-threaded event loop), documented in the module header AND enforced by `scripts/check-worker-pool-atomicity.sh` (wired into `bun run verify`), which rejects importing `worker_threads` in any consuming file and inserting `await` between the `nextIdx` read and write. `MUST_ABORT_ERROR_TAGS` set is seeded with `BUDGET_EXHAUSTED` from `src/core/budget/budget-tracker.ts`; tagged errors (matched via `err.tag === 'BUDGET_EXHAUSTED'` to avoid cross-module import) bypass `onError` and hard-abort the pool via `AbortController.abort()` to in-flight `onItem` — the budget cap is a structural ceiling under concurrency. `failures[]` shape is `{idx, label, error}` records (NOT full items; callers supply `failureLabel(item) => string`) for bounded memory under huge brains. Pinned by `test/worker-pool.test.ts` + `test/scripts/check-worker-pool-atomicity.test.ts`. Drives every `--workers N` bulk command.
|
||||
|
||||
+61
@@ -394,6 +394,18 @@ async function main() {
|
||||
if (op.localOnly) {
|
||||
refuseThinClient(command, cfgPre!.remote_mcp!.mcp_url);
|
||||
}
|
||||
// A thin client has no local mounts — an explicit --brain cannot be
|
||||
// honored and must not be silently dropped (same loud-beats-silent rule
|
||||
// as applyThinClientSourceScope's --source refusal). Ambient tiers
|
||||
// (GBRAIN_BRAIN_ID / .gbrain-mount) are ignored here, matching the
|
||||
// source axis's ambient-with-nowhere-to-send behavior.
|
||||
if (cliOpts.brain) {
|
||||
console.error(
|
||||
'--brain is not supported on a thin-client install: the remote server is a single brain. ' +
|
||||
'Remove the flag, or run from a machine with local mounts (gbrain mounts list).',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
// #2098: the local path resolves --source / GBRAIN_SOURCE / .gbrain-source
|
||||
// inside makeContext (ctx.sourceId), which this route never reaches — so
|
||||
// scope must be mapped onto the op's source_id wire param before the call.
|
||||
@@ -1004,6 +1016,10 @@ export async function makeContext(engine: BrainEngine, params: Record<string, un
|
||||
// table). Matches dispatch.ts's auto-fill so the contract holds across
|
||||
// every transport.
|
||||
sourceId: sourceId ?? 'default',
|
||||
// Brain axis: the id connectEngine resolved for this process. Module
|
||||
// state, NEVER params — caller-supplied params.brain must not select a
|
||||
// brain (that would be an untrusted-caller cross-brain hole over MCP).
|
||||
brainId: activeBrainId,
|
||||
...(localFederated ? { localFederatedSourceIds: localFederated } : {}),
|
||||
};
|
||||
}
|
||||
@@ -2340,7 +2356,52 @@ async function dispatchReadOnlyCommand(engine: BrainEngine, command: string, arg
|
||||
import { buildGatewayConfig } from './core/ai/build-gateway-config.ts';
|
||||
export { buildGatewayConfig };
|
||||
|
||||
/**
|
||||
* Which brain this process's engine targets. Set by connectEngine after brain
|
||||
* resolution; read by makeContext so ctx.brainId carries the audit id. Never
|
||||
* derived from op params — an untrusted caller must not be able to name a
|
||||
* brain (same fail-closed shape as the #3524 remote source sentinel).
|
||||
*/
|
||||
let activeBrainId: string = 'host';
|
||||
|
||||
/**
|
||||
* Connect to a mounted brain (brain axis, non-host). Routes through
|
||||
* BrainRegistry so:
|
||||
* - an unknown/disabled mount id throws UnknownBrainError. Fail-closed:
|
||||
* the pre-fix CLI silently fell back to the host brain, returning
|
||||
* confident wrong answers (mirror of #3524's explicit --source decision);
|
||||
* - postgres mounts get a per-instance pool, never the db.ts singleton;
|
||||
* - NO migrations run against the mount — schema is the publisher's job
|
||||
* (same decision as BrainRegistry.initMountBrain). Write access control
|
||||
* is the mount's own DB credential grants: a read-only role rejects
|
||||
* writes at the database; gbrain does not re-implement that client-side.
|
||||
* The AI gateway still configures from the HOST config (the caller's API
|
||||
* keys + model tiers) — embedding/expansion spend stays the caller's, and
|
||||
* a mount's DB-plane model config is never merged into the caller's gateway.
|
||||
*/
|
||||
async function connectMountEngine(brainId: string): Promise<BrainEngine> {
|
||||
const config = loadConfig();
|
||||
if (config) {
|
||||
const { configureGateway } = await import('./core/ai/gateway.ts');
|
||||
configureGateway(buildGatewayConfig(config));
|
||||
}
|
||||
const { loadRegistry } = await import('./core/brain-registry.ts');
|
||||
const handle = await loadRegistry().getBrain(brainId);
|
||||
activeBrainId = brainId;
|
||||
return handle.engine;
|
||||
}
|
||||
|
||||
async function connectEngine(opts?: { probeOnly?: boolean }): Promise<BrainEngine> {
|
||||
// Brain axis: resolve WHICH DATABASE this invocation targets before touching
|
||||
// the host engine. --brain (global flag) / GBRAIN_BRAIN_ID / .gbrain-mount /
|
||||
// mount-path-prefix resolve via the canonical 6-tier chain — the mirror of
|
||||
// the source axis in makeContext. connectEngine is the single choke point
|
||||
// every local CLI command routes through (shared ops, CLI-only commands,
|
||||
// and the search-dashboard path), so routing lands here once.
|
||||
const { resolveBrainId } = await import('./core/brain-resolver.ts');
|
||||
const brainId = resolveBrainId(getCliOptions().brain);
|
||||
if (brainId !== 'host') return connectMountEngine(brainId);
|
||||
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
console.error('No brain configured. Run: gbrain init');
|
||||
|
||||
+81
-4
@@ -2874,10 +2874,17 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
: await resolveSlugByPathOrSourcePath(engine, from, undefined);
|
||||
// The new path doesn't yet have a row, so resolve from path only.
|
||||
const newSlug = resolveSlugForPath(to);
|
||||
// #3056: the cheap rename is OBSERVED, not assumed. A zero-row UPDATE
|
||||
// doesn't throw, and a thrown collision used to be swallowed by an
|
||||
// empty catch — both fell through to importFile, which created/updated
|
||||
// the row at the new path while the old row stayed behind live. Both
|
||||
// shapes now fall through to the reconcile below.
|
||||
let renameApplied = false;
|
||||
try {
|
||||
await engine.updateSlug(oldSlug, newSlug, renameOpts);
|
||||
renameApplied = (await engine.updateSlug(oldSlug, newSlug, renameOpts)) > 0;
|
||||
} catch {
|
||||
// Slug doesn't exist or collision, treat as add
|
||||
// Destination slug occupied or invalid — treat as add; the reconcile
|
||||
// below removes the stale old row once the destination materialized.
|
||||
}
|
||||
// Reimport at new path (picks up content changes). Wrapped to match the
|
||||
// deletes/adds loops: a malformed renamed file is recorded to failedFiles
|
||||
@@ -2890,9 +2897,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
|
||||
// repo (committed symlink pointing out).
|
||||
const filePath = join(gitContextRoot, to);
|
||||
let importResult: Awaited<ReturnType<typeof importFile>> | undefined;
|
||||
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
|
||||
try {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
|
||||
importResult = result;
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
else if (result.status === 'skipped' && (result as { error?: string }).error) {
|
||||
failedFiles.push({ path: to, error: String((result as { error?: string }).error) });
|
||||
@@ -2901,9 +2910,68 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
failedFiles.push({ path: to, error: e instanceof Error ? e.message : String(e) });
|
||||
}
|
||||
}
|
||||
// #3056 reconcile: the rename fell back to add semantics, so the row
|
||||
// that still represents the OLD path is the stale half of the rename
|
||||
// (git reported the old path gone; a plain delete of that path would
|
||||
// remove this row). Two safety rails, both from the #3252 review:
|
||||
//
|
||||
// 1. Delete only after the destination demonstrably materialized —
|
||||
// `imported`, or an errorless `skipped` AT the new slug. Identity
|
||||
// dedup can skip against the OLD row (result.slug === oldSlug),
|
||||
// in which case nothing landed at newSlug and deleting the old
|
||||
// row would destroy the only copy.
|
||||
// 2. Locate the stale row POSITIVELY by `source_path = from`, never
|
||||
// by the oldSlug guess — after a collision, a path-derived
|
||||
// fallback slug could name an unrelated (e.g. manually curated)
|
||||
// row. No source_path match → nothing is deleted (this also means
|
||||
// code-strategy imports, which don't populate source_path, fall
|
||||
// back safely to leaving the old row rather than guessing).
|
||||
//
|
||||
// A failed delete records a `<rename:…>` SENTINEL (not an ordinary
|
||||
// path failure): the gate hard-blocks the bookmark, and — unlike a
|
||||
// plain path row — the auto-skip valve can never chronic-skip it after
|
||||
// N attempts, which would advance the bookmark and make a transient
|
||||
// delete outage a permanent duplicate. The sentinel clears through the
|
||||
// ordinary success path once the rename converges on a later run.
|
||||
let reconcileFailed = false;
|
||||
if (!renameApplied && importResult !== undefined) {
|
||||
const destMaterialized = importResult.status === 'imported' ||
|
||||
(importResult.status === 'skipped' && !importResult.error && importResult.slug === newSlug);
|
||||
if (destMaterialized) {
|
||||
try {
|
||||
const staleMap = await engine.resolveSlugsByPaths([from], { sourceId: opts.sourceId ?? DEFAULT_SOURCE_ID });
|
||||
const staleSlug = staleMap.get(from);
|
||||
if (staleSlug !== undefined && staleSlug !== newSlug) {
|
||||
await engine.deletePage(staleSlug, renameOpts);
|
||||
deletedSlugs.add(staleSlug); // never hand a deleted slug to auto-embed
|
||||
serr(` [sync] rename reconciled: removed stale row ${staleSlug} (${from} -> ${to} fell back to add).`);
|
||||
} else if (staleSlug === undefined) {
|
||||
serr(` [sync] rename fallback: no row has source_path ${from}; stale row (if any) left in place.`);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
reconcileFailed = true;
|
||||
failedFiles.push({
|
||||
path: `<rename:${to}>`,
|
||||
error: `rename reconcile failed (stale row for ${from} not removed): ` +
|
||||
`${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
serr(
|
||||
` [sync] rename fallback: ${from} -> ${to} did not materialize at ${newSlug} ` +
|
||||
`(import ${importResult.status}); old row left in place.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// Converged (cheap rename, clean reconcile, or nothing to reconcile):
|
||||
// clear any `<rename:…>` sentinel a previous failing run recorded.
|
||||
if (!reconcileFailed) succeededPaths.push(`<rename:${to}>`);
|
||||
pagesAffected.push(newSlug);
|
||||
deletedSlugs.delete(newSlug); // #1284: rename landed on a previously-deleted slug → embeddable again
|
||||
await markCompleted(to);
|
||||
// A failed reconcile must NOT checkpoint: banking `to` would make the
|
||||
// resume filter skip this rename on the retry run, turning a transient
|
||||
// delete failure into a permanent duplicate — the exact bug being fixed.
|
||||
if (!reconcileFailed) await markCompleted(to);
|
||||
progress.tick(1, newSlug);
|
||||
}
|
||||
progress.finish();
|
||||
@@ -3362,7 +3430,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
if (!gate.advanced) {
|
||||
const codeBreakdown = formatCodeBreakdown(failedFiles);
|
||||
if (gate.sentinelBlocked) {
|
||||
// Two sentinel classes block here: `<head>` (pin ancestry broken) and
|
||||
// `<rename:…>` (#3056 — a rename-reconcile delete failed and advancing
|
||||
// would permanently bank the duplicate). Pick the message by which fired.
|
||||
if (gate.sentinelBlocked && failedFiles.some(f => f.path === '<head>')) {
|
||||
serr(
|
||||
`\nSync blocked: repository history changed during sync (force-push / reset).\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
@@ -3370,6 +3441,12 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`a commit that doesn't match the indexed tree. Re-run sync to re-pin against ` +
|
||||
`current HEAD.`,
|
||||
);
|
||||
} else if (gate.sentinelBlocked) {
|
||||
serr(
|
||||
`\nSync blocked: a rename left a stale duplicate that could not be removed:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`The next 'gbrain sync' retries the reconcile from the same diff.`,
|
||||
);
|
||||
} else {
|
||||
const fileFailCount = failedFiles.filter(f => isSkippablePath(f.path)).length;
|
||||
serr(
|
||||
|
||||
@@ -29,6 +29,15 @@ export interface CliOptions {
|
||||
* the reranker. Has no effect on other commands.
|
||||
*/
|
||||
explain: boolean;
|
||||
/**
|
||||
* `--brain <id>` — which BRAIN (database) this invocation targets: 'host'
|
||||
* or a mount id from ~/.gbrain/mounts.json. Parsed here (stripped before
|
||||
* per-command parsing, like --source) so it can never collide with
|
||||
* per-op flag parsing. `null` = no explicit flag; connectEngine resolves
|
||||
* the ambient tiers (GBRAIN_BRAIN_ID / .gbrain-mount / mount-path / 'host')
|
||||
* via src/core/brain-resolver.ts.
|
||||
*/
|
||||
brain: string | null;
|
||||
}
|
||||
|
||||
export const DEFAULT_CLI_OPTIONS: CliOptions = {
|
||||
@@ -37,8 +46,29 @@ export const DEFAULT_CLI_OPTIONS: CliOptions = {
|
||||
progressInterval: 1000,
|
||||
timeoutMs: null,
|
||||
explain: false,
|
||||
brain: null,
|
||||
};
|
||||
|
||||
/**
|
||||
* Brain-id shape. Same regex as brain-registry's BRAIN_ID_RE (kept in sync;
|
||||
* brain-resolver.ts follows the same convention). 'host' matches. Validated
|
||||
* at parse time so an invalid id fails LOUDLY here — and so childGlobalFlags
|
||||
* can safely splice the value into execSync('gbrain ...') command strings.
|
||||
*/
|
||||
const BRAIN_ID_RE = /^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$/;
|
||||
|
||||
function parseBrainValue(val: string | undefined): string {
|
||||
if (val === undefined || val.length === 0 || val.startsWith('-')) {
|
||||
throw new Error('--brain requires a value (a mount id from `gbrain mounts list`, or "host").');
|
||||
}
|
||||
if (!BRAIN_ID_RE.test(val)) {
|
||||
throw new Error(
|
||||
`Invalid --brain value "${val}". Must match [a-z0-9-]{1,32}, start+end alphanumeric.`,
|
||||
);
|
||||
}
|
||||
return val;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse recognized global flags from the front / anywhere in argv and return
|
||||
* the resolved options plus the remaining argv (with global flags stripped).
|
||||
@@ -114,6 +144,20 @@ export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: s
|
||||
cliOpts.explain = true;
|
||||
continue;
|
||||
}
|
||||
// --brain <id> / --brain=<id> — brain (database) axis. Exact-match only:
|
||||
// `--brain-wide-max-cost-usd` (skillopt) and other `--brain-*` flags pass
|
||||
// through to per-command parsers untouched. A missing or malformed value
|
||||
// THROWS rather than falling through — a dropped --brain silently routes
|
||||
// to the wrong database (the exact bug class this flag's wiring fixes).
|
||||
if (a === '--brain') {
|
||||
cliOpts.brain = parseBrainValue(argv[i + 1]);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (a.startsWith('--brain=')) {
|
||||
cliOpts.brain = parseBrainValue(a.slice('--brain='.length));
|
||||
continue;
|
||||
}
|
||||
rest.push(a);
|
||||
}
|
||||
|
||||
@@ -204,6 +248,13 @@ export function childGlobalFlags(cliOpts?: CliOptions): string {
|
||||
if (opts.progressInterval !== DEFAULT_CLI_OPTIONS.progressInterval) {
|
||||
parts.push(`--progress-interval=${opts.progressInterval}`);
|
||||
}
|
||||
// Brain routing must survive into child `gbrain ...` subprocesses: the env
|
||||
// and dotfile tiers self-propagate (children inherit env + cwd), but an
|
||||
// explicit --brain does not — without this, a parent routed to a mount
|
||||
// spawns children that silently operate on the host brain. The value is
|
||||
// BRAIN_ID_RE-validated at parse time, so splicing it into an exec string
|
||||
// is safe.
|
||||
if (opts.brain) parts.push(`--brain=${opts.brain}`);
|
||||
return parts.length > 0 ? ' ' + parts.join(' ') : '';
|
||||
}
|
||||
|
||||
|
||||
+6
-1
@@ -1951,8 +1951,13 @@ export interface BrainEngine {
|
||||
* preserved via stable page_id). `opts.sourceId` scopes the UPDATE — without
|
||||
* it, the bare `WHERE slug = old` matches every row across every source and
|
||||
* would either rename them all OR violate the (source_id, slug) UNIQUE.
|
||||
*
|
||||
* Returns the number of rows moved. 0 means the old slug had no row in the
|
||||
* scoped source — an UPDATE that matches nothing does NOT throw, so callers
|
||||
* that need to know whether the rename actually happened (the sync rename
|
||||
* path, #3056) must check the return value rather than rely on the catch.
|
||||
*/
|
||||
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void>;
|
||||
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number>;
|
||||
rewriteLinks(oldSlug: string, newSlug: string): Promise<void>;
|
||||
|
||||
/**
|
||||
|
||||
@@ -5332,12 +5332,16 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// pages_with_timeline) and v0.10.3 graph layer (link_coverage, timeline_coverage,
|
||||
// most_connected). Both coexist: master's brain_score is the composite
|
||||
// dashboard, v0.10.3 metrics give entity-page-level granularity.
|
||||
// #1305: every page-scoped count here excludes soft-deleted rows — same
|
||||
// posture as getStats — so brain_score moves when the user deletes pages.
|
||||
// Chunk/link counts stay raw (storage until the purge phase), matching
|
||||
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
|
||||
const { rows: [h] } = await this.db.query(`
|
||||
WITH entity_pages AS (
|
||||
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
|
||||
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
|
||||
)
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
||||
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
||||
0 as stale_pages,
|
||||
@@ -5362,7 +5366,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT p.slug,
|
||||
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
|
||||
FROM pages p
|
||||
WHERE p.type IN ('entity', 'person', 'company')
|
||||
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
|
||||
ORDER BY link_count DESC
|
||||
LIMIT 5
|
||||
`);
|
||||
@@ -5381,6 +5385,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
|
||||
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
`);
|
||||
|
||||
const r = h as Record<string, unknown>;
|
||||
@@ -5475,15 +5480,18 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Sync
|
||||
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
|
||||
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
|
||||
newSlug = validateSlug(newSlug);
|
||||
const sourceId = opts?.sourceId ?? 'default';
|
||||
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
|
||||
// in sources B/C/D (mirrors postgres-engine.ts).
|
||||
await this.db.query(
|
||||
const result = await this.db.query(
|
||||
`UPDATE pages SET slug = $1, updated_at = now() WHERE slug = $2 AND source_id = $3`,
|
||||
[newSlug, oldSlug, sourceId]
|
||||
);
|
||||
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
|
||||
// the only way callers can see the no-op.
|
||||
return result.affectedRows ?? 0;
|
||||
}
|
||||
|
||||
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
|
||||
|
||||
@@ -5432,12 +5432,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
// no outbound links). The raw islanded list is filtered through the same
|
||||
// policy as `gbrain orphans` so convention pages do not count against
|
||||
// dashboard health.
|
||||
// #1305: every page-scoped count here excludes soft-deleted rows — same
|
||||
// posture as getStats — so brain_score moves when the user deletes pages.
|
||||
// Chunk/link counts stay raw (storage until the purge phase), matching
|
||||
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
|
||||
const [h] = await sql`
|
||||
WITH entity_pages AS (
|
||||
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
|
||||
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
|
||||
)
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
||||
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
||||
0 as stale_pages,
|
||||
@@ -5459,7 +5463,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
SELECT p.slug,
|
||||
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
|
||||
FROM pages p
|
||||
WHERE p.type IN ('entity', 'person', 'company')
|
||||
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
|
||||
ORDER BY link_count DESC
|
||||
LIMIT 5
|
||||
`;
|
||||
@@ -5478,6 +5482,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
|
||||
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
`;
|
||||
|
||||
const pageCount = Number(h.page_count);
|
||||
@@ -5569,14 +5574,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Sync
|
||||
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
|
||||
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
|
||||
newSlug = validateSlug(newSlug);
|
||||
const sql = this.sql;
|
||||
const sourceId = opts?.sourceId ?? 'default';
|
||||
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
|
||||
// in sources B/C/D (which would either rename them all OR fail the
|
||||
// (source_id, slug) UNIQUE if the new slug already exists in another source).
|
||||
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
|
||||
const result = await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
|
||||
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
|
||||
// the only way callers can see the no-op.
|
||||
return result.count ?? 0;
|
||||
}
|
||||
|
||||
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
|
||||
|
||||
@@ -48,6 +48,32 @@ import {
|
||||
|
||||
export const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
|
||||
/**
|
||||
* Which detail levels get the compiled_truth boost (#3430).
|
||||
*
|
||||
* ONLY `low`. The documented contract (`src/core/operations.ts`) is
|
||||
* "low (compiled truth only), medium (default, all with dedup), high (all
|
||||
* chunks)" — so `low` is the level that privileges compiled truth, and both
|
||||
* `medium` and `high` are supposed to see everything on equal footing.
|
||||
*
|
||||
* This was previously spelled `detail !== 'high'`, i.e. written as though
|
||||
* `high` were the special case. Because COMPILED_TRUTH_BOOST is applied AFTER
|
||||
* RRF normalization, and RRF's whole range over a 100-deep pool is 1/60 → 1/160,
|
||||
* a 2.0x multiplier is not a tilt — break-even is `2/(60+r) >= 1/60`, so any
|
||||
* boosted chunk inside the first 60 ranks outranks an unboosted rank-1 chunk.
|
||||
* At the default detail that made search categorically compiled-truth-only:
|
||||
* a page whose answer lived in a `fenced_code` chunk returned the prose chunk,
|
||||
* and the code chunk fell out of the window entirely.
|
||||
*
|
||||
* Extracted as a named predicate rather than left inline at three call sites so
|
||||
* the detail→boost mapping is directly testable. An inline expression can only
|
||||
* be covered through a full `hybridSearch` round trip, which is why the
|
||||
* original inversion went unnoticed.
|
||||
*/
|
||||
export function shouldBoostCompiledTruth(detail: string | null | undefined): boolean {
|
||||
return detail === 'low';
|
||||
}
|
||||
const pendingCacheWrites = new Set<Promise<unknown>>();
|
||||
|
||||
/**
|
||||
@@ -1169,7 +1195,7 @@ export async function hybridSearch(
|
||||
const noEmbedLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
|
||||
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
|
||||
noEmbedResults = rrfFusionWeighted(noEmbedLists, shouldBoostCompiledTruth(detailResolved));
|
||||
}
|
||||
if (noEmbedResults.length > 0) {
|
||||
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
|
||||
@@ -1413,7 +1439,7 @@ export async function hybridSearch(
|
||||
const fallbackLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
|
||||
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
|
||||
fallbackResults = rrfFusionWeighted(fallbackLists, shouldBoostCompiledTruth(detail));
|
||||
}
|
||||
if (fallbackResults.length > 0) {
|
||||
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
|
||||
@@ -1500,7 +1526,7 @@ export async function hybridSearch(
|
||||
// arms BEFORE fusion so the compiled-truth authority boost skips them.
|
||||
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
|
||||
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
let fused = rrfFusionWeighted(allLists, shouldBoostCompiledTruth(detail));
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive.
|
||||
// v0.36 (D9): hydrate from the active embedding column so rescore happens
|
||||
|
||||
@@ -766,7 +766,7 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// written between the #3391 stale-fix (which changes which chunks count as
|
||||
// current) and the operator's migration run. Same one-time global cold-miss
|
||||
// pattern as the bumps above.
|
||||
export const KNOBS_HASH_VERSION = 13;
|
||||
export const KNOBS_HASH_VERSION = 14;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
/**
|
||||
* `--brain <id>` must actually route to the named mounted brain.
|
||||
*
|
||||
* The bug: docs/architecture/brains-and-sources.md promises
|
||||
* `gbrain query "X" --brain media-team` runs against the team's DB, and
|
||||
* src/core/brain-resolver.ts implements the full 6-tier chain — but nothing
|
||||
* ever CALLED the resolver from the CLI dispatch path. `--brain media-team`
|
||||
* was silently ignored (unknown flag) and the command ran against the HOST
|
||||
* brain, returning confident wrong answers. Same silent-wrong-target class
|
||||
* as #1712/#3524 on the source axis.
|
||||
*
|
||||
* These tests spawn the real CLI against a fake home with two distinct
|
||||
* PGLite brains (host + one mount), each seeded with a uniquely-slugged
|
||||
* page, and assert on WHICH brain's data comes back:
|
||||
* - control: no flag → host page (default unchanged);
|
||||
* - `--brain team-a` → the mount's page, not the host's;
|
||||
* - `--brain nope` (unregistered) → hard error, NOT a silent host fallback;
|
||||
* - `GBRAIN_BRAIN_ID=team-a` → the mount's page (env tier wired too).
|
||||
*
|
||||
* Pre-fix, the --brain/env spawns list the HOST page and the unknown-brain
|
||||
* spawn exits 0 — all three fail behaviorally on an unfixed tree.
|
||||
*
|
||||
* Serial because it spawns subprocesses + writes tmpdirs.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
const REPO = new URL('..', import.meta.url).pathname.replace(/\/$/, '');
|
||||
|
||||
let home: string;
|
||||
let mountsPath: string;
|
||||
|
||||
async function seedBrain(databasePath: string, slug: string): Promise<void> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite', database_path: databasePath });
|
||||
await engine.initSchema();
|
||||
await engine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: slug,
|
||||
compiled_truth: `content of ${slug}`,
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.disconnect();
|
||||
}
|
||||
|
||||
function cliEnv(extra: Record<string, string> = {}): Record<string, string> {
|
||||
return {
|
||||
...process.env as Record<string, string>,
|
||||
HOME: home,
|
||||
GBRAIN_HOME: home,
|
||||
GBRAIN_MOUNTS_PATH: mountsPath,
|
||||
GBRAIN_SKIP_STARTUP_HOOKS: '1',
|
||||
// Neutralize ambient routing signals from the invoking shell/CI.
|
||||
GBRAIN_BRAIN_ID: '',
|
||||
GBRAIN_SOURCE: '',
|
||||
GBRAIN_DATABASE_URL: '',
|
||||
DATABASE_URL: '',
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
|
||||
async function runCli(
|
||||
args: string[],
|
||||
env: Record<string, string>,
|
||||
timeoutMs = 90_000,
|
||||
): Promise<{ exitCode: number; stdout: string; stderr: string }> {
|
||||
const proc = Bun.spawn(['bun', 'run', `${REPO}/src/cli.ts`, ...args], {
|
||||
cwd: REPO,
|
||||
env,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const killer = setTimeout(() => {
|
||||
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
}, timeoutMs);
|
||||
try {
|
||||
const [stdout, stderr, exitCode] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
]);
|
||||
return { exitCode, stdout, stderr };
|
||||
} finally {
|
||||
clearTimeout(killer);
|
||||
}
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
home = mkdtempSync(join(tmpdir(), 'gbrain-brain-flag-'));
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
mkdirSync(join(home, 'team-a-clone'), { recursive: true });
|
||||
|
||||
const hostDb = join(home, '.gbrain', 'brain.pglite');
|
||||
const teamDb = join(home, 'team-a.pglite');
|
||||
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({ engine: 'pglite', database_path: hostDb, embedding_dimensions: 1536 }) + '\n',
|
||||
);
|
||||
mountsPath = join(home, '.gbrain', 'mounts.json');
|
||||
writeFileSync(
|
||||
mountsPath,
|
||||
JSON.stringify({
|
||||
version: 1,
|
||||
mounts: [
|
||||
{
|
||||
id: 'team-a',
|
||||
path: join(home, 'team-a-clone'),
|
||||
engine: 'pglite',
|
||||
database_path: teamDb,
|
||||
enabled: true,
|
||||
},
|
||||
],
|
||||
}) + '\n',
|
||||
);
|
||||
|
||||
// Two brains, two distinct pages. WHICH slug comes back tells us WHICH
|
||||
// database the CLI actually queried.
|
||||
await seedBrain(hostDb, 'host-page');
|
||||
await seedBrain(teamDb, 'team-page');
|
||||
}, 240_000);
|
||||
|
||||
afterAll(() => {
|
||||
try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||
});
|
||||
|
||||
describe('--brain routes the CLI to the named mounted brain', () => {
|
||||
test('control: no brain signal → host brain (default unchanged)', async () => {
|
||||
const r = await runCli(['list'], cliEnv());
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stdout).toContain('host-page');
|
||||
expect(r.stdout).not.toContain('team-page');
|
||||
}, 120_000);
|
||||
|
||||
test('--brain team-a → the mount database, not host', async () => {
|
||||
const r = await runCli(['list', '--brain', 'team-a'], cliEnv());
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stdout).toContain('team-page');
|
||||
expect(r.stdout).not.toContain('host-page');
|
||||
}, 120_000);
|
||||
|
||||
test('--brain <unknown> hard-errors — never a silent host fallback', async () => {
|
||||
const r = await runCli(['list', '--brain', 'nope'], cliEnv());
|
||||
expect(r.exitCode).not.toBe(0);
|
||||
expect(r.stdout + r.stderr).toMatch(/Unknown brain/i);
|
||||
// The silent-wrong-results bug: pre-fix this listed the host's pages.
|
||||
expect(r.stdout).not.toContain('host-page');
|
||||
}, 120_000);
|
||||
|
||||
test('GBRAIN_BRAIN_ID=team-a env tier is wired through the same seam', async () => {
|
||||
const r = await runCli(['list'], cliEnv({ GBRAIN_BRAIN_ID: 'team-a' }));
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stdout).toContain('team-page');
|
||||
expect(r.stdout).not.toContain('host-page');
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
// ── Trust boundary: brain selection is NEVER caller-controlled ────────────
|
||||
//
|
||||
// Brain routing happens at engine-connect time in the local CLI process
|
||||
// (trusted, remote === false). An untrusted caller over MCP must have no way
|
||||
// to name a brain: no op declares a brain param, and neither context builder
|
||||
// reads one from params. Fail-closed pins for the new surface.
|
||||
|
||||
describe('untrusted callers cannot cross brains', () => {
|
||||
test('no operation exposes a brain/brain_id param an MCP caller could set', async () => {
|
||||
const { operations } = await import('../src/core/operations.ts');
|
||||
for (const op of operations) {
|
||||
expect(`${op.name}:${'brain' in op.params}`).toBe(`${op.name}:false`);
|
||||
expect(`${op.name}:${'brain_id' in op.params}`).toBe(`${op.name}:false`);
|
||||
}
|
||||
});
|
||||
|
||||
test('makeContext ignores caller-supplied params.brain (stays on the connected engine)', async () => {
|
||||
const { makeContext } = await import('../src/cli.ts');
|
||||
const stub = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async () => [],
|
||||
getConfig: async () => null,
|
||||
} as any;
|
||||
const ctx = await makeContext(stub, { brain: 'team-a', brain_id: 'team-a' });
|
||||
expect(ctx.engine).toBe(stub);
|
||||
// Local process default is the host brain; params must not move it.
|
||||
expect(ctx.brainId ?? 'host').toBe('host');
|
||||
});
|
||||
|
||||
test('remote dispatch context never derives a brain from params (fail-closed)', async () => {
|
||||
const { buildOperationContext } = await import('../src/mcp/dispatch.ts');
|
||||
const stub = { kind: 'pglite' } as any;
|
||||
const ctx = buildOperationContext(stub, { brain: 'team-a', brain_id: 'team-a' }, {
|
||||
remote: true,
|
||||
sourceId: 'default',
|
||||
});
|
||||
expect(ctx.engine).toBe(stub);
|
||||
expect(ctx.brainId).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -65,7 +65,7 @@ describe('parseGlobalFlags', () => {
|
||||
|
||||
test('all global flags combined', () => {
|
||||
const r = parseGlobalFlags(['--quiet', '--progress-json', '--progress-interval=250', 'sync']);
|
||||
expect(r.cliOpts).toEqual({ quiet: true, progressJson: true, progressInterval: 250, timeoutMs: null, explain: false });
|
||||
expect(r.cliOpts).toEqual({ quiet: true, progressJson: true, progressInterval: 250, timeoutMs: null, explain: false, brain: null });
|
||||
expect(r.rest).toEqual(['sync']);
|
||||
});
|
||||
|
||||
@@ -96,7 +96,7 @@ describe('getCliOptions / setCliOptions singleton', () => {
|
||||
|
||||
test('setCliOptions applies + getCliOptions returns a copy', () => {
|
||||
_resetCliOptionsForTest();
|
||||
setCliOptions({ quiet: false, progressJson: true, progressInterval: 250, timeoutMs: null, explain: false });
|
||||
setCliOptions({ quiet: false, progressJson: true, progressInterval: 250, timeoutMs: null, explain: false, brain: null });
|
||||
expect(getCliOptions().progressJson).toBe(true);
|
||||
expect(getCliOptions().progressInterval).toBe(250);
|
||||
});
|
||||
@@ -156,12 +156,12 @@ describe('CLI integration: progress streams to the right channel', () => {
|
||||
|
||||
describe('cliOptsToProgressOptions', () => {
|
||||
test('--quiet → quiet mode', () => {
|
||||
const opts = cliOptsToProgressOptions({ quiet: true, progressJson: false, progressInterval: 1000, timeoutMs: null, explain: false });
|
||||
const opts = cliOptsToProgressOptions({ quiet: true, progressJson: false, progressInterval: 1000, timeoutMs: null, explain: false, brain: null });
|
||||
expect(opts.mode).toBe('quiet');
|
||||
});
|
||||
|
||||
test('--progress-json → json mode with interval', () => {
|
||||
const opts = cliOptsToProgressOptions({ quiet: false, progressJson: true, progressInterval: 500, timeoutMs: null, explain: false });
|
||||
const opts = cliOptsToProgressOptions({ quiet: false, progressJson: true, progressInterval: 500, timeoutMs: null, explain: false, brain: null });
|
||||
expect(opts.mode).toBe('json');
|
||||
expect(opts.minIntervalMs).toBe(500);
|
||||
});
|
||||
@@ -173,7 +173,7 @@ describe('cliOptsToProgressOptions', () => {
|
||||
});
|
||||
|
||||
test('quiet takes priority over progressJson', () => {
|
||||
const opts = cliOptsToProgressOptions({ quiet: true, progressJson: true, progressInterval: 1000, timeoutMs: null, explain: false });
|
||||
const opts = cliOptsToProgressOptions({ quiet: true, progressJson: true, progressInterval: 1000, timeoutMs: null, explain: false, brain: null });
|
||||
expect(opts.mode).toBe('quiet');
|
||||
});
|
||||
});
|
||||
@@ -224,3 +224,54 @@ describe('--timeout flag', () => {
|
||||
expect(r.cliOpts.timeoutMs).toBe(null);
|
||||
});
|
||||
});
|
||||
|
||||
describe('--brain flag (brain axis routing)', () => {
|
||||
test('--brain <id> space form: parsed + stripped from rest', () => {
|
||||
const r = parseGlobalFlags(['query', 'X', '--brain', 'media-team']);
|
||||
expect(r.cliOpts.brain).toBe('media-team');
|
||||
expect(r.rest).toEqual(['query', 'X']);
|
||||
});
|
||||
|
||||
test('--brain=<id> equals form: parsed + stripped from rest', () => {
|
||||
const r = parseGlobalFlags(['--brain=media-team', 'query', 'X']);
|
||||
expect(r.cliOpts.brain).toBe('media-team');
|
||||
expect(r.rest).toEqual(['query', 'X']);
|
||||
});
|
||||
|
||||
test('--brain host is a valid explicit value', () => {
|
||||
const r = parseGlobalFlags(['stats', '--brain', 'host']);
|
||||
expect(r.cliOpts.brain).toBe('host');
|
||||
});
|
||||
|
||||
test('missing value throws (loud, never a silent host fallback)', () => {
|
||||
expect(() => parseGlobalFlags(['query', 'X', '--brain'])).toThrow(/--brain requires a value/);
|
||||
expect(() => parseGlobalFlags(['--brain=', 'query'])).toThrow(/--brain requires a value/);
|
||||
// A following flag is not a value.
|
||||
expect(() => parseGlobalFlags(['--brain', '--quiet'])).toThrow(/--brain requires a value/);
|
||||
});
|
||||
|
||||
test('malformed id throws (validated at parse time)', () => {
|
||||
expect(() => parseGlobalFlags(['--brain', 'Bad_Id!'])).toThrow(/Invalid --brain value/);
|
||||
expect(() => parseGlobalFlags(['--brain=$(rm -rf /)'])).toThrow(/Invalid --brain value/);
|
||||
});
|
||||
|
||||
test('--brain-* per-command flags pass through untouched (skillopt collision guard)', () => {
|
||||
const r = parseGlobalFlags(['skillopt', '--brain-wide-max-cost-usd', '5']);
|
||||
expect(r.cliOpts.brain).toBe(null);
|
||||
expect(r.rest).toEqual(['skillopt', '--brain-wide-max-cost-usd', '5']);
|
||||
});
|
||||
|
||||
test('default brain is null (ambient resolution applies)', () => {
|
||||
const r = parseGlobalFlags(['query', 'X']);
|
||||
expect(r.cliOpts.brain).toBe(null);
|
||||
});
|
||||
});
|
||||
|
||||
describe('childGlobalFlags propagates --brain', () => {
|
||||
test('explicit brain rides into child gbrain subprocess commands', async () => {
|
||||
const { childGlobalFlags } = await import('../src/core/cli-options.ts');
|
||||
expect(childGlobalFlags({ ...DEFAULT_CLI_OPTIONS, brain: 'media-team' }))
|
||||
.toContain('--brain=media-team');
|
||||
expect(childGlobalFlags({ ...DEFAULT_CLI_OPTIONS })).not.toContain('--brain');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => {
|
||||
test('KNOBS_HASH_VERSION is 14 (cross-modal still appended; 13→14 compiled_truth boost scope #3430)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
@@ -146,7 +146,8 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
|
||||
// finally reaches asymmetric providers — pre-fix rows were keyed on
|
||||
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
// #3430: 13→14 compiled_truth boost no longer applies at detail=medium.
|
||||
expect(KNOBS_HASH_VERSION).toBe(14);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -41,7 +41,7 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null, brain: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
|
||||
@@ -47,7 +47,7 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null, brain: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
|
||||
@@ -46,7 +46,7 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null, brain: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
|
||||
@@ -73,7 +73,7 @@ beforeAll(async () => {
|
||||
await engine.initSchema();
|
||||
// Default CLI options (quiet enough that the progress reporter doesn't
|
||||
// pollute the capture buffer beyond what the assertions need).
|
||||
setCliOptions({ quiet: false, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
setCliOptions({ quiet: false, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null, brain: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* #1305 — getHealth() must exclude soft-deleted pages from every
|
||||
* page-scoped count, the same posture getStats() has had since v0.26.5.
|
||||
*
|
||||
* Pre-fix, getHealth counted raw `pages` rows: page_count and orphan_pages
|
||||
* included soft-deleted pages, the entity_pages CTE kept deleted entities in
|
||||
* the link/timeline coverage denominators and in most_connected, and
|
||||
* brain_score therefore never moved when a user soft-deleted pages.
|
||||
*
|
||||
* Boundary (deliberate): chunk- and link-scoped counts (embed_coverage,
|
||||
* missing_embeddings, link_count, dead_links) stay RAW — they occupy storage
|
||||
* until the autopilot purge phase, matching getStats. Destructive-removal
|
||||
* counts (purge paths, #2235) also deliberately count all rows and are
|
||||
* untouched here.
|
||||
*
|
||||
* Runs against PGLite — the fixed SQL shapes are identical in both engines.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
for (const t of ['links', 'content_chunks', 'timeline_entries', 'tags', 'page_versions', 'pages']) {
|
||||
await (engine as any).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
});
|
||||
|
||||
async function seedNote(slug: string): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `content of ${slug}`, frontmatter: {} });
|
||||
}
|
||||
|
||||
async function pageId(slug: string): Promise<number> {
|
||||
return (await (engine as any).db.query(`SELECT id FROM pages WHERE slug=$1`, [slug])).rows[0].id;
|
||||
}
|
||||
|
||||
describe('#1305 — getHealth excludes soft-deleted pages', () => {
|
||||
test('page_count and orphan_pages match getStats after soft-delete (the issue repro)', async () => {
|
||||
for (let i = 0; i < 10; i++) await seedNote(`wiki/note-${i}`);
|
||||
for (let i = 0; i < 6; i++) await engine.softDeletePage(`wiki/note-${i}`);
|
||||
|
||||
const stats = await engine.getStats();
|
||||
const health = await engine.getHealth();
|
||||
expect(stats.page_count).toBe(4);
|
||||
// Pre-fix: 10 (raw rows). getHealth must agree with getStats.
|
||||
expect(health.page_count).toBe(4);
|
||||
// Pre-fix: 10 — deleted pages stayed in the islanded scan.
|
||||
expect(health.orphan_pages).toBe(4);
|
||||
});
|
||||
|
||||
test('brain_score moves when the user soft-deletes the islanded pages', async () => {
|
||||
// 2 connected pages + 8 islanded ones.
|
||||
await seedNote('wiki/hub');
|
||||
await seedNote('wiki/leaf');
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
|
||||
[await pageId('wiki/hub'), await pageId('wiki/leaf')],
|
||||
);
|
||||
for (let i = 0; i < 8; i++) await seedNote(`wiki/clutter-${i}`);
|
||||
|
||||
const before = await engine.getHealth();
|
||||
for (let i = 0; i < 8; i++) await engine.softDeletePage(`wiki/clutter-${i}`);
|
||||
const after = await engine.getHealth();
|
||||
|
||||
// Pre-fix both assertions fail: orphan_pages stayed 8 and brain_score
|
||||
// was byte-identical before/after the delete.
|
||||
expect(after.orphan_pages).toBe(0);
|
||||
expect(after.brain_score).toBeGreaterThan(before.brain_score);
|
||||
});
|
||||
|
||||
test('entity coverage denominators and most_connected exclude deleted entities', async () => {
|
||||
// Live entity: inbound link + timeline entry → full coverage.
|
||||
await engine.putPage('people/alice-example', { type: 'person', title: 'Alice', compiled_truth: 'a person', frontmatter: {} });
|
||||
await engine.putPage('people/bob-example', { type: 'person', title: 'Bob', compiled_truth: 'another person', frontmatter: {} });
|
||||
await seedNote('wiki/mentions-alice');
|
||||
const aliceId = await pageId('people/alice-example');
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
|
||||
[await pageId('wiki/mentions-alice'), aliceId],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, summary) VALUES ($1, '2026-01-01', 'met alice')`,
|
||||
[aliceId],
|
||||
);
|
||||
|
||||
await engine.softDeletePage('people/bob-example');
|
||||
const h = await engine.getHealth();
|
||||
|
||||
// Pre-fix: bob stayed in the entity_pages CTE → coverage 0.5 each,
|
||||
// and bob appeared in most_connected.
|
||||
expect(h.link_coverage).toBe(1);
|
||||
expect(h.timeline_coverage).toBe(1);
|
||||
expect(h.most_connected.map((c) => c.slug)).not.toContain('people/bob-example');
|
||||
});
|
||||
|
||||
test('chunk storage counts stay raw (the deliberate boundary)', async () => {
|
||||
await seedNote('wiki/kept');
|
||||
await seedNote('wiki/gone');
|
||||
for (const slug of ['wiki/kept', 'wiki/gone']) {
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text) VALUES ($1, 0, 'chunk')`,
|
||||
[await pageId(slug)],
|
||||
);
|
||||
}
|
||||
await engine.softDeletePage('wiki/gone');
|
||||
|
||||
const h = await engine.getHealth();
|
||||
// Soft-deleted pages' chunks still occupy storage until purge; the
|
||||
// missing_embeddings count keeps seeing them, same as getStats.
|
||||
expect(h.missing_embeddings).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('is 13 (12→13 embedding-provider migration invalidates rows written against the prior embedding space, #3390)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
it('is 14 (13→14 compiled_truth boost no longer applies at detail=medium, so pre-fix rankings must be unreachable, #3430)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(14);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* #3430: the compiled_truth boost must not apply at `detail=medium`.
|
||||
*
|
||||
* `COMPILED_TRUTH_BOOST = 2.0` is applied AFTER RRF score normalization. RRF's
|
||||
* entire dynamic range over a 100-deep pool is 1/60 → 1/160 (a factor of 2.67),
|
||||
* so a 2.0x multiplier consumes roughly three quarters of it. Break-even is
|
||||
* `2/(60+r) >= 1/60`, i.e. r <= 60 — so ANY boosted chunk in the first 60 ranks
|
||||
* outranks an unboosted rank-1 chunk. That is a categorical filter, not a tilt:
|
||||
* a page whose actual answer is in a `fenced_code` chunk returns the prose
|
||||
* chunk instead, and the code chunk leaves the result window entirely.
|
||||
*
|
||||
* The gate was written as `detail !== 'high'` — "high is special" — but the
|
||||
* documented contract in `src/core/operations.ts` is:
|
||||
*
|
||||
* low (compiled truth only), medium (default, all with dedup), high (all chunks)
|
||||
*
|
||||
* which makes LOW the special one. `low` already restricts to compiled_truth,
|
||||
* so a boost there is a no-op among equals; `medium` and `high` are both
|
||||
* supposed to see everything. Hence `detail === 'low'`.
|
||||
*
|
||||
* These tests pin the arithmetic, not the constant — they would still fail if
|
||||
* someone reintroduced a boost at medium with a different multiplier or behind
|
||||
* a score floor, which is why they assert final RANK rather than score.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { rrfFusion, RRF_K, shouldBoostCompiledTruth } from '../src/core/search/hybrid.ts';
|
||||
import { KNOBS_HASH_VERSION } from '../src/core/search/mode.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function chunk(slug: string, chunkSource: string): SearchResult {
|
||||
return { slug, chunk_source: chunkSource, chunk_text: 'x', title: slug, score: 0 } as unknown as SearchResult;
|
||||
}
|
||||
|
||||
/** One vector arm: the correct answer at rank 0, then `n` compiled_truth chunks. */
|
||||
function poolWithAnswerFirst(n: number): SearchResult[] {
|
||||
const list = [chunk('code/answer', 'fenced_code')];
|
||||
for (let i = 0; i < n; i++) list.push(chunk(`prose/p${i}`, 'compiled_truth'));
|
||||
return list;
|
||||
}
|
||||
|
||||
function rankOfAnswer(results: SearchResult[]): number {
|
||||
return results.findIndex((r) => r.slug === 'code/answer');
|
||||
}
|
||||
|
||||
describe('#3430: the detail→boost mapping itself', () => {
|
||||
// These are the assertions that actually FAIL on master. The rrfFusion tests
|
||||
// below pin the arithmetic but pass either way, because they pass the boost
|
||||
// flag explicitly — they cannot see how hybridSearch decides it. This is the
|
||||
// wiring.
|
||||
test('ONLY detail=low boosts compiled_truth', () => {
|
||||
expect(shouldBoostCompiledTruth('low')).toBe(true);
|
||||
expect(shouldBoostCompiledTruth('medium')).toBe(false);
|
||||
expect(shouldBoostCompiledTruth('high')).toBe(false);
|
||||
});
|
||||
|
||||
test('an absent detail does not boost — medium is the documented default', () => {
|
||||
// Callers that omit detail get medium semantics, so the unset case must
|
||||
// match medium, not low. A `!== 'high'` spelling gets this backwards.
|
||||
expect(shouldBoostCompiledTruth(undefined)).toBe(false);
|
||||
expect(shouldBoostCompiledTruth(null)).toBe(false);
|
||||
});
|
||||
|
||||
test('an unrecognized detail value does not boost', () => {
|
||||
// Fail-open toward showing everything rather than silently filtering.
|
||||
expect(shouldBoostCompiledTruth('')).toBe(false);
|
||||
expect(shouldBoostCompiledTruth('LOW')).toBe(false);
|
||||
expect(shouldBoostCompiledTruth('detailed')).toBe(false);
|
||||
});
|
||||
|
||||
test('the cache version was bumped so pre-fix rankings are unreachable', () => {
|
||||
// Results are cached AFTER fusion, so rows written under the old boost
|
||||
// semantics would otherwise be served under the new ones for the whole TTL.
|
||||
// 13 was the pre-fix value.
|
||||
expect(KNOBS_HASH_VERSION).toBeGreaterThanOrEqual(14);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3430: compiled_truth boost scope', () => {
|
||||
test('boost OFF (detail=medium/high) keeps the vector-ranked answer at rank 0', () => {
|
||||
// The regression this file exists for. Pre-fix, medium passed applyBoost=true
|
||||
// and the answer landed at rank n — outside a 20-result window for n >= 20.
|
||||
for (const n of [10, 20, 40, 80]) {
|
||||
const fused = rrfFusion([poolWithAnswerFirst(n)], RRF_K, false);
|
||||
expect(rankOfAnswer(fused), `n=${n}: answer must stay first without the boost`).toBe(0);
|
||||
}
|
||||
});
|
||||
|
||||
test('boost ON demonstrates the categorical displacement it causes', () => {
|
||||
// Documents WHY the boost cannot be on at medium. Not an endorsement of
|
||||
// these numbers — a characterization of the mechanism, so a future reader
|
||||
// sees the cost rather than re-deriving it.
|
||||
const observed = [10, 20, 40].map((n) => ({
|
||||
n,
|
||||
rank: rankOfAnswer(rrfFusion([poolWithAnswerFirst(n)], RRF_K, true)),
|
||||
}));
|
||||
// Displacement scales with pool composition: the answer is pushed back by
|
||||
// roughly one position per boosted chunk ahead of the break-even rank.
|
||||
for (const { n, rank } of observed) {
|
||||
expect(rank, `n=${n}: boosted chunks should displace the answer`).toBeGreaterThan(0);
|
||||
}
|
||||
// And past ~20 compiled_truth chunks it leaves a default-size window.
|
||||
expect(observed.find((o) => o.n === 20)!.rank).toBeGreaterThanOrEqual(20);
|
||||
});
|
||||
|
||||
test('with the boost off, compiled_truth still wins when the vector arm ranks it first', () => {
|
||||
// Guard against over-correcting: removing the boost must not penalize
|
||||
// compiled_truth, only stop privileging it.
|
||||
const list = [chunk('prose/answer', 'compiled_truth'), chunk('code/other', 'fenced_code')];
|
||||
const fused = rrfFusion([list], RRF_K, false);
|
||||
expect(fused[0].slug).toBe('prose/answer');
|
||||
});
|
||||
});
|
||||
@@ -413,7 +413,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// #3390/#3391: bumped 12→13 for the embedding-provider migration wave —
|
||||
// legacy callers hash prov=default before AND after a provider swap, so
|
||||
// pre-migration cache rows must become unreachable on upgrade.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
// v0.42.67.x bumped 13→14: the compiled_truth boost no longer applies at
|
||||
// detail=medium (#3430). Cached rows were ranked under the old semantics,
|
||||
// so they must become unreachable rather than be served under the new ones.
|
||||
expect(KNOBS_HASH_VERSION).toBe(14);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -578,8 +581,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION is 13 (12→13 embedding-migration wave, #3390/#3391)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
test('KNOBS_HASH_VERSION is 14 (13→14 compiled_truth boost scope fix, #3430)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(14);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -64,7 +64,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// pre-fix document-side query vectors must not be served.
|
||||
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
|
||||
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
// #3430: 13→14 — the compiled_truth boost no longer applies at
|
||||
// detail=medium. Results are cached after fusion, so rows ranked under
|
||||
// the old boost semantics must not be served under the new ones.
|
||||
expect(KNOBS_HASH_VERSION).toBe(14);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
/**
|
||||
* #3056 — sync rename path: a failed `updateSlug` must not leave a live
|
||||
* duplicate of the renamed page behind.
|
||||
*
|
||||
* Before the fix, the rename loop swallowed `updateSlug` failures with an
|
||||
* empty catch ("treat as add") and could not see a zero-row UPDATE at all
|
||||
* (updateSlug returned void). The run then fell through to importFile,
|
||||
* which created/updated the row at the new path — while the old row stayed
|
||||
* behind, live, with its slug occupied. Nothing was logged, no counter
|
||||
* moved, and the duplicate was permanent.
|
||||
*
|
||||
* The fix reconciles: when the cheap rename didn't move a row AND the
|
||||
* destination demonstrably materialized, the stale old row is located
|
||||
* positively by `source_path = from` and deleted. Two safety rails:
|
||||
*
|
||||
* - dedup-skip protection: identity dedup can skip the import against
|
||||
* the OLD row, in which case nothing landed at the destination and
|
||||
* deleting the old row would destroy the only copy — no reconcile.
|
||||
* - no slug-guess deletes: the stale row is found by source_path only;
|
||||
* an unrelated row that happens to sit at the guessed slug survives.
|
||||
*
|
||||
* A failed reconcile delete lands in failedFiles so the existing failure
|
||||
* gate blocks the bookmark and the next run retries the same rename diff.
|
||||
*/
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { execSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const repos: string[] = [];
|
||||
// Serial-file requirement: blocked runs write real rows to the sync-failure
|
||||
// ledger under the gbrain home — isolate it per test so the operator's
|
||||
// actual ledger is never touched (GBRAIN_HOME is the isolation lever;
|
||||
// process.env.HOME does not redirect Bun's os.homedir()).
|
||||
let tmpHome: string;
|
||||
const originalGbrainHome = process.env.GBRAIN_HOME;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-3056-home-'));
|
||||
process.env.GBRAIN_HOME = tmpHome;
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalGbrainHome !== undefined) process.env.GBRAIN_HOME = originalGbrainHome;
|
||||
else delete process.env.GBRAIN_HOME;
|
||||
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
while (repos.length) {
|
||||
const d = repos.pop();
|
||||
if (d) rmSync(d, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function personMd(title: string, body: string): string {
|
||||
return ['---', 'type: person', `title: ${title}`, '---', '', body].join('\n');
|
||||
}
|
||||
|
||||
/** Create a temp git repo seeded with the given files + an initial commit. */
|
||||
function mkRepo(files: Record<string, string>): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-3056-'));
|
||||
repos.push(dir);
|
||||
execSync('git init', { cwd: dir, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: dir, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: dir, stdio: 'pipe' });
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
mkdirSync(join(dir, rel, '..'), { recursive: true });
|
||||
writeFileSync(join(dir, rel), content);
|
||||
}
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: dir, stdio: 'pipe' });
|
||||
return dir;
|
||||
}
|
||||
|
||||
const SYNC_OPTS = { noPull: true, noEmbed: true, noExtract: true, sourceId: 'default' } as const;
|
||||
|
||||
async function countPages(): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ n: number | string }>(
|
||||
`SELECT count(*)::int AS n FROM pages WHERE source_id = 'default'`,
|
||||
);
|
||||
return Number(rows[0]?.n ?? 0);
|
||||
}
|
||||
|
||||
describe('updateSlug engine contract (#3056)', () => {
|
||||
test('returns 1 when the old slug row is moved', async () => {
|
||||
await engine.putPage('people/old', {
|
||||
type: 'person', title: 'Old', compiled_truth: 'body',
|
||||
}, { sourceId: 'default' });
|
||||
const moved = await engine.updateSlug('people/old', 'people/new', { sourceId: 'default' });
|
||||
expect(moved).toBe(1);
|
||||
expect(await engine.getPage('people/new')).not.toBeNull();
|
||||
});
|
||||
|
||||
test('returns 0 when the old slug has no row (the silent no-op case)', async () => {
|
||||
const moved = await engine.updateSlug('people/ghost', 'people/new', { sourceId: 'default' });
|
||||
expect(moved).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3056: rename fallback reconciles the stale old row', () => {
|
||||
test('collision: destination slug occupied → stale old row deleted after import lands', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(await engine.getPage('people/carol')).not.toBeNull();
|
||||
|
||||
// A pre-existing row already occupies the rename destination, so
|
||||
// updateSlug throws (source_id, slug) UNIQUE and the loop falls back.
|
||||
await engine.putPage('people/dana', {
|
||||
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
|
||||
}, { sourceId: 'default' });
|
||||
|
||||
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(result.status).toBe('synced');
|
||||
|
||||
// The destination carries the renamed file's content...
|
||||
const dana = await engine.getPage('people/dana');
|
||||
expect(dana).not.toBeNull();
|
||||
expect(dana!.compiled_truth).toContain('Carol is a person.');
|
||||
|
||||
// ...and the stale old row is gone — no live duplicate.
|
||||
expect(await engine.getPage('people/carol')).toBeNull();
|
||||
expect(await countPages()).toBe(1);
|
||||
});
|
||||
|
||||
test('dedup-skip against the old row must NOT reconcile: the only copy survives', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// frontmatter.id gives identity dedup a handle: the import at the new
|
||||
// path can skip as "identical to <old row>" — in which case NOTHING
|
||||
// landed at the destination and deleting the old row would destroy the
|
||||
// only copy of the content.
|
||||
const md = ['---', 'type: person', 'title: Carol', 'id: ext-3056', '---', '', 'Carol is a person.'].join('\n');
|
||||
const repo = mkRepo({ 'people/carol.md': md });
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(await engine.getPage('people/carol')).not.toBeNull();
|
||||
|
||||
// Destination occupied → updateSlug throws → fallback path.
|
||||
await engine.putPage('people/dana', {
|
||||
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
|
||||
}, { sourceId: 'default' });
|
||||
|
||||
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
|
||||
// The import skipped against the OLD row (identity dedup), so the
|
||||
// destination never materialized with the renamed content — the
|
||||
// reconcile must not have deleted the old row, which still holds the
|
||||
// only copy.
|
||||
const carol = await engine.getPage('people/carol');
|
||||
expect(carol).not.toBeNull();
|
||||
expect(carol!.compiled_truth).toContain('Carol is a person.');
|
||||
});
|
||||
|
||||
test('reconcile never deletes by slug guess: unrelated manual row survives', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
|
||||
// The file's real row drifts to a divergent slug with no source_path
|
||||
// (unlocatable), and an UNRELATED manually-curated page happens to sit
|
||||
// at the path-derived slug a naive reconcile would guess.
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET slug = 'people/carol-divergent', source_path = NULL
|
||||
WHERE source_id = 'default' AND slug = 'people/carol'`,
|
||||
);
|
||||
await engine.putPage('people/carol', {
|
||||
type: 'person', title: 'Manual Carol', compiled_truth: 'hand-authored, not from the file',
|
||||
}, { sourceId: 'default' });
|
||||
// Destination occupied → updateSlug throws UNIQUE → fallback path.
|
||||
await engine.putPage('people/dana', {
|
||||
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
|
||||
}, { sourceId: 'default' });
|
||||
|
||||
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(result.status).toBe('synced');
|
||||
|
||||
// The destination materialized with the file's content...
|
||||
const dana = await engine.getPage('people/dana');
|
||||
expect(dana).not.toBeNull();
|
||||
expect(dana!.compiled_truth).toContain('Carol is a person.');
|
||||
// ...but no row had source_path = from, so the reconcile deleted
|
||||
// NOTHING: the unrelated manual row at the guessed slug survives.
|
||||
const manual = await engine.getPage('people/carol');
|
||||
expect(manual).not.toBeNull();
|
||||
expect(manual!.compiled_truth).toContain('hand-authored');
|
||||
});
|
||||
|
||||
test('happy path: clean git mv rename keeps page_id and touches nothing else', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
const before = await engine.getPage('people/carol');
|
||||
expect(before).not.toBeNull();
|
||||
|
||||
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(result.status).toBe('synced');
|
||||
|
||||
const after = await engine.getPage('people/dana');
|
||||
expect(after).not.toBeNull();
|
||||
expect(after!.id).toBe(before!.id); // cheap-path rename preserved the row
|
||||
expect(await engine.getPage('people/carol')).toBeNull();
|
||||
expect(await countPages()).toBe(1);
|
||||
});
|
||||
|
||||
test('reconcile failure blocks the bookmark and the next run retries to convergence', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
|
||||
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
await engine.putPage('people/dana', {
|
||||
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
|
||||
}, { sourceId: 'default' });
|
||||
|
||||
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
// Inject a transient failure into the reconcile delete.
|
||||
const origDelete = engine.deletePage.bind(engine);
|
||||
engine.deletePage = async () => { throw new Error('injected transient delete failure'); };
|
||||
let blocked;
|
||||
try {
|
||||
blocked = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
} finally {
|
||||
engine.deletePage = origDelete;
|
||||
}
|
||||
|
||||
// The failed reconcile is not checkpointed past: the run blocks and the
|
||||
// stale duplicate is still visible. The failure is recorded as a
|
||||
// `<rename:…>` SENTINEL, which the auto-skip valve can never
|
||||
// chronic-skip — an outage lasting longer than the threshold must not
|
||||
// quietly bank the duplicate.
|
||||
expect(blocked.status).toBe('blocked_by_failures');
|
||||
expect(blocked.failedFiles).toBe(1);
|
||||
expect(await engine.getPage('people/carol')).not.toBeNull();
|
||||
const { loadSyncFailures } = await import('../src/core/sync-failure-ledger.ts');
|
||||
const openSentinels = loadSyncFailures().filter(
|
||||
f => f.path === '<rename:people/dana.md>' && f.state === 'open',
|
||||
);
|
||||
expect(openSentinels).toHaveLength(1);
|
||||
|
||||
// Next run (failure gone) retries the same rename diff and converges.
|
||||
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
|
||||
expect(result.status).toBe('synced');
|
||||
expect(await engine.getPage('people/carol')).toBeNull();
|
||||
const dana = await engine.getPage('people/dana');
|
||||
expect(dana).not.toBeNull();
|
||||
expect(dana!.compiled_truth).toContain('Carol is a person.');
|
||||
expect(await countPages()).toBe(1);
|
||||
|
||||
// The convergence also clears the sentinel row — doctor must not keep
|
||||
// warning about a rename that has since reconciled.
|
||||
const remaining = loadSyncFailures().filter(
|
||||
f => f.path === '<rename:people/dana.md>' && f.state === 'open',
|
||||
);
|
||||
expect(remaining).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
@@ -39,6 +39,7 @@ const DEFAULT_CLI_OPTS: CliOptions = {
|
||||
progressInterval: 1000,
|
||||
timeoutMs: null,
|
||||
explain: false,
|
||||
brain: null,
|
||||
};
|
||||
|
||||
let tmpHome: string;
|
||||
|
||||
Reference in New Issue
Block a user