mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-18 09:48:17 +00:00
Compare commits
31
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a5da889f44 | ||
|
|
d698b44438 | ||
|
|
1d0b5ed816 | ||
|
|
4c71a76c0a | ||
|
|
354c8c36a9 | ||
|
|
dbf2b3f562 | ||
|
|
9ed53e4e1c | ||
|
|
9f7244a77f | ||
|
|
6ec3dd410e | ||
|
|
bcf3b73dcf | ||
|
|
23e0541d9b | ||
|
|
c873ce3014 | ||
|
|
f3e78fd2fb | ||
|
|
4528bfa79c | ||
|
|
6498b872ea | ||
|
|
324c355318 | ||
|
|
184b6cb8a1 | ||
|
|
912407bef1 | ||
|
|
89f226eb38 | ||
|
|
f1031d5a0b | ||
|
|
3a5c4c194c | ||
|
|
d165e99f0b | ||
|
|
8b325041ee | ||
|
|
a46f28a63e | ||
|
|
f72de97943 | ||
|
|
f8d11f67a3 | ||
|
|
93cfb37540 | ||
|
|
9fe4628d02 | ||
|
|
1833d95896 | ||
|
|
42375bded5 | ||
|
|
a8e6b1d177 |
@@ -2,6 +2,29 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.63.0] - 2026-07-20
|
||||
|
||||
**Schema commands now open the local brain you actually configured.**
|
||||
|
||||
If your PGLite brain lives at a custom path, commands such as `gbrain schema stats` previously ignored that path and could inspect the default brain instead. That made a healthy configured brain look empty or report the wrong schema counts. Schema commands now use the same complete database configuration as the rest of GBrain. PostgreSQL behavior is unchanged, and no migration is required.
|
||||
|
||||
### How to use it
|
||||
|
||||
Upgrade, then run the schema command normally:
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain schema stats --json
|
||||
```
|
||||
|
||||
The reported page and type counts now come from the `database_path` in `~/.gbrain/config.json` when the engine is PGLite.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
- **Schema CLI commands preserve configured PGLite paths.** Engine construction and connection now receive the canonical complete engine configuration, including both `database_path` and `database_url` where applicable.
|
||||
- **CLI tests are isolated from ambient database URLs.** Schema subprocess tests explicitly clear inherited PostgreSQL URL variables, and a persistent-PGLite regression test proves `schema stats` reads the configured database rather than the default brain.
|
||||
|
||||
## [0.42.62.0] - 2026-07-17
|
||||
|
||||
**If your brain holds more than one source, everything now lands in the right one. Link extraction, timeline extraction, background cycles, and webhook captures used to quietly file some of their output under the default source; all of those paths now carry the correct source identity. Background agent jobs got tougher too: a failed database reconnect can no longer wedge the engine, and workers recover from dropped connections instead of crash-looping. If you run the admin dashboard behind a reverse proxy, the live activity panel finally connects. Long agent conversations cost less because repeated context is reused between turns on Anthropic calls. Local LiteLLM proxies work out of the box. Nested sources scan correctly again instead of reporting zero files. And the project's automated checks now include dependency vulnerability scanning, static code-security analysis, and signed provenance for release builds. Thirty merged changes in all, the largest batch to date, each one reviewed and verified against the live codebase before landing.**
|
||||
|
||||
@@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -2113,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
|
||||
+1
-1
@@ -144,7 +144,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.62.0",
|
||||
"version": "0.42.63.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^1.19.13",
|
||||
"fast-uri": "^3.1.2",
|
||||
|
||||
+32
-1
@@ -998,6 +998,13 @@ const THIN_CLIENT_REFUSED_COMMANDS = new Set([
|
||||
// - `code-def`/`code-refs`/`code-callers`/`code-callees` have NO MCP ops
|
||||
// in operations.ts:2630-2671; cannot be "fixed by routing" yet
|
||||
'pages', 'files', 'eval', 'code-def', 'code-refs', 'code-callers', 'code-callees',
|
||||
// scratch-DB audit: `config` get/set operate on the host brain's config
|
||||
// plane (DB rows / host file-plane). On a thin client they fabricated an
|
||||
// ephemeral local PGLite (full migration replay per call) and read/wrote
|
||||
// config nobody would ever see. NOTE: `jobs` is deliberately NOT here —
|
||||
// it gets a partial dispatch (list/get route over MCP engine-free, the
|
||||
// rest refuse) in the main dispatch before connectEngine().
|
||||
'config',
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -1035,6 +1042,9 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
|
||||
'code-refs': '`code-refs` has no MCP op yet. Run on the host.',
|
||||
'code-callers': '`code-callers` has no MCP op yet. Run on the host.',
|
||||
'code-callees': '`code-callees` has no MCP op yet. Run on the host.',
|
||||
// scratch-DB audit additions
|
||||
config: "config reads/writes the host brain's config plane. Edit the host's .gbrain/config.json (file-plane keys) or run on the host with GBRAIN_HOME set.",
|
||||
jobs: '`jobs list` and `jobs get <id>` are thin-client routable; this subcommand runs against the host queue. Use the submit_job / list_jobs / get_job MCP tools from your agent, or run on the host with GBRAIN_HOME set.',
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -1593,6 +1603,27 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
// Thin-client `jobs` dispatch: `list` and `get` route over MCP (v0.32
|
||||
// routing branches in commands/jobs.ts) and never touch a local engine —
|
||||
// but falling through to connectEngine() below fabricates an empty
|
||||
// scratch PGLite in the thin-client GBRAIN_HOME and replays the entire
|
||||
// migration chain on every invocation before the remote call even runs.
|
||||
// Dispatch them engine-free here; every other jobs subcommand is
|
||||
// host-queue-bound, so refuse with a pinpoint hint instead of building
|
||||
// the scratch store.
|
||||
if (command === 'jobs') {
|
||||
const cfgJobs = loadConfig();
|
||||
if (isThinClient(cfgJobs)) {
|
||||
const jobsSub = args[0];
|
||||
if (jobsSub === 'list' || jobsSub === 'get') {
|
||||
const { runJobs } = await import('./commands/jobs.ts');
|
||||
await runJobs(null, args);
|
||||
return;
|
||||
}
|
||||
refuseThinClient('jobs', cfgJobs!.remote_mcp!.mcp_url);
|
||||
}
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
@@ -2258,7 +2289,7 @@ IMPORT/EXPORT
|
||||
import <dir> [--no-embed] Import markdown directory
|
||||
sync [--repo <path>] [flags] Git-to-brain incremental sync
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
sync --install-cron Install persistent sync daemon
|
||||
See also: autopilot --install (continuous daemon).
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
|
||||
@@ -109,7 +109,21 @@ function logError(phase: string, e: unknown) {
|
||||
*/
|
||||
export function resolveGbrainCliPath(): string {
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
// #2747: `env: process.env` is required under Bun. Bun's execSync
|
||||
// snapshots process.env at Bun's OWN startup, not at call time — a
|
||||
// runtime PATH mutation (dotenv/config loading, shell-profile sourcing
|
||||
// in a wrapper, etc.) happening between Bun boot and this call is
|
||||
// invisible to `which` without explicitly forwarding the current env.
|
||||
// This is why "which gbrain" succeeds when run standalone (fresh Bun
|
||||
// process, no prior mutation) but can fail from inside autopilot's own
|
||||
// process at this exact call site. Same fix already applied to
|
||||
// detectTini() in spawn-helpers.ts (see its comment) — this call site
|
||||
// was missed.
|
||||
const which = execSync('which gbrain', {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
env: process.env,
|
||||
}).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on $PATH — fall through */ }
|
||||
|
||||
@@ -123,7 +137,14 @@ export function resolveGbrainCliPath(): string {
|
||||
return arg1;
|
||||
}
|
||||
|
||||
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
|
||||
// #2747: include what we actually saw so an operator (or a future bug
|
||||
// report) doesn't have to guess whether PATH/execPath/argv[1] looked
|
||||
// sane at the moment of failure.
|
||||
throw new Error(
|
||||
'Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH ' +
|
||||
'(e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly. ' +
|
||||
`Debug: PATH=${JSON.stringify(process.env.PATH ?? '')} execPath=${JSON.stringify(exec)} argv1=${JSON.stringify(arg1)}`,
|
||||
);
|
||||
}
|
||||
|
||||
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
|
||||
|
||||
@@ -1069,7 +1069,8 @@ export async function runExtractConversationFactsCore(
|
||||
}
|
||||
// Fall through to receipt+rollup write so the partial run is
|
||||
// still observable in extract_health doctor + extracts/ pages.
|
||||
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
|
||||
// ...but not under --dry-run: a preview must not persist cache state.
|
||||
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
|
||||
// Return partial result — caller (CLI / Minion) decides how to
|
||||
// surface. NOT a thrown failure.
|
||||
return result;
|
||||
@@ -1081,7 +1082,9 @@ export async function runExtractConversationFactsCore(
|
||||
// (queryable + citable per D-EXTRACT-17/19) AND UPSERTs the per-day
|
||||
// rollup row (best-effort cache per F-OUT-19). Both are best-effort —
|
||||
// failures stderr-warn but never fail the parent operation.
|
||||
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
|
||||
// --dry-run must not persist cache/knowledge state: skip the rollup UPSERT +
|
||||
// receipt-page write so a preview leaves no extract cache row behind.
|
||||
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
|
||||
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
|
||||
// call isSyncable at all — so it descended into `node_modules/`, emitted
|
||||
// markdown files from there, AND ignored the canonical exclusion list
|
||||
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// subtrees before recursion (saving IO), and isSyncable filters the emit
|
||||
// set against the canonical markdown-strategy rules.
|
||||
const files: { path: string; relPath: string }[] = [];
|
||||
|
||||
+62
-8
@@ -11,6 +11,7 @@ import {
|
||||
isCodeFilePath,
|
||||
isMarkdownFilePath,
|
||||
isImageFilePath as isImageFilePathFromSync,
|
||||
matchesAnyGlob,
|
||||
pruneDir,
|
||||
SYNC_SKIP_FILES,
|
||||
type SyncStrategy,
|
||||
@@ -47,7 +48,25 @@ export interface RunImportResult {
|
||||
export async function runImport(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string; managedBookmark?: boolean } = {},
|
||||
opts: {
|
||||
commit?: string;
|
||||
strategy?: SyncStrategy;
|
||||
sourceId?: string;
|
||||
managedBookmark?: boolean;
|
||||
/**
|
||||
* #753/#774: glob patterns to exclude from the import (same semantics as
|
||||
* `isSyncable`'s `exclude` — matched against the dir-relative path).
|
||||
* Threaded by performFullSync for `gbrain sync --exclude`.
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* #753/#774 monorepo subdir-source support: when set, slugs and
|
||||
* `source_path` are computed relative to this root (the git repo root)
|
||||
* instead of `dir` (the sync scope), so `wiki/page1.md` lands as slug
|
||||
* `wiki/page1` consistently across full and incremental sync.
|
||||
*/
|
||||
slugRoot?: string;
|
||||
} = {},
|
||||
): Promise<RunImportResult> {
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const fresh = args.includes('--fresh');
|
||||
@@ -190,13 +209,30 @@ export async function runImport(
|
||||
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
|
||||
const _walkT0 = Date.now();
|
||||
console.error(`[gbrain phase] import.collect_files start dir=${dir} strategy=${strategy}`);
|
||||
const allFiles = collectSyncableFiles(dir, { strategy });
|
||||
let allFiles = collectSyncableFiles(dir, { strategy });
|
||||
console.error(
|
||||
`[gbrain phase] import.collect_files done ${Date.now() - _walkT0}ms files=${allFiles.length}`,
|
||||
);
|
||||
const fileTypeLabel = strategy === 'code' ? 'code'
|
||||
: strategy === 'auto' ? 'syncable' : 'markdown';
|
||||
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
|
||||
// #753/#774: apply --exclude glob patterns (threaded by performFullSync).
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
const beforeExclude = allFiles.length;
|
||||
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
|
||||
console.log(
|
||||
`Found ${allFiles.length} ${fileTypeLabel} files ` +
|
||||
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
|
||||
);
|
||||
// NAV-4: everything excluded is almost always a mistyped pattern — warn.
|
||||
if (beforeExclude > 0 && allFiles.length === 0) {
|
||||
console.warn(
|
||||
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
|
||||
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
|
||||
}
|
||||
|
||||
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
|
||||
// See src/core/sort-newest-first.ts for the policy.
|
||||
@@ -242,6 +278,11 @@ export async function runImport(
|
||||
|
||||
async function processFile(eng: BrainEngine, filePath: string) {
|
||||
const relativePath = relative(dir, filePath);
|
||||
// #753/#774: slug + source_path base. When performFullSync syncs a
|
||||
// monorepo subdir, slugRoot is the git root so slugs stay git-root-
|
||||
// relative (matching the incremental path's git-diff paths). The
|
||||
// checkpoint (`completed`) stays dir-relative — resumeFilter's contract.
|
||||
const importRelPath = opts.slugRoot ? relative(opts.slugRoot, filePath) : relativePath;
|
||||
// v0.31.2 (D5): per-file slow-path log. Fires only when a single
|
||||
// file takes >5s. The user's hang surfaces as one file taking
|
||||
// forever — without this, the agent can't see which file.
|
||||
@@ -252,8 +293,8 @@ export async function runImport(
|
||||
// up images when GBRAIN_EMBEDDING_MULTIMODAL=true so this branch is
|
||||
// unreachable when the gate is off; defense-in-depth check anyway.
|
||||
const result = isImageFilePath(relativePath) && process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'
|
||||
? await importImageFile(eng, filePath, relativePath, { noEmbed, sourceId })
|
||||
: await importFile(eng, filePath, relativePath, { noEmbed, sourceId, activePack: importActivePack });
|
||||
? await importImageFile(eng, filePath, importRelPath, { noEmbed, sourceId })
|
||||
: await importFile(eng, filePath, importRelPath, { noEmbed, sourceId, activePack: importActivePack });
|
||||
const _fileMs = Date.now() - _fileT0;
|
||||
if (_fileMs > 5000) {
|
||||
console.error(`[gbrain phase] import.process_file slow ${_fileMs}ms ${relativePath}`);
|
||||
@@ -269,7 +310,9 @@ export async function runImport(
|
||||
if (result.error && result.error !== 'unchanged') {
|
||||
console.error(` Skipped ${relativePath}: ${result.error}`);
|
||||
// Bug 9 — non-"unchanged" skips carry a real error reason.
|
||||
failures.push({ path: relativePath, error: result.error });
|
||||
// #774: ledger paths use the slug base so an incremental sync's
|
||||
// success at the same (git-root-relative) path clears the row.
|
||||
failures.push({ path: importRelPath, error: result.error });
|
||||
} else {
|
||||
// 'unchanged' or no-error skip: content_hash matched a prior
|
||||
// successful import, so this file IS done for checkpoint purposes.
|
||||
@@ -287,7 +330,7 @@ export async function runImport(
|
||||
}
|
||||
errors++;
|
||||
skipped++;
|
||||
failures.push({ path: relativePath, error: msg });
|
||||
failures.push({ path: importRelPath, error: msg });
|
||||
}
|
||||
processed++;
|
||||
tickProgress();
|
||||
@@ -526,10 +569,21 @@ function isCollectibleForWalker(
|
||||
strategy: SyncStrategy,
|
||||
multimodalOn: boolean,
|
||||
): boolean {
|
||||
// #2607: apply the SAME segment-level prune gate as incremental sync's
|
||||
// `classifySync` (core/sync.ts). The FS walk below prunes at descent time,
|
||||
// but the git fast path enumerates via `git ls-files` and historically
|
||||
// filtered only by extension — so `sync --full` imported (and resurrected
|
||||
// previously-deleted) pages under dot-dirs / vendored trees that incremental
|
||||
// sync excludes. Full and incremental must agree on the exclusion set.
|
||||
// (In the FS-walk route `path` is a basename, so this is the same dot-file
|
||||
// check pruneDir already applied there — no behavior change on that route.)
|
||||
const segments = path.split('/');
|
||||
if (segments.some((seg) => !pruneDir(seg))) return false;
|
||||
|
||||
// Metafiles are directory scaffolding (READMEs / index / log / schema /
|
||||
// resolver), not typed brain pages — same exclusion `sync`'s `isSyncable`
|
||||
// applies. Guards both the FS-walk and the git-fast-path collection routes.
|
||||
const basename = path.split('/').pop() || '';
|
||||
const basename = segments[segments.length - 1] || '';
|
||||
if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false;
|
||||
|
||||
switch (strategy) {
|
||||
|
||||
+20
-1
@@ -132,9 +132,23 @@ function formatJobDetail(job: MinionJob): string {
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export async function runJobs(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
|
||||
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
|
||||
// with remote MCP routing (`list`, `get`) so no scratch local engine is
|
||||
// ever built. Any other subcommand arriving with a null engine is a
|
||||
// routing bug upstream of this function — refuse instead of crashing
|
||||
// inside MinionQueue.
|
||||
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
|
||||
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
|
||||
process.exit(1);
|
||||
}
|
||||
// Null only ever reaches the MCP-routed `list`/`get` branches, which
|
||||
// never touch the engine — narrowed once here so the host-only cases
|
||||
// below typecheck unchanged.
|
||||
const engine = engineOrNull as BrainEngine;
|
||||
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(`gbrain jobs — Minions job queue
|
||||
|
||||
@@ -217,6 +231,8 @@ HANDLER TYPES (built in)
|
||||
return;
|
||||
}
|
||||
|
||||
// The constructor just stores the reference; on the null (thin-client
|
||||
// list/get) paths no queue method is ever reached.
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
switch (sub) {
|
||||
@@ -1780,6 +1796,7 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: effectiveBrainDir,
|
||||
pull,
|
||||
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
...(sourceId ? { sourceId } : {}),
|
||||
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
|
||||
yieldBetweenPhases: async () => {
|
||||
@@ -1817,6 +1834,7 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: repoPath,
|
||||
pull: false, // brain-wide DB/maintenance work never git-pulls
|
||||
signal: job.signal,
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
phases,
|
||||
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
|
||||
});
|
||||
@@ -1962,6 +1980,7 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: repoPath,
|
||||
phases: [phase as any],
|
||||
signal: job.signal,
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
});
|
||||
return { phase, status: report.status, report };
|
||||
};
|
||||
|
||||
+20
-11
@@ -9,6 +9,7 @@ import { listRecipes, getRecipe } from '../core/ai/recipes/index.ts';
|
||||
import { configureGateway, embedOne, isAvailable as gwIsAvailable, chat as gwChat } from '../core/ai/gateway.ts';
|
||||
import { probeOllama, probeLMStudio } from '../core/ai/probes.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
|
||||
import { AIConfigError, AITransientError } from '../core/ai/errors.ts';
|
||||
import type { Recipe } from '../core/ai/types.ts';
|
||||
|
||||
@@ -33,16 +34,19 @@ interface ProviderOption {
|
||||
|
||||
function configureFromEnv(): void {
|
||||
const config = loadConfig();
|
||||
configureGateway({
|
||||
embedding_model: config?.embedding_model,
|
||||
embedding_dimensions: config?.embedding_dimensions,
|
||||
expansion_model: config?.expansion_model,
|
||||
chat_model: config?.chat_model,
|
||||
chat_fallback_chain: config?.chat_fallback_chain,
|
||||
base_urls: config?.provider_base_urls,
|
||||
provider_chat_options: config?.provider_chat_options,
|
||||
env: { ...process.env },
|
||||
});
|
||||
// Route through buildGatewayConfig — the single ownership seam that folds
|
||||
// file-plane API keys (openrouter_api_key, zeroentropy_api_key, ...) into
|
||||
// the gateway env — instead of hand-assembling AIGatewayConfig field by
|
||||
// field. Hand-building it here let this diagnostic report a provider as
|
||||
// missing env even when ~/.gbrain/config.json had it and the real gateway
|
||||
// path resolved it fine (#2728). Pre-init (no file-plane config yet) falls
|
||||
// back to a bare env passthrough so the command still works before
|
||||
// `gbrain init`.
|
||||
if (config) {
|
||||
configureGateway(buildGatewayConfig(config));
|
||||
return;
|
||||
}
|
||||
configureGateway({ env: { ...process.env } });
|
||||
}
|
||||
|
||||
export function envReady(recipe: Recipe, env: NodeJS.ProcessEnv = process.env): boolean {
|
||||
@@ -137,7 +141,12 @@ EXAMPLES
|
||||
}
|
||||
|
||||
function runList(_args: string[]): void {
|
||||
console.log(formatRecipeTable(listRecipes()));
|
||||
// Same env the gateway actually sees (file-plane keys folded in), not bare
|
||||
// process.env — keeps this table's STATUS column honest with what
|
||||
// `providers test` (and the real init/gateway path) would report.
|
||||
const cfg = loadConfig();
|
||||
const env = cfg ? buildGatewayConfig(cfg).env : process.env;
|
||||
console.log(formatRecipeTable(listRecipes(), env));
|
||||
}
|
||||
|
||||
async function runTest(args: string[]): Promise<void> {
|
||||
|
||||
@@ -179,10 +179,16 @@ export async function runReindexSearchVector(
|
||||
}
|
||||
|
||||
// Recreate trigger functions. The strings are intentionally identical to
|
||||
// the v123 migration body — keeping them in lockstep is the contract.
|
||||
// the v124 migration body — keeping them in lockstep is the contract.
|
||||
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
|
||||
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
|
||||
// hardening from every brain that runs this command.
|
||||
//
|
||||
// #2704: compiled_truth (the unbounded whole-page body) is deliberately
|
||||
// NOT indexed here — it overflows Postgres's 1MB tsvector cap on large
|
||||
// pages, and content_chunks.search_vector (populated separately, chunk-
|
||||
// grain, well under the cap) is what searchKeyword() actually queries.
|
||||
// See migrate.ts's v124 for the full rationale; keep this copy in sync.
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
@@ -195,7 +201,6 @@ export async function runReindexSearchVector(
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
+18
-12
@@ -105,13 +105,19 @@ function printHelp(): void {
|
||||
async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>, args: string[]): Promise<void> {
|
||||
const { json, timeoutMs } = parseFlags(args);
|
||||
|
||||
let submitted: { id: number; name: string; state: string };
|
||||
// submit_job / get_job return the MinionJob row verbatim — the lifecycle
|
||||
// field is `status` (src/core/minions/types.ts), not `state`. Reading
|
||||
// `state` here made every poll see `undefined`, so the terminal check
|
||||
// never matched and ping always exhausted its timeout (exit 1) even when
|
||||
// the cycle completed. The ping's own JSON *output* keys (`state`,
|
||||
// `last_state`) are kept as-is for consumers.
|
||||
let submitted: { id: number; name: string; status: string };
|
||||
try {
|
||||
const res = await callRemoteTool(config, 'submit_job', {
|
||||
name: 'autopilot-cycle',
|
||||
data: { phases: ['sync', 'extract', 'embed'] },
|
||||
});
|
||||
submitted = unpackToolResult<{ id: number; name: string; state: string }>(res);
|
||||
submitted = unpackToolResult<{ id: number; name: string; status: string }>(res);
|
||||
} catch (e) {
|
||||
return failPing(e, json);
|
||||
}
|
||||
@@ -122,43 +128,43 @@ async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>,
|
||||
|
||||
const startMs = Date.now();
|
||||
let attempt = 0;
|
||||
let lastState = submitted.state;
|
||||
let lastState = submitted.status;
|
||||
while (Date.now() - startMs < timeoutMs) {
|
||||
const elapsed = Date.now() - startMs;
|
||||
const intervalMs = elapsed < 30_000 ? 1_000 : elapsed < 5 * 60_000 + 30_000 ? 5_000 : 10_000;
|
||||
await sleep(intervalMs);
|
||||
attempt++;
|
||||
|
||||
let job: { id: number; state: string; failed_reason?: string };
|
||||
let job: { id: number; status: string; failed_reason?: string };
|
||||
try {
|
||||
const res = await callRemoteTool(config, 'get_job', { id: submitted.id });
|
||||
job = unpackToolResult<{ id: number; state: string; failed_reason?: string }>(res);
|
||||
job = unpackToolResult<{ id: number; status: string; failed_reason?: string }>(res);
|
||||
} catch (e) {
|
||||
// Network blip mid-poll: log and keep going. Surface only if persistent.
|
||||
if (!json) console.error(` poll #${attempt} failed (${e instanceof Error ? e.message : String(e)}); continuing...`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (job.state !== lastState) {
|
||||
lastState = job.state;
|
||||
if (!json) console.error(` job #${submitted.id} → ${job.state}`);
|
||||
if (job.status !== lastState) {
|
||||
lastState = job.status;
|
||||
if (!json) console.error(` job #${submitted.id} → ${job.status}`);
|
||||
}
|
||||
|
||||
const terminal = ['completed', 'failed', 'dead', 'cancelled'];
|
||||
if (terminal.includes(job.state)) {
|
||||
const ok = job.state === 'completed';
|
||||
if (terminal.includes(job.status)) {
|
||||
const ok = job.status === 'completed';
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
status: ok ? 'success' : 'error',
|
||||
job_id: submitted.id,
|
||||
state: job.state,
|
||||
state: job.status,
|
||||
...(job.failed_reason ? { failed_reason: job.failed_reason } : {}),
|
||||
elapsed_ms: Date.now() - startMs,
|
||||
}));
|
||||
} else {
|
||||
console.log(ok
|
||||
? `\nautopilot-cycle complete (${Math.round((Date.now() - startMs) / 1000)}s).`
|
||||
: `\nautopilot-cycle ended ${job.state}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
|
||||
: `\nautopilot-cycle ended ${job.status}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
|
||||
}
|
||||
process.exit(ok ? 0 : 1);
|
||||
}
|
||||
|
||||
@@ -48,7 +48,7 @@ import {
|
||||
} from '../core/schema-pack/index.ts';
|
||||
import type { SchemaPackManifest, PackPrimitive } from '../core/schema-pack/manifest-v1.ts';
|
||||
import { PACK_PRIMITIVES } from '../core/schema-pack/manifest-v1.ts';
|
||||
import { gbrainPath, loadConfig, configPath } from '../core/config.ts';
|
||||
import { gbrainPath, loadConfig, configPath, toEngineConfig } from '../core/config.ts';
|
||||
|
||||
export async function runSchema(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
@@ -434,16 +434,12 @@ function parseFlags(args: string[]): ParsedFlags {
|
||||
|
||||
async function withConnectedEngine<T>(fn: (engine: import('../core/engine.ts').BrainEngine) => Promise<T>): Promise<T> {
|
||||
const { createEngine } = await import('../core/engine-factory.ts');
|
||||
const cfg = loadConfig() ?? {};
|
||||
const engineKind = (cfg as { engine?: string }).engine === 'postgres' ? 'postgres' : 'pglite';
|
||||
const cfg = loadConfig() ?? { engine: 'pglite' as const };
|
||||
// PR #1321 (closed) defensive fix retained: build the EngineConfig once and
|
||||
// pass it to BOTH createEngine and engine.connect. The factory captures
|
||||
// config at construction; explicit re-pass at connect() is defense in depth
|
||||
// against future engine implementations that read URL from connect-time.
|
||||
const connectConfig: import('../core/types.ts').EngineConfig = {
|
||||
engine: engineKind,
|
||||
database_url: (cfg as { database_url?: string }).database_url,
|
||||
};
|
||||
const connectConfig = toEngineConfig(cfg);
|
||||
const engine = await createEngine(connectConfig);
|
||||
await engine.connect(connectConfig);
|
||||
try {
|
||||
|
||||
+611
-58
@@ -1,4 +1,4 @@
|
||||
import { existsSync, readFileSync, writeFileSync, statSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, statSync, realpathSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
@@ -9,6 +9,7 @@ import { createInterface } from 'readline';
|
||||
import {
|
||||
isSyncable,
|
||||
unsyncableReason,
|
||||
matchesAnyGlob,
|
||||
resolveSlugForPath,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeFailures,
|
||||
@@ -742,6 +743,27 @@ export interface SyncOpts {
|
||||
sourceId?: string;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
/**
|
||||
* #753/#774 — sync only files under this subdirectory of the git repo.
|
||||
* Git operations (pull, diff, rev-parse) still run against the repo root
|
||||
* (discovered via `git rev-parse --show-toplevel`); file walking, imports,
|
||||
* deletes and renames are scoped to the subpath. Slugs are git-root-relative
|
||||
* (`wiki/page1.md` → slug `wiki/page1`) so full and incremental syncs of
|
||||
* the same scope agree. Enables N logical sources in one git repo.
|
||||
*
|
||||
* SECURITY (NAV-1/NAV-2): the resolved subpath must realpath-resolve inside
|
||||
* the git root — `../escape` and symlinked subdirs pointing outside the repo
|
||||
* are rejected before any git op runs.
|
||||
*/
|
||||
srcSubpath?: string;
|
||||
/**
|
||||
* #753/#774 — glob patterns for files to exclude from sync (repeatable
|
||||
* `--exclude` on the CLI). Matched against the scope-relative path in both
|
||||
* the full-sync and incremental paths. Excluded files are never imported;
|
||||
* exclusion does NOT delete previously-imported pages (conservative,
|
||||
* matching the #1433 metafile posture).
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* Number of parallel workers for the import phase. When > 1, each worker
|
||||
* gets its own small Postgres connection pool and files are dispatched via
|
||||
@@ -897,14 +919,210 @@ export function buildAutoEmbedArgs(slugs: string[], sourceId?: string): string[]
|
||||
* 100 MiB is generous but still bounded — a 100K-file diff with long
|
||||
* paths tops out around 10–20 MiB in practice.
|
||||
*/
|
||||
function git(repoPath: string, args: string[], configs: string[] = []): string {
|
||||
function git(repoPath: string, args: string[], configs: string[] = [], timeoutMs = 30000): string {
|
||||
return execFileSync('git', buildGitInvocation(repoPath, args, configs), {
|
||||
encoding: 'utf-8',
|
||||
timeout: 30000,
|
||||
timeout: timeoutMs,
|
||||
maxBuffer: 100 * 1024 * 1024,
|
||||
}).trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* #753/#774: walk up from inputPath to the nearest git repo root via
|
||||
* `git -C <path> rev-parse --show-toplevel`. Handles worktrees and submodules
|
||||
* natively (git itself resolves them). Throws a user-friendly error when no
|
||||
* git repo is found.
|
||||
*/
|
||||
export function discoverGitRoot(inputPath: string): string {
|
||||
try {
|
||||
return git(inputPath, ['rev-parse', '--show-toplevel']);
|
||||
} catch {
|
||||
throw new Error(
|
||||
`Not inside a git repository: ${inputPath}. GBrain sync requires a git-initialized repo (or a subdirectory of one).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2964: snapshot the CURRENT on-disk state of a gbrain-owned brain dir as
|
||||
* a baseline commit — used both right after a self-healing `git init` (no
|
||||
* `.git` at all) and to recover a repo left with `.git` but zero commits
|
||||
* (an interrupted prior self-heal, or a `git init` from some other source
|
||||
* that never got a first commit). Respects `.gitignore` (written first) so
|
||||
* future incremental syncs diff against what's actually here rather than
|
||||
* an empty tree — an empty initial commit would make every existing file
|
||||
* look "added" again on the next sync, even though the full-sync pass that
|
||||
* follows already imported them from disk directly.
|
||||
*
|
||||
* `--no-gpg-sign` + explicit `-c user.name/user.email`: this runs from a
|
||||
* headless nightly cron/launchd invocation, which has no reason to have
|
||||
* git signing/identity configured, and must not block on an unavailable
|
||||
* signing agent or pinentry prompt.
|
||||
*
|
||||
* db_only exclusion is recomputed directly and passed to `git add` as
|
||||
* negative pathspecs, rather than relying solely on `manageGitignore`
|
||||
* having written `.gitignore` successfully: that helper is deliberately
|
||||
* best-effort (a broken gbrain.yml parse, or an unwritable .gitignore,
|
||||
* only warns and returns — the right default for its OTHER callers, where
|
||||
* .gitignore management is a side effect that must never kill the sync
|
||||
* job). For a commit we are about to create ourselves, "fail open" there
|
||||
* would mean silently committing db_only content into git history. Fail
|
||||
* closed instead: db_only exclusion doesn't depend on the .gitignore
|
||||
* write having succeeded. `loadStorageConfig` throwing (unreadable
|
||||
* gbrain.yml, or a semantic overlap) propagates — better to leave this
|
||||
* self-heal wedged with a clear error than commit unknown content.
|
||||
*/
|
||||
function createSyncBaselineCommit(repoPath: string): void {
|
||||
// #2964: db_only exclusion is computed directly from loadStorageConfig
|
||||
// and passed to `git add` as pathspecs — deliberately NOT via
|
||||
// manageGitignore/.gitignore, for two independent reasons:
|
||||
//
|
||||
// 1. Ordering (Codex review round 6, P1): `collectSyncableFiles` — the
|
||||
// file enumeration `performFullSync` runs right after this function
|
||||
// returns — honors `.gitignore` via `git ls-files --exclude-standard`.
|
||||
// Writing db_only entries into `.gitignore` BEFORE that first import
|
||||
// would silently exclude those pages from the database entirely.
|
||||
// That's the exact bug class `runSync`'s existing "manage .gitignore
|
||||
// ONLY on successful sync" ordering (this file, `manageGitignoreAtGitRoot`
|
||||
// callers below — itself a prior Codex P1 fix) exists to prevent. Leave
|
||||
// `.gitignore` untouched here; the existing post-sync flow writes it
|
||||
// once this sync completes, same as it does for every other sync.
|
||||
// 2. Fail-closed (rounds 5-6): `manageGitignore`'s "warn and return" on a
|
||||
// broken gbrain.yml/unwritable .gitignore is the right default for its
|
||||
// OTHER callers (a side effect that must never kill the sync job), but
|
||||
// wrong for a commit we are creating ourselves — silently committing
|
||||
// db_only content into git history.
|
||||
const storageConfig = loadStorageConfig(repoPath);
|
||||
const dbOnlyDirs = storageConfig?.db_only ?? [];
|
||||
// Sniff-test fail-closed (round 6, P2): `loadStorageConfig` warns-and-
|
||||
// returns an EMPTY config for syntactically-valid-but-unsupported YAML
|
||||
// (e.g. flow-style `db_only: [dir/]` — the narrow custom parser only
|
||||
// handles block-style lists), which would silently resolve zero
|
||||
// exclusions from a file that clearly intended some. If gbrain.yml
|
||||
// exists and mentions db_only (or its deprecated pre-v0.22.11 alias
|
||||
// `supabase_only` — same keep-out-of-git semantics, still a supported
|
||||
// backward-compat key per storage-config.ts) but nothing resolved from
|
||||
// it, refuse rather than guess "genuinely empty" vs "syntax ignored".
|
||||
//
|
||||
// Known false-positive (round 8 review): a genuinely, intentionally
|
||||
// empty `db_only: []` mentioning the word also refuses, and can't be
|
||||
// told apart from the unsupported-syntax case — `loadStorageConfig`
|
||||
// returns the IDENTICAL `{db_tracked:[],db_only:[]}` for both (verified
|
||||
// directly: flow-style `[dir/]` and literal `[]` both collapse to that
|
||||
// same shape). Distinguishing them would mean teaching this function
|
||||
// about the parser's internal line-recognition rules, which belongs in
|
||||
// storage-config.ts, not here. Accepted trade-off: the false-positive
|
||||
// cost is low and self-resolving (the brain stays wedged with a clear,
|
||||
// actionable error until the user drops the pointless empty stanza or
|
||||
// fixes their syntax; retried on every subsequent sync); the
|
||||
// false-negative this guards against — silently committing db_only
|
||||
// content into permanent git history — is high-cost and hard to undo.
|
||||
if (dbOnlyDirs.length === 0) {
|
||||
const yamlPath = join(repoPath, 'gbrain.yml');
|
||||
const yamlContent = existsSync(yamlPath) ? readFileSync(yamlPath, 'utf-8') : '';
|
||||
// A YAML KEY line (`db_only:` / `supabase_only:`, ignoring leading
|
||||
// whitespace and `#` comments), not a bare substring search — round 9,
|
||||
// P2: a comment or unrelated prose value that happens to mention the
|
||||
// word (e.g. `# db_only handling TBD`) must not trip this guard on an
|
||||
// otherwise-genuinely-config-free gbrain.yml.
|
||||
const mentionsUnresolvedKey = yamlContent.split('\n').some((line) => {
|
||||
const trimmed = line.trim();
|
||||
return !trimmed.startsWith('#') && /^(db_only|supabase_only)\s*:/.test(trimmed);
|
||||
});
|
||||
if (mentionsUnresolvedKey) {
|
||||
throw new Error(
|
||||
`${yamlPath} mentions db_only but no directories resolved from it — refusing to ` +
|
||||
`auto-commit (cannot tell "genuinely empty" from "unsupported syntax silently ignored"). ` +
|
||||
`Fix gbrain.yml's storage.db_only syntax, or git-init this directory manually.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// #2964 (round 9, P1): every db_only dir is ALWAYS pathspec-excluded,
|
||||
// unconditionally — never pre-filtered against what an existing
|
||||
// `.gitignore` claims to already cover. An earlier version checked
|
||||
// `git check-ignore -q dir` first and skipped the pathspec when it
|
||||
// already reported "ignored" (to dodge the advisory error below), but
|
||||
// `check-ignore` on a directory can say "ignored" even when a
|
||||
// pre-existing `.gitignore` re-includes a child via negation (e.g.
|
||||
// `private-cache/*` + `!private-cache/index.md`) — the filter would
|
||||
// then skip excluding it via pathspec, and `git add -A` would stage
|
||||
// that re-included child despite the whole directory being declared
|
||||
// db_only. Our OWN pathspec exclusion is unconditional and doesn't
|
||||
// consult `.gitignore` at all, so it can't be defeated by ANY
|
||||
// .gitignore content, negated or not. `:(exclude,literal)dir` (not the
|
||||
// `:!dir` shorthand) so a db_only dir name that itself starts with a
|
||||
// pathspec magic character like `:` is excluded literally rather than
|
||||
// reinterpreted (round 9, P2).
|
||||
const excludePathspecs = dbOnlyDirs.map((dir) => `:(exclude,literal)${dir}`);
|
||||
// Clear the index before staging (round 6, P1): the unborn-HEAD
|
||||
// recovery site can reach this function with a repo whose index
|
||||
// already has entries staged from some OTHER prior operation (a manual
|
||||
// `git add`, an interrupted workflow) before gbrain ever touched it.
|
||||
// `add -A` only adds/updates — it does not drop an already-staged path
|
||||
// that our exclusion pathspecs above now want excluded. `read-tree
|
||||
// --empty` resets the index without touching the working tree; a
|
||||
// no-op on a freshly-`git init`-ed repo, whose index is already empty.
|
||||
git(repoPath, ['read-tree', '--empty']);
|
||||
try {
|
||||
// #2964: 10 minutes, not the shared git() helper's 30s default — this
|
||||
// full-tree `git add -A` walks a legacy brain that may hold years of
|
||||
// accumulated content. A 30s timeout would abort staging after `git
|
||||
// init` already created `.git`, leaving an unborn repo that every
|
||||
// subsequent sync would retry (and time out identically) forever;
|
||||
// the unborn-HEAD recovery path exists for OTHER causes of that
|
||||
// state, not to be this one's normal first outcome.
|
||||
git(repoPath, ['add', '-A', '--', '.', ...excludePathspecs], [], 600_000);
|
||||
} catch (err) {
|
||||
// Now that exclusion is always applied (never pre-filtered), an
|
||||
// explicit pathspec exclusion for a path a pre-existing `.gitignore`
|
||||
// ALSO happens to cover trips git's advice.addIgnoredFile: nonzero
|
||||
// exit + "paths ignored by one of your .gitignore files, use -f",
|
||||
// even though the add otherwise fully succeeded (verified directly:
|
||||
// `git status --short` right after this exact error shows every
|
||||
// non-excluded path staged correctly). Recognize and swallow ONLY
|
||||
// this exact advisory; anything else (timeout, permission denied,
|
||||
// real corruption) rethrows.
|
||||
const stderr = err && typeof err === 'object' && 'stderr' in err ? String((err as { stderr: unknown }).stderr) : '';
|
||||
if (!stderr.includes('ignored by one of your .gitignore files')) throw err;
|
||||
}
|
||||
git(
|
||||
repoPath,
|
||||
// --no-verify only skips pre-commit/commit-msg — prepare-commit-msg
|
||||
// and (worse, since it runs AFTER the commit object already exists,
|
||||
// synchronously inside this same git invocation) post-commit are
|
||||
// NOT covered by it. An operator's global core.hooksPath or
|
||||
// init.templateDir can wire either, expecting project tooling,
|
||||
// prompting interactively, or hanging — none of which a headless
|
||||
// self-heal commit can satisfy, and a hanging post-commit hook would
|
||||
// burn the 600s budget above without even being the slow step.
|
||||
// `-c core.hooksPath=/dev/null` (in configs, below) makes git look
|
||||
// for hook scripts inside a location that can't contain any,
|
||||
// disabling the entire hooks path for this one invocation — the
|
||||
// complete form of what --no-verify only partially covers, kept for
|
||||
// explicitness on the two hooks it does name.
|
||||
[
|
||||
'commit', '--quiet', '--allow-empty', '--no-gpg-sign', '--no-verify',
|
||||
'-m', 'gbrain: initial commit (auto-init by sync)',
|
||||
],
|
||||
['user.name=gbrain', 'user.email=gbrain@localhost', 'core.hooksPath=/dev/null'],
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* #774 NAV-1 TOCTOU: true only if filePath realpath-resolves inside gitRoot.
|
||||
* Guards symlink escape at the per-file level (a committed symlink whose
|
||||
* target lives outside the repo), not just at scope entry.
|
||||
*/
|
||||
function isPathSafe(filePath: string, gitRoot: string): boolean {
|
||||
try {
|
||||
const real = realpathSync(filePath);
|
||||
const rootReal = realpathSync(gitRoot);
|
||||
return real === rootReal || real.startsWith(rootReal + '/');
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function hasOriginRemote(repoPath: string): boolean {
|
||||
try {
|
||||
execFileSync('git', buildGitInvocation(repoPath, ['remote', 'get-url', 'origin']), {
|
||||
@@ -956,6 +1174,65 @@ async function readSyncAnchor(
|
||||
return await engine.getConfig(`sync.${which}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* #2964: is `repoPath` gbrain's own default-brain anchor, as opposed to a
|
||||
* path some caller merely happened to pass through unchanged?
|
||||
*
|
||||
* `!opts.sourceId` alone is NOT sufficient — and neither is rejecting
|
||||
* `opts.sourceId` outright: migration `sources_table_additive` (v20)
|
||||
* seeds a `'default'` source row whose `local_path` is copied FROM
|
||||
* `config.sync.repo_path` on every brain that has ever run it (i.e.
|
||||
* effectively all of them by now), and `writeSyncAnchor` keeps that row's
|
||||
* `local_path` current on every sync thereafter. So on a real installed
|
||||
* brain, `resolveSourceForDir` (dream cycle) and the CLI's bare `gbrain
|
||||
* sync` both resolve `sourceId: 'default'`, NOT `undefined` — rejecting
|
||||
* all non-empty `sourceId` (an earlier, insufficiently-reviewed version
|
||||
* of this check) made self-heal never fire on that real path either,
|
||||
* masked in tests only because a freshly-`initSchema()`'d test brain's
|
||||
* `'default'` row has a null `local_path` (Codex review round 5).
|
||||
*
|
||||
* The actual boundary: `'default'` is gbrain's own bootstrap identity,
|
||||
* not something a caller names — a DIFFERENT, non-default `sourceId` is
|
||||
* what an explicit `sources add <id> --path <dir>` registration (a
|
||||
* user's own external directory) looks like, and that's what must keep
|
||||
* failing loudly. So: permit `sourceId` when it's exactly `undefined` or
|
||||
* `'default'`, reject any other id, and for BOTH permitted cases prove
|
||||
* ownership by VALUE — reread the live anchor for that same identity
|
||||
* (`sources.default.local_path` when sourceId='default', else
|
||||
* `config.sync.repo_path`) and require the resolved `repoPath` to
|
||||
* REALPATH-equal it (not raw string equality: `dream`'s `resolveBrainDir`
|
||||
* normalizes via `path.resolve`, so a trailing slash or `..` in the
|
||||
* stored anchor must not defeat the match — Codex review round 5, P2).
|
||||
* An arbitrary caller-supplied path (e.g. an admin-scope
|
||||
* `submit_job({name:'sync', data:{repoPath}})`) only passes this check
|
||||
* if it already equals gbrain's own anchor by realpath identity — at
|
||||
* which point self-healing it is exactly the legitimate case, not an
|
||||
* escalation.
|
||||
*
|
||||
* `opts.srcSubpath` disqualifies unconditionally: a subpath-scoped sync
|
||||
* only wants THAT subdirectory captured, but the self-heal baseline
|
||||
* commit runs `git add -A` at the git root (there's no file list yet to
|
||||
* scope it to — collection happens after this point) — see the P2 review
|
||||
* finding on `createSyncBaselineCommit`'s callers.
|
||||
*/
|
||||
async function isAnchorOwnedSyncPath(
|
||||
engine: BrainEngine,
|
||||
opts: SyncOpts,
|
||||
repoPath: string,
|
||||
): Promise<boolean> {
|
||||
if (opts.srcSubpath) return false;
|
||||
if (opts.sourceId && opts.sourceId !== 'default') return false;
|
||||
const anchor = await readSyncAnchor(engine, opts.sourceId, 'repo_path');
|
||||
if (anchor === null) return false;
|
||||
try {
|
||||
return realpathSync(anchor) === realpathSync(repoPath);
|
||||
} catch {
|
||||
// Anchor or repoPath doesn't realpath-resolve (dangling/nonexistent) —
|
||||
// can't prove identity, so don't self-heal.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function writeSyncAnchor(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
@@ -1567,17 +1844,72 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
}
|
||||
|
||||
// Validate git repo
|
||||
if (!existsSync(join(repoPath, '.git'))) {
|
||||
throw new Error(`Not a git repository: ${repoPath}. GBrain sync requires a git-initialized repo.`);
|
||||
// #753/#774: discover the git root instead of requiring `.git` at repoPath
|
||||
// directly. Supports subdir-of-git-repo sources (monorepo pattern): either
|
||||
// an explicit `--src-subpath` under a git-root repoPath, or a repoPath that
|
||||
// IS a subdirectory (auto-discovery). Two axes fall out:
|
||||
// - gitContextRoot: ALL git operations (pull, rev-parse, diff, cat-file)
|
||||
// - syncScopeRoot: file walking, imports, deletes, renames
|
||||
// In the common case (repoPath == git root, no subpath) they are identical.
|
||||
serr(`[gbrain phase] sync.discover_git_root`);
|
||||
// #2964: a legacy `sync.repo_path`-anchored default brain can reach here
|
||||
// having never been `git init`-ed — e.g. a brain-pages dir that predates
|
||||
// git-backed sync, or one rsync'd from another machine without its
|
||||
// `.git`. gbrain owns that directory outright, so self-heal by
|
||||
// initializing it in place instead of failing the sync phase every
|
||||
// single run. Mirrors the recloneIfMissing self-recovery above for
|
||||
// owned remote clones. Ownership is proven by VALUE (resolved repoPath
|
||||
// equals gbrain's persisted anchor) via `isAnchorOwnedSyncPath`, not by
|
||||
// the mere absence of `opts.sourceId`/`opts.repoPath` — see that
|
||||
// function's docstring. `!opts.dryRun`: a preview must never write.
|
||||
let gitContextRoot: string;
|
||||
try {
|
||||
gitContextRoot = realpathSync(discoverGitRoot(repoPath));
|
||||
} catch (err) {
|
||||
if (
|
||||
opts.dryRun ||
|
||||
opts.signal?.aborted ||
|
||||
!existsSync(repoPath) ||
|
||||
!(await isAnchorOwnedSyncPath(engine, opts, repoPath))
|
||||
) {
|
||||
throw err;
|
||||
}
|
||||
serr(`[gbrain] auto-recovery: git-initializing brain dir ${repoPath} (no git repo found).`);
|
||||
git(repoPath, ['init', '--quiet']);
|
||||
createSyncBaselineCommit(repoPath);
|
||||
gitContextRoot = realpathSync(discoverGitRoot(repoPath));
|
||||
}
|
||||
const rawScopeRoot = opts.srcSubpath ? join(repoPath, opts.srcSubpath) : repoPath;
|
||||
if (!existsSync(rawScopeRoot)) {
|
||||
throw new Error(`Sync scope does not exist: ${rawScopeRoot}`);
|
||||
}
|
||||
const syncScopeRoot = realpathSync(rawScopeRoot);
|
||||
// NAV-1/NAV-2 scope-entry guard: the realpath-resolved scope must live
|
||||
// inside the realpath-resolved git root. Catches `--src-subpath ../escape`
|
||||
// AND a symlinked subdir pointing outside the repo, before any git op runs.
|
||||
if (syncScopeRoot !== gitContextRoot && !syncScopeRoot.startsWith(gitContextRoot + '/')) {
|
||||
throw new Error(
|
||||
`Sync scope ${syncScopeRoot} resolves outside git repo ${gitContextRoot}. ` +
|
||||
`Refusing to sync: possible path traversal via --src-subpath.`,
|
||||
);
|
||||
}
|
||||
// Relative path from git root to sync scope ('' when scope == root).
|
||||
const syncScopeRelPath = syncScopeRoot === gitContextRoot ? '' : relative(gitContextRoot, syncScopeRoot);
|
||||
const scoped = syncScopeRelPath !== '';
|
||||
// Anchor written back to sync state (sources.local_path / sync.repo_path):
|
||||
// the SCOPE path, so a follow-up bare `gbrain sync` auto-discovers the same
|
||||
// scope. Unchanged (the caller's repoPath spelling) when no --src-subpath.
|
||||
const anchorPath = opts.srcSubpath ? rawScopeRoot : repoPath;
|
||||
const fullSyncRoots = { gitContextRoot, syncScopeRoot, anchorPath };
|
||||
|
||||
serr(`[gbrain phase] sync.detect_head`);
|
||||
// Detect detached HEAD up front so the working-tree fallback fires for both
|
||||
// the default sync and `--no-pull` callers. Only the actual git pull is
|
||||
// gated on opts.noPull.
|
||||
const detachedHead = isDetachedHead(repoPath);
|
||||
const detachedHead = isDetachedHead(gitContextRoot);
|
||||
if (detachedHead && !opts.noPull) {
|
||||
// Print the caller's repoPath spelling (not the realpathed git root) —
|
||||
// it's what the operator recognizes, and tests pin it.
|
||||
serr(`Detached HEAD on ${repoPath}; skipping git pull. Syncing from local working tree.`);
|
||||
}
|
||||
|
||||
@@ -1587,7 +1919,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// hardening that cloneRepo applies. Route through pullRepo from
|
||||
// git-remote.ts so the flag set is consistent across initial clone and
|
||||
// ongoing pulls — single source of truth for the defensive flags.
|
||||
const originRemotePresent = !opts.noPull && !detachedHead ? hasOriginRemote(repoPath) : false;
|
||||
const originRemotePresent = !opts.noPull && !detachedHead ? hasOriginRemote(gitContextRoot) : false;
|
||||
if (!opts.noPull && !detachedHead && !originRemotePresent) {
|
||||
serr(`No origin remote on ${repoPath}; skipping git pull. Syncing from local working tree.`);
|
||||
}
|
||||
@@ -1626,8 +1958,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// We pass a safe default (the operator's full --timeout if set, else
|
||||
// pullRepo's own 300s default). The catch below distinguishes
|
||||
// timeout (ETIMEDOUT / SIGTERM on err.cause) from ordinary pull
|
||||
// failure.
|
||||
pullRepo(repoPath);
|
||||
// failure. Pull applies to the whole git repo (gitContextRoot), not
|
||||
// just the sync scope — git has no per-subdir pull.
|
||||
pullRepo(gitContextRoot);
|
||||
serr(`[gbrain phase] sync.git_pull done ${Date.now() - _t0}ms`);
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -1668,11 +2001,56 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// Get current HEAD
|
||||
let headCommit: string;
|
||||
try {
|
||||
headCommit = git(repoPath, ['rev-parse', 'HEAD']);
|
||||
headCommit = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
} catch {
|
||||
throw new Error(`No commits in repo ${repoPath}. Make at least one commit before syncing.`);
|
||||
// #2964: unborn-HEAD recovery. `.git` exists (discoverGitRoot succeeded
|
||||
// above) but there are zero commits — e.g. a prior self-heal `git init`
|
||||
// ran but the process died before the baseline commit landed, leaving
|
||||
// this brain permanently wedged on "No commits in repo" every night
|
||||
// thereafter. Finish the same baseline-commit self-heal the
|
||||
// discoverGitRoot catch above would have done, gated the same way
|
||||
// (ownership proven by value, never on a dry-run preview) PLUS a scope
|
||||
// check: `discoverGitRoot` walks UP from `repoPath`, so it can resolve
|
||||
// to an ANCESTOR repo, not `repoPath` itself (most plausible for a
|
||||
// `--src-subpath` sync, but `isAnchorOwnedSyncPath` already refuses
|
||||
// that case — kept here too as defense in depth against any other path
|
||||
// where gitContextRoot could diverge from repoPath). Committing at an
|
||||
// ancestor (`git add -A` at gitContextRoot) would capture sibling
|
||||
// files well outside the sync scope — refuse instead of guessing.
|
||||
if (
|
||||
opts.dryRun ||
|
||||
opts.signal?.aborted ||
|
||||
gitContextRoot !== realpathSync(repoPath) ||
|
||||
!(await isAnchorOwnedSyncPath(engine, opts, repoPath))
|
||||
) {
|
||||
throw new Error(`No commits in repo ${repoPath}. Make at least one commit before syncing.`);
|
||||
}
|
||||
serr(`[gbrain] auto-recovery: repo has no commits yet, creating baseline commit ${gitContextRoot}.`);
|
||||
createSyncBaselineCommit(gitContextRoot);
|
||||
headCommit = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
}
|
||||
|
||||
// #2964: self-heal deliberately does NOT special-case db_only/.gitignore
|
||||
// interaction beyond the COMMIT itself (createSyncBaselineCommit's
|
||||
// pathspec exclusion, which stands on its own regardless of what
|
||||
// .gitignore says). db_only content is documented as DB-sourced ("bulk
|
||||
// machine-generated content... written to disk as a local cache", see
|
||||
// docs/storage-tiering.md) — it reaches the database via ingest-specific
|
||||
// paths, never via gbrain sync's git-diff-based file collection, and
|
||||
// `.gitignore` management there is entirely about keeping db_only out of
|
||||
// git history, not about what sync imports. An earlier version of this
|
||||
// fix (Codex review rounds 6-7) tried to also guarantee db_only markdown
|
||||
// gets imported on this first sync and that .gitignore gets written
|
||||
// post-success even when called outside runSync — solving a problem
|
||||
// that, per the docs above, isn't actually in scope for what sync is
|
||||
// for. Reverted in round 8 review discussion in favor of this simpler
|
||||
// design: after self-heal, the import + any subsequent .gitignore
|
||||
// management behave EXACTLY the same as for any other brain, self-healed
|
||||
// or not (runSync's existing post-success manageGitignoreAtGitRoot call
|
||||
// covers the CLI path identically either way; the dream cycle not
|
||||
// calling it is a separate, pre-existing characteristic of the dream
|
||||
// cycle in general, not something this fix introduces or worsens).
|
||||
|
||||
// #1970: bookmark reachability. The ONLY thing that should force a full
|
||||
// reconcile is a truly-absent object; a present-but-non-ancestor bookmark
|
||||
// (history rewrite: force-push, master→main consolidation, squash) is still
|
||||
@@ -1690,7 +2068,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (lastCommit) {
|
||||
let objectPresent = true;
|
||||
try {
|
||||
git(repoPath, ['cat-file', '-t', lastCommit]);
|
||||
git(gitContextRoot, ['cat-file', '-t', lastCommit]);
|
||||
} catch {
|
||||
objectPresent = false;
|
||||
}
|
||||
@@ -1699,7 +2077,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// back to the authoritative full reconcile (which now also purges stale
|
||||
// pages for deleted files; see performFullSync's delete-reconcile pass).
|
||||
serr(`Sync anchor ${lastCommit.slice(0, 8)} object missing (gc'd after history rewrite). Running full reimport.`);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
// Observability only — NOT control flow. A non-ancestor bookmark is still
|
||||
@@ -1707,7 +2085,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// failure mode (#1970) is visible in the logs.
|
||||
let isAncestor = true;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', lastCommit, headCommit]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', lastCommit, headCommit]);
|
||||
} catch {
|
||||
isAncestor = false;
|
||||
}
|
||||
@@ -1722,7 +2100,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
// First sync
|
||||
if (!lastCommit) {
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
// v0.42.x (#1794): resumable incremental sync — resolve the PINNED target.
|
||||
@@ -1744,7 +2122,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (storedTarget) {
|
||||
let pinReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', storedTarget, headCommit]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', storedTarget, headCommit]);
|
||||
pinReachable = true;
|
||||
} catch {
|
||||
pinReachable = false;
|
||||
@@ -1778,7 +2156,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const currentVersion = String(CHUNKER_VERSION);
|
||||
const versionMismatch = storedVersion !== null && storedVersion !== currentVersion;
|
||||
const versionNeverSet = storedVersion === null && opts.sourceId !== undefined;
|
||||
const detachedWorkingTreeManifest = detachedHead ? buildDetachedWorkingTreeManifest(repoPath) : null;
|
||||
const detachedWorkingTreeManifest = detachedHead ? buildDetachedWorkingTreeManifest(gitContextRoot) : null;
|
||||
const hasDetachedWorkingTreeChanges = detachedWorkingTreeManifest !== null &&
|
||||
(detachedWorkingTreeManifest.added.length > 0 ||
|
||||
detachedWorkingTreeManifest.modified.length > 0 ||
|
||||
@@ -1814,7 +2192,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`[sync] chunker_version gate: stored=${storedVersion ?? 'unset'}, current=${currentVersion}. ` +
|
||||
`Forcing full re-chunk pass (git HEAD unchanged but pipeline version advanced).`,
|
||||
);
|
||||
const result = await performFullSync(engine, repoPath, headCommit, opts);
|
||||
const result = await performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
await writeChunkerVersion(engine, opts.sourceId, currentVersion);
|
||||
return result;
|
||||
}
|
||||
@@ -1835,7 +2213,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// buffer, and a gc'd anchor object can't be diffed at all. On either
|
||||
// `unavailable`, fall back to the authoritative full reconcile instead of
|
||||
// throwing — a slow correct reconcile beats a hard error or a silent walk.
|
||||
const delta = computeSyncDelta(repoPath, lastCommit, pin, {
|
||||
const delta = computeSyncDelta(gitContextRoot, lastCommit, pin, {
|
||||
detachedManifest: detachedWorkingTreeManifest,
|
||||
});
|
||||
if (delta.status === 'unavailable') {
|
||||
@@ -1843,30 +2221,60 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`[sync] delta ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} unavailable ` +
|
||||
`(${delta.reason}) — falling back to full reconcile.`,
|
||||
);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
const manifest = delta.manifest;
|
||||
|
||||
// Filter to syncable files (strategy-aware)
|
||||
// #753/#774 scope filter: git-diff paths are git-root-relative; when a
|
||||
// subpath scope is active, only paths under it participate. Back-compat:
|
||||
// syncScopeRelPath is '' when scope == root, so inScope is always true and
|
||||
// the filters below reduce to the pre-#774 behavior exactly.
|
||||
const inScope = (p: string): boolean =>
|
||||
!scoped || p === syncScopeRelPath || p.startsWith(syncScopeRelPath + '/');
|
||||
// --exclude patterns match the SCOPE-relative path (what the user of a
|
||||
// scoped source thinks in), same form runImport matches on full sync.
|
||||
const scopeRel = (p: string): string =>
|
||||
scoped && p.startsWith(syncScopeRelPath + '/') ? p.slice(syncScopeRelPath.length + 1) : p;
|
||||
const excluded = (p: string): boolean =>
|
||||
opts.exclude !== undefined && opts.exclude.length > 0 && matchesAnyGlob(scopeRel(p), opts.exclude);
|
||||
|
||||
// Filter to syncable files (strategy-aware + scope-aware + exclude-aware)
|
||||
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
|
||||
// #1970 (F-C): a rename whose DESTINATION is unsyncable drops out of BOTH
|
||||
// `renamed` (only `r.to` is kept below) AND `deleted` (git emits it as `R`,
|
||||
// not `D`), leaving the OLD page stale. Fold the source side into the delete
|
||||
// set. isSyncable(r.from) excludes metafiles automatically, so a rename of a
|
||||
// metafile is left untouched (matching the #1433 metafile-skip invariant).
|
||||
// #774: a rename whose destination LEFT the scope is the same class — the
|
||||
// old page's backing file is gone from this source's slice of the repo.
|
||||
const renamedToUnsyncable = manifest.renamed
|
||||
.filter(r => isSyncable(r.from, syncOpts) && !isSyncable(r.to, syncOpts))
|
||||
.filter(r => inScope(r.from) && isSyncable(r.from, syncOpts) &&
|
||||
!(inScope(r.to) && isSyncable(r.to, syncOpts)))
|
||||
.map(r => r.from);
|
||||
const filtered: SyncManifest = {
|
||||
added: manifest.added.filter(p => isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p, syncOpts)),
|
||||
added: manifest.added.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
|
||||
deleted: unique([
|
||||
...manifest.deleted.filter(p => isSyncable(p, syncOpts)),
|
||||
...manifest.deleted.filter(p => inScope(p) && isSyncable(p, syncOpts)),
|
||||
...renamedToUnsyncable,
|
||||
]),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to, syncOpts)),
|
||||
renamed: manifest.renamed.filter(r => inScope(r.to) && !excluded(r.to) && isSyncable(r.to, syncOpts)),
|
||||
};
|
||||
|
||||
// NAV-4: warn when --exclude filtered out every candidate change — almost
|
||||
// always a mistyped pattern, and otherwise indistinguishable from
|
||||
// "up to date" in the output.
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
const excludeCandidates = [...manifest.added, ...manifest.modified]
|
||||
.filter(p => inScope(p) && isSyncable(p, syncOpts));
|
||||
if (excludeCandidates.length > 0 && excludeCandidates.every(excluded)) {
|
||||
console.warn(
|
||||
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
|
||||
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Delete pages that became un-syncable (modified but filtered out).
|
||||
// v0.20.0 Cathedral II SP-5: resolveSlugForPath picks the right slug shape
|
||||
// (markdown vs code) based on the chunker's classifier, so a Rust file that
|
||||
@@ -1890,14 +2298,20 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// delete the page. That's the same pre-fix behavior — removing the
|
||||
// page requires `gbrain pages purge-deleted` or a direct MCP delete.
|
||||
// Filed as v0.42+ follow-up for a `gbrain pages remove <slug>` surface.
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p, syncOpts));
|
||||
const unsyncableModified = manifest.modified.filter(p => inScope(p) && !isSyncable(p, syncOpts));
|
||||
// v0.18.0+ multi-source: scope getPage + deletePage to opts.sourceId so
|
||||
// unsyncable cleanup in source A doesn't accidentally sweep same-slug
|
||||
// pages in sources B/C/D.
|
||||
const pageOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
for (const path of unsyncableModified) {
|
||||
// v0.41.13 #1433: never delete on metafile classification.
|
||||
if (unsyncableReason(path, syncOpts) === 'metafile') continue;
|
||||
// #2404 hardening: same for 'pruned-dir' — a page under a pruned
|
||||
// directory can only exist via a deliberate put_page (sync never
|
||||
// imports those paths), so "the file was modified" is not evidence
|
||||
// the page is stale. Deleting here silently destroyed put-created
|
||||
// pages every time their materialized file landed in a commit.
|
||||
const reason = unsyncableReason(path, syncOpts);
|
||||
if (reason === 'metafile' || reason === 'pruned-dir') continue;
|
||||
const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId);
|
||||
try {
|
||||
const existing = await engine.getPage(slug, pageOpts);
|
||||
@@ -1938,7 +2352,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// (#1794): advance to the PINNED target, and clear any checkpoint (a resume
|
||||
// whose remaining range turned out to have no syncable changes still
|
||||
// completes cleanly here).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(gitContextRoot, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
@@ -2325,8 +2739,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// throw here crashes the whole sync mid-run and freezes the checkpoint,
|
||||
// defeating --skip-failed. A `skipped` result carrying an error is also
|
||||
// captured so the failure is recorded rather than silently dropped.
|
||||
const filePath = join(repoPath, to);
|
||||
if (existsSync(filePath)) {
|
||||
// Paths from git diff are relative to gitContextRoot; join from there.
|
||||
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
|
||||
// repo (committed symlink pointing out).
|
||||
const filePath = join(gitContextRoot, to);
|
||||
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
|
||||
try {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
@@ -2411,8 +2828,8 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
progress.start('sync.imports', importsToDo.length);
|
||||
|
||||
// Core import logic shared by serial and parallel paths.
|
||||
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
|
||||
const syncRepoPath = repoPath!;
|
||||
// Paths from git diff are relative to gitContextRoot; join from there.
|
||||
const syncRepoPath = gitContextRoot;
|
||||
// paced-backfill (T3 / C9 / CX4): ONE shared pacer across all worker
|
||||
// engines. This is the multi-pool permit case — each parallel worker owns a
|
||||
// separate PostgresEngine, so a single worker count can't bound TOTAL
|
||||
@@ -2500,6 +2917,16 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
// #774 NAV-1 TOCTOU: re-validate the file's realpath at import time so a
|
||||
// committed symlink pointing outside the repo (or one swapped in after
|
||||
// the scope-entry check) is never read. Recorded as a failure —
|
||||
// fail-closed: the bookmark won't advance past a symlink escape.
|
||||
if (!isPathSafe(filePath, gitContextRoot)) {
|
||||
failedFiles.push({ path, error: 'path resolves outside git repo (symlink escape)' });
|
||||
progressAt.last = Date.now();
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
// v0.41.37.0 #1569: per-file BEGIN heartbeat, emitted BEFORE importFile so a
|
||||
// hang names the stalling file (the progress.tick below only fires AFTER
|
||||
// importFile returns — useless when one file wedges). Off by default
|
||||
@@ -2703,11 +3130,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// - pin NOT an ancestor of HEAD (history REWRITE / reset / force-push) →
|
||||
// the tree we imported against is gone. Block; do not advance.
|
||||
try {
|
||||
const currentHead = git(repoPath, ['rev-parse', 'HEAD']);
|
||||
const currentHead = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
if (currentHead !== pin) {
|
||||
let pinStillReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', pin, currentHead]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', pin, currentHead]);
|
||||
pinStillReachable = true;
|
||||
} catch {
|
||||
pinStillReachable = false;
|
||||
@@ -2748,9 +3175,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// "fresh". The checkpoint rows clear here — CONVERGENCE CONTRACT: sync
|
||||
// convergence == IMPORT convergence; downstream extract/facts/embed is
|
||||
// decoupled (its own resumable stale sweeps).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(gitContextRoot, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
@@ -2799,7 +3226,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// checkpoint is INTENTIONALLY left in place — the banked completed set lets
|
||||
// the next run skip the drained files and re-attempt only the failures.
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
// v0.42.x (#1794): surface banked progress so a blocked run doesn't read as
|
||||
// total loss (last_commit is unchanged by design; the checkpoint is banked).
|
||||
serr(
|
||||
@@ -2870,8 +3297,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (!opts.noExtract && totalChanges <= 100 && pagesAffected.length > 0) {
|
||||
try {
|
||||
const { extractLinksForSlugs, extractTimelineForSlugs, stampExtracted } = await import('./extract.ts');
|
||||
const linksCreated = await extractLinksForSlugs(engine, repoPath, pagesAffected, extractOpts);
|
||||
const timelineCreated = await extractTimelineForSlugs(engine, repoPath, pagesAffected, extractOpts);
|
||||
// #774: pages' source_path is git-root-relative, so extract resolves
|
||||
// files from gitContextRoot (== repoPath realpath when unscoped).
|
||||
const linksCreated = await extractLinksForSlugs(engine, gitContextRoot, pagesAffected, extractOpts);
|
||||
const timelineCreated = await extractTimelineForSlugs(engine, gitContextRoot, pagesAffected, extractOpts);
|
||||
if (linksCreated > 0 || timelineCreated > 0) {
|
||||
slog(` Extracted: ${linksCreated} links, ${timelineCreated} timeline entries`);
|
||||
}
|
||||
@@ -2976,11 +3405,21 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
async function performFullSync(
|
||||
engine: BrainEngine,
|
||||
repoPath: string,
|
||||
// #753/#774: the three roots resolved once at the top of performSyncInner.
|
||||
// gitContextRoot — git repo root (git ops, slug base for scoped syncs)
|
||||
// syncScopeRoot — where files are walked/imported (== gitContextRoot
|
||||
// when no subpath scope is active)
|
||||
// anchorPath — what gets written back to sync.repo_path/local_path
|
||||
roots: { gitContextRoot: string; syncScopeRoot: string; anchorPath: string },
|
||||
headCommit: string,
|
||||
opts: SyncOpts,
|
||||
): Promise<SyncResult> {
|
||||
// Dry-run: walk the repo, count syncable files, return without writing.
|
||||
const { gitContextRoot, syncScopeRoot, anchorPath } = roots;
|
||||
// Scoped sync → slugs/source_path are git-root-relative (matches the
|
||||
// incremental path's git-diff paths). Unscoped → undefined (dir-relative,
|
||||
// the pre-#774 behavior, byte-for-byte).
|
||||
const slugRoot = syncScopeRoot !== gitContextRoot ? gitContextRoot : undefined;
|
||||
// Dry-run: walk the scope, count syncable files, return without writing.
|
||||
// Fixes the silent-write-on-dry-run bug where performFullSync called
|
||||
// runImport unconditionally regardless of opts.dryRun.
|
||||
//
|
||||
@@ -2990,11 +3429,14 @@ async function performFullSync(
|
||||
// code --dry-run` always reported zero files even when ~1500 code
|
||||
// files were waiting.
|
||||
if (opts.dryRun) {
|
||||
const allFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' });
|
||||
let allFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' });
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(syncScopeRoot, abs), opts.exclude));
|
||||
}
|
||||
slog(
|
||||
`Full-sync dry run (strategy=${opts.strategy ?? 'markdown'}): ` +
|
||||
`${allFiles.length} file(s) would be imported ` +
|
||||
`from ${repoPath} @ ${headCommit.slice(0, 8)}.`,
|
||||
`from ${syncScopeRoot} @ ${headCommit.slice(0, 8)}.`,
|
||||
);
|
||||
return {
|
||||
status: 'dry_run',
|
||||
@@ -3017,21 +3459,24 @@ async function performFullSync(
|
||||
// sync and the jobs handler.
|
||||
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
|
||||
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
|
||||
slog(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
slog(`Running full import of ${syncScopeRoot}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
const importArgs = [syncScopeRoot];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
|
||||
// v0.31.2: thread strategy through so code-strategy first sync
|
||||
// actually enumerates code files (closes bug 1).
|
||||
// v0.30.x: thread sourceId so performFullSync routes pages to the named
|
||||
// source (incremental path already does this).
|
||||
// #753/#774: thread exclude (--exclude CLI) + slugRoot (monorepo subdir).
|
||||
const _fullImportT0 = Date.now();
|
||||
serr(`[gbrain phase] sync.fullsync.import start strategy=${opts.strategy ?? 'markdown'}`);
|
||||
const result = await runImport(engine, importArgs, {
|
||||
commit: headCommit,
|
||||
strategy: opts.strategy,
|
||||
sourceId: opts.sourceId,
|
||||
exclude: opts.exclude,
|
||||
slugRoot,
|
||||
// issue #1939: performFullSync owns the failure ledger + bookmark via the
|
||||
// shared gate below; don't let runImport double-record or write its own.
|
||||
managedBookmark: true,
|
||||
@@ -3055,9 +3500,9 @@ async function performFullSync(
|
||||
const advanceFull = async (): Promise<void> => {
|
||||
// Persist sync state so the next sync is incremental. Routed through
|
||||
// writeSyncAnchor so --source pins the right sources row.
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(gitContextRoot));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
};
|
||||
|
||||
@@ -3084,7 +3529,7 @@ async function performFullSync(
|
||||
);
|
||||
}
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: null,
|
||||
@@ -3140,16 +3585,24 @@ async function performFullSync(
|
||||
// backslash paths while a stored source_path can hold git-derived forward
|
||||
// slashes; without normalization every file-backed page mismatches, looks
|
||||
// stale, and the reconcile wipes the whole source.
|
||||
const currentFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(repoPath, abs));
|
||||
// #774: scoped syncs store git-root-relative source_paths (slugRoot), so
|
||||
// relativize the walk to the same base — otherwise every page mismatches
|
||||
// and the mass-delete valve trips on a perfectly healthy scoped source.
|
||||
const currentFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(slugRoot ?? syncScopeRoot, abs));
|
||||
const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>(
|
||||
`SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`,
|
||||
[sid],
|
||||
);
|
||||
// #774: a scoped full sync is authoritative ONLY for its scope — pages
|
||||
// whose source_path lives outside the subpath (e.g. from an earlier
|
||||
// root-level sync of this source) are out of this walk's sight and must
|
||||
// not be treated as stale.
|
||||
const scopePrefix = slugRoot ? relative(gitContextRoot, syncScopeRoot) + '/' : '';
|
||||
const plan = planReconcileDeletes(
|
||||
rows,
|
||||
currentFiles,
|
||||
p => isSyncable(p, reconcileSyncOpts),
|
||||
p => (scopePrefix === '' || p.startsWith(scopePrefix)) && isSyncable(p, reconcileSyncOpts),
|
||||
);
|
||||
if (plan.staleSlugs.length > 0 && plan.massDelete && !massReconcileAllowed()) {
|
||||
// #2828 mass-delete safety valve: a reconcile that would sweep more than
|
||||
@@ -3169,9 +3622,45 @@ async function performFullSync(
|
||||
`GBRAIN_ALLOW_MASS_RECONCILE=1 to restore the old behavior.`,
|
||||
);
|
||||
} else if (plan.staleSlugs.length > 0) {
|
||||
// #2426: a stale page whose source_path was NEVER committed to git is
|
||||
// DB-only write-through (the file was written into the clone but never
|
||||
// committed/pushed, then lost — e.g. a fresh clone). "Absent from git"
|
||||
// is the SYMPTOM of that bug, not evidence the content is disposable.
|
||||
// Keep those pages and re-export their markdown to the working tree so
|
||||
// they're file-backed again; only pages whose file once existed in git
|
||||
// history (i.e. was genuinely deleted) are reconcile-deleted.
|
||||
const everCommitted = listEverCommittedPaths(gitContextRoot);
|
||||
const pathBySlug = new Map(rows.map(r => [r.slug, r.source_path]));
|
||||
let deletableSlugs = plan.staleSlugs;
|
||||
const dbOnlySlugs: string[] = [];
|
||||
if (everCommitted) {
|
||||
deletableSlugs = [];
|
||||
for (const slug of plan.staleSlugs) {
|
||||
const sp = pathBySlug.get(slug);
|
||||
if (sp && !everCommitted.has(sp.replace(/\\/g, '/'))) dbOnlySlugs.push(slug);
|
||||
else deletableSlugs.push(slug);
|
||||
}
|
||||
}
|
||||
if (dbOnlySlugs.length > 0) {
|
||||
let reExported = 0;
|
||||
try {
|
||||
const { writePageThrough } = await import('../core/write-through.ts');
|
||||
for (const slug of dbOnlySlugs) {
|
||||
const r = await writePageThrough(engine, slug, { sourceId: sid });
|
||||
if (r.written) reExported++;
|
||||
}
|
||||
} catch { /* best-effort — pages are preserved either way */ }
|
||||
serr(
|
||||
`\n Kept ${dbOnlySlugs.length} page(s) whose markdown was never committed to git ` +
|
||||
`(DB-only write-through — not deleting).` +
|
||||
(reExported > 0 ? ` Re-exported ${reExported} of them to the working tree.` : '') +
|
||||
`\n Commit + push them (e.g. scripts/brain-commit-push.sh, or 'gbrain sources harden') ` +
|
||||
`so the next sync sees them as file-backed.`,
|
||||
);
|
||||
}
|
||||
const deleteScopedOpts = { sourceId: sid };
|
||||
for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
for (let i = 0; i < deletableSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = deletableSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
try {
|
||||
const deleted = await engine.deletePages(batch, deleteScopedOpts);
|
||||
reconciledDeletes += deleted.length;
|
||||
@@ -3290,6 +3779,34 @@ export function planReconcileDeletes(
|
||||
return { staleSlugs, reconcilableCount: reconcilable.length, massDelete };
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: every repo-relative path that ever appeared as an ADD in git history
|
||||
* (rename detection off, so a `git mv` destination still counts as an add).
|
||||
* Used by the full-sync reconcile to distinguish "file was committed and later
|
||||
* deleted" (genuine delete → reconcile) from "file was NEVER committed"
|
||||
* (DB-only write-through → preserve). Returns null when `repoPath` isn't a git
|
||||
* work tree or git is unavailable — callers keep the plain-directory behavior.
|
||||
* Forward-slash-normalized to match `normalizeReconcilePath` membership tests.
|
||||
*/
|
||||
export function listEverCommittedPaths(repoPath: string): Set<string> | null {
|
||||
let stdout: string;
|
||||
try {
|
||||
stdout = execFileSync(
|
||||
'git',
|
||||
['-C', repoPath, '-c', 'core.quotepath=off', 'log', '--all', '--no-renames',
|
||||
'--diff-filter=A', '--format=', '--name-only'],
|
||||
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
|
||||
);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const set = new Set<string>();
|
||||
for (const line of stdout.split('\n')) {
|
||||
if (line) set.add(line.replace(/\\/g, '/'));
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve
|
||||
* behavior for the rare intentional bulk removal. Env-only (an incident-time
|
||||
@@ -3400,6 +3917,19 @@ export function composeAbortSignals(
|
||||
return AbortSignal.any(live);
|
||||
}
|
||||
|
||||
/**
|
||||
* #753/#774: `.gitignore` must be managed at the git ROOT — when a source's
|
||||
* local_path (or --repo) points at a monorepo subdirectory, writing ignore
|
||||
* entries into the subdir would create a stray `.gitignore` git doesn't
|
||||
* consult for the repo-level db_only rules. Best-effort: falls back to the
|
||||
* given path when git discovery fails (manageGitignore no-ops on non-repos).
|
||||
*/
|
||||
function manageGitignoreAtGitRoot(path: string, engineKind?: 'pglite' | 'postgres'): void {
|
||||
let root = path;
|
||||
try { root = discoverGitRoot(path); } catch { /* best-effort */ }
|
||||
manageGitignore(root, engineKind);
|
||||
}
|
||||
|
||||
export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
// v0.40 Federated Sync v2: `gbrain sync trigger` subcommand
|
||||
// Routes to runSyncTrigger which queues a 'sync' minion job with
|
||||
@@ -3432,6 +3962,13 @@ Options:
|
||||
--repo <path> Path to the brain repo. Defaults to the path
|
||||
saved by 'gbrain init'.
|
||||
--full Force a full re-sync (rare; usually incremental).
|
||||
--src-subpath <dir> Sync only this subdirectory of the git repo (monorepo
|
||||
pattern: N logical sources in one repo). Git pull/diff
|
||||
run at the repo root; imports are scoped to the subdir
|
||||
and slugs stay root-relative (wiki/page1). Passing the
|
||||
subdirectory directly as --repo also works.
|
||||
--exclude <glob> Exclude files matching the glob from sync (repeatable;
|
||||
matched against the scope-relative path).
|
||||
--dry-run Show what would be synced without writing.
|
||||
--skip-failed Acknowledge previously-recorded sync failures so
|
||||
the bookmark can advance past unparseable files.
|
||||
@@ -3591,6 +4128,20 @@ See also:
|
||||
process.exit(1);
|
||||
}
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
// #753/#774: monorepo subdir-source flags. --exclude is repeatable.
|
||||
const srcSubpath = args.find((a, i) => args[i - 1] === '--src-subpath') || undefined;
|
||||
const excludePatterns: string[] = [];
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--exclude' && i + 1 < args.length) excludePatterns.push(args[i + 1]);
|
||||
}
|
||||
if (syncAll && (srcSubpath || excludePatterns.length > 0)) {
|
||||
console.error(
|
||||
`--src-subpath/--exclude scope a single sync invocation; they cannot be combined with --all. ` +
|
||||
`For --all runs, register the subdirectory as the source's local_path instead ` +
|
||||
`(gbrain sources add <id> --path <repo>/<subdir>).`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
|
||||
const parallelStr = args.find((a, i) => args[i - 1] === '--parallel');
|
||||
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
|
||||
@@ -3850,7 +4401,7 @@ See also:
|
||||
result.status !== 'blocked_by_failures' &&
|
||||
result.status !== 'partial'
|
||||
) {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
manageGitignoreAtGitRoot(src.local_path!, engine.kind);
|
||||
}
|
||||
// D18: auto-enqueue embed-backfill per source (unless opted out).
|
||||
// v0.41.13.0 (T7 / D-V3-5): partial excluded — the next clean sync
|
||||
@@ -4038,6 +4589,8 @@ See also:
|
||||
const opts: SyncOpts = {
|
||||
repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, sourceId,
|
||||
strategy: strategyArg, concurrency,
|
||||
srcSubpath,
|
||||
exclude: excludePatterns.length > 0 ? excludePatterns : undefined,
|
||||
signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
|
||||
};
|
||||
|
||||
@@ -4121,7 +4674,7 @@ See also:
|
||||
) {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
manageGitignoreAtGitRoot(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
// v0.42.42.0 (#2139, Step 4b): the inline gate auto-deferred this run's
|
||||
@@ -4170,7 +4723,7 @@ See also:
|
||||
) {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
manageGitignoreAtGitRoot(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
|
||||
+12
-6
@@ -101,12 +101,18 @@ async function getPageId(engine: BrainEngine, slug: string, sourceId?: string):
|
||||
return rows[0].id;
|
||||
}
|
||||
|
||||
async function resolveTakesSourceId(engine: BrainEngine): Promise<string | undefined> {
|
||||
try {
|
||||
return await resolveSourceId(engine, null);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
// Fail-closed (#2698 residual, TODOS.md): `resolveSourceId` only ever
|
||||
// throws when a source WAS explicitly in play — an invalid or
|
||||
// unregistered `GBRAIN_SOURCE`, a `.gbrain-source` dotfile pointing at a
|
||||
// source that doesn't exist, or a genuine DB error — never for "nothing
|
||||
// configured" (that path resolves cleanly to the seeded `'default'`
|
||||
// source, tier 6 of resolveSourceId). Swallowing those errors here used
|
||||
// to fall back to the unscoped slug-only page lookup, silently
|
||||
// reintroducing the pre-#2698 cross-source write bug whenever resolution
|
||||
// merely errored instead of resolving cleanly. Let it propagate so the
|
||||
// write is blocked instead of silently unscoped.
|
||||
async function resolveTakesSourceId(engine: BrainEngine): Promise<string> {
|
||||
return resolveSourceId(engine, null);
|
||||
}
|
||||
|
||||
function readBodyOrEmpty(path: string): string {
|
||||
|
||||
+63
-1
@@ -2982,6 +2982,26 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
|
||||
const providerOptions: Record<string, any> = {};
|
||||
if (useCache) {
|
||||
// Call-level `providerOptions.anthropic.cacheControl` is NOT a no-op:
|
||||
// @ai-sdk/anthropic 3.0.47+ passes it through as a top-level
|
||||
// `cache_control` field on the Anthropic request body, which the
|
||||
// Messages API resolves as its documented "auto-cache the last
|
||||
// cacheable block in the request" shorthand (see Anthropic's
|
||||
// prompt-caching docs — "top-level auto-caching ... is the simplest
|
||||
// option when you don't need fine-grained placement"). Keep it: it's
|
||||
// what gives a growing multi-turn conversation (toolLoop()) a rolling
|
||||
// cache breakpoint on each turn's tail for free, without us having to
|
||||
// hand-roll the marker-walking logic subagent.ts's raw-SDK path uses.
|
||||
//
|
||||
// But "last cacheable block" is the wrong block for gbrain#2490's
|
||||
// actual callers (page-summary, skillopt, enrich): those are
|
||||
// single-turn calls with a STABLE system prompt and a DIFFERENT user
|
||||
// message every time, so the auto-marker lands on the ever-varying
|
||||
// tail — every call WRITES a fresh cache entry and never READS a prior
|
||||
// one (cache_read_input_tokens stays 0 forever). Caching the stable
|
||||
// prefix needs an EXPLICIT breakpoint on the system block itself,
|
||||
// which is applied below via a `SystemModelMessage` (round-trips its
|
||||
// own `providerOptions`) instead of a bare string.
|
||||
providerOptions.anthropic = { cacheControl: { type: 'ephemeral' } };
|
||||
}
|
||||
// OpenAI prompt_cache_key (native-openai only): a stable per-prefix routing
|
||||
@@ -3000,6 +3020,30 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
}
|
||||
applyConfiguredChatProviderOptions(providerOptions, cfg, recipe.id, modelId);
|
||||
|
||||
// Derive ONE canonical cache-control value AFTER config merging and reuse
|
||||
// it for every breakpoint (system block, last tool def, call-level). If
|
||||
// `provider_chat_options.anthropic.cacheControl` overrides the TTL (e.g.
|
||||
// `{ type: 'ephemeral', ttl: '1h' }`), that override lands in
|
||||
// `providerOptions.anthropic.cacheControl` via the deep-merge above —
|
||||
// reusing it here (instead of hardcoding `{ type: 'ephemeral' }` per
|
||||
// breakpoint) keeps every marker in the request on the same TTL.
|
||||
const cacheControlValue: { type: 'ephemeral'; ttl?: '5m' | '1h' } | undefined = useCache
|
||||
? (providerOptions.anthropic?.cacheControl ?? { type: 'ephemeral' })
|
||||
: undefined;
|
||||
|
||||
// Anthropic-only secondary breakpoint: mark the LAST tool def too (mirrors
|
||||
// subagent.ts's raw-SDK path — Anthropic caches everything up to and
|
||||
// including the last `cache_control` block it sees in the request, so
|
||||
// marking the last tool extends the cached prefix through the whole tool
|
||||
// list). `tool.providerOptions.anthropic.cacheControl` is the shape
|
||||
// @ai-sdk/anthropic 3.x reads for tool-def breakpoints.
|
||||
if (cacheControlValue && opts.tools && opts.tools.length > 0 && tools) {
|
||||
const lastTool = tools[opts.tools[opts.tools.length - 1]!.name];
|
||||
if (lastTool) {
|
||||
lastTool.providerOptions = { anthropic: { cacheControl: cacheControlValue } };
|
||||
}
|
||||
}
|
||||
|
||||
let _budgetRecorded = false;
|
||||
const _recordBudget = (modelLabel: string, inputTokens: number, outputTokens: number): void => {
|
||||
if (!tracker || _budgetRecorded) return;
|
||||
@@ -3016,10 +3060,28 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
}
|
||||
};
|
||||
|
||||
// The actual Anthropic system-prompt cache breakpoint. A bare string
|
||||
// `system` produces `{ role: 'system', content }` with no `providerOptions`
|
||||
// field (ai@6's convertToLanguageModelPrompt), so @ai-sdk/anthropic's
|
||||
// getCacheControl(providerOptions) on that block always resolves to
|
||||
// nothing. Passing a `SystemModelMessage` object instead — the shape `ai`
|
||||
// documents specifically for "additional provider options (e.g. for
|
||||
// caching)" — round-trips `providerOptions` onto that block. Byte-identical
|
||||
// to the old bare-string form when useCache is false. Reuses
|
||||
// `cacheControlValue` (the config-merged value) so this breakpoint's TTL
|
||||
// always matches the last-tool and call-level breakpoints.
|
||||
const systemParam = cacheControlValue && opts.system
|
||||
? {
|
||||
role: 'system' as const,
|
||||
content: opts.system,
|
||||
providerOptions: { anthropic: { cacheControl: cacheControlValue } },
|
||||
}
|
||||
: opts.system;
|
||||
|
||||
try {
|
||||
const result = await _generateTextTransport({
|
||||
model,
|
||||
system: opts.system,
|
||||
system: systemParam,
|
||||
messages: toModelMessages(repairToolPairing(opts.messages)) as any,
|
||||
tools: opts.tools && opts.tools.length > 0 ? tools : undefined,
|
||||
maxOutputTokens: opts.maxTokens ?? defaultMaxOutputTokens(modelStr),
|
||||
|
||||
@@ -23,6 +23,8 @@ import { zhipu } from './zhipu.ts';
|
||||
import { azureOpenAI } from './azure-openai.ts';
|
||||
import { zeroentropyai } from './zeroentropyai.ts';
|
||||
import { llamaServerReranker } from './llama-server-reranker.ts';
|
||||
import { moonshot } from './moonshot.ts';
|
||||
import { mistral } from './mistral.ts';
|
||||
|
||||
const ALL: Recipe[] = [
|
||||
openai,
|
||||
@@ -42,6 +44,8 @@ const ALL: Recipe[] = [
|
||||
zhipu,
|
||||
azureOpenAI,
|
||||
zeroentropyai,
|
||||
moonshot,
|
||||
mistral,
|
||||
];
|
||||
|
||||
/** Map from `provider:id` key to recipe. */
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Mistral AI exposes an OpenAI-compatible API at https://api.mistral.ai/v1
|
||||
* (/embeddings + /chat/completions). EU-hosted — the reason this recipe
|
||||
* exists: a brain that must stay inside EU jurisdiction can run embed +
|
||||
* expansion + chat on a single provider without a US hop.
|
||||
*
|
||||
* Verified against the live API on 2026-07-19 (model catalog, embedding
|
||||
* dimensions, dimension-parameter rejection, and the batch ceiling — see
|
||||
* the notes on each field below).
|
||||
*
|
||||
* DIMENSIONS — mistral-embed is FIXED 1024 and accepts NO dimension
|
||||
* parameter at all. Both spellings are rejected upstream:
|
||||
* {"dimensions": 512} -> 400 extra_forbidden (not in the API schema)
|
||||
* {"output_dimension": 512} -> 400 "This model does not support output_dimension"
|
||||
* The generic `openai-compatible` branch of dims.ts:dimsProviderOptions()
|
||||
* already falls through to `return undefined` for these model ids, so no
|
||||
* dimension field is emitted. Do NOT add mistral-embed to any of the
|
||||
* flexible-dim allowlists there — it would 400 every embed call. Same
|
||||
* contract as voyage-4-nano, for the same reason.
|
||||
*
|
||||
* codestral-embed / codestral-embed-2505 are deliberately NOT listed: they
|
||||
* return 1536 dims, and a touchpoint carries a single `default_dims`.
|
||||
* Mixing them under a 1024 declaration is the mixed-dim footgun
|
||||
* embedding-dim-check.ts exists to catch. They are code-retrieval models
|
||||
* anyway; a prose brain wants mistral-embed.
|
||||
*/
|
||||
export const mistral: Recipe = {
|
||||
id: 'mistral',
|
||||
name: 'Mistral AI',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://api.mistral.ai/v1',
|
||||
auth_env: {
|
||||
required: ['MISTRAL_API_KEY'],
|
||||
setup_url: 'https://console.mistral.ai/api-keys',
|
||||
},
|
||||
touchpoints: {
|
||||
embedding: {
|
||||
models: ['mistral-embed', 'mistral-embed-2312'],
|
||||
default_dims: 1024,
|
||||
// Mistral's published list price. Advisory only — canonical embedding
|
||||
// spend accounting lives in src/core/embedding-pricing.ts.
|
||||
cost_per_1m_tokens_usd: 0.1,
|
||||
price_last_verified: '2026-07-19',
|
||||
// Measured ceiling, not a doc guess: the /embeddings endpoint accepts a
|
||||
// 65,286-token batch and rejects 66,960 with
|
||||
// 400 code 3210 "Too many tokens overall, split into more batches."
|
||||
// -> the real cap is 65,536 (64K) tokens per request.
|
||||
max_batch_tokens: 65_536,
|
||||
// chars_per_token is a DIVISOR in splitByTokenBudget()
|
||||
// (estTokens = text.length / charsPerToken), so a LOWER value is the
|
||||
// conservative direction. The module default of 4 is an English-prose
|
||||
// assumption; German prose measured 3.58 here, and code/JSON/CJK runs
|
||||
// denser still. 2 keeps the estimate above the real token count for
|
||||
// every content shape we see.
|
||||
chars_per_token: 2,
|
||||
// With safety_factor 0.5 the pre-split budget is 32,768 estimated
|
||||
// tokens = 65,536 chars. Worst realistic density (~1.5 chars/token)
|
||||
// puts that at ~43.7K real tokens — still clear of the 64K ceiling.
|
||||
safety_factor: 0.5,
|
||||
},
|
||||
expansion: {
|
||||
models: ['ministral-3b-latest', 'mistral-small-latest'],
|
||||
price_last_verified: '2026-07-19',
|
||||
},
|
||||
chat: {
|
||||
models: [
|
||||
'mistral-small-latest', 'mistral-medium-latest', 'mistral-large-latest',
|
||||
'ministral-3b-latest', 'ministral-8b-latest', 'magistral-small-latest',
|
||||
],
|
||||
supports_tools: true,
|
||||
// Same call as the Moonshot recipe: ordinary tool calls are fine, but
|
||||
// gbrain's subagent loop stays Anthropic-pinned for stable tool_use_id
|
||||
// behavior across crashes/replays.
|
||||
supports_subagent_loop: false,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 262144,
|
||||
price_last_verified: '2026-07-19',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://console.mistral.ai/api-keys, then `export MISTRAL_API_KEY=...` and use `mistral:mistral-embed` (1024 dims) for embeddings.',
|
||||
};
|
||||
@@ -0,0 +1,42 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Moonshot AI / Kimi Open Platform. Kimi exposes an OpenAI-compatible
|
||||
* /v1/chat/completions API at https://api.moonshot.ai/v1.
|
||||
*
|
||||
* Verified against Kimi API docs and live /v1/models on 2026-06-23.
|
||||
* The recipe is local-production glue until upstream GBrain carries a native
|
||||
* Moonshot recipe; keep it registered in the local patch registry.
|
||||
*/
|
||||
export const moonshot: Recipe = {
|
||||
id: 'moonshot',
|
||||
name: 'Moonshot AI / Kimi',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://api.moonshot.ai/v1',
|
||||
auth_env: {
|
||||
required: ['MOONSHOT_API_KEY'],
|
||||
setup_url: 'https://platform.kimi.ai/console/api-keys',
|
||||
},
|
||||
touchpoints: {
|
||||
expansion: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
// Kimi pricing varies by current promotional/account terms; do not use
|
||||
// this advisory field for budget enforcement. Canonical budget pricing
|
||||
// belongs in src/core/model-pricing.ts when verified for the account.
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
chat: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
supports_tools: true,
|
||||
// Kimi tool calling is enough for ordinary chat/tool calls. GBrain's
|
||||
// subagent loop remains Anthropic-pinned because upstream requires stable
|
||||
// Anthropic-style tool_use_id behavior across crashes/replays.
|
||||
supports_subagent_loop: false,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 256000,
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://platform.kimi.ai/console/api-keys, then `export MOONSHOT_API_KEY=...` and use `moonshot:kimi-k2.7-code`.',
|
||||
};
|
||||
@@ -100,7 +100,15 @@ function gbrainHome(): string {
|
||||
* core→commands import). which gbrain → process.execPath → argv[1] → "gbrain". */
|
||||
function resolveGbrainCliPath(): string {
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
// #2747: `env: process.env` required under Bun — see the sibling copy
|
||||
// of this function in commands/autopilot.ts for the full explanation
|
||||
// (Bun snapshots process.env at its own startup; execSync without an
|
||||
// explicit env is blind to any PATH mutation since then).
|
||||
const which = execSync('which gbrain', {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
env: process.env,
|
||||
}).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on PATH */ }
|
||||
const exec = process.execPath ?? '';
|
||||
@@ -183,15 +191,17 @@ if [ "\${1:-}" = "--push-only" ]; then
|
||||
fi
|
||||
|
||||
_msg="\${1:?usage: brain-commit-push.sh <message> <path> [paths...]}"; shift || true
|
||||
# Pull first so the local tree is current before we stage.
|
||||
git fetch origin >/dev/null 2>&1 || true
|
||||
git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; }
|
||||
|
||||
# EXPLICIT paths only — never a blind 'git add -A' (would risk committing
|
||||
# secrets, temp files, or unrelated edits).
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2
|
||||
fi
|
||||
# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage)
|
||||
# aborted on any dirty tree — 'cannot pull with rebase: You have unstaged
|
||||
# changes' — so the helper could never commit a MODIFIED page (exactly the
|
||||
# write-through case). Stage + commit first; brain_push below already handles
|
||||
# a remote that advanced (push -> rejected -> pull --rebase -> push).
|
||||
git add -- "$@"
|
||||
if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi
|
||||
git commit -m "$_msg"
|
||||
@@ -337,6 +347,47 @@ function uninstallLocalHook(repoPath: string): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the gbrain durability post-commit hook is installed — i.e. the
|
||||
* user opted this repo into push-durability via `gbrain sources harden`.
|
||||
* Cheap (one git-config read + one file read); used as the gate for
|
||||
* write-through auto-commit (#2426).
|
||||
*/
|
||||
export function isDurabilityHardened(repoPath: string): boolean {
|
||||
try {
|
||||
const { dir } = resolveHooksDir(repoPath);
|
||||
const hookPath = join(dir, 'post-commit');
|
||||
return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: best-effort commit of a single write-through artifact so DB writes
|
||||
* reach git (the post-commit hook then background-pushes). Pre-fix,
|
||||
* write-through `.md` accumulated uncommitted forever: it never reached the
|
||||
* remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full`
|
||||
* delete-reconcile treated the never-committed pages as disposable.
|
||||
*
|
||||
* Path-limited (`git commit -- <path>`) so unrelated staged/dirty edits are
|
||||
* never swept into the commit. Never throws; returns false on any failure
|
||||
* (index.lock contention, nothing changed, detached states) — the DB row and
|
||||
* the on-disk file remain the durable sinks either way.
|
||||
*/
|
||||
export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean {
|
||||
try {
|
||||
const rel = relative(repoPath, absPath);
|
||||
if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false;
|
||||
const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const;
|
||||
execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts);
|
||||
execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Committed helper ────────────────────────────────────────────────────────
|
||||
|
||||
function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } {
|
||||
|
||||
+33
-26
@@ -409,6 +409,11 @@ export interface ScanOpts {
|
||||
visitDir?: (dirPath: string) => void;
|
||||
}
|
||||
|
||||
/** Timeout-arm winner for the COUNT-vs-deadline race in scanBrainSources.
|
||||
* A unique object so it can never collide with a legitimate COUNT result
|
||||
* (number | null). Module-private. */
|
||||
const DEADLINE_SENTINEL: unique symbol = Symbol('gbrain.scan.deadline');
|
||||
|
||||
export async function scanBrainSources(
|
||||
engine: BrainEngine,
|
||||
opts: ScanOpts = {},
|
||||
@@ -480,41 +485,43 @@ export async function scanBrainSources(
|
||||
// pool can make this await hang past the budget. Without the race, we'd
|
||||
// wait indefinitely AND defeat the wall-clock guarantee.
|
||||
let dbPageCount: number | null = null;
|
||||
// Set when the deadline race's timeout arm wins: the verdict that the
|
||||
// budget is spent, independent of any later Date.now() reading. Timer
|
||||
// callbacks on loaded runners can fire measurably EARLY relative to the
|
||||
// wall clock (a +1ms pad was drifted past in practice — see the flake
|
||||
// lineage in test/brain-writer-partial-scan.test.ts and issue #2946), so
|
||||
// the hung-COUNT path must not re-derive "did the deadline fire?" from
|
||||
// the clock the timer just raced against.
|
||||
let deadlineHit = false;
|
||||
if (opts.dbPageCountForSource) {
|
||||
try {
|
||||
if (opts.deadline) {
|
||||
const remainingMs = opts.deadline - Date.now();
|
||||
if (remainingMs <= 0) {
|
||||
dbPageCount = null;
|
||||
deadlineHit = true;
|
||||
} else {
|
||||
// Race COUNT against the deadline so a hung query can't eat the budget.
|
||||
//
|
||||
// Boundary overshoot (+1ms): the post-await deadline check at line
|
||||
// ~512 uses `Date.now() >= deadline`. setTimeout fires AT OR AFTER
|
||||
// the requested delay, so in theory the check always passes. In
|
||||
// practice on heavily-loaded CI runners (8 parallel shards × 4
|
||||
// concurrent test files = ~32 concurrent bun processes) we saw
|
||||
// intermittent failures where the timer callback resolved
|
||||
// microseconds BEFORE the wall-clock boundary, leaving Date.now()
|
||||
// a tick below deadline and the skip-check evaluating false. The
|
||||
// src-a scan then ran on a populated dir before src-b's
|
||||
// between-source check caught up — causing
|
||||
// `firstSource.status === 'skipped'` to receive 'scanned'.
|
||||
//
|
||||
// Adding 1ms guarantees the timer fires past the deadline by at
|
||||
// least one millisecond regardless of runner timer drift. Cost is
|
||||
// 1ms additional wall-clock latency on hung COUNT queries, which
|
||||
// is operationally negligible. Flake repro:
|
||||
// https://github.com/garrytan/gbrain/actions/runs/77611667786
|
||||
dbPageCount = await Promise.race([
|
||||
// Race COUNT against the deadline so a hung query can't eat the
|
||||
// budget. The timeout arm resolves a private sentinel — NOT null —
|
||||
// so a deadline win is distinguishable from a COUNT that resolved
|
||||
// null (failed/absent count keeps its existing semantics).
|
||||
const raced = await Promise.race([
|
||||
opts.dbPageCountForSource(src.id),
|
||||
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs + 1)),
|
||||
new Promise<typeof DEADLINE_SENTINEL>(resolve =>
|
||||
setTimeout(() => resolve(DEADLINE_SENTINEL), remainingMs)),
|
||||
]);
|
||||
if (raced === DEADLINE_SENTINEL) {
|
||||
dbPageCount = null;
|
||||
deadlineHit = true;
|
||||
} else {
|
||||
dbPageCount = raced;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
dbPageCount = await opts.dbPageCountForSource(src.id);
|
||||
}
|
||||
} catch {
|
||||
// A throwing COUNT is a failed count, not a deadline verdict.
|
||||
dbPageCount = null;
|
||||
}
|
||||
}
|
||||
@@ -524,11 +531,11 @@ export async function scanBrainSources(
|
||||
// status='partial' with files_scanned=0, which is misleading ("partial
|
||||
// scan" when actually nothing was scanned). Mark this source + remainder
|
||||
// as 'skipped' so the doctor message is honest.
|
||||
// `>=` matches the between-source check above (line 445). The Promise.race
|
||||
// setTimeout resolves null at exactly `remainingMs` from now, so post-await
|
||||
// Date.now() often equals deadline within integer-ms precision — strict `>`
|
||||
// missed those landings on CI and let the next scanOneSource run anyway.
|
||||
if (opts.signal?.aborted || (opts.deadline && Date.now() >= opts.deadline)) {
|
||||
// `deadlineHit` is the authoritative verdict for the hung-COUNT path (the
|
||||
// sentinel above); the wall-clock re-check (`>=`, matching the
|
||||
// between-source check at line ~445) still covers a COUNT that RESOLVED
|
||||
// slowly enough to eat the budget without the timer winning.
|
||||
if (opts.signal?.aborted || deadlineHit || (opts.deadline && Date.now() >= opts.deadline)) {
|
||||
if (abortedAtSource === null) {
|
||||
abortedAtSource = src.id;
|
||||
}
|
||||
|
||||
@@ -26,7 +26,17 @@ export interface ChronicleJudgeInput {
|
||||
effectiveDate: string | null; // depth page effective_date (deterministic when)
|
||||
attendees: string[]; // deterministic who from frontmatter
|
||||
}
|
||||
export interface ChronicleJudgeResult { events: ChronicleEventProposal[] }
|
||||
export interface ChronicleJudgeResult {
|
||||
events: ChronicleEventProposal[];
|
||||
/**
|
||||
* #2606 — distinct judge-failure signal so an unusable response is never
|
||||
* recorded as a legitimate `no_events`:
|
||||
* - 'truncated': the model hit the output-token cap (stopReason 'length');
|
||||
* the JSON array was cut mid-stream and must not be parsed as complete.
|
||||
* - 'parse_failed': the model returned text but no valid JSON array.
|
||||
*/
|
||||
failure?: 'truncated' | 'parse_failed';
|
||||
}
|
||||
export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise<ChronicleJudgeResult>;
|
||||
|
||||
export interface ChronicleExtractResult {
|
||||
@@ -126,6 +136,12 @@ export async function runChronicleExtract(
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' };
|
||||
}
|
||||
|
||||
// #2606: a truncated or unparseable judge response is a FAILURE, not an
|
||||
// empty page. Record it as a distinct skipped reason so operators (and
|
||||
// retries) can tell it apart from a genuine no_events.
|
||||
if (result?.failure) {
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` };
|
||||
}
|
||||
const proposals = Array.isArray(result?.events) ? result.events : [];
|
||||
if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 };
|
||||
// PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes.
|
||||
@@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli
|
||||
Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}.
|
||||
Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown — just the JSON array.`;
|
||||
|
||||
/**
|
||||
* #2606: default output-token cap for the judge. Raised from the original
|
||||
* 1500 (which event-dense pages overflowed, silently truncating the JSON
|
||||
* array). Override via `chronicle.judge_max_tokens`.
|
||||
*/
|
||||
const DEFAULT_JUDGE_MAX_TOKENS = 4000;
|
||||
|
||||
function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
return async (input) => {
|
||||
const { isAvailable, chat } = await import('../ai/gateway.ts');
|
||||
if (!isAvailable('chat')) return { events: [] };
|
||||
const body = (input.body || '').slice(0, 12_000);
|
||||
// #2606: configurable cap so event-dense pages have headroom.
|
||||
let maxTokens = DEFAULT_JUDGE_MAX_TOKENS;
|
||||
const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null);
|
||||
if (capRaw) {
|
||||
const n = parseInt(capRaw, 10);
|
||||
if (Number.isFinite(n) && n > 0) maxTokens = n;
|
||||
}
|
||||
let text: string;
|
||||
try {
|
||||
const res = await chat({
|
||||
@@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
`${input.title}\n\n${body}\n</page>\n\n` +
|
||||
`Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`,
|
||||
}],
|
||||
maxTokens: 1500,
|
||||
maxTokens,
|
||||
});
|
||||
if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] };
|
||||
// #2606: output hit the token cap — the JSON array is cut mid-stream.
|
||||
// Do NOT feed it to the parser as if complete; surface the truncation.
|
||||
if (res.stopReason === 'length') return { events: [], failure: 'truncated' };
|
||||
text = res.text;
|
||||
} catch (err) {
|
||||
if ((err as Error)?.name === 'AbortError') throw err;
|
||||
return { events: [] };
|
||||
}
|
||||
const parsed = parseJudgeJson(text);
|
||||
// #2606: non-empty model text with no parseable JSON array is a parse
|
||||
// failure, distinct from the model legitimately answering `[]`.
|
||||
if (parsed === null) return { events: [], failure: 'parse_failed' };
|
||||
return { events: parsed };
|
||||
};
|
||||
}
|
||||
|
||||
/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] {
|
||||
if (!text) return [];
|
||||
/**
|
||||
* Tolerant JSON-array extraction from a model response (mirrors facts parser).
|
||||
*
|
||||
* #2606: returns `null` on parse FAILURE (empty text, no `[...]` found,
|
||||
* JSON.parse throw, non-array result) so callers can distinguish "the model
|
||||
* said no events" (a legitimate `[]`) from "the response was unusable".
|
||||
*/
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] | null {
|
||||
if (!text) return null;
|
||||
let s = text.trim();
|
||||
const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i);
|
||||
if (fence) s = fence[1].trim();
|
||||
const start = s.indexOf('[');
|
||||
const end = s.lastIndexOf(']');
|
||||
if (start === -1 || end === -1 || end < start) return [];
|
||||
if (start === -1 || end === -1 || end < start) return null;
|
||||
try {
|
||||
const arr = JSON.parse(s.slice(start, end + 1));
|
||||
return Array.isArray(arr) ? arr : [];
|
||||
return Array.isArray(arr) ? arr : null;
|
||||
} catch {
|
||||
return [];
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -114,6 +114,26 @@ function resolveFlushGraceMs(): number {
|
||||
/** Default per-sink drain budget (matches drainAllBackgroundWorkForCliExit). */
|
||||
const DEFAULT_DRAIN_TIMEOUT_MS = 2_000;
|
||||
|
||||
/**
|
||||
* Resolve the per-sink drain budget: `GBRAIN_DRAIN_TIMEOUT_MS` env override
|
||||
* (slow-provider escape hatch, same env-only pattern as
|
||||
* GBRAIN_TEARDOWN_DEADLINE_MS) over the 2000ms default. An explicit
|
||||
* `drainTimeoutMs` from a call site still wins — the env replaces only the
|
||||
* DEFAULT. The 2s default assumes a sub-second cloud chat provider; a
|
||||
* self-hosted model (e.g. ollama at 10-20s per completion) can never finish a
|
||||
* fire-and-forget facts:absorb extraction inside it, so every one-shot CLI
|
||||
* exit — sync timers especially — aborts the in-flight chat and the
|
||||
* extraction never lands, retrying (and re-aborting) on each subsequent sync
|
||||
* of the same page. Raising the budget via env lets those installs drain
|
||||
* instead of abort; computeTeardownDeadlineMs already scales the backstop
|
||||
* from the resolved value, so the deadline widens with it.
|
||||
*/
|
||||
export function resolveDrainTimeoutMs(): number {
|
||||
const env = Number(process.env.GBRAIN_DRAIN_TIMEOUT_MS);
|
||||
if (Number.isFinite(env) && env > 0) return env;
|
||||
return DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backstop deadline for drain + disconnect COMBINED, computed from the bounds
|
||||
* it guards so it fires only when a component violated its own bound (#2084
|
||||
@@ -262,7 +282,10 @@ export function flushThenExit(code: number, opts: FlushThenExitOpts = {}): void
|
||||
export interface FinishCliTeardownOpts {
|
||||
/** Engine to disconnect. A disconnect throw is warned + swallowed (D3). */
|
||||
engine: { disconnect(): Promise<void> };
|
||||
/** Per-sink drain budget. Default 2000 (the registry default). */
|
||||
/**
|
||||
* Per-sink drain budget. Default: `GBRAIN_DRAIN_TIMEOUT_MS` env override,
|
||||
* else 2000 (the registry default).
|
||||
*/
|
||||
drainTimeoutMs?: number;
|
||||
/** Test seam — wins over the env override and the computed formula. */
|
||||
deadlineMs?: number;
|
||||
@@ -284,7 +307,7 @@ export interface FinishCliTeardownOpts {
|
||||
* exit in here, and it means a component violated its own bound.
|
||||
*/
|
||||
export async function finishCliTeardown(opts: FinishCliTeardownOpts): Promise<void> {
|
||||
const drainTimeoutMs = opts.drainTimeoutMs ?? DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
const drainTimeoutMs = opts.drainTimeoutMs ?? resolveDrainTimeoutMs();
|
||||
const warn = opts.warn ?? ((m: string) => console.warn(m));
|
||||
const drain = opts.drain ?? drainAllBackgroundWorkForCliExit;
|
||||
const deadlineMs =
|
||||
|
||||
@@ -915,6 +915,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'dream.synthesize.verdict_model',
|
||||
'dream.synthesize.max_prompt_tokens',
|
||||
'dream.synthesize.max_chunks_per_transcript',
|
||||
// #2415: top-level namespace for synthesize/patterns output (default 'wiki').
|
||||
'dream.synthesize.output_root',
|
||||
'dream.synthesize.subagent_timeout_ms',
|
||||
'dream.synthesize.subagent_wait_timeout_ms',
|
||||
'dream.patterns.lookback_days',
|
||||
@@ -971,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// operator had to discover --force by reading source. Same class as the
|
||||
// spend-controls registration above.
|
||||
'auto_chronicle',
|
||||
// #2606: chronicle judge output-token cap (default 4000). Event-dense
|
||||
// pages overflowed the old hardcoded 1500 and were misrecorded as
|
||||
// no_events; the cap is now configurable and truncation is surfaced.
|
||||
'chronicle.judge_max_tokens',
|
||||
// Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate
|
||||
// consent reads this key, and enabling it is the documented path to
|
||||
// `gbrain takes extract --from-pages` — same unregistered-key class.
|
||||
|
||||
@@ -479,6 +479,16 @@ export interface CycleOpts {
|
||||
* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).
|
||||
*/
|
||||
sourceId?: string;
|
||||
/**
|
||||
* Absolute wall-clock deadline (epoch ms) of the enclosing minion job,
|
||||
* from `MinionJobContext.deadlineAtMs` (the claim-time `timeout_at`
|
||||
* stamp). Phases that spawn bounded sub-work (patterns' subagent) clamp
|
||||
* their own timeouts to the REMAINING time so one phase's fixed
|
||||
* worst-case can't blow past the job budget and dead-letter the whole
|
||||
* cycle mid-phase (#2781). Unset for direct callers (`gbrain dream`) —
|
||||
* phases then use their configured timeouts unchanged.
|
||||
*/
|
||||
deadlineAtMs?: number | null;
|
||||
}
|
||||
|
||||
// ─── Lock primitives ───────────────────────────────────────────────
|
||||
@@ -1682,6 +1692,9 @@ export async function runCycle(
|
||||
from: opts.synthFrom,
|
||||
to: opts.synthTo,
|
||||
bypassDreamGuard: opts.synthBypassDreamGuard,
|
||||
// #1586: scope synthesized writes to the cycle's resolved source
|
||||
// (explicit --source wins, else derived from the checkout dir).
|
||||
sourceId: cycleSourceId,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -1885,6 +1898,7 @@ export async function runCycle(
|
||||
brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
deadlineAtMs: opts.deadlineAtMs ?? null,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
|
||||
+97
-35
@@ -19,7 +19,7 @@
|
||||
*/
|
||||
|
||||
import { join, dirname } from 'node:path';
|
||||
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
@@ -27,6 +27,9 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.
|
||||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
// #2415: allow-list + output-root resolution shared with the synthesize
|
||||
// phase — both phases must agree on the configured namespace.
|
||||
import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts';
|
||||
import { probeChatModel } from '../ai/gateway.ts';
|
||||
import { normalizeModelId } from '../model-id.ts';
|
||||
|
||||
@@ -34,6 +37,57 @@ export interface PatternsPhaseOpts {
|
||||
brainDir: string;
|
||||
dryRun: boolean;
|
||||
yieldDuringPhase?: () => Promise<void>;
|
||||
/**
|
||||
* Absolute deadline (epoch ms) of the enclosing minion job, or null for
|
||||
* direct callers (`gbrain dream`). When set, the subagent's job timeout
|
||||
* and the wait timeout are clamped so the phase finishes (or times out)
|
||||
* BEFORE the parent job's budget expires — a fixed 30/35-min default
|
||||
* inside an interval-derived cycle budget dead-letters the whole cycle
|
||||
* mid-phase and starves every tail phase (#2781).
|
||||
*/
|
||||
deadlineAtMs?: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop-margin reserved under the parent deadline when clamping subagent
|
||||
* budgets. NOT a promise that tail phases complete — the cycle is allowed
|
||||
* to go partial and resume next tick. This only guarantees the phase's
|
||||
* wait returns and the handler unwinds cleanly before the worker's abort
|
||||
* fires: wait poll interval (5s) + worker force-evict grace (30s) + lock
|
||||
* and DB cleanup headroom.
|
||||
*/
|
||||
export const CYCLE_DEADLINE_RESERVE_MS = 60 * 1000;
|
||||
|
||||
/**
|
||||
* Smallest remaining budget worth submitting a subagent for. Below this,
|
||||
* the LLM call is near-certain to be killed mid-flight — wasted spend and
|
||||
* a guaranteed-timeout child — so the phase skips honestly instead
|
||||
* (`insufficient_cycle_budget`) and the next cycle retries with a fresh
|
||||
* budget.
|
||||
*/
|
||||
export const MIN_PATTERNS_SUBAGENT_BUDGET_MS = 2 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Clamp the configured subagent budgets to the remaining parent-job time.
|
||||
* Both timeouts derive from the SAME absolute child deadline
|
||||
* (`deadlineAtMs - reserve`) so the child job's kill switch and our wait
|
||||
* agree. Returns null when the remaining budget is below the minimum —
|
||||
* caller should skip the phase without submitting.
|
||||
*/
|
||||
export function clampSubagentBudgets(
|
||||
config: { subagentTimeoutMs: number; subagentWaitTimeoutMs: number },
|
||||
deadlineAtMs: number | null | undefined,
|
||||
nowMs: number,
|
||||
): { timeoutMs: number; waitTimeoutMs: number } | null {
|
||||
if (deadlineAtMs == null) {
|
||||
return { timeoutMs: config.subagentTimeoutMs, waitTimeoutMs: config.subagentWaitTimeoutMs };
|
||||
}
|
||||
const childBudgetMs = deadlineAtMs - CYCLE_DEADLINE_RESERVE_MS - nowMs;
|
||||
if (childBudgetMs < MIN_PATTERNS_SUBAGENT_BUDGET_MS) return null;
|
||||
return {
|
||||
timeoutMs: Math.min(config.subagentTimeoutMs, childBudgetMs),
|
||||
waitTimeoutMs: Math.min(config.subagentWaitTimeoutMs, childBudgetMs),
|
||||
};
|
||||
}
|
||||
|
||||
export async function runPhasePatterns(
|
||||
@@ -49,7 +103,7 @@ export async function runPhasePatterns(
|
||||
}
|
||||
|
||||
// Gather reflections within lookback window.
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays);
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot);
|
||||
if (reflections.length < config.minEvidence) {
|
||||
return skipped(
|
||||
'insufficient_evidence',
|
||||
@@ -81,22 +135,35 @@ export async function runPhasePatterns(
|
||||
return skipped('no_provider', `pattern detection skipped: ${probe.detail}`);
|
||||
}
|
||||
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
}
|
||||
|
||||
// #2781: budget the subagent from the REMAINING parent-job time, not
|
||||
// the fixed config default. Checked after the cheap gates (disabled /
|
||||
// insufficient_evidence / no_provider) so a skip for budget reasons
|
||||
// only fires when the phase would otherwise have submitted.
|
||||
const budgets = clampSubagentBudgets(config, opts.deadlineAtMs, Date.now());
|
||||
if (budgets === null) {
|
||||
return skipped(
|
||||
'insufficient_cycle_budget',
|
||||
`remaining cycle budget under ${Math.round(MIN_PATTERNS_SUBAGENT_BUDGET_MS / 1000)}s ` +
|
||||
`(reserve ${Math.round(CYCLE_DEADLINE_RESERVE_MS / 1000)}s); next cycle retries with a fresh budget`,
|
||||
);
|
||||
}
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence),
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
timeout_ms: config.subagentTimeoutMs,
|
||||
timeout_ms: budgets.timeoutMs,
|
||||
};
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
@@ -105,13 +172,23 @@ export async function runPhasePatterns(
|
||||
let outcome: string;
|
||||
try {
|
||||
const final = await waitForCompletion(queue, job.id, {
|
||||
timeoutMs: config.subagentWaitTimeoutMs,
|
||||
timeoutMs: budgets.waitTimeoutMs,
|
||||
pollMs: 5 * 1000,
|
||||
});
|
||||
outcome = final.status;
|
||||
} catch (e) {
|
||||
if (e instanceof TimeoutError) outcome = 'timeout';
|
||||
else throw e;
|
||||
if (e instanceof TimeoutError) {
|
||||
outcome = 'timeout';
|
||||
// The child's own timeout_ms clock starts at ITS claim, not at
|
||||
// submit — a child that sat queued behind other work can outlive
|
||||
// the parent deadline this wait was clamped to. Cancel it so the
|
||||
// subagent can't keep spending/writing after the phase gave up
|
||||
// (waiting child → cancelled immediately; active child → lock
|
||||
// stripped, worker abort fires on next renew tick).
|
||||
try { await queue.cancelJob(job.id); } catch { /* best-effort */ }
|
||||
} else {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.yieldDuringPhase) {
|
||||
@@ -182,6 +259,8 @@ interface PatternsConfig {
|
||||
lookbackDays: number;
|
||||
minEvidence: number;
|
||||
model: string;
|
||||
/** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */
|
||||
outputRoot: string;
|
||||
/** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */
|
||||
subagentTimeoutMs: number;
|
||||
/** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */
|
||||
@@ -216,6 +295,7 @@ async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig>
|
||||
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
|
||||
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
|
||||
model,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs: await getNumberConfig(
|
||||
engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS,
|
||||
),
|
||||
@@ -236,16 +316,19 @@ interface ReflectionRef {
|
||||
async function gatherReflections(
|
||||
engine: BrainEngine,
|
||||
lookbackDays: number,
|
||||
outputRoot = 'wiki',
|
||||
): Promise<ReflectionRef[]> {
|
||||
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
|
||||
// #2415: reflections live under the configured output root (bound as a
|
||||
// parameter; outputRoot is slug-grammar-validated by loadOutputRoot).
|
||||
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
|
||||
`SELECT slug, title, compiled_truth
|
||||
FROM pages
|
||||
WHERE slug LIKE 'wiki/personal/reflections/%'
|
||||
WHERE slug LIKE $2
|
||||
AND updated_at >= $1::timestamptz
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT 100`,
|
||||
[since],
|
||||
[since, `${outputRoot}/personal/reflections/%`],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
slug: r.slug,
|
||||
@@ -256,7 +339,7 @@ async function gatherReflections(
|
||||
|
||||
// ── Prompt ────────────────────────────────────────────────────────────
|
||||
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string {
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const corpus = reflections
|
||||
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
|
||||
@@ -266,15 +349,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number):
|
||||
|
||||
OUTPUT POLICY
|
||||
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks).
|
||||
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
|
||||
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- Pattern slug format: \`${outputRoot}/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
|
||||
|
||||
DO NOT WRITE
|
||||
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
|
||||
- Patterns with <${minEvidence} reflections cited.
|
||||
- Anything outside wiki/personal/patterns/.
|
||||
- Anything outside ${outputRoot}/personal/patterns/.
|
||||
|
||||
CONTEXT
|
||||
- Today: ${today}
|
||||
@@ -365,27 +448,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||||
);
|
||||
}
|
||||
|
||||
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const candidates = [
|
||||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||||
];
|
||||
for (const path of candidates) {
|
||||
if (!existsSync(path)) continue;
|
||||
try {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// ── Status helpers ───────────────────────────────────────────────────
|
||||
|
||||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||||
|
||||
@@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts';
|
||||
import type { ProgressReporter } from '../progress.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts';
|
||||
// #2163: concept pages route through importFromContent (the same
|
||||
// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage,
|
||||
// so they land in the retrieval surface (content_chunks + embeddings) where
|
||||
// source-boost's 1.3× 'concepts/' weighting can actually reach them.
|
||||
import { importFromContent } from '../import-file.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
|
||||
const DEFAULT_BUDGET_USD = 1.5;
|
||||
const TIER_T1_MIN = 10;
|
||||
@@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts(
|
||||
|
||||
if (!opts.dryRun) {
|
||||
const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug;
|
||||
await engine.putPage(`concepts/${title}`, {
|
||||
title: title.replace(/-/g, ' '),
|
||||
type: 'concept',
|
||||
compiled_truth: narrative,
|
||||
frontmatter: {
|
||||
type: 'concept',
|
||||
// #2163: serialize to markdown and import via the canonical pipeline so
|
||||
// the page is chunked (+ embedded when a provider is configured) —
|
||||
// mirrors put_page's isAvailable('embedding') → noEmbed gate.
|
||||
const md = serializeMarkdown(
|
||||
{
|
||||
tier: group.tier,
|
||||
mention_count: group.atomTitles.length,
|
||||
composite_score: group.atomTitles.length,
|
||||
synthesized_at: new Date().toISOString(),
|
||||
synthesized_by: 'synthesize_concepts-v0.41',
|
||||
},
|
||||
timeline: '',
|
||||
narrative,
|
||||
'',
|
||||
{ type: 'concept', title: title.replace(/-/g, ' '), tags: [] },
|
||||
);
|
||||
await importFromContent(engine, `concepts/${title}`, md, {
|
||||
noEmbed: !isAvailable('embedding'),
|
||||
});
|
||||
}
|
||||
conceptsWritten++;
|
||||
|
||||
+119
-21
@@ -244,6 +244,14 @@ export interface SynthesizePhaseOpts {
|
||||
* the synthesize loop. Caller must opt in explicitly.
|
||||
*/
|
||||
bypassDreamGuard?: boolean;
|
||||
/**
|
||||
* #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts —
|
||||
* explicit --source wins, else derived from the checkout dir). Threaded to
|
||||
* every subagent child as `source_id` so put_page writes land in this
|
||||
* source, and stamped onto collected refs so reverse-writes read the
|
||||
* correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
export async function runPhaseSynthesize(
|
||||
@@ -399,7 +407,7 @@ export async function runPhaseSynthesize(
|
||||
|
||||
// Fan-out: submit one subagent per worth-processing transcript (or one
|
||||
// per chunk for transcripts that exceed the model's per-prompt budget).
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
@@ -462,10 +470,13 @@ export async function runPhaseSynthesize(
|
||||
: config.model;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot),
|
||||
model: subagentModel,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
// #1586: scope every child tool call to the cycle's resolved source
|
||||
// so put_page writes land there instead of the hardcoded 'default'.
|
||||
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
|
||||
};
|
||||
// Idempotency key parity:
|
||||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||||
@@ -524,20 +535,29 @@ export async function runPhaseSynthesize(
|
||||
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
|
||||
// even if Sonnet drops the chunk suffix.
|
||||
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
|
||||
// (source, slug) row (currently always 'default' from subagent put_page).
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
|
||||
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
|
||||
// source (children write there via SubagentHandlerData.source_id).
|
||||
const cycleSourceId = opts.sourceId ?? 'default';
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
|
||||
|
||||
const summaryDate = opts.date ?? today();
|
||||
|
||||
// #2569: persist the dream-output identity marker into the DB frontmatter
|
||||
// of every child-written page BEFORE reverse-rendering, so generated pages
|
||||
// are queryable (`frontmatter->>'dream_generated'`) and a later put_page
|
||||
// write-through (which re-renders from the DB row) can't erase the stamp.
|
||||
await stampDreamProvenance(engine, writtenRefs, summaryDate);
|
||||
|
||||
// Dual-write: reverse-render each DB row → markdown file.
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId);
|
||||
|
||||
// Summary index page (deterministic; orchestrator-written via direct
|
||||
// engine.putPage so no allow-list path needed).
|
||||
const summaryDate = opts.date ?? today();
|
||||
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
|
||||
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
|
||||
const writtenSlugs = writtenRefs.map(r => r.slug);
|
||||
if (SUMMARY_SLUG_RE.test(summarySlug)) {
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId);
|
||||
}
|
||||
|
||||
// Write completion timestamp ON SUCCESS only.
|
||||
@@ -595,10 +615,29 @@ interface SynthConfig {
|
||||
* `dream.synthesize.max_chunks_per_transcript`.
|
||||
*/
|
||||
maxChunksPerTranscript: number;
|
||||
/**
|
||||
* #2415: top-level namespace for synthesized output (reflections, originals,
|
||||
* patterns). Config key `dream.synthesize.output_root`; default 'wiki' —
|
||||
* zero behavior change unless set. No trailing slash. Must satisfy the slug
|
||||
* grammar; invalid values fall back to 'wiki' with a stderr warning.
|
||||
*/
|
||||
outputRoot: string;
|
||||
subagentTimeoutMs: number;
|
||||
subagentWaitTimeoutMs: number;
|
||||
}
|
||||
|
||||
/** #2415: shared output-root resolution (synthesize + patterns phases). */
|
||||
export async function loadOutputRoot(engine: BrainEngine): Promise<string> {
|
||||
const raw = await engine.getConfig('dream.synthesize.output_root');
|
||||
if (!raw) return 'wiki';
|
||||
const trimmed = raw.trim().replace(/^\/+|\/+$/g, '');
|
||||
if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed;
|
||||
process.stderr.write(
|
||||
`[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`,
|
||||
);
|
||||
return 'wiki';
|
||||
}
|
||||
|
||||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
const enabledRaw = await engine.getConfig('dream.synthesize.enabled');
|
||||
const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
|
||||
@@ -672,6 +711,7 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||||
maxPromptTokens,
|
||||
maxChunksPerTranscript,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs,
|
||||
subagentWaitTimeoutMs,
|
||||
};
|
||||
@@ -704,7 +744,13 @@ async function checkCooldown(
|
||||
|
||||
// ── Allow-list source of truth ───────────────────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
/**
|
||||
* #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the
|
||||
* configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki'
|
||||
* returns the globs verbatim. Shared by the patterns phase (imported there —
|
||||
* the two phases must enforce the same allow-list).
|
||||
*/
|
||||
export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise<string[]> {
|
||||
// Search a few known locations relative to the binary / repo. The first
|
||||
// hit wins; if none found, return [].
|
||||
const candidates = [
|
||||
@@ -718,7 +764,10 @@ async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
if (outputRoot === 'wiki') return globs as string[];
|
||||
return (globs as string[]).map(g =>
|
||||
g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g,
|
||||
);
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
@@ -966,6 +1015,7 @@ function buildSynthesisPrompt(
|
||||
chunkIdx: number,
|
||||
chunkTotal: number,
|
||||
priorContradictionsBlock = '',
|
||||
outputRoot = 'wiki',
|
||||
): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
@@ -994,10 +1044,10 @@ OUTPUT POLICY (ALL of these are required)
|
||||
|
||||
TASKS
|
||||
A. Reflections (self-knowledge, pattern recognition, emotional processing):
|
||||
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
|
||||
B. Originals (new ideas, frames, theses, mental models):
|
||||
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
|
||||
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages).
|
||||
|
||||
@@ -1038,6 +1088,7 @@ async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
chunkInfo: Map<number, { idx: number; hash6: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<Array<{ slug: string; source_id: string }>> {
|
||||
if (childIds.length === 0) return [];
|
||||
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
|
||||
@@ -1047,10 +1098,10 @@ async function collectChildPutPageSlugs(
|
||||
//
|
||||
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
|
||||
// put_page tool schema doesn't expose source_id (subagents are scoped to
|
||||
// a single source); default to 'default' for the current dream-cycle
|
||||
// product behavior. Threading the source_id through reverseWriteRefs
|
||||
// guarantees getPage targets the correct (source, slug) row instead of
|
||||
// the first DB match.
|
||||
// a single source). #1586: the orchestrator scopes each child to the
|
||||
// cycle's resolved source via SubagentHandlerData.source_id, and stamps
|
||||
// the SAME source here so reverseWriteRefs / provenance reads target the
|
||||
// correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||||
`SELECT job_id,
|
||||
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
|
||||
@@ -1066,7 +1117,7 @@ async function collectChildPutPageSlugs(
|
||||
const ci = chunkInfo.get(r.job_id);
|
||||
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
|
||||
}
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1095,12 +1146,52 @@ async function hasLegacySingleChunkCompletion(
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
// ── Dream-provenance DB stamp (#2569) ────────────────────────────────
|
||||
|
||||
/**
|
||||
* Persist the dream-output identity marker (`dream_generated: true` +
|
||||
* `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page
|
||||
* a synthesize child wrote. Render-time `frontmatterOverrides` alone only
|
||||
* reach the markdown FILE — the DB row stayed unstamped, so DB consumers
|
||||
* couldn't enumerate generated pages and a later put_page write-through
|
||||
* (which re-renders from the DB row) silently erased the marker.
|
||||
*
|
||||
* Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb —
|
||||
* never JSON.stringify into a ::jsonb cast; engine-parity safe, no new
|
||||
* engine method). Best-effort per row: a stamp failure never kills the
|
||||
* phase (the render-time override still covers the file).
|
||||
*/
|
||||
async function stampDreamProvenance(
|
||||
engine: BrainEngine,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
cycleDate: string,
|
||||
): Promise<void> {
|
||||
if (refs.length === 0) return;
|
||||
const { executeRawJsonb } = await import('../sql-query.ts');
|
||||
for (const { slug, source_id } of refs) {
|
||||
try {
|
||||
await executeRawJsonb(
|
||||
engine,
|
||||
`UPDATE pages
|
||||
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
|
||||
WHERE slug = $1 AND source_id = $2`,
|
||||
[slug, source_id],
|
||||
[{ dream_generated: true, dream_cycle_date: cycleDate }],
|
||||
);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||||
|
||||
async function reverseWriteRefs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
nativeSourceId = 'default',
|
||||
): Promise<number> {
|
||||
let count = 0;
|
||||
for (const { slug, source_id } of refs) {
|
||||
@@ -1111,10 +1202,11 @@ async function reverseWriteRefs(
|
||||
const tags = await engine.getTags(slug, { sourceId: source_id });
|
||||
try {
|
||||
const md = renderPageToMarkdown(page, tags);
|
||||
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Default-source
|
||||
// pages stay at brainDir/<slug>.md so single-source brains see no change.
|
||||
const filePath = source_id === 'default'
|
||||
// v0.32.8 F6: foreign-source pages land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Pages belonging to
|
||||
// the cycle's own source (#1586: brainDir IS that source's checkout —
|
||||
// legacy 'default' when unscoped) stay at brainDir/<slug>.md.
|
||||
const filePath = source_id === nativeSourceId
|
||||
? join(brainDir, `${slug}.md`)
|
||||
: join(brainDir, '.sources', source_id, `${slug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
@@ -1161,6 +1253,7 @@ async function writeSummaryPage(
|
||||
summaryDate: string,
|
||||
writtenSlugs: string[],
|
||||
childOutcomes: Array<{ jobId: number; status: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<void> {
|
||||
const completed = childOutcomes.filter(c => c.status === 'completed').length;
|
||||
const failed = childOutcomes.length - completed;
|
||||
@@ -1198,13 +1291,15 @@ async function writeSummaryPage(
|
||||
// unnecessarily; we go straight to the engine.
|
||||
const { parseMarkdown } = await import('../markdown.ts');
|
||||
const parsed = parseMarkdown(fullMarkdown);
|
||||
// #1586: summary lands in the cycle's resolved source too — otherwise the
|
||||
// children live in the named source while the index drifts to 'default'.
|
||||
await engine.putPage(summarySlug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
}, { sourceId });
|
||||
|
||||
// Also write to disk (orchestrator dual-write).
|
||||
try {
|
||||
@@ -1269,4 +1364,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P
|
||||
// double-encoded jsonb regression). Not part of the runtime contract.
|
||||
export const __testing = {
|
||||
collectChildPutPageSlugs,
|
||||
buildSynthesisPrompt,
|
||||
stampDreamProvenance,
|
||||
reverseWriteRefs,
|
||||
};
|
||||
|
||||
@@ -37,6 +37,9 @@ export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
|
||||
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
|
||||
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
|
||||
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
|
||||
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-19)
|
||||
'mistral:mistral-embed': { pricePerMTok: 0.10 },
|
||||
'mistral:mistral-embed-2312': { pricePerMTok: 0.10 },
|
||||
};
|
||||
|
||||
export type PriceLookupResult =
|
||||
|
||||
@@ -936,6 +936,27 @@ export interface BrainEngine {
|
||||
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
/**
|
||||
* fix/title-retrieval-arm (D1): page-grain title candidate arm.
|
||||
*
|
||||
* content_chunks.search_vector never includes the page TITLE (it is
|
||||
* doc_comment + symbol_name_qualified + chunk_text), so a page whose
|
||||
* title tokens are absent from its body is unreachable by searchKeyword.
|
||||
* This arm queries the PAGE-GRAIN DOCUMENT vector pages.search_vector —
|
||||
* NOT titles alone: per trg_pages_search_vector it is title (weight 'A')
|
||||
* + compiled_truth ('B') + timeline text ('C'). Ranked by ts_rank_cd,
|
||||
* the 'A'-weighted title dominates, but body/timeline matches also
|
||||
* produce (lower-ranked) candidates. Returns page-grain hits joined to
|
||||
* ONE representative chunk per page (compiled_truth preferred, else
|
||||
* lowest chunk_index) so rows are shaped like searchKeyword's output and
|
||||
* can enter RRF fusion in hybridSearch.
|
||||
*
|
||||
* Deliberately NO query-length gating — unlike the alias hop (≤6-token
|
||||
* guard) and the title-phrase re-rank boost, this arm must GENERATE
|
||||
* candidates for long exact-title queries, which is exactly where
|
||||
* chunk-grain AND FTS is weakest.
|
||||
*/
|
||||
searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
/**
|
||||
* Hydrate embeddings for chunks already known by id. v0.36 (D9):
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { TakeBatchInput, TakeKind } from './engine.ts';
|
||||
import { chat, isAvailable } from './ai/gateway.ts';
|
||||
import { chat, getChatModel, isAvailable } from './ai/gateway.ts';
|
||||
|
||||
export const ALLOWED_PAGE_TYPES = [
|
||||
'concept', 'atom', 'lore', 'briefing', 'writing', 'originals',
|
||||
@@ -190,7 +190,11 @@ export async function extractTakesFromPages(
|
||||
let response: { text: string };
|
||||
try {
|
||||
response = await chat({
|
||||
model: opts.model ?? 'anthropic:claude-haiku-4-5',
|
||||
// #2997 — default to the configured chat model (file-plane gateway
|
||||
// config, same idiom as enrich.ts) instead of hardcoded cloud Haiku.
|
||||
// On OAuth/local-only installs the hardcoded model made every takes
|
||||
// extraction die with llm_unavailable despite a working chat_model.
|
||||
model: opts.model || getChatModel(),
|
||||
system: CLASSIFIER_SYSTEM,
|
||||
messages: [
|
||||
{
|
||||
|
||||
+78
-3
@@ -135,7 +135,10 @@ export const MIGRATIONS: Migration[] = [
|
||||
}
|
||||
}
|
||||
}
|
||||
if (renamed > 0) console.log(` Renamed ${renamed} slugs`);
|
||||
// Migration progress goes to stderr — stdout must stay clean for
|
||||
// callers parsing JSON (e.g. `gbrain doctor --json | jq`); migrations
|
||||
// can run lazily inside ANY command's first DB connect.
|
||||
if (renamed > 0) process.stderr.write(` Renamed ${renamed} slugs\n`);
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -5572,7 +5575,10 @@ export const MIGRATIONS: Migration[] = [
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
if (lang === 'english') {
|
||||
console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`);
|
||||
// stderr, NOT stdout: migrations run lazily inside any command's
|
||||
// first DB connect — a console.log here polluted `doctor --json`
|
||||
// stdout and broke jq consumers (heavy-tests fm_wallclock).
|
||||
process.stderr.write(` v123: trigger functions recreated with language='english' (default — no backfill needed)\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -5593,7 +5599,76 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`);
|
||||
process.stderr.write(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows\n`);
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 124,
|
||||
name: 'page_search_vector_drop_compiled_truth',
|
||||
// #2704: a single markdown page whose compiled_truth exceeds Postgres's
|
||||
// hard 1,048,575-byte tsvector cap made update_page_search_vector()
|
||||
// throw "string is too long for tsvector" INSIDE the pages UPSERT
|
||||
// transaction — not a per-file ledger entry, a transaction abort. The
|
||||
// whole source's sync checkpoint stayed pinned (Sync BLOCKED) until the
|
||||
// oversized file was fixed or manually skipped, even though every
|
||||
// OTHER file in the run imported fine.
|
||||
//
|
||||
// Fix: drop compiled_truth (the unbounded whole-page body) from this
|
||||
// trigger. It was already redundant — content_chunks.search_vector
|
||||
// (Cathedral II Layer 3, v0.20.0) is the ACTUAL keyword-search source:
|
||||
// searchKeyword() in postgres-engine.ts/pglite-engine.ts ranks and
|
||||
// queries `cc.search_vector` exclusively; `pages.search_vector` is
|
||||
// written by this trigger but never read by any query in this
|
||||
// codebase (verified: no `pages.search_vector`/bare `search_vector`
|
||||
// appears on either side of a WHERE/ts_rank anywhere outside this
|
||||
// trigger's own definition and the reindex/backfill machinery that
|
||||
// maintains it). And chunking already bounds each chunk_text well
|
||||
// under the tsvector limit (chunkText() targets embedding-sized
|
||||
// pieces, several orders of magnitude smaller than 1MB) — the overflow
|
||||
// was specific to the whole-page grain this trigger no longer builds.
|
||||
//
|
||||
// title + timeline (both naturally small — a compiled_truth-sized
|
||||
// title or timeline field would be its own bug) stay, so
|
||||
// pages.search_vector keeps carrying SOME signal rather than going
|
||||
// fully inert; a future PR can drop the column outright once its
|
||||
// last non-search consumer (if any turns up) is confirmed gone.
|
||||
//
|
||||
// No backfill: existing rows keep whatever search_vector they already
|
||||
// computed until their next UPDATE (harmless — nothing reads this
|
||||
// column, so staleness has zero behavioral effect). The brains that
|
||||
// actually hit this bug never successfully wrote a value for the
|
||||
// oversized page in the first place, so there's nothing stale to fix
|
||||
// for them specifically — the NEXT sync of that exact file is what
|
||||
// proves the fix, not a backfill of already-working rows.
|
||||
//
|
||||
// Function body mirrors reindex-search-vector.ts's recreatePagesFn
|
||||
// (documented contract there: keep both in lockstep) and the fresh-
|
||||
// install baselines in pglite-schema.ts / schema-embedded.ts — all
|
||||
// four updated in the same commit as this migration.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const lang = getFtsLanguage();
|
||||
await engine.executeRaw(`
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`);
|
||||
process.stderr.write(` v124: update_page_search_vector() no longer indexes compiled_truth (was overflowing tsvector on large pages, #2704)
|
||||
`);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
@@ -272,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
config,
|
||||
brainId: data.brain_id,
|
||||
allowedSlugPrefixes: data.allowed_slug_prefixes,
|
||||
// #1586: cycle-resolved source scope for tool-call OperationContexts.
|
||||
sourceId: data.source_id,
|
||||
});
|
||||
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
|
||||
? filterAllowedTools(registry, data.allowed_tools)
|
||||
|
||||
@@ -471,12 +471,34 @@ export class MinionQueue {
|
||||
});
|
||||
}
|
||||
|
||||
/** Re-queue a failed or dead job for retry. */
|
||||
/**
|
||||
* Re-queue a failed or dead job for retry.
|
||||
*
|
||||
* #2783: an explicit `jobs retry` is an operator asserting "run this
|
||||
* fresh" — so it clears `started_at` (re-stamped on re-claim via
|
||||
* `claim()`'s `COALESCE(started_at, now())`, `queue.ts:620`) and resets
|
||||
* `attempts_made`/`attempts_started` to 0. Without this, `started_at`
|
||||
* kept the ORIGINAL first-claim time, so `handleWallClockTimeouts()`
|
||||
* (anchored on `now() - started_at`, `queue.ts:729-749`) could measure
|
||||
* from long before the retry — a retry issued more than `timeout_ms * 2`
|
||||
* after the original claim was dead-lettered again in under a second,
|
||||
* with `attempts_made` already past `max_attempts`. This made retry
|
||||
* useless for exactly the case it exists for: recovering work after an
|
||||
* outage that outlasted the job's timeout.
|
||||
*
|
||||
* Also resets `stalled_counter` (Codex review): `handleStalled()`
|
||||
* dead-letters once `stalled_counter + 1 >= max_stalled` (`queue.ts:1190`).
|
||||
* A job dead-lettered BY stall exhaustion, left un-reset, would hit that
|
||||
* same threshold on its very first lock expiry after retry — a job
|
||||
* killed by 3 stalls doesn't get a fresh stall budget, contradicting
|
||||
* "run this fresh" the same way the unreset attempt counters did.
|
||||
*/
|
||||
async retryJob(id: number): Promise<MinionJob | null> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET status = 'waiting', error_text = NULL,
|
||||
lock_token = NULL, lock_until = NULL, delay_until = NULL,
|
||||
finished_at = NULL, updated_at = now()
|
||||
finished_at = NULL, started_at = NULL, attempts_made = 0,
|
||||
attempts_started = 0, stalled_counter = 0, updated_at = now()
|
||||
WHERE id = $1 AND status IN ('failed', 'dead')
|
||||
RETURNING *`,
|
||||
[id]
|
||||
|
||||
@@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts';
|
||||
import { operations } from '../../operations.ts';
|
||||
import type { Operation, OperationContext } from '../../operations.ts';
|
||||
import { paramDefToSchema } from '../../../mcp/tool-defs.ts';
|
||||
import { validateSourceId } from '../../utils.ts';
|
||||
import type { ToolCtx, ToolDef } from '../types.ts';
|
||||
|
||||
/**
|
||||
@@ -201,6 +202,13 @@ export interface BuildBrainToolsOpts {
|
||||
* SubagentHandlerData.allowed_slug_prefixes via the handler.
|
||||
*/
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
/**
|
||||
* Brain source every tool-call OperationContext is scoped to (#1586).
|
||||
* Trusted (flows from SubagentHandlerData.source_id, which only
|
||||
* PROTECTED_JOB_NAMES-gated submitters can set); validated at build time.
|
||||
* Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
interface OpContextDeps {
|
||||
@@ -211,6 +219,7 @@ interface OpContextDeps {
|
||||
signal?: AbortSignal;
|
||||
brainId?: string;
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
@@ -224,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
},
|
||||
dryRun: false,
|
||||
remote: true, // match MCP trust boundary for auto-link skip
|
||||
sourceId: 'default', // v0.34 D4: required; subagent tools default to host source
|
||||
// #1586: cycle-resolved source when provided; legacy host default else.
|
||||
sourceId: deps.sourceId ?? 'default',
|
||||
jobId: deps.jobId,
|
||||
subagentId: deps.subagentId,
|
||||
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
|
||||
@@ -248,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name),
|
||||
);
|
||||
|
||||
// #1586: fail fast on a malformed source id before any tool executes
|
||||
// (defense-in-depth — the seam is trusted, but the value round-trips
|
||||
// through the job payload).
|
||||
if (opts.sourceId !== undefined) validateSourceId(opts.sourceId);
|
||||
|
||||
return picked.map<ToolDef>(op => {
|
||||
const schema = op.name === 'put_page'
|
||||
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
|
||||
@@ -277,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
signal: ctx.signal,
|
||||
brainId: opts.brainId,
|
||||
allowedSlugPrefixes: opts.allowedSlugPrefixes,
|
||||
sourceId: opts.sourceId,
|
||||
});
|
||||
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
|
||||
return op.handler(opCtx, params);
|
||||
|
||||
@@ -200,6 +200,12 @@ export interface MinionJobContext {
|
||||
attempts_made: number;
|
||||
/** AbortSignal for cooperative cancellation (fires on timeout, cancel, pause, or lock loss). */
|
||||
signal: AbortSignal;
|
||||
/** Absolute wall-clock deadline (epoch ms) from the claim-time `timeout_at` stamp,
|
||||
* or null when the job has no per-job timeout. This is the DB's ground truth —
|
||||
* the same instant handleTimeouts() dead-letters against — so handlers that
|
||||
* spawn bounded sub-work (e.g. autopilot-cycle's subagent phases) can budget
|
||||
* from the REMAINING time instead of a fixed constant that may exceed it. */
|
||||
deadlineAtMs: number | null;
|
||||
/** AbortSignal that fires only on worker process SIGTERM/SIGINT. Handlers sensitive
|
||||
* to deploy restarts (e.g. the shell handler, which must run a SIGTERM → 5s → SIGKILL
|
||||
* sequence on its child) listen to this in addition to `signal`. Most handlers can
|
||||
@@ -455,6 +461,17 @@ export interface SubagentHandlerData {
|
||||
* and direct CLI submitters set it.
|
||||
*/
|
||||
allowed_slug_prefixes?: string[];
|
||||
/**
|
||||
* Brain source the subagent's tool calls are scoped to (#1586).
|
||||
*
|
||||
* When set, every tool-call `OperationContext.sourceId` uses this value
|
||||
* instead of the legacy 'default', so put_page writes land in the cycle's
|
||||
* resolved source. Same trust story as `allowed_slug_prefixes`:
|
||||
* PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and
|
||||
* direct CLI submitters can set it. Validated via `validateSourceId` at
|
||||
* tool-registry build time.
|
||||
*/
|
||||
source_id?: string;
|
||||
/**
|
||||
* v0.41 Approach C: opt out of the auto-generated tool-usage preamble
|
||||
* that `buildSystemPrompt()` splices into `system`. Default behavior
|
||||
|
||||
@@ -900,15 +900,22 @@ export class MinionWorker extends EventEmitter {
|
||||
|
||||
// Per-job wall-clock timeout (timer-armed only if `timeout_ms` was
|
||||
// set on the job; the grace-evict pattern above now lives outside
|
||||
// this branch).
|
||||
// this branch). The delay derives from the claim-time `timeout_at`
|
||||
// stamp when present so this timer, the DB sweeper (handleTimeouts),
|
||||
// and the handler-visible `deadlineAtMs` all agree on ONE absolute
|
||||
// deadline instead of three clocks started at slightly different
|
||||
// instants.
|
||||
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
if (job.timeout_ms != null) {
|
||||
const delayMs = job.timeout_at != null
|
||||
? Math.max(0, job.timeout_at.getTime() - Date.now())
|
||||
: job.timeout_ms;
|
||||
timeoutTimer = setTimeout(() => {
|
||||
if (!abort.signal.aborted) {
|
||||
console.warn(`Job ${job.id} (${job.name}) hit per-job timeout (${job.timeout_ms}ms), aborting`);
|
||||
abort.abort(new Error('timeout'));
|
||||
}
|
||||
}, job.timeout_ms);
|
||||
}, delayMs);
|
||||
}
|
||||
|
||||
const promise = this.executeJob(job, lockToken, abort, lockTimer)
|
||||
@@ -964,6 +971,7 @@ export class MinionWorker extends EventEmitter {
|
||||
data: job.data,
|
||||
attempts_made: job.attempts_made,
|
||||
signal: abort.signal,
|
||||
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
|
||||
shutdownSignal: this.shutdownAbort.signal,
|
||||
updateProgress: async (progress: unknown) => {
|
||||
await this.queue.updateProgress(job.id, lockToken, progress);
|
||||
|
||||
+135
-5
@@ -55,7 +55,7 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
|
||||
import { finalizeLastSeen } from './chronicle/last-seen.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
|
||||
import {
|
||||
normalizeEngineColumn,
|
||||
buildVectorCastFragment,
|
||||
@@ -1630,7 +1630,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
const keywordSql =
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
@@ -1654,10 +1654,140 @@ export class PGLiteEngine implements BrainEngine {
|
||||
${buildBestPerPagePoolCte('ranked')}
|
||||
SELECT * FROM best_per_page
|
||||
ORDER BY score DESC, page_id ASC, chunk_id ASC
|
||||
LIMIT $3 OFFSET $4`,
|
||||
params
|
||||
);
|
||||
LIMIT $3 OFFSET $4`;
|
||||
|
||||
let { rows } = await this.db.query(keywordSql, params);
|
||||
// D2 fix (fix/title-retrieval-arm): websearch AND semantics at chunk
|
||||
// grain mean one non-co-occurring token zeroes keyword recall. When the
|
||||
// strict query returns nothing, retry ONCE with OR-of-terms. Strict-AND
|
||||
// results always win when non-empty (no change for working queries).
|
||||
// Opt-in via SearchOpts.orFallback (Reviewer F1): only hybridSearch's
|
||||
// recall arm relaxes; precision consumers (countMentions,
|
||||
// link-extraction, eval) keep the strict-AND contract.
|
||||
if (rows.length === 0 && opts?.orFallback) {
|
||||
const orQuery = buildOrFallbackWebsearchQuery(query);
|
||||
if (orQuery) {
|
||||
const fallbackParams = [...params];
|
||||
fallbackParams[0] = orQuery;
|
||||
({ rows } = await this.db.query(keywordSql, fallbackParams));
|
||||
}
|
||||
}
|
||||
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
/**
|
||||
* fix/title-retrieval-arm (D1): page-grain title candidate arm. See the
|
||||
* BrainEngine interface doc for the full contract. Queries
|
||||
* pages.search_vector (title weight 'A' dominates ts_rank_cd by
|
||||
* construction) with the same page-grain filters the keyword arm applies
|
||||
* (type/types/excludeSlugs/date/source scoping, hard-excludes,
|
||||
* visibility), joined to one representative chunk per page. Applies the
|
||||
* same AND→OR recall fallback as searchKeyword. NO query-length gate —
|
||||
* long exact-title queries are the case this arm exists for.
|
||||
*
|
||||
* CJK queries fall through to websearch FTS here (a single-token CJK
|
||||
* query CAN exact-match a single-token CJK title); the richer CJK ILIKE
|
||||
* fallback stays keyword-arm-only.
|
||||
*/
|
||||
async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
// language/symbolKind are chunk-grain code filters with no page-grain
|
||||
// meaning; a code-scoped query gets no title candidates rather than
|
||||
// rows that silently violate the caller's filter.
|
||||
if (opts?.language || opts?.symbolKind) return [];
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const detailLow = opts?.detail === 'low';
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const params: unknown[] = [query, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.type) {
|
||||
params.push(opts.type);
|
||||
extraFilter += ` AND p.type = $${params.length}`;
|
||||
}
|
||||
if (opts?.types && opts.types.length > 0) {
|
||||
params.push(opts.types);
|
||||
extraFilter += ` AND p.type = ANY($${params.length}::text[])`;
|
||||
}
|
||||
if (opts?.exclude_slugs?.length) {
|
||||
params.push(opts.exclude_slugs);
|
||||
extraFilter += ` AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
if (opts?.afterDate) {
|
||||
params.push(opts.afterDate);
|
||||
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
|
||||
}
|
||||
if (opts?.beforeDate) {
|
||||
params.push(opts.beforeDate);
|
||||
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
|
||||
}
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
params.push(opts.sourceIds);
|
||||
extraFilter += ` AND p.source_id = ANY($${params.length}::text[])`;
|
||||
} else if (opts?.sourceId) {
|
||||
params.push(opts.sourceId);
|
||||
extraFilter += ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
|
||||
// Page grain — one row per page by construction, so no best_per_page
|
||||
// pooling CTE is needed. The LEFT JOIN LATERAL picks the representative
|
||||
// chunk (compiled_truth first, then lowest chunk_index); COALESCEs keep
|
||||
// chunkless pages retrievable (the extreme D1 case: a title with no
|
||||
// body) with the alias-hop row shape (chunk_id 0, empty chunk_text).
|
||||
// Accepted limitations (Reviewer F5/F6): the synthetic chunkless row
|
||||
// inherits the compiled-truth RRF boost and dedups on empty chunk_text;
|
||||
// and detail='low' filters only the REPRESENTATIVE — pages without a
|
||||
// compiled_truth chunk still surface (unlike the keyword arm's filter).
|
||||
const titlesSql =
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
COALESCE(rep.id, 0) as chunk_id,
|
||||
COALESCE(rep.chunk_index, 0) as chunk_index,
|
||||
COALESCE(rep.chunk_text, '') as chunk_text,
|
||||
COALESCE(rep.chunk_source, 'compiled_truth') as chunk_source,
|
||||
ts_rank_cd(p.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM pages p
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_source
|
||||
FROM content_chunks cc
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.modality = 'text'
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
ORDER BY (cc.chunk_source = 'compiled_truth') DESC, cc.chunk_index ASC
|
||||
LIMIT 1
|
||||
) rep ON true
|
||||
WHERE p.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC, p.id ASC
|
||||
LIMIT $2 OFFSET $3`;
|
||||
|
||||
let { rows } = await this.db.query(titlesSql, params);
|
||||
if (rows.length === 0) {
|
||||
const orQuery = buildOrFallbackWebsearchQuery(query);
|
||||
if (orQuery) {
|
||||
const fallbackParams = [...params];
|
||||
fallbackParams[0] = orQuery;
|
||||
({ rows } = await this.db.query(titlesSql, fallbackParams));
|
||||
}
|
||||
}
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
|
||||
@@ -1022,6 +1022,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + schema-embedded.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -1033,7 +1039,6 @@ BEGIN
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
+615
-335
File diff suppressed because it is too large
Load Diff
@@ -830,6 +830,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- Function to rebuild search_vector for a page
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + pglite-schema.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS \$\$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -843,7 +849,6 @@ BEGIN
|
||||
-- Build weighted tsvector
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
+116
-17
@@ -34,6 +34,7 @@ import { normalizeAlias } from './alias-normalize.ts';
|
||||
import { stampEvidence } from './evidence.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
import { enforceTokenBudget } from './token-budget.ts';
|
||||
import { warnOncePerProcess } from '../utils.ts';
|
||||
import { recordSearchTelemetry } from './telemetry.ts';
|
||||
import {
|
||||
weightsForIntent,
|
||||
@@ -740,6 +741,21 @@ export interface HybridSearchOpts extends SearchOpts {
|
||||
* a fresh per-call deadline. Not part of the public contract.
|
||||
*/
|
||||
_queryEmbedDeadline?: QueryEmbedDeadline;
|
||||
|
||||
/**
|
||||
* INTERNAL — cache-consult outcome threaded from `hybridSearchCached` into
|
||||
* the inner `hybridSearch` so the ONE telemetry record per search (emitted
|
||||
* by the inner function) carries the cache classification: 'miss' when the
|
||||
* semantic cache was consulted and had no row, 'disabled' when the consult
|
||||
* was skipped (cache off, walk/near-symbol/non-default-column/adaptive
|
||||
* skip, or the lookup embed failed). Folded into the RECORDED meta only —
|
||||
* `onMeta` payloads are unchanged. Direct `hybridSearch` callers leave it
|
||||
* undefined and keep recording with no cache field (they never consulted
|
||||
* the cache). The cache-HIT record is emitted by `hybridSearchCached`
|
||||
* itself, since the inner function never runs on a hit. Not part of the
|
||||
* public contract.
|
||||
*/
|
||||
_telemetryCacheStatus?: 'miss' | 'disabled';
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -917,6 +933,11 @@ export async function hybridSearch(
|
||||
// it never has to read config. Engines normalize string-or-descriptor
|
||||
// via normalizeEngineColumn; the descriptor path is the strict one.
|
||||
embeddingColumn: resolvedCol,
|
||||
// D2 fix (fix/title-retrieval-arm, Reviewer F1): the hybrid keyword arm
|
||||
// is a recall arm — opt in to the engine's AND→OR zero-recall fallback.
|
||||
// Direct searchKeyword consumers (countMentions, link-extraction, eval)
|
||||
// do NOT set this and keep the strict-AND contract.
|
||||
orFallback: true,
|
||||
};
|
||||
// Track what actually ran for the optional onMeta callback (v0.25.0).
|
||||
// Caller leaves onMeta undefined → these flags are computed but never
|
||||
@@ -943,7 +964,15 @@ export async function hybridSearch(
|
||||
// swallow — capture telemetry is best-effort
|
||||
}
|
||||
try {
|
||||
recordSearchTelemetry(engine, meta, { results_count: lastResultsCount, rank1_score: lastRank1Score });
|
||||
// #2952 — fold the cache-consult outcome (threaded by hybridSearchCached)
|
||||
// into the RECORDED meta only. None of the inner return paths set a
|
||||
// `cache` field themselves, so this is the sole source of the miss /
|
||||
// disabled classification; `onMeta` consumers above still receive the
|
||||
// meta unchanged (the cached wrapper emits its own merged meta to them).
|
||||
const recordedMeta = opts?._telemetryCacheStatus
|
||||
? { ...meta, cache: { status: opts._telemetryCacheStatus } }
|
||||
: meta;
|
||||
recordSearchTelemetry(engine, recordedMeta, { results_count: lastResultsCount, rank1_score: lastRank1Score });
|
||||
} catch {
|
||||
// swallow — telemetry must never break the search hot path.
|
||||
}
|
||||
@@ -967,8 +996,31 @@ export async function hybridSearch(
|
||||
const earlyModality = (opts?.crossModal && opts.crossModal !== 'auto')
|
||||
? opts.crossModal
|
||||
: (suggestions.suggestedModality ?? 'text');
|
||||
const keywordResults: SearchResult[] =
|
||||
earlyModality === 'image' ? [] : await engine.searchKeyword(query, searchOpts);
|
||||
// D1 fix (fix/title-retrieval-arm): page-grain title candidate arm,
|
||||
// fetched CONCURRENTLY with the keyword arm (Reviewer F7 — independent
|
||||
// engine queries). The chunk FTS vector never includes the page title, so
|
||||
// an exact-title query can be unretrievable by keyword — this arm queries
|
||||
// pages.search_vector (title weight 'A') directly. Runs regardless of
|
||||
// query token count: the alias hop (≤6-token guard) and the title-phrase
|
||||
// boost are re-rank-only, so LONG exact-title queries — where strict-AND
|
||||
// chunk FTS is weakest — need a candidate GENERATOR. Fail-open WITH
|
||||
// SIGNAL (Reviewer F2): a SQL error (e.g. a pre-search_vector brain)
|
||||
// degrades to no title candidates, but warns once per process so a
|
||||
// broken engine arm cannot ship dark.
|
||||
const [keywordResults, titleResults]: [SearchResult[], SearchResult[]] =
|
||||
earlyModality === 'image'
|
||||
? [[], []]
|
||||
: await Promise.all([
|
||||
engine.searchKeyword(query, searchOpts),
|
||||
engine.searchTitles(query, searchOpts).catch((err: unknown) => {
|
||||
warnOncePerProcess(
|
||||
'search-titles-arm-failed',
|
||||
`[gbrain] searchTitles arm failed (fail-open, title candidates skipped): ` +
|
||||
`${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
return [] as SearchResult[];
|
||||
}),
|
||||
]);
|
||||
|
||||
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
|
||||
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
|
||||
@@ -1046,14 +1098,16 @@ export async function hybridSearch(
|
||||
if (!isAvailable('embedding', providerProbe)) {
|
||||
// v0.43 — fuse the relational arm with keyword so typed-edge answers
|
||||
// survive on the no-embedding-provider path (the relational win is most
|
||||
// valuable exactly when vector is unavailable).
|
||||
// valuable exactly when vector is unavailable). The title arm fuses here
|
||||
// too — an exact-title lookup on a keyless install is precisely where
|
||||
// chunk-grain keyword FTS alone fails (D1).
|
||||
let noEmbedResults = keywordResults;
|
||||
if (relationalList.length > 0) {
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
noEmbedResults = rrfFusionWeighted(
|
||||
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
|
||||
detailResolved !== 'high',
|
||||
);
|
||||
const noEmbedLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
|
||||
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
|
||||
}
|
||||
if (noEmbedResults.length > 0) {
|
||||
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
|
||||
@@ -1280,14 +1334,15 @@ export async function hybridSearch(
|
||||
// post-fusion stages here too — without it, salience='on' silently
|
||||
// does nothing on embed failures.
|
||||
// v0.43: fuse the relational arm with keyword via RRF so typed-edge
|
||||
// answers survive even when vector is unavailable.
|
||||
// answers survive even when vector is unavailable. The title arm fuses
|
||||
// here too (same rationale as the no-embedding-provider path — D1).
|
||||
let fallbackResults = keywordResults;
|
||||
if (relationalList.length > 0) {
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
fallbackResults = rrfFusionWeighted(
|
||||
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
|
||||
detail !== 'high',
|
||||
);
|
||||
const fallbackLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
|
||||
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
|
||||
}
|
||||
if (fallbackResults.length > 0) {
|
||||
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
|
||||
@@ -1352,6 +1407,15 @@ export async function hybridSearch(
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
|
||||
// D1 fix (fix/title-retrieval-arm) — title candidate arm as a third
|
||||
// weighted list. Fuses at the keyword arm's intent-effective k (same
|
||||
// lexical-evidence class, no new tunable). Mirrors the keyword list's
|
||||
// inclusion rules: fetch was gated on earlyModality, so no extra modality
|
||||
// check here. Empty for non-matching queries → pure no-op.
|
||||
if (titleResults.length > 0) {
|
||||
allLists.push({ list: titleResults, k: keywordK });
|
||||
}
|
||||
|
||||
// v0.43 — relational recall arm (fourth RRF arm), built above so it also
|
||||
// contributes on the keyword-only fallback path. Neutral weight (baseRrfK):
|
||||
// competes evenly with keyword/vector, not dominating. Empty for
|
||||
@@ -1744,8 +1808,17 @@ export async function hybridSearchCached(
|
||||
...(hit.meta?.embedding_column ? { embedding_column: hit.meta.embedding_column } : {}),
|
||||
...(hit.meta?.adaptive_return ? { adaptive_return: hit.meta.adaptive_return } : {}),
|
||||
...(hit.meta?.autocut ? { autocut: hit.meta.autocut } : {}),
|
||||
// Per-call budget: prefer the STORED budget record, which carries
|
||||
// the true dropped count from the write-time cut — the
|
||||
// re-application above ran on an already-cut set and reads
|
||||
// dropped=0 (same masking as the miss path's finalMeta). Safe
|
||||
// unconditionally: tokenBudget is folded into knobsHash (`tb=`),
|
||||
// so a hit only ever serves a lookup with the identical resolved
|
||||
// budget as the write — the outer pass can never cut further.
|
||||
// budgetMeta stays as the fallback for legacy rows stored without
|
||||
// a budget record.
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
? { token_budget: hit.meta?.token_budget ?? budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
@@ -1753,6 +1826,21 @@ export async function hybridSearchCached(
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort
|
||||
}
|
||||
// #2952 — a cache hit never reaches the inner hybridSearch (the only
|
||||
// other telemetry site), so record the search HERE or it vanishes from
|
||||
// stats entirely (count, results, tokens, rank-1 — not just the hit
|
||||
// counter). Same rank-1 rule as the inner return paths. Tokens are
|
||||
// gated on the MODE-resolved budget, mirroring the inner paths' `if
|
||||
// (resolvedMode.tokenBudget > 0)` meta condition — otherwise a
|
||||
// tokenmax (budget-off) brain would record real tokens on hits but 0
|
||||
// on misses, skewing avg-tokens upward as the hit rate rises (codex).
|
||||
recordSearchTelemetry(engine, cachedMeta, {
|
||||
results_count: budgeted.length,
|
||||
...(resolvedForCache.tokenBudget && resolvedForCache.tokenBudget > 0
|
||||
? { tokens_estimate: budgetMeta.used }
|
||||
: {}),
|
||||
rank1_score: budgeted[0] ? (budgeted[0].base_score ?? budgeted[0].score) : undefined,
|
||||
});
|
||||
return budgeted;
|
||||
}
|
||||
}
|
||||
@@ -1768,6 +1856,10 @@ export async function hybridSearchCached(
|
||||
// v0.42.20.0 (Fix 3) — share the query-embed deadline so the inner embed
|
||||
// doesn't start a fresh 6s budget after the cache-lookup already spent it.
|
||||
_queryEmbedDeadline: queryEmbedDl,
|
||||
// #2952 — classify this search's telemetry record (emitted by the inner
|
||||
// function) with the cache-consult outcome. 'hit' already returned above,
|
||||
// so only miss/disabled reach this call.
|
||||
_telemetryCacheStatus: cacheStatus === 'disabled' ? 'disabled' : 'miss',
|
||||
onMeta: (m) => {
|
||||
innerMetaBox.current = m;
|
||||
// Do NOT call userOnMeta here — we'll emit a merged meta below
|
||||
@@ -1794,8 +1886,15 @@ export async function hybridSearchCached(
|
||||
...(innerMeta?.embedding_column ? { embedding_column: innerMeta.embedding_column } : {}),
|
||||
...(innerMeta?.adaptive_return ? { adaptive_return: innerMeta.adaptive_return } : {}),
|
||||
...(innerMeta?.autocut ? { autocut: innerMeta.autocut } : {}),
|
||||
// Per-call budget: prefer the INNER meta's budget record. The inner
|
||||
// hybridSearch already enforced the same resolved budget (per-call wins
|
||||
// in resolveSearchMode), so the re-application above sees an
|
||||
// already-cut set and its meta reads dropped=0 — masking the real cut
|
||||
// from onMeta consumers (the `dropped` under-report the restored
|
||||
// search-lite test caught). The outer pass stays as the enforcement
|
||||
// for the cache-HIT path, where no inner run exists.
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
? { token_budget: innerMeta?.token_budget ?? budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
|
||||
@@ -206,6 +206,51 @@ export function buildBestPerPagePoolCte(candidateCte: string): string {
|
||||
)`;
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// AND→OR keyword-recall fallback (fix/title-retrieval-arm, D2)
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Build a relaxed OR-of-terms websearch string for the keyword-arm recall
|
||||
* fallback.
|
||||
*
|
||||
* `websearch_to_tsquery('english', query)` joins unquoted terms with `&`
|
||||
* (AND). At chunk grain, one query token that doesn't co-occur in any
|
||||
* single chunk zeroes keyword recall with no fallback. When the strict
|
||||
* AND query returns zero rows, engines retry ONCE with the string this
|
||||
* builder returns — the same tokens joined with websearch's `OR` keyword,
|
||||
* which compiles to `|`.
|
||||
*
|
||||
* Why rebuild via websearch syntax instead of hand-assembling a tsquery:
|
||||
* websearch_to_tsquery never raises on malformed input, applies the same
|
||||
* stemming/stopword pipeline as the document side, and an all-stopword
|
||||
* token list degrades to an empty tsquery (matches nothing) instead of a
|
||||
* SQL error — the empty-tsquery guard comes free.
|
||||
*
|
||||
* Returns null when relaxation is pointless or unsafe:
|
||||
* - fewer than 2 tokens survive tokenization (OR of one term is the same
|
||||
* query as AND of one term);
|
||||
* - the raw query uses websearch OPERATORS (Reviewer F3): a `-term`
|
||||
* negation would be RESURRECTED as a positive OR term, and a quoted
|
||||
* phrase would degrade to a bag of words — both invert caller intent,
|
||||
* so operator queries get no fallback at all.
|
||||
* Tokenization splits on non-alphanumeric runs (Unicode-aware). Literal
|
||||
* OR/AND words are dropped so they can't be re-parsed as operators
|
||||
* mid-list.
|
||||
*/
|
||||
export function buildOrFallbackWebsearchQuery(query: string): string | null {
|
||||
// F3 operator guard: any double quote, or a dash LEADING a token
|
||||
// (whitespace/start boundary — interior hyphens like "foo-bar" are fine).
|
||||
if (query.includes('"') || /(^|\s)-\S/.test(query)) return null;
|
||||
const tokens = query
|
||||
.normalize('NFKC')
|
||||
.split(/[^\p{L}\p{N}]+/u)
|
||||
.filter(Boolean)
|
||||
.filter(t => { const u = t.toUpperCase(); return u !== 'OR' && u !== 'AND'; });
|
||||
if (tokens.length < 2) return null;
|
||||
return tokens.join(' OR ');
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// v0.29.1 — Recency component SQL builder
|
||||
// ============================================================
|
||||
|
||||
+9
-4
@@ -219,7 +219,7 @@ function globToRegex(pattern: string): RegExp {
|
||||
return new RegExp(regex);
|
||||
}
|
||||
|
||||
function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
export function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
if (!patterns || patterns.length === 0) return false;
|
||||
const normalized = path.replace(/\\/g, '/');
|
||||
return patterns.some((pattern) => globToRegex(pattern).test(normalized));
|
||||
@@ -255,7 +255,12 @@ const PRUNE_DIR_NAMES = new Set<string>([
|
||||
// with the first-sync walker in commands/import.ts.
|
||||
'venv',
|
||||
'.raw',
|
||||
'ops',
|
||||
// NOTE (#2404): `'ops'` used to be in this list (a v0.2.0-era carve-out for
|
||||
// one brain layout). Matching the bare segment pruned EVERY user `ops/`
|
||||
// directory at any depth — sync silently deleted `ops/*` pages and never
|
||||
// imported `ops/*` files, while the bundled daily-task-manager skill
|
||||
// prescribes `ops/tasks` as its canonical page. `ops/` is ordinary content;
|
||||
// do NOT re-add it. Only generated/vendored trees belong here.
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -352,8 +357,8 @@ function classifySync(path: string, opts: SyncableOptions = {}): SyncableReason
|
||||
if (!isAllowedByStrategy(path, strategy)) return 'strategy';
|
||||
|
||||
// Skip every path segment that pruneDir would block walkers from descending
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars,
|
||||
// `node_modules/` (latent bug fix), and `ops/` at any depth.
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, and
|
||||
// vendor/generated trees (`node_modules/`, `vendor/`, …) at any depth.
|
||||
const segments = path.split('/');
|
||||
if (segments.some(p => !pruneDir(p))) return 'pruned-dir';
|
||||
|
||||
|
||||
@@ -974,6 +974,19 @@ export interface SearchOpts {
|
||||
* client) → `sourceIds`; otherwise `ctx.sourceId` (scalar) → `sourceId`.
|
||||
*/
|
||||
sourceIds?: string[];
|
||||
/**
|
||||
* fix/title-retrieval-arm (D2, Reviewer F1): opt-in AND→OR keyword-recall
|
||||
* fallback. When true, `searchKeyword` retries ONCE with OR-of-terms after
|
||||
* the strict websearch AND query returns zero rows (strict results always
|
||||
* win when non-empty). Default false/undefined = strict-AND only — the
|
||||
* pre-fix contract. hybridSearch opts in for its keyword arm; precision
|
||||
* consumers (enrichment countMentions, link-extraction resolution, eval
|
||||
* paths) MUST NOT set this: OR-matches would inflate mention counts and
|
||||
* relax link-candidate resolution ("John Smith" matching every John and
|
||||
* every Smith). `searchTitles` has its own page-grain fallback and
|
||||
* ignores this flag.
|
||||
*/
|
||||
orFallback?: boolean;
|
||||
/**
|
||||
* v0.27.1 / v0.36 (D11): target column for vector search. Two shapes:
|
||||
*
|
||||
|
||||
@@ -27,6 +27,7 @@ import { randomBytes } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts';
|
||||
import { isWriteTargetContained } from './path-confine.ts';
|
||||
import { isDurabilityHardened, commitWriteThroughFile } from './brain-repo-durability.ts';
|
||||
|
||||
/** Minimal logger surface — structurally compatible with operations.ts `Logger`. */
|
||||
export interface WriteThroughLogger {
|
||||
@@ -36,6 +37,13 @@ export interface WriteThroughLogger {
|
||||
export interface WriteThroughResult {
|
||||
written: boolean;
|
||||
path?: string;
|
||||
/**
|
||||
* True when the write was also committed to git (#2426). Only attempted on
|
||||
* repos hardened via `gbrain sources harden` (durability hook installed);
|
||||
* the hook then background-pushes the commit. Best-effort — a false/absent
|
||||
* value never blocks the write.
|
||||
*/
|
||||
committed?: boolean;
|
||||
/**
|
||||
* Non-error reasons the file was not written:
|
||||
* - no_repo_configured: the resolved target (source `local_path` or, for a
|
||||
@@ -157,7 +165,20 @@ export async function writePageThrough(
|
||||
throw writeErr;
|
||||
}
|
||||
|
||||
return { written: true, path: filePath };
|
||||
// #2426: on a durability-hardened repo (user ran `gbrain sources harden`),
|
||||
// commit the artifact so it reaches git — pre-fix, write-through content
|
||||
// stayed uncommitted forever: never pushed, `last_sync_at` frozen, and
|
||||
// silently deleted by a later `sync --full` delete-reconcile. The local
|
||||
// post-commit hook background-pushes the commit. Best-effort: a commit
|
||||
// failure never fails the write (the DB row + file are the durable sinks).
|
||||
let committed = false;
|
||||
try {
|
||||
if (isDurabilityHardened(writeRoot)) {
|
||||
committed = commitWriteThroughFile(writeRoot, filePath, slug);
|
||||
}
|
||||
} catch { /* best-effort */ }
|
||||
|
||||
return { written: true, path: filePath, ...(committed ? { committed } : {}) };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`);
|
||||
|
||||
+6
-1
@@ -826,6 +826,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- Function to rebuild search_vector for a page
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + pglite-schema.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -839,7 +845,6 @@ BEGIN
|
||||
-- Build weighted tsvector
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* gbrain#2490 — gateway.chat() never caches a stable system prompt across
|
||||
* varying single-turn calls (page-summary, skillopt, enrich).
|
||||
*
|
||||
* Root cause: `chat()` passed `system` as a bare string and relied solely on
|
||||
* a CALL-LEVEL `providerOptions.anthropic.cacheControl`. On `ai@6` +
|
||||
* `@ai-sdk/anthropic@3.x`, that call-level marker is real — it's serialized
|
||||
* as a top-level `cache_control` field on the Anthropic request body, which
|
||||
* the Messages API resolves via its documented "auto-cache the LAST
|
||||
* cacheable block in the request" shorthand (see Anthropic's prompt-caching
|
||||
* docs). For a single-turn call with a stable system prompt and a DIFFERENT
|
||||
* user message every time, "the last cacheable block" is that ever-varying
|
||||
* user message — every call WRITES a fresh cache entry there and never
|
||||
* READS a prior one, so `cache_read_input_tokens` stays 0 forever even
|
||||
* though a `cache_control` breakpoint genuinely reaches Anthropic.
|
||||
*
|
||||
* Fix: ALSO pass `system` as a `SystemModelMessage` object (`{ role:
|
||||
* 'system', content, providerOptions }`) when caching is requested — the
|
||||
* shape `ai` documents specifically for attaching provider options to the
|
||||
* system block — and mark the last tool def's own `providerOptions` too
|
||||
* (mirrors the already-correct raw-SDK path in `subagent.ts`). The
|
||||
* call-level marker is KEPT (not removed): it's what gives `toolLoop()`'s
|
||||
* growing multi-turn conversation a rolling cache breakpoint on each turn's
|
||||
* tail, which the explicit system/tool markers alone don't provide.
|
||||
*
|
||||
* These tests pin the FIX by inspecting the exact args handed to the
|
||||
* `generateText` transport (via `__setGenerateTextTransportForTests`),
|
||||
* not by asserting on `providerOptions` alone — that field is exactly what
|
||||
* the bug made you believe was sufficient.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import {
|
||||
chat,
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setGenerateTextTransportForTests,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
|
||||
describe('gbrain#2490 — Anthropic cache breakpoint placement', () => {
|
||||
beforeEach(() => {
|
||||
resetGateway();
|
||||
__setGenerateTextTransportForTests(null);
|
||||
});
|
||||
|
||||
async function captureTransportArgs(
|
||||
opts: Partial<Parameters<typeof chat>[0]> = {},
|
||||
): Promise<any> {
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
env: { ANTHROPIC_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
...opts,
|
||||
});
|
||||
return captured;
|
||||
}
|
||||
|
||||
test('cacheSystem:true puts a real breakpoint on the system block (SystemModelMessage, not a bare string)', async () => {
|
||||
const args = await captureTransportArgs({ system: 'You are a helpful assistant.', cacheSystem: true });
|
||||
|
||||
// The regression: `system` used to stay a bare string forever, which
|
||||
// carries no per-block `providerOptions` — no breakpoint could ever land.
|
||||
expect(typeof args.system).not.toBe('string');
|
||||
expect(args.system).toEqual({
|
||||
role: 'system',
|
||||
content: 'You are a helpful assistant.',
|
||||
providerOptions: { anthropic: { cacheControl: { type: 'ephemeral' } } },
|
||||
});
|
||||
});
|
||||
|
||||
test('cacheSystem:true ALSO keeps the call-level cache_control on top-level providerOptions (rolling-conversation cache for toolLoop)', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: true });
|
||||
|
||||
// Not removed: @ai-sdk/anthropic serializes this as the Anthropic API's
|
||||
// documented top-level "auto-cache the last cacheable block" shorthand,
|
||||
// which is what gives a growing multi-turn toolLoop() conversation a
|
||||
// rolling cache breakpoint on each turn's tail. The explicit
|
||||
// system-block marker (asserted above) is what actually fixes gbrain#2490
|
||||
// for single-turn callers — the two coexist, marking different blocks.
|
||||
expect(args.providerOptions?.anthropic?.cacheControl).toEqual({ type: 'ephemeral' });
|
||||
});
|
||||
|
||||
test('cacheSystem:true marks the LAST tool def with its own providerOptions.anthropic.cacheControl', async () => {
|
||||
const args = await captureTransportArgs({
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
tools: [
|
||||
{ name: 'search', description: 'search', inputSchema: { type: 'object', properties: {} } },
|
||||
{ name: 'put_page', description: 'put_page', inputSchema: { type: 'object', properties: {} } },
|
||||
],
|
||||
});
|
||||
|
||||
expect(args.tools.search.providerOptions).toBeUndefined();
|
||||
expect(args.tools.put_page.providerOptions).toEqual({
|
||||
anthropic: { cacheControl: { type: 'ephemeral' } },
|
||||
});
|
||||
});
|
||||
|
||||
test('cacheSystem:false (default) leaves system a byte-identical bare string — no behavior change', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: false });
|
||||
expect(args.system).toBe('SYS');
|
||||
expect(args.providerOptions).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem omitted entirely leaves system a byte-identical bare string — no behavior change', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS' });
|
||||
expect(args.system).toBe('SYS');
|
||||
expect(args.providerOptions).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true with no system prompt does not synthesize an empty cached system block', async () => {
|
||||
const args = await captureTransportArgs({ cacheSystem: true });
|
||||
expect(args.system).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true with no tools does not throw and leaves tools undefined', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: true });
|
||||
expect(args.tools).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true on a non-Anthropic model is silently ignored (supports_prompt_cache=false)', async () => {
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'openai:gpt-4o-mini',
|
||||
env: { OPENAI_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'openai:gpt-4o-mini',
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
// Still a bare string — the recipe doesn't support prompt caching, so
|
||||
// useCache is false regardless of the caller's request.
|
||||
expect(captured.system).toBe('SYS');
|
||||
});
|
||||
|
||||
test('a configured cacheControl TTL override applies to every breakpoint, not just the call-level one', async () => {
|
||||
// Codex review finding: with three independently-hardcoded `{type:
|
||||
// 'ephemeral'}` markers, a `provider_chat_options.anthropic.cacheControl`
|
||||
// TTL override (e.g. `ttl: '1h'`) would only reach the call-level marker
|
||||
// via applyConfiguredChatProviderOptions()'s deep-merge — the system and
|
||||
// tool markers would stay implicit 5m, mixing TTLs across breakpoints in
|
||||
// the same request. Assert all three markers derive from ONE canonical
|
||||
// value instead.
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
provider_chat_options: {
|
||||
anthropic: { cacheControl: { type: 'ephemeral', ttl: '1h' } },
|
||||
},
|
||||
env: { ANTHROPIC_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
tools: [{ name: 'search', description: 'search', inputSchema: { type: 'object', properties: {} } }],
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
const expected = { type: 'ephemeral', ttl: '1h' };
|
||||
expect(captured.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
expect((captured.system as any)?.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
expect(captured.tools?.search?.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
});
|
||||
});
|
||||
@@ -297,6 +297,14 @@ describe('chat touchpoint — provider_chat_options passthrough', () => {
|
||||
});
|
||||
|
||||
test('anthropic cacheControl survives provider_chat_options merging', async () => {
|
||||
// gbrain#2490: this call-level cacheControl is real (not a no-op) —
|
||||
// @ai-sdk/anthropic serializes it as the Anthropic API's documented
|
||||
// top-level "auto-cache the last cacheable block" shorthand. It's kept
|
||||
// alongside the fix (an explicit breakpoint on the system message's own
|
||||
// providerOptions — see test/ai/gateway-cache-breakpoint.test.ts) because
|
||||
// it's what gives toolLoop()'s growing multi-turn conversation a rolling
|
||||
// cache breakpoint on each turn's tail. See gateway.ts's `useCache` block
|
||||
// for the full explanation of why both markers are needed.
|
||||
const providerOptions = await captureProviderOptions({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
provider_chat_options: {
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Mistral recipe smoke.
|
||||
*
|
||||
* The load-bearing assertion here is the negative one: mistral-embed rejects
|
||||
* every dimension parameter with HTTP 400, so dimsProviderOptions() must emit
|
||||
* no dimension field for it. Same contract as voyage-4-nano, pinned the same
|
||||
* way (see the negative regression assertion in test/ai/gateway.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
|
||||
import { lookupEmbeddingPrice } from '../../src/core/embedding-pricing.ts';
|
||||
|
||||
describe('recipe: mistral', () => {
|
||||
test('registered with expected OpenAI-compatible shape', () => {
|
||||
const r = getRecipe('mistral');
|
||||
expect(r).toBeDefined();
|
||||
expect(r!.id).toBe('mistral');
|
||||
expect(r!.tier).toBe('openai-compat');
|
||||
expect(r!.implementation).toBe('openai-compatible');
|
||||
expect(r!.base_url_default).toBe('https://api.mistral.ai/v1');
|
||||
expect(r!.auth_env?.required).toEqual(['MISTRAL_API_KEY']);
|
||||
});
|
||||
|
||||
test('embedding touchpoint pins the measured 1024 dims and 64K batch ceiling', () => {
|
||||
const e = getRecipe('mistral')!.touchpoints.embedding;
|
||||
expect(e).toBeDefined();
|
||||
expect(e!.models).toContain('mistral-embed');
|
||||
expect(e!.default_dims).toBe(1024);
|
||||
// Measured: a 65,286-token batch is accepted, 66,960 returns 400 code 3210.
|
||||
expect(e!.max_batch_tokens).toBe(65_536);
|
||||
// chars_per_token is a DIVISOR in splitByTokenBudget(), so a lower value
|
||||
// is the conservative direction. The module default of 4 is an English
|
||||
// assumption and overshoots on denser prose.
|
||||
expect(e!.chars_per_token).toBe(2);
|
||||
});
|
||||
|
||||
test('NEGATIVE: no dimension parameter is emitted for mistral-embed', () => {
|
||||
// Mistral rejects both spellings:
|
||||
// {"dimensions": N} -> 400 extra_forbidden
|
||||
// {"output_dimension": N} -> 400 "does not support output_dimension"
|
||||
// If a future change adds mistral-embed to a flexible-dim allowlist in
|
||||
// dims.ts, this assertion fails before it reaches users as a 400 on every
|
||||
// embed call.
|
||||
expect(dimsProviderOptions('openai-compatible', 'mistral-embed', 1024)).toBeUndefined();
|
||||
expect(dimsProviderOptions('openai-compatible', 'mistral-embed-2312', 1024)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('embedding models resolve to a known price', () => {
|
||||
// An unknown price makes the embedding spend cap fail closed.
|
||||
expect(lookupEmbeddingPrice('mistral:mistral-embed').kind).toBe('known');
|
||||
expect(lookupEmbeddingPrice('mistral:mistral-embed-2312').kind).toBe('known');
|
||||
});
|
||||
|
||||
test('chat and expansion touchpoints accept their configured models', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
expect(r.touchpoints.chat!.supports_tools).toBe(true);
|
||||
expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false);
|
||||
expect(() => assertTouchpoint(r, 'chat', 'mistral-small-latest')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'expansion', 'ministral-3b-latest')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'embedding', 'mistral-embed')).not.toThrow();
|
||||
});
|
||||
|
||||
test('codestral-embed is deliberately absent (1536 dims would mix under a 1024 declaration)', () => {
|
||||
const e = getRecipe('mistral')!.touchpoints.embedding!;
|
||||
expect(e.models).not.toContain('codestral-embed');
|
||||
expect(e.models).not.toContain('codestral-embed-2505');
|
||||
});
|
||||
|
||||
test('default auth: MISTRAL_API_KEY set -> Bearer token', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
const auth = defaultResolveAuth(r, { MISTRAL_API_KEY: 'fake-mistral-key' }, 'embedding');
|
||||
expect(auth.headerName).toBe('Authorization');
|
||||
expect(auth.token).toBe('Bearer fake-mistral-key');
|
||||
});
|
||||
|
||||
test('default auth: missing MISTRAL_API_KEY -> AIConfigError', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
expect(() => defaultResolveAuth(r, {}, 'embedding')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Moonshot/Kimi local recipe smoke.
|
||||
*
|
||||
* This pins the governed production exception GBrain-Local-003: GBrain can
|
||||
* route configured Kimi chat/expansion IDs through Moonshot's OpenAI-compatible
|
||||
* endpoint without treating `moonshot` as an unknown provider.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('recipe: moonshot', () => {
|
||||
test('registered with expected OpenAI-compatible shape', () => {
|
||||
const r = getRecipe('moonshot');
|
||||
expect(r).toBeDefined();
|
||||
expect(r!.id).toBe('moonshot');
|
||||
expect(r!.tier).toBe('openai-compat');
|
||||
expect(r!.implementation).toBe('openai-compatible');
|
||||
expect(r!.base_url_default).toBe('https://api.moonshot.ai/v1');
|
||||
expect(r!.auth_env?.required).toEqual(['MOONSHOT_API_KEY']);
|
||||
});
|
||||
|
||||
test('chat and expansion touchpoints include Kimi K2.7 Code', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(r.touchpoints.chat).toBeDefined();
|
||||
expect(r.touchpoints.expansion).toBeDefined();
|
||||
expect(r.touchpoints.chat!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.expansion!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.chat!.supports_tools).toBe(true);
|
||||
expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false);
|
||||
});
|
||||
|
||||
test('configured Kimi model is accepted for chat and expansion', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => assertTouchpoint(r, 'chat', 'kimi-k2.7-code')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'expansion', 'kimi-k2.7-code')).not.toThrow();
|
||||
});
|
||||
|
||||
test('default auth: MOONSHOT_API_KEY set -> Bearer token', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
const auth = defaultResolveAuth(r, { MOONSHOT_API_KEY: 'fake-moonshot-key' }, 'chat');
|
||||
expect(auth.headerName).toBe('Authorization');
|
||||
expect(auth.token).toBe('Bearer fake-moonshot-key');
|
||||
});
|
||||
|
||||
test('default auth: missing MOONSHOT_API_KEY -> AIConfigError', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => defaultResolveAuth(r, {}, 'chat')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
@@ -146,6 +146,41 @@ describe('buildBrainTools', () => {
|
||||
),
|
||||
).rejects.toBeInstanceOf(OperationError);
|
||||
});
|
||||
|
||||
// #1586: sourceId threads through buildBrainTools → buildOpContext →
|
||||
// put_page → importFromContent, so subagent writes land in the cycle's
|
||||
// resolved source instead of the hardcoded 'default'.
|
||||
test('execute() on put_page writes to the configured sourceId (#1586)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
|
||||
VALUES ('mybrain', 'My Brain', '/tmp/mybrain', '{}'::jsonb, false, now())
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 42,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
|
||||
sourceId: 'mybrain',
|
||||
});
|
||||
const putPage = tools.find(t => t.name === 'brain_put_page');
|
||||
const ctx: ToolCtx = { engine, jobId: 1, remote: true };
|
||||
await putPage!.execute(
|
||||
{ slug: 'wiki/personal/reflections/2026-07-17-scoped', content: '---\ntitle: Scoped\n---\nbody' },
|
||||
ctx,
|
||||
);
|
||||
const rows = await engine.executeRaw<{ source_id: string }>(
|
||||
`SELECT source_id FROM pages WHERE slug = 'wiki/personal/reflections/2026-07-17-scoped'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].source_id).toBe('mybrain');
|
||||
});
|
||||
|
||||
test('buildBrainTools rejects a malformed sourceId at build time (#1586)', () => {
|
||||
expect(() =>
|
||||
buildBrainTools({ subagentId: 1, engine, config, sourceId: '../evil' }),
|
||||
).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('filterAllowedTools', () => {
|
||||
|
||||
@@ -90,6 +90,36 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
|
||||
} catch (e: any) { code = e.status ?? 1; }
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('#2426 — commits a MODIFIED tracked file even when the remote advanced (commit before pull)', () => {
|
||||
// Pre-fix, the helper ran `git pull --rebase` BEFORE staging, so any dirty
|
||||
// tree (a modified/enriched page — exactly the write-through case) aborted
|
||||
// with 'cannot pull with rebase: You have unstaged changes' (exit 3). The
|
||||
// helper could only ever commit untracked-NEW files, never modifications.
|
||||
// Remove the post-commit hook so its background push can't race the
|
||||
// helper's own push (macOS has no flock to serialize them) — this test
|
||||
// targets the HELPER's ordering; hook behavior is covered below.
|
||||
rmSync(join(work, '.git', 'hooks', 'post-commit'));
|
||||
// Advance the remote from a second clone so a pull is genuinely needed.
|
||||
const other = mkdtempSync(join(root, 'other-'));
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' });
|
||||
git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other');
|
||||
writeFileSync(join(other, 'remote.md'), 'from other\n');
|
||||
git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main');
|
||||
|
||||
// Dirty MODIFICATION of a tracked file in the hardened clone (write-through shape).
|
||||
writeFileSync(join(work, 'README.md'), 'modified by write-through\n');
|
||||
execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'wt: README', 'README.md'], {
|
||||
cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env,
|
||||
});
|
||||
|
||||
// Both the remote's commit and ours are on origin/main.
|
||||
const subjects = git(bare, 'log', '--format=%s', 'main');
|
||||
expect(subjects).toContain('wt: README');
|
||||
expect(subjects).toContain('remote change');
|
||||
// Working tree is clean — the modification was committed, not stranded.
|
||||
expect(git(work, 'status', '--porcelain', 'README.md')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('post-commit hook (D9 local, D7 self-contained)', () => {
|
||||
|
||||
@@ -174,6 +174,21 @@ describe('scanBrainSources partial-scan state', () => {
|
||||
expect(report.aborted_at_source).toBe('src-a');
|
||||
});
|
||||
|
||||
// #2946: the deadline race's timeout arm resolves a SENTINEL, not null —
|
||||
// a COUNT that legitimately resolves null (failed/absent count) before the
|
||||
// deadline must NOT be mistaken for a deadline hit: the source still gets
|
||||
// scanned, with db_page_count simply unavailable.
|
||||
test('COUNT resolving null before the deadline is not a deadline verdict — source still scans', async () => {
|
||||
const start = Date.now();
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: start + 5_000,
|
||||
dbPageCountForSource: async () => null,
|
||||
});
|
||||
const firstSource = report.per_source.find(r => r.source_id === 'src-a')!;
|
||||
expect(firstSource.status).toBe('scanned');
|
||||
expect(firstSource.db_page_count == null).toBe(true);
|
||||
});
|
||||
|
||||
// Codex adversarial #4 regression: even when dbPageCountForSource itself
|
||||
// would hang indefinitely, the Promise.race against the deadline must
|
||||
// resolve null and the scan must abort cleanly.
|
||||
|
||||
@@ -43,8 +43,10 @@ beforeAll(() => {
|
||||
writeFileSync(join(root, '.obsidian', 'workspace.json'), '{}');
|
||||
mkdirSync(join(root, 'people', 'pedro.raw'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'pedro.raw', 'source.md'), '---\ntitle: should not visit\n---\n');
|
||||
// ops/ is ORDINARY content (#2404) — walker MUST descend (it used to be
|
||||
// wrongly pruned, silently excluding user runbooks / ops/tasks).
|
||||
mkdirSync(join(root, 'ops', 'logs'), { recursive: true });
|
||||
writeFileSync(join(root, 'ops', 'logs', 'run.md'), '# nope\n');
|
||||
writeFileSync(join(root, 'ops', 'logs', 'run.md'), '---\ntitle: Run\n---\n\nbody\n');
|
||||
// Nested node_modules — must also be pruned, not just at the root.
|
||||
mkdirSync(join(root, 'people', 'tools', 'node_modules', 'inner'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'tools', 'node_modules', 'inner', 'a.md'), '---\ntitle: nope\n---\n');
|
||||
@@ -95,8 +97,11 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => {
|
||||
walkDir(root, (f) => { files.push(f); }, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/people'))).toBe(true);
|
||||
expect(visited.some(d => d.endsWith('/concepts/subdir'))).toBe(true);
|
||||
// ops/ is ordinary content — descended, not pruned (#2404).
|
||||
expect(visited.some(d => d.endsWith('/ops/logs'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true);
|
||||
// And explicitly does NOT visit the file under node_modules.
|
||||
expect(files.some(f => f.includes('/node_modules/'))).toBe(false);
|
||||
});
|
||||
@@ -107,7 +112,7 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => {
|
||||
// visitDir would be called with node_modules paths.
|
||||
const descents: string[] = [];
|
||||
walkDir(root, () => {}, (d) => descents.push(d));
|
||||
const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian|ops)(\/|$)/.test(d) || /\.raw$/.test(d));
|
||||
const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian)(\/|$)/.test(d) || /\.raw$/.test(d));
|
||||
expect(vendor).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -119,13 +124,20 @@ describe('collectFiles (frontmatter.ts) — descent-time pruning parity', () =>
|
||||
expect(visited.some(d => d.includes('/node_modules'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into .git, .obsidian, *.raw, or ops', () => {
|
||||
test('does NOT descend into .git, .obsidian, or *.raw', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/.git'))).toBe(false);
|
||||
expect(visited.some(d => d.includes('/.obsidian'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('.raw'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('DOES descend into ops/ — ordinary content, not a vendor tree (#2404)', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(true);
|
||||
const files = collectFiles(root);
|
||||
expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT descend into git submodule directories', () => {
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { isChronicleEligible } from '../src/core/chronicle/eligibility.ts';
|
||||
import { runChronicleExtract, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts';
|
||||
import { runChronicleExtract, parseJudgeJson, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts';
|
||||
import { runChronicleBackstop } from '../src/core/chronicle/backstop.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -111,6 +111,44 @@ describe('runChronicleExtract', () => {
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: none });
|
||||
expect(r.status).toBe('no_events');
|
||||
});
|
||||
|
||||
// #2606: a truncated or unparseable judge response must NOT be recorded as
|
||||
// a legitimate no_events — it gets a distinct skipped reason.
|
||||
test('truncated judge output → skipped/judge_truncated, not no_events (#2606)', async () => {
|
||||
const truncated: ChronicleJudge = async () => ({ events: [], failure: 'truncated' });
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: truncated });
|
||||
expect(r.status).toBe('skipped');
|
||||
expect(r.reason).toBe('judge_truncated');
|
||||
expect(await countEvents()).toBe(0);
|
||||
});
|
||||
|
||||
test('unparseable judge output → skipped/judge_parse_failed (#2606)', async () => {
|
||||
const parseFailed: ChronicleJudge = async () => ({ events: [], failure: 'parse_failed' });
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: parseFailed });
|
||||
expect(r.status).toBe('skipped');
|
||||
expect(r.reason).toBe('judge_parse_failed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseJudgeJson failure signalling (#2606)', () => {
|
||||
test('a legitimate empty array parses to []', () => {
|
||||
expect(parseJudgeJson('[]')).toEqual([]);
|
||||
expect(parseJudgeJson('```json\n[]\n```')).toEqual([]);
|
||||
});
|
||||
|
||||
test('a valid array round-trips', () => {
|
||||
const arr = parseJudgeJson('[{"when":"2026-06-18","who":[],"what":"x","kind":"meeting"}]');
|
||||
expect(Array.isArray(arr)).toBe(true);
|
||||
expect(arr!.length).toBe(1);
|
||||
});
|
||||
|
||||
test('empty / no-array / truncated / non-array responses return null', () => {
|
||||
expect(parseJudgeJson('')).toBeNull();
|
||||
expect(parseJudgeJson('I found no events worth extracting.')).toBeNull();
|
||||
// Truncated mid-array (the maxTokens-cap shape from the issue).
|
||||
expect(parseJudgeJson('[{"when":"2026-06-18","who":["a"],"what":"long ev')).toBeNull();
|
||||
expect(parseJudgeJson('{"events": 1}')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runChronicleBackstop gating', () => {
|
||||
|
||||
@@ -174,3 +174,47 @@ describe('regression — local config still passes through normally', () => {
|
||||
expect(r.stdout).not.toContain('"mode":"thin-client"');
|
||||
});
|
||||
});
|
||||
|
||||
describe('thin-client scratch-DB guard — jobs partial dispatch + config refusal', () => {
|
||||
test('`gbrain config set x y` is refused with pinpoint hint', async () => {
|
||||
seedThinClientConfig();
|
||||
const r = await run(['config', 'set', 'search.reranker.enabled', 'false']);
|
||||
expect(r.exitCode).toBe(1);
|
||||
expect(r.stderr).toContain('gbrain config');
|
||||
expect(r.stderr).toContain('not routable');
|
||||
expect(r.stderr).toContain('thin-client of https://brain-host.example/mcp');
|
||||
});
|
||||
|
||||
test('`gbrain jobs work` is refused with pinpoint hint (host-queue-bound)', async () => {
|
||||
seedThinClientConfig();
|
||||
const r = await run(['jobs', 'work']);
|
||||
expect(r.exitCode).toBe(1);
|
||||
expect(r.stderr).toContain('gbrain jobs');
|
||||
expect(r.stderr).toContain('not routable');
|
||||
expect(r.stderr).toContain('thin-client of https://brain-host.example/mcp');
|
||||
});
|
||||
|
||||
test('`gbrain jobs get` never fabricates a scratch local engine', async () => {
|
||||
// The regression this pins: on a thin-client install with a PGLite
|
||||
// engine key, `jobs get` connected a LOCAL engine before its remote
|
||||
// routing branch ran — creating an empty scratch PGLite store in the
|
||||
// thin-client GBRAIN_HOME and replaying the entire migration chain
|
||||
// ("Schema version 1 → N") on every invocation. The remote call to
|
||||
// brain-host.example will fail (unreachable) — irrelevant here. What
|
||||
// matters: no local store is created and no migration replay runs.
|
||||
seedThinClientConfig({ engine: 'pglite' });
|
||||
const r = await run(['jobs', 'get', '999']);
|
||||
const { existsSync } = await import('fs');
|
||||
expect(existsSync(join(tmp, '.gbrain', 'brain.pglite'))).toBe(false);
|
||||
expect(r.stdout + r.stderr).not.toContain('Schema version');
|
||||
expect(r.stdout + r.stderr).not.toContain('migration(s) pending');
|
||||
});
|
||||
|
||||
test('`gbrain jobs list` never fabricates a scratch local engine', async () => {
|
||||
seedThinClientConfig({ engine: 'pglite' });
|
||||
const r = await run(['jobs', 'list']);
|
||||
const { existsSync } = await import('fs');
|
||||
expect(existsSync(join(tmp, '.gbrain', 'brain.pglite'))).toBe(false);
|
||||
expect(r.stdout + r.stderr).not.toContain('Schema version');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
finishCliTeardown,
|
||||
flushThenExit,
|
||||
computeTeardownDeadlineMs,
|
||||
resolveDrainTimeoutMs,
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
setCliExitVerdict,
|
||||
currentExitCode,
|
||||
@@ -115,6 +116,67 @@ describe('computeTeardownDeadlineMs', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveDrainTimeoutMs', () => {
|
||||
test('defaults to the 2000ms registry budget', () => {
|
||||
expect(resolveDrainTimeoutMs()).toBe(2_000);
|
||||
});
|
||||
|
||||
test('GBRAIN_DRAIN_TIMEOUT_MS env override wins over the default', async () => {
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: '30000' }, async () => {
|
||||
expect(resolveDrainTimeoutMs()).toBe(30_000);
|
||||
});
|
||||
});
|
||||
|
||||
test('garbage, zero, and negative env values fall back to the default', async () => {
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: 'banana' }, async () => {
|
||||
expect(resolveDrainTimeoutMs()).toBe(2_000);
|
||||
});
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: '0' }, async () => {
|
||||
expect(resolveDrainTimeoutMs()).toBe(2_000);
|
||||
});
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: '-5' }, async () => {
|
||||
expect(resolveDrainTimeoutMs()).toBe(2_000);
|
||||
});
|
||||
});
|
||||
|
||||
test('finishCliTeardown drains with the env-resolved budget when no explicit drainTimeoutMs', async () => {
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: '12345' }, async () => {
|
||||
let drainBudget = -1;
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => {} },
|
||||
deadlineMs: 250,
|
||||
drain: async ({ timeoutMs }) => {
|
||||
drainBudget = timeoutMs;
|
||||
},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
});
|
||||
expect(drainBudget).toBe(12_345);
|
||||
});
|
||||
});
|
||||
|
||||
test('an explicit drainTimeoutMs still wins over the env override', async () => {
|
||||
await withEnv({ GBRAIN_DRAIN_TIMEOUT_MS: '12345' }, async () => {
|
||||
let drainBudget = -1;
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => {} },
|
||||
drainTimeoutMs: 777,
|
||||
deadlineMs: 250,
|
||||
drain: async ({ timeoutMs }) => {
|
||||
drainBudget = timeoutMs;
|
||||
},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
});
|
||||
expect(drainBudget).toBe(777);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — clean path', () => {
|
||||
test('drains with the injected budget, disconnects, returns; no exit, no warn', async () => {
|
||||
const calls: string[] = [];
|
||||
|
||||
@@ -98,6 +98,33 @@ describe('WARN-6 — main `gbrain --help` lists capture/brainstorm/lsd', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2795 — `sync --install-cron` help line no longer promises an unbuilt feature', () => {
|
||||
test('main `gbrain --help` does not advertise install-cron', () => {
|
||||
// Pre-fix: `sync --install-cron Install persistent sync daemon` was
|
||||
// listed in the top-level help with no flag parsing or handler behind
|
||||
// it anywhere in src/commands/sync.ts — `gbrain sync --install-cron`
|
||||
// silently ran an ordinary sync instead of installing anything.
|
||||
const { stdout, status } = runCli(['--help']);
|
||||
expect(status).toBe(0);
|
||||
expect(stdout).not.toContain('install-cron');
|
||||
expect(stdout).not.toContain('Install persistent sync daemon');
|
||||
});
|
||||
|
||||
test('main `gbrain --help` points sync users at the real continuous-daemon command', () => {
|
||||
const { stdout } = runCli(['--help']);
|
||||
// autopilot --install already runs sync+extract+embed on a schedule
|
||||
// (docs/architecture/KEY_FILES.md); point discoverability there instead
|
||||
// of promising a separate sync-only cron installer that never existed.
|
||||
expect(stdout).toMatch(/sync --watch \[--interval N\][^\n]*\n\s*See also: autopilot --install/);
|
||||
});
|
||||
|
||||
test('`gbrain sync --help` never listed install-cron either', () => {
|
||||
const { stdout, status } = runCli(['sync', '--help']);
|
||||
expect(status).toBe(0);
|
||||
expect(stdout).not.toContain('install-cron');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1175 — main `gbrain --help` SOURCES block matches the real subcommand set', () => {
|
||||
test('archive and its lifecycle siblings are listed', () => {
|
||||
const { stdout, status } = runCli(['--help']);
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* #2415 — configurable dream output namespace (`dream.synthesize.output_root`).
|
||||
*
|
||||
* The synthesize + patterns phases previously hardcoded `wiki/` in the
|
||||
* subagent prompt slug templates, the patterns reflection lookup, and the
|
||||
* trusted-workspace allow-list loaded from skills/_brain-filing-rules.json.
|
||||
* This suite pins:
|
||||
* - default 'wiki' → byte-identical prompt + verbatim filing-rule globs
|
||||
* (zero behavior change unless the key is set);
|
||||
* - a custom root remaps prompt slug templates and the allow-list globs;
|
||||
* - loadOutputRoot validates against the slug grammar (bad values fall
|
||||
* back to 'wiki');
|
||||
* - the patterns phase gathers reflections under the configured root.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { __testing, loadAllowedSlugPrefixes, loadOutputRoot } from '../src/core/cycle/synthesize.ts';
|
||||
import { runPhasePatterns } from '../src/core/cycle/patterns.ts';
|
||||
import type { DiscoveredTranscript } from '../src/core/cycle/transcript-discovery.ts';
|
||||
|
||||
const { buildSynthesisPrompt } = __testing;
|
||||
|
||||
const transcript: DiscoveredTranscript = {
|
||||
filePath: '/tmp/t.txt',
|
||||
basename: 't',
|
||||
content: 'User: hello world',
|
||||
contentHash: 'abcdef0123456789',
|
||||
inferredDate: '2026-07-17',
|
||||
} as DiscoveredTranscript;
|
||||
|
||||
describe('#2415: buildSynthesisPrompt output root', () => {
|
||||
test('defaults to wiki/ slug templates', () => {
|
||||
const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1);
|
||||
expect(prompt).toContain('wiki/personal/reflections/2026-07-17-');
|
||||
expect(prompt).toContain('wiki/originals/ideas/2026-07-17-');
|
||||
});
|
||||
|
||||
test('custom root replaces wiki/ in both slug templates', () => {
|
||||
const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1, '', 'notes');
|
||||
expect(prompt).toContain('notes/personal/reflections/2026-07-17-');
|
||||
expect(prompt).toContain('notes/originals/ideas/2026-07-17-');
|
||||
expect(prompt).not.toContain('wiki/personal/reflections/');
|
||||
expect(prompt).not.toContain('wiki/originals/ideas/');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2415: loadAllowedSlugPrefixes remap', () => {
|
||||
// Runs from the repo root, so skills/_brain-filing-rules.json resolves.
|
||||
test("default 'wiki' returns the filing-rule globs verbatim", async () => {
|
||||
const globs = await loadAllowedSlugPrefixes();
|
||||
expect(globs).toContain('wiki/personal/reflections/*');
|
||||
expect(globs).toContain('dream-cycle-summaries/*');
|
||||
});
|
||||
|
||||
test('custom root remaps only wiki/-rooted globs', async () => {
|
||||
const globs = await loadAllowedSlugPrefixes('notes');
|
||||
expect(globs).toContain('notes/personal/reflections/*');
|
||||
expect(globs).toContain('notes/originals/*');
|
||||
expect(globs).toContain('notes/personal/patterns/*');
|
||||
// Non-wiki globs pass through untouched.
|
||||
expect(globs).toContain('dream-cycle-summaries/*');
|
||||
expect(globs.some(g => g.startsWith('wiki/'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2415: loadOutputRoot validation + patterns gather scope', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('unset → wiki; trailing slash trimmed; invalid → wiki fallback', async () => {
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
await engine.setConfig('dream.synthesize.output_root', 'notes/');
|
||||
expect(await loadOutputRoot(engine)).toBe('notes');
|
||||
await engine.setConfig('dream.synthesize.output_root', '../escape');
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
await engine.setConfig('dream.synthesize.output_root', 'Bad_Root');
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
});
|
||||
|
||||
test('patterns phase gathers reflections under the configured root', async () => {
|
||||
await engine.setConfig('dream.synthesize.output_root', 'notes');
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await engine.putPage(`notes/personal/reflections/2026-07-17-r${i}`, {
|
||||
type: 'note',
|
||||
title: `R${i}`,
|
||||
compiled_truth: `reflection ${i}`,
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
}
|
||||
// A wiki/-rooted reflection must NOT be counted under the custom root.
|
||||
await engine.putPage('wiki/personal/reflections/2026-07-17-old', {
|
||||
type: 'note',
|
||||
title: 'Old',
|
||||
compiled_truth: 'legacy reflection',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
const result = await runPhasePatterns(engine, { brainDir: '/tmp', dryRun: true });
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.details?.reflections_considered).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,168 @@
|
||||
/**
|
||||
* #2781 — patterns phase budgets its subagent from the REMAINING parent-job
|
||||
* time instead of a fixed 30/35-min default that can exceed any
|
||||
* interval-derived cycle budget and dead-letter the whole cycle mid-phase.
|
||||
*
|
||||
* Layers:
|
||||
* 1. Unit tests on the exported pure `clampSubagentBudgets`.
|
||||
* 2. A real-queue check that `claim` stamps `timeout_at` (the DB ground
|
||||
* truth `deadlineAtMs` derives from) and leaves it null when the job
|
||||
* has no per-job timeout.
|
||||
* 3. Structural assertions pinning the wiring: worker → context →
|
||||
* handler → runCycle → patterns (matches the house style of
|
||||
* test/cycle-patterns.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import {
|
||||
clampSubagentBudgets,
|
||||
CYCLE_DEADLINE_RESERVE_MS,
|
||||
MIN_PATTERNS_SUBAGENT_BUDGET_MS,
|
||||
} from '../src/core/cycle/patterns.ts';
|
||||
|
||||
const CONFIG = {
|
||||
subagentTimeoutMs: 30 * 60 * 1000,
|
||||
subagentWaitTimeoutMs: 35 * 60 * 1000,
|
||||
};
|
||||
|
||||
describe('clampSubagentBudgets', () => {
|
||||
const now = 1_000_000_000_000; // fixed epoch ms; the function takes nowMs explicitly
|
||||
|
||||
test('null deadline → config passthrough (direct `gbrain dream` back-compat)', () => {
|
||||
expect(clampSubagentBudgets(CONFIG, null, now)).toEqual({
|
||||
timeoutMs: CONFIG.subagentTimeoutMs,
|
||||
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
|
||||
});
|
||||
expect(clampSubagentBudgets(CONFIG, undefined, now)).toEqual({
|
||||
timeoutMs: CONFIG.subagentTimeoutMs,
|
||||
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
|
||||
});
|
||||
});
|
||||
|
||||
test('deadline far away → config values win (no clamping)', () => {
|
||||
const deadline = now + 2 * 60 * 60 * 1000; // 2h out
|
||||
expect(clampSubagentBudgets(CONFIG, deadline, now)).toEqual({
|
||||
timeoutMs: CONFIG.subagentTimeoutMs,
|
||||
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
|
||||
});
|
||||
});
|
||||
|
||||
test('deadline inside config window → BOTH timeouts clamp to the same child budget', () => {
|
||||
const deadline = now + 10 * 60 * 1000; // 10 min out
|
||||
const childBudget = deadline - CYCLE_DEADLINE_RESERVE_MS - now; // 9 min
|
||||
const budgets = clampSubagentBudgets(CONFIG, deadline, now);
|
||||
expect(budgets).toEqual({ timeoutMs: childBudget, waitTimeoutMs: childBudget });
|
||||
// The child's own kill switch never outlives the parent budget.
|
||||
expect(budgets!.timeoutMs).toBeLessThanOrEqual(deadline - now);
|
||||
});
|
||||
|
||||
test('remaining budget below minimum → null (caller skips, no submit)', () => {
|
||||
const deadline = now + CYCLE_DEADLINE_RESERVE_MS + MIN_PATTERNS_SUBAGENT_BUDGET_MS - 1;
|
||||
expect(clampSubagentBudgets(CONFIG, deadline, now)).toBeNull();
|
||||
});
|
||||
|
||||
test('boundary: exactly the minimum budget → submit allowed', () => {
|
||||
const deadline = now + CYCLE_DEADLINE_RESERVE_MS + MIN_PATTERNS_SUBAGENT_BUDGET_MS;
|
||||
expect(clampSubagentBudgets(CONFIG, deadline, now)).toEqual({
|
||||
timeoutMs: MIN_PATTERNS_SUBAGENT_BUDGET_MS,
|
||||
waitTimeoutMs: MIN_PATTERNS_SUBAGENT_BUDGET_MS,
|
||||
});
|
||||
});
|
||||
|
||||
test('deadline already past → null, never a negative timeout', () => {
|
||||
expect(clampSubagentBudgets(CONFIG, now - 1000, now)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('claim stamps timeout_at (deadlineAtMs ground truth)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' }); // in-memory
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('job with timeout_ms → claim sets timeout_at ≈ now + timeout_ms', async () => {
|
||||
const before = Date.now();
|
||||
await queue.add('sync', {}, { timeout_ms: 600_000 });
|
||||
const claimed = await queue.claim('tok-dl-1', 30000, 'default', ['sync']);
|
||||
const after = Date.now();
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.timeout_at).not.toBeNull();
|
||||
const at = claimed!.timeout_at!.getTime();
|
||||
expect(at).toBeGreaterThanOrEqual(before + 600_000 - 5_000);
|
||||
expect(at).toBeLessThanOrEqual(after + 600_000 + 5_000);
|
||||
});
|
||||
|
||||
test('job without timeout_ms and no handler default → timeout_at stays null', async () => {
|
||||
// 'sync' is not in the long-handler default set, so no stamp either way.
|
||||
await queue.add('sync', { which: 'no-timeout' });
|
||||
// Drain the possibly-remaining job from the prior test first.
|
||||
let claimed = await queue.claim('tok-dl-2', 30000, 'default', ['sync']);
|
||||
while (claimed && claimed.timeout_ms != null) {
|
||||
claimed = await queue.claim('tok-dl-2', 30000, 'default', ['sync']);
|
||||
}
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.timeout_ms).toBeNull();
|
||||
expect(claimed!.timeout_at).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('deadline plumbing wiring (structural)', () => {
|
||||
const workerSrc = readFileSync(new URL('../src/core/minions/worker.ts', import.meta.url), 'utf-8');
|
||||
const jobsSrc = readFileSync(new URL('../src/commands/jobs.ts', import.meta.url), 'utf-8');
|
||||
const cycleSrc = readFileSync(new URL('../src/core/cycle.ts', import.meta.url), 'utf-8');
|
||||
const patternsSrc = readFileSync(new URL('../src/core/cycle/patterns.ts', import.meta.url), 'utf-8');
|
||||
|
||||
test('worker exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
|
||||
expect(workerSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
|
||||
});
|
||||
|
||||
test('worker arms its abort timer from timeout_at when present (one absolute deadline)', () => {
|
||||
expect(workerSrc).toContain('job.timeout_at.getTime() - Date.now()');
|
||||
});
|
||||
|
||||
test('autopilot-cycle, global-maintenance AND phase-wrapper handlers thread deadlineAtMs into runCycle', () => {
|
||||
const matches = jobsSrc.match(/deadlineAtMs: job\.deadlineAtMs/g) ?? [];
|
||||
expect(matches.length).toBe(3);
|
||||
});
|
||||
|
||||
test('runCycle forwards deadlineAtMs to the patterns phase', () => {
|
||||
expect(cycleSrc).toContain('deadlineAtMs: opts.deadlineAtMs ?? null');
|
||||
});
|
||||
|
||||
test('patterns submits + waits with the CLAMPED budgets, not raw config', () => {
|
||||
expect(patternsSrc).toContain('timeout_ms: budgets.timeoutMs');
|
||||
expect(patternsSrc).toContain('timeoutMs: budgets.waitTimeoutMs');
|
||||
expect(patternsSrc).not.toContain('timeout_ms: config.subagentTimeoutMs');
|
||||
expect(patternsSrc).not.toContain('timeoutMs: config.subagentWaitTimeoutMs');
|
||||
});
|
||||
|
||||
test('patterns cancels the child on wait timeout (child clock starts at ITS claim)', () => {
|
||||
// A child that sat queued can outlive the parent deadline the wait was
|
||||
// clamped to; the timeout path must strip it so it can't keep spending.
|
||||
expect(patternsSrc).toContain('queue.cancelJob(job.id)');
|
||||
});
|
||||
|
||||
test('patterns skips honestly when the remaining budget is too small', () => {
|
||||
expect(patternsSrc).toContain('insufficient_cycle_budget');
|
||||
// Budget gate sits AFTER the provider probe so a no-provider brain
|
||||
// still reports no_provider (cheaper, more actionable reason).
|
||||
const probeIdx = patternsSrc.indexOf("skipped('no_provider'");
|
||||
// lastIndexOf: the doc comment on MIN_PATTERNS_SUBAGENT_BUDGET_MS
|
||||
// mentions the reason string too; the CALL SITE is the later hit.
|
||||
const budgetIdx = patternsSrc.lastIndexOf('insufficient_cycle_budget');
|
||||
expect(probeIdx).toBeGreaterThan(0);
|
||||
expect(budgetIdx).toBeGreaterThan(probeIdx);
|
||||
});
|
||||
});
|
||||
@@ -74,8 +74,12 @@ describe('patterns phase wiring', () => {
|
||||
});
|
||||
|
||||
describe('patterns scope filter', () => {
|
||||
test('filters reflections by slug LIKE wiki/personal/reflections/%', () => {
|
||||
expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'");
|
||||
test('filters reflections by slug LIKE <output_root>/personal/reflections/%', () => {
|
||||
// #2415: the namespace root is configurable (dream.synthesize.output_root,
|
||||
// default 'wiki') and bound as a parameter — the scope filter itself and
|
||||
// the reflections sub-path stay pinned.
|
||||
expect(patternsSrc).toContain('slug LIKE $2');
|
||||
expect(patternsSrc).toContain('/personal/reflections/%');
|
||||
});
|
||||
|
||||
test('orders by updated_at DESC for recency-bias', () => {
|
||||
|
||||
@@ -21,7 +21,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { __testing } from '../src/core/cycle/synthesize.ts';
|
||||
|
||||
const { collectChildPutPageSlugs } = __testing;
|
||||
const { collectChildPutPageSlugs, stampDreamProvenance } = __testing;
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
@@ -103,4 +103,52 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', ()
|
||||
// Function silently drops rows whose slug resolves to null/empty.
|
||||
expect(refs.map((r: { slug: string }) => r.slug)).not.toContain('no-slug');
|
||||
});
|
||||
|
||||
// #1586: refs are stamped with the cycle's resolved source, not a
|
||||
// hardcoded 'default'.
|
||||
test('stamps refs with the provided cycle sourceId (#1586)', async () => {
|
||||
const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map(), 'mybrain');
|
||||
expect(refs.length).toBeGreaterThan(0);
|
||||
for (const r of refs) expect(r.source_id).toBe('mybrain');
|
||||
});
|
||||
|
||||
test('defaults to source_id=default when no sourceId is passed (legacy)', async () => {
|
||||
const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map());
|
||||
expect(refs.length).toBeGreaterThan(0);
|
||||
for (const r of refs) expect(r.source_id).toBe('default');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2569: stampDreamProvenance persists the marker into DB frontmatter', () => {
|
||||
test('merges dream_generated + dream_cycle_date into pages.frontmatter', async () => {
|
||||
await engine.putPage('wiki/originals/ideas/2026-07-17-stamp-me-abc123', {
|
||||
type: 'note',
|
||||
title: 'Stamp me',
|
||||
compiled_truth: 'body',
|
||||
timeline: '',
|
||||
frontmatter: { keep_me: 'yes' },
|
||||
});
|
||||
await stampDreamProvenance(
|
||||
engine as any,
|
||||
[{ slug: 'wiki/originals/ideas/2026-07-17-stamp-me-abc123', source_id: 'default' }],
|
||||
'2026-07-17',
|
||||
);
|
||||
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
|
||||
`SELECT frontmatter AS fm FROM pages WHERE slug = 'wiki/originals/ideas/2026-07-17-stamp-me-abc123'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
const fm = rows[0].fm as Record<string, unknown>;
|
||||
// The stamp lands as real JSONB values (queryable via ->>), not a
|
||||
// double-encoded string scalar.
|
||||
expect(fm.dream_generated).toBe(true);
|
||||
expect(fm.dream_cycle_date).toBe('2026-07-17');
|
||||
// Merge, not replace: pre-existing frontmatter keys survive.
|
||||
expect(fm.keep_me).toBe('yes');
|
||||
});
|
||||
|
||||
test('is idempotent and never throws for a missing page', async () => {
|
||||
const refs = [{ slug: 'wiki/originals/ideas/does-not-exist', source_id: 'default' }];
|
||||
await stampDreamProvenance(engine as any, refs, '2026-07-17'); // no throw
|
||||
await stampDreamProvenance(engine as any, refs, '2026-07-17'); // idempotent
|
||||
});
|
||||
});
|
||||
|
||||
@@ -316,4 +316,32 @@ describe('v0.41 T6: runPhaseSynthesizeConcepts via stubbed chat', () => {
|
||||
);
|
||||
expect(rows[0].compiled_truth).toContain('Custom synthesized narrative');
|
||||
});
|
||||
|
||||
// #2163: concept pages must enter the retrieval surface. The write routes
|
||||
// through importFromContent (the same parse→chunk pipeline put_page uses),
|
||||
// so content_chunks rows exist and source-boost's 1.3× 'concepts/' weight
|
||||
// has something to boost. (Embeddings are skipped in this env — no
|
||||
// provider — but chunks + search_vector land regardless.)
|
||||
test('concept pages are chunked (#2163)', async () => {
|
||||
const atoms = Array.from({ length: 12 }, (_, i) => ({
|
||||
slug: `c${i}`,
|
||||
title: `Chunk atom ${i}`,
|
||||
body: `Chunky body ${i}.`,
|
||||
concept_refs: ['chunked-concept'],
|
||||
}));
|
||||
const chat = stubChat('A concept narrative long enough to produce at least one chunk.');
|
||||
await runPhaseSynthesizeConcepts(engine, { _atoms: atoms, _chat: chat });
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n
|
||||
FROM content_chunks c JOIN pages p ON p.id = c.page_id
|
||||
WHERE p.slug = 'concepts/chunked-concept'`,
|
||||
);
|
||||
expect(Number(rows[0].n)).toBeGreaterThan(0);
|
||||
// Page metadata survives the importFromContent round-trip.
|
||||
const page = await engine.executeRaw<{ type: string; fm: Record<string, unknown> }>(
|
||||
`SELECT type, frontmatter AS fm FROM pages WHERE slug = 'concepts/chunked-concept'`,
|
||||
);
|
||||
expect(page[0].type).toBe('concept');
|
||||
expect((page[0].fm as Record<string, unknown>).tier).toBe('T1');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* Postgres-only regression for addCodeEdges jsonb encoding (#2968).
|
||||
*
|
||||
* Bun SQL mis-encodes `::jsonb[]` array binds: each element arrives as a
|
||||
* double-encoded JSON string (jsonb_typeof = 'string'), not an object. The
|
||||
* symbol resolver's `edge_metadata || jsonb_build_object(...)` UPDATE then
|
||||
* concatenates onto a string scalar and produces a jsonb array, so
|
||||
* resolved_chunk_id never lands and code_callers/code_callees return nothing.
|
||||
*
|
||||
* PGLite cannot reproduce this class (its addCodeEdges always used per-row
|
||||
* placeholders), so this is DATABASE_URL-gated per the engine-parity
|
||||
* convention. Pins the per-row `$n::text::jsonb` shape: every inserted
|
||||
* edge_metadata must be jsonb_typeof = 'object' and round-trip its fields.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { setupDB, teardownDB, hasDatabase } from './helpers.ts';
|
||||
import type { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeIfDB = skip ? describe.skip : describe;
|
||||
|
||||
let engine: PostgresEngine;
|
||||
let chunkA: number;
|
||||
let chunkB: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (skip) return;
|
||||
engine = await setupDB();
|
||||
|
||||
await engine.putPage('src-a-ts', {
|
||||
type: 'code', page_kind: 'code',
|
||||
title: 'src/a.ts (typescript)',
|
||||
compiled_truth: 'export function run() { return helper(); }',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('src-a-ts', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'export function run() { return helper(); }',
|
||||
chunk_source: 'compiled_truth',
|
||||
language: 'typescript',
|
||||
symbol_name: 'run',
|
||||
symbol_type: 'function',
|
||||
symbol_name_qualified: 'run',
|
||||
}]);
|
||||
|
||||
await engine.putPage('src-b-ts', {
|
||||
type: 'code', page_kind: 'code',
|
||||
title: 'src/b.ts (typescript)',
|
||||
compiled_truth: 'export function helper() { return 1; }',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('src-b-ts', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'export function helper() { return 1; }',
|
||||
chunk_source: 'compiled_truth',
|
||||
language: 'typescript',
|
||||
symbol_name: 'helper',
|
||||
symbol_type: 'function',
|
||||
symbol_name_qualified: 'helper',
|
||||
}]);
|
||||
|
||||
chunkA = (await engine.getChunks('src-a-ts'))[0]!.id;
|
||||
chunkB = (await engine.getChunks('src-b-ts'))[0]!.id;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (skip) return;
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
describeIfDB('addCodeEdges jsonb encoding — Postgres regression (#2968)', () => {
|
||||
test('resolved edges land as jsonb objects, not double-encoded strings', async () => {
|
||||
const inserted = await engine.addCodeEdges([{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: chunkB,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'helper',
|
||||
edge_type: 'calls',
|
||||
edge_metadata: { line: 1, via: 'direct' },
|
||||
}]);
|
||||
expect(inserted).toBe(1);
|
||||
|
||||
const rows = await engine.executeRaw<{ kind: string; line: string | null }>(
|
||||
`SELECT jsonb_typeof(edge_metadata) AS kind, edge_metadata->>'line' AS line
|
||||
FROM code_edges_chunk
|
||||
WHERE from_chunk_id = $1 AND to_chunk_id = $2 AND edge_type = 'calls'`,
|
||||
[chunkA, chunkB],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0]!.kind).toBe('object');
|
||||
expect(rows[0]!.line).toBe('1');
|
||||
});
|
||||
|
||||
test('unresolved edges land as jsonb objects (empty metadata defaults to {})', async () => {
|
||||
const inserted = await engine.addCodeEdges([
|
||||
{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: null,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'phantom',
|
||||
edge_type: 'calls',
|
||||
edge_metadata: { line: 2 },
|
||||
},
|
||||
{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: null,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'ghost',
|
||||
edge_type: 'calls',
|
||||
},
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
|
||||
const rows = await engine.executeRaw<{ to_symbol_qualified: string; kind: string }>(
|
||||
`SELECT to_symbol_qualified, jsonb_typeof(edge_metadata) AS kind
|
||||
FROM code_edges_symbol
|
||||
WHERE from_chunk_id = $1`,
|
||||
[chunkA],
|
||||
);
|
||||
expect(rows.length).toBe(2);
|
||||
for (const row of rows) {
|
||||
expect(row.kind).toBe('object');
|
||||
}
|
||||
});
|
||||
|
||||
test('resolver-style || UPDATE keeps object shape (the corruption symptom)', async () => {
|
||||
// The production resolver runs `edge_metadata || jsonb_build_object(...)`.
|
||||
// On a double-encoded string scalar this yields a jsonb ARRAY and the
|
||||
// resolved_chunk_id key never becomes readable. Pin the healthy path.
|
||||
await engine.executeRaw(
|
||||
`UPDATE code_edges_symbol
|
||||
SET edge_metadata = edge_metadata || jsonb_build_object('resolved_chunk_id', $1::int)
|
||||
WHERE from_chunk_id = $2 AND to_symbol_qualified = 'phantom'`,
|
||||
[chunkB, chunkA],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ kind: string; resolved: string | null }>(
|
||||
`SELECT jsonb_typeof(edge_metadata) AS kind,
|
||||
edge_metadata->>'resolved_chunk_id' AS resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE from_chunk_id = $1 AND to_symbol_qualified = 'phantom'`,
|
||||
[chunkA],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0]!.kind).toBe('object');
|
||||
expect(rows[0]!.resolved).toBe(String(chunkB));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* Unit tests for the setupDB production guard (assertSafeE2eDatabaseUrl).
|
||||
* Pure — no database connection; runs with or without DATABASE_URL set.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { assertSafeE2eDatabaseUrl } from './helpers.ts';
|
||||
|
||||
const NO_ENV = {} as Record<string, string | undefined>;
|
||||
|
||||
describe('assertSafeE2eDatabaseUrl', () => {
|
||||
test('allows the canonical CI test database', () => {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl('postgresql://postgres:postgres@localhost:5433/gbrain_test', NO_ENV),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('allows test as any word segment', () => {
|
||||
for (const name of ['test', 'test_gbrain', 'e2e-test', 'gbrain_test_2', 'TEST_DB']) {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl(`postgresql://u:p@localhost:5432/${name}`, NO_ENV),
|
||||
).not.toThrow();
|
||||
}
|
||||
});
|
||||
|
||||
test('refuses production-looking database names', () => {
|
||||
for (const name of ['gbrain', 'postgres', 'prod', 'gbrain_live', 'contest', 'latest']) {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl(`postgresql://u:p@localhost:5432/${name}`, NO_ENV),
|
||||
).toThrow(/does not look like a test database/);
|
||||
}
|
||||
});
|
||||
|
||||
test('refuses a Supabase-style pooler URL with a bare postgres db', () => {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl(
|
||||
'postgresql://postgres.ref:pw@aws-0-us-east-1.pooler.supabase.com:6543/postgres',
|
||||
NO_ENV,
|
||||
),
|
||||
).toThrow(/does not look like a test database/);
|
||||
});
|
||||
|
||||
test('explicit exact-name override opts a non-test database in', () => {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl('postgresql://u:p@localhost:5432/gbrain', {
|
||||
GBRAIN_E2E_ALLOW_DB: 'gbrain',
|
||||
}),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('override must match the exact database name', () => {
|
||||
expect(() =>
|
||||
assertSafeE2eDatabaseUrl('postgresql://u:p@localhost:5432/gbrain', {
|
||||
GBRAIN_E2E_ALLOW_DB: 'other_db',
|
||||
}),
|
||||
).toThrow(/does not look like a test database/);
|
||||
});
|
||||
|
||||
test('refuses unparseable URLs and missing database names', () => {
|
||||
expect(() => assertSafeE2eDatabaseUrl('not a url', NO_ENV)).toThrow(/not a parseable URL/);
|
||||
expect(() => assertSafeE2eDatabaseUrl('postgresql://u:p@localhost:5432/', NO_ENV)).toThrow(
|
||||
/no database name/,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -225,6 +225,66 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
expect(pgChanged || pgliteChanged).toBe(true);
|
||||
});
|
||||
|
||||
// fix/title-retrieval-arm (Reviewer F2): the title arm must behave
|
||||
// identically on both engines — including the D1 case where the title
|
||||
// tokens never appear in any chunk. Without this case the Postgres
|
||||
// implementation would only ever execute behind hybridSearch's fail-open
|
||||
// catch and a break could ship dark on the production brain. Runs in CI
|
||||
// via scripts/run-e2e.sh (docker-provisioned Postgres); skips gracefully
|
||||
// when DATABASE_URL is not configured.
|
||||
test('searchTitles parity: exact-title hit with title tokens absent from body', async () => {
|
||||
const seed = async (eng: BrainEngine) => {
|
||||
await eng.putPage('wiki/title-arm-parity', {
|
||||
type: 'note',
|
||||
title: 'Vermilion Icebreaker Compendium',
|
||||
compiled_truth: 'A document body that never mentions those words.',
|
||||
timeline: '',
|
||||
});
|
||||
await eng.upsertChunks('wiki/title-arm-parity', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'A document body that never mentions those words.',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(33),
|
||||
token_count: 9,
|
||||
}] satisfies ChunkInput[]);
|
||||
};
|
||||
await seed(pgEngine);
|
||||
await seed(pgliteEngine);
|
||||
|
||||
const q = 'Vermilion Icebreaker Compendium';
|
||||
// Premise on both engines: chunk-grain keyword cannot see the page
|
||||
// (also pins the F1 contract — no orFallback flag means strict AND).
|
||||
expect((await pgEngine.searchKeyword(q, { limit: 5 })).map((r: SearchResult) => r.slug))
|
||||
.not.toContain('wiki/title-arm-parity');
|
||||
expect((await pgliteEngine.searchKeyword(q, { limit: 5 })).map((r: SearchResult) => r.slug))
|
||||
.not.toContain('wiki/title-arm-parity');
|
||||
|
||||
const pg = await pgEngine.searchTitles(q, { limit: 5 });
|
||||
const pglite = await pgliteEngine.searchTitles(q, { limit: 5 });
|
||||
expect(pg.map((r: SearchResult) => r.slug)).toContain('wiki/title-arm-parity');
|
||||
expect(pglite.map((r: SearchResult) => r.slug)).toContain('wiki/title-arm-parity');
|
||||
|
||||
// Row-shape parity: identical representative chunk on both engines.
|
||||
const pgHit = pg.find((r: SearchResult) => r.slug === 'wiki/title-arm-parity')!;
|
||||
const pgliteHit = pglite.find((r: SearchResult) => r.slug === 'wiki/title-arm-parity')!;
|
||||
expect(pgHit.chunk_source).toBe('compiled_truth');
|
||||
expect(pgliteHit.chunk_source).toBe(pgHit.chunk_source);
|
||||
expect(pgliteHit.chunk_text).toBe(pgHit.chunk_text);
|
||||
});
|
||||
|
||||
// fix/title-retrieval-arm (Reviewer F1): the AND→OR fallback is opt-in.
|
||||
// Default searchKeyword stays strict on BOTH engines; orFallback: true
|
||||
// rescues the one-bad-token query identically.
|
||||
test('searchKeyword orFallback parity: default strict, opt-in rescues', async () => {
|
||||
const q = 'fat code thin harness zzzabsenttoken';
|
||||
for (const eng of [pgEngine, pgliteEngine]) {
|
||||
const strict = await eng.searchKeyword(q, { limit: 5 });
|
||||
expect(strict.length).toBe(0);
|
||||
const relaxed = await eng.searchKeyword(q, { limit: 5, orFallback: true });
|
||||
expect(relaxed.map((r: SearchResult) => r.slug)).toContain('concepts/fat-code-thin-harness');
|
||||
}
|
||||
});
|
||||
|
||||
// v0.39.3.0 T3 — provenance write+read parity (WARN-8 + CV5).
|
||||
// Both engines must write the same 4 provenance columns (source_kind,
|
||||
// source_uri, ingested_via, ingested_at) on putPage AND surface them
|
||||
|
||||
@@ -66,6 +66,40 @@ export function hasDatabase(): boolean {
|
||||
return !!DATABASE_URL;
|
||||
}
|
||||
|
||||
/**
|
||||
* Production guard: setupDB() TRUNCATEs every data table on whatever
|
||||
* DATABASE_URL points at, and run-e2e.sh deliberately preserves an exported
|
||||
* DATABASE_URL — so a developer with a production URL in their environment
|
||||
* would wipe their real brain by running the suite. Refuse unless the
|
||||
* database name identifies itself as a test database ("test" as a word
|
||||
* segment, e.g. gbrain_test — the CI/.env.testing.example convention), or
|
||||
* the operator explicitly opts the exact name in via GBRAIN_E2E_ALLOW_DB.
|
||||
*
|
||||
* Exported for unit testing; pure — no connection is made.
|
||||
*/
|
||||
export function assertSafeE2eDatabaseUrl(
|
||||
url: string,
|
||||
env: Record<string, string | undefined> = process.env,
|
||||
): void {
|
||||
let dbName: string;
|
||||
try {
|
||||
dbName = decodeURIComponent(new URL(url).pathname.replace(/^\//, ''));
|
||||
} catch {
|
||||
throw new Error(`E2E guard: DATABASE_URL is not a parseable URL; refusing to run destructive setup.`);
|
||||
}
|
||||
if (!dbName) {
|
||||
throw new Error(`E2E guard: DATABASE_URL has no database name; refusing to run destructive setup.`);
|
||||
}
|
||||
if (/(^|[_-])test([_-]|$)/i.test(dbName)) return;
|
||||
if (env.GBRAIN_E2E_ALLOW_DB && env.GBRAIN_E2E_ALLOW_DB === dbName) return;
|
||||
throw new Error(
|
||||
`E2E guard: database "${dbName}" does not look like a test database ` +
|
||||
`(expected "test" as a name segment, e.g. gbrain_test). setupDB() would ` +
|
||||
`TRUNCATE every data table in it. If this is intentional, set ` +
|
||||
`GBRAIN_E2E_ALLOW_DB=${dbName} to opt in explicitly.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to DB, run schema init, truncate all tables.
|
||||
* Call in beforeAll() of each test file.
|
||||
@@ -74,6 +108,7 @@ export async function setupDB(): Promise<PostgresEngine> {
|
||||
if (!DATABASE_URL) {
|
||||
throw new Error('DATABASE_URL not set. Copy .env.testing.example to .env.testing and configure it.');
|
||||
}
|
||||
assertSafeE2eDatabaseUrl(DATABASE_URL);
|
||||
|
||||
// Disconnect any prior connection (clean slate)
|
||||
await db.disconnect();
|
||||
|
||||
@@ -49,6 +49,7 @@ function makeFakeJobCtx(data: Record<string, unknown>): MinionJobContext {
|
||||
data,
|
||||
attempts_made: 1,
|
||||
signal: new AbortController().signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: new AbortController().signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
|
||||
@@ -279,6 +279,7 @@ async function makeCrashedCtx(jobId: number, prompt: string, modelId: string): P
|
||||
data: { prompt, model: modelId },
|
||||
attempts_made: 1, // crashed once
|
||||
signal: abortCtrl.signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: shutdownCtrl.signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
|
||||
@@ -92,6 +92,7 @@ async function makeFakeJob(opts: FakeJobOpts): Promise<{ jobId: number; ctx: Min
|
||||
data: { prompt: opts.prompt, model: opts.model, allowed_tools: opts.allowed_tools },
|
||||
attempts_made: 0,
|
||||
signal: abortCtrl.signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: shutdownCtrl.signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async (t) => { tokenSink.push(t); },
|
||||
|
||||
@@ -68,7 +68,7 @@ async function makeJob(prompt: string, model: string): Promise<{ jobId: number;
|
||||
const jobId = rows[0].id;
|
||||
const ctx: MinionJobContext = {
|
||||
id: jobId, name: 'subagent', data: { prompt, model }, attempts_made: 1,
|
||||
signal: new AbortController().signal, shutdownSignal: new AbortController().signal,
|
||||
signal: new AbortController().signal, deadlineAtMs: null, shutdownSignal: new AbortController().signal,
|
||||
updateProgress: async () => {}, updateTokens: async () => {}, log: async () => {},
|
||||
isActive: async () => true, readInbox: async () => [],
|
||||
};
|
||||
|
||||
@@ -224,7 +224,7 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
|
||||
expect(bob).toBeNull();
|
||||
});
|
||||
|
||||
test('sync skips non-syncable files (README, hidden, .raw)', async () => {
|
||||
test('sync skips non-syncable files (README, hidden, .raw) but imports ops/ (#2404)', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
@@ -249,8 +249,9 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
|
||||
const raw = await engine.getPage('.raw/data');
|
||||
expect(raw).toBeNull();
|
||||
|
||||
// ops/ is ordinary content and DOES sync (#2404).
|
||||
const ops = await engine.getPage('ops/deploy');
|
||||
expect(ops).toBeNull();
|
||||
expect(ops).not.toBeNull();
|
||||
});
|
||||
|
||||
test('sync stores last_commit and last_run in config', async () => {
|
||||
|
||||
@@ -306,6 +306,7 @@ describe('runExtractConversationFactsCore', () => {
|
||||
// truncation semantics than the canonical reset helper.
|
||||
await engine.executeRaw(`DELETE FROM facts WHERE source LIKE 'cli:extract-conversation-facts%'`);
|
||||
await engine.executeRaw(`DELETE FROM op_checkpoints WHERE op = 'extract-conversation-facts'`);
|
||||
await engine.executeRaw(`DELETE FROM extract_rollup_7d`);
|
||||
await engine.executeRaw(`DELETE FROM pages WHERE slug LIKE 'conversations/%' OR slug LIKE 'people/alice%'`);
|
||||
// Set facts.extraction_enabled=true so kill-switch doesn't refuse.
|
||||
await engine.setConfig('facts.extraction_enabled', 'true');
|
||||
@@ -365,6 +366,21 @@ describe('runExtractConversationFactsCore', () => {
|
||||
expect(result.segments_processed).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('dry-run does not write the extract_rollup_7d cache row', async () => {
|
||||
// Regression: --dry-run promises "no DB writes" but writeRunReceiptAndRollup
|
||||
// upsert-ed extract_rollup_7d unconditionally. A preview must not mutate the DB.
|
||||
await runExtractConversationFactsCore(engine, {
|
||||
sourceId: 'default',
|
||||
slug: 'conversations/imessage/alice-example',
|
||||
dryRun: true,
|
||||
sleepMs: 0,
|
||||
});
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM extract_rollup_7d WHERE kind = 'facts.conversation' AND source_id = 'default'`,
|
||||
);
|
||||
expect(Number(rows[0]?.count ?? 0)).toBe(0);
|
||||
});
|
||||
|
||||
test('non-conversation pages are skipped', async () => {
|
||||
const result = await runExtractConversationFactsCore(engine, {
|
||||
sourceId: 'default',
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* Takes-extraction model resolution regression (#2997).
|
||||
*
|
||||
* extractTakesFromPages hardcoded `anthropic:claude-haiku-4-5` as the
|
||||
* classifier model. On OAuth/local-only installs (no ANTHROPIC_API_KEY;
|
||||
* chat routed through a gateway model) every extraction died with
|
||||
* llm_unavailable even though a working chat_model was configured.
|
||||
*
|
||||
* Pins the fix's resolution order AND its config plane:
|
||||
* opts.model → getChatModel() (file-plane gateway config, the enrich.ts
|
||||
* idiom) — NOT the DB config plane (engine.getConfig('chat_model')).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setChatTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { extractTakesFromPages } from '../src/core/extract-takes-from-pages.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const seenModels: string[] = [];
|
||||
let pageN = 0;
|
||||
|
||||
/** Each test seeds a fresh uncovered page so the extraction loop fires. */
|
||||
async function seedPage(): Promise<void> {
|
||||
const body = 'An opinion-bearing body long enough to clear the 200-char eligibility floor. '.repeat(5);
|
||||
await engine.putPage(`concepts/model-resolution-${pageN++}`, {
|
||||
type: 'concept', title: `M${pageN}`, compiled_truth: body, frontmatter: {},
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
__setChatTransportForTests(async (opts) => {
|
||||
seenModels.push(opts.model ?? '(unset)');
|
||||
return {
|
||||
text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]',
|
||||
blocks: [{ type: 'text' as const, text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]' }],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: opts.model ?? '(unset)',
|
||||
providerId: 'test',
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__setChatTransportForTests(null);
|
||||
resetGateway();
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
seenModels.length = 0;
|
||||
});
|
||||
|
||||
describe('extractTakesFromPages — model resolution (#2997)', () => {
|
||||
test('defaults to the configured chat_model from the file-plane gateway config', async () => {
|
||||
configureGateway({
|
||||
chat_model: 'openai:gpt-config-plane-test',
|
||||
env: { OPENAI_API_KEY: 'sk-test-model-resolution' },
|
||||
});
|
||||
// A conflicting DB-plane value must be IGNORED — model config is the
|
||||
// config-file plane (getChatModel), not the brain DB config table.
|
||||
await engine.setConfig('chat_model', 'wrong:db-plane-model');
|
||||
await seedPage();
|
||||
|
||||
const r = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 });
|
||||
expect(r.pages_scanned).toBe(1);
|
||||
expect(seenModels).toEqual(['openai:gpt-config-plane-test']);
|
||||
});
|
||||
|
||||
test('explicit opts.model wins over the configured chat_model', async () => {
|
||||
configureGateway({
|
||||
chat_model: 'openai:gpt-config-plane-test',
|
||||
env: { OPENAI_API_KEY: 'sk-test-model-resolution' },
|
||||
});
|
||||
await seedPage();
|
||||
|
||||
const r = await extractTakesFromPages(engine, {
|
||||
bootstrapEnabled: true,
|
||||
maxPages: 50,
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
});
|
||||
expect(r.pages_scanned).toBe(1);
|
||||
expect(seenModels).toEqual(['anthropic:claude-haiku-4-5']);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { MIGRATIONS } from '../src/core/migrate.ts';
|
||||
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
@@ -24,9 +24,11 @@ describe('configurable_fts_language migration', () => {
|
||||
expect(ftsMig?.version).toBeGreaterThan(115);
|
||||
});
|
||||
|
||||
test('fts migration is the latest migration', () => {
|
||||
expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION);
|
||||
});
|
||||
// #2704 (v124, page_search_vector_drop_compiled_truth) landed after this
|
||||
// migration — "is the latest migration" was only ever true at the
|
||||
// moment v123 was added and would break on every subsequent migration,
|
||||
// so it's removed rather than bumped to a hardcoded v124. The
|
||||
// registration + shape assertions below don't depend on migration order.
|
||||
|
||||
test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
|
||||
@@ -48,6 +48,7 @@ function fakeJob(data: Record<string, unknown>): MinionJobContext {
|
||||
data,
|
||||
attempts_made: 0,
|
||||
signal: controller.signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: controller.signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
/**
|
||||
* Cache-HIT budget-meta provenance — companion to the miss-path fix.
|
||||
*
|
||||
* With a per-call tokenBudget, the miss path stores an already-budgeted
|
||||
* result set; a subsequent HIT re-applies the same budget to that trimmed
|
||||
* payload (a structural no-op: tokenBudget is folded into knobsHash, so a
|
||||
* hit only ever serves a lookup with the identical resolved budget as the
|
||||
* write) — and pre-fix published that no-op pass's meta, reporting
|
||||
* dropped=0 while the miss that produced the very same result set reported
|
||||
* the real cut. This file drives a real store→hit roundtrip (mocked
|
||||
* `embedQuery` for a deterministic vector, real PGLite SemanticQueryCache)
|
||||
* and pins that the hit's token_budget matches the miss's.
|
||||
*
|
||||
* Serial: mock.module + gateway/global-env mutation (isolation guard R2).
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, describe, expect, mock, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import * as realEmbedding from '../src/core/embedding.ts';
|
||||
|
||||
/** Deterministic 1536d unit vector — identical for every call, so the
|
||||
* second consult matches the first write at cosine 1.0. */
|
||||
function fixedEmbedding(): Float32Array {
|
||||
const arr = new Float32Array(1536);
|
||||
for (let i = 0; i < 1536; i++) arr[i] = Math.sin(1 + i * 0.001);
|
||||
let norm = 0;
|
||||
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
}
|
||||
|
||||
// Mock BEFORE importing hybrid.ts (spread keeps every other export live).
|
||||
mock.module('../src/core/embedding.ts', () => ({
|
||||
...realEmbedding,
|
||||
embed: async () => fixedEmbedding(),
|
||||
embedQuery: async () => fixedEmbedding(),
|
||||
}));
|
||||
|
||||
// Import AFTER mocking.
|
||||
const { hybridSearchCached, awaitPendingSearchCacheWrites } =
|
||||
await import('../src/core/search/hybrid.ts');
|
||||
const { configureGateway, resetGateway } = await import('../src/core/ai/gateway.ts');
|
||||
const { PGLiteEngine } = await import('../src/core/pglite-engine.ts');
|
||||
|
||||
let engine: InstanceType<typeof PGLiteEngine>;
|
||||
let tmpHome: string;
|
||||
const savedGbrainHome = process.env.GBRAIN_HOME;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Hermetic config home so the developer's real ~/.gbrain/config.json
|
||||
// can't leak an embedding_model that flips the cache consult to
|
||||
// 'disabled' via isCacheSafe.
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-hit-budget-meta-'));
|
||||
process.env.GBRAIN_HOME = tmpHome;
|
||||
|
||||
// Pin the gateway to a 1536d provider BEFORE initSchema so the
|
||||
// query_cache.embedding column is sized for the mock vectors. The fake
|
||||
// key is never used — embedQuery is mocked above.
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-fake' },
|
||||
});
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Three keyword-findable pages, ~200 tokens each, mixed types so dedup's
|
||||
// type-diversity layer keeps all of them. putPage never chunks —
|
||||
// searchKeyword joins content_chunks, so chunks are explicit.
|
||||
const longText = 'x'.repeat(800);
|
||||
const fixtures: Array<[string, string, string]> = [
|
||||
['alice-foo', 'Alice Foo', 'person'],
|
||||
['bob-bar', 'Bob Bar', 'company'],
|
||||
['carol-baz', 'Carol Baz', 'note'],
|
||||
];
|
||||
for (const [slug, title, type] of fixtures) {
|
||||
const truth = `${title} is a builder. ${longText}`;
|
||||
await engine.putPage(slug, { type, title, compiled_truth: truth });
|
||||
await engine.upsertChunks(slug, [
|
||||
{ chunk_index: 0, chunk_text: truth, chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
}
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedGbrainHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = savedGbrainHome;
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
resetGateway();
|
||||
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
describe('cache HIT — token_budget provenance', () => {
|
||||
test('hit reports the same cut the miss reported, not the no-op re-application', async () => {
|
||||
// Miss: budget 250 keeps ~1 of 3 rows (~209 tokens each); the meta
|
||||
// carries the real cut from the inner enforcement.
|
||||
let missMeta: import('../src/core/types.ts').HybridSearchMeta | undefined;
|
||||
const missResults = await hybridSearchCached(engine, 'builder', {
|
||||
limit: 10,
|
||||
tokenBudget: 250,
|
||||
onMeta: (m) => { missMeta = m; },
|
||||
});
|
||||
expect(missResults.length).toBeGreaterThan(0);
|
||||
expect(missMeta?.cache?.status).toBe('miss');
|
||||
expect(missMeta?.token_budget?.budget).toBe(250);
|
||||
const missDropped = missMeta?.token_budget?.dropped;
|
||||
expect(missDropped).toBeGreaterThan(0);
|
||||
|
||||
await awaitPendingSearchCacheWrites();
|
||||
|
||||
// Hit: identical query + knobs (tokenBudget is part of knobsHash, so
|
||||
// this is the ONLY kind of lookup the stored row can serve). The
|
||||
// published budget record must match the miss's — pre-fix it was the
|
||||
// outer no-op pass's meta with dropped=0.
|
||||
let hitMeta: import('../src/core/types.ts').HybridSearchMeta | undefined;
|
||||
const hitResults = await hybridSearchCached(engine, 'builder', {
|
||||
limit: 10,
|
||||
tokenBudget: 250,
|
||||
onMeta: (m) => { hitMeta = m; },
|
||||
});
|
||||
expect(hitMeta?.cache?.status).toBe('hit');
|
||||
expect(hitResults.length).toBe(missResults.length);
|
||||
expect(hitMeta?.token_budget?.budget).toBe(250);
|
||||
expect(hitMeta?.token_budget?.dropped).toBe(missDropped);
|
||||
expect(hitMeta?.token_budget?.kept).toBe(missMeta?.token_budget?.kept);
|
||||
});
|
||||
});
|
||||
@@ -41,7 +41,11 @@ beforeAll(async () => {
|
||||
{
|
||||
slug: 'bob-bar',
|
||||
page: {
|
||||
type: 'person',
|
||||
// Mixed types across the fixture keep dedup Layer 3 (no page type
|
||||
// above 60% of results) out of this test's way — an all-person set
|
||||
// would be capped to 2 of 3 and couple these assertions to the
|
||||
// diversity policy.
|
||||
type: 'company',
|
||||
title: 'Bob Bar',
|
||||
compiled_truth: `Bob Bar is a builder. ${longText}`,
|
||||
},
|
||||
@@ -49,7 +53,7 @@ beforeAll(async () => {
|
||||
{
|
||||
slug: 'carol-baz',
|
||||
page: {
|
||||
type: 'person',
|
||||
type: 'note',
|
||||
title: 'Carol Baz',
|
||||
compiled_truth: `Carol Baz is a builder. ${longText}`,
|
||||
},
|
||||
@@ -57,6 +61,13 @@ beforeAll(async () => {
|
||||
];
|
||||
for (const p of pages) {
|
||||
await engine.putPage(p.slug, p.page);
|
||||
// putPage never chunks — searchKeyword joins content_chunks, so a
|
||||
// page without explicit chunks is invisible to the keyword arm and
|
||||
// every result-dependent assertion below runs against an empty set.
|
||||
// (Pattern: test/chunk-grain-fts.test.ts.)
|
||||
await engine.upsertChunks(p.slug, [
|
||||
{ chunk_index: 0, chunk_text: p.page.compiled_truth!, chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
}
|
||||
// Force keyword-only fallback by unsetting the embedding provider key.
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
@@ -103,10 +114,10 @@ describe('hybridSearchCached \u2014 token budget', () => {
|
||||
limit: 10,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// Don't assert non-empty here — keyword tokenization depends on the
|
||||
// pglite analyzer config. What matters: meta is shaped right and
|
||||
// budget metadata is absent when budget isn't set.
|
||||
expect(results).toBeDefined();
|
||||
// Non-empty matters: pre-fix the fixture had no chunks, so this ran
|
||||
// against an empty result set and the absent-budget assertion was
|
||||
// trivially true.
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
expect(meta?.token_budget).toBeUndefined();
|
||||
});
|
||||
|
||||
@@ -117,19 +128,21 @@ describe('hybridSearchCached \u2014 token budget', () => {
|
||||
tokenBudget: 250,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
expect(meta?.token_budget).toBeDefined();
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
});
|
||||
|
||||
test('tight budget cuts the result set', async () => {
|
||||
// First find out the result count without a budget so the assertion
|
||||
// is robust to the fixture’s actual chunking.
|
||||
// All three fixture pages match 'builder' (mixed types, so dedup's
|
||||
// type-diversity layer keeps all of them), and the unbounded set MUST
|
||||
// have enough rows for the cut to be observable. Pre-fix this was a
|
||||
// silent `return` when fewer than 2 rows came back — and with no
|
||||
// chunks in the fixture, zero rows ALWAYS came back, so the cut
|
||||
// assertions below had never executed anywhere.
|
||||
const unbounded = await hybridSearchCached(engine, 'builder', { limit: 10 });
|
||||
// Skip the cut test if the fixture happens to return only one row
|
||||
// (keyword search may dedupe by page); the budget enforcement itself
|
||||
// is exhaustively unit-tested in test/token-budget.test.ts.
|
||||
if (unbounded.length < 2) return;
|
||||
expect(unbounded.length).toBeGreaterThanOrEqual(2);
|
||||
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'builder', {
|
||||
@@ -137,10 +150,16 @@ describe('hybridSearchCached \u2014 token budget', () => {
|
||||
tokenBudget: 250, // enough for ~1 row of fixture data
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
expect(results.length).toBeLessThan(unbounded.length);
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
expect(meta?.token_budget?.dropped).toBeGreaterThan(0);
|
||||
// The budget must hold: cumulative cost <= budget.
|
||||
// Exact accounting: every row the budget removed is a reported drop —
|
||||
// dropped > 0 alone would accept any wrong positive count (codex).
|
||||
expect(meta?.token_budget?.dropped).toBe(unbounded.length - results.length);
|
||||
// The budget must hold with a real (non-zero) cost: cumulative cost
|
||||
// <= budget, and used=0 would mean the accounting never ran.
|
||||
expect(meta?.token_budget?.used).toBeGreaterThan(0);
|
||||
expect(meta?.token_budget?.used).toBeLessThanOrEqual(250);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
/**
|
||||
* #2607 — the `sync --full` git fast path applies the same prune gate as
|
||||
* incremental sync.
|
||||
*
|
||||
* Bug class: `collectSyncableFiles` on a git work tree takes the
|
||||
* `git ls-files` fast path, which historically filtered ONLY by
|
||||
* strategy/extension + .gitignore — no `pruneDir`, so `sync --full`
|
||||
* imported (and resurrected previously-soft-deleted) pages under dot-dirs
|
||||
* and vendored trees that incremental sync's `isSyncable` excludes. The two
|
||||
* enumeration modes cycled content in and out depending on which ran last.
|
||||
*
|
||||
* Fix: `isCollectibleForWalker` (shared by the git fast path AND the FS-walk
|
||||
* emit filter) now rejects any path with a segment `pruneDir` would block —
|
||||
* the same segment rule `classifySync` applies on the incremental path.
|
||||
*
|
||||
* No PGLite needed: `collectSyncableFiles` is pure filesystem + git.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join, relative } from 'path';
|
||||
import { collectSyncableFiles } from '../src/commands/import.ts';
|
||||
import { isSyncable } from '../src/core/sync.ts';
|
||||
|
||||
let repo: string;
|
||||
|
||||
function rel(files: string[]): string[] {
|
||||
return files.map((f) => relative(repo, f));
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
repo = mkdtempSync(join(tmpdir(), 'gbrain-fastpath-'));
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
// Ordinary content — must be collected.
|
||||
mkdirSync(join(repo, 'notes'), { recursive: true });
|
||||
writeFileSync(join(repo, 'notes/real.md'), '---\ntitle: Real\n---\nbody\n');
|
||||
mkdirSync(join(repo, 'ops'), { recursive: true });
|
||||
writeFileSync(join(repo, 'ops/tasks.md'), '---\ntitle: Tasks\n---\nbody\n');
|
||||
|
||||
// TRACKED files under excluded trees — `git ls-files` returns these, so
|
||||
// only the prune gate keeps them out (this is the #2607 divergence).
|
||||
mkdirSync(join(repo, '.obsidian'), { recursive: true });
|
||||
writeFileSync(join(repo, '.obsidian/plugin-notes.md'), 'not a page\n');
|
||||
mkdirSync(join(repo, 'vendor/pkg'), { recursive: true });
|
||||
writeFileSync(join(repo, 'vendor/pkg/notes.md'), 'vendored\n');
|
||||
mkdirSync(join(repo, 'node_modules/dep'), { recursive: true });
|
||||
writeFileSync(join(repo, 'node_modules/dep/CHANGELOG.md'), 'dep changelog\n');
|
||||
mkdirSync(join(repo, 'people/pedro.raw'), { recursive: true });
|
||||
writeFileSync(join(repo, 'people/pedro.raw/source.md'), 'raw sidecar\n');
|
||||
|
||||
// Metafiles — excluded on both routes (pre-existing #345 behavior).
|
||||
writeFileSync(join(repo, 'README.md'), '# repo\n');
|
||||
writeFileSync(join(repo, 'notes/index.md'), '# index\n');
|
||||
|
||||
execSync('git add -A -f && git commit -m "fixture"', { cwd: repo, stdio: 'pipe' });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
if (repo) rmSync(repo, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('#2607 — git fast path excludes what incremental sync excludes', () => {
|
||||
test('tracked files under pruned dirs are NOT collected', () => {
|
||||
const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' }));
|
||||
expect(files).toContain('notes/real.md');
|
||||
expect(files).toContain('ops/tasks.md'); // ordinary content (#2404)
|
||||
expect(files).not.toContain('.obsidian/plugin-notes.md');
|
||||
expect(files).not.toContain('vendor/pkg/notes.md');
|
||||
expect(files).not.toContain('node_modules/dep/CHANGELOG.md');
|
||||
expect(files).not.toContain('people/pedro.raw/source.md');
|
||||
// Metafiles stay excluded too.
|
||||
expect(files).not.toContain('README.md');
|
||||
expect(files).not.toContain('notes/index.md');
|
||||
});
|
||||
|
||||
test('full-sync enumeration agrees with incremental isSyncable for every collected file', () => {
|
||||
// The single-source-of-truth contract: nothing the full path collects may
|
||||
// be something the incremental path would refuse to sync.
|
||||
const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' }));
|
||||
for (const f of files) {
|
||||
expect({ path: f, syncable: isSyncable(f) }).toEqual({ path: f, syncable: true });
|
||||
}
|
||||
expect(files.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -58,6 +58,7 @@ function makeJob(data: Record<string, unknown>): MinionJobContext {
|
||||
data,
|
||||
attempts_made: 1,
|
||||
signal: new AbortController().signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: new AbortController().signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* Migrations must never write to stdout — regression for the heavy-tests
|
||||
* fm_wallclock failure (run 29731426470).
|
||||
*
|
||||
* Migrations run lazily inside ANY command's first DB connect (initSchema →
|
||||
* runMigrations), including JSON-emitting commands like `gbrain doctor --json`.
|
||||
* The v123 FTS migration (#2941) printed its completion notice via
|
||||
* `console.log`, which landed as the first line of `doctor --json` stdout and
|
||||
* broke every jq consumer ("Invalid numeric literal at line 1, column 7").
|
||||
* runMigrations' own contract (see the comment above its progress writes)
|
||||
* routes ALL migration noise to stderr.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runMigrations } from '../src/core/migrate.ts';
|
||||
|
||||
describe('migration output stays off stdout', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('re-running pending migrations (v122 → latest) writes nothing to stdout', async () => {
|
||||
// Rewind the version stamp so the v123 handler actually re-executes —
|
||||
// the exact state a CI Postgres/older brain is in when doctor connects.
|
||||
await engine.setConfig('version', '122');
|
||||
|
||||
const stdoutWrites: string[] = [];
|
||||
const origWrite = process.stdout.write.bind(process.stdout);
|
||||
const origLog = console.log;
|
||||
process.stdout.write = ((chunk: unknown, ...rest: unknown[]) => {
|
||||
stdoutWrites.push(String(chunk));
|
||||
return (origWrite as (...a: unknown[]) => boolean)(chunk, ...rest);
|
||||
}) as typeof process.stdout.write;
|
||||
console.log = (...args: unknown[]) => { stdoutWrites.push(args.map(String).join(' ')); };
|
||||
|
||||
try {
|
||||
const res = await runMigrations(engine);
|
||||
// Load-bearing: the migration must have actually run for the stdout
|
||||
// assertion to prove anything.
|
||||
expect(res.applied).toBeGreaterThanOrEqual(1);
|
||||
} finally {
|
||||
process.stdout.write = origWrite;
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
expect(stdoutWrites).toEqual([]);
|
||||
}, 60000);
|
||||
|
||||
test('migrate.ts contains no console.log (all migration noise goes to stderr)', () => {
|
||||
const src = readFileSync(join(import.meta.dir, '../src/core/migrate.ts'), 'utf8');
|
||||
const offenders = src
|
||||
.split('\n')
|
||||
.map((line, i) => ({ line, n: i + 1 }))
|
||||
.filter(({ line }) => line.includes('console.log('));
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -52,6 +52,7 @@ function makeCtx(
|
||||
data,
|
||||
attempts_made: 0,
|
||||
signal: opts.signal ?? new AbortController().signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: opts.shutdownSignal ?? new AbortController().signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
|
||||
@@ -735,6 +735,99 @@ describe('MinionQueue: Cancel & Retry', () => {
|
||||
expect(retried!.status).toBe('waiting');
|
||||
expect(retried!.error_text).toBeNull();
|
||||
});
|
||||
|
||||
// #2783: retry must reset started_at/attempts_made/attempts_started/
|
||||
// stalled_counter — an explicit `jobs retry` is an operator asserting
|
||||
// "run this fresh".
|
||||
test('retry resets started_at/attempts_made/attempts_started/stalled_counter', async () => {
|
||||
const job = await queue.add('sync', {}, { max_attempts: 3, max_stalled: 3 });
|
||||
await queue.claim('tok1', 30000, 'default', ['sync']);
|
||||
await queue.failJob(job.id, 'tok1', 'error', 'dead');
|
||||
// Simulate the original claim having stamped started_at long ago,
|
||||
// attempts already elevated, and a near-exhausted stall budget —
|
||||
// matching what a real dead job (killed by wall-clock OR by stall
|
||||
// exhaustion) looks like.
|
||||
await engine.executeRaw(
|
||||
"UPDATE minion_jobs SET started_at = now() - interval '1 hour', stalled_counter = 2 WHERE id = $1",
|
||||
[job.id],
|
||||
);
|
||||
const retried = await queue.retryJob(job.id);
|
||||
expect(retried!.status).toBe('waiting');
|
||||
expect(retried!.started_at).toBeNull();
|
||||
expect(retried!.attempts_made).toBe(0);
|
||||
expect(retried!.attempts_started).toBe(0);
|
||||
expect(retried!.stalled_counter).toBe(0);
|
||||
});
|
||||
|
||||
// #2783 repro: retry issued long after the original claim must NOT be
|
||||
// immediately dead-lettered by the wall-clock sweep on re-claim.
|
||||
test('retry survives handleWallClockTimeouts after re-claim, even long after the original attempt', async () => {
|
||||
const job = await queue.add('sync', {}, { max_attempts: 3 });
|
||||
await engine.executeRaw('UPDATE minion_jobs SET timeout_ms = 1000 WHERE id = $1', [job.id]);
|
||||
await queue.claim('tok1', 30000, 'default', ['sync']);
|
||||
// Original attempt dies from a wall-clock timeout — matches the issue's
|
||||
// repro (an outage that outlasts timeout_ms).
|
||||
await engine.executeRaw(
|
||||
"UPDATE minion_jobs SET started_at = now() - interval '10 seconds' WHERE id = $1",
|
||||
[job.id],
|
||||
);
|
||||
const firstDead = await queue.handleWallClockTimeouts(30000);
|
||||
expect(firstDead.length).toBe(1);
|
||||
expect(firstDead[0].status).toBe('dead');
|
||||
|
||||
// Outage clears; operator retries — LONG after the original claim time
|
||||
// (this is the exact scenario that used to dead-letter in <1s: without
|
||||
// the fix, started_at would still be the original claim's timestamp).
|
||||
await queue.retryJob(job.id);
|
||||
const reclaimed = await queue.claim('tok2', 30000, 'default', ['sync']);
|
||||
expect(reclaimed).not.toBeNull();
|
||||
expect(reclaimed!.attempts_made).toBe(0);
|
||||
|
||||
// The sweep must NOT kill it immediately this time — started_at was
|
||||
// re-stamped fresh on re-claim (claim()'s COALESCE(started_at, now())).
|
||||
const stillAlive = await queue.handleWallClockTimeouts(30000);
|
||||
expect(stillAlive.length).toBe(0);
|
||||
expect((await queue.getJob(job.id))!.status).toBe('active');
|
||||
});
|
||||
|
||||
// #2783 repro (stall side): a job dead-lettered by stall exhaustion must
|
||||
// get a fresh stall budget on retry, not immediately re-die on its first
|
||||
// stall after being re-claimed.
|
||||
test('retry survives one stall after re-claim, even after the original stall budget was exhausted', async () => {
|
||||
const job = await queue.add('sync', {}, { max_attempts: 3, max_stalled: 2 });
|
||||
|
||||
// Exhaust the stall budget the same way the existing stall test does:
|
||||
// one requeue stall, then one dead-lettering stall.
|
||||
await queue.claim('tok1', 30000, 'default', ['sync']);
|
||||
await engine.executeRaw(
|
||||
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
|
||||
[job.id],
|
||||
);
|
||||
await queue.handleStalled();
|
||||
await queue.claim('tok2', 30000, 'default', ['sync']);
|
||||
await engine.executeRaw(
|
||||
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
|
||||
[job.id],
|
||||
);
|
||||
const r2 = await queue.handleStalled();
|
||||
expect(r2.dead.length).toBe(1);
|
||||
expect(r2.dead[0].status).toBe('dead');
|
||||
expect(r2.dead[0].stalled_counter).toBe(2); // == max_stalled — exhausted
|
||||
|
||||
// Operator retries. Without the stalled_counter reset, the very next
|
||||
// stall would immediately satisfy `stalled_counter + 1 >= max_stalled`
|
||||
// and dead-letter again despite "run this fresh".
|
||||
const retried = await queue.retryJob(job.id);
|
||||
expect(retried!.stalled_counter).toBe(0);
|
||||
await queue.claim('tok3', 30000, 'default', ['sync']);
|
||||
await engine.executeRaw(
|
||||
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
|
||||
[job.id],
|
||||
);
|
||||
const r3 = await queue.handleStalled();
|
||||
expect(r3.requeued.length).toBe(1); // fresh budget — requeued, not dead
|
||||
expect(r3.dead.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// --- Pause / Resume (5 tests) ---
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* #2704 — a single markdown page whose compiled_truth exceeds Postgres's
|
||||
* hard 1,048,575-byte tsvector cap made update_page_search_vector() throw
|
||||
* "string is too long for tsvector" INSIDE the pages UPSERT transaction,
|
||||
* blocking the whole source's sync checkpoint (Sync BLOCKED) even though
|
||||
* every other file in the run imported fine.
|
||||
*
|
||||
* v124 (migrate.ts) drops compiled_truth from the trigger — it was
|
||||
* already redundant with content_chunks.search_vector (chunk-grain,
|
||||
* populated separately and well under the tsvector cap), which is what
|
||||
* searchKeyword() actually queries.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
// #2704: the 1,048,575-byte tsvector cap is on to_tsvector's SERIALIZED
|
||||
// OUTPUT (lexemes + position lists), not the raw input byte length —
|
||||
// repeating the same few words produces a tiny deduplicated vector
|
||||
// regardless of input size (verified: a 2.2MB string of 5 repeated words
|
||||
// does NOT overflow). Genuinely diverse, mostly-unique tokens are what
|
||||
// blows the output past the cap, matching a real large export (a Google
|
||||
// Docs dump, a long mailing-list thread) where the words don't repeat
|
||||
// like lorem-ipsum filler does.
|
||||
const OVERSIZED_BODY = Array.from({ length: 200_000 }, (_, i) => `token${i.toString(36)}`).join(' '); // ~2MB, ~2.7MB serialized tsvector
|
||||
|
||||
describe('#2704: oversized page body no longer overflows pages.search_vector', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
test('putPage with a >1MB compiled_truth succeeds (previously threw "string is too long for tsvector")', async () => {
|
||||
expect(OVERSIZED_BODY.length).toBeGreaterThan(1_048_575);
|
||||
|
||||
const page = await engine.putPage('oversized-page', {
|
||||
type: 'note',
|
||||
title: 'Oversized Page',
|
||||
compiled_truth: OVERSIZED_BODY,
|
||||
});
|
||||
|
||||
expect(page).not.toBeNull();
|
||||
expect(page.slug).toBe('oversized-page');
|
||||
}, 30_000);
|
||||
|
||||
test('an oversized page is still keyword-searchable via chunk-grain search after import', async () => {
|
||||
// Mirrors import-file.ts: chunking is what actually feeds
|
||||
// content_chunks.search_vector, independent of the pages-level
|
||||
// trigger this fix touches. A distinctive token near the start proves
|
||||
// the chunk (not just the page row) is queryable.
|
||||
const distinctiveBody = `zzdistinctivetoken2704 ${OVERSIZED_BODY}`;
|
||||
await engine.putPage('oversized-searchable', {
|
||||
type: 'note',
|
||||
title: 'Oversized Searchable',
|
||||
compiled_truth: distinctiveBody,
|
||||
});
|
||||
const { chunkText } = await import('../src/core/chunkers/recursive.ts');
|
||||
let chunkIndex = 0;
|
||||
const chunks = chunkText(distinctiveBody).map((c) => ({
|
||||
chunk_index: chunkIndex++,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
}));
|
||||
await engine.upsertChunks('oversized-searchable', chunks);
|
||||
|
||||
const results = await engine.searchKeyword('zzdistinctivetoken2704');
|
||||
expect(results.some((r) => r.slug === 'oversized-searchable')).toBe(true);
|
||||
}, 30_000);
|
||||
|
||||
test('normal-sized page search_vector still carries title/timeline signal (not fully inert)', async () => {
|
||||
await engine.putPage('small-page', {
|
||||
type: 'note',
|
||||
title: 'zzTitleToken2704',
|
||||
compiled_truth: 'short body',
|
||||
});
|
||||
const rows = await engine.executeRaw<{ has_vector: boolean }>(
|
||||
`SELECT search_vector IS NOT NULL AS has_vector FROM pages WHERE slug = 'small-page'`,
|
||||
);
|
||||
expect(rows[0]?.has_vector).toBe(true);
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,181 @@
|
||||
/**
|
||||
* withScopedReadTransaction — opt-in Postgres RLS source-scope binding
|
||||
* (GBRAIN_RLS_SCOPE_BINDING, lands community PR #2387).
|
||||
*
|
||||
* Behavioral pins, no real DB (fake postgres.js sql handle):
|
||||
* - flag OFF (default): TRUE pass-through — callback receives the shared
|
||||
* pool handle directly, no sql.begin(), no set_config. This is the
|
||||
* #1794-class guard: reads must not gain a per-read pool hold.
|
||||
* - flag OFF + alwaysTransaction (the search methods' SET LOCAL path):
|
||||
* sql.begin() opens, still no set_config — identical to master's wrap.
|
||||
* - flag ON: sql.begin() + SELECT set_config('app.scopes', $1, true)
|
||||
* with federated-array > scalar > '*' precedence, and the CSV value
|
||||
* carried as a BOUND PARAMETER, never interpolated into the SQL text.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { PostgresEngine } from '../src/core/postgres-engine.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
type Recorded = { text: string; params: unknown[] };
|
||||
|
||||
function makeFakeSql() {
|
||||
const queries: Recorded[] = [];
|
||||
let beginCalls = 0;
|
||||
const record = (strings: TemplateStringsArray, ...params: unknown[]) => {
|
||||
// Join the literal segments with a placeholder marker so the test can
|
||||
// assert the exact SQL text shape around each bound parameter.
|
||||
queries.push({ text: strings.join('${}'), params });
|
||||
return Promise.resolve([]);
|
||||
};
|
||||
const sql = ((strings: TemplateStringsArray, ...params: unknown[]) =>
|
||||
record(strings, ...params)) as unknown as Record<string, unknown> & {
|
||||
(strings: TemplateStringsArray, ...params: unknown[]): Promise<unknown[]>;
|
||||
begin: (cb: (tx: unknown) => Promise<unknown>) => Promise<unknown>;
|
||||
};
|
||||
const tx = ((strings: TemplateStringsArray, ...params: unknown[]) =>
|
||||
record(strings, ...params)) as unknown as Record<string, unknown>;
|
||||
sql.begin = async (cb: (t: unknown) => Promise<unknown>) => {
|
||||
beginCalls++;
|
||||
return await cb(tx);
|
||||
};
|
||||
return { sql, tx, queries, beginCalls: () => beginCalls };
|
||||
}
|
||||
|
||||
function makeEngine(fake: ReturnType<typeof makeFakeSql>) {
|
||||
const e = new PostgresEngine();
|
||||
(e as unknown as { _sql: unknown })._sql = fake.sql;
|
||||
(e as unknown as { _connectionStyle: string })._connectionStyle = 'instance';
|
||||
// private method, invoked directly for the pin
|
||||
return e as unknown as {
|
||||
withScopedReadTransaction<T>(
|
||||
sourceIds: string[] | undefined,
|
||||
sourceId: string | undefined,
|
||||
cb: (tx: unknown) => Promise<T>,
|
||||
opts?: { alwaysTransaction?: boolean },
|
||||
): Promise<T>;
|
||||
};
|
||||
}
|
||||
|
||||
function setConfigQueries(queries: Recorded[]): Recorded[] {
|
||||
return queries.filter((q) => q.text.includes('set_config'));
|
||||
}
|
||||
|
||||
describe('withScopedReadTransaction / flag off (default)', () => {
|
||||
test('true pass-through: callback gets the shared pool handle, no begin, no set_config', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let received: unknown;
|
||||
const result = await engine.withScopedReadTransaction(undefined, 'src-a', async (tx) => {
|
||||
received = tx;
|
||||
return 42;
|
||||
});
|
||||
expect(result).toBe(42);
|
||||
expect(received).toBe(fake.sql); // the pool handle itself, not a tx
|
||||
expect(fake.beginCalls()).toBe(0);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('explicit "0" is off too', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '0' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(['a', 'b'], undefined, async () => null);
|
||||
expect(fake.beginCalls()).toBe(0);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('alwaysTransaction keeps master\'s sql.begin() wrap, still no set_config', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let received: unknown;
|
||||
await engine.withScopedReadTransaction(
|
||||
undefined,
|
||||
'src-a',
|
||||
async (tx) => {
|
||||
received = tx;
|
||||
return null;
|
||||
},
|
||||
{ alwaysTransaction: true },
|
||||
);
|
||||
expect(fake.beginCalls()).toBe(1);
|
||||
expect(received).toBe(fake.tx); // a transaction handle this time
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('withScopedReadTransaction / flag on', () => {
|
||||
test('emits set_config(\'app.scopes\', ...) inside a transaction, before the callback', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let queriesAtCallback = -1;
|
||||
await engine.withScopedReadTransaction(undefined, 'src-a', async () => {
|
||||
queriesAtCallback = fake.queries.length;
|
||||
return null;
|
||||
});
|
||||
expect(fake.beginCalls()).toBe(1);
|
||||
const sc = setConfigQueries(fake.queries);
|
||||
expect(sc).toHaveLength(1);
|
||||
expect(sc[0].params).toEqual(['src-a']);
|
||||
// set_config was emitted before the callback ran
|
||||
expect(queriesAtCallback).toBe(1);
|
||||
expect(fake.queries[0]).toBe(sc[0]);
|
||||
});
|
||||
});
|
||||
|
||||
test('"true" also enables', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: 'true' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(undefined, 'src-a', async () => null);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('federated array wins over scalar: CSV of sourceIds', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(['a', 'b', 'c'], 'ignored-scalar', async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['a,b,c']);
|
||||
});
|
||||
});
|
||||
|
||||
test('empty federated array falls back to scalar', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction([], 'src-b', async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['src-b']);
|
||||
});
|
||||
});
|
||||
|
||||
test("unscoped (no sourceIds, no sourceId) binds '*'", async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(undefined, undefined, async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['*']);
|
||||
});
|
||||
});
|
||||
|
||||
test('the scopes CSV is a BOUND PARAMETER, never interpolated into SQL text', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
const hostile = "x','y'); DROP TABLE pages; --";
|
||||
await engine.withScopedReadTransaction(undefined, hostile, async () => null);
|
||||
const sc = setConfigQueries(fake.queries)[0];
|
||||
// Exact literal-segment shape: the value slot is the tagged-template hole.
|
||||
expect(sc.text).toBe("SELECT set_config('app.scopes', ${}, true)");
|
||||
expect(sc.params).toEqual([hostile]);
|
||||
expect(sc.text).not.toContain(hostile);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -48,14 +48,34 @@ describe('postgres-engine / search path timeout isolation', () => {
|
||||
expect(bare).toBeNull();
|
||||
});
|
||||
|
||||
test('searchKeyword wraps its query in sql.begin()', () => {
|
||||
test('searchKeyword wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => {
|
||||
// Post-RLS-scope-binding invariant: the search methods route through
|
||||
// withScopedReadTransaction with alwaysTransaction: true, which
|
||||
// guarantees a sql.begin() wrap in BOTH modes — flag off (identical to
|
||||
// master's pre-helper wrap) and flag on (scoped transaction with
|
||||
// set_config). See the helper tests in
|
||||
// test/postgres-engine-rls-scope.test.ts for the behavioral pins.
|
||||
const fn = extractMethod(SRC, 'searchKeyword');
|
||||
expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/);
|
||||
expect(fn).toMatch(/withScopedReadTransaction\s*\(/);
|
||||
expect(fn).toMatch(/alwaysTransaction:\s*true/);
|
||||
});
|
||||
|
||||
test('searchVector wraps its query in sql.begin()', () => {
|
||||
test('searchVector wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => {
|
||||
const fn = extractMethod(SRC, 'searchVector');
|
||||
expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/);
|
||||
expect(fn).toMatch(/withScopedReadTransaction\s*\(/);
|
||||
expect(fn).toMatch(/alwaysTransaction:\s*true/);
|
||||
});
|
||||
|
||||
test('withScopedReadTransaction owns the sql.begin() wrap (and only opens it when needed)', () => {
|
||||
// (extractMethod can't grab this one: `private async ...<T>(`.)
|
||||
const stripped = stripComments(SRC);
|
||||
// The transaction lives in the helper...
|
||||
expect(stripped).toMatch(/this\.sql\.begin\s*\(/);
|
||||
// ...and the flag-off / non-alwaysTransaction path is a true
|
||||
// pass-through on the shared pool — no per-read transaction hold.
|
||||
expect(stripped).toMatch(
|
||||
/if\s*\(!this\.rlsScopeBindingEnabled\s*&&\s*!opts\?\.alwaysTransaction\)\s*\{\s*return\s+await\s+callback\(this\.sql\);/,
|
||||
);
|
||||
});
|
||||
|
||||
test('both search methods use SET LOCAL for the timeout', () => {
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Regression guard: `gbrain remote ping` must poll the MinionJob `status`
|
||||
* field, never `state`.
|
||||
*
|
||||
* submit_job and get_job (src/core/operations.ts) return the MinionJob row
|
||||
* verbatim, whose lifecycle field is `status`
|
||||
* (src/core/minions/types.ts). remote.ts once typed and read `state`
|
||||
* instead: every poll then saw `undefined`, the terminal check
|
||||
* (`['completed','failed','dead','cancelled'].includes(job.state)`) never
|
||||
* matched, and ping exhausted its full --timeout and exited 1 even when
|
||||
* the autopilot-cycle had completed — printing
|
||||
* "Job #N is still undefined." on the way out.
|
||||
*
|
||||
* Source-audit style (same idiom as thin-client-routing-audit.test.ts):
|
||||
* pins the reads without needing a live MCP transport.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
const REMOTE_TS_PATH = join(import.meta.dir, '..', 'src', 'commands', 'remote.ts');
|
||||
const REMOTE_SOURCE = readFileSync(REMOTE_TS_PATH, 'utf8');
|
||||
|
||||
describe('remote ping polls MinionJob.status, not .state', () => {
|
||||
test('no `.state` property reads on job objects remain', () => {
|
||||
// Catches `submitted.state`, `job.state` — any resurrection of the
|
||||
// wrong field. The ping's JSON *output* keys (`state:`, `last_state:`)
|
||||
// are object-literal keys, not property reads, and don't match this.
|
||||
expect(REMOTE_SOURCE).not.toMatch(/\b(?:job|submitted)\.state\b/);
|
||||
});
|
||||
|
||||
test('poll loop reads job.status', () => {
|
||||
expect(REMOTE_SOURCE).toMatch(/\bjob\.status\b/);
|
||||
expect(REMOTE_SOURCE).toMatch(/\bsubmitted\.status\b/);
|
||||
});
|
||||
|
||||
test('terminal-state check tests job.status', () => {
|
||||
expect(REMOTE_SOURCE).toMatch(/terminal\.includes\(job\.status\)/);
|
||||
});
|
||||
|
||||
test('unpack generics type the lifecycle field as status', () => {
|
||||
// Both the submit and poll unpack sites must carry `status: string` in
|
||||
// their type argument, and none may reintroduce `state: string`.
|
||||
const unpackShapes = REMOTE_SOURCE.match(/unpackToolResult<\{[^}]*\}>/g) ?? [];
|
||||
const jobShapes = unpackShapes.filter((s) => s.includes('id: number'));
|
||||
expect(jobShapes.length).toBeGreaterThanOrEqual(2);
|
||||
for (const shape of jobShapes) {
|
||||
expect(shape).toContain('status: string');
|
||||
expect(shape).not.toContain('state: string');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* Regression for schema CLI engine routing.
|
||||
*
|
||||
* Serial because it opens a persistent PGLite database and then hands that
|
||||
* database to a CLI subprocess. The subprocess must read the configured path,
|
||||
* not silently fall back to the default brain.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
const REPO_ROOT = join(import.meta.dir, '..');
|
||||
|
||||
describe('gbrain schema configured PGLite routing', () => {
|
||||
test('schema stats reads database_path from config', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gbrain-schema-db-path-'));
|
||||
const gbrainDir = join(home, '.gbrain');
|
||||
const dbPath = join(gbrainDir, 'configured-brain.pglite');
|
||||
mkdirSync(gbrainDir, { recursive: true });
|
||||
|
||||
const engine = new PGLiteEngine();
|
||||
try {
|
||||
await engine.connect({ engine: 'pglite', database_path: dbPath });
|
||||
await engine.initSchema();
|
||||
await engine.putPage('people/alice-example', {
|
||||
type: 'person',
|
||||
title: 'Alice Example',
|
||||
compiled_truth: 'Example page',
|
||||
});
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
|
||||
writeFileSync(
|
||||
join(gbrainDir, 'config.json'),
|
||||
JSON.stringify({ engine: 'pglite', database_path: dbPath, schema_pack: 'gbrain-base' }),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
try {
|
||||
const result = spawnSync(
|
||||
'bun',
|
||||
['run', 'src/cli.ts', 'schema', 'stats', '--json'],
|
||||
{
|
||||
cwd: REPO_ROOT,
|
||||
encoding: 'utf-8',
|
||||
env: {
|
||||
...process.env,
|
||||
GBRAIN_DATABASE_URL: '',
|
||||
DATABASE_URL: '',
|
||||
GBRAIN_HOME: home,
|
||||
},
|
||||
timeout: 60_000,
|
||||
},
|
||||
);
|
||||
expect(result.status).toBe(0);
|
||||
const stats = JSON.parse(result.stdout ?? '');
|
||||
expect(stats.aggregate.total_pages).toBe(1);
|
||||
expect(stats.aggregate.by_type).toContainEqual({ type: 'person', count: 1 });
|
||||
} finally {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
}
|
||||
}, 90_000);
|
||||
});
|
||||
@@ -36,7 +36,13 @@ function gbrain(
|
||||
// bun's spawnSync does NOT inherit env mutations done via process.env = ...,
|
||||
// so pass env explicitly. CLAUDE.md flags this pattern as load-bearing for
|
||||
// any subprocess test that needs GBRAIN_HOME isolation.
|
||||
const env = { ...process.env, GBRAIN_HOME: DEFAULT_GBRAIN_HOME, ...extraEnv };
|
||||
const env = {
|
||||
...process.env,
|
||||
GBRAIN_DATABASE_URL: '',
|
||||
DATABASE_URL: '',
|
||||
GBRAIN_HOME: DEFAULT_GBRAIN_HOME,
|
||||
...extraEnv,
|
||||
};
|
||||
const result = spawnSync('bun', ['run', 'src/cli.ts', ...args], {
|
||||
cwd: REPO_ROOT,
|
||||
encoding: 'utf-8',
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
/**
|
||||
* Regression wiring test for #2952 — cache classification reaches telemetry.
|
||||
*
|
||||
* Pre-fix, `recordSearchTelemetry` fired only from bare `hybridSearch`, whose
|
||||
* meta never carries a `cache` field, and a cache HIT returned from
|
||||
* `hybridSearchCached` before any record at all. Net effect on a live brain:
|
||||
* `search stats` reported `0 hit / 0 miss` forever while the `query_cache`
|
||||
* table grew, and hit searches vanished from count/results/tokens/rank-1.
|
||||
*
|
||||
* This file drives the REAL pipeline (PGLite brain, real SemanticQueryCache
|
||||
* store→lookup roundtrip, mocked `embedQuery` for a deterministic vector) and
|
||||
* pins the decision matrix:
|
||||
*
|
||||
* - consulted + no row → recorded once with cache_miss
|
||||
* - consulted + row → recorded once with cache_hit (plus results/rank-1)
|
||||
* - consult skipped → recorded once with neither counter
|
||||
* - bare hybridSearch → recorded once with neither counter (unchanged)
|
||||
*
|
||||
* Serial: mock.module + gateway/global-env mutation (isolation guard R2).
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, mock, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import * as realEmbedding from '../src/core/embedding.ts';
|
||||
|
||||
/** Deterministic 1536d unit vector — same for every call, so an identical
|
||||
* query's second consult matches its first write at cosine 1.0. */
|
||||
function fixedEmbedding(): Float32Array {
|
||||
const arr = new Float32Array(1536);
|
||||
for (let i = 0; i < 1536; i++) arr[i] = Math.sin(1 + i * 0.001);
|
||||
let norm = 0;
|
||||
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
}
|
||||
|
||||
// Pluggable behavior so individual tests can simulate an embed-provider
|
||||
// failure (the 'disabled'-via-catch flavor). null → deterministic vector.
|
||||
let embedBehavior: (() => Promise<Float32Array>) | null = null;
|
||||
|
||||
// Mock the embedding seam BEFORE importing hybrid.ts so both the cache-lookup
|
||||
// embed and the inner vector-arm embed resolve without a provider call. Spread
|
||||
// the real module so every other export stays live.
|
||||
mock.module('../src/core/embedding.ts', () => ({
|
||||
...realEmbedding,
|
||||
embed: async () => (embedBehavior ? embedBehavior() : fixedEmbedding()),
|
||||
embedQuery: async () => (embedBehavior ? embedBehavior() : fixedEmbedding()),
|
||||
}));
|
||||
|
||||
// Import AFTER mocking.
|
||||
const { hybridSearch, hybridSearchCached, awaitPendingSearchCacheWrites, _resetPendingSearchCacheWritesForTests } =
|
||||
await import('../src/core/search/hybrid.ts');
|
||||
const { getTelemetryWriter, _resetTelemetryWriterForTest } = await import('../src/core/search/telemetry.ts');
|
||||
const { configureGateway, resetGateway } = await import('../src/core/ai/gateway.ts');
|
||||
const { PGLiteEngine } = await import('../src/core/pglite-engine.ts');
|
||||
|
||||
let engine: InstanceType<typeof PGLiteEngine>;
|
||||
let tmpHome: string;
|
||||
const savedGbrainHome = process.env.GBRAIN_HOME;
|
||||
|
||||
interface Counters {
|
||||
c: number;
|
||||
hit: number;
|
||||
miss: number;
|
||||
rank1: number;
|
||||
results: number;
|
||||
tokens: number;
|
||||
}
|
||||
|
||||
/** Flush the writer and read the summed counters back from the table. */
|
||||
async function readCounters(): Promise<Counters> {
|
||||
await getTelemetryWriter().flush();
|
||||
const rows = await engine.executeRaw<Counters>(
|
||||
`SELECT COALESCE(SUM(count), 0)::int AS c,
|
||||
COALESCE(SUM(cache_hit), 0)::int AS hit,
|
||||
COALESCE(SUM(cache_miss), 0)::int AS miss,
|
||||
COALESCE(SUM(count_rank1), 0)::int AS rank1,
|
||||
COALESCE(SUM(sum_results), 0)::int AS results,
|
||||
COALESCE(SUM(sum_tokens), 0)::int AS tokens
|
||||
FROM search_telemetry`,
|
||||
);
|
||||
return rows[0];
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
// Hermetic config home so the developer's real ~/.gbrain/config.json can't
|
||||
// leak an embedding_model that flips isCacheSafe → 'disabled'.
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-cache-telemetry-'));
|
||||
process.env.GBRAIN_HOME = tmpHome;
|
||||
|
||||
// Pin the gateway to a 1536d provider BEFORE initSchema so the
|
||||
// query_cache.embedding column is sized for the mock vectors, and so
|
||||
// isAvailable('embedding') lets the cache consult proceed. The fake key is
|
||||
// never used — embedQuery is mocked above. (Pattern:
|
||||
// test/query-cache-knobs-hash.serial.test.ts.)
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-fake' },
|
||||
});
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Keyword-findable fixtures so the inner search returns rows (a non-empty
|
||||
// result set is what arms the cache writeback). searchKeyword joins
|
||||
// content_chunks, so pages need explicit chunks — putPage alone leaves the
|
||||
// chunk table empty (pattern: test/chunk-grain-fts.test.ts).
|
||||
await engine.putPage('alice-foo', {
|
||||
type: 'person',
|
||||
title: 'Alice Foo',
|
||||
compiled_truth: 'Alice Foo is a builder who ships search telemetry fixtures.',
|
||||
});
|
||||
await engine.upsertChunks('alice-foo', [
|
||||
{ chunk_index: 0, chunk_text: 'Alice Foo is a builder who ships search telemetry fixtures.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('bob-bar', {
|
||||
type: 'person',
|
||||
title: 'Bob Bar',
|
||||
compiled_truth: 'Bob Bar is a builder who reviews cache wiring fixtures.',
|
||||
});
|
||||
await engine.upsertChunks('bob-bar', [
|
||||
{ chunk_index: 0, chunk_text: 'Bob Bar is a builder who reviews cache wiring fixtures.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedGbrainHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = savedGbrainHome;
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
resetGateway();
|
||||
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
embedBehavior = null;
|
||||
_resetTelemetryWriterForTest();
|
||||
_resetPendingSearchCacheWritesForTests();
|
||||
await engine.executeRaw('DELETE FROM search_telemetry');
|
||||
await engine.executeRaw('DELETE FROM query_cache');
|
||||
});
|
||||
|
||||
describe('hybridSearchCached — telemetry carries the cache outcome', () => {
|
||||
test('miss then hit: one record per search, classified, hit keeps results/rank-1 telemetry', async () => {
|
||||
// Call 1 — cache consulted, empty → miss.
|
||||
const first = await hybridSearchCached(engine, 'alice telemetry fixtures', { limit: 5 });
|
||||
expect(first.length).toBeGreaterThan(0);
|
||||
await awaitPendingSearchCacheWrites();
|
||||
|
||||
// Sanity: the writeback actually landed, so call 2 exercises a REAL hit
|
||||
// (a broken writeback would otherwise fail the hit assertion ambiguously).
|
||||
const cacheRows = await engine.executeRaw<{ n: number }>(
|
||||
'SELECT COUNT(*)::int AS n FROM query_cache',
|
||||
);
|
||||
expect(cacheRows[0].n).toBeGreaterThan(0);
|
||||
|
||||
const afterMiss = await readCounters();
|
||||
expect(afterMiss.c).toBe(1);
|
||||
expect(afterMiss.miss).toBe(1);
|
||||
expect(afterMiss.hit).toBe(0);
|
||||
expect(afterMiss.rank1).toBe(1);
|
||||
expect(afterMiss.results).toBeGreaterThan(0);
|
||||
|
||||
// Call 2 — identical query + knobs, deterministic embedding → hit.
|
||||
let meta: import('../src/core/types.ts').HybridSearchMeta | undefined;
|
||||
const second = await hybridSearchCached(engine, 'alice telemetry fixtures', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.cache?.status).toBe('hit');
|
||||
expect(second.length).toBeGreaterThan(0);
|
||||
|
||||
const afterHit = await readCounters();
|
||||
// Pre-fix both sides of this were wrong: hit stayed 0 forever AND the hit
|
||||
// search was missing from count entirely (c would read 1, not 2).
|
||||
expect(afterHit.c).toBe(2);
|
||||
expect(afterHit.miss).toBe(1);
|
||||
expect(afterHit.hit).toBe(1);
|
||||
// The hit search contributes results/rank-1/tokens telemetry too.
|
||||
expect(afterHit.rank1).toBe(2);
|
||||
expect(afterHit.results).toBeGreaterThan(afterMiss.results);
|
||||
// Token parity (codex): the hit serves the SAME result set the miss
|
||||
// stored, so its token contribution must EQUAL the miss's — a hit/miss
|
||||
// accounting asymmetry (e.g. hits counting tokens the miss convention
|
||||
// skips) would break this exact-delta check.
|
||||
expect(afterHit.tokens - afterMiss.tokens).toBe(afterMiss.tokens);
|
||||
});
|
||||
|
||||
test('lookup-embed failure: consult degrades to disabled — recorded once, neither counter', async () => {
|
||||
embedBehavior = async () => { throw new Error('embed provider down'); };
|
||||
// The failed consult must not break the search: keyword fallback serves.
|
||||
const results = await hybridSearchCached(engine, 'bob cache wiring', { limit: 5 });
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
|
||||
const counters = await readCounters();
|
||||
expect(counters.c).toBe(1);
|
||||
expect(counters.hit).toBe(0);
|
||||
expect(counters.miss).toBe(0);
|
||||
expect(counters.rank1).toBe(1);
|
||||
});
|
||||
|
||||
test('consult skipped (useCache:false): recorded once, neither counter', async () => {
|
||||
const results = await hybridSearchCached(engine, 'bob cache wiring', { limit: 5, useCache: false });
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
|
||||
const counters = await readCounters();
|
||||
expect(counters.c).toBe(1);
|
||||
expect(counters.hit).toBe(0);
|
||||
expect(counters.miss).toBe(0);
|
||||
// Telemetry otherwise unchanged: the search still counts fully.
|
||||
expect(counters.rank1).toBe(1);
|
||||
expect(counters.results).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('bare hybridSearch — direct callers unchanged', () => {
|
||||
test('records once with no cache classification', async () => {
|
||||
let meta: import('../src/core/types.ts').HybridSearchMeta | undefined;
|
||||
const results = await hybridSearch(engine, 'bob cache wiring', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// The onMeta contract is untouched: no cache field is injected into the
|
||||
// caller-visible meta (the fold happens on the recorded copy only).
|
||||
expect(meta?.cache).toBeUndefined();
|
||||
|
||||
const counters = await readCounters();
|
||||
expect(counters.c).toBe(1);
|
||||
expect(counters.hit).toBe(0);
|
||||
expect(counters.miss).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,296 @@
|
||||
/**
|
||||
* fix/title-retrieval-arm — D1 title candidate arm + D2 AND→OR keyword fallback.
|
||||
*
|
||||
* The disease (3-lane diagnostic, 2026-07): page titles never enter the
|
||||
* keyword-searchable text. content_chunks.search_vector is doc_comment +
|
||||
* symbol_name_qualified + chunk_text — no title — so an exact-title query
|
||||
* whose tokens are absent from the body had ZERO keyword recall, and every
|
||||
* existing title mechanism (title boost, exact-match boost, alias hop) is
|
||||
* re-rank-only: none can GENERATE the missing candidate. Compounding it,
|
||||
* websearch_to_tsquery AND semantics at chunk grain meant one
|
||||
* non-co-occurring token zeroed the whole keyword arm with no fallback.
|
||||
*
|
||||
* Fixes under test:
|
||||
* C1 — engine.searchTitles: page-grain candidates from pages.search_vector
|
||||
* (title weight 'A'), joined to one representative chunk, fused into
|
||||
* hybridSearch as a keyword-class RRF list. No query-length gate.
|
||||
* C2 — searchKeyword retries ONCE with OR-of-terms when strict AND
|
||||
* returns zero rows; strict results always win when non-empty.
|
||||
*
|
||||
* Hermetic PGLite. The gateway is pinned with an EMPTY env so embedding is
|
||||
* deterministically unavailable — hybridSearch takes the keyword(+title)
|
||||
* no-embed path with zero network, regardless of host API keys.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
import { hybridSearch } from '../../src/core/search/hybrid.ts';
|
||||
import { buildOrFallbackWebsearchQuery } from '../../src/core/search/sql-ranking.ts';
|
||||
import { configureGateway } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
const DIM = 1536;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Pin 1536-d (matches the preload schema default) with an EMPTY env so
|
||||
// isAvailable('embedding') is false → hybridSearch never embeds.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: DIM,
|
||||
env: {},
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({}); // in-memory
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
// Restore the preload-equivalent gateway for sibling files in this shard.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: DIM,
|
||||
env: { ...process.env },
|
||||
});
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Page whose TITLE tokens never appear in its body/chunks (the D1 shape). */
|
||||
async function seedTitleOnlyPage(): Promise<void> {
|
||||
await engine.putPage('projects/chronomancer', {
|
||||
type: 'note',
|
||||
title: 'Chronomancer Codex Ledger',
|
||||
compiled_truth: 'A reference document about scheduling practices and planning.',
|
||||
});
|
||||
await engine.upsertChunks('projects/chronomancer', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'A reference document about scheduling practices and planning.',
|
||||
chunk_source: 'compiled_truth',
|
||||
},
|
||||
]);
|
||||
}
|
||||
|
||||
describe('searchTitles — D1 title candidate arm', () => {
|
||||
test('exact-title query retrieves a page whose title tokens are absent from its body', async () => {
|
||||
await seedTitleOnlyPage();
|
||||
|
||||
// Premise check: the chunk-grain keyword arm CANNOT see this page for
|
||||
// this query, even with the OR fallback (no title token is in any chunk).
|
||||
const kw = await engine.searchKeyword('Chronomancer Codex Ledger', { limit: 10 });
|
||||
expect(kw.map(r => r.slug)).not.toContain('projects/chronomancer');
|
||||
|
||||
// The title arm can.
|
||||
const hits = await engine.searchTitles('Chronomancer Codex Ledger', { limit: 10 });
|
||||
expect(hits.map(r => r.slug)).toContain('projects/chronomancer');
|
||||
const hit = hits.find(r => r.slug === 'projects/chronomancer')!;
|
||||
expect(hit.title).toBe('Chronomancer Codex Ledger');
|
||||
expect(hit.score).toBeGreaterThan(0);
|
||||
// Shaped like a keyword-arm row: representative chunk attached.
|
||||
expect(hit.chunk_text).toContain('reference document');
|
||||
expect(hit.chunk_source).toBe('compiled_truth');
|
||||
});
|
||||
|
||||
test('long 10-content-token exact-title query still retrieves (no token-count gate)', async () => {
|
||||
const longTitle = 'Emerald Falcon Doctrine Quarterly Synthesis Report Alpha Bravo Charlie Delta';
|
||||
await engine.putPage('reports/emerald-falcon', {
|
||||
type: 'note',
|
||||
title: longTitle,
|
||||
compiled_truth: 'An annual planning artifact.',
|
||||
});
|
||||
await engine.upsertChunks('reports/emerald-falcon', [
|
||||
{ chunk_index: 0, chunk_text: 'An annual planning artifact.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
const hits = await engine.searchTitles(longTitle, { limit: 10 });
|
||||
expect(hits.map(r => r.slug)).toContain('reports/emerald-falcon');
|
||||
});
|
||||
|
||||
test('representative chunk prefers compiled_truth, else lowest chunk_index', async () => {
|
||||
await engine.putPage('notes/mixed-chunks', {
|
||||
type: 'note',
|
||||
title: 'Obsidian Waterfall Registry',
|
||||
compiled_truth: 'body text here',
|
||||
});
|
||||
await engine.upsertChunks('notes/mixed-chunks', [
|
||||
{ chunk_index: 0, chunk_text: 'timeline entry text', chunk_source: 'timeline' },
|
||||
{ chunk_index: 1, chunk_text: 'compiled body text', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
const hits = await engine.searchTitles('Obsidian Waterfall Registry', { limit: 5 });
|
||||
const hit = hits.find(r => r.slug === 'notes/mixed-chunks')!;
|
||||
expect(hit.chunk_source).toBe('compiled_truth');
|
||||
expect(hit.chunk_index).toBe(1);
|
||||
|
||||
await engine.putPage('notes/timeline-only', {
|
||||
type: 'note',
|
||||
title: 'Cobalt Meridian Atlas',
|
||||
compiled_truth: 'unrelated body',
|
||||
});
|
||||
await engine.upsertChunks('notes/timeline-only', [
|
||||
{ chunk_index: 5, chunk_text: 'later timeline', chunk_source: 'timeline' },
|
||||
{ chunk_index: 2, chunk_text: 'earlier timeline', chunk_source: 'timeline' },
|
||||
]);
|
||||
const tlHits = await engine.searchTitles('Cobalt Meridian Atlas', { limit: 5 });
|
||||
const tlHit = tlHits.find(r => r.slug === 'notes/timeline-only')!;
|
||||
expect(tlHit.chunk_index).toBe(2); // lowest index when no compiled_truth chunk
|
||||
});
|
||||
|
||||
test('respects soft-delete visibility and source scoping', async () => {
|
||||
await seedTitleOnlyPage();
|
||||
|
||||
// Source scope that doesn't own the page → filtered out at SQL level.
|
||||
const scoped = await engine.searchTitles('Chronomancer Codex Ledger', {
|
||||
limit: 10,
|
||||
sourceId: 'some-other-source',
|
||||
});
|
||||
expect(scoped.length).toBe(0);
|
||||
|
||||
// Soft-deleted pages disappear (visibility clause).
|
||||
await engine.softDeletePage('projects/chronomancer');
|
||||
const afterDelete = await engine.searchTitles('Chronomancer Codex Ledger', { limit: 10 });
|
||||
expect(afterDelete.map(r => r.slug)).not.toContain('projects/chronomancer');
|
||||
});
|
||||
|
||||
test('respects hard-exclude slug prefixes (test/ is excluded by default)', async () => {
|
||||
await engine.putPage('test/hidden-fixture', {
|
||||
type: 'note',
|
||||
title: 'Zanzibar Protocol Manifest',
|
||||
compiled_truth: 'fixture body',
|
||||
});
|
||||
const hits = await engine.searchTitles('Zanzibar Protocol Manifest', { limit: 10 });
|
||||
expect(hits.map(r => r.slug)).not.toContain('test/hidden-fixture');
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchKeyword — D2 AND→OR fallback', () => {
|
||||
async function seedQuantumPage(): Promise<void> {
|
||||
await engine.putPage('notes/quantum', {
|
||||
type: 'note',
|
||||
title: 'Quantum Notes',
|
||||
compiled_truth: 'quantum lattice harmonics resonance experiments',
|
||||
});
|
||||
await engine.upsertChunks('notes/quantum', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'quantum lattice harmonics resonance experiments',
|
||||
chunk_source: 'compiled_truth',
|
||||
},
|
||||
]);
|
||||
}
|
||||
|
||||
test('one bad token no longer zeroes keyword recall (orFallback: true rescues)', async () => {
|
||||
await seedQuantumPage();
|
||||
// Strict AND fails ('zzzmissingtoken' is nowhere); OR fallback rescues.
|
||||
const hits = await engine.searchKeyword('quantum lattice harmonics zzzmissingtoken', {
|
||||
limit: 10,
|
||||
orFallback: true,
|
||||
});
|
||||
expect(hits.map(r => r.slug)).toContain('notes/quantum');
|
||||
});
|
||||
|
||||
test('WITHOUT the orFallback flag the one-bad-token query returns zero (F1: strict default)', async () => {
|
||||
await seedQuantumPage();
|
||||
// Precision consumers (countMentions, link-extraction, eval) call
|
||||
// searchKeyword without the flag — their strict-AND contract must hold.
|
||||
const hits = await engine.searchKeyword('quantum lattice harmonics zzzmissingtoken', { limit: 10 });
|
||||
expect(hits.length).toBe(0);
|
||||
});
|
||||
|
||||
test('strict-AND results stay preferred: no OR dilution when AND matches', async () => {
|
||||
await seedQuantumPage();
|
||||
await engine.putPage('notes/partial', {
|
||||
type: 'note',
|
||||
title: 'Partial Overlap',
|
||||
compiled_truth: 'quantum computing conference recap',
|
||||
});
|
||||
await engine.upsertChunks('notes/partial', [
|
||||
{ chunk_index: 0, chunk_text: 'quantum computing conference recap', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
// All four tokens co-occur only in notes/quantum → strict AND non-empty
|
||||
// → the OR retry must NOT fire (even with the flag SET), so the
|
||||
// partial-overlap page stays out.
|
||||
const hits = await engine.searchKeyword('quantum lattice harmonics resonance', {
|
||||
limit: 10,
|
||||
orFallback: true,
|
||||
});
|
||||
expect(hits.map(r => r.slug)).toContain('notes/quantum');
|
||||
expect(hits.map(r => r.slug)).not.toContain('notes/partial');
|
||||
});
|
||||
|
||||
test('single unmatched token returns empty (OR of one term is pointless)', async () => {
|
||||
await seedQuantumPage();
|
||||
const hits = await engine.searchKeyword('zzznothinghere', { limit: 10, orFallback: true });
|
||||
expect(hits.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildOrFallbackWebsearchQuery — pure', () => {
|
||||
test('joins tokens with OR', () => {
|
||||
expect(buildOrFallbackWebsearchQuery('alpha beta')).toBe('alpha OR beta');
|
||||
});
|
||||
test('returns null for <2 tokens', () => {
|
||||
expect(buildOrFallbackWebsearchQuery('alpha')).toBeNull();
|
||||
expect(buildOrFallbackWebsearchQuery('')).toBeNull();
|
||||
expect(buildOrFallbackWebsearchQuery(' ')).toBeNull();
|
||||
});
|
||||
test('F3: refuses queries with websearch operators (negation must not resurrect)', () => {
|
||||
// A `-bar` exclusion relaxed to `foo OR bar` would MATCH the excluded
|
||||
// term; a quoted phrase would degrade to a bag of words. No fallback.
|
||||
expect(buildOrFallbackWebsearchQuery('foo -bar')).toBeNull();
|
||||
expect(buildOrFallbackWebsearchQuery('"alpha beta" gamma')).toBeNull();
|
||||
expect(buildOrFallbackWebsearchQuery('"alpha beta" -gamma')).toBeNull();
|
||||
});
|
||||
test('interior hyphens are not operators — still relaxed', () => {
|
||||
expect(buildOrFallbackWebsearchQuery('alpha-beta gamma')).toBe('alpha OR beta OR gamma');
|
||||
});
|
||||
test('drops literal OR/AND words so they cannot re-parse as operators', () => {
|
||||
expect(buildOrFallbackWebsearchQuery('alpha or beta')).toBe('alpha OR beta');
|
||||
expect(buildOrFallbackWebsearchQuery('alpha AND beta')).toBe('alpha OR beta');
|
||||
// Only operator words survive tokenization → nothing left to relax.
|
||||
expect(buildOrFallbackWebsearchQuery('or and')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch wiring — title arm reaches the fused result set', () => {
|
||||
test('exact-title query surfaces the page through hybridSearch (keyword-only path)', async () => {
|
||||
await seedTitleOnlyPage();
|
||||
const results = await hybridSearch(engine, 'Chronomancer Codex Ledger', { limit: 5 });
|
||||
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
|
||||
});
|
||||
|
||||
test('long exact-title query (>=8 content tokens) surfaces through hybridSearch', async () => {
|
||||
const longTitle = 'Emerald Falcon Doctrine Quarterly Synthesis Report Alpha Bravo Charlie Delta';
|
||||
await engine.putPage('reports/emerald-falcon', {
|
||||
type: 'note',
|
||||
title: longTitle,
|
||||
compiled_truth: 'An annual planning artifact.',
|
||||
});
|
||||
await engine.upsertChunks('reports/emerald-falcon', [
|
||||
{ chunk_index: 0, chunk_text: 'An annual planning artifact.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
const results = await hybridSearch(engine, longTitle, { limit: 5 });
|
||||
expect(results.map(r => r.slug)).toContain('reports/emerald-falcon');
|
||||
});
|
||||
|
||||
test('body-only queries still work (no regression from the extra arm)', async () => {
|
||||
await seedTitleOnlyPage();
|
||||
const results = await hybridSearch(engine, 'scheduling practices planning', { limit: 5 });
|
||||
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
|
||||
});
|
||||
|
||||
test('hybrid keyword arm still opts into the OR fallback (F1: QA-verified behavior preserved)', async () => {
|
||||
await seedTitleOnlyPage();
|
||||
// One bad token against body text: direct searchKeyword (no flag) finds
|
||||
// nothing, but hybridSearch sets orFallback for its recall arm.
|
||||
const q = 'scheduling practices zzzmissingtoken';
|
||||
expect((await engine.searchKeyword(q, { limit: 5 })).length).toBe(0);
|
||||
const results = await hybridSearch(engine, q, { limit: 5 });
|
||||
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
|
||||
});
|
||||
});
|
||||
@@ -40,6 +40,7 @@ function ctxWithInbox(
|
||||
data,
|
||||
attempts_made: 0,
|
||||
signal: new AbortController().signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: new AbortController().signal,
|
||||
async updateProgress(p: unknown) { progress.push(p); },
|
||||
async updateTokens() {},
|
||||
|
||||
@@ -90,6 +90,7 @@ async function makeCtx(input: unknown): Promise<MinionJobContext> {
|
||||
data: (input as Record<string, unknown>) ?? {},
|
||||
attempts_made: 0,
|
||||
signal: ac.signal,
|
||||
deadlineAtMs: null,
|
||||
shutdownSignal: shutdown.signal,
|
||||
async updateProgress() {},
|
||||
async updateTokens() {},
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
/**
|
||||
* #2964 — sync phase self-heals a never-git-initialized default brain dir.
|
||||
*
|
||||
* A legacy `sync.repo_path`-anchored default brain can reach `performSync`
|
||||
* pointed at a directory that was never `git init`-ed (predates git-backed
|
||||
* sync, or was rsync'd from another machine without its `.git`). Before
|
||||
* this fix, `discoverGitRoot` threw unconditionally and the dream cycle's
|
||||
* sync phase failed every night with no self-recovery, even though
|
||||
* `doctor`'s sync checks reported "ok" (for an unrelated reason — they
|
||||
* only look at the `sources` table in a way this brain shape doesn't hit).
|
||||
*
|
||||
* gbrain owns that directory outright, so the fix self-heals by `git
|
||||
* init`-ing it and capturing the current on-disk state as the sync
|
||||
* baseline. Ownership is proven by VALUE — the resolved `repoPath` must
|
||||
* realpath-equal gbrain's own anchor — not by whether
|
||||
* `opts.repoPath`/`opts.sourceId` happen to be set:
|
||||
*
|
||||
* - Gating on `!opts.repoPath` (round 3) would have made self-heal never
|
||||
* fire on `runPhaseSync` (dream cycle), which always resolves the
|
||||
* anchor itself and passes it through explicitly as `opts.repoPath`.
|
||||
* - Gating on `!opts.sourceId` (round 4) would ALSO never fire in
|
||||
* practice: migration `sources_table_additive` seeds a `'default'`
|
||||
* source row whose `local_path` mirrors `sync.repo_path` on every
|
||||
* brain that's run it (i.e. virtually all installed brains today), so
|
||||
* both the dream cycle and bare `gbrain sync` resolve
|
||||
* `sourceId: 'default'`, never `undefined`, in reality — a fresh test
|
||||
* brain's null `local_path` masked this (Codex review round 5).
|
||||
*
|
||||
* The actual boundary implemented by `isAnchorOwnedSyncPath`: `sourceId`
|
||||
* must be `undefined` OR exactly `'default'` (gbrain's own bootstrap
|
||||
* identity — a DIFFERENT id is what an explicit `sources add <id> --path
|
||||
* <dir>` registration of a user's own external directory looks like),
|
||||
* AND the resolved `repoPath` must realpath-equal the LIVE anchor for
|
||||
* that same identity. A caller-supplied path that does not match (a
|
||||
* registered non-default source, or an admin-scope
|
||||
* `submit_job({name:'sync', data:{repoPath}})` MCP call with an
|
||||
* unrelated path) must keep failing loudly rather than being silently
|
||||
* git-initialized without consent.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
function mdPage(title: string, body = 'Content.'): string {
|
||||
return `---\ntype: note\ntitle: ${title}\n---\n\n${body}`;
|
||||
}
|
||||
|
||||
describe('#2964: sync auto-inits a never-git-initialized default brain dir', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let dir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
dir = mkdtempSync(join(tmpdir(), 'gbrain-2964-'));
|
||||
writeFileSync(join(dir, 'page1.md'), mdPage('Page 1'));
|
||||
writeFileSync(join(dir, 'page2.md'), mdPage('Page 2'));
|
||||
// The self-heal-eligible anchor: gbrain's own persisted config, not a
|
||||
// caller-supplied --repo / job.data.repoPath (those are proven by
|
||||
// VALUE against this anchor, not by mere absence — see file docstring).
|
||||
await engine.setConfig('sync.repo_path', dir);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (dir) rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('anchor-resolved sync (no repoPath, no sourceId) on a non-git dir auto-inits git and imports files', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
|
||||
const result = await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
expect(existsSync(join(dir, '.git'))).toBe(true);
|
||||
expect(await engine.getPage('page1')).not.toBeNull();
|
||||
expect(await engine.getPage('page2')).not.toBeNull();
|
||||
});
|
||||
|
||||
test('explicit repoPath matching the anchor still auto-inits (mirrors gbrain dream\'s sync phase)', async () => {
|
||||
// cycle.ts's runPhaseSync (the actual dream-cycle call site this bug
|
||||
// was filed against) always passes `repoPath: brainDir` explicitly —
|
||||
// it already resolved the anchor itself upstream and threads it
|
||||
// through. Gating self-heal on `!opts.repoPath` would silently never
|
||||
// fire here; ownership must be proven by matching the anchor's VALUE.
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
|
||||
const result = await performSync(engine, { repoPath: dir, noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
expect(existsSync(join(dir, '.git'))).toBe(true);
|
||||
});
|
||||
|
||||
test('a second sync after auto-init sees no changes (baseline commit captured current on-disk state)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const first = await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
expect(first.added).toBe(2);
|
||||
|
||||
// No new files, no explicit `full` — a real incremental sync against the
|
||||
// auto-init baseline. Before this fix there was no baseline to diff
|
||||
// against (sync errored outright); a naive fix that skipped the initial
|
||||
// commit would make this call re-report both files as "added" again.
|
||||
const second = await performSync(engine, { noPull: true, noEmbed: true });
|
||||
expect(second.status).not.toBe('first_sync');
|
||||
expect(second.added).toBe(0);
|
||||
expect(second.modified).toBe(0);
|
||||
});
|
||||
|
||||
test("sourceId='default' whose local_path mirrors the anchor still auto-inits (P1: the real installed-brain shape)", async () => {
|
||||
// Migration sources_table_additive seeds a 'default' source row with
|
||||
// local_path copied from sync.repo_path on every brain that's run it
|
||||
// — i.e. this, not a bare no-sourceId call, is what runPhaseSync/CLI
|
||||
// `gbrain sync` actually resolve to on a real installed brain.
|
||||
await engine.executeRaw(`UPDATE sources SET local_path = $1 WHERE id = 'default'`, [dir]);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
|
||||
const result = await performSync(engine, {
|
||||
repoPath: dir,
|
||||
sourceId: 'default',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
expect(existsSync(join(dir, '.git'))).toBe(true);
|
||||
});
|
||||
|
||||
test('a registered non-default local source (sourceId != default, no remote_url) on a non-git dir still throws — not auto-inited', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config) VALUES ('mysource', 'mysource', $1, '{}'::jsonb)`,
|
||||
[dir],
|
||||
);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, {
|
||||
repoPath: dir,
|
||||
sourceId: 'mysource',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
}),
|
||||
).rejects.toThrow(/git repository/i);
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
});
|
||||
|
||||
test('a caller-supplied repoPath that does NOT match the anchor still throws (P1: MCP submit_job arbitrary-path guard)', async () => {
|
||||
// Mirrors jobs.ts: submit_job({name:'sync', data:{repoPath}}) reaches
|
||||
// performSyncInner with sourceId left undefined whenever repoPath
|
||||
// doesn't match a registered source's local_path. Self-heal must not
|
||||
// fire for a path that isn't gbrain's own anchor, even with no
|
||||
// sourceId set — only exact anchor-value equality (the previous test)
|
||||
// is eligible.
|
||||
const other = mkdtempSync(join(tmpdir(), 'gbrain-2964-other-'));
|
||||
writeFileSync(join(other, 'unrelated.md'), mdPage('Unrelated'));
|
||||
try {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, { repoPath: other, noPull: true, noEmbed: true, full: true }),
|
||||
).rejects.toThrow(/git repository/i);
|
||||
expect(existsSync(join(other, '.git'))).toBe(false);
|
||||
} finally {
|
||||
rmSync(other, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--src-subpath on the anchor-resolved path still throws — not auto-inited (P2: subpath scope guard)', async () => {
|
||||
// A self-heal baseline commit runs `git add -A` at the git root before
|
||||
// any subpath-scoped file collection happens, so it would capture
|
||||
// sibling directories a --src-subpath sync never intended to touch.
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, {
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
}),
|
||||
).rejects.toThrow(/git repository/i);
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
});
|
||||
|
||||
test('--dry-run on the anchor-resolved path throws without writing anything to disk', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, { repoPath: dir, dryRun: true, noPull: true, noEmbed: true, full: true }),
|
||||
).rejects.toThrow(/git repository/i);
|
||||
// The whole point of --dry-run is "preview only" — it must never git-init
|
||||
// or commit on our behalf, even though this is otherwise self-heal-eligible.
|
||||
expect(existsSync(join(dir, '.git'))).toBe(false);
|
||||
});
|
||||
|
||||
test('unborn-HEAD recovery: a bare `git init` with zero commits (interrupted prior self-heal) still completes', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const { execSync } = await import('child_process');
|
||||
// Simulate a self-heal that ran `git init` but died before the baseline
|
||||
// commit landed (process killed, disk full, etc.) — `.git` exists so
|
||||
// discoverGitRoot succeeds, but `git rev-parse HEAD` still fails.
|
||||
execSync('git init -q', { cwd: dir });
|
||||
|
||||
const result = await performSync(engine, { repoPath: dir, noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
expect(execSync('git rev-parse HEAD', { cwd: dir }).toString().trim()).not.toBe('');
|
||||
});
|
||||
|
||||
test('db_only paths are excluded from the baseline commit even without gbrain.yml write support (P2: fail-closed exclusion)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const { mkdirSync } = await import('fs');
|
||||
const { execSync } = await import('child_process');
|
||||
mkdirSync(join(dir, 'private-cache'));
|
||||
writeFileSync(join(dir, 'private-cache', 'secret.bin'), 'binary-ish content');
|
||||
writeFileSync(
|
||||
join(dir, 'gbrain.yml'),
|
||||
'storage:\n db_only:\n - private-cache\n',
|
||||
);
|
||||
|
||||
await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(existsSync(join(dir, '.git'))).toBe(true);
|
||||
const tracked = execSync('git ls-files', { cwd: dir }).toString();
|
||||
expect(tracked).not.toContain('private-cache');
|
||||
});
|
||||
|
||||
test('db_only exclusion applies even when a pre-existing .gitignore already covers the same dir (round 9 P1: unconditional pathspec)', async () => {
|
||||
// Regression for the "check-ignore pre-filter" version of this logic:
|
||||
// when a dir is ALSO already covered by an existing .gitignore, git's
|
||||
// `-A` bails with an advisory "paths ignored... use -f" even though
|
||||
// the add otherwise succeeds. Exclusion must be unconditional and the
|
||||
// advisory must not surface as a hard failure.
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const { mkdirSync } = await import('fs');
|
||||
const { execSync } = await import('child_process');
|
||||
mkdirSync(join(dir, 'private-cache'));
|
||||
writeFileSync(join(dir, 'private-cache', 'secret.bin'), 'binary-ish content');
|
||||
writeFileSync(join(dir, 'gbrain.yml'), 'storage:\n db_only:\n - private-cache\n');
|
||||
writeFileSync(join(dir, '.gitignore'), 'private-cache/\n');
|
||||
|
||||
const result = await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
const tracked = execSync('git ls-files', { cwd: dir }).toString();
|
||||
expect(tracked).not.toContain('private-cache');
|
||||
});
|
||||
|
||||
test('a comment merely mentioning db_only does not false-positive the sniff test (round 9 P2)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// No `storage:` section at all — just a comment mentioning the word.
|
||||
// A bare substring search would wrongly refuse this brain forever.
|
||||
writeFileSync(join(dir, 'gbrain.yml'), '# db_only handling: TBD, not configured yet\n');
|
||||
|
||||
const result = await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
});
|
||||
|
||||
test('a gbrain.yml that mentions db_only but resolves no dirs refuses the baseline commit (round 6 P2: unsupported-syntax sniff test)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const { execSync } = await import('child_process');
|
||||
// Flow-style array — valid YAML, but the narrow custom parser only
|
||||
// handles block-style lists, so loadStorageConfig warns and resolves
|
||||
// an empty db_only list rather than throwing.
|
||||
writeFileSync(join(dir, 'gbrain.yml'), 'storage:\n db_only: [private-cache/]\n');
|
||||
|
||||
await expect(
|
||||
performSync(engine, { noPull: true, noEmbed: true, full: true }),
|
||||
).rejects.toThrow(/db_only/i);
|
||||
// `git init` (site 1's first step) already ran before the sniff-test
|
||||
// guard (inside createSyncBaselineCommit) refused — that's fine, it's
|
||||
// the same "unborn repo" state the round-6-P1 index-rebuild test above
|
||||
// recovers from on a later retry, which would hit this same guard and
|
||||
// refuse again until gbrain.yml is fixed. What must NOT happen is a
|
||||
// commit landing with unknown/unexcluded content.
|
||||
expect(existsSync(join(dir, '.git'))).toBe(true);
|
||||
let hasCommit = true;
|
||||
try {
|
||||
execSync('git rev-parse HEAD', { cwd: dir, stdio: 'pipe' });
|
||||
} catch {
|
||||
hasCommit = false;
|
||||
}
|
||||
expect(hasCommit).toBe(false);
|
||||
});
|
||||
|
||||
test('unborn-HEAD recovery drops stale staged content the exclusion pathspec now wants excluded (round 6 P1: index rebuild)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const { mkdirSync } = await import('fs');
|
||||
const { execSync } = await import('child_process');
|
||||
mkdirSync(join(dir, 'private-cache'));
|
||||
writeFileSync(join(dir, 'private-cache', 'secret.bin'), 'binary-ish content');
|
||||
writeFileSync(join(dir, 'gbrain.yml'), 'storage:\n db_only:\n - private-cache\n');
|
||||
// Simulate an interrupted workflow that left this file staged in an
|
||||
// unborn repo BEFORE gbrain's self-heal ever ran.
|
||||
execSync('git init -q', { cwd: dir });
|
||||
execSync('git add private-cache/secret.bin', { cwd: dir });
|
||||
|
||||
await performSync(engine, { noPull: true, noEmbed: true, full: true });
|
||||
|
||||
const tracked = execSync('git ls-files', { cwd: dir }).toString();
|
||||
expect(tracked).not.toContain('private-cache');
|
||||
});
|
||||
|
||||
});
|
||||
@@ -28,7 +28,8 @@ describe('#1433 — isSyncable / unsyncableReason are duals of one classifier',
|
||||
{ path: 'RESOLVER.md', expected: 'metafile', note: 'top-level master routing config (closes #345)' },
|
||||
{ path: 'brain/RESOLVER.md', expected: 'metafile', note: 'RESOLVER.md anywhere is metafile (closes #345)' },
|
||||
{ path: 'people/alice.txt', expected: 'strategy', note: '.txt rejected by markdown strategy' },
|
||||
{ path: 'ops/scratch/note.md', expected: 'pruned-dir', note: 'ops/ is pruned' },
|
||||
{ path: 'ops/scratch/note.md', expected: null, note: 'ops/ is ordinary content, not pruned (#2404)' },
|
||||
{ path: 'vendor/pkg/note.md', expected: 'pruned-dir', note: 'vendor/ is pruned' },
|
||||
{ path: '.git/notes.md', expected: 'pruned-dir', note: 'hidden dir pruned' },
|
||||
{ path: 'node_modules/foo/README.md', expected: 'pruned-dir', note: 'node_modules pruned' },
|
||||
];
|
||||
|
||||
@@ -0,0 +1,385 @@
|
||||
/**
|
||||
* #753/#774 — --src-subpath + --exclude monorepo subdir-source support.
|
||||
*
|
||||
* A single git repo can hold N logical sources at subdirectories (wiki/,
|
||||
* memory/, ...). `gbrain sync --src-subpath wiki` (or passing the subdir
|
||||
* directly as the repo path) scopes file walking + imports to the subdir
|
||||
* while git operations (pull, rev-parse, diff) run at the discovered repo
|
||||
* root. Slugs stay git-root-relative (`wiki/page1`) so full and incremental
|
||||
* syncs of the same scope agree.
|
||||
*
|
||||
* Security pins (the point of the feature's guards):
|
||||
* NAV-1/NAV-2 — `--src-subpath ../escape` and a symlinked subdir pointing
|
||||
* outside the repo are realpath-checked and rejected before any git op.
|
||||
* NAV-1 TOCTOU — per-file realpath checks during the incremental import
|
||||
* drain (see the isPathSafe guard in sync.ts's importOnePath).
|
||||
* NAV-4 — an --exclude set that filters out everything warns loudly.
|
||||
*
|
||||
* Regression note: against pre-#774 master every subdir test fails with
|
||||
* "Not a git repository".
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, symlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
// Helper: create a minimal valid markdown file
|
||||
function mdPage(title: string, body = 'Content.'): string {
|
||||
return `---\ntype: note\ntitle: ${title}\n---\n\n${body}`;
|
||||
}
|
||||
|
||||
// Helper: init a git repo with author identity
|
||||
function gitInit(dir: string): void {
|
||||
execSync('git init', { cwd: dir, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: dir, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: dir, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
// Helper: stage + commit everything in a git repo
|
||||
function gitCommit(dir: string, msg = 'initial'): void {
|
||||
execSync('git add -A', { cwd: dir, stdio: 'pipe' });
|
||||
execSync(`git commit -m "${msg}"`, { cwd: dir, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
describe('sync monorepo subdir-source support (#753/#774)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-monorepo-'));
|
||||
gitInit(repoPath);
|
||||
mkdirSync(join(repoPath, 'wiki'), { recursive: true });
|
||||
mkdirSync(join(repoPath, 'memory'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'wiki', 'page1.md'), mdPage('Wiki Page 1'));
|
||||
writeFileSync(join(repoPath, 'wiki', 'page2.md'), mdPage('Wiki Page 2'));
|
||||
writeFileSync(join(repoPath, 'memory', 'note1.md'), mdPage('Memory Note 1'));
|
||||
writeFileSync(join(repoPath, 'memory', 'note2.md'), mdPage('Memory Note 2'));
|
||||
gitCommit(repoPath);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// Back-compat: sync at git root (no srcSubpath) still works
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('back-compat: sync at git root without srcSubpath imports all files', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(4); // wiki/page1 + wiki/page2 + memory/note1 + memory/note2
|
||||
// Slug shape unchanged for git-root syncs.
|
||||
expect(await engine.getPage('wiki/page1')).not.toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// Auto-discovery: repoPath IS a non-git-root subdir
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('auto-discovery: repoPath is a git subdir — discoverGitRoot succeeds', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// Pass the wiki/ subdir directly as repoPath (no explicit srcSubpath).
|
||||
// Pre-#774: throws "Not a git repository".
|
||||
// Post-#774: gitContextRoot = repo root, syncScopeRoot = wiki/.
|
||||
const result = await performSync(engine, {
|
||||
repoPath: join(repoPath, 'wiki'),
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2); // only wiki/page1 + wiki/page2
|
||||
// Slugs are git-root-relative in BOTH spellings (subdir repoPath and
|
||||
// --src-subpath) so full and incremental syncs of the same scope agree.
|
||||
expect(await engine.getPage('wiki/page1')).not.toBeNull();
|
||||
expect(await engine.getPage('page1')).toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// srcSubpath explicit flag: scope to subdir from git root
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('--src-subpath wiki: only wiki/ files are imported', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
// Verify the imported slugs are from wiki/ only (git-root-relative)
|
||||
const wikiPage = await engine.getPage('wiki/page1');
|
||||
expect(wikiPage).not.toBeNull();
|
||||
const memoryPage = await engine.getPage('memory/note1');
|
||||
expect(memoryPage).toBeNull();
|
||||
});
|
||||
|
||||
test('--src-subpath memory: only memory/ files are imported', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'memory',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(2);
|
||||
const memoryPage = await engine.getPage('memory/note1');
|
||||
expect(memoryPage).not.toBeNull();
|
||||
const wikiPage = await engine.getPage('wiki/page1');
|
||||
expect(wikiPage).toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// Two sources in one repo, scoped independently
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('2 sources in 1 repo: sync each scope independently, no cross-contamination', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
const wikiResult = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(wikiResult.status).toBe('first_sync');
|
||||
expect(wikiResult.added).toBe(2);
|
||||
|
||||
// Reset only page state, keep the engine connected for second sync
|
||||
await resetPgliteState(engine);
|
||||
|
||||
const memResult = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'memory',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(memResult.status).toBe('first_sync');
|
||||
expect(memResult.added).toBe(2);
|
||||
|
||||
// After memory sync, memory pages exist and wiki pages don't
|
||||
expect(await engine.getPage('memory/note1')).not.toBeNull();
|
||||
expect(await engine.getPage('wiki/page1')).toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// Incremental sync respects the scope (the gap #774 left untested)
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('incremental --src-subpath: only in-scope diff paths are processed', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
const first = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(first.status).toBe('first_sync');
|
||||
|
||||
// Commit 2: touch one file in each scope + add one wiki file.
|
||||
writeFileSync(join(repoPath, 'wiki', 'page1.md'), mdPage('Wiki Page 1', 'Updated.'));
|
||||
writeFileSync(join(repoPath, 'memory', 'note1.md'), mdPage('Memory Note 1', 'Updated.'));
|
||||
writeFileSync(join(repoPath, 'wiki', 'page3.md'), mdPage('Wiki Page 3'));
|
||||
gitCommit(repoPath, 'second');
|
||||
|
||||
const second = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
});
|
||||
expect(second.status).toBe('synced');
|
||||
expect(second.added).toBe(1); // wiki/page3 only — memory change filtered by scope
|
||||
expect(second.modified).toBe(1); // wiki/page1
|
||||
expect(await engine.getPage('wiki/page3')).not.toBeNull();
|
||||
expect(await engine.getPage('memory/note1')).toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// Path-traversal sanitization (NAV-1 + NAV-2)
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('path-traversal: --src-subpath ../escape is rejected before any git op', async () => {
|
||||
const outsideDir = mkdtempSync(join(tmpdir(), 'gbrain-escape-'));
|
||||
try {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: '../' + outsideDir.split('/').pop(),
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
}),
|
||||
).rejects.toThrow(/outside git repo|does not exist/i);
|
||||
} finally {
|
||||
rmSync(outsideDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('path-traversal: symlink subdir pointing outside repo is rejected (NAV-1 TOCTOU)', async () => {
|
||||
const outsideDir = mkdtempSync(join(tmpdir(), 'gbrain-sym-target-'));
|
||||
writeFileSync(join(outsideDir, 'secret.md'), mdPage('Secret'));
|
||||
const symlinkPath = join(repoPath, 'symlink-escape');
|
||||
try {
|
||||
symlinkSync(outsideDir, symlinkPath);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await expect(
|
||||
performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'symlink-escape',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
}),
|
||||
).rejects.toThrow(/outside git repo/i);
|
||||
} finally {
|
||||
rmSync(outsideDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('path-traversal: absolute --src-subpath outside the repo is rejected', async () => {
|
||||
const outsideDir = mkdtempSync(join(tmpdir(), 'gbrain-abs-escape-'));
|
||||
writeFileSync(join(outsideDir, 'secret.md'), mdPage('Secret'));
|
||||
try {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// path.join(repoPath, '/abs/path') keeps the traversal relative, but a
|
||||
// crafted subpath can still resolve outside via ..-segments; both are
|
||||
// caught by the same realpath containment check.
|
||||
await expect(
|
||||
performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: join('..', '..', outsideDir.slice(1)),
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
}),
|
||||
).rejects.toThrow(/outside git repo|does not exist/i);
|
||||
} finally {
|
||||
rmSync(outsideDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// --exclude: repeatable glob pattern flag
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('--exclude: single pattern excludes matching files from full sync', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// Sync wiki/ but exclude page2.md (patterns are scope-relative)
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
exclude: ['page2.md'],
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(1); // only page1 (page2 excluded)
|
||||
expect(await engine.getPage('wiki/page1')).not.toBeNull();
|
||||
expect(await engine.getPage('wiki/page2')).toBeNull();
|
||||
});
|
||||
|
||||
test('--exclude: glob pattern with wildcard', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// Exclude all files matching *2.md
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
exclude: ['*2.md'],
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
expect(result.added).toBe(1); // only page1 (page2 excluded by *2.md)
|
||||
});
|
||||
|
||||
test('--exclude applies to the incremental path too', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const first = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
expect(first.status).toBe('first_sync');
|
||||
|
||||
writeFileSync(join(repoPath, 'wiki', 'draft-a.md'), mdPage('Draft A'));
|
||||
writeFileSync(join(repoPath, 'wiki', 'page3.md'), mdPage('Wiki Page 3'));
|
||||
gitCommit(repoPath, 'drafts');
|
||||
|
||||
const second = await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
exclude: ['draft-*.md'],
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
});
|
||||
expect(second.status).toBe('synced');
|
||||
expect(second.added).toBe(1); // page3 only; draft-a excluded
|
||||
expect(await engine.getPage('wiki/page3')).not.toBeNull();
|
||||
expect(await engine.getPage('wiki/draft-a')).toBeNull();
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// --exclude '**/*' emits warning (NAV-4)
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
test('--exclude **/* emits warning when all files are excluded (NAV-4)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const warnMessages: string[] = [];
|
||||
const origWarn = console.warn;
|
||||
console.warn = (...args: unknown[]) => {
|
||||
warnMessages.push(args.join(' '));
|
||||
origWarn(...args);
|
||||
};
|
||||
try {
|
||||
await performSync(engine, {
|
||||
repoPath,
|
||||
srcSubpath: 'wiki',
|
||||
exclude: ['**/*'],
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
full: true,
|
||||
});
|
||||
} finally {
|
||||
console.warn = origWarn;
|
||||
}
|
||||
const hasExcludeWarn = warnMessages.some(m => m.includes('--exclude') || m.includes('No files matched'));
|
||||
expect(hasExcludeWarn).toBe(true);
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user