Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 3487e4b255 fix(embed): thread write column into sumStaleChunkChars — sync cost gate followed the legacy predicate (#1262)
Review follow-up: the PR threaded the write-side column through
countStaleChunks/listStaleChunks but not sumStaleChunkChars, so the
sync cost gate counted an alt-column brain's fully-embedded corpus as
phantom backlog on every gate (inflating the --full estimate and the
deferred-mode backlog note). Both engines already share
buildStaleChunkWhere, so this is a type widening + one call-site
thread + a contrast assertion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 10:59:36 -07:00
159ddc3249 fix(embed): dynamic embedding column write target for upsertChunks + stale scans (#1262)
PostgresEngine/PGLiteEngine.upsertChunks hardcoded the legacy `embedding`
column (INSERT list + `::vector` cast + ON CONFLICT clauses), so a brain
with a registered alternate embedding column (e.g. halfvec(2560)) failed
every put_page/import/sync/embed write with a dimension mismatch — the
embedding_columns registry had read-side consumers only (PR #1164).

Fix (write-side symmetric to the read-side resolver):
- `resolveWriteColumn(cfg)` + `resolveWriteColumnForEngine(engine)` in
  search/embedding-column.ts: user-declared registry entry whose provider
  matches the current embedding model wins; no match => undefined (legacy
  column). Builtins are never consulted so the multimodal builtin can't
  capture text writes.
- `upsertChunks` accepts a caller-resolved `embeddingColumn` descriptor in
  BOTH engines; the target column + cast (`::vector` / `::halfvec(N)`)
  and the v0.40.3.0 D24 ON CONFLICT race-fix CASE arms follow the column.
- Stale scans follow the write column: `countStaleChunks`,
  `listStaleChunks` (both cursor arms), and the shared
  buildStaleChunkWhere accept the descriptor — without this, an
  alt-column brain re-selects (and re-pays for) already-embedded chunks
  forever.
- Boundary threading: runEmbedCore (embedPage/embedAll/embedAllStale),
  embedStaleForSource + the embed-backfill handler, importFromContent /
  importCodeFile / withImportTransaction, and the contextual-retrieval
  re-embed transaction all resolve once and pass the descriptor.
- `preflightDimMismatch` skips the legacy-column dim comparison when a
  non-default write column resolved (it would otherwise hard-block embed
  runs on alt-column brains).

Tests: resolveWriteColumn unit coverage; PGLite e2e for the write target,
D24 preserve-on-reupsert, stale-scan contrast, and an `embed --stale
--dry-run` convergence integration; Postgres e2e twins (DATABASE_URL-gated).

Takeover of PR #1263 rebased onto current master (D24 ON CONFLICT
semantics, batchRetry wrapper, signature-stale + embed-backfill paths).

Fixes #1262

Co-authored-by: DmitryBMsk <DmitryBMsk@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 15:04:38 -07:00
88 changed files with 925 additions and 2488 deletions
+1 -5
View File
@@ -206,11 +206,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
# 22, not 15: under parallel PR load the PGLite WASM cold-starts stretch a
# shard past 15 min while every test is still passing — the timeout then
# cancels the job and the test-status gate reads it as a failure. 13 runs
# died this way on 2026-07-21/22 alone.
timeout-minutes: 22
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
+2 -2
View File
@@ -71,8 +71,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
+1 -3
View File
@@ -131,9 +131,7 @@ into gbrain so other clients can scaffold it. Default behavior:
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted. Harvest must preserve
the top-level OpenClaw-native plugin fields (`id`, `configSchema`, `contracts`)
because OpenClaw validates those before it can install the package.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
@@ -233,14 +233,13 @@ keep it or `git checkout` to throw it away. Nothing is committed for you.
**For a skill that ships with gbrain** (anything under the gbrain repo's own
`skills/`): SkillOpt refuses to overwrite it by default and writes the winner to
`skills/<name>/skillopt/proposed.md` instead (while keeping `best.md` as the
optimizer's current-best pointer), so an optimization pass can never silently
mutate a skill other people depend on. Two ways to handle that:
`skills/<name>/skillopt/best.md` instead, so an optimization pass can never
silently mutate a skill other people depend on. Two ways to handle that:
```bash
# See the proposed improvement without touching SKILL.md (works for ANY skill):
gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
# → writes skills/meeting-prep/skillopt/proposed.md, updates best.md, and prints the proposal path.
# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+2 -2
View File
@@ -1565,8 +1565,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
-1
View File
@@ -1,5 +1,4 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.32.3.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
+1 -2
View File
@@ -266,5 +266,4 @@ editorial pass.
(e.g. `src/commands/<slug>.ts` if the host SKILL.md declares it
in frontmatter)
- gbrain's `openclaw.plugin.json` — adds the slug to `skills:`
array, sorted alphabetically, without removing OpenClaw-native plugin fields
like `id`, `configSchema`, or `contracts`
array, sorted alphabetically
+1 -3
View File
@@ -57,8 +57,6 @@ This mode guarantees:
- `skills/manifest.json` lists every skill directory
- `skills/RESOLVER.md` references every skill in the manifest
- `openclaw.plugin.json` `skills[]` round-trips with both
- `openclaw.plugin.json` keeps OpenClaw install-required native plugin fields
(`id`, object `configSchema`, and `contracts.contextEngines` when applicable)
- No MECE violations (duplicate triggers across skills)
### Phases
@@ -74,7 +72,7 @@ This mode guarantees:
### Automation
```bash
bun test test/skills-conformance.test.ts test/resolver.test.ts test/openclaw-plugin-manifest.test.ts
bun test test/skills-conformance.test.ts test/resolver.test.ts
```
The CI-gated check is the package.json `test` script.
+1 -8
View File
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -78,8 +78,6 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// maintain (#3015) prints its own usage block (modes + not-auto-applied list).
'maintain',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
@@ -1759,11 +1757,6 @@ async function handleCliOnly(command: string, args: string[]) {
await runOrphans(engine, args);
break;
}
case 'maintain': {
const { runMaintain } = await import('./commands/maintain.ts');
await runMaintain(engine, args);
break;
}
// v0.32.7 CJK wave — post-upgrade markdown re-chunk sweep.
// v0.36 Phase 3 wave — `gbrain reindex --multimodal` re-embeds content_chunks
// into the unified Voyage multimodal-3 column.
-9
View File
@@ -1186,15 +1186,6 @@ function writeWrapperScript(repoPath: string): string {
# OPENAI/ANTHROPIC keys exported in zshenv reach autopilot.
[ -f ~/.zshenv ] && source ~/.zshenv 2>/dev/null
source ~/.zshrc 2>/dev/null || source ~/.bashrc 2>/dev/null || true
# Belt-and-suspenders PATH fix. ~/.bashrc ships with a non-interactive guard
# (\`case $- in *i*) ;; *) return;; esac\`) that exits early when launched from
# cron/systemd/launchd — so its PATH exports never reach this subprocess.
# Without bun on PATH, the exec'd gbrain (a \`#!/usr/bin/env bun\` script) fails
# silently with "env: bun: No such file or directory" and leaves a stale
# lockfile that blocks every subsequent tick. Prepending ~/.bun/bin here
# keeps the wrapper self-contained regardless of which init file the OS
# loaded.
export PATH="$HOME/.bun/bin:$PATH"
exec '${safeGbrainPath}' autopilot --repo '${safeRepoPath}'
`;
writeFileSync(wrapperPath, wrapper, { mode: 0o755 });
-10
View File
@@ -37,19 +37,9 @@ export async function findCodeDef(
// trigger) are first-class definitions in the SQL sense. The chunker's
// normalizeSymbolType maps create_table → 'table' etc, so adding the SQL
// kinds here is what makes `gbrain code-def users` work against SQL.
// Method-level + member definitions. normalizeSymbolType only canonicalizes
// some node types; the rest fall through `type.replace(/_/g, ' ')`, so
// tree-sitter's method_declaration → 'method declaration', struct_specifier →
// 'struct specifier', protocol_declaration → 'protocol declaration', etc.
// Without these, code-def is blind to every method, constructor, field, C
// struct, and Swift protocol — which is most of an OO codebase. The plain
// 'struct' entry above never matched for the same reason (C emits the
// 'struct specifier' fallback form).
const DEF_TYPES = [
'function', 'class', 'interface', 'type', 'enum', 'struct', 'trait', 'module', 'contract',
'table', 'view', 'index', 'procedure', 'schema', 'database', 'trigger',
'method declaration', 'method definition', 'constructor declaration',
'field declaration', 'field definition', 'struct specifier', 'protocol declaration',
];
const params: unknown[] = [symbol, limit];
let whereLang = '';
+2 -1
View File
@@ -4963,7 +4963,8 @@ export async function buildChecks(
message:
`${unmatched}/${sample.length} conversation pages (${unmatchedPct.toFixed(1)}%) match NO built-in pattern. ` +
`Breakdown: ${breakdown}. ` +
`Investigate: gbrain conversation-parser scan <slug>`,
`Investigate: gbrain conversation-parser scan <slug> | ` +
`Enable LLM fallback (opt-in): gbrain config set conversation_parser.llm_fallback_enabled true`,
});
} else {
checks.push({
+47 -47
View File
@@ -1,6 +1,7 @@
import type { BrainEngine } from '../core/engine.ts';
import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts';
import type { ChunkInput } from '../core/types.ts';
import type { ChunkInput, ResolvedColumn } from '../core/types.ts';
import { resolveWriteColumnForEngine } from '../core/search/embedding-column.ts';
import { chunkText } from '../core/chunkers/recursive.ts';
import { createProgress, type ProgressReporter } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -183,8 +184,13 @@ export class EmbeddingDimMismatchError extends Error {
* fresh-install bug class at the very first invocation instead of letting
* the worker pool hammer N pages with raw 22000 errors.
*/
async function preflightDimMismatch(engine: BrainEngine, dryRun: boolean): Promise<void> {
async function preflightDimMismatch(engine: BrainEngine, dryRun: boolean, embeddingColumn?: ResolvedColumn): Promise<void> {
if (dryRun) return; // dry-run never embeds, no risk
// #1262: an alt-column brain writes to `embeddingColumn`, not the legacy
// `embedding` column — the legacy column's dims are irrelevant, and the
// registry entry (validated at resolve time) pins the target's dims. Only
// the legacy default path needs the schema-vs-gateway dim comparison.
if (embeddingColumn && embeddingColumn.name !== 'embedding') return;
const { readContentChunksEmbeddingDim, embeddingMismatchMessage } = await import('../core/embedding-dim-check.ts');
const { getEmbeddingDimensions, getEmbeddingModel } = await import('../core/ai/gateway.ts');
let existing;
@@ -238,7 +244,12 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
// v0.37.11.0 (Lane D.2): pre-flight dim-mismatch check. Catches the headline
// fresh-install bug class before the worker pool spends 20 parallel calls
// hitting raw Postgres dimension errors.
await preflightDimMismatch(engine, !!opts.dryRun);
// #1262: resolve the write-side embedding column ONCE at the boundary
// (merged config + gateway model) and thread the descriptor through every
// upsertChunks / stale-scan below. undefined => legacy `embedding` column.
const embeddingColumn = await resolveWriteColumnForEngine(engine);
await preflightDimMismatch(engine, !!opts.dryRun, embeddingColumn);
const result: EmbedResult = {
embedded: 0,
@@ -253,7 +264,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
for (const s of opts.slugs) {
if (isAborted(opts.signal)) break; // #1737: stop the per-slug loop on abort
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal);
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal, embeddingColumn);
} catch (e: unknown) {
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
@@ -347,7 +358,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
catchUp: opts.catchUp,
pacer,
paceMaxConcurrency,
}, opts.signal);
}, opts.signal, embeddingColumn);
} finally {
// E1: surface pacing telemetry (human + structured) when pacing was on.
const snap = pacer.snapshot();
@@ -376,7 +387,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
return result;
}
if (opts.slug) {
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId, opts.signal);
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId, opts.signal, embeddingColumn);
return result;
}
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
@@ -521,8 +532,13 @@ async function embedPage(
result: EmbedResult,
sourceId?: string,
signal?: AbortSignal,
embeddingColumn?: ResolvedColumn,
) {
const opts = sourceId ? { sourceId } : undefined;
// #1262: write-side descriptor rides only on WRITE calls (upsertChunks).
const chunkOpts = (sourceId || embeddingColumn)
? { ...(sourceId && { sourceId }), ...(embeddingColumn && { embeddingColumn }) }
: undefined;
const page = await engine.getPage(slug, opts);
if (!page) {
throw new Error(`Page not found: ${slug}`);
@@ -554,7 +570,7 @@ async function embedPage(
}
if (inputs.length > 0) {
await engine.upsertChunks(slug, inputs, opts);
await engine.upsertChunks(slug, inputs, chunkOpts);
chunks = await engine.getChunks(slug, opts);
}
}
@@ -581,7 +597,7 @@ async function embedPage(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
const updated: ChunkInput[] = chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -589,7 +605,7 @@ async function embedPage(
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(slug, updated, opts);
await engine.upsertChunks(slug, updated, chunkOpts);
// v0.41.31: stamp provenance so a later model/dims swap is detectable as
// stale. embedPage is the per-slug path used by `gbrain embed <slug>` AND
// by `gbrain sync`'s post-import embed step (runEmbedCore({slugs})).
@@ -605,31 +621,6 @@ async function embedPage(
slog(`${slug}: embedded ${toEmbed.length} chunks`);
}
/**
* Carry code-chunk metadata (language, symbol_name, symbol_type, line range,
* parent scope, doc comment, qualified name) from a loaded Chunk back into a
* ChunkInput destined for upsertChunks.
*
* Issue #769: every re-embed used to strip these fields, and upsertChunks
* overwrites (does not COALESCE) the metadata columns from EXCLUDED, so
* each pass clobbered code-def's primary index to NULL. Pulling the
* preservation into one helper keeps the three re-embed call sites
* (embedPage, embedAll non-stale, embedAllStale) in lock-step.
*/
function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
return {
...base,
language: loaded.language ?? undefined,
symbol_name: loaded.symbol_name ?? undefined,
symbol_type: loaded.symbol_type ?? undefined,
start_line: loaded.start_line ?? undefined,
end_line: loaded.end_line ?? undefined,
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
doc_comment: loaded.doc_comment ?? undefined,
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
};
}
async function embedAll(
engine: BrainEngine,
staleOnly: boolean,
@@ -647,6 +638,7 @@ async function embedAll(
paceMaxConcurrency?: number;
},
signal?: AbortSignal,
embeddingColumn?: ResolvedColumn,
) {
// v0.41.31: current embedding provenance signature. Stamped onto pages
// when their chunks are (re)embedded so a later model/dimension swap is
@@ -669,7 +661,7 @@ async function embedAll(
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
// #1737: thread the external abort signal so the cycle embed phase bails.
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature, signal);
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature, signal, embeddingColumn);
}
// --all path: pacer (no-op when off). E-1: lower the worker count to the
@@ -742,17 +734,18 @@ async function embedAll(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
// doesn't clobber language/symbol_name/symbol_type to NULL.
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
// Preserve ALL chunks, only update embeddings for stale ones
const updated: ChunkInput[] = chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
embedding: embeddingMap.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts));
await observed(pacer, () => engine.upsertChunks(page.slug, updated, {
...(pageSourceId && { sourceId: pageSourceId }),
...(embeddingColumn && { embeddingColumn }),
}));
// v0.41.31: stamp embedding provenance so a later model swap is
// detectable as stale.
await observed(pacer, () =>
@@ -832,10 +825,16 @@ async function embedAllStale(
},
signature?: string,
externalSignal?: AbortSignal,
embeddingColumn?: ResolvedColumn,
) {
// D7: thread sourceId so source-scoped runs only count + visit
// that source's NULL embeddings.
const sourceOpt = sourceId ? { sourceId } : undefined;
// #1262: the stale predicate follows the write-side column — without it an
// alt-column brain would perpetually re-select (and re-pay for) chunks whose
// target column is already populated.
const sourceOpt = (sourceId || embeddingColumn)
? { ...(sourceId && { sourceId }), ...(embeddingColumn && { embeddingColumn }) }
: undefined;
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
// swap). dry-run must NOT mutate, so it counts signature-stale via the
@@ -994,6 +993,7 @@ async function embedAllStale(
afterUpdatedAt,
}),
...(sourceId && { sourceId }),
...(embeddingColumn && { embeddingColumn }),
}),
);
if (batch.length === 0) {
@@ -1039,17 +1039,17 @@ async function embedAllStale(
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
}
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
// to NULL on every cycle.
const merged: ChunkInput[] = existing.map(c => preserveCodeMetadata(c, {
const merged: ChunkInput[] = existing.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
await observed(pacer, () => engine.upsertChunks(slug, merged, {
sourceId: keySourceId,
...(embeddingColumn && { embeddingColumn }),
}));
// v0.41.31: stamp provenance after the page's chunks are embedded —
// but only when EVERY chunk was stale (fully re-embedded this pass).
// A partially-stale page keeps preserved chunks of unknown/old
@@ -1120,7 +1120,7 @@ async function embedAllStale(
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
const remaining = await engine.countStaleChunks(
signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined),
signature ? { signature, ...sourceOpt } : sourceOpt,
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
+2 -13
View File
@@ -1651,7 +1651,7 @@ async function extractTimelineFromDB(
* make re-extraction idempotent). EVERY processed page is stamped, including
* zero-link pages — they WERE processed.
*/
export async function extractStaleFromDB(
async function extractStaleFromDB(
engine: BrainEngine,
opts: {
dryRun: boolean;
@@ -1743,18 +1743,7 @@ export async function extractStaleFromDB(
// `page.updated_at.toISOString()` — the JS Date is ms-truncated, so the
// µs-precision DB updated_at stayed strictly greater and the page never
// cleared on Postgres. Stamping the exact value makes them equal.
//
// BUT the stamp must also clear the version-staleness clause
// (`links_extracted_at < versionTs`). A page whose updated_at predates
// versionTs would otherwise be stamped below the threshold and read as
// stale forever — a permanent re-extract loop that never clears the lag.
// GREATEST(updated_at, versionTs) preserves the race semantics (a real
// future edit advances updated_at > versionTs >= stamp → re-extracts)
// while lifting old pages to the threshold so they clear.
const stampIso = page.updated_at.getTime() >= Date.parse(versionTs)
? page.updated_at_iso
: versionTs;
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: stampIso });
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at_iso });
}
// Flush NON-swallowing (CDX-4): a throw here propagates out of the sweep so
+1 -10
View File
@@ -98,17 +98,8 @@ export function findBareTweetHits(compiledTruth: string, slug: string): BareTwee
}
// If the line already contains a tweet URL, it's cited — skip
if (URL_NEARBY_RE.test(line)) continue;
// If the line carries an explicit source citation (e.g.
// "[Source: X, @handle, 2026-05-28]"), it's already attributed — skip.
// Catches instructional/example lines in recipe docs that demonstrate
// the CORRECT citation format. (v0.42.x)
if (/\[\s*source:/i.test(line)) continue;
// Strip inline-code spans (`...`) before matching: phrases shown as
// inline-code templates in docs are examples, not bare claims. The
// fenced-code skip above only covers ``` blocks, not inline backticks.
const lineForMatch = line.replace(/`[^`]*`/g, '');
for (const re of BARE_TWEET_PHRASES) {
const m = lineForMatch.match(re);
const m = line.match(re);
if (m) {
hits.push({ slug, line: i + 1, rawLine: line.trim(), phrase: m[0] });
break; // one finding per line is enough
+1 -7
View File
@@ -1664,13 +1664,7 @@ export async function registerBuiltinHandlers(
worker.register('backlinks', async (job) => {
const { runBacklinksCore } = await import('./backlinks.ts');
// Default to 'check', not 'fix': backlinks jobs submitted with an empty
// payload (e.g. the sync→embed→backlinks chains enqueued after ingestion)
// must never rewrite tracked brain pages with generated "Referenced in"
// timeline bullets. Mirrors the documented intent in src/core/cycle.ts
// (runPhaseBacklinks). The filesystem fixer stays available explicitly
// via '{"action":"fix"}' or `gbrain check-backlinks fix`.
const action: 'check' | 'fix' = job.data.action === 'fix' ? 'fix' : 'check';
const action: 'check' | 'fix' = job.data.action === 'check' ? 'check' : 'fix';
const dir = typeof job.data.dir === 'string'
? job.data.dir
: (await engine.getConfig('sync.repo_path')) ?? '.';
+1 -6
View File
@@ -127,12 +127,7 @@ export function lintContent(content: string, filePath: string, opts: LintContent
}
// Rule: Wrapping code fences (```markdown ... ```)
// Detector intentionally has NO /m flag so ^/$ match start/end of the whole
// file, not inner lines. Keeps detector in sync with fixContent() below,
// which also has no /m flag. Without this, lint reports "fixable" false
// positives on any page that simply contains a ```markdown code block, but
// fixContent can never strip them (its regex only matches whole-file wrappers).
if (content.match(/^```(?:markdown|md)\s*\n/) && content.match(/\n```\s*$/)) {
if (content.match(/^```(?:markdown|md)\s*\n/m) && content.match(/\n```\s*$/m)) {
issues.push({
file: filePath, line: 1, rule: 'code-fence-wrap',
message: 'Page wrapped in ```markdown code fences (LLM artifact)',
-224
View File
@@ -1,224 +0,0 @@
/**
* gbrain maintain conservative self-healing maintenance.
*
* This command automates the safe parts of the operator runbook:
* - stale link/timeline extraction
* - stale per-source dream cycles when doctor reports cycle_freshness
*
* It deliberately does NOT mutate source files, apply schema-pack upgrades, or
* invent semantic hub links. Those need review or a separate command with an
* auditable proposal surface.
*/
import { existsSync } from 'fs';
import type { BrainEngine } from '../core/engine.ts';
import type { BrainHealth } from '../core/types.ts';
import { buildChecks, computeDoctorReport, type DoctorReport, type Check } from './doctor.ts';
import { extractStaleFromDB } from './extract.ts';
import { runCycle, type CycleReport } from '../core/cycle.ts';
type ActionStatus = 'ok' | 'would_apply' | 'applied' | 'blocked' | 'skipped';
export interface MaintenanceAction {
name: string;
status: ActionStatus;
message: string;
details?: Record<string, unknown>;
}
export interface MaintainOptions {
json: boolean;
safe: boolean;
dryRun: boolean;
help: boolean;
}
export interface MaintainReport {
mode: 'dry-run' | 'safe';
before: {
health: BrainHealth;
doctor: DoctorReport;
};
actions: MaintenanceAction[];
after: {
health: BrainHealth;
doctor: DoctorReport;
};
}
export function parseMaintainArgs(args: string[]): MaintainOptions {
const safe = args.includes('--safe');
return {
json: args.includes('--json'),
safe,
dryRun: args.includes('--dry-run') || !safe,
help: args.includes('--help') || args.includes('-h'),
};
}
export function extractCycleFreshnessSourceIds(checks: Check[]): string[] {
const ids = new Set<string>();
for (const check of checks) {
if (check.name !== 'cycle_freshness' || check.status === 'ok') continue;
const re = /Source '([^']+)' last cycled/g;
for (const match of check.message.matchAll(re)) {
const id = match[1]?.trim();
if (id) ids.add(id);
}
}
return [...ids].sort();
}
async function buildDoctorReport(engine: BrainEngine): Promise<DoctorReport> {
const checks = await buildChecks(engine, ['--json', '--scope=brain']);
return computeDoctorReport(checks);
}
async function runStaleExtraction(
engine: BrainEngine,
beforeHealth: BrainHealth,
dryRun: boolean,
): Promise<MaintenanceAction> {
if (beforeHealth.stale_pages <= 0) {
return { name: 'extract_stale', status: 'ok', message: 'No stale pages.' };
}
if (dryRun) {
return {
name: 'extract_stale',
status: 'would_apply',
message: `Would run DB-backed stale extraction for ${beforeHealth.stale_pages} page(s).`,
details: { stale_pages: beforeHealth.stale_pages },
};
}
const result = await extractStaleFromDB(engine, {
dryRun: false,
jsonMode: false,
includeFrontmatter: false,
catchUp: false,
});
return {
name: 'extract_stale',
status: 'applied',
message: `Processed ${result.pagesProcessed} stale page(s); ${result.staleRemaining} remain.`,
details: {
links_created: result.linksCreated,
timeline_created: result.timelineCreated,
pages_processed: result.pagesProcessed,
stale_remaining: result.staleRemaining,
},
};
}
async function runCycleFreshnessMaintenance(
engine: BrainEngine,
beforeDoctor: DoctorReport,
dryRun: boolean,
): Promise<MaintenanceAction[]> {
const sourceIds = extractCycleFreshnessSourceIds(beforeDoctor.checks);
if (sourceIds.length === 0) {
return [{ name: 'cycle_freshness', status: 'ok', message: 'All sources cycled recently.' }];
}
if (dryRun) {
return sourceIds.map((sourceId) => ({
name: 'cycle_freshness',
status: 'would_apply',
message: `Would run source-scoped dream cycle for ${sourceId}.`,
details: { source_id: sourceId },
}));
}
const sources = await engine.listAllSources();
const actions: MaintenanceAction[] = [];
for (const sourceId of sourceIds) {
const source = sources.find((s) => s.id === sourceId);
const localPath = source?.local_path ?? null;
const brainDir = localPath && existsSync(localPath) ? localPath : null;
const report: CycleReport = await runCycle(engine, {
brainDir,
dryRun: false,
pull: false,
sourceId,
});
actions.push({
name: 'cycle_freshness',
status: report.status === 'failed' ? 'blocked' : 'applied',
message: `Ran source-scoped dream cycle for ${sourceId}: ${report.status}.`,
details: {
source_id: sourceId,
brain_dir: brainDir,
cycle_status: report.status,
phases: report.phases.map((p) => ({ phase: p.phase, status: p.status })),
},
});
}
return actions;
}
export async function runMaintain(engine: BrainEngine, args: string[]): Promise<MaintainReport | void> {
const opts = parseMaintainArgs(args);
if (opts.help) {
console.log(`Usage: gbrain maintain [--safe] [--dry-run] [--json]
Conservative self-healing maintenance.
Modes:
--dry-run Preview safe actions without writes. Default when --safe is absent.
--safe Apply safe actions: stale extraction and source cycle freshness.
--json Emit a structured before/action/after report.
Not auto-applied:
source-file frontmatter fixes, schema-pack upgrades, atom-pack changes,
semantic hub-link guesses, and destructive cleanup.
`);
return;
}
const beforeHealth = await engine.getHealth();
const beforeDoctor = await buildDoctorReport(engine);
const actions: MaintenanceAction[] = [];
actions.push(await runStaleExtraction(engine, beforeHealth, opts.dryRun));
actions.push(...await runCycleFreshnessMaintenance(engine, beforeDoctor, opts.dryRun));
const afterHealth = await engine.getHealth();
const afterDoctor = await buildDoctorReport(engine);
const report: MaintainReport = {
mode: opts.dryRun ? 'dry-run' : 'safe',
before: { health: beforeHealth, doctor: beforeDoctor },
actions,
after: { health: afterHealth, doctor: afterDoctor },
};
if (opts.json) {
console.log(JSON.stringify(report, null, 2));
} else {
printMaintainReport(report);
}
return report;
}
function printMaintainReport(report: MaintainReport): void {
console.log(`GBrain maintain (${report.mode})`);
console.log(
`Before: brain_score=${Math.round(report.before.health.brain_score)}/100 ` +
`stale=${report.before.health.stale_pages} islands=${report.before.health.orphan_pages} ` +
`doctor=${report.before.doctor.status}`,
);
for (const action of report.actions) {
console.log(` ${action.status}: ${action.name}${action.message}`);
}
console.log(
`After: brain_score=${Math.round(report.after.health.brain_score)}/100 ` +
`stale=${report.after.health.stale_pages} islands=${report.after.health.orphan_pages} ` +
`doctor=${report.after.doctor.status}`,
);
if (report.mode === 'dry-run') {
console.log('Run `gbrain maintain --safe` to apply safe actions.');
}
}
+1 -14
View File
@@ -536,20 +536,7 @@ function shouldSkipProvider(modelStr: string, skip: string[]): boolean {
export async function runModels(engine: BrainEngine, args: string[]): Promise<void> {
const json = args.includes('--json');
// args is `subArgs` from cli.ts `handleCliOnly` — the leading 'models'
// token has already been stripped. The subcommand is at args[0], NOT
// args[1]. Pre-fix this check was `args[1]`, so `gbrain models doctor`
// silently fell through to the read view. The doctor probe path was
// unreachable from the CLI.
//
// --help honored FIRST so `gbrain models doctor --help` shows usage
// instead of running network probes (which would spend tokens or
// exit nonzero when the user only asked for help). Pre-fix the
// args[1] ternary happened to dodge this by always falling through
// to the args.includes('--help') branch; the args[0] rewrite needs
// explicit ordering to preserve that behavior.
const hasHelp = args.includes('--help') || args.includes('-h') || args[0] === 'help';
const sub = hasHelp ? 'help' : args[0] === 'doctor' ? 'doctor' : 'read';
const sub = args[1] === 'doctor' ? 'doctor' : args[1] === 'help' || args.includes('--help') || args.includes('-h') ? 'help' : 'read';
if (sub === 'help') {
process.stdout.write(
+55 -10
View File
@@ -15,11 +15,6 @@
import type { BrainEngine } from '../core/engine.ts';
import { createProgress, startHeartbeat } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import {
shouldExcludeFromOrphanReporting,
loadOrphanPolicyOverrides,
type OrphanPolicyOverrides,
} from '../core/orphan-policy.ts';
// --- Types ---
@@ -37,14 +32,65 @@ export interface OrphanResult {
excluded: number;
}
// --- Filter constants ---
/** Slug suffixes that are always auto-generated root files */
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
/** Page slugs that are pseudo-pages by convention */
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
/** Slug segment that marks raw sources */
const RAW_SEGMENT = '/raw/';
/** Slug prefixes where no inbound links is expected */
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'openclaw/config/',
];
/** First slug segments where no inbound links is expected */
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
]);
// --- Filter logic ---
/**
* Returns true if a slug should be excluded from orphan reporting by default.
* These are pages where having no inbound links is expected / not a content problem.
*/
export function shouldExclude(slug: string, overrides?: OrphanPolicyOverrides): boolean {
return shouldExcludeFromOrphanReporting(slug, overrides);
export function shouldExclude(slug: string): boolean {
// Pseudo-pages (exact match)
if (PSEUDO_SLUGS.has(slug)) return true;
// Auto-generated suffix patterns
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
// Raw source slugs
if (slug.includes(RAW_SEGMENT)) return true;
// Deny-prefix slugs
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
// First-segment exclusions
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
return false;
}
/**
@@ -110,7 +156,6 @@ export async function findOrphans(
let allOrphans: { slug: string; title: string; domain: string | null }[];
let total: number;
let excludedAll: number;
const overrides = includePseudo ? undefined : await loadOrphanPolicyOverrides(engine);
try {
allOrphans = await engine.findOrphanPages(
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
@@ -139,7 +184,7 @@ export async function findOrphans(
total = liveRows.length;
excludedAll = includePseudo
? 0
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug, overrides) ? 1 : 0), 0);
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
} finally {
stopHb();
progress.finish();
@@ -147,7 +192,7 @@ export async function findOrphans(
const filtered = includePseudo
? allOrphans
: allOrphans.filter(row => !shouldExclude(row.slug, overrides));
: allOrphans.filter(row => !shouldExclude(row.slug));
const orphans: OrphanPage[] = filtered.map(row => ({
slug: row.slug,
+2 -17
View File
@@ -843,21 +843,6 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// reverse proxies / tunnels; default to localhost for dev.
const issuerUrl = new URL(publicUrl || `http://localhost:${port}`);
// MCP authorization spec (2025-06-18 draft §5.1) and RFC 9728 require the
// protected resource server to return its discovery metadata URL in the
// WWW-Authenticate header on 401 responses:
//
// WWW-Authenticate: Bearer resource_metadata="<URL>"
//
// Clients (claude.ai, Cursor, every other MCP-aware OAuth client) use that
// URL to find the authorization-server discovery doc + token endpoint
// without the user having to paste those URLs manually. Pre-fix the header
// shipped `Bearer error="invalid_token", ...` with no resource_metadata
// parameter, so MCP clients couldn't begin the OAuth flow from a fresh
// 401 — they would silently fail to connect with a generic "couldn't
// reach the MCP server" error.
const resourceMetadataUrl = `${issuerUrl.toString().replace(/\/$/, '')}/.well-known/oauth-protected-resource`;
// F9: cookie `secure` flag honors both the request's TLS state (req.secure
// is set when express trust-proxy lands an X-Forwarded-Proto: https) AND
// the operator's declared issuer protocol (so a Cloudflare-tunnel deploy
@@ -1616,7 +1601,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
res.status(405).json({ jsonrpc: '2.0', error: { code: -32000, message: 'Method not allowed' }, id: null });
});
app.post('/mcp', requireBearerAuth({ verifier: oauthProvider, resourceMetadataUrl }), async (req: Request, res: Response) => {
app.post('/mcp', requireBearerAuth({ verifier: oauthProvider }), async (req: Request, res: Response) => {
const startTime = Date.now();
const authInfo = (req as any).auth as AuthInfo;
@@ -1959,7 +1944,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
app.post(
'/ingest',
ingestRateLimiter,
requireBearerAuth({ verifier: oauthProvider, requiredScopes: ['write'], resourceMetadataUrl }),
requireBearerAuth({ verifier: oauthProvider, requiredScopes: ['write'] }),
express.raw({ type: '*/*', limit: ingestMaxBytes }),
async (req: Request, res: Response) => {
const startTime = Date.now();
+8 -1
View File
@@ -576,9 +576,16 @@ async function runInlineCostGate(
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB hiccup
// never blocks the sync. Signature-aware (model/dims swap surfaces here).
// #1262: follow the write-side embedding column — otherwise an alt-column
// brain's fully-embedded corpus counts as phantom backlog on every gate.
let staleChars = 0;
try {
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
const { resolveWriteColumnForEngine } = await import('../core/search/embedding-column.ts');
const embeddingColumn = await resolveWriteColumnForEngine(engine);
staleChars = await engine.sumStaleChunkChars({
signature: currentEmbeddingSignature(),
...(embeddingColumn && { embeddingColumn }),
});
} catch {
staleChars = 0;
}
+2 -2
View File
@@ -6,7 +6,7 @@
* degrades to gather-only output with a warning if missing.
*/
import type { BrainEngine } from '../core/engine.ts';
import { runThink, persistSynthesis, stripGapsSection } from '../core/think/index.ts';
import { runThink, persistSynthesis } from '../core/think/index.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
@@ -157,7 +157,7 @@ prints what would have been the input (exit 0).
// Human-readable output
console.log(`# ${question}\n`);
console.log(stripGapsSection(result.answer));
console.log(result.answer);
console.log('');
if (result.gaps.length > 0) {
console.log('## Gaps');
-9
View File
@@ -263,15 +263,6 @@ export function dimsProviderOptions(
if (modelId === 'text-embedding-v3' || modelId === 'embedding-3') {
return { openaiCompatible: { dimensions: dims } };
}
// Qwen3-Embedding family on Ollama (and any other openai-compatible
// provider serving it) supports Matryoshka truncation via `dimensions`.
// Native sizes: 0.6B=1024, 4B=2560, 8B=4096. Without `dimensions`,
// Ollama returns the native size and brains configured for narrower
// widths hard-fail with a dim-mismatch error. Pattern match the bare
// model name + any `:tag` (e.g. `qwen3-embedding:4b`, `qwen3-embedding:0.6b`).
if (modelId === 'qwen3-embedding' || modelId.startsWith('qwen3-embedding:')) {
return { openaiCompatible: { dimensions: dims } };
}
// MiniMax embo-01 takes a `type: 'db' | 'query'` field for asymmetric
// retrieval. Today still hardcoded to 'db' for back-compat — opting
// into the new inputType seam is a follow-up (see plan's deferred
+1 -27
View File
@@ -599,8 +599,6 @@ function warnRecipesMissingBatchTokens(): void {
// LiteLLM proxy, llama-server) — they ship without a static cap because
// the cap depends on a user-launched server. Warning is noise for them.
if (embedding.no_batch_cap === true) continue;
// A declared item-count cap is a real batch cap — no warning needed.
if (embedding.max_batch_items !== undefined) continue;
if (_warnedRecipes.has(recipe.id)) continue;
_warnedRecipes.add(recipe.id);
// eslint-disable-next-line no-console
@@ -1519,17 +1517,10 @@ export async function embed(texts: string[], opts?: EmbedOpts): Promise<Float32A
// Pre-split is gated on max_batch_tokens. Recipes without it (e.g. OpenAI)
// ride the fast path: one embedMany call, no recursion safety net.
const tokenBatches = maxBatchTokens
const batches = maxBatchTokens
? splitByTokenBudget(truncated, Math.floor(maxBatchTokens * effectiveSafetyFactor(recipe)), charsPerToken)
: [truncated];
// Hard COUNT cap (e.g. llama-server's "maximum allowed batch size 32").
// Token budget can't bound item count, so re-split any oversized batch.
const maxBatchItems = embedding?.max_batch_items;
const batches = maxBatchItems
? tokenBatches.flatMap(b => capBatchItems(b, maxBatchItems))
: tokenBatches;
const allEmbeddings: Float32Array[] = [];
let _embedThrew = false;
try {
@@ -1605,23 +1596,6 @@ export function splitByTokenBudget(
return batches;
}
/**
* Split a batch into sub-batches of at most `maxItems` inputs. Enforces a
* hard COUNT cap that the token-budget split can't (many tiny inputs fit
* under any token budget). Used for endpoints like llama.cpp's llama-server
* that reject requests exceeding their launch batch size.
*
* @internal exported for tests; not part of the public gateway API.
*/
export function capBatchItems(texts: string[], maxItems: number): string[][] {
if (maxItems <= 0 || texts.length <= maxItems) return [texts];
const batches: string[][] = [];
for (let i = 0; i < texts.length; i += maxItems) {
batches.push(texts.slice(i, i + maxItems));
}
return batches;
}
/**
* Returns true if the error looks like a provider batch-token-limit error.
*
+3 -6
View File
@@ -35,12 +35,9 @@ export const llamaServer: Recipe = {
trust_custom_dims: true, // #2271: user knows the launched model's native dim
cost_per_1m_tokens_usd: 0,
price_last_verified: '2026-05-10',
// llama-server enforces a hard request-COUNT cap equal to its launch
// batch size (`--batch-size`, default 32): it rejects requests with
// more inputs with `batch size N > maximum allowed batch size 32`.
// The token-budget split can't bound item count, so cap it here. A
// server launched with a larger `-b` can raise this. v0.32 (#779).
max_batch_items: 32,
// llama-server's batch capacity is set by `--ctx-size` at launch
// time; no static cap to declare. v0.32 (#779).
no_batch_cap: true,
},
},
/**
-10
View File
@@ -54,16 +54,6 @@ export interface EmbeddingTouchpoint {
* `max_batch_tokens` is also set.
*/
safety_factor?: number;
/**
* Maximum number of inputs per embedding request. Some endpoints enforce a
* hard COUNT cap independent of token budget notably llama.cpp's
* `llama-server`, which rejects requests with more inputs than its launch
* batch size (e.g. `batch size 100 > maximum allowed batch size 32`). The
* token-budget pre-split cannot bound item count (many tiny chunks fit under
* any token budget), so this is enforced as a separate hard re-split after
* the token split. When unset, no count cap is applied.
*/
max_batch_items?: number;
/**
* v0.27.1: when true, at least one model in this recipe accepts image
* inputs via a multimodal embedding endpoint (e.g. Voyage's
+2 -6
View File
@@ -400,12 +400,8 @@ function pickMax(current: number, max: number, status: RemediationStatus | undef
// ---------------------------------------------------------------------
// Idempotency key construction (D9 — content-hash, no time-slot).
// Same params produce the same key across runs (exactly-once dedup via the
// queue's unique partial index). The --remediate loop attempts each step at
// most once per run and excludes terminally-failed ids from later rechecks
// (selectActiveRecs in remediation-step.ts); it does NOT mutate this key. The
// `:r<N>` failed-row replay suffix was designed but never implemented and is
// unnecessary given queue-level max_attempts.
// Same params produce the same key across runs. Failed-row replay
// appends `:r<N>` (caller responsibility — handled by --remediate loop).
// ---------------------------------------------------------------------
function idemKey(source: string, job: string, params: Record<string, unknown>): string {
+3 -13
View File
@@ -217,19 +217,9 @@ export function parseResolverEntries(resolverContent: string): ResolverEntry[] {
// `skillsDir/*/SKILL.md` when manifest.json is missing — the scenario
// needed for AGENTS.md-only OpenClaw deployments. See D-CX-12 / F-ENG-1.
/**
* Simple YAML frontmatter parser extracts triggers array if present.
*
* Normalizes CRLF LF before parsing so Windows checkouts (where
* `core.autocrlf=true` is the default) parse correctly. Without this,
* the `^---\n` and `^triggers:\s*\n` regexes never match because the
* file content is `---\r\n` / `triggers:\r\n`, and every skill on
* Windows is reported as `mece_gap` regardless of its actual content.
* CI runs on Ubuntu-only so the bug only surfaces in user environments.
*/
export function extractTriggers(skillContent: string): string[] {
const content = skillContent.replace(/\r\n/g, '\n');
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
/** Simple YAML frontmatter parser — extracts triggers array if present. */
function extractTriggers(skillContent: string): string[] {
const fmMatch = skillContent.match(/^---\n([\s\S]*?)\n---/);
if (!fmMatch) return [];
const fm = fmMatch[1];
const triggersMatch = fm.match(/^triggers:\s*\n((?:\s+-\s+.+\n?)*)/m);
+2 -79
View File
@@ -168,15 +168,6 @@ export interface CodeChunkOptions {
largeChunkThresholdTokens?: number;
fallbackChunkSizeWords?: number;
fallbackOverlapWords?: number;
/**
* Hard upper bound (estimated tokens) on any single emitted chunk. A node
* the AST splitter can't break up (a giant object/array literal, a single
* huge assignment, a massive template literal) would otherwise be emitted
* whole and rejected by the embedder ("input exceeds context length").
* Chunks over this budget are recursively re-split. Default 2000 fits the
* smallest common embedder context (e.g. nomic-embed-text, 2048).
*/
maxChunkTokens?: number;
}
/**
@@ -558,7 +549,6 @@ export function parseWithTimeout(
}
const DEFAULT_CHUNKER_TIMEOUT_MS = 30_000;
const DEFAULT_MAX_CHUNK_TOKENS = 2000;
function resolveChunkerTimeoutMs(): number {
const raw = process.env.GBRAIN_CHUNKER_TIMEOUT_MS;
@@ -716,9 +706,9 @@ export async function chunkCodeTextFull(
}
if (chunks.length === 0) {
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
}
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
return { chunks: mergeSmallSiblings(chunks, chunkTarget), edges: rawEdges };
} catch {
return { chunks: fallbackChunks(source, filePath, language, opts), edges: [] };
} finally {
@@ -824,73 +814,6 @@ function buildMergedChunk(group: CodeChunk[], index: number): CodeChunk {
};
}
/**
* Final safety net: guarantee no emitted chunk exceeds the embedder's context
* budget. tree-sitter splitting (splitLargeNode) can only break up a node that
* exposes a `body` with >= 2 named children. A node without one a giant
* object/array literal, a single huge assignment, a massive template literal
* is emitted whole, producing a chunk far larger than the embedder accepts.
* The embedder then rejects it ("input exceeds context length") and the chunk
* is never embedded. Recursively re-split any over-budget chunk; fall back to a
* hard character split for pathological no-whitespace content (e.g. a minified
* one-liner) where word/line splitting can't get under budget.
*/
function capOversizedChunks(
chunks: CodeChunk[],
filePath: string,
language: SupportedCodeLanguage,
opts: CodeChunkOptions,
): CodeChunk[] {
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
const out: CodeChunk[] = [];
for (const c of chunks) {
if (estimateTokens(c.text) <= cap) {
out.push({ ...c, index: out.length });
continue;
}
// Strip the structured header ("[Lang] path:N-M symbol\n\n") so the splitter
// works on the raw body; buildChunk re-adds a header to each piece.
const body = c.text.replace(/^\[[^\]]+\] [^\n]+\n\n/, '');
for (const piece of splitToTokenBudget(body, cap, opts)) {
if (!piece.trim()) continue;
out.push(buildChunk({
body: piece,
filePath,
language,
symbolName: c.metadata.symbolName,
symbolType: c.metadata.symbolType,
startLine: c.metadata.startLine,
endLine: c.metadata.endLine,
index: out.length,
parentSymbolPath: c.metadata.parentSymbolPath,
}));
}
}
return out;
}
/** Split `text` into pieces each estimated <= cap tokens. Word/line-aware
* (recursiveChunk) first; a hard character split is the last resort for
* content with no whitespace to break on. */
function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions): string[] {
const out: string[] = [];
const pieces = recursiveChunk(text, {
chunkSize: opts.fallbackChunkSizeWords ?? 300,
chunkOverlap: opts.fallbackOverlapWords ?? 50,
}).map((p) => p.text);
for (const piece of pieces) {
if (estimateTokens(piece) <= cap) {
out.push(piece);
continue;
}
// ~3.5 chars/token is a conservative cl100k estimate for source text.
const charBudget = Math.max(1, Math.floor(cap * 3.5));
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
}
return out;
}
// ---------- Internals ----------
function fallbackChunks(
+1 -33
View File
@@ -620,10 +620,7 @@ export function loadConfig(): GBrainConfig | null {
* size the schema and must be stable across engine connect.
*/
export async function loadConfigWithEngine(
engine: {
getConfig(key: string): Promise<string | null | undefined>;
listConfigKeys?(prefix: string): Promise<string[]>;
},
engine: { getConfig(key: string): Promise<string | null | undefined> },
base?: GBrainConfig | null,
): Promise<GBrainConfig | null> {
// Codex /ship finding #3: when there's no file config AND no env DB URL,
@@ -660,31 +657,11 @@ export async function loadConfigWithEngine(
return undefined;
}
}
async function dbPrefixMap(prefix: string): Promise<Record<string, string> | undefined> {
if (typeof engine.listConfigKeys !== 'function') return undefined;
let keys: string[];
try {
keys = await engine.listConfigKeys(prefix);
} catch {
return undefined;
}
const out: Record<string, string> = {};
for (const key of keys.sort()) {
if (!key.startsWith(prefix)) continue;
const leaf = key.slice(prefix.length);
if (!leaf) continue;
const value = await dbStr(key);
if (value !== undefined) out[leaf] = value;
}
return Object.keys(out).length > 0 ? out : undefined;
}
const dbMultimodal = await dbBool('embedding_multimodal');
const dbMultimodalModel = await dbStr('embedding_multimodal_model');
const dbOcr = await dbBool('embedding_image_ocr');
const dbOcrModel = await dbStr('embedding_image_ocr_model');
const dbProviderBaseUrls = await dbPrefixMap('provider_base_urls.');
// v0.36 (D7) — embedding-column registry merge. Stored as JSON string in
// the config table. Parse + shape-check here; full registry validation
// (regex on keys, type/dim/provider field shapes) runs in the resolver at
@@ -708,15 +685,6 @@ export async function loadConfigWithEngine(
if (merged.embedding_image_ocr_model === undefined && dbOcrModel !== undefined) {
merged.embedding_image_ocr_model = dbOcrModel;
}
if (dbProviderBaseUrls !== undefined) {
const next = { ...(merged.provider_base_urls ?? {}) };
for (const [providerId, baseUrl] of Object.entries(dbProviderBaseUrls)) {
if (next[providerId] === undefined) next[providerId] = baseUrl;
}
if (Object.keys(next).length > 0) {
merged.provider_base_urls = next;
}
}
if (merged.embedding_columns === undefined && dbEmbeddingColumns !== undefined) {
try {
const parsed = JSON.parse(dbEmbeddingColumns);
+10 -22
View File
@@ -54,7 +54,6 @@ import {
import {
generatePerChunkSynopsis,
SYNOPSIS_PROMPT_VERSION,
SYNOPSIS_DOC_MAX_CHARS,
type GeneratePerChunkSynopsisResult,
} from './page-summary.ts';
import {
@@ -62,6 +61,7 @@ import {
type SynopsisFailureKind,
} from './audit-synopsis.ts';
import type { BrainEngine } from './engine.ts';
import { resolveWriteColumnForEngine } from './search/embedding-column.ts';
import type { ChunkInput, CRMode, Page } from './types.ts';
import type { SourceRow } from './sources-ops.ts';
@@ -104,17 +104,8 @@ function getEmbeddingModelTag(): string {
export function computeCorpusGeneration(args: {
crMode: CRMode;
haikuModel: string;
/**
* Resolved `SYNOPSIS_DOC_MAX_CHARS` for per_chunk_synopsis runs. When
* present, folded into the hash so changes to
* `GBRAIN_SYNOPSIS_DOC_MAX_CHARS` invalidate the prior cache cleanly.
* Omit for `crMode !== 'per_chunk_synopsis'` title / none modes
* don't consult the cap and the field stays out of the hash for
* back-compat with pre-cap embeddings.
*/
synopsisDocMaxChars?: number;
}): string {
const h = createHash('sha256')
return createHash('sha256')
.update(args.crMode)
.update('|')
.update(String(SYNOPSIS_PROMPT_VERSION))
@@ -123,11 +114,9 @@ export function computeCorpusGeneration(args: {
.update('|')
.update(String(TITLE_WRAPPER_VERSION))
.update('|')
.update(getEmbeddingModelTag());
if (args.synopsisDocMaxChars !== undefined) {
h.update('|doc_cap=').update(String(args.synopsisDocMaxChars));
}
return h.digest('hex').slice(0, 16);
.update(getEmbeddingModelTag())
.digest('hex')
.slice(0, 16);
}
/**
@@ -265,11 +254,7 @@ export async function reembedPageWithContextualRetrieval(
args.pageSlug,
args.sourceId,
resolution.mode,
computeCorpusGeneration({
crMode: resolution.mode,
haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL,
synopsisDocMaxChars: resolution.mode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
}),
computeCorpusGeneration({ crMode: resolution.mode, haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL }),
);
return { kind: 'skipped', reason: 'no_chunks' };
}
@@ -298,14 +283,17 @@ export async function reembedPageWithContextualRetrieval(
const corpus_generation = computeCorpusGeneration({
crMode: attemptMode,
haikuModel,
synopsisDocMaxChars: attemptMode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
});
// ── PHASE 2: single DB transaction ───────────────────────────
try {
// #1262: contextual re-embeds write TEXT embeddings — thread the
// caller-resolved write column like every other embed path.
const embeddingColumn = await resolveWriteColumnForEngine(args.engine);
await args.engine.transaction(async (tx) => {
await tx.upsertChunks(args.pageSlug, phase1.embeddedChunks, {
sourceId: args.sourceId,
...(embeddingColumn && { embeddingColumn }),
});
await tx.updatePageContextualRetrievalState(
args.pageSlug,
+13 -2
View File
@@ -18,7 +18,7 @@
*/
import type { BrainEngine } from './engine.ts';
import type { ChunkInput } from './types.ts';
import type { ChunkInput, ResolvedColumn } from './types.ts';
import { embedBatchWithBackoff } from '../commands/embed.ts';
import { type DbPacer, createNoopPacer, observed } from './db-pacer.ts';
import { AbortError } from './abort-check.ts';
@@ -61,6 +61,13 @@ export interface EmbedStaleOpts {
* Omit to keep the legacy `embedding IS NULL`-only behavior.
*/
embeddingSignature?: string;
/**
* #1262: caller-resolved write-side embedding column. Threaded into BOTH
* listStaleChunks (staleness predicate) and upsertChunks (write target) so
* an alt-column brain converges instead of re-selecting embedded rows.
* Resolve at the boundary via `resolveWriteColumnForEngine()`.
*/
embeddingColumn?: ResolvedColumn;
/**
* DB-contention pacer (paced-backfill). When enabled it (a) supplies the
* worker count via the caller passing `concurrency = bundle.maxConcurrency`
@@ -156,6 +163,7 @@ export async function embedStaleForSource(
afterPageId,
afterChunkIndex,
sourceId,
...(opts.embeddingColumn && { embeddingColumn: opts.embeddingColumn }),
}),
);
if (batch.length === 0) {
@@ -223,7 +231,10 @@ export async function embedStaleForSource(
doc_comment: c.doc_comment ?? undefined,
symbol_name_qualified: c.symbol_name_qualified ?? undefined,
}));
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
await observed(pacer, () => engine.upsertChunks(slug, merged, {
sourceId: keySourceId,
...(opts.embeddingColumn && { embeddingColumn: opts.embeddingColumn }),
}));
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
// re-embedded this pass) — a partially-stale page keeps preserved
// chunks of unknown provenance, so don't claim current. After the
+22 -13
View File
@@ -12,6 +12,7 @@ import type {
BrainStats, BrainHealth,
IngestLogEntry, IngestLogInput,
EngineConfig,
ResolvedColumn,
CodeEdgeInput, CodeEdgeResult,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
@@ -987,8 +988,13 @@ export interface BrainEngine {
* Postgres rolls back automatically on conn drop, so commit-ambiguous
* failure replays to the same end state. Callers MUST NOT wrap externally;
* see {@link BatchOpts} retry-contract block.
*
* `opts.embeddingColumn` (optional) selects the content_chunks column that
* receives TEXT embeddings (#1262). The caller resolves the descriptor at
* the import/embed boundary via `resolveWriteColumn()`; engines never read
* config or choose columns themselves. Omitted => legacy `embedding`.
*/
upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string } & BatchOpts): Promise<void>;
upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string; embeddingColumn?: ResolvedColumn } & BatchOpts): Promise<void>;
/**
* Read every chunk for a page. `opts.sourceId` source-scopes the page
* lookup; without it, multi-source brains return chunks from every
@@ -1005,8 +1011,13 @@ export interface BrainEngine {
* counts across every source in the brain. Operators running
* `gbrain embed --stale --source media-corpus` expect only that
* source's NULLs touched; the caller threads `sourceId` here.
*
* `opts.embeddingColumn` switches the staleness predicate from the legacy
* `embedding` column to the resolved write-side column, so alt-column
* brains do not perpetually re-select rows whose target column is already
* populated (#1262). Must match the eventual upsertChunks target.
*/
countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number>;
countStaleChunks(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number>;
/**
* Sum of LENGTH(chunk_text) over stale chunks the character-count
* backlog the embed phase / embed-backfill will process. Sibling of
@@ -1020,8 +1031,13 @@ export interface BrainEngine {
* model signature (a model/dims swap). NULL signature is GRANDFATHERED
* (never counted) so the post-migration corpus isn't flagged en masse.
* Omit `signature` for the legacy `embedding IS NULL`-only count.
*
* `opts.embeddingColumn` switches the staleness predicate to the resolved
* write-side column (#1262) same contract as countStaleChunks so the
* sync cost gate doesn't count an alt-column brain's fully-embedded corpus
* as phantom backlog.
*/
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number>;
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number>;
/**
* Stamp `pages.embedding_signature = signature` for one page. Called after
* a page's chunks are (re)embedded so a later model swap can detect it as
@@ -1069,6 +1085,9 @@ export interface BrainEngine {
// both round-trip TIMESTAMPTZ as Date | string; ISO string is the
// common denominator on the wire).
afterUpdatedAt?: string | null;
// #1262: staleness predicate targets this column when set (must match
// countStaleChunks and the eventual upsertChunks write target).
embeddingColumn?: ResolvedColumn;
}): Promise<StaleChunkRow[]>;
/**
* Delete every chunk for a page. Internal page-id lookup is sourceId-scoped
@@ -1218,21 +1237,11 @@ export interface BrainEngine {
*
* Uses the `%` trigram operator (GIN-indexed) + the standard `similarity()`
* function. Both engines support pg_trgm (PGLite 0.3+, Postgres always).
*
* `sourceId` constrains the search to a single source and filters out
* soft-deleted pages. Mirrors the same filters `tryFuzzyMatch` in
* `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Omit for the
* historical unscoped behavior live-mode callers that already know
* the source should pass it to avoid cross-source slug suggestions that
* get silently dropped at the FK filter downstream. Batch-mode callers
* (e.g. `gbrain extract`) intentionally omit it to build a cross-source
* resolution map.
*/
findByTitleFuzzy(
name: string,
dirPrefix?: string,
minSimilarity?: number,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null>;
/**
* v0.34.1 (#861 P0 leak seal): `opts.sourceId` / `opts.sourceIds`
+25 -9
View File
@@ -10,7 +10,8 @@ import { findChunkForOffset } from './chunkers/edge-extractor.ts';
import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts';
import { embedBatch, embedMultimodal, currentEmbeddingSignature } from './embedding.ts';
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
import type { ChunkInput, PageInput, PageType } from './types.ts';
import type { ChunkInput, PageInput, PageType, ResolvedColumn } from './types.ts';
import { resolveWriteColumnForEngine } from './search/embedding-column.ts';
import { computeEffectiveDate } from './effective-date.ts';
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
import { logSlugFallback } from './audit-slug-fallback.ts';
@@ -733,11 +734,6 @@ export async function importFromContent(
: computeCorpusGeneration({
crMode: effectiveCRMode,
haikuModel: 'anthropic:claude-haiku-4-5-20251001',
// Inline import-file path never uses per_chunk_synopsis (refuses
// upstream); pass undefined so the doc-cap field stays out of
// the hash here. Per_chunk_synopsis runs through the Minion
// backfill handler which threads SYNOPSIS_DOC_MAX_CHARS through
// the service layer.
});
// Transaction wraps all DB writes. Every per-page tx call carries the
@@ -745,6 +741,14 @@ export async function importFromContent(
// schema DEFAULT — required for multi-source brains; harmless ('default')
// for single-source callers.
const txOpts = sourceId ? { sourceId } : undefined;
// #1262: resolve the write-side embedding column once (merged config +
// gateway model) BEFORE the transaction; the descriptor rides only on
// upsertChunks so text embeddings land in the registered column.
const chunkWriteColumn = await resolveWriteColumnForEngine(engine);
const chunkOpts: { sourceId?: string; embeddingColumn?: ResolvedColumn } | undefined =
(sourceId || chunkWriteColumn)
? { ...(sourceId && { sourceId }), ...(chunkWriteColumn && { embeddingColumn: chunkWriteColumn }) }
: undefined;
await engine.transaction(async (tx) => {
if (existing) await tx.createVersion(slug, txOpts);
@@ -829,7 +833,7 @@ export async function importFromContent(
}
if (chunks.length > 0) {
await tx.upsertChunks(slug, chunks, txOpts);
await tx.upsertChunks(slug, chunks, chunkOpts);
// v0.41.31: stamp embedding provenance when this import actually
// embedded (not --no-embed), so a later model/dims swap is detectable
// as stale via embed --stale. The deferred/backfill + per-slug embed
@@ -1069,6 +1073,12 @@ export async function importCodeFile(
const title = `${relativePath} (${lang})`;
const sourceId = opts.sourceId;
const txOpts = sourceId ? { sourceId } : undefined;
// #1262: write-side embedding column descriptor (rides only on upsertChunks).
const chunkWriteColumn = await resolveWriteColumnForEngine(engine);
const chunkOpts: { sourceId?: string; embeddingColumn?: ResolvedColumn } | undefined =
(sourceId || chunkWriteColumn)
? { ...(sourceId && { sourceId }), ...(chunkWriteColumn && { embeddingColumn: chunkWriteColumn }) }
: undefined;
const byteLength = Buffer.byteLength(content, 'utf-8');
if (byteLength > MAX_FILE_SIZE) {
@@ -1188,7 +1198,7 @@ export async function importCodeFile(
await tx.addTag(slug, lang, txOpts);
if (chunks.length > 0) {
await tx.upsertChunks(slug, chunks, txOpts);
await tx.upsertChunks(slug, chunks, chunkOpts);
// v0.41.31: stamp embedding provenance ONLY when every chunk was
// freshly embedded with the current model this call (no reuse-by-hash
// carrying old-model vectors). Mixed pages stay unstamped rather than
@@ -1337,6 +1347,12 @@ export async function withImportTransaction(
): Promise<void> {
const sourceId = spec.sourceId ?? 'default';
const txOpts = spec.sourceId ? { sourceId: spec.sourceId } : undefined;
// #1262: write-side embedding column descriptor (rides only on upsertChunks).
const chunkWriteColumn = await resolveWriteColumnForEngine(engine);
const chunkOpts: { sourceId?: string; embeddingColumn?: ResolvedColumn } | undefined =
(spec.sourceId || chunkWriteColumn)
? { ...(spec.sourceId && { sourceId: spec.sourceId }), ...(chunkWriteColumn && { embeddingColumn: chunkWriteColumn }) }
: undefined;
await engine.transaction(async (tx) => {
if (spec.hadExisting) await tx.createVersion(spec.slug, txOpts);
await tx.putPage(spec.slug, spec.page, txOpts);
@@ -1352,7 +1368,7 @@ export async function withImportTransaction(
}
if (spec.chunks !== undefined) {
if (spec.chunks.length > 0) {
await tx.upsertChunks(spec.slug, spec.chunks, txOpts);
await tx.upsertChunks(spec.slug, spec.chunks, chunkOpts);
} else {
await tx.deleteChunks(spec.slug, txOpts);
}
+3 -22
View File
@@ -489,22 +489,7 @@ export async function extractPageLinks(
// text inside `[[...]]` before any `|`), NOT the display alias
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
// `struktura`, not "the project". The display text is for context only.
//
// The literal may be path-qualified (`[[notes/struktura]]`). The FS
// path (resolveSlugAll) strips the dirname before its basename lookup,
// but this path passed the raw literal to an index keyed by final
// segments only — so every slash-containing wikilink outside
// DIR_PATTERN silently resolved to nothing. Query by the final
// segment, then use the written path as a disambiguation filter
// (the analogue of the FS ancestor walk honoring the written path):
// a match must end with the literal, so `[[notes/struktura]]` can
// resolve to `vault/notes/struktura` but never to `wiki/struktura`.
const slashIdx = ref.slug.lastIndexOf('/');
const basename = slashIdx === -1 ? ref.slug : ref.slug.slice(slashIdx + 1);
let matches = await resolver.resolveBasenameMatches(basename);
if (slashIdx !== -1) {
matches = matches.filter(m => m === ref.slug || m.endsWith(`/${ref.slug}`));
}
const matches = await resolver.resolveBasenameMatches(ref.slug);
if (matches.length === 0) continue;
const idx = content.indexOf(ref.slug);
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
@@ -980,14 +965,10 @@ export function makeResolver(
// Step 3: pg_trgm fuzzy title match — both modes. Tries each hint in
// order; first hint with a ≥0.55 similarity match wins. If no hints,
// try the whole pages table. When opts.sourceId is set, the fuzzy
// search is constrained to that source (and skips soft-deleted pages)
// so cross-source slug suggestions don't get silently dropped at the
// FK filter downstream. Mirrors the same scope fix `tryFuzzyMatch` got
// via #1436.
// try the whole pages table.
const searchHints = hints.length > 0 ? hints : [undefined];
for (const hint of searchHints) {
const match = await engine.findByTitleFuzzy(trimmed, hint, 0.55, opts.sourceId);
const match = await engine.findByTitleFuzzy(trimmed, hint, 0.55);
if (match) {
cache.set(cacheKey, match.slug);
return match.slug;
@@ -35,6 +35,7 @@ import { tryAcquireDbLock } from '../../db-lock.ts';
import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts';
import { withBudgetTracker } from '../../ai/gateway.ts';
import { embedStaleForSource } from '../../embed-stale.ts';
import { resolveWriteColumnForEngine } from '../../search/embedding-column.ts';
import { currentEmbeddingSignature } from '../../embedding.ts';
import { type DbPacer, createDbPacer, createNoopPacer } from '../../db-pacer.ts';
import { resolvePaceMode, loadPaceModeConfig, readPaceEnv } from '../../pace-mode.ts';
@@ -164,12 +165,16 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) {
// the supervisor, so pacing it is the headline win.
const { pacer, concurrency } = await resolveBackfillPacer(engine, job.data);
// #1262: resolve the write-side embedding column once at the job boundary.
const embeddingColumn = await resolveWriteColumnForEngine(engine);
try {
const result = await withBudgetTracker(tracker, async () =>
embedStaleForSource(engine, sourceId, {
batchSize,
signal: job.signal,
pacer,
...(embeddingColumn && { embeddingColumn }),
...(concurrency !== undefined && { concurrency }),
// v0.41.31: re-embed pages whose model signature drifted + stamp
// provenance as chunks land.
-8
View File
@@ -87,11 +87,6 @@ function walkMarkdownAndMdxFiles(
for (const entry of entries) {
if (truncated) return;
if (entry.startsWith('.')) continue;
// Skip heavy non-content dirs so the walk doesn't exhaust the time
// budget on dependency/build trees (node_modules can be 50k+ files
// with zero .md). These are never gbrain page sources.
if (entry === 'node_modules' || entry === 'dist' || entry === 'build' ||
entry === '.next' || entry === 'vendor' || entry === 'target') continue;
const full = join(d, entry);
let isDir = false;
try {
@@ -100,9 +95,6 @@ function walkMarkdownAndMdxFiles(
continue;
}
if (isDir) {
// Time check on directory descent too, so a deep dependency-free
// tree still respects the deadline even before any .md is found.
if (Date.now() >= deadlineMs) { truncated = true; return; }
walk(full);
continue;
}
+1 -5
View File
@@ -1153,11 +1153,7 @@ async function runAutoLink(
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
// Issue #972 (codex [P1]): pass sourceId so basename resolution stays
// within this page's source — no cross-source basename edges. Also scopes
// the fuzzy fallback (findByTitleFuzzy) to the same source the put_page is
// targeting — without it, cross-source slug suggestions get silently dropped
// at the FK filter and the link looks like it failed to resolve. Twin of
// #1436's `tryFuzzyMatch` fix.
// within this page's source — no cross-source basename edges.
const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId });
// Issue #972: opt-in bare-wikilink basename resolution. Off by default.
const globalBasename = await isGlobalBasenameEnabled(engine);
-116
View File
@@ -1,116 +0,0 @@
/**
* Shared orphan-reporting exclusion policy.
*
* These are pages where "no inbound links" is expected and should not count
* against health. Keep this in core so the CLI orphan report and engine health
* dashboard cannot drift.
*
* Defaults are GBrain-wide conventions only. Brain-specific exclusions
* (private folder names, one-off fixture slugs) belong in the brain's own
* config, not here:
*
* gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
* gbrain config set orphans.exclude_slugs "some-one-off-page"
*/
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
const RAW_SEGMENT = '/raw/';
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'_templates/',
'openclaw/config/',
'extracts/',
];
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
'dreaming',
'daily',
]);
const ROOT_DATE_SLUG = /^\d{4}-\d{2}-\d{2}(?:-.+)?$/;
function isAgentWorkspaceConvention(slug: string): boolean {
if (!slug.startsWith('agents/')) return false;
if (slug.includes('/memory/dreaming/')) return true;
return /^agents\/[^/]+\/(?:agents|identity|soul|tools|user|heartbeat|dreams|dormant)$/.test(slug);
}
/** Per-brain additions to the convention defaults (from config). */
export interface OrphanPolicyOverrides {
excludePrefixes?: string[];
excludeSlugs?: string[];
}
/** Config keys for per-brain orphan exclusions (comma-separated values). */
export const ORPHAN_EXCLUDE_PREFIXES_KEY = 'orphans.exclude_prefixes';
export const ORPHAN_EXCLUDE_SLUGS_KEY = 'orphans.exclude_slugs';
function parseList(value: string | null): string[] {
if (!value) return [];
return value.split(',').map(s => s.trim()).filter(Boolean);
}
/**
* Load per-brain orphan exclusions from the brain config table. Callers with
* an engine in hand (getHealth, `gbrain orphans`) pass the result as the
* second argument to shouldExcludeFromOrphanReporting.
*/
export async function loadOrphanPolicyOverrides(
engine: { getConfig(key: string): Promise<string | null> },
): Promise<OrphanPolicyOverrides> {
const [prefixes, slugs] = await Promise.all([
engine.getConfig(ORPHAN_EXCLUDE_PREFIXES_KEY),
engine.getConfig(ORPHAN_EXCLUDE_SLUGS_KEY),
]);
return { excludePrefixes: parseList(prefixes), excludeSlugs: parseList(slugs) };
}
export function shouldExcludeFromOrphanReporting(
slug: string,
overrides?: OrphanPolicyOverrides,
): boolean {
if (PSEUDO_SLUGS.has(slug)) return true;
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
if (slug.includes(RAW_SEGMENT)) return true;
if (slug.includes('/daily/')) return true;
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
if (ROOT_DATE_SLUG.test(slug)) return true;
if (slug.startsWith('_brain-')) return true;
if (isAgentWorkspaceConvention(slug)) return true;
if (overrides) {
if (overrides.excludeSlugs?.includes(slug)) return true;
for (const prefix of overrides.excludePrefixes ?? []) {
if (slug.startsWith(prefix)) return true;
}
}
return false;
}
+1 -36
View File
@@ -44,33 +44,6 @@ const HAIKU_MAX_TOKENS = 200;
/** Default model when caller doesn't override. Resolves through the gateway. */
const DEFAULT_SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';
/**
* Hard cap on `documentText` length (chars) before send.
*
* 2026-05-25 fix wave: small local chat models (Gemma 4 E2B, Qwen3 4B) get
* dramatically slower on long contexts even with 131K-token windows declared.
* A 73K-char page synopsis on Gemma 4 E2B takes 60-120s, exceeding the
* worker's default 30s `lockDuration` and tripping `lock-lost` errors.
*
* Truncate to a budget that fits a small model's effective throughput while
* preserving enough document context for the synopsis to be useful. Truncates
* the TAIL because the head (title, frontmatter, intro) carries the
* document-level anchor the synopsis needs.
*
* Override per workload via `GBRAIN_SYNOPSIS_DOC_MAX_CHARS`. Default 32768
* (~8K tokens at 4 chars/tok) keeps small-model synopsis under ~30s.
* Anthropic Haiku is unaffected at this cap; bump higher when running
* frontier models if you want richer document anchoring.
*/
export const SYNOPSIS_DOC_MAX_CHARS = (() => {
const env = process.env.GBRAIN_SYNOPSIS_DOC_MAX_CHARS;
if (env && /^\d+$/.test(env)) {
const n = parseInt(env, 10);
if (n >= 512 && n <= 1_048_576) return n;
}
return 32768;
})();
/**
* Synopsis prompt version. Folded into corpus_generation so prompt edits
* invalidate prior embeddings via the v0.40.3.0 query_cache.page_generations
@@ -215,19 +188,11 @@ function buildUserPrompt(
documentText: string,
chunkText: string,
): string {
// Tail-truncate `documentText` to `SYNOPSIS_DOC_MAX_CHARS` so small local
// chat models don't stall on >100KB pages. Head preserved (title block,
// frontmatter, intro paragraphs carry the document-level anchor).
let trimmedDoc = documentText;
if (documentText.length > SYNOPSIS_DOC_MAX_CHARS) {
trimmedDoc = documentText.slice(0, SYNOPSIS_DOC_MAX_CHARS) +
`\n\n[... ${documentText.length - SYNOPSIS_DOC_MAX_CHARS} chars truncated for synopsis budget ...]`;
}
return [
`<page_title>${pageTitle}</page_title>`,
'',
'<full_document>',
trimmedDoc,
documentText,
'</full_document>',
'',
'<chunk>',
+70 -96
View File
@@ -40,6 +40,7 @@ import type {
BrainStats, BrainHealth,
IngestLogEntry, IngestLogInput,
EngineConfig,
ResolvedColumn,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult,
@@ -57,8 +58,6 @@ import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
import {
normalizeEngineColumn,
buildVectorCastFragment,
@@ -1060,16 +1059,6 @@ export class PGLiteEngine implements BrainEngine {
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename, source_kind, source_uri, ingested_via, ingested_at`,
[sourceId, slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename, chunkerVersion, sourcePath, sourceKind, sourceUri, ingestedVia, ingestedAt]
);
// PGLite can return zero rows from INSERT ... ON CONFLICT DO UPDATE ...
// RETURNING in no-op/trigger edge cases, which made rowToPage(undefined)
// throw "undefined is not an object (evaluating 'row.deleted_at')" and
// skip the file during sync. The row WAS written, so re-read instead of
// crashing.
if (rows.length === 0) {
const reread = await this.getPage(slug, { sourceId });
if (reread) return reread;
throw new Error(`putPage: RETURNING produced no row for ${sourceId}/${slug}`);
}
return rowToPage(rows[0] as Record<string, unknown>);
}
@@ -2242,12 +2231,20 @@ export class PGLiteEngine implements BrainEngine {
}
// Chunks
async upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string } & BatchOpts): Promise<void> {
async upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string; embeddingColumn?: ResolvedColumn } & BatchOpts): Promise<void> {
return this.batchRetry(opts?.auditSite ?? 'upsertChunks', opts?.signal, () => this._upsertChunksOnce(slug, chunks, opts), chunks.length);
}
private async _upsertChunksOnce(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string }): Promise<void> {
private async _upsertChunksOnce(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string; embeddingColumn?: ResolvedColumn }): Promise<void> {
const sourceId = opts?.sourceId ?? 'default';
// #1262: caller-resolved write target for TEXT embeddings. Descriptor
// names are identifier-validated + quoted by buildVectorCastFragment;
// omitted => legacy `embedding vector`. Mirrors postgres-engine.ts.
const targetFragment = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn)
: undefined;
const targetCol = targetFragment?.col ?? 'embedding';
const embeddingCast = targetFragment?.castSql.replace('$1::', '') ?? 'vector';
// Source-scope the page-id lookup so duplicate slugs in different sources
// do not return multiple rows or target the wrong page.
@@ -2282,7 +2279,7 @@ export class PGLiteEngine implements BrainEngine {
// list. Image chunks pass embedding=null + embedding_image=Float32Array
// (1024-dim Voyage). Text/code chunks pass embedding=Float32Array +
// embedding_image=null. Default modality='text' when omitted.
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified, modality, embedding_image)';
const cols = `(page_id, chunk_index, chunk_text, chunk_source, ${targetCol}, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified, modality, embedding_image)`;
const rowParts: string[] = [];
const params: unknown[] = [];
let paramIdx = 1;
@@ -2300,7 +2297,7 @@ export class PGLiteEngine implements BrainEngine {
const modality = chunk.modality ?? 'text';
// Inline ::vector NULL literals to avoid a per-branch placeholder.
const embeddingPh = embeddingStr ? `$${paramIdx++}::vector` : 'NULL';
const embeddingPh = embeddingStr ? `$${paramIdx++}::${embeddingCast}` : 'NULL';
const embeddedAtPh = embeddingStr ? 'now()' : 'NULL';
const embeddingImagePh = embeddingImageStr ? `$${paramIdx++}::vector` : 'NULL';
@@ -2334,41 +2331,37 @@ export class PGLiteEngine implements BrainEngine {
// v0.40.3.0 D24 NULL→non-NULL race fix mirrors postgres-engine.ts. Two writers
// racing on the same chunk previously raced last-write-wins; the fix lets the
// fresher `embedded_at` win in the text-unchanged branch.
//
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
// only embedding-shaped fields doesn't clobber metadata to NULL.
await this.db.query(
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
chunk_text = EXCLUDED.chunk_text,
chunk_source = EXCLUDED.chunk_source,
embedding = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.embedding
${targetCol} = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.${targetCol}
WHEN content_chunks.${targetCol} IS NULL THEN EXCLUDED.${targetCol}
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.embedding
ELSE content_chunks.embedding
THEN EXCLUDED.${targetCol}
ELSE content_chunks.${targetCol}
END,
model = COALESCE(EXCLUDED.model, content_chunks.model),
token_count = EXCLUDED.token_count,
embedded_at = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
WHEN content_chunks.embedding IS NULL AND EXCLUDED.embedding IS NOT NULL THEN EXCLUDED.embedded_at
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.${targetCol} IS NULL THEN NULL
WHEN content_chunks.${targetCol} IS NULL AND EXCLUDED.${targetCol} IS NOT NULL THEN EXCLUDED.embedded_at
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params
@@ -2393,14 +2386,19 @@ export class PGLiteEngine implements BrainEngine {
* drift (NULL grandfathered never stale). Shared by countStaleChunks +
* sumStaleChunkChars so they can't drift.
*/
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): { where: string; params: unknown[] } {
// #1262: staleness targets the caller-resolved write column when set
// (identifier-validated + quoted); legacy `embedding` otherwise.
const staleCol = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn).col
: 'embedding';
const params: unknown[] = [];
const conds: string[] = [];
if (opts?.signature !== undefined) {
params.push(opts.signature);
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
conds.push(`(cc.${staleCol} IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
} else {
conds.push(`cc.embedding IS NULL`);
conds.push(`cc.${staleCol} IS NULL`);
}
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
if (opts?.sourceId !== undefined) {
@@ -2410,7 +2408,7 @@ export class PGLiteEngine implements BrainEngine {
return { where: conds.join(' AND '), params };
}
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async countStaleChunks(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number> {
// D7: source-scoped count for `gbrain embed --stale --source X`. Always
// JOIN pages so embed-skip + signature predicates apply. PGLite is
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
@@ -2426,7 +2424,7 @@ export class PGLiteEngine implements BrainEngine {
return Number(count);
}
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number> {
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
// length for the sync cost preview. ::bigint guards int4 overflow.
const { where, params } = this.buildStaleChunkWhere(opts);
@@ -2479,11 +2477,17 @@ export class PGLiteEngine implements BrainEngine {
sourceId?: string;
orderBy?: 'page_id' | 'updated_desc';
afterUpdatedAt?: string | null;
embeddingColumn?: ResolvedColumn;
}): Promise<StaleChunkRow[]> {
const limit = opts?.batchSize ?? 2000;
const afterPid = opts?.afterPageId ?? 0;
const afterIdx = opts?.afterChunkIndex ?? -1;
const orderBy = opts?.orderBy ?? 'page_id';
// #1262: staleness follows the caller-resolved write column (validated +
// quoted identifier); legacy `embedding` otherwise.
const staleCol = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn).col
: 'embedding';
// v0.41.18.0 (A13, codex #9): --priority recent path. See postgres-engine
// sibling for full rationale. Same composite cursor + ORDER BY.
@@ -2497,7 +2501,7 @@ export class PGLiteEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT $1`,
@@ -2508,7 +2512,7 @@ export class PGLiteEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < $1::timestamptz
@@ -2527,7 +2531,7 @@ export class PGLiteEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND p.source_id = $1
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
@@ -2539,7 +2543,7 @@ export class PGLiteEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND p.source_id = $1
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
@@ -2564,7 +2568,7 @@ export class PGLiteEngine implements BrainEngine {
cc.model, cc.token_count, p.source_id, cc.page_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (cc.page_id, cc.chunk_index) > ($1, $2)
ORDER BY cc.page_id, cc.chunk_index
@@ -2578,7 +2582,7 @@ export class PGLiteEngine implements BrainEngine {
cc.model, cc.token_count, p.source_id, cc.page_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE cc.${staleCol} IS NULL
AND p.source_id = $1
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (cc.page_id, cc.chunk_index) > ($2, $3)
@@ -2922,41 +2926,22 @@ export class PGLiteEngine implements BrainEngine {
name: string,
dirPrefix?: string,
minSimilarity: number = 0.55,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null> {
// Inline threshold comparison instead of `SET LOCAL pg_trgm.similarity_threshold`.
// The GUC only scopes to the current transaction and pglite auto-commits each
// .query() call, so the SET LOCAL would be a no-op. Using similarity() >= $N
// directly gives predictable behavior. Tie-breaker: sort by slug so re-runs
// pick the same winner.
//
// `sourceId` + `deleted_at IS NULL` mirror the filters `tryFuzzyMatch` in
// `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Without them,
// fuzzy resolution could suggest cross-source slugs that the caller then
// silently drops at the FK filter — making it look like the match failed
// when in fact it picked the wrong page.
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
const { rows } = sourceId
? await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
AND source_id = $4
AND deleted_at IS NULL
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity, sourceId]
)
: await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity]
);
const { rows } = await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity]
);
if (rows.length === 0) return null;
const row = rows[0] as { slug: string; sim: number };
return { slug: row.slug, similarity: row.sim };
@@ -5242,10 +5227,15 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound). The raw
-- list is filtered in TS using the shared orphan-reporting policy.
0 as orphan_pages,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
@@ -5270,20 +5260,10 @@ export class PGLiteEngine implements BrainEngine {
LIMIT 5
`);
const { rows: islandedRows } = await this.db.query(`
SELECT p.slug
FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`);
const r = h as Record<string, unknown>;
const pageCount = Number(r.page_count);
const embedCoverage = Number(r.embed_coverage);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = (islandedRows as { slug: string }[])
.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const orphanPages = Number(r.orphan_pages);
const deadLinks = Number(r.dead_links);
const linkCount = Number(r.link_count);
const pagesWithTimeline = Number(r.pages_with_timeline);
@@ -5311,7 +5291,7 @@ export class PGLiteEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: stalePages,
stale_pages: Number(r.stale_pages),
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,
@@ -5866,11 +5846,6 @@ export class PGLiteEngine implements BrainEngine {
params.push(escaped);
prefixCondition = `AND p.slug LIKE $${params.length} ESCAPE '\\'`;
}
// TIM-37: exclude briefing pages from their own Brain Pulse. See the
// matching block in postgres-engine.ts getRecentSalience() for context.
const excludeBriefings = !(slugPrefix && slugPrefix.startsWith('briefings'))
? `AND p.slug NOT LIKE 'briefings/%'`
: '';
params.push(limit);
const limitParam = `$${params.length}`;
@@ -5906,7 +5881,6 @@ export class PGLiteEngine implements BrainEngine {
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= $1::timestamptz
${prefixCondition}
${excludeBriefings}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limitParam}`,
+73 -90
View File
@@ -50,6 +50,7 @@ import type {
BrainStats, BrainHealth,
IngestLogEntry, IngestLogInput,
EngineConfig,
ResolvedColumn,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult,
@@ -67,8 +68,6 @@ import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -2382,13 +2381,21 @@ export class PostgresEngine implements BrainEngine {
}
// Chunks
async upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string } & BatchOpts): Promise<void> {
async upsertChunks(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string; embeddingColumn?: ResolvedColumn } & BatchOpts): Promise<void> {
return this.batchRetry(opts?.auditSite ?? 'upsertChunks', opts?.signal, () => this._upsertChunksOnce(slug, chunks, opts), chunks.length);
}
private async _upsertChunksOnce(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string }): Promise<void> {
private async _upsertChunksOnce(slug: string, chunks: ChunkInput[], opts?: { sourceId?: string; embeddingColumn?: ResolvedColumn }): Promise<void> {
const sql = this.sql;
const sourceId = opts?.sourceId ?? 'default';
// #1262: caller-resolved write target for TEXT embeddings. Descriptor
// names are identifier-validated + quoted by buildVectorCastFragment;
// omitted => legacy `embedding vector`.
const targetFragment = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn)
: undefined;
const targetCol = targetFragment?.col ?? 'embedding';
const embeddingCast = targetFragment?.castSql.replace('$1::', '') ?? 'vector';
// Source-scope the page-id lookup. Without this filter, multi-source
// brains where the slug exists in 2+ sources return >1 row and the
@@ -2415,7 +2422,7 @@ export class PostgresEngine implements BrainEngine {
// scope metadata through upserts.
// v0.27.1 (Phase 8): added `modality` + `embedding_image` to the column
// list. Image chunks pass embedding=null + embedding_image=Float32Array.
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified, modality, embedding_image)';
const cols = `(page_id, chunk_index, chunk_text, chunk_source, ${targetCol}, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified, modality, embedding_image)`;
const rows: string[] = [];
const params: unknown[] = [];
let paramIdx = 1;
@@ -2432,7 +2439,7 @@ export class PostgresEngine implements BrainEngine {
: null;
const modality = chunk.modality ?? 'text';
const embeddingPh = embeddingStr ? `$${paramIdx++}::vector` : 'NULL';
const embeddingPh = embeddingStr ? `$${paramIdx++}::${embeddingCast}` : 'NULL';
const embeddedAtPh = embeddingStr ? 'now()' : 'NULL';
const embeddingImagePh = embeddingImageStr ? `$${paramIdx++}::vector` : 'NULL';
@@ -2475,44 +2482,37 @@ export class PostgresEngine implements BrainEngine {
// - new is fresher (embedded_at > existing.embedded_at) → take new
// - otherwise → keep existing (slower writer with stale embedding loses)
// Mirrored in pglite-engine.ts; pinned by test/e2e/concurrent-embed-race.test.ts.
//
// Code-chunk metadata columns (language / symbol_name / symbol_type / line range /
// parent_symbol_path / doc_comment / symbol_name_qualified) follow the SAME chunk_text-gated
// CASE pattern as `embedding` (#769). Re-chunk (chunk_text changed) trusts EXCLUDED outright;
// pure re-embed (chunk_text unchanged) COALESCEs so a caller that only carries embedding
// doesn't clobber metadata to NULL. Without this, every embed --stale pass nuked code-def's
// primary index for thousands of chunks at once.
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
chunk_text = EXCLUDED.chunk_text,
chunk_source = EXCLUDED.chunk_source,
embedding = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.embedding
${targetCol} = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.${targetCol}
WHEN content_chunks.${targetCol} IS NULL THEN EXCLUDED.${targetCol}
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.embedding
ELSE content_chunks.embedding
THEN EXCLUDED.${targetCol}
ELSE content_chunks.${targetCol}
END,
model = COALESCE(EXCLUDED.model, content_chunks.model),
token_count = EXCLUDED.token_count,
embedded_at = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
WHEN content_chunks.embedding IS NULL AND EXCLUDED.embedding IS NOT NULL THEN EXCLUDED.embedded_at
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.${targetCol} IS NULL THEN NULL
WHEN content_chunks.${targetCol} IS NULL AND EXCLUDED.${targetCol} IS NOT NULL THEN EXCLUDED.embedded_at
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params as Parameters<typeof sql.unsafe>[1],
@@ -2539,14 +2539,19 @@ export class PostgresEngine implements BrainEngine {
* embedding_signature drift (NULL grandfathered). Shared by
* countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling).
*/
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): { where: string; params: unknown[] } {
// #1262: staleness targets the caller-resolved write column when set
// (identifier-validated + quoted); legacy `embedding` otherwise.
const staleCol = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn).col
: 'embedding';
const params: unknown[] = [];
const conds: string[] = [];
if (opts?.signature !== undefined) {
params.push(opts.signature);
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
conds.push(`(cc.${staleCol} IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
} else {
conds.push(`cc.embedding IS NULL`);
conds.push(`cc.${staleCol} IS NULL`);
}
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
if (opts?.sourceId !== undefined) {
@@ -2556,7 +2561,7 @@ export class PostgresEngine implements BrainEngine {
return { where: conds.join(' AND '), params };
}
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async countStaleChunks(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number> {
// Always JOIN pages so the embed_skip + signature predicates apply.
// D7: source_id scoping. v0.41.31: optional signature widens staleness
// to embedding_signature drift (NULL grandfathered).
@@ -2574,7 +2579,7 @@ export class PostgresEngine implements BrainEngine {
});
}
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; embeddingColumn?: ResolvedColumn }): Promise<number> {
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
// length for the sync cost preview. ::bigint guards int4 overflow.
const { where, params } = this.buildStaleChunkWhere(opts);
@@ -2627,11 +2632,18 @@ export class PostgresEngine implements BrainEngine {
sourceId?: string;
orderBy?: 'page_id' | 'updated_desc';
afterUpdatedAt?: string | null;
embeddingColumn?: ResolvedColumn;
}): Promise<StaleChunkRow[]> {
const limit = opts?.batchSize ?? 2000;
const afterPid = opts?.afterPageId ?? 0;
const afterIdx = opts?.afterChunkIndex ?? -1;
const orderBy = opts?.orderBy ?? 'page_id';
// #1262: staleness follows the caller-resolved write column (validated +
// quoted identifier); legacy `embedding` otherwise. Interpolated below as
// an unsafe FRAGMENT (identifiers can't be bound parameters).
const staleCol = opts?.embeddingColumn
? buildVectorCastFragment(opts.embeddingColumn).col
: 'embedding';
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
@@ -2648,7 +2660,7 @@ export class PostgresEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT ${limit}
@@ -2658,7 +2670,7 @@ export class PostgresEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < ${afterUpdated}::timestamptz
@@ -2676,7 +2688,7 @@ export class PostgresEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND p.source_id = ${opts.sourceId}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
@@ -2687,7 +2699,7 @@ export class PostgresEngine implements BrainEngine {
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND p.source_id = ${opts.sourceId}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
@@ -2707,7 +2719,7 @@ export class PostgresEngine implements BrainEngine {
cc.model, cc.token_count, p.source_id, cc.page_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
ORDER BY cc.page_id, cc.chunk_index
@@ -2720,7 +2732,7 @@ export class PostgresEngine implements BrainEngine {
cc.model, cc.token_count, p.source_id, cc.page_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
WHERE ${tx.unsafe(`cc.${staleCol} IS NULL`)}
AND p.source_id = ${opts.sourceId}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
@@ -3082,7 +3094,6 @@ export class PostgresEngine implements BrainEngine {
name: string,
dirPrefix?: string,
minSimilarity: number = 0.55,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null> {
const sql = this.sql;
// Use the `similarity()` function directly with an explicit threshold
@@ -3095,33 +3106,15 @@ export class PostgresEngine implements BrainEngine {
// Tie-breaker: sort by slug after similarity so re-runs return the
// same winner when multiple pages score equally (prevents churn
// in put_page auto-link reconciliation).
//
// `sourceId` + `deleted_at IS NULL` mirror the filters `tryFuzzyMatch`
// in `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Without
// them, fuzzy resolution could suggest cross-source slugs that the
// caller then silently drops at the FK filter in
// `operations.ts:reconcileLinks` (the `allSlugs` filter) — making it
// look like the match failed when in fact it picked the wrong page.
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
const rows = sourceId
? await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
AND source_id = ${sourceId}
AND deleted_at IS NULL
ORDER BY sim DESC, slug ASC
LIMIT 1
`
: await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
ORDER BY sim DESC, slug ASC
LIMIT 1
`;
const rows = await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
ORDER BY sim DESC, slug ASC
LIMIT 1
`;
if (rows.length === 0) return null;
const row = rows[0] as { slug: string; sim: number };
return { slug: row.slug, similarity: row.sim };
@@ -5341,9 +5334,11 @@ export class PostgresEngine implements BrainEngine {
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
@@ -5352,8 +5347,13 @@ export class PostgresEngine implements BrainEngine {
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
0 as orphan_pages,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
@@ -5377,18 +5377,9 @@ export class PostgresEngine implements BrainEngine {
LIMIT 5
`;
const islandedRows = await sql<{ slug: string }[]>`
SELECT p.slug
FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`;
const pageCount = Number(h.page_count);
const embedCoverage = Number(h.embed_coverage);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = islandedRows.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const orphanPages = Number(h.orphan_pages);
const deadLinks = Number(h.dead_links);
const linkCount = Number(h.link_count);
const pagesWithTimeline = Number(h.pages_with_timeline);
@@ -5416,7 +5407,7 @@ export class PostgresEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: stalePages,
stale_pages: Number(h.stale_pages),
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
@@ -6172,13 +6163,6 @@ export class PostgresEngine implements BrainEngine {
const prefixCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
// TIM-37: exclude briefing pages from their own Brain Pulse. The cron
// briefing writes to 90_Briefings/, gets re-ingested, and would otherwise
// top tomorrow's salience as pure self-reference. Suppress unless the
// caller explicitly asked for the briefings/ prefix.
const excludeBriefings = !(slugPrefix && slugPrefix.startsWith('briefings'))
? sql`AND p.slug NOT LIKE 'briefings/%'`
: sql``;
// v0.29.1: third score term via buildRecencyComponentSql. Default
// 'flat' = v0.29.0 behavior (1 / (1 + days_old)). 'on' opts into the
// per-prefix decay map (concepts/ evergreen, daily/ aggressive, etc.).
@@ -6212,7 +6196,6 @@ export class PostgresEngine implements BrainEngine {
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= ${boundaryIso}::timestamptz
${prefixCondition}
${excludeBriefings}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limit}
+3 -35
View File
@@ -39,14 +39,9 @@ export type RemediationStatus = 'remediable' | 'human_only' | 'blocked';
* References other steps via depends_on.
* job Minion handler name. Must match a registered handler.
* params passed verbatim to the handler.
* idempotency_key content-hash dedup key. Same (job, params) same
* key (exactly-once: the queue's unique partial index
* means a re-submit returns the existing row, terminal
* or not). The --remediate loop therefore attempts a
* step at most once per run and excludes terminally-
* failed ids from later rechecks (see selectActiveRecs);
* it does NOT mutate this key. The next scheduled run
* retries with a fresh abort set.
* idempotency_key content-hash dedup key. Same (job, params)
* same key. Across retries (--remediate re-runs)
* pre-existing failed jobs append `:r<N>` suffix.
* severity drives ordering.
* est_seconds upper-bound runtime estimate for budgeting.
* est_usd_cost USD cost estimate when applicable.
@@ -155,30 +150,3 @@ export function makeRemediationStep(opts: {
status: opts.status ?? 'remediable',
};
}
/**
* Select the steps that are still actionable on the current --remediate
* pass. The loop recomputes recommendations from fresh health after every
* step (D7); this filters that recomputed set to:
* - status === 'remediable' (the only kind that executes), and
* - ids NOT already in `abortedIds`.
*
* The second clause is the loop-termination guarantee. A step that reached a
* terminal NON-completed state this run (failed / dead / cancelled) is in
* `abortedIds`. Because idempotency keys are content-stable and queue.add
* already exhausted `max_attempts`, re-dispatching such a step returns the
* SAME terminal row forever so if its underlying metric never improves the
* recommendation re-fires every recheck and the loop spins (the failure a
* cancelled job triggers under `onboard --auto`). Excluding aborted ids makes
* progress monotonic: each step is attempted at most once per run, then
* abandoned until the next scheduled run (which starts with an empty set).
*
* Pure + side-effect-free so the loop-progress invariant is unit-testable
* without the (deferred) Minion-worker stub harness.
*/
export function selectActiveRecs(
recs: RemediationStep[],
abortedIds: ReadonlySet<string>,
): RemediationStep[] {
return recs.filter((r) => r.status === 'remediable' && !abortedIds.has(r.id));
}
+6 -28
View File
@@ -15,7 +15,7 @@ import type { BrainEngine } from '../engine.ts';
import {
computeRecommendations,
} from '../brain-score-recommendations.ts';
import { selectActiveRecs, type RemediationStep } from '../remediation-step.ts';
import type { RemediationStep } from '../remediation-step.ts';
import { loadRecommendationContext } from './context.ts';
import { computeRemediationPlan } from './plan.ts';
import type {
@@ -246,23 +246,6 @@ export async function runRemediation(
continue;
}
// Forward-progress guard (loop-termination fix). A step in abortedIds
// already reached a terminal NON-completed state earlier this run.
// Idempotency keys are content-stable (same job+params → same key) and
// queue.add already retried via max_attempts, so re-dispatching returns
// the SAME terminal row forever — an infinite loop when the step's metric
// never improves (the failure a cancelled job triggers under
// onboard --auto). selectActiveRecs below normally keeps aborted ids out
// of recs; this is the explicit, locally-checkable backstop. Skip without
// re-adding to abortedIds (already present → aborted_count unaffected).
if (abortedIds.has(step.id)) {
const result: StepResult = { step: stepCount, id: step.id, job_id: null, status: 'skipped_already_aborted' };
submitted.push(result);
hooks.onStepEnd?.(result);
recs.shift();
continue;
}
hooks.onStepStart?.(stepCount, totalSteps, step);
try {
const isProtected = !!step.protected;
@@ -317,17 +300,12 @@ export async function runRemediation(
}
recs.shift();
// D7: scoped recheck — re-compute the plan from a fresh health snapshot.
// selectActiveRecs drops completed steps (the metric improved, so the
// recommendation no longer fires) AND excludes ids already in abortedIds,
// so a terminally-failed/cancelled step is attempted at most once per run
// instead of being re-introduced every recheck. The `:r<N>` retry-suffix
// replay referenced by older comments was never implemented and is
// unnecessary: queue.add already retries via max_attempts, and the next
// scheduled run starts fresh (empty abortedIds).
// D7: scoped recheck — re-compute plan from fresh health snapshot.
// The next plan may drop completed steps and re-introduce failed
// steps with bumped retry suffix (D1).
if (recs.length === 0 || stepCount >= maxJobs) break;
const freshHealth = await engine.getHealth();
recs = selectActiveRecs(computeRecommendations(freshHealth, ctx), abortedIds);
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
}
};
@@ -345,7 +323,7 @@ export async function runRemediation(
// Clear checkpoint on a clean run (no budget abort). Failed steps in the
// submitted set don't disqualify the cleanup — they re-surface on the
// next scheduled run, which starts with a fresh abortedIds set.
// next plan with bumped suffixes.
if (!budgetAbort) {
clearRemediationCheckpoint(planHash);
}
+1 -2
View File
@@ -67,8 +67,7 @@ export interface RemediationOpts {
/**
* Result of one step. status mirrors Minion job terminal states + a few
* synthetic ones ('skipped_dep_aborted', 'skipped_already_aborted',
* 'skipped_completed_in_checkpoint', 'dry_run').
* synthetic ones ('skipped_dep_aborted', 'skipped_completed_in_checkpoint').
*/
export interface StepResult {
step: number;
+74
View File
@@ -443,6 +443,80 @@ export function resolveEmbeddingColumn(
};
}
/**
* Resolves the WRITE-side embedding column for the currently configured
* embedding model (#1262). The read-side resolver above answers "which
* column does this query search?"; this one answers "which column should
* newly produced text embeddings land in?".
*
* Unlike read-side search, writes take no per-call column override. The
* import/embed boundary resolves once from merged config + gateway state
* and passes the descriptor into `engine.upsertChunks`; engines stay
* config-free (same contract as the read-side descriptor).
*
* Behavior:
* - no user-declared `embedding_columns` => undefined (legacy brain,
* writes keep targeting the default `embedding` column)
* - a user-declared entry whose `provider` matches the current
* embedding model => that entry's descriptor
* - no provider match => undefined (fall back to legacy `embedding`)
*
* Only USER-declared entries are consulted never the cfg-derived
* builtins. The `embedding_image` builtin's provider is the multimodal
* model; matching it here would misroute text embeddings into the image
* column. The no-match fallback is intentional: switching models before
* registering a matching column must not silently write vectors into an
* arbitrary column.
*/
export function resolveWriteColumn(cfg: GBrainConfig): ResolvedColumn | undefined {
const userColumns = cfg.embedding_columns;
if (
!userColumns ||
typeof userColumns !== 'object' ||
Array.isArray(userColumns) ||
Object.keys(userColumns).length === 0
) {
return undefined;
}
// Same model-resolution chain as the registry builtin: cfg > gateway > default.
let gwModel: string | undefined;
try {
const gw = require('../ai/gateway.ts') as typeof import('../ai/gateway.ts');
gwModel = gw.getEmbeddingModel();
} catch {
// Gateway unconfigured — fall through to the canonical default.
}
const currentModel = cfg.embedding_model ?? gwModel ?? DEFAULT_EMBEDDING_MODEL;
for (const [name, entry] of Object.entries(userColumns)) {
if (!entry) continue;
validateColumnKey(name);
validateColumnConfig(name, entry);
if (entry.provider !== currentModel) continue;
return {
name,
type: entry.type,
dimensions: entry.dimensions,
embeddingModel: entry.provider,
};
}
return undefined;
}
/**
* Engine-boundary convenience: merged config (file/env + DB plane)
* resolveWriteColumn. Dynamic import keeps config.ts out of this module's
* static graph (mirrors the gateway require above).
*/
export async function resolveWriteColumnForEngine(
engine: { getConfig(key: string): Promise<string | null | undefined> },
): Promise<ResolvedColumn | undefined> {
const { loadConfigWithEngine } = await import('../config.ts');
const cfg = await loadConfigWithEngine(engine);
return cfg ? resolveWriteColumn(cfg) : undefined;
}
/**
* True when the resolved column is the default `embedding` name.
* Name-based check; does not compare embedding space.
+4 -10
View File
@@ -93,13 +93,7 @@ import { resolveLrSchedule } from './lr-schedule.ts';
import { preflight, formatPreflightReport } from './preflight.ts';
import { isRejected, loadRejectedBuffer, makeRejectedEntry, saveRejectedBuffer } from './rejected-buffer.ts';
import { runReflect, runOneShotRewrite, describeJudges } from './reflect.ts';
import {
acceptCandidate,
proposedPath as proposedFilePath,
revertAllPending,
skillPath,
writeProposed,
} from './version-store.ts';
import { acceptCandidate, bestPath, revertAllPending, skillPath, writeProposed } from './version-store.ts';
import { runValidationGate, scoreSkillOnTasks } from './validate-gate.ts';
import { ROLLOUT_SUCCESS_THRESHOLD } from './types.ts';
import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask } from './types.ts';
@@ -708,9 +702,9 @@ async function runOptimizationLoop(
// to the catch's assignment values only (it can't prove the async callback ran).
const finalOutcome = outcome as 'accepted' | 'no_improvement' | 'aborted' | 'errored';
if (!mutateDecision.mutate && finalOutcome === 'accepted') {
// writeProposed() emitted both the best pointer and the stable review
// artifact in the accept branch. SKILL.md remains untouched.
proposedPath = proposedFilePath(skillsDir, skillName);
// best.md was written by writeProposed() in the accept branch (no-mutate
// path); it doubles as proposed.md for human review. SKILL.md untouched.
proposedPath = bestPath(skillsDir, skillName);
} else if (mutateDecision.mutate) {
mutatedSkillFile = finalOutcome === 'accepted';
}
+9 -15
View File
@@ -23,7 +23,6 @@
*
* history.json
* best.md
* proposed.md
* versions/
* v0001_e1_s1.md
* v0002_e1_s2.md
@@ -53,10 +52,6 @@ export function bestPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'best.md');
}
export function proposedPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'proposed.md');
}
export function skillPath(skillsDir: string, skillName: string): string {
return path.join(skillsDir, skillName, 'SKILL.md');
}
@@ -176,18 +171,17 @@ export function acceptCandidate(input: AcceptInput): AcceptResult {
}
/**
* Write the candidate to both `best.md` and `proposed.md` WITHOUT touching
* SKILL.md or the history ledger. `best.md` remains the optimizer's current
* best pointer; `proposed.md` is the stable human-review artifact promised by
* `--no-mutate`. Returns the proposal path. Each write is atomic (.tmp + rename).
* Write the candidate to `best.md` (which doubles as `proposed.md`) WITHOUT
* touching SKILL.md or the history ledger. Used by the `--no-mutate` /
* bundled-without-allow paths: the optimizer found a better candidate but the
* caller opted out of in-place mutation, so we surface it for human review.
* Returns the path written. Atomic (.tmp + rename).
*/
export function writeProposed(skillsDir: string, skillName: string, candidateText: string): string {
const best = bestPath(skillsDir, skillName);
const proposed = proposedPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(best), { recursive: true });
atomicWrite(best, candidateText);
atomicWrite(proposed, candidateText);
return proposed;
const p = bestPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(p), { recursive: true });
atomicWrite(p, candidateText);
return p;
}
/**
+1 -35
View File
@@ -553,40 +553,6 @@ export async function runThink(
};
}
/**
* Strip a "## Gaps" section from an answer body.
*
* `think` returns gaps in the structured `gaps` array, which the CLI and the
* persisted synthesis page render exactly once. The system prompt also used to
* ask for a "Gaps" section inside the answer prose, so a model that still emits
* one would make the output show "## Gaps" twice once from the prose, once
* from the structured array. This removes the prose section so the structured
* array stays the single source of truth.
*
* Matches a heading line `## Gaps` (level 2-6, case-insensitive) and removes it
* through the next heading of the same-or-higher level, or end of string.
* Returns the input unchanged when there is no such section.
*/
export function stripGapsSection(answer: string): string {
if (!answer) return answer;
const lines = answer.split('\n');
let start = -1;
let level = 0;
for (let i = 0; i < lines.length; i++) {
const m = /^(#{2,6})\s+gaps\s*$/i.exec(lines[i]);
if (m) { start = i; level = m[1].length; break; }
}
if (start === -1) return answer;
let end = lines.length;
for (let i = start + 1; i < lines.length; i++) {
const h = /^(#{1,6})\s+\S/.exec(lines[i]);
if (h && h[1].length <= level) { end = i; break; }
}
const kept = [...lines.slice(0, start), ...lines.slice(end)].join('\n');
// Drop trailing blank lines left by removing a trailing section.
return kept.replace(/\s+$/, '');
}
/**
* Persist a synthesis page + its evidence. Returns the saved slug.
* Synthesis pages are written under `synthesis/<slugified-question>-<date>.md`.
@@ -616,7 +582,7 @@ export async function persistSynthesis(
const body = [
`# ${result.question}`,
'',
stripGapsSection(result.answer),
result.answer,
'',
result.gaps.length > 0 ? '## Gaps\n\n' + result.gaps.map(g => `- ${g}`).join('\n') : '',
].filter(Boolean).join('\n');
+6 -6
View File
@@ -52,19 +52,19 @@ Hard rules:
rather than asserting it as established. Confidence is part of the data.
- If two takes contradict (different holders, opposite claims), surface BOTH in a "Conflicts"
section. Never silently pick one.
- If the brain doesn't contain data needed to answer, do NOT make it up. Record each
missing piece in the structured "gaps" array (below), not as a section in the answer prose.
- If you cannot answer because the brain doesn't contain the relevant data, say so in the
"Gaps" section. List the specific missing pieces. Do not make up answers.
- Never instruct the user (no "you should" / "I recommend X"). The brain reports; the user decides.
- Output MUST be valid JSON matching the schema below. No prose outside JSON.
Output schema:
{
"answer": "<markdown body. Inline citations like [slug#row] or [slug]. Sections: Answer, Conflicts (optional). Do NOT add a Gaps section here — gaps belong in the gaps array.>",
"answer": "<markdown body. Inline citations like [slug#row] or [slug]. Sections: Answer, Conflicts (optional), Gaps>",
"citations": [
{"page_slug": "people/alice-example", "row_num": 3, "citation_index": 1},
{"page_slug": "companies/acme-example", "row_num": null, "citation_index": 2}
],
"gaps": ["a specific, self-contained missing-or-stale data point, citing the [slug] where relevant", "another specific gap"]
"gaps": ["specific missing data point 1", "specific missing data point 2"]
}
The "row_num" field is required for take citations and MUST be null for page-only citations.`;
@@ -83,7 +83,7 @@ export function buildThinkSystemPrompt(opts: ThinkSystemPromptOpts = {}): string
lines.push(`\nThis is a temporal question. Order key claims chronologically when it helps the reader.`);
}
if (opts.willSave) {
lines.push(`\nThis synthesis will be persisted as a brain page. Aim for completeness — cover the Answer and any Conflicts thoroughly, and list every missing piece in the structured "gaps" array.`);
lines.push(`\nThis synthesis will be persisted as a brain page. Aim for completeness — cover Answer, Conflicts, and Gaps thoroughly.`);
}
if (opts.withCalibration) {
lines.push(
@@ -92,7 +92,7 @@ export function buildThinkSystemPrompt(opts: ThinkSystemPromptOpts = {}): string
lines.push(`- Name both the user's PRIOR (default reasoning) AND the COUNTER-PRIOR from their hedged-domain self.`);
lines.push(`- Reference active bias tags by name when relevant ("this fits the over-confident-geography pattern").`);
lines.push(`- Do NOT silently substitute the debiased answer. ALWAYS surface both priors transparently.`);
lines.push(`- Track-record sentences belong in a "Calibration" section in the answer body, after the Conflicts section (if present).`);
lines.push(`- Track-record sentences belong in a "Calibration" section in the answer body, between Conflicts and Gaps.`);
}
return lines.join('\n');
}
+15 -16
View File
@@ -63,26 +63,25 @@ interface PluginCtx {
[key: string]: unknown;
}
export function register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
}
const entry: PluginEntry = {
id: 'gbrain-context-engine',
name: 'GBrain Context Engine',
description: 'Deterministic temporal/spatial context injection on every turn',
register,
register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
},
};
export default entry;
-36
View File
@@ -34,7 +34,6 @@ import {
resetGateway,
embed,
splitByTokenBudget,
capBatchItems,
isTokenLimitError,
__setEmbedTransportForTests,
__getShrinkStateForTests,
@@ -152,41 +151,6 @@ describe('splitByTokenBudget (pure helper)', () => {
});
});
describe('capBatchItems (hard COUNT cap helper)', () => {
test('batch at or under the cap is returned as a single batch (no copy of contents)', () => {
const texts = ['a', 'b', 'c'];
expect(capBatchItems(texts, 3)).toEqual([texts]);
expect(capBatchItems(texts, 10)).toEqual([texts]);
});
test('oversized batch splits into chunks of at most maxItems', () => {
const texts = Array.from({ length: 100 }, (_, i) => `t${i}`);
const result = capBatchItems(texts, 32);
expect(result.map(b => b.length)).toEqual([32, 32, 32, 4]);
expect(result.every(b => b.length <= 32)).toBe(true);
});
test('exact multiple splits evenly with no trailing empty batch', () => {
const texts = Array.from({ length: 64 }, (_, i) => `t${i}`);
expect(capBatchItems(texts, 32).map(b => b.length)).toEqual([32, 32]);
});
test('order is preserved across the split (concatenation round-trips)', () => {
const texts = Array.from({ length: 70 }, (_, i) => `t${i}`);
expect(capBatchItems(texts, 32).flat()).toEqual(texts);
});
test('maxItems <= 0 is a no-op (single batch) — never produces empty/infinite batches', () => {
const texts = ['a', 'b', 'c'];
expect(capBatchItems(texts, 0)).toEqual([texts]);
expect(capBatchItems(texts, -5)).toEqual([texts]);
});
test('empty input returns a single empty batch', () => {
expect(capBatchItems([], 32)).toEqual([[]]);
});
});
describe('isTokenLimitError (pure helper)', () => {
test('matches Voyage error format', () => {
expect(isTokenLimitError(VOYAGE_TOKEN_LIMIT_ERROR)).toBe(true);
@@ -28,8 +28,8 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
resetGateway();
});
test('Ollama, LiteLLM declare no_batch_cap: true', () => {
for (const id of ['ollama', 'litellm']) {
test('Ollama, LiteLLM, llama-server all declare no_batch_cap: true', () => {
for (const id of ['ollama', 'litellm', 'llama-server']) {
const r = getRecipe(id);
expect(r, `${id} not registered`).toBeDefined();
expect(
@@ -39,16 +39,6 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
}
});
test('llama-server declares a hard item-count cap (max_batch_items: 32)', () => {
// llama.cpp enforces a request-COUNT cap equal to its launch --batch-size
// (default 32); declaring max_batch_items both bounds batches AND suppresses
// the missing-max_batch_tokens warning. Replaces the prior no_batch_cap flag.
const r = getRecipe('llama-server');
expect(r, 'llama-server not registered').toBeDefined();
expect(r!.touchpoints.embedding?.max_batch_items).toBe(32);
expect(r!.touchpoints.embedding?.no_batch_cap).toBeUndefined();
});
test('configureGateway does NOT warn for ollama/litellm/llama-server', () => {
warnSpy.mockClear();
resetGateway();
-41
View File
@@ -1,41 +0,0 @@
/**
* Ollama Matryoshka dims passthrough.
*
* Several embedding models served via Ollama (Qwen3-Embedding family) support
* Matryoshka truncation through the `dimensions` field on /v1/embeddings.
* Without this passthrough, gbrain ignores user-selected reduced dims and the
* provider returns its native size, causing dim-mismatch failures against
* brains configured for smaller widths.
*/
import { describe, expect, test } from 'bun:test';
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
describe('dims: ollama Matryoshka models', () => {
test('qwen3-embedding:4b threads dimensions=1536', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:4b', 1536))
.toEqual({ openaiCompatible: { dimensions: 1536 } });
});
test('qwen3-embedding:0.6b threads dimensions=512', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:0.6b', 512))
.toEqual({ openaiCompatible: { dimensions: 512 } });
});
test('qwen3-embedding:8b threads dimensions=2048', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:8b', 2048))
.toEqual({ openaiCompatible: { dimensions: 2048 } });
});
test('bare qwen3-embedding (no quant tag) also recognized', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 1024))
.toEqual({ openaiCompatible: { dimensions: 1024 } });
});
test('unrelated openai-compat model returns undefined (regression guard)', () => {
expect(dimsProviderOptions('openai-compatible', 'nomic-embed-text', 768))
.toBeUndefined();
expect(dimsProviderOptions('openai-compatible', 'mxbai-embed-large', 1024))
.toBeUndefined();
});
});
-26
View File
@@ -99,29 +99,3 @@ describe('autopilot wrapper script — env source order (v0.36.1.x #966)', () =>
expect(src).toMatch(/source\s+~\/\.zshrc/);
});
});
// v0.42.x: the wrapper must export PATH with ~/.bun/bin before exec'ing
// gbrain. The exec'd gbrain has a `#!/usr/bin/env bun` shebang, and the
// standard Debian ~/.bashrc ships a non-interactive guard
// (`case $- in *i*) ;; *) return;; esac`) that exits early when cron/launchd/
// systemd invokes bash non-interactively — so the PATH exports that
// operators put in ~/.bashrc never reach this subprocess. Without the
// explicit export the wrapper silently dies with `env: bun: No such file
// or directory`, leaves a stale lockfile, and blocks every subsequent tick
// for the 10-min stale-lock window. Regression: see Hermes `cron doctor`
// reports — this caused a 1-week nightly-cycle outage on at least one
// operator machine before being diagnosed.
describe('autopilot wrapper script — bun PATH export (v0.42.x regression)', () => {
test('wrapper exports ~/.bun/bin onto PATH before the exec', async () => {
const { readFileSync } = await import('fs');
const src = readFileSync('src/commands/autopilot.ts', 'utf8');
// The export line must appear inside the writeWrapperScript heredoc.
expect(src).toMatch(/export\s+PATH="\$HOME\/\.bun\/bin:\$PATH"/);
// The export must precede the exec line, otherwise env never sees it.
const exportIdx = src.search(/export\s+PATH="\$HOME\/\.bun\/bin/);
const execIdx = src.search(/exec\s+'\${safeGbrainPath}'/);
expect(exportIdx).toBeGreaterThan(0);
expect(execIdx).toBeGreaterThan(0);
expect(exportIdx).toBeLessThan(execIdx);
});
});
-47
View File
@@ -1,47 +0,0 @@
/**
* Structural regression for the backlinks Minion handler default.
*
* Backlinks jobs submitted with an EMPTY payload (the syncembedbacklinks
* chains enqueued after every ingestion) must run as 'check', never 'fix'.
* The pre-fix handler inverted the default (`=== 'check' ? 'check' : 'fix'`),
* so every routine post-ingestion job rewrote tracked brain pages with
* generated "Referenced in" timeline bullets contradicting the documented
* intent in src/core/cycle.ts (runPhaseBacklinks): "Maintenance cycles must
* not rewrite tracked brain pages with generated 'Referenced in' timeline
* bullets."
*
* Source-grep is the right tool here (see fix-wave-structural.test.ts): the
* handler dynamically imports runBacklinksCore and walks a real repo dir, so
* a behavioral test would require heavy mocking that hides the regression
* behind a test seam. The rule is "this specific default must stay 'check'".
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
describe('backlinks Minion handler — empty payload defaults to check, not fix', () => {
const src = readFileSync('src/commands/jobs.ts', 'utf8');
// Isolate the backlinks register block so assertions can't accidentally
// match another handler's action parsing.
const blockMatch = src.match(
/worker\.register\('backlinks',[\s\S]*?runBacklinksCore\(\{[\s\S]*?\}\);/
);
test('the backlinks handler block exists', () => {
expect(blockMatch).not.toBeNull();
});
test("default action is 'check' (explicit opt-in required for 'fix')", () => {
const block = blockMatch![0];
expect(block).toMatch(
/job\.data\.action\s*===\s*'fix'\s*\?\s*'fix'\s*:\s*'check'/
);
});
test('the inverted (fix-by-default) shape stays absent', () => {
const block = blockMatch![0];
expect(block).not.toMatch(
/job\.data\.action\s*===\s*'check'\s*\?\s*'check'\s*:\s*'fix'/
);
});
});
-34
View File
@@ -6,7 +6,6 @@ import {
checkResolvable,
parseResolverEntries,
extractDelegationTargets,
extractTriggers,
} from "../src/core/check-resolvable.ts";
const SKILLS_DIR = join(import.meta.dir, "..", "skills");
@@ -196,39 +195,6 @@ describe("parseResolverEntries", () => {
});
});
describe("extractTriggers", () => {
const LF_FRONTMATTER =
"---\nname: query\ndescription: Test\ntriggers:\n - \"what do we know\"\n - \"tell me about\"\ntools:\n - search\n---\n\n# Body\n";
test("parses triggers from LF-terminated frontmatter", () => {
const triggers = extractTriggers(LF_FRONTMATTER);
expect(triggers).toEqual(["what do we know", "tell me about"]);
});
test("parses triggers from CRLF-terminated frontmatter (Windows checkouts)", () => {
// Regression: `core.autocrlf=true` is the Windows default. Without
// CRLF→LF normalization, every Windows skill is reported as a false
// mece_gap warning because the `^---\n` regex never matches `---\r\n`.
const crlf = LF_FRONTMATTER.replace(/\n/g, "\r\n");
const triggers = extractTriggers(crlf);
expect(triggers).toEqual(["what do we know", "tell me about"]);
});
test("returns [] when frontmatter is missing", () => {
expect(extractTriggers("# Just a body, no frontmatter\n")).toEqual([]);
});
test("returns [] when triggers field is absent from frontmatter", () => {
const fm = "---\nname: query\ndescription: Test\ntools:\n - search\n---\n";
expect(extractTriggers(fm)).toEqual([]);
});
test("strips surrounding quotes from trigger values", () => {
const fm = "---\nname: x\ntriggers:\n - \"double quoted\"\n - 'single quoted'\n - unquoted\n---\n";
expect(extractTriggers(fm)).toEqual(["double quoted", "single quoted", "unquoted"]);
});
});
describe("checkResolvable — real skills directory", () => {
const report = checkResolvable(SKILLS_DIR);
+1 -39
View File
@@ -8,15 +8,13 @@
//
// PGLite-only: in-memory engine, no DATABASE_URL needed.
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
import {
__setRerankTransportForTests,
configureGateway,
getEmbeddingModel,
getMultimodalModel,
rerank,
resetGateway,
} from '../src/core/ai/gateway.ts';
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
@@ -54,16 +52,10 @@ afterAll(async () => {
beforeEach(async () => {
resetGateway();
__setRerankTransportForTests(null);
// Clear any prior config rows so tests are independent. setConfig with
// empty string is treated as undefined by loadConfigWithEngine (per
// dbStr semantics), so this is safe to call between tests.
await engine.setConfig('embedding_multimodal_model', '');
await engine.setConfig('provider_base_urls.llama-server-reranker', '');
});
afterEach(() => {
__setRerankTransportForTests(null);
});
describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing', () => {
@@ -130,34 +122,4 @@ describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getMultimodalModel()).toBeUndefined();
});
test('DB-set provider_base_urls.llama-server-reranker flows to gateway.rerank URL', async () => {
await engine.setConfig('provider_base_urls.llama-server-reranker', 'http://127.0.0.1:8091/v1');
const baseConfig: GBrainConfig = {
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
};
const merged = await loadConfigWithEngine(engine, baseConfig);
configureGateway(buildGatewayConfig(merged!));
let capturedUrl = '';
__setRerankTransportForTests(async (url) => {
capturedUrl = url;
return new Response(JSON.stringify({ results: [{ index: 0, relevance_score: 0.9 }] }), {
status: 200,
headers: { 'content-type': 'application/json' },
});
});
await rerank({
query: 'q',
documents: ['d'],
model: 'llama-server-reranker:qwen3-reranker-4b',
});
expect(capturedUrl).toBe('http://127.0.0.1:8091/v1/rerank');
});
});
-4
View File
@@ -20,10 +20,6 @@ import {
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E doctor --progress-json tests (DATABASE_URL not set)');
}
const CLI = join(import.meta.dir, '..', '..', 'src', 'cli.ts');
describeE2E('gbrain doctor --progress-json (E2E)', () => {
+133
View File
@@ -241,3 +241,136 @@ describe('buildVectorCastFragment — engine SQL composer (D3)', () => {
expect(castSql).toBe('$1::halfvec(2560)');
});
});
describe('PGLite engine: upsertChunks write-side ResolvedColumn descriptor (#1262)', () => {
test('halfvec descriptor writes the text embedding to the alternate column, not legacy embedding', async () => {
await engine.putPage('docs/write-alt-pglite', {
type: 'concept',
title: 'Write alt column PGLite',
compiled_truth: 'PGLite write-side alternate embedding column test.',
});
const descriptor: ResolvedColumn = {
name: 'embedding_ze',
type: 'halfvec',
dimensions: 2560,
embeddingModel: 'zeroentropyai:zembed-1',
};
await engine.upsertChunks('docs/write-alt-pglite', [
{
chunk_index: 0,
chunk_text: 'PGLite write-side alternate embedding column test.',
chunk_source: 'compiled_truth',
embedding: new Float32Array(2560).fill(0.25),
},
], { embeddingColumn: descriptor });
const rows = await engine.executeRaw<{
has_default: boolean;
has_ze: boolean;
has_embedded_at: boolean;
}>(
`SELECT embedding IS NOT NULL AS has_default,
embedding_ze IS NOT NULL AS has_ze,
embedded_at IS NOT NULL AS has_embedded_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = 'docs/write-alt-pglite'`,
);
expect(rows.length).toBe(1);
expect(rows[0].has_default).toBe(false);
expect(rows[0].has_ze).toBe(true);
expect(rows[0].has_embedded_at).toBe(true);
});
test('text-unchanged re-upsert without a vector preserves the alternate-column embedding', async () => {
const descriptor: ResolvedColumn = {
name: 'embedding_ze',
type: 'halfvec',
dimensions: 2560,
embeddingModel: 'zeroentropyai:zembed-1',
};
// Same chunk_text, no embedding: the ON CONFLICT CASE must keep the
// existing alternate-column vector (D24 semantics follow the column).
await engine.upsertChunks('docs/write-alt-pglite', [
{
chunk_index: 0,
chunk_text: 'PGLite write-side alternate embedding column test.',
chunk_source: 'compiled_truth',
},
], { embeddingColumn: descriptor });
const rows = await engine.executeRaw<{ has_ze: boolean }>(
`SELECT embedding_ze IS NOT NULL AS has_ze
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = 'docs/write-alt-pglite'`,
);
expect(rows).toEqual([{ has_ze: true }]);
});
});
describe('PGLite: embed --stale converges on an alt-column brain (#1262)', () => {
test('boundary resolves the write column; stale scan does not re-select embedded rows', async () => {
const { runEmbedCore } = await import('../../src/commands/embed.ts');
const local = new PGLiteEngine();
const previousHome = process.env.GBRAIN_HOME;
process.env.GBRAIN_HOME = `/tmp/gbrain-write-col-stale-${Date.now()}`;
try {
await local.connect({});
await local.initSchema();
await (local as any).db.exec(
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_ze halfvec(2560)`,
);
const descriptor: ResolvedColumn = {
name: 'embedding_ze',
type: 'halfvec',
dimensions: 2560,
embeddingModel: 'zeroentropyai:zembed-1',
};
await local.setConfig('embedding_columns', JSON.stringify({
embedding_ze: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' },
}));
configureGateway({
embedding_model: 'zeroentropyai:zembed-1',
embedding_dimensions: 2560,
env: {},
});
await local.putPage('docs/stale-alt-pglite', {
type: 'concept',
title: 'Dynamic stale column',
compiled_truth: 'A chunk that is embedded only in the dynamic column.',
});
await local.upsertChunks('docs/stale-alt-pglite', [
{
chunk_index: 0,
chunk_text: 'A chunk that is embedded only in the dynamic column.',
chunk_source: 'compiled_truth',
embedding: new Float32Array(2560).fill(0.25),
},
], { embeddingColumn: descriptor });
// Engine-level contrast: legacy predicate still sees the row as stale;
// the alt-column predicate does not.
expect(await local.countStaleChunks()).toBe(1);
expect(await local.countStaleChunks({ embeddingColumn: descriptor })).toBe(0);
// sumStaleChunkChars feeds the sync cost gate — same predicate contract.
expect(await local.sumStaleChunkChars()).toBeGreaterThan(0);
expect(await local.sumStaleChunkChars({ embeddingColumn: descriptor })).toBe(0);
expect(await local.listStaleChunks({ embeddingColumn: descriptor, batchSize: 100 })).toHaveLength(0);
expect(await local.listStaleChunks({ batchSize: 100 })).toHaveLength(1);
// Boundary-level: `embed --stale --dry-run` resolves the write column
// from merged config + gateway and reports NOTHING to embed. Without
// the fix this reports 1 (perpetual re-embed loop).
const result = await runEmbedCore(local, { stale: true, dryRun: true });
expect(result.would_embed).toBe(0);
} finally {
await local.disconnect();
if (previousHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = previousHome;
resetGateway();
}
});
});
@@ -224,4 +224,54 @@ if (!dbUrl) {
await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${v}'::vector WHERE id = ${dogId}`);
});
});
describe('Postgres: upsertChunks write-side ResolvedColumn descriptor (#1262)', () => {
const descriptor: ResolvedColumn = {
name: 'embedding_ze',
type: 'halfvec',
dimensions: 2560,
embeddingModel: 'zeroentropyai:zembed-1',
};
test('halfvec descriptor writes the text embedding to the alternate column, not legacy embedding', async () => {
await engine.putPage('docs/write-alt-postgres', {
type: 'concept',
title: 'Write alt column Postgres',
compiled_truth: 'Postgres write-side alternate embedding column test.',
});
await engine.upsertChunks('docs/write-alt-postgres', [
{
chunk_index: 0,
chunk_text: 'Postgres write-side alternate embedding column test.',
chunk_source: 'compiled_truth',
embedding: new Float32Array(2560).fill(0.25),
},
], { embeddingColumn: descriptor });
const rows = await engine.executeRaw<{
has_default: boolean;
has_ze: boolean;
}>(
`SELECT embedding IS NOT NULL AS has_default,
embedding_ze IS NOT NULL AS has_ze
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = 'docs/write-alt-postgres'`,
);
expect(rows.length).toBe(1);
expect(rows[0].has_default).toBe(false);
expect(rows[0].has_ze).toBe(true);
}, 30_000);
test('stale scan follows the write-side column (count + list parity with the write target)', async () => {
// Legacy predicate: cat/dog/write-alt rows all have embedding NULL.
expect(await engine.countStaleChunks()).toBeGreaterThan(0);
// Alt-column predicate: every chunk has embedding_ze populated.
expect(await engine.countStaleChunks({ embeddingColumn: descriptor })).toBe(0);
expect(await engine.listStaleChunks({ embeddingColumn: descriptor, batchSize: 100 })).toHaveLength(0);
expect((await engine.listStaleChunks({ batchSize: 100 })).length).toBeGreaterThan(0);
// updated_desc arm uses the same predicate.
expect(await engine.listStaleChunks({ embeddingColumn: descriptor, orderBy: 'updated_desc', batchSize: 100 })).toHaveLength(0);
}, 30_000);
});
}
-53
View File
@@ -172,59 +172,6 @@ describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag ON → path-qualified wikilink outside DIR_PATTERN resolves via DB path', async () => {
// `[[notes/struktura]]` — `notes` is not in DIR_PATTERN, so the ref
// reaches the generic pass with its dirname intact. Regression: the DB
// path queried the basename index with the raw literal (which is keyed
// by final segments only), so path-qualified wikilinks outside
// DIR_PATTERN silently produced zero edges while the FS path resolved
// the identical content.
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'Background in [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'notes/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
expect(strk!.link_source).toBe('wikilink-resolved');
});
test('path-qualified wikilink never attaches to a basename-only sibling', async () => {
// Both notes/struktura and wiki/struktura exist. The author wrote
// `[[notes/struktura]]` — the written path must exclude wiki/struktura
// (a bare `[[struktura]]` would legitimately match both).
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('wiki/struktura', {
type: 'concept' as any, title: 'Struktura Wiki',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/x');
const basenameLinks = outLinks
.filter(l => l.link_type === 'wikilink_basename')
.map(l => l.to_slug);
expect(basenameLinks).toEqual(['notes/struktura']);
});
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
+5 -71
View File
@@ -29,15 +29,9 @@ afterAll(async () => {
});
async function truncateAll() {
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'config', 'pages']) {
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) {
await (engine as any).db.exec(`DELETE FROM ${t}`);
}
// Re-seed the two config keys this file touches back to their documented
// defaults (both default to ON). This makes every test deterministic even if
// an earlier test threw before its finally restored auto_link/auto_timeline,
// and even though absent-key already resolves truthy via isAuto*Enabled.
await engine.setConfig('auto_link', 'true');
await engine.setConfig('auto_timeline', 'true');
}
function makeContext(): OperationContext {
@@ -83,12 +77,10 @@ describe('E2E graph quality (v0.10.1 pipeline)', () => {
await runExtract(engine, ['links', '--source', 'db']);
await runExtract(engine, ['timeline', '--source', 'db']);
// Verify graph populated. Concrete floors derived from the seeded fixtures:
// resolvable entity refs: alice->acme, bob->acme, standup->alice, standup->bob = 4
// timeline lines: alice(2) + bob(1) + acme(1) + standup(1) = 5
// Verify graph populated.
const stats = await engine.getStats();
expect(stats.link_count).toBeGreaterThanOrEqual(4);
expect(stats.timeline_entry_count).toBeGreaterThanOrEqual(5);
expect(stats.link_count).toBeGreaterThan(0);
expect(stats.timeline_entry_count).toBeGreaterThan(0);
// Verify typed link inference.
const aliceLinks = await engine.getLinks('people/alice');
@@ -99,16 +91,7 @@ describe('E2E graph quality (v0.10.1 pipeline)', () => {
const bobAcme = bobLinks.find(l => l.to_slug === 'companies/acme');
expect(bobAcme?.link_type).toBe('invested_in');
// The standup meeting references both Alice and Bob as attendees. Assert the
// exact attendee edges are present and typed 'attended' (a plain .every()
// would silently pass if a meeting->company edge were misclassified or if the
// attendee edges were missing entirely).
const meetingLinks = await engine.getLinks('meetings/standup');
const attended = new Set(
meetingLinks.filter(l => l.link_type === 'attended').map(l => l.to_slug),
);
expect(attended.has('people/alice')).toBe(true);
expect(attended.has('people/bob')).toBe(true);
expect(meetingLinks.every(l => l.link_type === 'attended')).toBe(true);
});
@@ -135,9 +118,7 @@ Attendees: [Alice](people/alice). Discussed [Acme](companies/acme).
// The response should include auto_links results.
expect((result as any).auto_links).toBeDefined();
const autoLinks = (result as any).auto_links;
// The page references exactly two seeded, resolvable targets (Alice + Acme),
// so exactly two links are created.
expect(autoLinks.created).toBe(2);
expect(autoLinks.created).toBeGreaterThan(0);
expect(autoLinks.errors).toBe(0);
// Verify links actually exist in DB.
@@ -302,53 +283,6 @@ Mention of [Alice](people/alice).
expect(paths[0].link_type).toBe('works_at');
});
test('graph-query traversal: direction out and both, plus depth:2 multi-hop', async () => {
// Seed a 2-hop chain: alice -works_at-> acme -partnered_with-> beta.
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('companies/acme', { type: 'company', title: 'Acme', compiled_truth: '', timeline: '' });
await engine.putPage('companies/beta', { type: 'company', title: 'Beta', compiled_truth: '', timeline: '' });
await engine.addLink('people/alice', 'companies/acme', '', 'works_at');
await engine.addLink('companies/acme', 'companies/beta', '', 'partnered_with');
// direction:'out' from alice, depth 1 -> only the first hop.
const out1 = await engine.traversePaths('people/alice', { direction: 'out', depth: 1 });
expect(out1.length).toBe(1);
expect(out1[0].from_slug).toBe('people/alice');
expect(out1[0].to_slug).toBe('companies/acme');
expect(out1[0].depth).toBe(1);
// depth:2 -> both hops, depths 1 and 2.
const out2 = await engine.traversePaths('people/alice', { direction: 'out', depth: 2 });
const out2Edges = new Set(out2.map(p => `${p.from_slug}->${p.to_slug}@${p.depth}`));
expect(out2Edges.has('people/alice->companies/acme@1')).toBe(true);
expect(out2Edges.has('companies/acme->companies/beta@2')).toBe(true);
expect(out2.length).toBe(2);
// direction:'both' from acme depth 1 -> sees the inbound edge from alice AND
// the outbound edge to beta. Edges keep their natural from->to orientation.
const both = await engine.traversePaths('companies/acme', { direction: 'both', depth: 1 });
const bothEdges = new Set(both.map(p => `${p.from_slug}->${p.to_slug}`));
expect(bothEdges.has('people/alice->companies/acme')).toBe(true);
expect(bothEdges.has('companies/acme->companies/beta')).toBe(true);
});
test('graph-query cycle safety: A->B->A terminates and returns bounded results', async () => {
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: '', timeline: '' });
// Create a 2-cycle: alice -> bob -> alice.
await engine.addLink('people/alice', 'people/bob', '', 'knows');
await engine.addLink('people/bob', 'people/alice', '', 'knows');
// High depth must NOT loop forever; the visited-set guard bounds the walk.
const paths = await engine.traversePaths('people/alice', { direction: 'out', depth: 100 });
const edges = new Set(paths.map(p => `${p.from_slug}->${p.to_slug}`));
// Both edges of the cycle are reachable exactly once.
expect(edges.has('people/alice->people/bob')).toBe(true);
expect(edges.has('people/bob->people/alice')).toBe(true);
// Bounded: there are only two edges in the graph, so no path explosion.
expect(paths.length).toBe(2);
});
test('search backlink boost: well-connected pages rank higher', async () => {
// Create 3 pages all matching a search term, but with different inbound link counts.
await engine.putPage('topic/popular', {
-4
View File
@@ -21,10 +21,6 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E JSONB roundtrip tests (DATABASE_URL not set)');
}
describeE2E('E2E: JSONB roundtrip — v0.12.1 reliability wave', () => {
beforeAll(async () => { await setupDB(); });
afterAll(async () => { await teardownDB(); });
-1
View File
@@ -56,7 +56,6 @@ describe('E2E: MCP Tool Generation', () => {
expect(names).toContain('get_health');
expect(names).toContain('sync_brain');
expect(names).toContain('file_upload');
expect(names).toContain('find_orphans');
});
test('MCP server module can be imported', async () => {
+7 -64
View File
@@ -175,15 +175,6 @@ describeE2E('E2E: Search', () => {
for (const [query, score] of Object.entries(scores)) {
console.log(` "${query}": ${(score * 100).toFixed(0)}%`);
}
// Guard value: every known-item query must surface at least one ground-truth
// doc in the top 5. This is a deliberately loose floor (not a tuned P@5
// threshold) — it catches a total keyword-retrieval regression without
// breaking on every scoring/fixture tweak. Without it this test asserted
// nothing and a 0%-precision result passed silently.
for (const [query, score] of Object.entries(scores)) {
expect(score).toBeGreaterThan(0);
}
});
});
@@ -214,22 +205,10 @@ describeE2E('E2E: Links', () => {
}, 30_000);
test('traverse_graph finds connected pages', async () => {
// Self-contained: do not depend on a prior test's add_link. add_link is
// idempotent (ON CONFLICT DO NOTHING), so re-adding here is safe whether or
// not the round-trip test ran first, and the test no longer false-passes or
// false-fails based on describe-block ordering.
await callOp('add_link', {
from: 'people/sarah-chen',
to: 'companies/novamind',
link_type: 'founded',
});
const graph = await callOp('traverse_graph', { slug: 'people/sarah-chen', depth: 2 }) as any[];
// Links should already be added from prior test in this describe block
const graph = await callOp('traverse_graph', { slug: 'people/sarah-chen', depth: 2 }) as any;
expect(Array.isArray(graph)).toBe(true);
expect(graph.length).toBeGreaterThanOrEqual(1);
// Content assertion, not just shape: the linked company must be reachable.
const reachable = graph.map((n: any) => n.slug ?? n.to_slug ?? n.to_page_slug);
expect(reachable).toContain('companies/novamind');
});
test('remove_link removes the link', async () => {
@@ -490,14 +469,8 @@ describeE2E('E2E: Admin', () => {
test('get_health returns valid structure', async () => {
const health = await callOp('get_health') as any;
expect(health).toBeDefined();
// Value bounds, not just types: page_count must match the fixture inventory
// and embed_coverage is a 0..1 fraction (src/commands/doctor.ts multiplies
// by 100 and compares to 0.9). Type-only checks let embed_coverage: -9999
// through; these catch a genuinely broken health payload.
expect(health.page_count).toBe(16);
expect(Number.isFinite(health.embed_coverage)).toBe(true);
expect(health.embed_coverage).toBeGreaterThanOrEqual(0);
expect(health.embed_coverage).toBeLessThanOrEqual(1);
expect(typeof health.page_count).toBe('number');
expect(typeof health.embed_coverage).toBe('number');
});
});
@@ -515,17 +488,7 @@ describeE2E('E2E: Chunks & Resolution', () => {
test('get_chunks returns chunks for imported page', async () => {
const chunks = await callOp('get_chunks', { slug: 'people/sarah-chen' }) as any[];
expect(chunks.length).toBeGreaterThan(0);
// Content + ordering, not just truthiness (a whitespace-only chunk is truthy):
// every chunk has real text and a numeric index, the indexes are
// non-decreasing in return order, and the page's own name appears somewhere.
for (const c of chunks) {
expect(typeof c.chunk_text).toBe('string');
expect(c.chunk_text.trim().length).toBeGreaterThan(0);
expect(typeof c.chunk_index).toBe('number');
}
const indexes = chunks.map((c: any) => c.chunk_index);
expect(indexes).toEqual([...indexes].sort((x, y) => x - y));
expect(chunks.some((c: any) => c.chunk_text.includes('Sarah'))).toBe(true);
expect(chunks[0].chunk_text).toBeTruthy();
}, 30_000);
test('resolve_slugs finds partial match', async () => {
@@ -699,29 +662,9 @@ describeE2E('E2E: file_list LIMIT enforcement', () => {
}, 30_000);
test('file_list without slug also respects LIMIT 100', async () => {
// Self-sufficient: seed our own >100 rows rather than relying on the
// previous test's 150 rows surviving in the DB. A bun reorder, a focused
// `-t` run, or a failure mid-insert in the prior test would otherwise leave
// this asserting against an indeterminate row count.
const sql = getConn();
const seedSlug = 'test-limit-noslug';
await sql`
INSERT INTO pages (slug, title, type, compiled_truth, frontmatter)
VALUES (${seedSlug}, ${'Test Limit NoSlug'}, ${'note'}, ${'body'}, ${'{}'}::jsonb)
ON CONFLICT (source_id, slug) DO NOTHING
`;
for (let i = 0; i < 120; i++) {
await sql`
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
VALUES (${seedSlug}, ${'nf-' + String(i).padStart(3, '0') + '.txt'}, ${seedSlug + '/nf-' + i + '.txt'}, ${'text/plain'}, ${100}, ${'nhash-' + i}, ${'{}'}::jsonb)
ON CONFLICT (storage_path) DO NOTHING
`;
}
const total = await sql`SELECT count(*)::int AS n FROM files`;
expect(Number(total[0].n)).toBeGreaterThan(100); // cap is actually exercised
// The 150 rows from the previous test are still in the DB
const files = await callOp('file_list', {}) as any[];
expect(files.length).toBe(100);
expect(files.length).toBeLessThanOrEqual(100);
});
});
+111 -149
View File
@@ -78,19 +78,6 @@ function freshTempHome(label: string) {
return dir;
}
// Restore HOME/PATH to the captured originals. Called from each test's
// `finally` so a throw mid-test can never leave HOME/PATH pointed at a temp
// dir for the rest of the bun process (which would silently break unrelated
// suites that read HOME). PATH keeps the shim prepended because the
// module-level shim install is what subsequent tests in this suite rely on;
// afterAll does the final teardown to the pristine origPath.
function restoreHomePath() {
if (origHome === undefined) delete process.env.HOME;
else process.env.HOME = origHome;
if (origPath === undefined) delete process.env.PATH;
else process.env.PATH = `${fakeBinDir}:${origPath ?? ''}`;
}
beforeAll(() => {
if (SKIP) {
console.log('[migration-flow.e2e] DATABASE_URL not set — skipping.');
@@ -113,15 +100,6 @@ afterAll(() => {
beforeEach(() => {
if (SKIP) return;
// Robust restore: if a prior test threw before its own finally ran (or
// before afterAll), HOME/PATH could still point at a dead temp dir. Reset
// them to the captured originals at the start of every test so a throw in
// one test can never leak a temp HOME/PATH into sibling suites that read
// them. freshTempHome() re-points HOME per test immediately after this.
if (origHome === undefined) delete process.env.HOME;
else process.env.HOME = origHome;
if (origPath === undefined) delete process.env.PATH;
else process.env.PATH = `${fakeBinDir}:${origPath ?? ''}`;
try { if (tmp) rmSync(tmp, { recursive: true, force: true }); } catch { /* best-effort */ }
});
@@ -136,160 +114,144 @@ const COMMON_OPTS = {
describeE2E('E2E: v0.11.0 orchestrator against live Postgres', () => {
test('fresh install flow: schema → smoke → prefs → host-rewrite → completed', async () => {
tmp = freshTempHome('fresh');
try {
const result = await v0_11_0.orchestrator(COMMON_OPTS);
const result = await v0_11_0.orchestrator(COMMON_OPTS);
// Orchestrator returns a structured result (status is `complete` when
// no pending-host-work TODOs fired, `partial` otherwise).
expect(result.version).toBe('0.11.0');
expect(['complete', 'partial']).toContain(result.status);
// Orchestrator returns a structured result (status is `complete` when
// no pending-host-work TODOs fired, `partial` otherwise).
expect(result.version).toBe('0.11.0');
expect(['complete', 'partial']).toContain(result.status);
// Phase D: preferences.json exists with 0o600 + mode=pain_triggered.
const prefsPath = join(tmp, '.gbrain', 'preferences.json');
expect(existsSync(prefsPath)).toBe(true);
expect(statSync(prefsPath).mode & 0o777).toBe(0o600);
const prefs = loadPreferences();
expect(prefs.minion_mode).toBe('pain_triggered');
expect(prefs.set_at).toBeTruthy();
expect(prefs.set_in_version).toBeTruthy();
// Phase D: preferences.json exists with 0o600 + mode=pain_triggered.
const prefsPath = join(tmp, '.gbrain', 'preferences.json');
expect(existsSync(prefsPath)).toBe(true);
expect(statSync(prefsPath).mode & 0o777).toBe(0o600);
const prefs = loadPreferences();
expect(prefs.minion_mode).toBe('pain_triggered');
expect(prefs.set_at).toBeTruthy();
expect(prefs.set_in_version).toBeTruthy();
// Bug 3 (v0.14.2) — orchestrator no longer writes completed.jsonl.
// The runner (apply-migrations.ts) persists the result after the
// orchestrator returns. A direct orchestrator call in E2E leaves the
// ledger empty; the runner path is tested separately in
// test/apply-migrations.test.ts + test/migration-resume.test.ts.
const completed = loadCompletedMigrations();
const v0110Entries = completed.filter(e => e.version === '0.11.0');
expect(v0110Entries.length).toBe(0);
// Bug 3 (v0.14.2) — orchestrator no longer writes completed.jsonl.
// The runner (apply-migrations.ts) persists the result after the
// orchestrator returns. A direct orchestrator call in E2E leaves the
// ledger empty; the runner path is tested separately in
// test/apply-migrations.test.ts + test/migration-resume.test.ts.
const completed = loadCompletedMigrations();
const v0110Entries = completed.filter(e => e.version === '0.11.0');
expect(v0110Entries.length).toBe(0);
// Phase F is skipped per COMMON_OPTS — autopilot should NOT have been
// installed on this host.
expect(result.autopilot_installed).toBe(false);
} finally {
restoreHomePath();
}
// Phase F is skipped per COMMON_OPTS — autopilot should NOT have been
// installed on this host.
expect(result.autopilot_installed).toBe(false);
}, 60_000);
test('idempotent rerun: second invocation is a safe no-op', async () => {
tmp = freshTempHome('rerun');
try {
const first = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(first.status);
const first = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(first.status);
const second = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(second.status);
const second = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(second.status);
// Bug 3 (v0.14.2) — orchestrator does not write completed.jsonl, so
// repeated direct invocations don't accumulate ledger entries. Assert
// the preferences state stays stable (the real idempotency signal for
// this orchestrator is "running again doesn't corrupt preferences").
expect(loadPreferences().minion_mode).toBe('pain_triggered');
const completed = loadCompletedMigrations();
expect(completed.filter(e => e.version === '0.11.0').length).toBe(0);
} finally {
restoreHomePath();
}
// Bug 3 (v0.14.2) — orchestrator does not write completed.jsonl, so
// repeated direct invocations don't accumulate ledger entries. Assert
// the preferences state stays stable (the real idempotency signal for
// this orchestrator is "running again doesn't corrupt preferences").
expect(loadPreferences().minion_mode).toBe('pain_triggered');
const completed = loadCompletedMigrations();
expect(completed.filter(e => e.version === '0.11.0').length).toBe(0);
}, 90_000);
test('host rewrite: builtin handlers auto-rewritten, non-builtins queued as JSONL TODOs', async () => {
tmp = freshTempHome('host-rewrite');
try {
// Fixture: AGENTS.md + cron/jobs.json with a mix of gbrain-builtin and
// non-builtin handlers.
const claudeDir = join(tmp, '.claude');
mkdirSync(claudeDir, { recursive: true });
writeFileSync(
join(claudeDir, 'AGENTS.md'),
'# Test AGENTS.md\n\nSome existing content referencing sessions_spawn routing.\n',
);
mkdirSync(join(claudeDir, 'cron'), { recursive: true });
writeFileSync(
join(claudeDir, 'cron', 'jobs.json'),
JSON.stringify({
jobs: [
{ schedule: '*/5 * * * *', kind: 'agentTurn', skill: 'sync' }, // builtin
{ schedule: '0 */30 * * *', kind: 'agentTurn', skill: 'ea-inbox-sweep' }, // non-builtin
{ schedule: '*/10 * * * *', kind: 'agentTurn', skill: 'embed' }, // builtin
{ schedule: '0 8 * * *', kind: 'agentTurn', skill: 'morning-briefing' }, // non-builtin
],
}, null, 2) + '\n',
);
// Fixture: AGENTS.md + cron/jobs.json with a mix of gbrain-builtin and
// non-builtin handlers.
const claudeDir = join(tmp, '.claude');
mkdirSync(claudeDir, { recursive: true });
writeFileSync(
join(claudeDir, 'AGENTS.md'),
'# Test AGENTS.md\n\nSome existing content referencing sessions_spawn routing.\n',
);
mkdirSync(join(claudeDir, 'cron'), { recursive: true });
writeFileSync(
join(claudeDir, 'cron', 'jobs.json'),
JSON.stringify({
jobs: [
{ schedule: '*/5 * * * *', kind: 'agentTurn', skill: 'sync' }, // builtin
{ schedule: '0 */30 * * *', kind: 'agentTurn', skill: 'ea-inbox-sweep' }, // non-builtin
{ schedule: '*/10 * * * *', kind: 'agentTurn', skill: 'embed' }, // builtin
{ schedule: '0 8 * * *', kind: 'agentTurn', skill: 'morning-briefing' }, // non-builtin
],
}, null, 2) + '\n',
);
const result = await v0_11_0.orchestrator(COMMON_OPTS);
const result = await v0_11_0.orchestrator(COMMON_OPTS);
// Builtins rewritten in place; non-builtins left alone.
const cronAfter = JSON.parse(readFileSync(join(claudeDir, 'cron', 'jobs.json'), 'utf-8'));
expect(cronAfter.jobs[0].kind).toBe('shell'); // sync (builtin)
expect(cronAfter.jobs[0].cmd).toContain('gbrain jobs submit sync');
expect(cronAfter.jobs[1].kind).toBe('agentTurn'); // ea-inbox-sweep (non-builtin)
expect(cronAfter.jobs[2].kind).toBe('shell'); // embed (builtin)
expect(cronAfter.jobs[3].kind).toBe('agentTurn'); // morning-briefing (non-builtin)
// Builtins rewritten in place; non-builtins left alone.
const cronAfter = JSON.parse(readFileSync(join(claudeDir, 'cron', 'jobs.json'), 'utf-8'));
expect(cronAfter.jobs[0].kind).toBe('shell'); // sync (builtin)
expect(cronAfter.jobs[0].cmd).toContain('gbrain jobs submit sync');
expect(cronAfter.jobs[1].kind).toBe('agentTurn'); // ea-inbox-sweep (non-builtin)
expect(cronAfter.jobs[2].kind).toBe('shell'); // embed (builtin)
expect(cronAfter.jobs[3].kind).toBe('agentTurn'); // morning-briefing (non-builtin)
// files_rewritten counts the 2 builtin rewrites.
expect(result.files_rewritten).toBeGreaterThanOrEqual(2);
// files_rewritten counts the 2 builtin rewrites.
expect(result.files_rewritten).toBeGreaterThanOrEqual(2);
// pending_host_work counts the 2 non-builtin TODOs.
expect(result.pending_host_work).toBe(2);
// pending_host_work counts the 2 non-builtin TODOs.
expect(result.pending_host_work).toBe(2);
// Status is "partial" because non-builtin TODOs remain.
expect(result.status).toBe('partial');
// Status is "partial" because non-builtin TODOs remain.
expect(result.status).toBe('partial');
// AGENTS.md got the marker injected.
const agentsMdAfter = readFileSync(join(claudeDir, 'AGENTS.md'), 'utf-8');
expect(agentsMdAfter).toContain('gbrain:subagent-routing v0.11.0');
expect(agentsMdAfter).toContain('skills/conventions/subagent-routing.md');
// AGENTS.md got the marker injected.
const agentsMdAfter = readFileSync(join(claudeDir, 'AGENTS.md'), 'utf-8');
expect(agentsMdAfter).toContain('gbrain:subagent-routing v0.11.0');
expect(agentsMdAfter).toContain('skills/conventions/subagent-routing.md');
// JSONL TODO file written under ~/.gbrain/migrations/.
const jsonlPath = join(tmp, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(jsonlPath)).toBe(true);
const lines = readFileSync(jsonlPath, 'utf-8').split('\n').filter(l => l.trim());
expect(lines.length).toBe(2);
const todos = lines.map(l => JSON.parse(l));
const handlers = todos.map(t => t.handler).sort();
expect(handlers).toEqual(['ea-inbox-sweep', 'morning-briefing']);
for (const todo of todos) {
expect(todo.type).toBe('cron-handler-needs-host-registration');
expect(todo.status).toBe('pending');
expect(todo.manifest_path).toContain('cron/jobs.json');
}
} finally {
restoreHomePath();
// JSONL TODO file written under ~/.gbrain/migrations/.
const jsonlPath = join(tmp, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(jsonlPath)).toBe(true);
const lines = readFileSync(jsonlPath, 'utf-8').split('\n').filter(l => l.trim());
expect(lines.length).toBe(2);
const todos = lines.map(l => JSON.parse(l));
const handlers = todos.map(t => t.handler).sort();
expect(handlers).toEqual(['ea-inbox-sweep', 'morning-briefing']);
for (const todo of todos) {
expect(todo.type).toBe('cron-handler-needs-host-registration');
expect(todo.status).toBe('pending');
expect(todo.manifest_path).toContain('cron/jobs.json');
}
}, 90_000);
test('resumable: partial run → orchestrator re-run → complete', async () => {
tmp = freshTempHome('resumable');
try {
// Simulate a stopgap-written partial entry BEFORE running the orchestrator.
mkdirSync(join(tmp, '.gbrain', 'migrations'), { recursive: true });
writeFileSync(
join(tmp, '.gbrain', 'migrations', 'completed.jsonl'),
JSON.stringify({
version: '0.11.0',
status: 'partial',
apply_migrations_pending: true,
mode: 'pain_triggered',
source: 'fix-v0.11.0.sh',
ts: new Date().toISOString(),
}) + '\n',
);
// Simulate a stopgap-written partial entry BEFORE running the orchestrator.
mkdirSync(join(tmp, '.gbrain', 'migrations'), { recursive: true });
writeFileSync(
join(tmp, '.gbrain', 'migrations', 'completed.jsonl'),
JSON.stringify({
version: '0.11.0',
status: 'partial',
apply_migrations_pending: true,
mode: 'pain_triggered',
source: 'fix-v0.11.0.sh',
ts: new Date().toISOString(),
}) + '\n',
);
// Orchestrator re-running on a partial → should succeed (schema apply
// and smoke are idempotent; prefs are preserved from the partial
// record; host-rewrite runs its safe-skip pass). Per Bug 3 (v0.14.2),
// the orchestrator itself doesn't append to completed.jsonl — the
// runner does. The stopgap's partial entry stays unchanged here.
const result = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(result.status);
// Orchestrator re-running on a partial → should succeed (schema apply
// and smoke are idempotent; prefs are preserved from the partial
// record; host-rewrite runs its safe-skip pass). Per Bug 3 (v0.14.2),
// the orchestrator itself doesn't append to completed.jsonl — the
// runner does. The stopgap's partial entry stays unchanged here.
const result = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(result.status);
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
// Just the stopgap partial — orchestrator doesn't add its own entry.
expect(v0110.length).toBe(1);
expect(v0110[0].status).toBe('partial');
expect(v0110[0].source).toBe('fix-v0.11.0.sh');
} finally {
restoreHomePath();
}
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
// Just the stopgap partial — orchestrator doesn't add its own entry.
expect(v0110.length).toBe(1);
expect(v0110[0].status).toBe('partial');
expect(v0110[0].source).toBe('fix-v0.11.0.sh');
}, 90_000);
});
+4 -14
View File
@@ -94,7 +94,7 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
}, 30_000);
// --- 2. Runaway handler: ignores AbortSignal, dead-lettered by handleTimeouts ---
test('runaway handler: ignores AbortSignal, handleTimeouts dead-letters it', async () => {
test('runaway handler: ignores AbortSignal, handleTimeouts dead-letters in <2s', async () => {
const { a, b } = await makeEngines();
try {
const queue = new MinionQueue(a);
@@ -133,14 +133,8 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
worker.stop();
await startP;
// Correctness gate: the job MUST be dead-lettered with the timeout reason.
// We intentionally do NOT assert a wall-clock upper bound (deadAt - started):
// on a loaded CI runner the stall/timeout sweep cadence varies, and the only
// thing that matters is that the runaway job terminates as 'dead'. The 3s poll
// deadline above is the real timeout — if the sweep is too slow, finalStatus
// stays '' and this toBe('dead') fails loudly.
expect(finalStatus).toBe('dead');
void deadAt; // retained for debugging; no timing assertion (flake-prone)
expect(deadAt - started).toBeLessThan(2000);
const final = await queue.getJob(job.id);
expect(final?.error_text).toMatch(/timeout exceeded/i);
@@ -310,7 +304,7 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
}, 60_000);
// --- 5. Cascade kill under load: cancelJob aborts all live descendants ---
test('cascade kill: cancelJob on parent aborts 10 live children', async () => {
test('cascade kill: cancelJob on parent aborts 10 live children within 2s', async () => {
const { a, b } = await makeEngines();
try {
const queue = new MinionQueue(a);
@@ -380,12 +374,8 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
worker.stop();
await startP;
// Correctness gate: all 10 cooperative handlers observed the abort and the
// DB shows every descendant + root cancelled. We do NOT assert a wall-clock
// upper bound on cancelElapsed — the 3s abort poll deadline above already
// bounds the wait, and asserting a tighter time flakes on shared runners.
expect(abortedChildren.size).toBe(10);
void cancelElapsed; // retained for debugging; no timing assertion (flake-prone)
expect(cancelElapsed).toBeLessThan(3000);
// DB truth: every descendant + root is 'cancelled'
const conn = getConn();
+3 -23
View File
@@ -78,11 +78,7 @@ describeE2E('v0.18.0 multi-source — Postgres schema shape (fresh install)', ()
);
expect(rows.length).toBe(1);
expect(rows[0].is_nullable).toBe('NO');
// Postgres renders a TEXT DEFAULT 'default' literal as `'default'::text`.
// Assert the exact stored expression rather than a loose substring so a
// drift in the schema DEFAULT (e.g. a different sentinel source id) fails
// here instead of silently passing.
expect(String(rows[0].column_default)).toBe("'default'::text");
expect(String(rows[0].column_default)).toContain('default');
});
test('composite UNIQUE pages(source_id, slug) replaces global UNIQUE(slug)', async () => {
@@ -296,18 +292,6 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
VALUES ('cascadetest', ${aliceId}, 'alice.pdf', 'cascadetest/people/alice/alice.pdf', 'fh1')`,
);
const aliceFile = await conn.unsafe(
`SELECT id FROM files WHERE source_id = 'cascadetest' AND storage_path = 'cascadetest/people/alice/alice.pdf'`,
);
const aliceFileId = aliceFile[0].id as number;
// file_migration_ledger row keyed on the file (FK file_id ON DELETE
// CASCADE). Removing the source cascades sources → files → ledger.
await conn.unsafe(
`INSERT INTO file_migration_ledger (file_id, storage_path_old, storage_path_new, status)
VALUES (${aliceFileId}, 'cascadetest/people/alice/alice.pdf', 'cascadetest/people/alice/alice.pdf', 'pending')
ON CONFLICT (file_id) DO NOTHING`,
);
// Sanity: everything exists
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(2);
@@ -315,7 +299,6 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM file_migration_ledger WHERE file_id = ${aliceFileId}`))[0].n).toBe(1);
// Remove the source.
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
@@ -327,7 +310,6 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM file_migration_ledger WHERE file_id = ${aliceFileId}`))[0].n).toBe(0);
// The sources row itself is gone.
const src = await conn.unsafe(`SELECT id FROM sources WHERE id = 'cascadetest'`);
@@ -396,10 +378,8 @@ describeE2E('v0.18.0 multi-source — sync --source routes through sources table
test('performSync with no sourceId falls back to global sync.repo_path', async () => {
const engine = getEngine();
// Self-contained: set the global config this test depends on directly
// instead of inheriting the side effect of the previous test. Without
// --source, performSync must read this global key.
await engine.setConfig('sync.repo_path', '/some/other/default/path');
// Global config is still '/some/other/default/path' from the
// previous test. Without --source, performSync uses it.
let err: Error | null = null;
try {
await performSync(engine, {});
-23
View File
@@ -112,27 +112,4 @@ describe('v0.29 E2E — getRecentSalience (Garry test)', () => {
const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'nope/does-not-exist/' });
expect(rows).toEqual([]);
});
// TIM-37: the daily briefing writes to the vault and re-ingests as
// `briefings/<date>`. Without this filter the briefing itself would top
// every subsequent Brain Pulse — self-reference with no signal.
describe('TIM-37 — briefings excluded from their own Brain Pulse', () => {
test('default query hides briefings/* slugs', async () => {
await engine.putPage('briefings/2026-05-19', {
type: 'note',
title: 'Daily Briefing — 2026-05-19',
compiled_truth: 'Auto-generated cron briefing.',
});
const rows = await engine.getRecentSalience({ days: 7, limit: 50 });
expect(rows.some(r => r.slug.startsWith('briefings/'))).toBe(false);
});
test('explicit slugPrefix=briefings/ still returns them', async () => {
const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'briefings/' });
expect(rows.length).toBeGreaterThan(0);
for (const r of rows) {
expect(r.slug.startsWith('briefings/')).toBe(true);
}
});
});
});
+2 -15
View File
@@ -128,17 +128,6 @@ describe('SearchResult fields', () => {
expect(r.chunk_index).toBeDefined();
expect(typeof r.chunk_index).toBe('number');
});
test('empty keyword query returns a defined array without throwing', async () => {
const results = await engine.searchKeyword('');
expect(Array.isArray(results)).toBe(true);
});
test('zero vector search returns a defined array without throwing', async () => {
const zeroVector = new Float32Array(1536);
const results = await engine.searchVector(zeroVector);
expect(Array.isArray(results)).toBe(true);
});
});
describe('detail parameter', () => {
@@ -156,11 +145,9 @@ describe('detail parameter', () => {
});
test('detail=low on vector search filters to compiled_truth', async () => {
// Use a timeline-direction embedding — detail=low filters to compiled_truth.
// Vector search returns every chunk with an embedding (ordered by distance),
// so the seeded compiled_truth chunks are non-empty and ALL compiled_truth.
// Use a timeline-direction embedding — with detail=low, should get no results
// or only compiled_truth results
const results = await engine.searchVector(basisEmbedding(1), { detail: 'low' });
expect(results.length).toBeGreaterThan(0);
for (const r of results) {
expect(r.chunk_source).toBe('compiled_truth');
}
+4 -4
View File
@@ -39,7 +39,6 @@ import { runSkillOpt } from '../../src/core/skillopt/orchestrator.ts';
import {
bestPath,
loadHistory,
proposedPath,
skillPath,
} from '../../src/core/skillopt/version-store.ts';
import { loadRejectedBuffer } from '../../src/core/skillopt/rejected-buffer.ts';
@@ -742,7 +741,7 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
} finally { fixture.cleanup(); }
});
test('--no-mutate writes proposed.md and best.md, leaves SKILL.md untouched', async () => {
test('--no-mutate writes proposed.md (best.md), leaves SKILL.md untouched', async () => {
const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
try {
installStub({
@@ -754,9 +753,10 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
const result = await runOnce(fixture, { noMutate: true });
expect(result.outcome).toBe('accepted');
expect(result.mutatedSkillFile).toBe(false);
expect(result.proposedPath).toBe(proposedPath(fixture.skillsDir, SKILL));
expect(result.proposedPath).toBeDefined();
// proposed.md (best.md) exists and carries the improvement.
expect(fs.existsSync(result.proposedPath!)).toBe(true);
expect(fs.readFileSync(result.proposedPath!, 'utf8')).toContain('## Citations');
expect(fs.readFileSync(bestPath(fixture.skillsDir, SKILL), 'utf8')).toContain('## Citations');
// SKILL.md on disk is UNCHANGED (still People-only).
const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
expect(skill).not.toContain('## Citations');
+3 -11
View File
@@ -73,7 +73,7 @@ describeE2E('E2E: Check-Update', () => {
expect(stdout).toContain('--json');
});
test('check-update --json contract holds regardless of real release state', async () => {
test('handles no-releases gracefully (current repo state)', async () => {
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'check-update', '--json'], {
cwd: new URL('../..', import.meta.url).pathname,
stdout: 'pipe',
@@ -84,16 +84,8 @@ describeE2E('E2E: Check-Update', () => {
expect(exitCode).toBe(0);
const output = JSON.parse(stdout);
// Don't pin update_available to a literal value — the repo may or may not
// have a published release. Assert the JSON shape instead.
expect(typeof output.update_available).toBe('boolean');
expect(output.current_version).toBe(VERSION);
if (output.latest_version != null) {
expect(typeof output.latest_version).toBe('string');
}
if (output.release_url != null) {
expect(typeof output.release_url).toBe('string');
}
// With no releases, should return false and an error
expect(output.update_available).toBe(false);
});
test('version comparison wiring works end-to-end', () => {
-104
View File
@@ -803,107 +803,3 @@ describe('embedAllStale --source threading (D7)', () => {
expect((firstCallOpts as { sourceId?: string }).sourceId).toBe('media-corpus');
});
});
// ────────────────────────────────────────────────────────────────
// Code metadata preservation across re-embed (regression for #769)
// ────────────────────────────────────────────────────────────────
//
// gbrain v0.30.1 and earlier silently clobbered code-chunk metadata
// (language, symbol_name, symbol_type, start_line, end_line,
// parent_symbol_path, doc_comment, symbol_name_qualified) on every
// re-embed pass. The chunker populated those columns at import time,
// but embed.ts loaded chunks via getChunks then mapped them to a
// stripped ChunkInput carrying only 5 fields. upsertChunks then
// OVERWROTE (not COALESCEd) the metadata columns from EXCLUDED, so
// re-embed wiped them to NULL. End result on a real brain: 4875 code
// pages, 47866 chunks, all with NULL language/symbol_name/symbol_type;
// code-def returned 0 hits across every indexed repo.
//
// All three runEmbed paths (--stale autopilot, --all, --slugs) must
// thread metadata through the re-upsert. Tests below assert that the
// engine.upsertChunks call carries the same metadata it loaded.
describe('runEmbed preserves code-chunk metadata across re-embed (regression for #769)', () => {
const fullCodeChunk = {
chunk_index: 0,
chunk_text: '[Java] foo/Bar.java:10-20 method baz',
chunk_source: 'compiled_truth' as const,
embedded_at: null,
token_count: 12,
language: 'java',
symbol_name: 'baz',
symbol_type: 'function',
start_line: 10,
end_line: 20,
parent_symbol_path: ['Bar'],
doc_comment: 'does the thing',
symbol_name_qualified: 'Bar.baz',
};
function metadataOf(chunk: any) {
return {
language: chunk.language,
symbol_name: chunk.symbol_name,
symbol_type: chunk.symbol_type,
start_line: chunk.start_line,
end_line: chunk.end_line,
parent_symbol_path: chunk.parent_symbol_path,
doc_comment: chunk.doc_comment,
symbol_name_qualified: chunk.symbol_name_qualified,
};
}
test('--stale (autopilot path) carries code metadata into upsertChunks', async () => {
const stale = [{
slug: 'code-page',
chunk_index: 0,
chunk_text: fullCodeChunk.chunk_text,
chunk_source: 'compiled_truth',
model: null,
token_count: 12,
}];
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
countStaleChunks: async () => 1,
listStaleChunks: async () => stale,
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--stale']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--all (full re-embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
listPages: async () => [{ slug: 'code-page' }],
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--all']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--slugs (per-page embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
getPage: async () => ({ slug: 'code-page', compiled_truth: 'x', timeline: '' }),
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--slugs', 'code-page']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
});
-26
View File
@@ -209,32 +209,6 @@ describe('gbrain extract --stale', () => {
expect(usRows[0]?.eq).toBe(true);
});
test('REGRESSION: page with updated_at BEFORE LINK_EXTRACTOR_VERSION_TS clears (no permanent-stale loop)', async () => {
// The v112 watermark column ships with no backfill, so every pre-existing
// page starts NULL-stale — and most pre-date the version bump. Pre-fix,
// extractStaleFromDB stamped links_extracted_at = read updated_at; for a
// page edited before LINK_EXTRACTOR_VERSION_TS the stamp landed BELOW the
// version threshold, so the version arm (links_extracted_at < versionTs)
// re-flagged it stale forever — an infinite re-extract loop that never
// cleared the lag (observed: 97% of pages stuck permanently).
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) leads [Acme](companies/acme).'));
// Backdate every page to BEFORE the extractor version timestamp.
await engine.executeRaw(`UPDATE pages SET updated_at = '2020-01-01T00:00:00Z'`);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(2);
await runExtract(engine, ['--stale']);
// Fixed: stamp = GREATEST(read updated_at, versionTs) → lifts old pages to
// the threshold so the version arm clears, while a real future edit still
// advances updated_at past the stamp (CDX-1 race protection preserved).
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
// Second run must ALSO find 0 — the defining symptom of the bug was that it
// never converged.
await runExtract(engine, ['--stale']);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
});
test('CDX-4 (D2): a link-flush throw aborts the sweep and leaves pages UNSTAMPED', async () => {
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) founded [Acme](companies/acme).'));
-108
View File
@@ -403,77 +403,6 @@ describe('extractPageLinks', () => {
expect(candidates).toEqual([]);
});
test('path-qualified wikilink outside DIR_PATTERN queries by final segment', async () => {
// `[[notes/struktura]]` (dir not in DIR_PATTERN) falls to the generic
// pass. The resolver's basename index is keyed by final path segments,
// so the lookup must strip the dirname — mirroring the FS path
// (resolveSlugAll). Regression: the raw literal was passed through,
// which never matched, so these links silently dropped.
const seen: string[] = [];
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) => {
seen.push(name);
return name === 'struktura' ? ['notes/struktura'] : [];
},
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(seen).toContain('struktura');
expect(seen).not.toContain('notes/struktura');
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
expect(candidates[0].linkType).toBe('wikilink_basename');
expect(candidates[0].linkSource).toBe('wikilink-resolved');
});
test('path-qualified wikilink keeps only matches ending with the written path', async () => {
// The written path disambiguates: `[[notes/struktura]]` must never
// attach to `wiki/struktura` even though both share the basename.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
});
test('path-qualified wikilink matches a deeper real slug by path suffix', async () => {
// The page lives at vault/notes/struktura; the author wrote the shorter
// tail `[[notes/struktura]]`. Suffix matching connects them, while the
// basename-only sibling `wiki/struktura` stays excluded.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['vault/notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['vault/notes/struktura']);
});
test('path-qualified self-link is dropped like the bare form', async () => {
// `[[notes/struktura]]` written on notes/struktura itself must not
// produce a self-loop (same guard as the bare `[[own-tail]]` case).
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'notes/struktura', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates).toEqual([]);
});
test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => {
// 2b refs (people/alice) take the verb-inferred type;
// 2c refs (struktura) take wikilink_basename. Same call.
@@ -1236,43 +1165,6 @@ describe('makeResolver — fallback chain', () => {
const out = await r.resolveBasenameMatches!('struktura');
expect(out.sort()).toEqual(['notes/struktura', 'struktura']);
});
test('opts.sourceId is forwarded to findByTitleFuzzy (twin of #1436 fix)', async () => {
// Captures every (name, dirPrefix, minSimilarity, sourceId) call so we
// can assert the resolver threads sourceId through. Without the wire-up,
// findByTitleFuzzy would be called with sourceId=undefined and the SQL
// could return cross-source slug suggestions that the FK filter
// downstream silently drops.
const calls: Array<{ name: string; dirPrefix?: string; minSimilarity?: number; sourceId?: string }> = [];
const engine = {
async getPage() { return null; },
async findByTitleFuzzy(name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) {
calls.push({ name, dirPrefix, minSimilarity, sourceId });
return null;
},
async searchKeyword() { return []; },
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
await r.resolve('Alice Example', 'people');
expect(calls.length).toBeGreaterThan(0);
expect(calls.every(c => c.sourceId === 'src-a')).toBe(true);
});
test('opts.sourceId omitted → findByTitleFuzzy receives undefined (back-compat)', async () => {
const calls: Array<{ sourceId?: string }> = [];
const engine = {
async getPage() { return null; },
async findByTitleFuzzy(_name: string, _dirPrefix?: string, _min?: number, sourceId?: string) {
calls.push({ sourceId });
return null;
},
async searchKeyword() { return []; },
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch' });
await r.resolve('Alice Example', 'people');
expect(calls.length).toBeGreaterThan(0);
expect(calls.every(c => c.sourceId === undefined)).toBe(true);
});
});
describe('FRONTMATTER_LINK_MAP integrity', () => {
-23
View File
@@ -32,29 +32,6 @@ describe('lintContent', () => {
expect(issues.some(i => i.rule === 'code-fence-wrap')).toBe(true);
});
test('no false positive: page CONTAINS an inner ```markdown code block', () => {
// Real-world case: a docs/SKILL page that shows a markdown example inline.
// Before this fix, the detector used the /m flag so ^/$ matched start/end
// of any line, which fired on any file that simply contained a ```markdown
// line. But fixContent's regex has no /m flag and can only strip whole-file
// wrappers, so the issue was reported as "fixable: true" yet never fixed.
const content =
'---\ntitle: Skill\n---\n\n# Skill\n\nExample input shape:\n\n' +
'```markdown\n# Inner page\nContent.\n```\n\nThat ends the example.\n';
const issues = lintContent(content, 'test.md');
expect(issues.filter(i => i.rule === 'code-fence-wrap')).toHaveLength(0);
});
test('no false positive: multiple inner ```markdown blocks', () => {
// Documentation pages frequently include several markdown examples.
const content =
'---\ntitle: Examples\n---\n\n# Examples\n\nFirst:\n\n' +
'```markdown\nfoo\n```\n\nSecond:\n\n' +
'```markdown\nbar\n```\n\nDone.\n';
const issues = lintContent(content, 'test.md');
expect(issues.filter(i => i.rule === 'code-fence-wrap')).toHaveLength(0);
});
test('detects placeholder dates', () => {
const content = '---\ntitle: Test\ntype: person\ncreated: YYYY-MM-DD\n---\n\n# Test';
const issues = lintContent(content, 'test.md');
-29
View File
@@ -9,7 +9,6 @@ import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
interface FakeEngine {
getConfig(key: string): Promise<string | null | undefined>;
listConfigKeys?(prefix: string): Promise<string[]>;
}
function makeEngine(map: Record<string, string | null | undefined>): FakeEngine {
@@ -17,9 +16,6 @@ function makeEngine(map: Record<string, string | null | undefined>): FakeEngine
async getConfig(key: string) {
return map[key];
},
async listConfigKeys(prefix: string) {
return Object.keys(map).filter(key => key.startsWith(prefix));
},
};
}
@@ -96,31 +92,6 @@ describe('loadConfigWithEngine (Phase 4 / F3)', () => {
expect(merged?.embedding_image_ocr).toBe(true);
});
test('DB provider_base_urls.<provider> fills the gateway base URL map', async () => {
const base: GBrainConfig = { engine: 'pglite' };
const engine = makeEngine({
'provider_base_urls.llama-server-reranker': 'http://127.0.0.1:8091/v1',
});
const merged = await loadConfigWithEngine(engine, base);
expect(merged?.provider_base_urls?.['llama-server-reranker']).toBe('http://127.0.0.1:8091/v1');
});
test('provider_base_urls merge is per-provider: file value wins and DB fills siblings', async () => {
const base: GBrainConfig = {
engine: 'pglite',
provider_base_urls: {
'llama-server-reranker': 'http://file.example/v1',
},
};
const engine = makeEngine({
'provider_base_urls.llama-server-reranker': 'http://db.example/v1',
'provider_base_urls.openrouter': 'http://openrouter.example/v1',
});
const merged = await loadConfigWithEngine(engine, base);
expect(merged?.provider_base_urls?.['llama-server-reranker']).toBe('http://file.example/v1');
expect(merged?.provider_base_urls?.openrouter).toBe('http://openrouter.example/v1');
});
test('engine.getConfig throwing is non-fatal — file/env config still returned', async () => {
const base: GBrainConfig = {
engine: 'pglite',
-62
View File
@@ -1,62 +0,0 @@
import { describe, expect, test } from 'bun:test';
import {
extractCycleFreshnessSourceIds,
parseMaintainArgs,
} from '../src/commands/maintain.ts';
import type { Check } from '../src/commands/doctor.ts';
describe('maintain args', () => {
test('defaults to dry-run unless --safe is explicit', () => {
expect(parseMaintainArgs([])).toMatchObject({
safe: false,
dryRun: true,
json: false,
});
});
test('--safe enables mutating safe mode', () => {
expect(parseMaintainArgs(['--safe', '--json'])).toMatchObject({
safe: true,
dryRun: false,
json: true,
});
});
test('--dry-run wins over --safe', () => {
expect(parseMaintainArgs(['--safe', '--dry-run'])).toMatchObject({
safe: true,
dryRun: true,
});
});
});
describe('cycle freshness source extraction', () => {
test('extracts stale source ids from doctor messages', () => {
const checks: Check[] = [
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'brain-sync-remote-teffur' last cycled 40h ago. Run `gbrain dream --source <id>`.",
},
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'wiki' last cycled 25h ago. Source 'wiki' last cycled 25h ago.",
},
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([
'brain-sync-remote-teffur',
'wiki',
]);
});
test('ignores ok and unrelated checks', () => {
const checks: Check[] = [
{ name: 'cycle_freshness', status: 'ok', message: "Source 'fresh' last cycled recently." },
{ name: 'frontmatter_integrity', status: 'warn', message: "Source 'wiki' has frontmatter issues." },
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([]);
});
});
-17
View File
@@ -1,17 +0,0 @@
import { describe, expect, it } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
describe('root OpenClaw plugin manifest', () => {
it('declares the id required by OpenClaw plugin installs', () => {
const manifest = JSON.parse(readFileSync(join(import.meta.dir, '..', 'openclaw.plugin.json'), 'utf8'));
const entrySource = readFileSync(join(import.meta.dir, '..', 'src', 'openclaw-context-engine.ts'), 'utf8');
const entryId = entrySource.match(/id:\s*'([^']+)'/)?.[1];
expect(manifest.id).toBe(entryId);
expect(manifest.configSchema).toBeDefined();
expect(typeof manifest.configSchema).toBe('object');
expect(manifest.contracts?.contextEngines).toContain('gbrain-context');
expect(entrySource).toContain('export function register');
});
});
-56
View File
@@ -186,67 +186,11 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () =>
expect(shouldExclude('entities/anonymous')).toBe(true);
expect(shouldExclude('atoms/fact-123')).toBe(true);
expect(shouldExclude('skills/gbrain-operations')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('workspace convention slugs are excluded', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('_templates/decision')).toBe(true);
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('regular slugs are NOT excluded', () => {
expect(shouldExclude('people/alice')).toBe(false);
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('writing/post-1')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
});
describe('getHealth orphan_pages uses shared exclusion policy', () => {
test('excluded convention islands do not count against health', async () => {
await engine.putPage('_templates/decision', {
type: 'template', title: 'Decision', compiled_truth: 'template', timeline: '', frontmatter: {},
});
await engine.putPage('skills/arya/source-check', {
type: 'concept', title: 'Skill', compiled_truth: 'skill', timeline: '', frontmatter: {},
});
await engine.putPage('agents/arya/identity', {
type: 'note', title: 'Identity', compiled_truth: 'identity', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
const health = await engine.getHealth();
expect(health.orphan_pages).toBe(1);
});
test('per-brain config overrides (orphans.exclude_*) also apply to health', async () => {
await engine.putPage('my-private-folder/secret-ref', {
type: 'note', title: 'Ref', compiled_truth: 'ref', timeline: '', frontmatter: {},
});
await engine.putPage('one-off-fixture-page', {
type: 'note', title: 'Fixture', compiled_truth: 'fixture', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
expect((await engine.getHealth()).orphan_pages).toBe(3);
await engine.setConfig('orphans.exclude_prefixes', 'my-private-folder/');
await engine.setConfig('orphans.exclude_slugs', 'one-off-fixture-page');
expect((await engine.getHealth()).orphan_pages).toBe(1);
await engine.unsetConfig('orphans.exclude_prefixes');
await engine.unsetConfig('orphans.exclude_slugs');
});
});
-38
View File
@@ -66,10 +66,6 @@ describe('shouldExclude', () => {
expect(shouldExclude('templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: _templates/', () => {
expect(shouldExclude('_templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: openclaw/config/', () => {
expect(shouldExclude('openclaw/config/agent')).toBe(true);
});
@@ -90,44 +86,10 @@ describe('shouldExclude', () => {
expect(shouldExclude('entities/product-hunt')).toBe(true);
});
test('excludes first-segment: skills, dreaming, and daily', () => {
expect(shouldExclude('skills/arya/source-check')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('excludes root date logs and agent workspace conventions', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('excludes generated extracts', () => {
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
});
test('brain-specific exclusions come from config overrides, not global defaults', () => {
// No baked-in defaults for these:
expect(shouldExclude('my-private-folder/some-secret-ref.md')).toBe(false);
expect(shouldExclude('one-off-fixture-page')).toBe(false);
// The per-brain config plane (orphans.exclude_prefixes / exclude_slugs):
const overrides = {
excludePrefixes: ['my-private-folder/'],
excludeSlugs: ['one-off-fixture-page'],
};
expect(shouldExclude('my-private-folder/some-secret-ref.md', overrides)).toBe(true);
expect(shouldExclude('one-off-fixture-page', overrides)).toBe(true);
expect(shouldExclude('people/jane-doe', overrides)).toBe(false);
});
test('does NOT exclude a normal content page', () => {
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('people/jane-doe')).toBe(false);
expect(shouldExclude('projects/gbrain')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
test('does NOT exclude a page ending with log-like text that is not /log', () => {
+2 -8
View File
@@ -218,21 +218,15 @@ describe('progress reporter', () => {
test('only one process-level signal handler installed across many reporters', () => {
// Baseline: one handler already installed by prior tests in this file.
const installedBefore = __signalHandlerInstalledForTest();
// liveReporters is module-global, so a reporter left running by ANOTHER
// test file in the same shard shows up here. Assert the DELTA (these 50
// lifecycles leak nothing) instead of an absolute zero — the absolute
// form flaked whenever shard composition changed and an unrelated file
// held a live reporter across this test.
const liveBefore = __liveReporterCountForTest();
const { stream } = sink(false);
for (let i = 0; i < 50; i++) {
const p = createProgress({ mode: 'json', stream, minIntervalMs: 0, minItems: 1 });
p.start(`phase_${i}`, 1);
p.finish();
}
// After 50 reporter lifecycles, still exactly one handler and no new live entries.
// After 50 reporter lifecycles, still exactly one handler and zero leaked live entries.
expect(__signalHandlerInstalledForTest()).toBe(installedBefore || true);
expect(__liveReporterCountForTest()).toBe(liveBefore);
expect(__liveReporterCountForTest()).toBe(0);
});
test('startHeartbeat() fires heartbeats and stop() clears', async () => {
-90
View File
@@ -1,90 +0,0 @@
// test/remediation-loop-termination.test.ts
//
// Regression test for the `onboard --auto` infinite loop (fix in
// src/core/remediation/run.ts). A step whose job reaches a terminal
// NON-completed state (failed / dead / cancelled) is recorded in the loop's
// `abortedIds`. Before the fix, the per-step recheck (D7) recomputed the plan
// from fresh health and re-introduced that step verbatim; because idempotency
// keys are content-stable (same job+params → same key) and the job already
// exhausted `max_attempts`, queue.add returned the SAME terminal row every
// iteration and the loop spun until `maxJobs` (default Infinity). A cancelled
// extract job under `onboard --auto` reproduced exactly this.
//
// `selectActiveRecs` is the extracted, pure progress-decision: it excludes
// aborted ids from the recomputed set so loop progress is monotonic. Full
// orchestrator e2e (real Minion handlers firing) stays deferred upstream until
// the per-handler stub seam lands (see test/e2e/onboard-full-flow.test.ts) —
// this pins the termination invariant at the decision boundary.
import { describe, expect, test } from 'bun:test';
import {
makeRemediationStep,
selectActiveRecs,
type RemediationStatus,
} from '../src/core/remediation-step.ts';
function step(id: string, status: RemediationStatus = 'remediable') {
return makeRemediationStep({
id,
job: id,
params: {},
severity: 'medium',
est_seconds: 10,
rationale: `step ${id}`,
status,
});
}
describe('selectActiveRecs — remediation loop termination', () => {
test('empty abortedIds: returns all remediable steps unchanged', () => {
const recs = [step('extract.all'), step('embed.stale')];
const out = selectActiveRecs(recs, new Set());
expect(out.map((r) => r.id)).toEqual(['extract.all', 'embed.stale']);
});
test('drops non-remediable steps (human_only / blocked)', () => {
const recs = [step('extract.all'), step('manual.only', 'human_only'), step('blk', 'blocked')];
const out = selectActiveRecs(recs, new Set());
expect(out.map((r) => r.id)).toEqual(['extract.all']);
});
test('excludes ids already in abortedIds (the core fix)', () => {
const recs = [step('extract.all'), step('embed.stale')];
const out = selectActiveRecs(recs, new Set(['extract.all']));
expect(out.map((r) => r.id)).toEqual(['embed.stale']);
});
test('infinite-loop scenario: a re-emitted aborted step is filtered out so recs drains', () => {
// Models the D7 recompute cycle. The recommendation keeps re-firing because
// its metric never improved (the job was cancelled), but once the step is in
// abortedIds it is excluded — so the active set strictly shrinks and the
// loop's `recs.length === 0` break fires instead of spinning forever.
const aborted = new Set<string>();
let recs = [step('extract.all')];
// iteration 1: step is dispatched, its job is cancelled → recorded aborted.
expect(selectActiveRecs(recs, aborted).map((r) => r.id)).toEqual(['extract.all']);
aborted.add('extract.all');
// recompute re-emits the SAME recommendation (health unchanged)…
recs = [step('extract.all')];
// …but selectActiveRecs now excludes it → empty → the loop terminates.
expect(selectActiveRecs(recs, aborted)).toEqual([]);
});
test('monotonic: once aborted, the id stays excluded across repeated rechecks', () => {
const aborted = new Set(['a']);
for (let i = 0; i < 5; i++) {
const out = selectActiveRecs([step('a'), step('b')], aborted);
expect(out.map((r) => r.id)).toEqual(['b']);
}
});
test('a NON-aborted sibling still runs while the aborted step is skipped', () => {
// Cascade sanity: aborting one step must not starve unrelated remediable
// steps — only the failed id is excluded.
const recs = [step('extract.all'), step('embed.stale'), step('backlinks.fix')];
const out = selectActiveRecs(recs, new Set(['embed.stale']));
expect(out.map((r) => r.id)).toEqual(['extract.all', 'backlinks.fix']);
});
});
+110 -1
View File
@@ -13,9 +13,10 @@
* throw on unknown string.
*/
import { describe, test, expect } from 'bun:test';
import { describe, test, expect, afterAll, afterEach } from 'bun:test';
import {
resolveEmbeddingColumn,
resolveWriteColumn,
getEmbeddingColumnRegistry,
buildVectorCastFragment,
quoteIdentifier,
@@ -34,6 +35,28 @@ import {
} from '../../src/core/search/embedding-column.ts';
import type { GBrainConfig } from '../../src/core/config.ts';
import type { ResolvedColumn } from '../../src/core/types.ts';
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
/**
* Teardown: reset AND re-apply the legacy preload config
* (test/helpers/legacy-embedding-preload.ts). A bare resetGateway() would
* leave the slot empty for the NEXT file's beforeAll (the preload's
* per-test beforeEach only fires before tests, not before beforeAll), which
* would make sibling PGLite fixtures initSchema at the 1280 default instead
* of the legacy 1536 their seed vectors assume.
*/
function restorePreloadGateway() {
resetGateway();
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
env: { ...process.env },
});
}
afterAll(() => {
restorePreloadGateway();
});
function cfg(overrides: Partial<GBrainConfig> = {}): GBrainConfig {
return { engine: 'pglite', ...overrides };
@@ -522,3 +545,89 @@ describe('codex /ship #4 — isCacheSafe (embedding-space-based skip)', () => {
expect(isCacheSafe(r, cfg())).toBe(true);
});
});
describe('resolveWriteColumn — write-side boundary resolution (#1262)', () => {
afterEach(() => {
restorePreloadGateway();
});
test('no registry / empty registry returns undefined (legacy single-column brain)', () => {
expect(resolveWriteColumn(cfg())).toBeUndefined();
expect(resolveWriteColumn(cfg({ embedding_columns: {} }))).toBeUndefined();
});
test('provider match via cfg.embedding_model returns the descriptor', () => {
const r = resolveWriteColumn(cfg({
embedding_model: 'voyage:voyage-3-large',
embedding_dimensions: 1024,
embedding_columns: {
embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
},
}));
expect(r).toEqual({
name: 'embedding_voyage',
type: 'vector',
dimensions: 1024,
embeddingModel: 'voyage:voyage-3-large',
});
});
test('provider match via gateway state (cfg.embedding_model unset) returns descriptor', () => {
configureGateway({
embedding_model: 'zeroentropyai:zembed-1',
embedding_dimensions: 2560,
env: {},
});
const r = resolveWriteColumn(cfg({
embedding_columns: {
embedding_ze: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' },
},
}));
expect(r).toEqual({
name: 'embedding_ze',
type: 'halfvec',
dimensions: 2560,
embeddingModel: 'zeroentropyai:zembed-1',
});
});
test('no provider match returns undefined instead of guessing a column', () => {
configureGateway({
embedding_model: 'zeroentropyai:zembed-1',
embedding_dimensions: 2560,
env: {},
});
const r = resolveWriteColumn(cfg({
embedding_columns: {
embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
},
}));
expect(r).toBeUndefined();
});
test('only USER-declared columns are consulted — multimodal builtin never captures text writes', () => {
// Current model equals the embedding_image BUILTIN's provider; a registry
// walk that consulted builtins would misroute text writes into the image
// column. resolveWriteColumn must return undefined here.
configureGateway({
embedding_model: 'voyage:voyage-multimodal-3',
embedding_dimensions: 1024,
env: {},
});
const r = resolveWriteColumn(cfg({
embedding_columns: {
embedding_other: { provider: 'openai:text-embedding-3-large', dimensions: 1536, type: 'vector' },
},
}));
expect(r).toBeUndefined();
});
test('malformed registry entry throws loud (same validation as the read side)', () => {
expect(() => resolveWriteColumn(cfg({
embedding_model: 'voyage:voyage-3-large',
embedding_columns: {
'bad"col': { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
} as never,
}))).toThrow(EmbeddingColumnConfigError);
});
});
-15
View File
@@ -12,11 +12,9 @@ import {
bestPath,
historyPath,
loadHistory,
proposedPath,
revertAllPending,
skillPath,
versionsDir,
writeProposed,
} from '../../src/core/skillopt/version-store.ts';
let tmpDir: string;
@@ -81,19 +79,6 @@ describe('acceptCandidate (D8 two-phase commit)', () => {
});
});
describe('writeProposed', () => {
test('writes distinct best and proposed artifacts without mutating SKILL.md (#2635)', () => {
const candidate = '---\nname: test\n---\nproposed body\n';
const written = writeProposed(tmpDir, SKILL, candidate);
expect(written).toBe(proposedPath(tmpDir, SKILL));
expect(fs.readFileSync(bestPath(tmpDir, SKILL), 'utf8')).toBe(candidate);
expect(fs.readFileSync(proposedPath(tmpDir, SKILL), 'utf8')).toBe(candidate);
expect(fs.readFileSync(skillPath(tmpDir, SKILL), 'utf8')).toContain('baseline body');
});
});
describe('revertAllPending (D8 crash recovery)', () => {
test('no-op when no pending rows', () => {
const reverted = revertAllPending(tmpDir, SKILL);
-90
View File
@@ -1,90 +0,0 @@
import { describe, expect, test } from 'bun:test';
import { stripGapsSection } from '../src/core/think/index.ts';
import { buildThinkSystemPrompt } from '../src/core/think/prompt.ts';
// `gbrain think` returns gaps in the structured `gaps` array, which both the
// CLI (`src/commands/think.ts`) and the persisted synthesis page
// (`persistSynthesis`) render exactly once. Older prompts also asked for a
// "Gaps" section inside the answer prose, so a model that still emits one made
// the output print "## Gaps" twice. `stripGapsSection` removes the prose
// section so the structured array is the single source of truth.
describe('stripGapsSection', () => {
test('removes a trailing "## Gaps" section', () => {
const answer = 'The answer with a claim [people/alice].\n\n## Gaps\n- no update since 2026-03-22 [projects/acme]\n- pricing not recorded';
const out = stripGapsSection(answer);
expect(out).not.toContain('## Gaps');
expect(out).not.toContain('no update since');
expect(out).toContain('The answer with a claim [people/alice].');
});
test('removes a level-3 "### Gaps" section', () => {
const out = stripGapsSection('Body text.\n\n### Gaps\n- missing thing');
expect(out).not.toMatch(/#+\s+Gaps/i);
expect(out).toBe('Body text.');
});
test('is case-insensitive', () => {
expect(stripGapsSection('Body.\n\n## GAPS\n- x')).toBe('Body.');
expect(stripGapsSection('Body.\n\n## gaps\n- x')).toBe('Body.');
});
test('returns the answer unchanged when there is no Gaps section', () => {
const answer = 'Just an answer.\n\n## Conflicts\n- a vs b';
expect(stripGapsSection(answer)).toBe(answer);
});
test('does not match a heading that merely starts with "Gaps"', () => {
const answer = 'Body.\n\n## Gaps in the coverage\n- this is real content';
expect(stripGapsSection(answer)).toBe(answer);
});
test('stops at the next same-or-higher heading (preserves later content)', () => {
const answer = 'Intro.\n\n## Gaps\n- missing x\n\n## Sources\n- [a]';
const out = stripGapsSection(answer);
expect(out).not.toContain('missing x');
expect(out).toContain('## Sources');
expect(out).toContain('- [a]');
});
test('handles empty / falsy input', () => {
expect(stripGapsSection('')).toBe('');
});
test('the bug repro: strip + structured render yields exactly one "## Gaps"', () => {
// Mirrors the render in src/commands/think.ts: print the (stripped) answer,
// then append one "## Gaps" block from the structured `gaps` array.
const answer = 'Answer prose [people/alice].\n\n## Gaps\n- the prose gap, slightly different wording';
const gaps = ['the structured gap'];
const rendered =
stripGapsSection(answer) + '\n\n## Gaps\n' + gaps.map((g) => `- ${g}`).join('\n');
expect((rendered.match(/## Gaps/g) ?? []).length).toBe(1);
expect(rendered).toContain('the structured gap');
});
});
describe('buildThinkSystemPrompt — gaps go in the structured array, not the answer body', () => {
test('the answer schema no longer lists "Gaps" as a body section', () => {
const out = buildThinkSystemPrompt({});
expect(out).not.toContain('Sections: Answer, Conflicts (optional), Gaps');
expect(out).toContain('gaps belong in the gaps array');
});
test('still requires the structured "gaps" array', () => {
const out = buildThinkSystemPrompt({});
expect(out).toContain('"gaps"');
});
test('preserves the Conflicts section and the Hard rules', () => {
const out = buildThinkSystemPrompt({});
expect(out).toContain('Conflicts');
expect(out).toContain('Hard rules:');
expect(out).toContain('Cite EVERY substantive claim');
});
test('willSave mode routes gaps to the structured array (no body Gaps section)', () => {
const out = buildThinkSystemPrompt({ willSave: true });
expect(out).not.toContain('cover Answer, Conflicts, and Gaps thoroughly');
expect(out).toContain('structured "gaps" array');
});
});