Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 bcdb435d73 fix(dream): deterministic last-writer attribution for colliding slugs
collectChildPutPageSlugs paired each slug to a jobId first-seen over an
unordered result set. When two children collide on a final slug, the pages
row holds the LAST put_page write, so the provenance stamp
(transcript_id/transcript_source/date) could attribute an arbitrary other
transcript. ORDER BY id + last-writer-wins aligns the stamp with the
surviving content; pinned by a collision test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 12:14:19 -07:00
Garry TanandClaude Fable 5 093d693502 test(progress): assert net-zero live-reporter delta, not absolute zero
The 'only one process-level signal handler' test asserted
__liveReporterCountForTest() === 0, which encodes 'no other test file in
this bun process left a live reporter' — a shard-composition property,
not this test's invariant. PR #3096's new test files reshuffled the LPT
shard packing so a shardmate's live reporter now lands before
progress.test.ts in CI shard 5, failing the test deterministically
(both run attempts). Snapshot the count before the 50 lifecycles and
assert the delta is zero, mirroring the installedBefore baseline the
test already uses for the signal handler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:22:03 -07:00
9268552d70 feat(dream): orchestrator-owned transcript metadata + transcriptSource discovery (#2285)
Takeover of #2286, rebased onto master so #1586's cycle-source scoping is
preserved (refs keep the cycleSourceId threading; the original branch's
rewritten collectChildPutPageSlugs hardcoded source_id 'default').

- DiscoveredTranscript carries transcriptSource, derived from the
  <corpus>/<source>/<date>/<id>.md layout; null for ad-hoc inputs.
- Transcript date inference now prefers the '| First message |' row in the
  transcript's '## Metadata' table (stable across mtime-restamping
  re-syncs), with the filename-regex date as fallback — implementing the
  cascade #2286 promised but didn't ship.
- The synthesize orchestrator stamps deterministic frontmatter
  (transcript_id, transcript_hash, transcript_source, chunk, date) through
  the existing #2569 stampDreamProvenance jsonb funnel — no second putPage
  write; reverseWriteRefs re-reads the row so DB and disk stay lockstep.
  Subagents retain authority over type/title/tags/body.

Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:25:51 -07:00
25 changed files with 401 additions and 1495 deletions
-38
View File
@@ -115,7 +115,6 @@ Full subcommand reference:
```
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
[--include <glob>...] [--exclude <glob>...]
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
--path must be a git repo (or a subdirectory of one) — see
"The git requirement for --path sources" below. --force
@@ -132,43 +131,6 @@ gbrain sources federate <id>
gbrain sources unfederate <id>
```
## Filtering what gets synced (--include / --exclude)
`--include` and `--exclude` on `gbrain sources add` accept repeatable glob
patterns and are honored by every subsequent sync AND lint of the source.
Common Obsidian vault setups need to exclude authoring scaffolding so it
doesn't pollute search:
```bash
# Skip Templates/, Drafts/, and the smart-env sidecar; everything else syncs.
gbrain sources add vault \
--path ~/Documents/vault --federated \
--exclude 'Templates/**' \
--exclude 'Drafts/**' \
--exclude '.smart-env/**'
# Or: only sync the people/ and companies/ subtrees of a CRM vault.
gbrain sources add crm \
--path ~/Documents/crm --no-federated \
--include 'people/**' \
--include 'companies/**'
```
Both persist into `sources.config.include_globs` / `exclude_globs` arrays.
The filter runs `include` first, then `exclude`, so a path inside
`people/**` is still rejected if it also matches `exclude_globs`. Globs use
the same matcher as the rest of gbrain's sync classifier (`matchesAnyGlob`
in `src/core/sync.ts`) and are matched against the source-root-relative
path. Exclusion is conservative: it never deletes previously-imported pages.
`gbrain sync --include <glob> --exclude <glob>` and
`gbrain lint <dir> --include <glob> --exclude <glob>` take the same
repeatable flags for one-off scope changes; for lint the persisted source
globs are auto-applied when the lint target matches a source's `local_path`.
Changing the persisted globs on an existing source triggers a full re-walk
on the next sync (the source's config fingerprint invalidates the
"already up to date" gate).
## The git requirement for --path sources
Every `--path` source must be a git repository (or live inside one — a
-11
View File
@@ -53,13 +53,6 @@ export async function runImport(
strategy?: SyncStrategy;
sourceId?: string;
managedBookmark?: boolean;
/**
* #2156: allow-list glob patterns — only dir-relative paths matching at
* least one pattern are imported. Applied BEFORE `exclude`. Threaded by
* performFullSync from `gbrain sync --include` / the source row's
* persisted `config.include_globs`.
*/
include?: string[];
/**
* #753/#774: glob patterns to exclude from the import (same semantics as
* `isSyncable`'s `exclude` — matched against the dir-relative path).
@@ -222,10 +215,6 @@ export async function runImport(
);
const fileTypeLabel = strategy === 'code' ? 'code'
: strategy === 'auto' ? 'syncable' : 'markdown';
// #2156: apply --include allow-list globs first (threaded by performFullSync).
if (opts.include && opts.include.length > 0) {
allFiles = allFiles.filter(abs => matchesAnyGlob(relative(dir, abs), opts.include));
}
// #753/#774: apply --exclude glob patterns (threaded by performFullSync).
if (opts.exclude && opts.exclude.length > 0) {
const beforeExclude = allFiles.length;
+12 -172
View File
@@ -17,7 +17,7 @@
*/
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
import { join, relative, resolve } from 'path';
import { join, relative } from 'path';
import { isAborted } from '../core/abort-check.ts';
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
import {
@@ -26,9 +26,7 @@ import {
DEFAULT_BYTES_WARN,
} from '../core/content-sanity.ts';
import { loadOperatorLiterals } from '../core/content-sanity-literals.ts';
import { loadConfig, loadConfigWithEngine, toEngineConfig, gbrainPath } from '../core/config.ts';
import { matchesAnyGlob } from '../core/sync.ts';
import { parseGlobList } from './sync.ts';
import { loadConfig, loadConfigWithEngine, gbrainPath } from '../core/config.ts';
import type { BrainEngine } from '../core/engine.ts';
export interface LintIssue {
@@ -380,89 +378,21 @@ async function resolveLintContentSanity(
};
}
/** Collect markdown files from a directory.
*
* When `opts.include` or `opts.exclude` are set, each candidate `.md` path's
* POSIX-style relative path (relative to `dir`) is matched against the same
* glob semantics sync uses (`matchesAnyGlob`). `include` allow-lists;
* `exclude` deny-lists. Empty or undefined arrays leave the filter
* unengaged. Symmetric with `isSyncable` in `src/core/sync.ts` so a
* source-config `exclude_globs` honored by `gbrain sync` is also honored
* by `gbrain lint` against the same dir.
*/
function collectPages(
dir: string,
opts: { include?: string[]; exclude?: string[] } = {},
): string[] {
const { include, exclude } = opts;
const haveInclude = !!(include && include.length > 0);
const haveExclude = !!(exclude && exclude.length > 0);
/** Collect markdown files from a directory */
function collectPages(dir: string): string[] {
const pages: string[] = [];
function walk(d: string) {
for (const entry of readdirSync(d)) {
if (entry.startsWith('.') || entry.startsWith('_')) continue;
const full = join(d, entry);
if (lstatSync(full).isDirectory()) walk(full);
else if (entry.endsWith('.md')) {
if (haveInclude || haveExclude) {
// Match against the path RELATIVE to `dir` (the source root),
// normalized to POSIX separators by matchesAnyGlob. A
// source-config glob like `Resources/veriff/**` is anchored at
// the source root; matching against the absolute path would
// require the user to anchor on their `$HOME` or repo prefix,
// which is brittle.
const rel = relative(dir, full);
if (haveInclude && !matchesAnyGlob(rel, include)) continue;
if (haveExclude && matchesAnyGlob(rel, exclude)) continue;
}
pages.push(full);
}
else if (entry.endsWith('.md')) pages.push(full);
}
}
walk(dir);
return pages.sort();
}
/** Look up the source row whose `local_path` resolves to the same absolute
* directory as `target`, and return its persisted `include_globs` /
* `exclude_globs` as parsed string arrays. Returns an empty object when no
* matching source exists, when the row has no globs configured, or when the
* lookup throws (best-effort — auto-resolution must never break standalone
* lint on brains without a sources table).
*
* Mirrors how `syncOneSource` lifts the same fields off `src.config` before
* threading them into `SyncOpts.include` / `SyncOpts.exclude`.
*/
async function resolveSourceGlobsForTarget(
engine: BrainEngine,
target: string,
): Promise<{ include?: string[]; exclude?: string[] }> {
try {
const absTarget = resolve(target);
const rows = await engine.executeRaw<{ config: unknown }>(
`SELECT config FROM sources
WHERE archived IS NOT TRUE
AND local_path IS NOT NULL
AND local_path = $1
LIMIT 1`,
[absTarget],
);
if (rows.length === 0) return {};
const cfg = (rows[0].config && typeof rows[0].config === 'object')
? rows[0].config as Record<string, unknown>
: {};
return {
include: parseGlobList(cfg.include_globs),
exclude: parseGlobList(cfg.exclude_globs),
};
} catch {
// Engine not connected, sources table missing on a fresh brain, RLS
// denial in an unusual scope — all best-effort. Lint proceeds without
// filtering rather than fail-closed.
return {};
}
}
export interface LintOpts {
target: string;
fix?: boolean;
@@ -484,22 +414,6 @@ export interface LintOpts {
* yields + checks this every 200 pages.
*/
signal?: AbortSignal;
/**
* Glob filters threaded into the file walker. When set, paths relative to
* `target` are matched against the patterns using the same semantics as
* `gbrain sync` (`matchesAnyGlob` in `src/core/sync.ts`). `include`
* allow-lists; `exclude` deny-lists; both unset == no filter.
*
* When BOTH are unset AND `engine` is provided, `runLintCore` attempts to
* auto-resolve them from the `sources` row whose `local_path` matches
* `target` — symmetric with `syncOneSource`, so a user who has run
* `gbrain sources add --exclude 'Resources/veriff/**'` sees the same
* exclusion applied to `gbrain lint <same-dir>` and to the cycle.lint
* phase without restating it on every invocation. Explicit caller-supplied
* arrays always win over the source-row lift.
*/
include?: string[];
exclude?: string[];
}
export interface LintResult {
@@ -526,21 +440,7 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
}
const isSingleFile = statSync(opts.target).isFile();
// Resolve glob filters. Explicit caller-supplied include/exclude win;
// otherwise lift from `sources.config.{include,exclude}_globs` when an
// engine is available and the target matches a known source's local_path.
// Single-file lints skip the resolve entirely — globs are a directory
// walk concern.
let include = opts.include;
let exclude = opts.exclude;
const haveExplicit = (include && include.length > 0) || (exclude && exclude.length > 0);
if (!isSingleFile && !haveExplicit && opts.engine) {
const resolved = await resolveSourceGlobsForTarget(opts.engine, opts.target);
include = resolved.include;
exclude = resolved.exclude;
}
const pages = isSingleFile ? [opts.target] : collectPages(opts.target, { include, exclude });
const pages = isSingleFile ? [opts.target] : collectPages(opts.target);
// Resolve content-sanity config once for this lint run (D1: lift DB
// config when reachable). Caller can pre-pass via opts.contentSanity
@@ -591,27 +491,14 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
}
export async function runLint(args: string[]) {
const target = args.find(a => !a.startsWith('--') && !args[args.indexOf(a) - 1]?.match(/^--(include|exclude)$/));
const target = args.find(a => !a.startsWith('--'));
const doFix = args.includes('--fix');
const dryRun = args.includes('--dry-run');
// Parse repeatable `--include <glob>` and `--exclude <glob>` flags.
// Symmetric with `gbrain sources add --include / --exclude` from PR #2157;
// explicit flags here override the source-config lift performed below for
// dir-mode lints.
const cliInclude: string[] = [];
const cliExclude: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] === '--include' && i + 1 < args.length) cliInclude.push(args[++i]);
else if (args[i] === '--exclude' && i + 1 < args.length) cliExclude.push(args[++i]);
}
if (!target) {
console.error('Usage: gbrain lint <dir|file.md> [--fix] [--dry-run] [--include <glob>]... [--exclude <glob>]...');
console.error(' --fix Auto-fix fixable issues (LLM preambles, code fences)');
console.error(' --dry-run Preview fixes without writing');
console.error(' --include <glob> Repeatable; only lint paths matching at least one pattern');
console.error(' --exclude <glob> Repeatable; skip paths matching any pattern (applied after --include)');
console.error('Usage: gbrain lint <dir|file.md> [--fix] [--dry-run]');
console.error(' --fix Auto-fix fixable issues (LLM preambles, code fences)');
console.error(' --dry-run Preview fixes without writing');
process.exit(1);
}
@@ -623,44 +510,7 @@ export async function runLint(args: string[]) {
// Single file or directory — print human detail as we go, then rely on
// Core for the aggregate numbers at the end.
const isSingleFile = statSync(target).isFile();
// Resolve glob filters for directory lints. Explicit CLI flags win;
// otherwise lift from `sources.config.{include,exclude}_globs` matching
// `target`. Connect a transient engine for the lookup only when (a) no
// explicit flags were passed AND (b) file/env config suggests an engine is
// available — mirrors the connect-disconnect pattern in
// `resolveLintContentSanity` (issue #1678: standalone CLI never shares the
// db.ts singleton, so create + dispose here is safe).
let runInclude: string[] | undefined = cliInclude.length > 0 ? cliInclude : undefined;
let runExclude: string[] | undefined = cliExclude.length > 0 ? cliExclude : undefined;
if (!isSingleFile && runInclude === undefined && runExclude === undefined) {
const base = loadConfig();
if (base?.database_url || base?.database_path) {
try {
const { createEngine } = await import('../core/engine-factory.ts');
const { connectWithRetry } = await import('../core/db.ts');
const engineCfg = toEngineConfig(base);
const engine = await createEngine(engineCfg);
try {
// Use the same connect path the rest of the CLI uses
// (`connectEngine` in cli.ts). `engine.connect({})` with empty
// opts drops the URL — confirmed by direct probe. `noRetry: true`
// keeps the standalone lint snappy (no retry tax when the brain
// happens to be unreachable; auto-resolve degrades to no-filter).
await connectWithRetry(engine, engineCfg, { noRetry: true });
const lifted = await resolveSourceGlobsForTarget(engine, target);
runInclude = lifted.include;
runExclude = lifted.exclude;
} finally {
await engine.disconnect().catch(() => { /* best-effort */ });
}
} catch {
// best-effort; fall through to no-filter
}
}
}
const pages = isSingleFile ? [target] : collectPages(target, { include: runInclude, exclude: runExclude });
const pages = isSingleFile ? [target] : collectPages(target);
// Progress on stderr. Stdout keeps the per-issue human output it always had.
const { createProgress } = await import('../core/progress.ts');
@@ -707,17 +557,7 @@ export async function runLint(args: string[]) {
// produces canonical numbers for the summary line).
// Pass contentSanity through so runLintCore skips its own resolve
// (we already resolved once for the human-detail loop above).
// Pass include/exclude so the aggregate scope matches the human-detail
// walk above — otherwise the summary line reports the unfiltered count
// even though the per-page details were already filtered.
const result = await runLintCore({
target,
fix: doFix,
dryRun,
contentSanity,
include: runInclude,
exclude: runExclude,
});
const result = await runLintCore({ target, fix: doFix, dryRun, contentSanity });
console.log(`\n${result.pages_scanned} pages scanned. ${result.total_issues} issue(s) in ${result.pages_with_issues} page(s).`);
if (doFix) {
console.log(`${dryRun ? '(dry run) ' : ''}${result.total_fixed} auto-fixed.`);
+1 -34
View File
@@ -122,8 +122,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
if (!id) {
console.error(
'Usage: gbrain sources add <id> [--path <path> | --url <https-url>] ' +
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>] [--force] ' +
'[--include <glob>...] [--exclude <glob>...]',
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>] [--force]',
);
process.exit(2);
}
@@ -136,12 +135,6 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
let patFile: string | undefined;
let noHarden = false;
let force = false;
// Repeatable. `--include 'people/**' --include 'companies/**'` accumulates.
// Persisted into sources.config.include_globs / .exclude_globs and read at
// sync time by commands/sync.ts so `Templates/`, `.smart-env/`, `Drafts/`
// and other vault scaffolding can be skipped without renaming directories.
const includeGlobs: string[] = [];
const excludeGlobs: string[] = [];
for (let i = 1; i < args.length; i++) {
const a = args[i];
@@ -154,24 +147,6 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
if (a === '--pat-file') { patFile = args[++i]; continue; }
if (a === '--no-harden') { noHarden = true; continue; }
if (a === '--force') { force = true; continue; }
if (a === '--include') {
const v = args[++i];
if (!v || v.startsWith('--')) {
console.error('Error: --include requires a glob argument (e.g. --include "people/**")');
process.exit(2);
}
includeGlobs.push(v);
continue;
}
if (a === '--exclude') {
const v = args[++i];
if (!v || v.startsWith('--')) {
console.error('Error: --exclude requires a glob argument (e.g. --exclude "Templates/**")');
process.exit(2);
}
excludeGlobs.push(v);
continue;
}
console.error(`Unknown flag: ${a}`);
process.exit(2);
}
@@ -192,8 +167,6 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
federated,
cloneDir,
force,
includeGlobs: includeGlobs.length > 0 ? includeGlobs : undefined,
excludeGlobs: excludeGlobs.length > 0 ? excludeGlobs : undefined,
});
// Topology A discovery: if the just-added source carries a brain-resident
@@ -217,12 +190,6 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
console.log(
` federated: ${fed}${fed ? ' — appears in cross-source default search' : ' — only searched when explicitly named via --source'}`,
);
if (includeGlobs.length > 0) {
console.log(` include globs: ${includeGlobs.join(', ')}`);
}
if (excludeGlobs.length > 0) {
console.log(` exclude globs: ${excludeGlobs.join(', ')}`);
}
// v0.42.44 — auto-harden managed clones for git durability the moment a brain
// repo is added with a PAT. Best-effort: NEVER fail `add` if hardening fails.
+18 -256
View File
@@ -1,7 +1,6 @@
import { existsSync, readFileSync, writeFileSync, statSync, realpathSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import { createHash } from 'crypto';
import type { BrainEngine } from '../core/engine.ts';
import { DELETE_BATCH_SIZE } from '../core/engine-constants.ts';
import { importFile } from '../core/import-file.ts';
@@ -757,23 +756,12 @@ export interface SyncOpts {
* are rejected before any git op runs.
*/
srcSubpath?: string;
/**
* #2156 glob patterns files must match to be synced (allow-list).
* Populated from the source row's persisted `config.include_globs`
* (set via `gbrain sources add --include <glob>`) or the repeatable
* `--include` CLI flag. Matched against the scope-relative path, same
* anchoring as `exclude`. `exclude` is applied after `include`: a path
* matching an include pattern is still rejected if it also matches an
* exclude pattern. Empty arrays are the same as undefined (no filter).
*/
include?: string[];
/**
* #753/#774 — glob patterns for files to exclude from sync (repeatable
* `--exclude` on the CLI; #2156: also populated from the source row's
* persisted `config.exclude_globs`). Matched against the scope-relative
* path in both the full-sync and incremental paths. Excluded files are
* never imported; exclusion does NOT delete previously-imported pages
* (conservative, matching the #1433 metafile posture).
* `--exclude` on the CLI). Matched against the scope-relative path in both
* the full-sync and incremental paths. Excluded files are never imported;
* exclusion does NOT delete previously-imported pages (conservative,
* matching the #1433 metafile posture).
*/
exclude?: string[];
/**
@@ -1165,29 +1153,6 @@ function unique<T>(items: T[]): T[] {
// `src/core/sync-delta.ts` (re-imported below) so the inline cost estimator
// prices detached sources through the same code the executor imports them with.
/**
* Defensive parse for the JSONB-loaded `config.include_globs` / `config.exclude_globs`
* arrays read off the sources row. The column is a free-form JSONB and could
* contain anything coerce to a string-only array, drop empties, and return
* undefined when the result has no useful entries so the caller can decide
* not to engage glob-filtering at all.
*/
export function parseGlobList(value: unknown): string[] | undefined {
if (!Array.isArray(value)) return undefined;
const globs = value.filter((v): v is string => typeof v === 'string' && v.length > 0);
return globs.length > 0 ? globs : undefined;
}
/**
* Union of CLI-supplied glob patterns (one-off, this invocation) and the
* source row's persisted config globs (every sync). Deduped; undefined when
* neither side has entries so `SyncOpts` stays unset and no filter engages.
*/
export function mergeGlobs(cli: string[], persisted: string[] | undefined): string[] | undefined {
const merged = [...new Set([...cli, ...(persisted ?? [])])];
return merged.length > 0 ? merged : undefined;
}
// v0.18.0 Step 5: source-scoped sync state helpers. When opts.sourceId
// is set, read/write the per-source row instead of the global config
// keys. These wrappers centralize the branch so every read/write site
@@ -1350,125 +1315,6 @@ async function writeChunkerVersion(
);
}
/**
* #2157 follow-on: detect when sources.config has shifted in a way that
* affects which paths the walker will include this run. The "Already up
* to date" gate at performSync's git-HEAD equality check honored chunker
* version match but ignored config drift a user who changes
* `sources.config.exclude_globs` mid-life got "Already up to date" on
* the next sync because git HEAD was unchanged, with no observable
* effect until `gbrain sync --full`.
*
* Fingerprint covers exactly the walk-affecting fields that flow from
* `sources.config` into `SyncOpts` at the syncOneSource call site:
* `strategy`, `include_globs`, `exclude_globs`. CLI-supplied --include
* / --exclude overrides do NOT participate they are one-off scope
* changes, not source state, and shouldn't invalidate the row's
* checkpoint. (A user running `gbrain sync --exclude X` on a row whose
* stored config has no X is intentionally narrowing this one pass; on
* the next no-flags sync, the row config governs again.)
*
* Array order is normalized (alphabetical, post-defensive-parse) so
* `["a/**", "b/**"]` and `["b/**", "a/**"]` fingerprint identically.
* `parseGlobList` shares the same defensive coercion as the call site
* that builds SyncOpts, so hand-edited or pre-normalization rows
* fingerprint to the same shape the walker actually sees.
*/
export function computeSourceConfigFingerprint(rawConfig: unknown): string {
const cfg = (rawConfig || {}) as {
strategy?: unknown;
include_globs?: unknown;
exclude_globs?: unknown;
};
const canonical = JSON.stringify({
strategy: typeof cfg.strategy === 'string' ? cfg.strategy : null,
include_globs: (parseGlobList(cfg.include_globs) ?? []).slice().sort(),
exclude_globs: (parseGlobList(cfg.exclude_globs) ?? []).slice().sort(),
});
return createHash('sha256').update(canonical).digest('hex');
}
/**
* Read the per-source fingerprint stamp. NULL on pre-migration rows or
* sources that have never been synced the gate treats NULL as
* "fingerprint unknown" and skips the invalidation check so first-time
* post-upgrade syncs don't spuriously force-full.
*/
export async function readConfigFingerprint(
engine: BrainEngine,
sourceId: string | undefined,
): Promise<string | null> {
if (!sourceId) return null;
const rows = await engine.executeRaw<{ config_fingerprint: string | null }>(
`SELECT config_fingerprint FROM sources WHERE id = $1`,
[sourceId],
);
return rows[0]?.config_fingerprint ?? null;
}
export async function writeConfigFingerprint(
engine: BrainEngine,
sourceId: string | undefined,
fingerprint: string,
): Promise<void> {
if (!sourceId) return;
await engine.executeRaw(
`UPDATE sources SET config_fingerprint = $1 WHERE id = $2`,
[fingerprint, sourceId],
);
}
/**
* Read the raw `sources.config` value for the named source. Returns an
* empty object for missing or never-configured rows. The reader is
* defensive about legacy double-encoded JSONB rows (`{"federated":true}`
* stored as a JSON string scalar, the #2339 class) the engine's
* `r.config` may arrive as either a string or an object, and both are
* normalized to an object before the fingerprint computation walks the
* keys.
*/
async function readSourceConfig(
engine: BrainEngine,
sourceId: string | undefined,
): Promise<unknown> {
if (!sourceId) return {};
const rows = await engine.executeRaw<{ config: unknown }>(
`SELECT config FROM sources WHERE id = $1`,
[sourceId],
);
const raw = rows[0]?.config;
if (raw === null || raw === undefined) return {};
if (typeof raw === 'string') {
try { return JSON.parse(raw); } catch { return {}; }
}
return raw;
}
/**
* Read-hash-stamp wrapper for sync-completion sites outside the gate's
* scope (e.g. `performFullSync`'s `advanceFull` closure, which doesn't
* see `performSync`'s cached `currentConfigFp` because it's a separate
* function). Reads the row's current config and stamps a fresh
* fingerprint.
*
* Race note: if `sources.config` was mutated between the gate's read
* and this stamp, the freshly-read value wins. The walker still used
* the gate-time effective globs (already captured into `opts.include` /
* `opts.exclude` upstream), so the stamp can drift from what was
* actually walked. In practice mid-sync mutations are rare and the
* NEXT sync will re-evaluate against the latest config anyway, so the
* minor staleness is acceptable and avoids threading the gate-time
* fingerprint through every helper signature.
*/
async function stampSourceConfigFingerprint(
engine: BrainEngine,
sourceId: string | undefined,
): Promise<void> {
if (!sourceId) return;
const cfg = await readSourceConfig(engine, sourceId);
await writeConfigFingerprint(engine, sourceId, computeSourceConfigFingerprint(cfg));
}
/**
* v0.40 Federated Sync v2: `gbrain sync trigger --source <id> [--priority high|normal|low]`
*
@@ -2317,25 +2163,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
detachedWorkingTreeManifest.deleted.length > 0 ||
detachedWorkingTreeManifest.renamed.length > 0);
// #2157 follow-on: parallel gate for sources.config drift. Without
// this, changing `sources.config.exclude_globs` (or include_globs /
// strategy) on a synced source had no observable effect on the next
// sync because git HEAD was unchanged — the "Already up to date"
// branch below returned without re-walking. Mismatch path mirrors the
// chunker_version gate exactly so both kinds of drift route through
// the same `performFullSync` recovery.
//
// NULL stored fingerprint is "never stamped" (pre-v125 brain OR fresh
// source whose first sync hasn't completed yet). Treated as
// pass-through in the up-to-date check — first post-upgrade sync
// stamps the column quietly so subsequent passes have a baseline.
const storedConfigFp = await readConfigFingerprint(engine, opts.sourceId);
const currentSourceConfig = await readSourceConfig(engine, opts.sourceId);
const currentConfigFp = computeSourceConfigFingerprint(currentSourceConfig);
const configMismatch = storedConfigFp !== null && storedConfigFp !== currentConfigFp;
const configNeverStamped = storedConfigFp === null && opts.sourceId !== undefined;
if (lastCommit === headCommit && !versionMismatch && !versionNeverSet && !hasDetachedWorkingTreeChanges && !configMismatch) {
if (lastCommit === headCommit && !versionMismatch && !versionNeverSet && !hasDetachedWorkingTreeChanges) {
// v0.42.52.0 (PR #22xx): bump last_sync_at as a heartbeat on every successful
// 0-changes sync. D4 invariant ("never advance last_commit on partial") is
// preserved: last_sync_at is a monitoring signal (doctor sync_freshness
@@ -2348,14 +2176,6 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
[opts.sourceId],
);
}
// First post-upgrade sync on a pre-v125 brain lands here with
// configNeverStamped=true; stamp the fingerprint so the gate has a
// baseline for the NEXT pass. A spurious re-walk on the upgrade
// pass would surprise users; quietly establishing the baseline does
// not.
if (configNeverStamped) {
await writeConfigFingerprint(engine, opts.sourceId, currentConfigFp);
}
return {
status: 'up_to_date',
fromCommit: lastCommit,
@@ -2367,21 +2187,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
};
}
if ((versionMismatch || versionNeverSet || configMismatch) && lastCommit === headCommit) {
const reasons: string[] = [];
if (versionMismatch || versionNeverSet) {
reasons.push(`chunker_version=${storedVersion ?? 'unset'}${currentVersion}`);
}
if (configMismatch) {
reasons.push(`config_fingerprint=${storedConfigFp?.slice(0, 8)}${currentConfigFp.slice(0, 8)}`);
}
if ((versionMismatch || versionNeverSet) && lastCommit === headCommit) {
slog(
`[sync] full re-walk forced (${reasons.join(', ')}): ` +
`git HEAD unchanged but a walk-affecting setting advanced.`,
`[sync] chunker_version gate: stored=${storedVersion ?? 'unset'}, current=${currentVersion}. ` +
`Forcing full re-chunk pass (git HEAD unchanged but pipeline version advanced).`,
);
const result = await performFullSync(engine, fullSyncRoots, headCommit, opts);
await writeChunkerVersion(engine, opts.sourceId, currentVersion);
await writeConfigFingerprint(engine, opts.sourceId, currentConfigFp);
return result;
}
@@ -2425,16 +2237,8 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
scoped && p.startsWith(syncScopeRelPath + '/') ? p.slice(syncScopeRelPath.length + 1) : p;
const excluded = (p: string): boolean =>
opts.exclude !== undefined && opts.exclude.length > 0 && matchesAnyGlob(scopeRel(p), opts.exclude);
// #2156: include globs are an allow-list, same scope-relative anchoring as
// exclude. Populated from the source row's persisted config.include_globs
// (or CLI --include). Deliberately NOT threaded into syncOpts/isSyncable:
// the unsyncable-cleanup loop below deletes pages for non-metafile
// classifications, and glob filtering must stay conservative (never delete
// previously-imported pages — the documented #1433 posture for --exclude).
const included = (p: string): boolean =>
opts.include === undefined || opts.include.length === 0 || matchesAnyGlob(scopeRel(p), opts.include);
// Filter to syncable files (strategy-aware + scope-aware + glob-aware)
// Filter to syncable files (strategy-aware + scope-aware + exclude-aware)
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
// #1970 (F-C): a rename whose DESTINATION is unsyncable drops out of BOTH
// `renamed` (only `r.to` is kept below) AND `deleted` (git emits it as `R`,
@@ -2448,13 +2252,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
!(inScope(r.to) && isSyncable(r.to, syncOpts)))
.map(r => r.from);
const filtered: SyncManifest = {
added: manifest.added.filter(p => inScope(p) && included(p) && !excluded(p) && isSyncable(p, syncOpts)),
modified: manifest.modified.filter(p => inScope(p) && included(p) && !excluded(p) && isSyncable(p, syncOpts)),
added: manifest.added.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
modified: manifest.modified.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
deleted: unique([
...manifest.deleted.filter(p => inScope(p) && isSyncable(p, syncOpts)),
...renamedToUnsyncable,
]),
renamed: manifest.renamed.filter(r => inScope(r.to) && included(r.to) && !excluded(r.to) && isSyncable(r.to, syncOpts)),
renamed: manifest.renamed.filter(r => inScope(r.to) && !excluded(r.to) && isSyncable(r.to, syncOpts)),
};
// NAV-4: warn when --exclude filtered out every candidate change — almost
@@ -2551,7 +2355,6 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(gitContextRoot, pin));
await engine.setConfig('sync.last_run', new Date().toISOString());
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
await writeConfigFingerprint(engine, opts.sourceId, currentConfigFp);
await clearOpCheckpoint(engine, ckpt.paths);
await clearOpCheckpoint(engine, ckpt.target);
return {
@@ -3376,7 +3179,6 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
await engine.setConfig('sync.last_run', new Date().toISOString());
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
await writeConfigFingerprint(engine, opts.sourceId, currentConfigFp);
await clearOpCheckpoint(engine, ckpt.paths);
await clearOpCheckpoint(engine, ckpt.target);
};
@@ -3628,9 +3430,6 @@ async function performFullSync(
// files were waiting.
if (opts.dryRun) {
let allFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' });
if (opts.include && opts.include.length > 0) {
allFiles = allFiles.filter(abs => matchesAnyGlob(relative(syncScopeRoot, abs), opts.include));
}
if (opts.exclude && opts.exclude.length > 0) {
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(syncScopeRoot, abs), opts.exclude));
}
@@ -3676,7 +3475,6 @@ async function performFullSync(
commit: headCommit,
strategy: opts.strategy,
sourceId: opts.sourceId,
include: opts.include,
exclude: opts.exclude,
slugRoot,
// issue #1939: performFullSync owns the failure ledger + bookmark via the
@@ -3706,7 +3504,6 @@ async function performFullSync(
await engine.setConfig('sync.last_run', new Date().toISOString());
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
await stampSourceConfigFingerprint(engine, opts.sourceId);
};
const fullGate = await applySyncFailureGate({
@@ -4170,12 +3967,8 @@ Options:
run at the repo root; imports are scoped to the subdir
and slugs stay root-relative (wiki/page1). Passing the
subdirectory directly as --repo also works.
--include <glob> Only sync files matching at least one glob (repeatable;
matched against the scope-relative path). Merged with
the source's persisted config.include_globs.
--exclude <glob> Exclude files matching the glob from sync (repeatable;
matched against the scope-relative path; applied after
--include). Merged with config.exclude_globs.
matched against the scope-relative path).
--dry-run Show what would be synced without writing.
--skip-failed Acknowledge previously-recorded sync failures so
the bookmark can advance past unparseable files.
@@ -4336,17 +4129,14 @@ See also:
}
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
// #753/#774: monorepo subdir-source flags. --exclude is repeatable.
// #2156: --include is the allow-list counterpart, same repeatable shape.
const srcSubpath = args.find((a, i) => args[i - 1] === '--src-subpath') || undefined;
const excludePatterns: string[] = [];
const includePatterns: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] === '--exclude' && i + 1 < args.length) excludePatterns.push(args[i + 1]);
if (args[i] === '--include' && i + 1 < args.length) includePatterns.push(args[i + 1]);
}
if (syncAll && (srcSubpath || excludePatterns.length > 0 || includePatterns.length > 0)) {
if (syncAll && (srcSubpath || excludePatterns.length > 0)) {
console.error(
`--src-subpath/--include/--exclude scope a single sync invocation; they cannot be combined with --all. ` +
`--src-subpath/--exclude scope a single sync invocation; they cannot be combined with --all. ` +
`For --all runs, register the subdirectory as the source's local_path instead ` +
`(gbrain sources add <id> --path <repo>/<subdir>).`,
);
@@ -4547,11 +4337,7 @@ See also:
const onAllSigint = () => { try { allInterrupt.abort(new Error('SIGINT')); } catch { /* */ } };
const runOne = async (src: typeof sources[number]): Promise<SyncResult> => {
const cfg = (src.config || {}) as {
strategy?: 'markdown' | 'code' | 'auto';
include_globs?: unknown;
exclude_globs?: unknown;
};
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
// D18: parallel path defers embed; auto-enqueue embed-backfill after.
// v0.42.42.0 (#2139): `autoDeferEmbeds` (the inline gate tripped in a
// non-TTY session) ALSO forces deferral — global by design (the gate's
@@ -4589,8 +4375,6 @@ See also:
skipFailed, retryFailed, noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
include: parseGlobList(cfg.include_globs),
exclude: parseGlobList(cfg.exclude_globs),
concurrency,
signal: composeAbortSignals(allInterrupt.signal, controller?.signal),
};
@@ -4802,27 +4586,11 @@ See also:
// lock released by its own finally) instead of a hard cut.
const singleSourceInterrupt = new AbortController();
const onSingleSourceSigint = () => { try { singleSourceInterrupt.abort(new Error('SIGINT')); } catch { /* */ } };
// Read persisted include/exclude globs from the source row, mirroring the
// --all fan-out's `runOne` closure above. Best-effort: a fetch failure
// falls through to "no glob filters", preserving pre-existing behavior.
// sourceId is always set here (resolveSourceWithTier ran above), so this
// path never silently runs without source-config awareness.
let sourceCfg: { include_globs?: unknown; exclude_globs?: unknown } = {};
try {
const { fetchSource } = await import('../core/sources-load.ts');
const src = await fetchSource(engine, sourceId);
if (src?.config && typeof src.config === 'object') {
sourceCfg = src.config as { include_globs?: unknown; exclude_globs?: unknown };
}
} catch { /* fall through to no filters */ }
const opts: SyncOpts = {
repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, sourceId,
strategy: strategyArg, concurrency,
srcSubpath,
// #2156: union of the repeatable CLI flags (one-off, this invocation
// only) and the source row's persisted config globs (every sync).
include: mergeGlobs(includePatterns, parseGlobList(sourceCfg.include_globs)),
exclude: mergeGlobs(excludePatterns, parseGlobList(sourceCfg.exclude_globs)),
exclude: excludePatterns.length > 0 ? excludePatterns : undefined,
signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
};
@@ -5050,11 +4818,7 @@ export async function syncOneSource(
noExtract?: boolean;
},
): Promise<{ result: SyncResult; log: string }> {
const cfg = (src.config || {}) as {
strategy?: 'markdown' | 'code' | 'auto';
include_globs?: unknown;
exclude_globs?: unknown;
};
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
const log = `\n--- Syncing source: ${src.name} ---\n`;
const repoOpts: SyncOpts = {
repoPath: src.local_path!,
@@ -5068,8 +4832,6 @@ export async function syncOneSource(
noSchemaPack: shared.noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
include: parseGlobList(cfg.include_globs),
exclude: parseGlobList(cfg.exclude_globs),
concurrency: shared.concurrency,
// lockId defaults to `gbrain-sync:${src.id}` via the invariant in
// performSync (no explicit override needed — sourceId triggers it).
+101 -20
View File
@@ -428,8 +428,13 @@ export async function runPhaseSynthesize(
const queue = new MinionQueue(engine);
const childIds: number[] = [];
/** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */
const chunkInfo = new Map<number, { idx: number; hash6: string }>();
/**
* Map child job_id transcript metadata. Drives D6 orchestrator-side
* slug rewrite for chunked transcripts AND the deterministic frontmatter
* stampDreamProvenance merges into each written page. Populated for
* every child (single-chunk children carry chunkTotal=1).
*/
const childMeta = new Map<number, ChildMeta>();
/** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */
const skipReports: Array<{ filePath: string; reason: string }> = [];
@@ -513,9 +518,14 @@ export async function runPhaseSynthesize(
{ allowProtectedSubmit: true },
);
childIds.push(child.id);
if (isChunked) {
chunkInfo.set(child.id, { idx: i, hash6 });
}
childMeta.set(child.id, {
idx: i,
hash6,
chunkTotal: chunks.length,
transcriptSource: t.transcriptSource,
transcriptId: stripContentVersionSuffix(t.basename),
inferredDate: t.inferredDate,
});
}
}
@@ -544,14 +554,14 @@ export async function runPhaseSynthesize(
// Collect slugs from put_page tool executions across the children
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
// D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of
// D6 orchestrator slug rewrite: childMeta drives post-hoc rewrite of
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
// even if Sonnet drops the chunk suffix.
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
// source (children write there via SubagentHandlerData.source_id).
const cycleSourceId = opts.sourceId ?? 'default';
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, childMeta, cycleSourceId);
const summaryDate = opts.date ?? today();
@@ -559,7 +569,12 @@ export async function runPhaseSynthesize(
// of every child-written page BEFORE reverse-rendering, so generated pages
// are queryable (`frontmatter->>'dream_generated'`) and a later put_page
// write-through (which re-renders from the DB row) can't erase the stamp.
await stampDreamProvenance(engine, writtenRefs, summaryDate);
// #2285: the stamp also carries the orchestrator-owned deterministic
// frontmatter (transcript_id, transcript_source, transcript_hash, date,
// chunk) derived from childMeta — subagent drift on those fields can't
// leak, and reverseWriteRefs below re-reads the row so the same fields
// land in the on-disk markdown.
await stampDreamProvenance(engine, writtenRefs, summaryDate, childMeta);
// Dual-write: reverse-render each DB row → markdown file.
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId);
@@ -1095,15 +1110,17 @@ function sanitizeForSlug(s: string): string {
* fake"): we no longer need detection because the rewrite enforces
* uniqueness at slug-write time.
*
* `chunkInfo` maps child job_id { chunk_index, hash6 }. Single-chunk
* children are absent from the map and pass through unchanged.
* `childMeta` maps child job_id per-child transcript metadata. Chunked
* children (chunkTotal > 1) get the slug rewrite; single-chunk children
* pass through unchanged. Each returned ref carries the job_id that wrote
* it so stampDreamProvenance can pair the slug back to its childMeta entry.
*/
async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
chunkInfo: Map<number, { idx: number; hash6: string }>,
childMeta: Map<number, ChildMeta>,
sourceId = 'default',
): Promise<Array<{ slug: string; source_id: string }>> {
): Promise<Array<{ slug: string; source_id: string; jobId: number }>> {
if (childIds.length === 0) return [];
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
// the orchestrator sees what each child wrote. COALESCE handles both
@@ -1122,16 +1139,73 @@ async function collectChildPutPageSlugs(
FROM subagent_tool_executions
WHERE job_id = ANY($1::int[])
AND tool_name = 'brain_put_page'
AND status = 'complete'`,
AND status = 'complete'
ORDER BY id`,
[childIds],
);
const rewritten = new Set<string>();
const rewritten = new Map<string, number>();
for (const r of rows) {
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
const ci = chunkInfo.get(r.job_id);
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
const meta = childMeta.get(r.job_id);
const finalSlug = meta && meta.chunkTotal > 1
? rewriteChunkedSlug(r.slug, meta.hash6, meta.idx)
: r.slug;
// Last writer wins, in execution-row order (ORDER BY id): if two children
// collide on a final slug, the pages row holds the LAST put_page write, so
// the stamp must attribute that child's transcript — not an arbitrary one.
rewritten.set(finalSlug, r.job_id);
}
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
return [...rewritten.entries()]
.sort(([a], [b]) => a.localeCompare(b))
.map(([slug, jobId]) => ({ slug, source_id: sourceId, jobId }));
}
/**
* Per-child orchestrator state. Drives D6 chunked-slug rewrite (idx + hash6)
* AND the deterministic frontmatter stampDreamProvenance merges into each
* written page. Populated for every child, not just chunked ones.
*/
interface ChildMeta {
idx: number;
hash6: string;
chunkTotal: number;
transcriptSource: string | null;
transcriptId: string;
inferredDate: string | null;
}
/**
* Strip the content-version suffix that claude-code-archive appends when a
* conversation is edited (`<uuid>--<contentHash>.md`). The session UUID is
* the stable transcript identifier; the suffix changes with content. Used to
* populate `transcript_id` so edits of the same session collapse to one id.
*/
function stripContentVersionSuffix(basename: string): string {
return basename.replace(/--[a-f0-9]+$/i, '');
}
/**
* Deterministic frontmatter for one synthesized page (#2285). Every field
* here is owned by the orchestrator the subagent's value for any of these
* is overwritten. The subagent retains authority over type / title / tags /
* body. `date` feeds the effective-date precedence chain
* (src/core/effective-date.ts) so re-imports keep the conversation date even
* when sync tools re-stamp file mtimes.
*/
function buildDeterministicFrontmatter(
meta: ChildMeta,
cycleDate: string,
): Record<string, unknown> {
const overrides: Record<string, unknown> = {
dream_generated: true,
dream_cycle_date: cycleDate,
transcript_id: meta.transcriptId,
transcript_hash: meta.hash6,
};
if (meta.transcriptSource) overrides.transcript_source = meta.transcriptSource;
if (meta.chunkTotal > 1) overrides.chunk = `${meta.idx + 1}/${meta.chunkTotal}`;
if (meta.inferredDate) overrides.date = meta.inferredDate;
return overrides;
}
/**
@@ -1177,12 +1251,19 @@ async function hasLegacySingleChunkCompletion(
*/
async function stampDreamProvenance(
engine: BrainEngine,
refs: Array<{ slug: string; source_id: string }>,
refs: Array<{ slug: string; source_id: string; jobId?: number }>,
cycleDate: string,
childMeta?: Map<number, ChildMeta>,
): Promise<void> {
if (refs.length === 0) return;
const { executeRawJsonb } = await import('../sql-query.ts');
for (const { slug, source_id } of refs) {
for (const { slug, source_id, jobId } of refs) {
// #2285: when the ref pairs back to a child, the stamp also carries the
// orchestrator-owned deterministic frontmatter for that transcript.
const meta = jobId !== undefined ? childMeta?.get(jobId) : undefined;
const stamp = meta
? buildDeterministicFrontmatter(meta, cycleDate)
: { dream_generated: true, dream_cycle_date: cycleDate };
try {
await executeRawJsonb(
engine,
@@ -1190,7 +1271,7 @@ async function stampDreamProvenance(
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
WHERE slug = $1 AND source_id = $2`,
[slug, source_id],
[{ dream_generated: true, dream_cycle_date: cycleDate }],
[stamp],
);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
+58 -5
View File
@@ -10,7 +10,7 @@
*/
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join, basename } from 'node:path';
import { join, basename, dirname } from 'node:path';
import { createHash } from 'node:crypto';
import { pruneDir } from '../sync.ts';
@@ -23,8 +23,22 @@ export interface DiscoveredTranscript {
content: string;
/** Filename basename without extension; used as a topic-slug seed. */
basename: string;
/** Inferred date if the basename matches `YYYY-MM-DD...` (or null). */
/**
* Inferred conversation date (YYYY-MM-DD) or null. Precedence: the
* `| First message | <ISO> |` row in the transcript's `## Metadata`
* table (stable across mtime-restamping re-syncs) wins; a leading
* `YYYY-MM-DD` in the basename is the fallback.
*/
inferredDate: string | null;
/**
* Transcript source archive name, derived from the path's grandparent
* directory (the immediate parent of the date directory). For the
* canonical layout `<corpus>/<source>/<date>/<id>.md` this yields the
* source-name segment e.g. `claude-code` for the claude-code-archive
* output, `meetings` for meeting recordings. Null when the file does
* not live under a `<source>/<date>/` pair (ad-hoc inputs).
*/
transcriptSource: string | null;
}
export interface DiscoverOpts {
@@ -161,6 +175,36 @@ function matchesAnyExclude(text: string, patterns: RegExp[]): boolean {
return false;
}
/**
* Content-based conversation date: the `| First message | <ISO timestamp> |`
* row claude-code-archive writes into the transcript's `## Metadata` table.
* Stable across rsync/Dropbox/Syncthing/B2 re-syncs that re-stamp mtime,
* unlike anything derived from file metadata. Returns YYYY-MM-DD or null.
*/
const FIRST_MESSAGE_RE = /^\|\s*First message\s*\|\s*(\d{4}-\d{2}-\d{2})/im;
export function inferContentDate(content: string): string | null {
const m = FIRST_MESSAGE_RE.exec(content);
return m ? m[1] : null;
}
/**
* Derive the archive source name from a transcript path. Returns the basename
* of the directory two levels above the file when the immediate parent is a
* date directory and the grandparent looks like a source-name slug (lowercase
* alphanumeric segments separated by hyphens); otherwise null. This pins the
* canonical claude-code-archive layout `<corpus>/<source>/<date>/<id>.md`
* without claiming a source for ad-hoc inputs that don't match.
*/
export function deriveTranscriptSource(filePath: string): string | null {
const parentName = basename(dirname(filePath));
if (!/^\d{4}-\d{2}-\d{2}/.test(parentName)) return null;
const grandparentName = basename(dirname(dirname(filePath)));
if (!grandparentName) return null;
if (!/^[a-z0-9]+(-[a-z0-9]+)*$/.test(grandparentName)) return null;
return grandparentName;
}
function listTextFiles(dir: string): string[] {
// Recursive walk with descent-time pruning (closes codex C12/C13 spec gap).
// Accepts BOTH .txt and .md per transcript-discovery's domain rules — does
@@ -225,8 +269,11 @@ export function discoverTranscripts(opts: DiscoverOpts): DiscoveredTranscript[]
const ext = filePath.endsWith('.md') ? '.md' : '.txt';
const baseName = basename(filePath, ext);
const dateMatch = DATE_RE.exec(baseName);
const inferredDate = dateMatch ? dateMatch[1] : null;
if (!isInDateRange(inferredDate, opts)) continue;
const filenameDate = dateMatch ? dateMatch[1] : null;
// Fast path: date-named files outside the window skip before the read.
// ponytail: a date-named file whose content date differs is filtered on
// its filename date — acceptable; archive layouts use UUID basenames.
if (filenameDate && !isInDateRange(filenameDate, opts)) continue;
let content: string;
try {
@@ -241,12 +288,17 @@ export function discoverTranscripts(opts: DiscoverOpts): DiscoveredTranscript[]
}
if (matchesAnyExclude(content, excludeRes)) continue;
// Content-metadata date wins (survives mtime restamps); filename next.
const inferredDate = inferContentDate(content) ?? filenameDate;
if (!isInDateRange(inferredDate, opts)) continue;
results.push({
filePath,
contentHash: hashContent(content),
content,
basename: baseName,
inferredDate,
transcriptSource: deriveTranscriptSource(filePath),
});
}
}
@@ -290,6 +342,7 @@ export function readSingleTranscript(
contentHash: hashContent(content),
content,
basename: baseName,
inferredDate: dateMatch ? dateMatch[1] : null,
inferredDate: inferContentDate(content) ?? (dateMatch ? dateMatch[1] : null),
transcriptSource: deriveTranscriptSource(filePath),
};
}
-26
View File
@@ -5671,32 +5671,6 @@ export const MIGRATIONS: Migration[] = [
`);
},
},
{
version: 125,
name: 'sources_config_fingerprint',
// #2157 follow-on: the "Already up to date" gate at sync.ts honors
// git-HEAD equality + chunker-version match but ignored source-config
// drift. A user who runs `gbrain sources add default --exclude
// 'Templates/**'` AFTER an initial sync got "Already up to date" on
// the next pass because git HEAD was unchanged — the new exclusion
// never reached the walk until `gbrain sync --full`.
//
// This column caches a SHA-256 fingerprint of the walk-affecting
// fields in `sources.config` (strategy + include_globs +
// exclude_globs); mismatches trigger a full re-walk via the same code
// path as a chunker_version bump.
//
// NULL on pre-migration rows is treated as "not yet stamped" by
// readConfigFingerprint, so the FIRST sync after upgrade is normal
// (no spurious force-full just because the column was added).
//
// Keep in sync with src/schema.sql and src/core/schema-embedded.ts.
idempotent: true,
sql: `
ALTER TABLE sources
ADD COLUMN IF NOT EXISTS config_fingerprint TEXT;
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
-7
View File
@@ -39,13 +39,6 @@ CREATE TABLE IF NOT EXISTS sources (
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
-- actually trigger re-chunking on upgrade.
chunker_version TEXT,
-- #2157 follow-on: SHA-256 fingerprint of the walk-affecting fields in
-- \`config\` (strategy + include_globs + exclude_globs). Mismatch forces a
-- full re-walk via the same code path as chunker_version, so a user who
-- changes \`sources.config.exclude_globs\` mid-life doesn't get "Already up
-- to date" on the next sync. NULL on pre-migration rows is treated as
-- "not yet stamped" and skips the gate (preserves first-run semantics).
config_fingerprint TEXT,
-- v0.26.5: soft-delete + recovery window. \`archive\` flips archived=true and
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
-25
View File
@@ -155,19 +155,6 @@ export interface AddSourceOpts {
* runs). Does NOT auto-`git init` anything see `addSource` docstring.
*/
force?: boolean;
/**
* Glob filters persisted into `sources.config.include_globs` /
* `sources.config.exclude_globs`. Read at sync time by
* `commands/sync.ts:syncOneSource` and the single-source path, threaded
* into `isSyncable` / `unsyncableReason` (their `SyncableOptions` shape
* has carried this contract since v0.41.13).
*
* Empty / unspecified arrays are not persisted at all (no `[]` written
* to the JSONB), which keeps the row identical to today for sources
* that don't use filtering.
*/
includeGlobs?: string[];
excludeGlobs?: string[];
}
export interface RemoveSourceOpts {
@@ -442,12 +429,6 @@ export async function addSource(
if (opts.federated !== null && opts.federated !== undefined) {
config.federated = opts.federated;
}
if (opts.includeGlobs && opts.includeGlobs.length > 0) {
config.include_globs = opts.includeGlobs;
}
if (opts.excludeGlobs && opts.excludeGlobs.length > 0) {
config.exclude_globs = opts.excludeGlobs;
}
const displayName = opts.name ?? opts.id;
try {
@@ -527,12 +508,6 @@ export async function addSource(
if (opts.federated !== null && opts.federated !== undefined) {
config.federated = opts.federated;
}
if (opts.includeGlobs && opts.includeGlobs.length > 0) {
config.include_globs = opts.includeGlobs;
}
if (opts.excludeGlobs && opts.excludeGlobs.length > 0) {
config.exclude_globs = opts.excludeGlobs;
}
const displayName = opts.name ?? opts.id;
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config)
-7
View File
@@ -219,13 +219,6 @@ function globToRegex(pattern: string): RegExp {
return new RegExp(regex);
}
/**
* Test a normalized POSIX-style path against an array of glob patterns. Returns
* true if any pattern matches. Empty / undefined `patterns` returns false (no
* filter engaged). Exported so non-sync surfaces (lint walker, future ingest
* variants) can apply the same glob semantics as `isSyncable` without
* re-declaring `globToRegex`.
*/
export function matchesAnyGlob(path: string, patterns?: string[]): boolean {
if (!patterns || patterns.length === 0) return false;
const normalized = path.replace(/\\/g, '/');
-7
View File
@@ -35,13 +35,6 @@ CREATE TABLE IF NOT EXISTS sources (
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
-- actually trigger re-chunking on upgrade.
chunker_version TEXT,
-- #2157 follow-on: SHA-256 fingerprint of the walk-affecting fields in
-- `config` (strategy + include_globs + exclude_globs). Mismatch forces a
-- full re-walk via the same code path as chunker_version, so a user who
-- changes `sources.config.exclude_globs` mid-life doesn't get "Already up
-- to date" on the next sync. NULL on pre-migration rows is treated as
-- "not yet stamped" and skips the gate (preserves first-run semantics).
config_fingerprint TEXT,
-- v0.26.5: soft-delete + recovery window. `archive` flips archived=true and
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
+1
View File
@@ -26,6 +26,7 @@ const transcript: DiscoveredTranscript = {
content: 'User: hello world',
contentHash: 'abcdef0123456789',
inferredDate: '2026-07-17',
transcriptSource: null,
} as DiscoveredTranscript;
describe('#2415: buildSynthesisPrompt output root', () => {
@@ -152,3 +152,94 @@ describe('#2569: stampDreamProvenance persists the marker into DB frontmatter',
await stampDreamProvenance(engine as any, refs, '2026-07-17'); // idempotent
});
});
describe('#2285: orchestrator-owned deterministic transcript frontmatter', () => {
const meta = {
idx: 1,
hash6: 'abc123',
chunkTotal: 3,
transcriptSource: 'claude-code',
transcriptId: 'session-uuid',
inferredDate: '2026-05-15',
};
test('collectChildPutPageSlugs pairs each ref back to the writing job', async () => {
const refs = await collectChildPutPageSlugs(
engine as any, [1001], new Map([[1001, { ...meta, chunkTotal: 1 }]]), 'mybrain',
);
expect(refs.length).toBeGreaterThan(0);
for (const r of refs) {
expect(r.jobId).toBe(1001);
expect(r.source_id).toBe('mybrain'); // #1586: cycle source, never hardcoded 'default'
}
});
test('slug collision across children attributes the LAST writer (matches surviving putPage)', async () => {
const db = (engine as any).db;
// Jobs 1001 then 1002 write the same slug; the pages row would hold
// 1002's content (last put_page wins), so the ref must carry jobId 1002.
await db.query(
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, status, input)
VALUES (1001, 9, 'tool_dup_a', 'brain_put_page', 'complete', $1::jsonb)`,
[JSON.stringify({ slug: 'wiki/agents/test/collision', body: 'first' })],
);
await db.query(
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, status, input)
VALUES (1002, 9, 'tool_dup_b', 'brain_put_page', 'complete', $1::jsonb)`,
[JSON.stringify({ slug: 'wiki/agents/test/collision', body: 'second' })],
);
const refs = await collectChildPutPageSlugs(engine as any, [1001, 1002], new Map());
const hit = refs.find((r: { slug: string }) => r.slug === 'wiki/agents/test/collision');
expect(hit?.jobId).toBe(1002);
});
test('stampDreamProvenance merges the transcript metadata into DB frontmatter', async () => {
const slug = 'wiki/originals/ideas/2026-07-17-transcript-meta-abc123';
await engine.putPage(slug, {
type: 'note',
title: 'Meta stamp',
compiled_truth: 'body',
timeline: '',
frontmatter: { keep_me: 'yes', transcript_id: 'subagent-drift' },
});
await stampDreamProvenance(
engine as any,
[{ slug, source_id: 'default', jobId: 42 }],
'2026-07-17',
new Map([[42, meta]]),
);
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
`SELECT frontmatter AS fm FROM pages WHERE slug = $1`, [slug],
);
const fm = rows[0].fm as Record<string, unknown>;
expect(fm.dream_generated).toBe(true);
expect(fm.dream_cycle_date).toBe('2026-07-17');
expect(fm.transcript_id).toBe('session-uuid'); // orchestrator wins over subagent drift
expect(fm.transcript_hash).toBe('abc123');
expect(fm.transcript_source).toBe('claude-code');
expect(fm.chunk).toBe('2/3');
expect(fm.date).toBe('2026-05-15');
expect(fm.keep_me).toBe('yes'); // subagent-owned keys survive
});
test('single-chunk children with no inferredDate stamp only the applicable fields', async () => {
const slug = 'wiki/originals/ideas/2026-07-17-minimal-meta-abc123';
await engine.putPage(slug, {
type: 'note', title: 'Minimal', compiled_truth: 'b', timeline: '', frontmatter: {},
});
await stampDreamProvenance(
engine as any,
[{ slug, source_id: 'default', jobId: 43 }],
'2026-07-17',
new Map([[43, { ...meta, chunkTotal: 1, transcriptSource: null, inferredDate: null }]]),
);
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
`SELECT frontmatter AS fm FROM pages WHERE slug = $1`, [slug],
);
const fm = rows[0].fm as Record<string, unknown>;
expect(fm.transcript_id).toBe('session-uuid');
expect(fm.chunk).toBeUndefined();
expect(fm.transcript_source).toBeUndefined();
expect(fm.date).toBeUndefined();
});
});
+2
View File
@@ -302,6 +302,7 @@ describe('judgeSignificance', () => {
content: 'A short conversation about something interesting.',
basename: 'x',
inferredDate: null,
transcriptSource: null,
};
}
@@ -415,6 +416,7 @@ describe('judgeSignificance — UTF-16 safety (v0.41.13)', () => {
content,
basename: 'long',
inferredDate: null,
transcriptSource: null,
};
}
@@ -45,6 +45,7 @@ const FIXTURE_TRANSCRIPT: DiscoveredTranscript = {
content: 'Synthetic transcript content for gateway-adapter parity tests.',
contentHash: 'sha-fixture-1',
inferredDate: '2026-05-24',
transcriptSource: null,
};
describe('makeJudgeClient — construction-time provider probe', () => {
@@ -0,0 +1,109 @@
/**
* #2285 transcript metadata discovery.
*
* Pins the two discovery-side additions:
* 1. `transcriptSource` derived from the `<source>/<date>/<file>` path
* layout; null for ad-hoc inputs that don't match.
* 2. Content-based date inference the `| First message | <ISO> |` row in
* the transcript's `## Metadata` table wins over the filename-regex
* date (stable across mtime-restamping re-syncs); filename is the
* fallback.
*
* Pure filesystem; no engine, no LLM.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, dirname } from 'node:path';
import {
discoverTranscripts,
readSingleTranscript,
deriveTranscriptSource,
inferContentDate,
} from '../../src/core/cycle/transcript-discovery.ts';
let tmpDir: string;
beforeEach(() => {
tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-transcript-meta-'));
});
afterEach(() => {
rmSync(tmpDir, { recursive: true, force: true });
});
function write(relPath: string, body: string): string {
const full = join(tmpDir, relPath);
mkdirSync(dirname(full), { recursive: true });
writeFileSync(full, body);
return full;
}
const FILLER = 'User: hello world. '.repeat(200);
const METADATA_BLOCK =
'## Metadata\n\n| Key | Value |\n| --- | --- |\n| First message | 2026-05-15T03:51:11.584Z |\n\n';
describe('deriveTranscriptSource', () => {
test('extracts the source slug from <source>/<date>/<file> layout', () => {
expect(deriveTranscriptSource('/corpus/claude-code/2026-06-12/abc.md')).toBe('claude-code');
expect(deriveTranscriptSource('/corpus/voice-notes/2026-06-12/xyz.md')).toBe('voice-notes');
});
test('null when the parent dir is not a date dir or grandparent is not a slug', () => {
expect(deriveTranscriptSource('/corpus/flat-file.md')).toBeNull();
expect(deriveTranscriptSource('/corpus/claude-code/not-a-date/abc.md')).toBeNull();
expect(deriveTranscriptSource('/corpus/Not A Slug/2026-06-12/abc.md')).toBeNull();
});
});
describe('inferContentDate', () => {
test('parses the | First message | row', () => {
expect(inferContentDate(METADATA_BLOCK)).toBe('2026-05-15');
});
test('null when absent', () => {
expect(inferContentDate(FILLER)).toBeNull();
});
});
describe('discoverTranscripts — transcriptSource + date cascade', () => {
test('populates transcriptSource per file; null for flat files', () => {
write('claude-code/2026-06-12/aaaa.md', FILLER);
write('2026-06-12-flat.md', FILLER);
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 100 });
const byBase = new Map(out.map(t => [t.basename, t.transcriptSource]));
expect(byBase.get('aaaa')).toBe('claude-code');
expect(byBase.get('2026-06-12-flat')).toBeNull();
});
test('content First-message date wins over the filename date', () => {
write('2026-01-01-named.md', METADATA_BLOCK + FILLER);
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 100 });
expect(out).toHaveLength(1);
expect(out[0].inferredDate).toBe('2026-05-15');
});
test('filename date remains the fallback when content has no metadata row', () => {
write('2026-01-01-named.md', FILLER);
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 100 });
expect(out[0].inferredDate).toBe('2026-01-01');
});
test('date filter matches on the content date for UUID-named transcripts', () => {
write('claude-code/2026-05-15/uuid-basename.md', METADATA_BLOCK + FILLER);
const hit = discoverTranscripts({ corpusDir: tmpDir, minChars: 100, date: '2026-05-15' });
expect(hit).toHaveLength(1);
const miss = discoverTranscripts({ corpusDir: tmpDir, minChars: 100, date: '2026-05-16' });
expect(miss).toHaveLength(0);
});
});
describe('readSingleTranscript — same metadata surface', () => {
test('carries transcriptSource and prefers the content date', () => {
const p = write('claude-code/2026-05-15/2026-01-01-single.md', METADATA_BLOCK + FILLER);
const t = readSingleTranscript(p, { minChars: 100 });
expect(t).not.toBeNull();
expect(t!.transcriptSource).toBe('claude-code');
expect(t!.inferredDate).toBe('2026-05-15');
});
});
-153
View File
@@ -1,153 +0,0 @@
/**
* `gbrain lint` source-glob filter walker integration.
*
* PR #2157 (commit cf9a3b18, `feat/sync-source-glob-filters`) wired
* `sources.config.include_globs` / `exclude_globs` into `gbrain sync` so a
* user could exclude `Resources/veriff/**` and have every subsequent sync
* honor it. The lint command walked the same source dirs blind and emitted
* findings against paths the user had already declared out of scope a
* half-finished feature.
*
* This patch extends the same persisted glob contract to lint:
* - `gbrain lint` gains `--include / --exclude` flags (parallel to sync).
* - `runLintCore` lifts `sources.config.{include,exclude}_globs` for any
* target whose absolute path matches a source row's `local_path`, so the
* cycle.lint phase + Minion lint handlers honor the same filter without
* restating it.
* - The walker in `collectPages` applies the filter using the SAME
* `matchesAnyGlob` helper sync uses, anchored at the target dir (so a
* persisted `Resources/veriff/**` glob written against the source root
* works without rewriting it as an absolute path).
*
* These tests pin the walker contract. The engine-side lift
* (`resolveSourceGlobsForTarget`) is best-effort by design (returns `{}` on
* any error) and is exercised by the dream-cycle lint phase end-to-end.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
// runLintCore is the library entry — the same surface the cycle.lint phase
// and Minion handlers call. Exercising it covers the walker via its real
// callsite; testing `collectPages` directly would skip the wiring.
import { runLintCore } from '../src/commands/lint.ts';
// A self-contained content-sanity stub so the test never touches a real
// engine / config file. Empty operator-literal list keeps the content-sanity
// pass silent so the only findings come from the structural rules
// (no-frontmatter etc.).
const STUB_CS = {
fail_on_throw: false,
warn_on_throw: false,
bytes_warn: 1024 * 1024,
operator_literals: [],
};
describe('runLintCore — source-glob walker filter', () => {
let root: string;
beforeAll(() => {
root = mkdtempSync(join(tmpdir(), 'gbrain-lint-globs-'));
// Three subtrees with mixed structured / archive-style content.
// All pages have `# Title` headers but no frontmatter so each one
// emits at least one `no-frontmatter` issue under the default rule set.
mkdirSync(join(root, 'Notes'), { recursive: true });
mkdirSync(join(root, 'Resources', 'veriff'), { recursive: true });
mkdirSync(join(root, 'Resources', 'prior-art', 'archive-v1'), { recursive: true });
writeFileSync(join(root, 'Notes', 'a.md'), '# A\nbody\n');
writeFileSync(join(root, 'Notes', 'b.md'), '# B\nbody\n');
writeFileSync(join(root, 'Resources', 'veriff', 'spec-1.md'), '# Veriff spec 1\nbody\n');
writeFileSync(join(root, 'Resources', 'veriff', 'spec-2.md'), '# Veriff spec 2\nbody\n');
writeFileSync(join(root, 'Resources', 'prior-art', 'archive-v1', 'old.md'), '# Old\nbody\n');
});
afterAll(() => {
rmSync(root, { recursive: true, force: true });
});
test('no filter — walks every .md (regression guard for default behavior)', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
});
expect(result.pages_scanned).toBe(5);
expect(result.pages_with_issues).toBeGreaterThan(0);
});
test('exclude glob skips matching paths (Resources/veriff/** off-limits)', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
exclude: ['Resources/veriff/**'],
});
// 5 total minus 2 veriff specs = 3 pages walked.
expect(result.pages_scanned).toBe(3);
});
test('exclude with multiple patterns is union (veriff + prior-art both skipped)', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
exclude: ['Resources/veriff/**', 'Resources/prior-art/**'],
});
// 5 total minus 3 (2 veriff + 1 archive-v1) = 2 pages walked.
expect(result.pages_scanned).toBe(2);
});
test('include glob narrows the walk to matching paths only', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
include: ['Notes/**'],
});
expect(result.pages_scanned).toBe(2);
});
test('exclude runs AFTER include (same precedence as `gbrain sync`)', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
include: ['**/*.md'],
exclude: ['Resources/**'],
});
// include lets everything through; exclude drops the 3 Resources/* files.
expect(result.pages_scanned).toBe(2);
});
test('empty include / exclude arrays do NOT engage the filter', async () => {
// Symmetric with `parseGlobList` returning undefined for empty input —
// an empty include would otherwise classify every path as a miss and
// silently zero out the lint scope. Pin the guard at the walker level.
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
include: [],
exclude: [],
});
expect(result.pages_scanned).toBe(5);
});
test('exclude semantics match sync — `**` matches across path segments', async () => {
const result = await runLintCore({
target: root,
contentSanity: STUB_CS,
exclude: ['**/spec-*.md'],
});
// Both Veriff specs match the deep glob; Notes + archive-v1 survive.
expect(result.pages_scanned).toBe(3);
});
test('single-file target bypasses the filter (file mode is not a walk)', async () => {
// A user lints one .md explicitly: filters are a directory-walk concern,
// so the file is processed even if its name would match an exclude.
const result = await runLintCore({
target: join(root, 'Resources', 'veriff', 'spec-1.md'),
contentSanity: STUB_CS,
exclude: ['Resources/veriff/**'],
});
expect(result.pages_scanned).toBe(1);
});
});
+7 -3
View File
@@ -216,17 +216,21 @@ describe('progress reporter', () => {
});
test('only one process-level signal handler installed across many reporters', () => {
// Baseline: one handler already installed by prior tests in this file.
// Baseline: one handler already installed by prior tests in this file, and
// possibly live reporters from OTHER test files sharing this bun process
// (shard composition is not this test's invariant — assert the delta, not
// an absolute zero, or shard reshuffles make this fail spuriously).
const installedBefore = __signalHandlerInstalledForTest();
const liveBefore = __liveReporterCountForTest();
const { stream } = sink(false);
for (let i = 0; i < 50; i++) {
const p = createProgress({ mode: 'json', stream, minIntervalMs: 0, minItems: 1 });
p.start(`phase_${i}`, 1);
p.finish();
}
// After 50 reporter lifecycles, still exactly one handler and zero leaked live entries.
// After 50 reporter lifecycles, still exactly one handler and zero NET leaked live entries.
expect(__signalHandlerInstalledForTest()).toBe(installedBefore || true);
expect(__liveReporterCountForTest()).toBe(0);
expect(__liveReporterCountForTest()).toBe(liveBefore);
});
test('startHeartbeat() fires heartbeats and stop() clears', async () => {
-6
View File
@@ -694,12 +694,6 @@ const COLUMN_EXEMPTIONS = new Set<string>([
'minion_jobs.quiet_hours',
'minion_jobs.stagger_key',
'sources.chunker_version',
// #2157 follow-on (migration v125). TEXT column read by performSync's
// `Already up to date` gate; not referenced by any CREATE INDEX. Same
// upgrade-path coverage as sources.chunker_version above: fresh installs
// get it via the CREATE TABLE in src/schema.sql + schema-embedded.ts;
// pre-existing brains get it via the idempotent ALTER TABLE in v125.
'sources.config_fingerprint',
'access_tokens.permissions',
'takes.resolved_quality',
'pages.emotional_weight_recomputed_at',
-121
View File
@@ -159,127 +159,6 @@ describe('sources add', () => {
await expect(runSources(engine, ['add', 'plans', '--path', '/tmp/gstack/plans']))
.rejects.toThrow(/overlaps with existing source "gstack"/);
});
// Glob filters — TODO #3 from the brettdavies fork recon. Pre-fix, the
// `SyncableOptions` shape in `src/core/sync.ts` had been carrying
// `include` / `exclude` since v0.41.13, but commands/sync.ts:1454 never
// populated them and `sources add` had no flag to persist them — so users
// had no way to tell gbrain to skip `Templates/` in an Obsidian vault.
test('--exclude persists glob into sources.config.exclude_globs', async () => {
const { engine, calls } = makeStub({
'SELECT id, name, local_path, last_commit, last_sync_at, config, created_at': [{
id: 'vault',
name: 'vault',
local_path: '/tmp/vault',
last_commit: null,
last_sync_at: null,
config: '{"exclude_globs":["Templates/**"]}',
created_at: new Date(),
}],
});
await runSources(engine, ['add', 'vault', '--path', '/tmp/vault', '--exclude', 'Templates/**']);
const insert = calls.find(c => c.sql.includes('INSERT INTO sources'));
expect(insert!.params[3]).toBe('{"exclude_globs":["Templates/**"]}');
});
test('--include persists glob into sources.config.include_globs', async () => {
const { engine, calls } = makeStub({
'SELECT id, name, local_path, last_commit, last_sync_at, config, created_at': [{
id: 'wiki',
name: 'wiki',
local_path: '/tmp/wiki',
last_commit: null,
last_sync_at: null,
config: '{"include_globs":["people/**"]}',
created_at: new Date(),
}],
});
await runSources(engine, ['add', 'wiki', '--path', '/tmp/wiki', '--include', 'people/**']);
const insert = calls.find(c => c.sql.includes('INSERT INTO sources'));
expect(insert!.params[3]).toBe('{"include_globs":["people/**"]}');
});
test('--exclude is repeatable; preserves order', async () => {
const { engine, calls } = makeStub({
'SELECT id, name, local_path, last_commit, last_sync_at, config, created_at': [{
id: 'vault',
name: 'vault',
local_path: '/tmp/vault',
last_commit: null,
last_sync_at: null,
config: '{}',
created_at: new Date(),
}],
});
await runSources(engine, [
'add', 'vault', '--path', '/tmp/vault',
'--exclude', 'Templates/**',
'--exclude', '.smart-env/**',
'--exclude', 'Drafts/**',
]);
const insert = calls.find(c => c.sql.includes('INSERT INTO sources'));
expect(insert!.params[3]).toBe('{"exclude_globs":["Templates/**",".smart-env/**","Drafts/**"]}');
});
test('--include and --exclude compose in one command (federated source with both filter axes)', async () => {
const { engine, calls } = makeStub({
'SELECT id, name, local_path, last_commit, last_sync_at, config, created_at': [{
id: 'vault',
name: 'vault',
local_path: '/tmp/vault',
last_commit: null,
last_sync_at: null,
config: '{"federated":true,"include_globs":["people/**"],"exclude_globs":["Templates/**"]}',
created_at: new Date(),
}],
});
await runSources(engine, [
'add', 'vault', '--path', '/tmp/vault', '--federated',
'--include', 'people/**',
'--exclude', 'Templates/**',
]);
const insert = calls.find(c => c.sql.includes('INSERT INTO sources'));
expect(insert!.params[3]).toBe(
'{"federated":true,"include_globs":["people/**"],"exclude_globs":["Templates/**"]}',
);
});
test('omitted glob flags leave config untouched (no [] entries persisted)', async () => {
// Regression guard: empty glob arrays must NOT be written. Otherwise a
// brain that never opts into filtering grows {"include_globs": [],
// "exclude_globs": []} cruft in every source row, and the parseGlobList
// path would return undefined anyway (the cruft is purely noise).
const { engine, calls } = makeStub({
'SELECT id, name, local_path, last_commit, last_sync_at, config, created_at': [{
id: 'gstack',
name: 'gstack',
local_path: '/tmp/gstack',
last_commit: null,
last_sync_at: null,
config: '{}',
created_at: new Date(),
}],
});
await runSources(engine, ['add', 'gstack', '--path', '/tmp/gstack']);
const insert = calls.find(c => c.sql.includes('INSERT INTO sources'));
expect(insert!.params[3]).toBe('{}');
});
test('--exclude requires a glob argument', async () => {
const { engine } = makeStub();
const code = await withExitCapture(() => runSources(engine, [
'add', 'vault', '--path', '/tmp/vault', '--exclude',
]));
expect(code).toBe(2);
});
test('--include rejects a flag-like value (--include --path looks like a typo)', async () => {
const { engine } = makeStub();
const code = await withExitCapture(() => runSources(engine, [
'add', 'vault', '--path', '/tmp/vault', '--include', '--federated',
]));
expect(code).toBe(2);
});
});
// ── add — #2707 git-repo validation (CLI wiring) ───────────────
-95
View File
@@ -1,95 +0,0 @@
/**
* #2157 follow-on (migration v125) end-to-end gate wiring.
*
* `test/sync-config-fingerprint.test.ts` pins the persistence + comparison
* primitives (compute/read/write). This file pins the WIRING inside
* `performSync`: with git HEAD unchanged, a drift in the walk-affecting
* `sources.config` fields must break out of the "Already up to date" early
* return and force a full re-walk and the re-stamped fingerprint must
* settle the gate back to `up_to_date` on the following pass. Deleting the
* `configMismatch` term from the gate condition fails this test; none of the
* primitive tests would catch that.
*/
import { test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { execFileSync } from 'child_process';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { performSync } from '../src/commands/sync.ts';
let engine: PGLiteEngine;
let repoPath: string;
function git(cwd: string, ...args: string[]) {
execFileSync('git', args, { cwd, stdio: 'pipe' });
}
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-fp-gate-'));
mkdirSync(join(repoPath, 'wiki'));
mkdirSync(join(repoPath, 'memory'));
writeFileSync(join(repoPath, 'wiki', 'page1.md'), '# Page 1\n\nbody\n');
writeFileSync(join(repoPath, 'memory', 'note1.md'), '# Note 1\n\nbody\n');
git(repoPath, 'init');
git(repoPath, 'add', '-A');
git(repoPath, '-c', 'user.email=t@example.com', '-c', 'user.name=t', 'commit', '-m', 'init');
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config) VALUES ($1, $2, $3, $4::text::jsonb)`,
['vault', 'vault', repoPath, JSON.stringify({ include_globs: ['wiki/**'] })],
);
}, 60_000);
afterAll(async () => {
await engine?.disconnect();
rmSync(repoPath, { recursive: true, force: true });
});
test('config-glob drift with unchanged git HEAD forces a re-walk, then settles', async () => {
// First sync: row config include_globs = ['wiki/**'], caller threads it
// (as syncOneSource / the single-source CLI path do). memory/* skipped.
const first = await performSync(engine, {
repoPath, sourceId: 'vault', include: ['wiki/**'],
noPull: true, noEmbed: true, full: true,
});
expect(first.status).toBe('first_sync');
expect(await engine.getPage('wiki/page1')).not.toBeNull();
expect(await engine.getPage('memory/note1')).toBeNull();
// No drift, HEAD unchanged: gate stays quiet.
const second = await performSync(engine, {
repoPath, sourceId: 'vault', include: ['wiki/**'],
noPull: true, noEmbed: true,
});
expect(second.status).toBe('up_to_date');
// User widens the persisted globs (what `gbrain sources add --include`
// writes). Git HEAD has NOT moved.
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify({ include_globs: ['wiki/**', 'memory/**'] }), 'vault'],
);
// Pre-fix this returned `up_to_date` (HEAD unchanged) and memory/note1
// stayed missing until a manual `--full`. The fingerprint gate must force
// the full re-walk instead.
const third = await performSync(engine, {
repoPath, sourceId: 'vault', include: ['wiki/**', 'memory/**'],
noPull: true, noEmbed: true,
});
expect(third.status).not.toBe('up_to_date');
expect(await engine.getPage('memory/note1')).not.toBeNull();
// Re-stamped fingerprint matches the current row: gate settles.
const fourth = await performSync(engine, {
repoPath, sourceId: 'vault', include: ['wiki/**', 'memory/**'],
noPull: true, noEmbed: true,
});
expect(fourth.status).toBe('up_to_date');
});
-234
View File
@@ -1,234 +0,0 @@
/**
* #2157 follow-on (migration v125 `sources.config_fingerprint`).
*
* The "Already up to date" gate at performSync's git-HEAD equality check
* honored chunker_version match but ignored `sources.config` drift.
* Changing `sources.config.exclude_globs` (or include_globs / strategy)
* had no observable effect on the next sync because git HEAD was
* unchanged the gate returned early and the new walk scope never
* applied. This file exercises the persistence shape + drift detection
* end-to-end on PGLite, including:
*
* - Migration v125 actually adds the column (regression guard against
* a future re-numbering or accidental deletion).
* - read/write round-trips preserve the value.
* - The fingerprint differs across the three walk-affecting fields
* and is order-insensitive on the array fields.
* - NULL fingerprint on pre-v125 rows treats as "not stamped" so a
* first post-upgrade sync doesn't spuriously force-full.
* - A toggle-and-revert leaves the stored fingerprint matching the
* current row, so the gate stays quiet.
*
* The wired-up gate behavior (force-full triggered on mismatch) is
* exercised by the existing sync end-to-end tests; here we pin the
* persistence + comparison primitives the gate depends on.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import {
computeSourceConfigFingerprint,
readConfigFingerprint,
writeConfigFingerprint,
} from '../src/commands/sync.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine?.disconnect();
});
/** Insert a fresh source row with the given config. Returns the id. */
async function makeSource(
id: string,
config: Record<string, unknown> = {},
): Promise<string> {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config) VALUES ($1, $2, $3, $4::text::jsonb)`,
[id, id, `/tmp/${id}`, JSON.stringify(config)],
);
return id;
}
describe('migration v125 — sources.config_fingerprint column', () => {
test('column exists on the sources table', async () => {
const rows = await engine.executeRaw<{ column_name: string }>(
`SELECT column_name FROM information_schema.columns
WHERE table_name = 'sources' AND column_name = 'config_fingerprint'`,
);
expect(rows).toHaveLength(1);
});
test('column is nullable (preserves pre-migration row semantics)', async () => {
const rows = await engine.executeRaw<{ is_nullable: string }>(
`SELECT is_nullable FROM information_schema.columns
WHERE table_name = 'sources' AND column_name = 'config_fingerprint'`,
);
expect(rows[0]?.is_nullable).toBe('YES');
});
});
describe('readConfigFingerprint / writeConfigFingerprint — persistence round-trip', () => {
test('round-trip: write then read returns the same value', async () => {
const id = await makeSource('rt-basic', { exclude_globs: ['Templates/**'] });
const fp = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'] });
await writeConfigFingerprint(engine, id, fp);
const got = await readConfigFingerprint(engine, id);
expect(got).toBe(fp);
});
test('NULL on never-stamped row (pre-v125 semantics)', async () => {
const id = await makeSource('rt-never');
const got = await readConfigFingerprint(engine, id);
expect(got).toBeNull();
});
test('undefined sourceId returns null (legacy non-source-scoped sync)', async () => {
const got = await readConfigFingerprint(engine, undefined);
expect(got).toBeNull();
});
test('write with undefined sourceId is a no-op (does not throw)', async () => {
// The legacy global-sync code path hits this branch; the guard must
// be silent rather than fail the sync run.
await writeConfigFingerprint(engine, undefined, 'deadbeef'.repeat(8));
// No assertion beyond "did not throw"; the function returns void.
});
test('overwrite: a second write replaces the prior fingerprint', async () => {
const id = await makeSource('rt-overwrite');
await writeConfigFingerprint(engine, id, 'a'.repeat(64));
await writeConfigFingerprint(engine, id, 'b'.repeat(64));
const got = await readConfigFingerprint(engine, id);
expect(got).toBe('b'.repeat(64));
});
});
describe('end-to-end drift simulation — the gate semantics this column enables', () => {
test('first stamp matches computed fingerprint of the row config', async () => {
const cfg = { exclude_globs: ['Templates/**', 'Photos/**'], strategy: 'markdown' };
const id = await makeSource('e2e-first-stamp', cfg);
const computed = computeSourceConfigFingerprint(cfg);
await writeConfigFingerprint(engine, id, computed);
expect(await readConfigFingerprint(engine, id)).toBe(computed);
});
test('exclude_globs mutation makes stored != current (drift detected)', async () => {
const before = { exclude_globs: ['Templates/**'] };
const after = { exclude_globs: ['Templates/**', 'Photos/**'] };
const id = await makeSource('e2e-exclude-drift', before);
const beforeFp = computeSourceConfigFingerprint(before);
await writeConfigFingerprint(engine, id, beforeFp);
// Simulate the user mutating sources.config via `gbrain sources add
// --exclude`. The gate's next read of (stored, computed-from-current)
// detects the drift and forces a re-walk.
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(after), id],
);
const afterFp = computeSourceConfigFingerprint(after);
const stored = await readConfigFingerprint(engine, id);
expect(stored).toBe(beforeFp);
expect(stored).not.toBe(afterFp);
});
test('toggle-and-revert: add then remove same pattern leaves stored matching current', async () => {
const original = { exclude_globs: ['Templates/**'] };
const id = await makeSource('e2e-toggle', original);
const originalFp = computeSourceConfigFingerprint(original);
await writeConfigFingerprint(engine, id, originalFp);
// Add a pattern (drift) then remove it (revert).
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify({ exclude_globs: ['Templates/**', 'Photos/**'] }), id],
);
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(original), id],
);
const revertedFp = computeSourceConfigFingerprint(original);
expect(revertedFp).toBe(originalFp);
expect(await readConfigFingerprint(engine, id)).toBe(originalFp);
// ⇒ Gate compares storedFp (==originalFp) to currentFp (==originalFp): no drift, no force-full.
});
test('include_globs drift detected independently', async () => {
const before = { include_globs: ['people/**'] };
const after = { include_globs: ['people/**', 'companies/**'] };
const id = await makeSource('e2e-include-drift', before);
await writeConfigFingerprint(engine, id, computeSourceConfigFingerprint(before));
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(after), id],
);
const stored = await readConfigFingerprint(engine, id);
const current = computeSourceConfigFingerprint(after);
expect(stored).not.toBe(current);
});
test('strategy drift detected', async () => {
const before = { strategy: 'markdown' };
const after = { strategy: 'code' };
const id = await makeSource('e2e-strategy-drift', before);
await writeConfigFingerprint(engine, id, computeSourceConfigFingerprint(before));
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(after), id],
);
expect(await readConfigFingerprint(engine, id))
.not.toBe(computeSourceConfigFingerprint(after));
});
test('mutating unrelated config field (federated) does NOT drift', async () => {
// The fingerprint hashes ONLY walk-affecting fields. Federation
// changes search visibility, not the walk set — must not invalidate
// the checkpoint.
const id = await makeSource('e2e-federated-toggle', {
federated: true,
exclude_globs: ['Templates/**'],
});
await writeConfigFingerprint(
engine,
id,
computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'] }),
);
await engine.executeRaw(
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[
JSON.stringify({ federated: false, exclude_globs: ['Templates/**'] }),
id,
],
);
const stored = await readConfigFingerprint(engine, id);
const current = computeSourceConfigFingerprint({
federated: false,
exclude_globs: ['Templates/**'],
});
expect(stored).toBe(current);
});
test('double-encoded JSONB config (the sources-add stringify bug) hashes equivalently to the parsed object', async () => {
// `gbrain sources add` writes `JSON.stringify(config)::jsonb`, which
// double-encodes the value into a JSON-string scalar (`"{\"x\":1}"`)
// rather than a proper JSONB object. The defensive reader in
// postgres-engine.ts:1274 + readSourceConfig parses the string back
// before the fingerprint sees it, so a double-encoded row and a
// properly-shaped row must fingerprint identically.
const cfg = { exclude_globs: ['Templates/**'], strategy: 'markdown' };
const direct = computeSourceConfigFingerprint(cfg);
// The pure compute fn handles a pre-parsed object; the persistence
// layer's job is to deliver a parsed object. We assert that the
// round-trip a real read would produce (parse the string scalar)
// hashes to the same value.
const parsed = JSON.parse(JSON.stringify(cfg));
expect(computeSourceConfigFingerprint(parsed)).toBe(direct);
});
});
-63
View File
@@ -355,69 +355,6 @@ describe('sync monorepo subdir-source support (#753/#774)', () => {
expect(await engine.getPage('wiki/draft-a')).toBeNull();
});
// ─────────────────────────────────────────────────────────────────────────
// --include: allow-list counterpart (#2156). Same scope-relative anchoring
// as --exclude; exclude applies after include.
// ─────────────────────────────────────────────────────────────────────────
test('--include: only matching files import on full sync', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const result = await performSync(engine, {
repoPath,
include: ['wiki/**'],
noPull: true,
noEmbed: true,
full: true,
});
expect(result.status).toBe('first_sync');
expect(result.added).toBe(2); // wiki/page1 + wiki/page2; memory/* miss the allow-list
expect(await engine.getPage('wiki/page1')).not.toBeNull();
expect(await engine.getPage('memory/note1')).toBeNull();
});
test('--include applies to the incremental path too', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const first = await performSync(engine, {
repoPath,
include: ['wiki/**'],
noPull: true,
noEmbed: true,
full: true,
});
expect(first.status).toBe('first_sync');
writeFileSync(join(repoPath, 'wiki', 'page3.md'), mdPage('Wiki Page 3'));
writeFileSync(join(repoPath, 'memory', 'note3.md'), mdPage('Memory Note 3'));
gitCommit(repoPath, 'more pages');
const second = await performSync(engine, {
repoPath,
include: ['wiki/**'],
noPull: true,
noEmbed: true,
});
expect(second.status).toBe('synced');
expect(second.added).toBe(1); // wiki/page3 only; memory/note3 misses the allow-list
expect(await engine.getPage('wiki/page3')).not.toBeNull();
expect(await engine.getPage('memory/note3')).toBeNull();
});
test('--exclude applies after --include (path in both is rejected)', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const result = await performSync(engine, {
repoPath,
include: ['wiki/**'],
exclude: ['wiki/page2.md'],
noPull: true,
noEmbed: true,
full: true,
});
expect(result.status).toBe('first_sync');
expect(result.added).toBe(1); // page1 only: page2 included then excluded
expect(await engine.getPage('wiki/page1')).not.toBeNull();
expect(await engine.getPage('wiki/page2')).toBeNull();
});
// ─────────────────────────────────────────────────────────────────────────
// --exclude '**/*' emits warning (NAV-4)
// ─────────────────────────────────────────────────────────────────────────
-212
View File
@@ -1,212 +0,0 @@
/**
* TODO #3 `parseGlobList` defensive parse.
*
* `sources.config` is a JSONB column with no schema. The runtime can find
* anything in `config.include_globs` / `config.exclude_globs`:
* - A user `gbrain sources add` wrote `["people/**"]` (the happy path).
* - A stray hand-edit wrote `"people/**"` (string, not array).
* - A future migration's null default.
* - A test fixture that left the column at `{}`.
*
* The parse must produce `string[] | undefined` so the downstream
* `SyncOpts.include` / `SyncOpts.exclude` are either undefined (no filter)
* or a non-empty list of usable globs. Returning `[]` would make
* `commands/sync.ts:1454` engage the filter loop with an empty allow-list
* that classifies every path as `include-glob-miss`.
*/
import { describe, test, expect } from 'bun:test';
import { parseGlobList, mergeGlobs, computeSourceConfigFingerprint } from '../src/commands/sync.ts';
describe('parseGlobList — JSONB-safe coercion to string[] | undefined', () => {
test('happy path: array of strings round-trips identically', () => {
expect(parseGlobList(['people/**', 'companies/**'])).toEqual(['people/**', 'companies/**']);
});
test('single-element array returned as-is', () => {
expect(parseGlobList(['Templates/**'])).toEqual(['Templates/**']);
});
test('non-array values return undefined (string, object, number, null)', () => {
expect(parseGlobList('Templates/**')).toBeUndefined();
expect(parseGlobList({ globs: ['Templates/**'] })).toBeUndefined();
expect(parseGlobList(42)).toBeUndefined();
expect(parseGlobList(null)).toBeUndefined();
expect(parseGlobList(undefined)).toBeUndefined();
});
test('empty array returns undefined (no engagement of the filter loop)', () => {
// Critical: a literal `[]` must not slip through. Empty `include` in
// SyncableOptions silently passes everything (good), but empty
// `exclude` is fine too — the real motivation is to keep `SyncOpts`
// unset so callers can ignore the field entirely. Symmetric with the
// `omitted glob flags leave config untouched` regression guard in
// sources.test.ts.
expect(parseGlobList([])).toBeUndefined();
});
test('mixed array drops non-string entries and keeps the rest', () => {
expect(parseGlobList(['people/**', 42, null, 'companies/**'])).toEqual([
'people/**',
'companies/**',
]);
});
test('empty strings dropped (a `""` glob would match every path)', () => {
expect(parseGlobList(['', 'people/**', ''])).toEqual(['people/**']);
});
test('array of only empty strings collapses to undefined', () => {
expect(parseGlobList(['', '', ''])).toBeUndefined();
});
});
describe('mergeGlobs — CLI flags union with persisted source-config globs', () => {
test('both sides present: union, deduped, CLI first', () => {
expect(mergeGlobs(['a/**', 'b/**'], ['b/**', 'c/**'])).toEqual(['a/**', 'b/**', 'c/**']);
});
test('CLI only', () => {
expect(mergeGlobs(['a/**'], undefined)).toEqual(['a/**']);
});
test('persisted only', () => {
expect(mergeGlobs([], ['Templates/**'])).toEqual(['Templates/**']);
});
test('neither side: undefined so SyncOpts stays unset', () => {
expect(mergeGlobs([], undefined)).toBeUndefined();
});
});
/**
* #2157 follow-on (sources_config_fingerprint, migration v125).
*
* computeSourceConfigFingerprint hashes the walk-affecting fields of
* sources.config (strategy + include_globs + exclude_globs) so the
* "Already up to date" gate at performSync's git-HEAD equality check
* can detect drift and force a re-walk. These cases pin the contract
* the gate depends on:
*
* - Deterministic over equivalent inputs (order-insensitive,
* defensively-coerced via parseGlobList).
* - Sensitive to each walk-affecting field separately.
* - Insensitive to fields the walker doesn't read (federated,
* unrelated keys).
* - A toggle-and-revert is a no-op (returns to the original hash).
*
* Without the canonicalization the gate would fire spuriously on
* cosmetic changes (e.g. a user re-ordering their exclude list) and
* miss real drift (e.g. an add-then-remove that nets to a different
* effective set than the stored fingerprint).
*/
describe('computeSourceConfigFingerprint — walk-affecting config drift detector', () => {
test('empty config produces a stable hash', () => {
const a = computeSourceConfigFingerprint({});
const b = computeSourceConfigFingerprint({});
expect(a).toBe(b);
expect(a).toMatch(/^[a-f0-9]{64}$/);
});
test('null / undefined / missing config all hash the same', () => {
const empty = computeSourceConfigFingerprint({});
expect(computeSourceConfigFingerprint(null)).toBe(empty);
expect(computeSourceConfigFingerprint(undefined)).toBe(empty);
});
test('same config → same hash (deterministic)', () => {
const cfg = { strategy: 'markdown', exclude_globs: ['Templates/**', 'Photos/**'] };
expect(computeSourceConfigFingerprint(cfg)).toBe(computeSourceConfigFingerprint(cfg));
});
test('array order does not affect hash (canonical sort)', () => {
const a = computeSourceConfigFingerprint({ exclude_globs: ['a/**', 'b/**', 'c/**'] });
const b = computeSourceConfigFingerprint({ exclude_globs: ['c/**', 'a/**', 'b/**'] });
expect(a).toBe(b);
});
test('exclude_globs change → different hash', () => {
const a = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'] });
const b = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**', 'Photos/**'] });
expect(a).not.toBe(b);
});
test('include_globs change → different hash', () => {
const a = computeSourceConfigFingerprint({ include_globs: ['people/**'] });
const b = computeSourceConfigFingerprint({ include_globs: ['people/**', 'companies/**'] });
expect(a).not.toBe(b);
});
test('strategy change → different hash', () => {
const a = computeSourceConfigFingerprint({ strategy: 'markdown' });
const b = computeSourceConfigFingerprint({ strategy: 'code' });
expect(a).not.toBe(b);
});
test('strategy unset vs set differ', () => {
const unset = computeSourceConfigFingerprint({});
const set = computeSourceConfigFingerprint({ strategy: 'markdown' });
expect(unset).not.toBe(set);
});
test('add-then-remove returns to original hash (toggle is a no-op)', () => {
const original = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'] });
const added = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**', 'Photos/**'] });
const reverted = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'] });
expect(added).not.toBe(original);
expect(reverted).toBe(original);
});
test('non-walk-affecting fields are ignored (federated, unrelated keys)', () => {
const a = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'], federated: true });
const b = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'], federated: false });
const c = computeSourceConfigFingerprint({ exclude_globs: ['Templates/**'], some_unrelated_key: 'value' });
expect(a).toBe(b);
expect(a).toBe(c);
});
test('non-string strategy coerced to null (defensive)', () => {
// A hand-edited row could leave `strategy: 42` or `strategy: {}` — both
// collapse to the same shape as `strategy: undefined` so the fingerprint
// doesn't reflect a value the walker can't honor anyway.
const empty = computeSourceConfigFingerprint({});
expect(computeSourceConfigFingerprint({ strategy: 42 })).toBe(empty);
expect(computeSourceConfigFingerprint({ strategy: {} })).toBe(empty);
expect(computeSourceConfigFingerprint({ strategy: null })).toBe(empty);
});
test('defensive parsing: mixed-type glob arrays hash same as cleaned arrays', () => {
// parseGlobList drops non-string + empty entries; the fingerprint must
// reflect what the walker actually uses, not what the raw row says.
const dirty = computeSourceConfigFingerprint({
exclude_globs: ['Templates/**', 42, null, '', 'Photos/**'],
});
const clean = computeSourceConfigFingerprint({
exclude_globs: ['Templates/**', 'Photos/**'],
});
expect(dirty).toBe(clean);
});
test('empty array and missing field hash identically', () => {
const missing = computeSourceConfigFingerprint({});
const emptyArray = computeSourceConfigFingerprint({ exclude_globs: [] });
const emptyAfterClean = computeSourceConfigFingerprint({ exclude_globs: ['', '', ''] });
expect(emptyArray).toBe(missing);
expect(emptyAfterClean).toBe(missing);
});
test('non-array exclude_globs (string, object) hash same as missing', () => {
const missing = computeSourceConfigFingerprint({});
expect(computeSourceConfigFingerprint({ exclude_globs: 'Templates/**' })).toBe(missing);
expect(computeSourceConfigFingerprint({ exclude_globs: { foo: 'bar' } })).toBe(missing);
});
test('SHA-256 output shape: 64 hex characters', () => {
const fp = computeSourceConfigFingerprint({
strategy: 'markdown',
include_globs: ['people/**'],
exclude_globs: ['Templates/**', '.git/**'],
});
expect(fp).toMatch(/^[a-f0-9]{64}$/);
});
});