diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 3bcd9861b..f548af4c2 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -5,6 +5,11 @@ on: branches: [master] pull_request: branches: [master] + # Manual dispatch lets a local dev/agent offload the suite to GitHub's + # on-demand runners from ANY branch (see scripts/ship-remote-tests.sh). + # Frees a load-saturated local machine (e.g. many Conductor agents running + # their own bun-test suites at once — load avg 120 on 16 cores). + workflow_dispatch: permissions: contents: read diff --git a/CHANGELOG.md b/CHANGELOG.md index 10af4533e..85a39a287 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,128 @@ All notable changes to GBrain will be documented in this file. +## [0.41.32.0] - 2026-05-30 + +**A quiet repo that's fully caught up no longer screams `SEVERELY STALE` in +`gbrain doctor`. Staleness now means "is there committed content the sync hasn't +ingested?" — not "how long has the wall clock been ticking since the last sync +ran."** + +If you keep a federated source that doesn't get a commit for days, the old check +kept escalating — 24h "stale", 72h "severely stale" — even though the sync had +everything the repo contained. It was pure wall-clock noise, and it trained you +to ignore the alert. Worse, a repo with a couple of stray untracked folders +(`companies/`, `media/`) tripped it even right after a sync, because the +freshness gate counted untracked files as "uncommitted work." + +Now the gate asks the right question. A source is caught up when its current +commit is the one the sync recorded (`HEAD == last_commit`) and there are no +uncommitted edits to *tracked* files. Untracked folders are ignored — they're +not part of the repo, and `gbrain sync` never imports them anyway. If that holds, +lag is `0` and the source reports clean, no matter how long ago the sync ran. + +### How to use it + +Nothing to turn on. Run `gbrain doctor` (or `gbrain sources status`) and a quiet, +caught-up source now shows fresh instead of stale. After `gbrain upgrade`, +migration v109 adds one column and the next `gbrain sync` starts populating it. + +### What you'd see + +| Scenario | Old metric | New metric | +|---|---|---| +| Quiet repo, caught up (newest commit predates last sync) | grows forever → SEVERELY STALE | **0 → fresh** | +| Repo with new commits the sync hasn't pulled | wall-clock | wall-clock → stale (correct) | +| HEAD force-pushed to an older-dated commit | could read "caught up" | **stale** (compares the commit hash, not its date) | +| Non-git path / never synced | wall-clock | wall-clock (unchanged) | +| Future `last_sync_at` (clock skew) | warns | warns (unchanged) | + +### Local vs remote, and the trust boundary we kept + +The local `gbrain doctor` (running on the machine that has your checkouts) reads +the live commit hash, so it always catches new commits the sync hasn't pulled — +your authoritative signal. The remote surfaces (`gbrain remote doctor`, +`federation_health`, and the `get_status_snapshot` MCP op) read a stored +`sources.newest_content_at` column written at sync time instead of running `git` +against a database-supplied path — preserving the v0.41.27.0 rule that a +remote-callable endpoint never shells out to a path an OAuth client could +influence. A `NULL` column falls back to wall-clock, so nothing regresses before +your next sync. + +### What we caught before merging + +An early version compared content *timestamps* (`newest content <= last sync`). +That's wrong when HEAD moves to a commit with an *older* author date — a rebase +that preserves dates, a branch rewind, an imported old commit — it would call a +genuinely-behind source "caught up." Switching to the commit *hash* fixes it and +also drops a fragile `git status` mtime-parsing path. The remote-path git +subprocess was gated back behind the trust boundary, and two regression tests +pin both: the headline untracked-folders case and the "remote path never shells +out to git" guarantee. + +### For contributors + +New `scripts/ship-remote-tests.sh` offloads the test suite to GitHub's on-demand +runners and blocks on the result with a real exit code (`gh run watch +--exit-status`). When your local machine is saturated (many agents running their +own `bun test` at once), run the gate in the cloud instead of fighting for CPU. +`test.yml` now also accepts `workflow_dispatch` so it can be triggered from any +branch. + +### To take advantage of v0.41.32.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain +doctor` warns about a partial migration: + +1. **Apply the migration:** + ```bash + gbrain apply-migrations --yes + ``` + This adds `sources.newest_content_at` (migration v109) — a metadata-only + column add, instant on any brain size. +2. **Run a sync so the column populates:** + ```bash + gbrain sync # or `gbrain sync --all` on a federated brain + ``` + Until a source syncs once post-upgrade, its remote staleness falls back to + the old wall-clock measure (the local doctor is already accurate via live git). +3. **Verify:** + ```bash + gbrain doctor --json | grep -A2 sync_freshness + gbrain sources status + ``` + A quiet, caught-up source should report `ok` / lag 0. +4. **If anything looks wrong,** file an issue with `gbrain doctor` output and the + contents of `~/.gbrain/upgrade-errors.jsonl` if it exists. + +### Itemized changes + +- **`src/core/git-head.ts`** — `isSourceUnchangedSinceSync`'s `requireCleanWorkingTree` + now accepts `'ignore-untracked'` (alongside `true`/`false`); the clean probe runs + `git status --porcelain --untracked-files=no` in that mode. This is the one-line + headline fix: untracked folders no longer defeat the freshness short-circuit. +- **`src/core/source-health.ts`** — new `newestCommitMs(localPath)` (HEAD committer + time via `git log -1 --format=%ct`, fail-open null) and pure `lagFromContentMs(contentMs, + lastSyncMs, nowMs)` comparator for the remote/column path; `computeAllSourceMetrics` + gains `{ probeContent }` (local opts into the live commit-hash probe, remote reads the + column). Dead `isSourceStale(src, intervalMs)` removed (only `autopilot-fanout.ts`'s + own variant was live). +- **`src/core/migrate.ts` v109** + `src/schema.sql` + `src/core/pglite-schema.ts` (+ + regenerated `schema-embedded.ts`) — `sources.newest_content_at TIMESTAMPTZ`. +- **`src/commands/sync.ts`** — `writeSyncAnchor` stamps `newest_content_at` (HEAD committer + time) in the same atomic UPDATE as `last_commit`/`last_sync_at`; `buildSyncStatusReport` + (the remote `get_status_snapshot` op) reads the column via `lagFromContentMs` — no git + subprocess. +- **`src/commands/doctor.ts`** — `checkSyncFreshness` short-circuit uses `'ignore-untracked'`; + the remote (non-`localOnly`) path computes lag from the stored column; the `< 0` clock-skew + check stays on raw wall-clock. +- **`src/commands/sources.ts`** — `gbrain sources status` opts into the live probe + (`probeContent: true`). +- **Tests** — `test/source-health.test.ts` (commit-hash caught-up incl. the old-dated-commit + regression, `lagFromContentMs` matrix, `newestCommitMs`, probeContent local/remote), + `test/doctor.test.ts` (T1 untracked-folders headline bug, T2 remote-never-shells-out trust + boundary), `test/sync-all-parallel.test.ts` (column-path staleness). +- **CI tooling** — `scripts/ship-remote-tests.sh` + `workflow_dispatch` on `test.yml`. ## [0.41.31.0] - 2026-05-30 **Your nightly `gbrain sync --all` cron stops getting blocked. It used to diff --git a/CLAUDE.md b/CLAUDE.md index 06718167f..6e90fcd78 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,7 +69,8 @@ strict behavior when unset. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. - `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). -- `src/core/git-head.ts` (v0.41.27.0) — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true, also requires the working tree to have no uncommitted changes (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) match the `last-retrieved.ts` precedent so unit tests stay R2-compliant (no `mock.module`). Uses `execFileSync` with array args — shell metachars in `local_path` cannot escape to a shell (the v0.41.27.0 superseded PR #1564 used `execSync` through `/bin/sh -c` with `JSON.stringify` for shell-escape, which is unsafe; the rebuild's regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel file is never created). Fail-open on every error: missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → returns false, preserving the caller's prior time-based behavior. Chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Designed for reuse: autopilot's per-source dispatch will want the same gate (filed as v0.41.27.1+ TODO in the plan). Pinned by `test/core/git-head.test.ts` (14 cases incl. the shell-injection regression guard). +- `src/core/git-head.ts` (v0.41.27.0) — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true, also requires the working tree to have no uncommitted changes (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) match the `last-retrieved.ts` precedent so unit tests stay R2-compliant (no `mock.module`). Uses `execFileSync` with array args — shell metachars in `local_path` cannot escape to a shell (the v0.41.27.0 superseded PR #1564 used `execSync` through `/bin/sh -c` with `JSON.stringify` for shell-escape, which is unsafe; the rebuild's regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel file is never created). Fail-open on every error: missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → returns false, preserving the caller's prior time-based behavior. Chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Designed for reuse: autopilot's per-source dispatch will want the same gate (filed as v0.41.27.1+ TODO in the plan). Pinned by `test/core/git-head.test.ts` (14 cases incl. the shell-injection regression guard). **v0.41.32.0 (supersedes #1623):** `GitFreshnessOpts.requireCleanWorkingTree` widened `boolean → boolean | 'ignore-untracked'`. In `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no`, so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" — sync's incremental path keys off the commit diff and never imports untracked files, so doctor agrees with sync. `GitCleanProbe` gains an `ignoreUntracked?` second arg. One-line headline fix for the false-SEVERE staleness bug. +- `src/core/source-health.ts` (v0.40, extended v0.41.32.0) — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. **v0.41.32.0 (supersedes #1623):** commit-relative staleness. New `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm). New pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash, catches HEAD moving to an old-dated commit which a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (v0.41.27.0 trust boundary). Dead `isSourceStale(src, intervalMs)` removed (only `autopilot-fanout.ts`'s own variant is live). Pinned by `test/source-health.test.ts`. - `src/core/git-remote.ts` (v0.35.3.0) — SSRF-hardened git invocations for remote-source `cloneRepo` and `pullRepo`. Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is **global config**, spread BEFORE the subcommand verb. New `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is **subcommand-scoped**, spread AFTER the verb. Pre-v0.35.3 a single combined `GIT_SSRF_FLAGS` array spread `--no-recurse-submodules` before the verb where real git rejects it with exit 129 ("unknown option"); the fake-git test harness exited 0 regardless of argv shape, so CI missed it for ~7 months and every remote-source clone/pull was silently broken. `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). - `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. @@ -262,8 +263,8 @@ strict behavior when unset. - `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). **v0.41.32.0 (supersedes #1623):** `checkSyncFreshness` short-circuit now passes `requireCleanWorkingTree: 'ignore-untracked'` (was `true`) — the headline fix: a quiet repo whose only dirt is untracked dirs is `unchanged`, not SEVERE. The SELECT widens to carry `newest_content_at`; the REMOTE (non-`localOnly`) path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column — NO git subprocess on a DB-supplied `local_path` (trust boundary intact). LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock (A1). `checkCycleFreshness` deliberately NOT content-relativized (CM2 — different axis, `last_full_cycle_at`; filed in TODOS as a probe-phase follow-up). Pinned by the "v0.41.32.0 — commit-relative staleness" describe in `test/doctor.test.ts` (T1 untracked-folders headline bug + T2/T2b/T2c remote-never-shells-out trust boundary). +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/TODOS.md b/TODOS.md index c04699441..d9f6b6af4 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,38 @@ # TODOS +## v0.41.32.0 content-relative staleness follow-ups (v0.42+) + +Filed from the v0.41.32.0 wave (supersedes #1623 — commit-relative sync +staleness). The wave fixes the LOCAL doctor/sources false-SEVERE and the +REMOTE surfaces via a durable `sources.newest_content_at` column. Two gaps +were deliberately scoped out (CM2 + the remote post-sync-divergence residual). + +- [ ] **v0.42+: lightweight local content-probe phase to keep `newest_content_at` fresh between syncs.** + - **What:** an autopilot/cron phase that, for each git-backed source, runs the + cheap `git log -1 --format=%ct` (HEAD committer time) and refreshes + `sources.newest_content_at` even when there's nothing to sync. + - **Why:** the REMOTE staleness path (`doctorReportRemote`'s `checkSyncFreshness`, + `federation_health`, the `get_status_snapshot` MCP op) reads the column and + cannot shell out to git (v0.41.27.0 trust boundary). The column is written at + sync time, so a commit landed AFTER the last sync is invisible to the remote + path until the next sync rewrites it — a narrow false-negative window. The + authoritative LOCAL cron doctor catches those (it probes live git), so this is + a remote-only freshness improvement, not a correctness hole. + - **Pros:** shrinks the remote false-negative window to the probe cadence; + keeps the trust boundary intact (probe runs on the trusted host, not from a + remote caller). + - **Cons:** a new background phase + its own tests + a cadence knob; only + matters for operators who rely on `gbrain remote doctor` instead of the local + cron doctor. + - **Context:** the helper already exists — `newestCommitMs(localPath)` in + `src/core/source-health.ts`. The phase just calls it per source and UPDATEs + the column. See the v0.41.32.0 plan at + `~/.claude/plans/system-instruction-you-are-working-vivid-gizmo.md`. + - **Also note:** `checkCycleFreshness` was deliberately left on wall-clock in + v0.41.32.0 (CM2 — it compares `last_full_cycle_at` via `listAllSources`, a + different axis from sync staleness). Content-relativizing it (a source whose + newest commit predates its last full cycle doesn't need re-cycling) is a + natural companion to this probe phase. Priority: P3. ## brainstorm/lsd --save source-awareness (v0.42+) Filed from the `--save` dual-sink hardening wave (route through the canonical diff --git a/VERSION b/VERSION index c9ccfb296..023ad4f0e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.31.0 \ No newline at end of file +0.41.32.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 9b7db17d8..00136cee8 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -211,7 +211,8 @@ strict behavior when unset. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. - `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). -- `src/core/git-head.ts` (v0.41.27.0) — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true, also requires the working tree to have no uncommitted changes (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) match the `last-retrieved.ts` precedent so unit tests stay R2-compliant (no `mock.module`). Uses `execFileSync` with array args — shell metachars in `local_path` cannot escape to a shell (the v0.41.27.0 superseded PR #1564 used `execSync` through `/bin/sh -c` with `JSON.stringify` for shell-escape, which is unsafe; the rebuild's regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel file is never created). Fail-open on every error: missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → returns false, preserving the caller's prior time-based behavior. Chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Designed for reuse: autopilot's per-source dispatch will want the same gate (filed as v0.41.27.1+ TODO in the plan). Pinned by `test/core/git-head.test.ts` (14 cases incl. the shell-injection regression guard). +- `src/core/git-head.ts` (v0.41.27.0) — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true, also requires the working tree to have no uncommitted changes (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) match the `last-retrieved.ts` precedent so unit tests stay R2-compliant (no `mock.module`). Uses `execFileSync` with array args — shell metachars in `local_path` cannot escape to a shell (the v0.41.27.0 superseded PR #1564 used `execSync` through `/bin/sh -c` with `JSON.stringify` for shell-escape, which is unsafe; the rebuild's regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel file is never created). Fail-open on every error: missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → returns false, preserving the caller's prior time-based behavior. Chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Designed for reuse: autopilot's per-source dispatch will want the same gate (filed as v0.41.27.1+ TODO in the plan). Pinned by `test/core/git-head.test.ts` (14 cases incl. the shell-injection regression guard). **v0.41.32.0 (supersedes #1623):** `GitFreshnessOpts.requireCleanWorkingTree` widened `boolean → boolean | 'ignore-untracked'`. In `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no`, so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" — sync's incremental path keys off the commit diff and never imports untracked files, so doctor agrees with sync. `GitCleanProbe` gains an `ignoreUntracked?` second arg. One-line headline fix for the false-SEVERE staleness bug. +- `src/core/source-health.ts` (v0.40, extended v0.41.32.0) — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. **v0.41.32.0 (supersedes #1623):** commit-relative staleness. New `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm). New pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash, catches HEAD moving to an old-dated commit which a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (v0.41.27.0 trust boundary). Dead `isSourceStale(src, intervalMs)` removed (only `autopilot-fanout.ts`'s own variant is live). Pinned by `test/source-health.test.ts`. - `src/core/git-remote.ts` (v0.35.3.0) — SSRF-hardened git invocations for remote-source `cloneRepo` and `pullRepo`. Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is **global config**, spread BEFORE the subcommand verb. New `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is **subcommand-scoped**, spread AFTER the verb. Pre-v0.35.3 a single combined `GIT_SSRF_FLAGS` array spread `--no-recurse-submodules` before the verb where real git rejects it with exit 129 ("unknown option"); the fake-git test harness exited 0 regardless of argv shape, so CI missed it for ~7 months and every remote-source clone/pull was silently broken. `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). - `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. @@ -404,8 +405,8 @@ strict behavior when unset. - `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). **v0.41.32.0 (supersedes #1623):** `checkSyncFreshness` short-circuit now passes `requireCleanWorkingTree: 'ignore-untracked'` (was `true`) — the headline fix: a quiet repo whose only dirt is untracked dirs is `unchanged`, not SEVERE. The SELECT widens to carry `newest_content_at`; the REMOTE (non-`localOnly`) path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column — NO git subprocess on a DB-supplied `local_path` (trust boundary intact). LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock (A1). `checkCycleFreshness` deliberately NOT content-relativized (CM2 — different axis, `last_full_cycle_at`; filed in TODOS as a probe-phase follow-up). Pinned by the "v0.41.32.0 — commit-relative staleness" describe in `test/doctor.test.ts` (T1 untracked-folders headline bug + T2/T2b/T2c remote-never-shells-out trust boundary). +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/package.json b/package.json index fb5a56b40..8e9e134d9 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.31.0" + "version": "0.41.32.0" } diff --git a/scripts/ship-remote-tests.sh b/scripts/ship-remote-tests.sh new file mode 100755 index 000000000..bfd4d83d5 --- /dev/null +++ b/scripts/ship-remote-tests.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# ship-remote-tests.sh — run the unit suite on GitHub's on-demand cloud +# runners instead of locally, and block until it finishes with a real +# pass/fail exit code. +# +# WHY: a local machine running many Conductor agents at once gets CPU/memory +# saturated (observed: load avg 120 on 16 cores, ~15 sibling `bun test` +# processes). The PGLite WASM test suite then OOMs (8-shard) or crawls +# (~12min for 1/3 of files vs ~85s normally). The suite already runs on +# GitHub's ephemeral runners on every PR push; this script makes a local +# caller (human or agent, e.g. /ship Step 5) AWAIT that cloud run exactly +# like a local `bun run test` — push, dispatch, `gh run watch --exit-status`. +# +# USAGE: +# scripts/ship-remote-tests.sh [--workflow test.yml] [--branch ] +# [--no-push] [--ref ] +# +# EXIT: mirrors the GitHub run — 0 on success, non-zero on failure (so it +# drops into a test gate unchanged). 2 = usage/precondition error. +# +# REQUIRES: `gh` authenticated; the workflow must declare `workflow_dispatch:` +# (test.yml does as of v0.41.32.0). +set -euo pipefail + +WORKFLOW="test.yml" +BRANCH="" +DO_PUSH=1 +REF="" + +while [ $# -gt 0 ]; do + case "$1" in + --workflow) WORKFLOW="$2"; shift 2 ;; + --branch) BRANCH="$2"; shift 2 ;; + --ref) REF="$2"; shift 2 ;; + --no-push) DO_PUSH=0; shift ;; + -h|--help) + sed -n '2,30p' "$0"; exit 0 ;; + *) echo "ship-remote-tests: unknown arg '$1'" >&2; exit 2 ;; + esac +done + +command -v gh >/dev/null 2>&1 || { echo "ship-remote-tests: gh CLI not found" >&2; exit 2; } +gh auth status >/dev/null 2>&1 || { echo "ship-remote-tests: gh not authenticated — run 'gh auth login'" >&2; exit 2; } + +[ -n "$BRANCH" ] || BRANCH="$(git branch --show-current 2>/dev/null || true)" +[ -n "$BRANCH" ] || { echo "ship-remote-tests: could not determine branch (detached HEAD?) — pass --branch" >&2; exit 2; } + +if [ "$DO_PUSH" = "1" ]; then + echo "ship-remote-tests: pushing $BRANCH ..." >&2 + git push -u origin "$BRANCH" +fi + +# Dispatch against the branch (or an explicit ref). Requires workflow_dispatch +# on the workflow. The HEAD sha lets us disambiguate OUR run from any +# concurrent pull_request run on the same branch. +HEAD_SHA="$(git rev-parse "${REF:-HEAD}")" +echo "ship-remote-tests: dispatching $WORKFLOW on $BRANCH @ ${HEAD_SHA:0:8} ..." >&2 +gh workflow run "$WORKFLOW" --ref "${REF:-$BRANCH}" >/dev/null + +# Poll for the dispatched run to register (cli/cli#8194: `gh run watch` can +# skip a not-yet-registered run, so we resolve the databaseId ourselves first). +RUN_ID="" +for _ in $(seq 1 30); do + RUN_ID="$(gh run list --workflow "$WORKFLOW" --branch "$BRANCH" \ + --event workflow_dispatch --limit 10 \ + --json databaseId,headSha,status \ + -q "[.[] | select(.headSha==\"$HEAD_SHA\")] | sort_by(.databaseId) | last | .databaseId" 2>/dev/null || true)" + [ -n "$RUN_ID" ] && [ "$RUN_ID" != "null" ] && break + sleep 3 +done + +if [ -z "$RUN_ID" ] || [ "$RUN_ID" = "null" ]; then + echo "ship-remote-tests: could not find the dispatched run after 90s." >&2 + echo " Check manually: gh run list --workflow $WORKFLOW --branch $BRANCH" >&2 + exit 2 +fi + +RUN_URL="$(gh run view "$RUN_ID" --json url -q .url 2>/dev/null || echo "")" +echo "ship-remote-tests: watching run $RUN_ID $RUN_URL" >&2 + +# Block until the cloud run finishes; mirror its pass/fail as our exit code. +if gh run watch "$RUN_ID" --exit-status; then + echo "ship-remote-tests: PASS $RUN_URL" >&2 + exit 0 +else + rc=$? + echo "ship-remote-tests: FAIL (exit $rc) $RUN_URL" >&2 + echo "--- failed logs ---" >&2 + gh run view "$RUN_ID" --log-failed 2>/dev/null | tail -120 >&2 || true + exit "$rc" +fi diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index ec1085254..f51216464 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -28,6 +28,9 @@ import { dirname, isAbsolute, join, resolve as resolvePath } from 'path'; import { fileURLToPath } from 'url'; import { existsSync, readFileSync, readdirSync, statSync } from 'fs'; import { isSourceUnchangedSinceSync } from '../core/git-head.ts'; +// v0.41.32.0: remote staleness reads the stored newest_content_at column via +// this pure comparator (no git subprocess on the HTTP MCP doctor path). +import { lagFromContentMs } from '../core/source-health.ts'; import { CHUNKER_VERSION } from '../core/chunkers/code.ts'; export interface Check { @@ -2710,8 +2713,11 @@ export async function checkSyncFreshness( last_sync_at: Date | null; last_commit: string | null; chunker_version: string | null; + newest_content_at: Date | null; }>( - `SELECT id, name, local_path, last_sync_at, last_commit, chunker_version FROM sources WHERE local_path IS NOT NULL`, + // v0.41.32.0: newest_content_at feeds the REMOTE (non-localOnly) lag so + // doctorReportRemote never shells out to git on a DB-supplied local_path. + `SELECT id, name, local_path, last_sync_at, last_commit, chunker_version, newest_content_at FROM sources WHERE local_path IS NOT NULL`, ); if (sources.length === 0) { @@ -2791,8 +2797,14 @@ export async function checkSyncFreshness( // v0.41.27.0: git short-circuit (D4 + D7 combined). Only fires when: // 1. caller opted in via localOnly=true (trust boundary) // 2. HEAD === last_commit (no new commits to sync) - // 3. working tree is clean (no uncommitted edits sync would re-walk) - // 4. chunker_version matches CURRENT (no post-upgrade re-chunk pending) + // 3. working tree has no TRACKED changes — untracked files ignored + // (v0.41.32.0: `'ignore-untracked'`. Sync's incremental path keys off + // the commit diff and never imports untracked files, so a quiet repo + // with stray untracked dirs is genuinely caught up. The pre-v0.41.30 + // `true` mode counted those as dirty and produced the false-SEVERE + // alarm this wave fixes.) + // 4. chunker_version matches CURRENT (no post-upgrade re-chunk pending — + // still ANDed, so a re-chunk need is never masked) // All four must hold; otherwise fall through to the time-based check. // The chunker version match is computed here (not in the helper) // because it depends on engine state, not git state. @@ -2800,7 +2812,7 @@ export async function checkSyncFreshness( const gitUnchanged = isSourceUnchangedSinceSync( source.local_path, source.last_commit, - { requireCleanWorkingTree: true }, + { requireCleanWorkingTree: 'ignore-untracked' }, ); const chunkerMatch = source.chunker_version === currentChunkerVersion; if (gitUnchanged && chunkerMatch) { @@ -2809,14 +2821,35 @@ export async function checkSyncFreshness( } } - const ageHours = Math.floor(ageMs / (1000 * 60 * 60)); + // v0.41.32.0: REMOTE path (doctorReportRemote, !localOnly) computes lag + // from the stored newest_content_at column — NO git subprocess on a + // DB-supplied local_path (preserves the v0.41.27.0 trust boundary). A + // quiet repo whose newest commit predates its last sync reports 0; NULL + // column → wall-clock fallback. LOCAL fall-through keeps wall-clock: the + // short-circuit already failed, so the source genuinely has work and + // "hours since last sync" is the right staleness measure. The `ageMs < 0` + // skew check above still runs on raw wall-clock for both paths (A1). + let thresholdAgeMs = ageMs; + if (!localOnly) { + const contentMs = source.newest_content_at + ? new Date(source.newest_content_at).getTime() + : null; + const lagSec = lagFromContentMs( + contentMs !== null && Number.isFinite(contentMs) ? contentMs : null, + lastSync, + now, + ); + thresholdAgeMs = lagSec === null ? ageMs : lagSec * 1000; + } + + const ageHours = Math.floor(thresholdAgeMs / (1000 * 60 * 60)); const ageDays = Math.floor(ageHours / 24); - if (ageMs > failMs) { + if (thresholdAgeMs > failMs) { issues.push(`Source ${display} last synced ${ageDays}d ago — brain search is stale!`); hasFailures = true; stale_count++; - } else if (ageMs > warnMs) { + } else if (thresholdAgeMs > warnMs) { issues.push(`Source ${display} last synced ${ageHours}h ago`); hasWarnings = true; stale_count++; diff --git a/src/commands/sources.ts b/src/commands/sources.ts index e7620048b..905150cb9 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -592,7 +592,9 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise { } return; } - const metrics = await computeAllSourceMetrics(engine, sources); + // Local CLI on the trusted host: probe the live commit hash so a quiet, + // caught-up source reports lag 0 instead of growing wall-clock (v0.41.32.0). + const metrics = await computeAllSourceMetrics(engine, sources, { probeContent: true }); if (json) { console.log(JSON.stringify({ schema_version: 1, sources: metrics }, null, 2)); diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 41244e5d8..c6db521bf 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -54,6 +54,11 @@ import { } from '../core/console-prefix.ts'; import { loadStorageConfig } from '../core/storage-config.ts'; import { getDefaultSourcePath } from '../core/source-resolver.ts'; +// v0.41.32.0: stamp the durable newest-COMMIT timestamp at sync time so the +// remote staleness path reads a column instead of shelling out to git. +// lagFromContentMs is the remote/column comparator (buildSyncStatusReport +// backs the get_status_snapshot MCP op — must NOT shell out to git). +import { newestCommitMs, lagFromContentMs } from '../core/source-health.ts'; import { sortNewestFirst } from '../core/sort-newest-first.ts'; export interface SyncResult { @@ -460,15 +465,32 @@ async function writeSyncAnchor( sourceId: string | undefined, which: 'repo_path' | 'last_commit', value: string, + // v0.41.32.0 (supersedes #1623): on `last_commit` advances, also stamp the + // durable newest-COMMIT timestamp (HEAD committer time, epoch ms) in the SAME + // atomic UPDATE as last_sync_at — no separate write to leave partial state, + // no clock-domain split (last_sync_at = DB now(); newest_content_at = the + // git-intrinsic committer time of the HEAD we just synced). `undefined` keeps + // the legacy 2-column write; `null` clears the column (git unavailable). + newestContentEpochMs?: number | null, ): Promise { if (sourceId) { const col = which === 'repo_path' ? 'local_path' : 'last_commit'; // last_sync_at bookmarked on every last_commit advance. if (which === 'last_commit') { - await engine.executeRaw( - `UPDATE sources SET last_commit = $1, last_sync_at = now() WHERE id = $2`, - [value, sourceId], - ); + if (newestContentEpochMs !== undefined) { + const iso = newestContentEpochMs === null + ? null + : new Date(newestContentEpochMs).toISOString(); + await engine.executeRaw( + `UPDATE sources SET last_commit = $1, last_sync_at = now(), newest_content_at = $3 WHERE id = $2`, + [value, sourceId, iso], + ); + } else { + await engine.executeRaw( + `UPDATE sources SET last_commit = $1, last_sync_at = now() WHERE id = $2`, + [value, sourceId], + ); + } } else { await engine.executeRaw( `UPDATE sources SET ${col} = $1 WHERE id = $2`, @@ -477,6 +499,9 @@ async function writeSyncAnchor( } return; } + // Legacy no-sourceId path (pre-v0.18 global config). Modern sync always + // resolves a sourceId (incl. 'default'), so newest_content_at is written via + // the sourceId branch above; the default source is not stuck on NULL. await engine.setConfig(`sync.${which}`, value); } @@ -1253,7 +1278,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise( - `SELECT id, last_commit, last_sync_at FROM sources WHERE id = ANY($1::text[])`, + `SELECT id, last_commit, last_sync_at, newest_content_at FROM sources WHERE id = ANY($1::text[])`, [sourceIds], ); const sourceMap = new Map(); @@ -3085,14 +3112,24 @@ export async function buildSyncStatusReport( const now = Date.now(); const out: SyncStatusReportSource[] = sources.map((src) => { const cfgEntry = (src.config || {}) as { syncEnabled?: boolean }; - const row = sourceMap.get(src.id) || { id: src.id, last_commit: null, last_sync_at: null }; + const row = sourceMap.get(src.id) || { id: src.id, last_commit: null, last_sync_at: null, newest_content_at: null }; const counts = countMap.get(src.id) || { pages: 0, chunks_total: 0, chunks_unembedded: 0 }; const lastSyncMs = row.last_sync_at ? (row.last_sync_at instanceof Date ? row.last_sync_at.getTime() : Date.parse(row.last_sync_at)) : null; - const stalenessHours = lastSyncMs !== null && Number.isFinite(lastSyncMs) - ? (now - lastSyncMs) / 3_600_000 + // v0.41.32.0: commit-relative staleness from the stored column — NO git + // subprocess (this function backs the remote get_status_snapshot MCP op, + // so it must honor the v0.41.27.0 trust boundary). A quiet repo whose + // newest commit predates its last sync reports 0; null column → wall-clock. + const contentMs = row.newest_content_at + ? (row.newest_content_at instanceof Date ? row.newest_content_at.getTime() : Date.parse(row.newest_content_at)) : null; + const lagSeconds = lagFromContentMs( + Number.isFinite(contentMs as number) ? (contentMs as number) : null, + lastSyncMs !== null && Number.isFinite(lastSyncMs) ? lastSyncMs : null, + now, + ); + const stalenessHours = lagSeconds === null ? null : lagSeconds / 3600; let stalenessClass: 'fresh' | 'stale' | 'severe' | 'unknown' = 'unknown'; if (stalenessHours !== null) { if (stalenessHours < 24) stalenessClass = 'fresh'; diff --git a/src/core/git-head.ts b/src/core/git-head.ts index 1e43b48bc..94955e79a 100644 --- a/src/core/git-head.ts +++ b/src/core/git-head.ts @@ -27,7 +27,11 @@ import { execFileSync } from 'node:child_process'; export type GitHeadProbe = (localPath: string) => string | null; // `null` distinguishes probe error from known-dirty (false). Doctor treats // both as "do not short-circuit", but tests need to assert which path fired. -export type GitCleanProbe = (localPath: string) => boolean | null; +// `ignoreUntracked` (v0.41.32.0): when true, untracked files (`git status` +// `??` rows) do NOT count as dirty — they are not part of the repo and sync's +// incremental path (commit-diff at sync.ts:1057) never imports them, so a +// quiet repo with stray untracked dirs is still "unchanged". +export type GitCleanProbe = (localPath: string, ignoreUntracked?: boolean) => boolean | null; const DEFAULT_HEAD_PROBE: GitHeadProbe = (localPath) => { try { @@ -42,9 +46,16 @@ const DEFAULT_HEAD_PROBE: GitHeadProbe = (localPath) => { } }; -const DEFAULT_CLEAN_PROBE: GitCleanProbe = (localPath) => { +const DEFAULT_CLEAN_PROBE: GitCleanProbe = (localPath, ignoreUntracked) => { try { - const out = execFileSync('git', ['-C', localPath, 'status', '--porcelain'], { + // `--untracked-files=no` makes `git status --porcelain` emit ONLY tracked + // changes. Empty output then means "clean ignoring untracked." This is the + // v0.41.32.0 fix for the false-SEVERE bug: untracked dirs (`?? companies/`, + // `?? media/`) on an otherwise-caught-up repo previously made the tree look + // dirty and defeated the short-circuit. + const args = ['-C', localPath, 'status', '--porcelain']; + if (ignoreUntracked) args.push('--untracked-files=no'); + const out = execFileSync('git', args, { encoding: 'utf8', timeout: 5000, stdio: ['ignore', 'pipe', 'ignore'], @@ -71,15 +82,17 @@ export function _setGitCleanProbeForTests(fn: GitCleanProbe | null): void { export interface GitFreshnessOpts { /** - * When true, additionally require working tree to be clean (no - * uncommitted changes). Doctor uses this to mirror `gbrain sync`'s - * working-tree-dirty gate at sync.ts:1075 — otherwise doctor would - * say "unchanged" for a repo with pending local edits that sync would - * actually re-walk on the next run. - * - * Default false (HEAD comparison only). Doctor-callers set true. + * Working-tree cleanliness requirement on top of the HEAD==lastCommit check: + * - `false`/omitted: HEAD comparison only. + * - `true`: require a fully clean tree (tracked AND untracked) — the + * v0.41.27.0 posture mirroring `gbrain sync`'s gate at sync.ts:1075. + * - `'ignore-untracked'` (v0.41.32.0): require no TRACKED changes but allow + * untracked files. This is what doctor/sources should use: sync's + * incremental path keys off the commit diff and never imports untracked + * files, so a quiet repo with stray untracked dirs is genuinely caught up. + * Fixes the false-SEVERE bug without weakening the commit-hash gate. */ - requireCleanWorkingTree?: boolean; + requireCleanWorkingTree?: boolean | 'ignore-untracked'; } /** @@ -102,7 +115,8 @@ export function isSourceUnchangedSinceSync( const head = _headProbe(localPath); if (head === null || head !== lastCommit) return false; if (opts?.requireCleanWorkingTree) { - const isClean = _cleanProbe(localPath); + const ignoreUntracked = opts.requireCleanWorkingTree === 'ignore-untracked'; + const isClean = _cleanProbe(localPath, ignoreUntracked); // null (probe error) AND false (known dirty) both fail the gate. if (isClean !== true) return false; } diff --git a/src/core/migrate.ts b/src/core/migrate.ts index b326ef980..dfd23b177 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4946,6 +4946,22 @@ export const MIGRATIONS: Migration[] = [ ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT NULL; `, }, + { + version: 109, + name: 'sources_newest_content_at', + // v0.41.32.0 (supersedes #1623): durable newest-COMMIT timestamp per source, + // written at sync time (HEAD committer time). The REMOTE staleness path + // (federation_health, get_status_snapshot MCP op) reads this column instead + // of shelling out to git on a DB-supplied local_path — preserving the + // v0.41.27.0 trust boundary while still killing the quiet-repo false-SEVERE + // alarm. ADD COLUMN with a NULL default is metadata-only on both engines + // (instant, no table rewrite). Mirror lives in pglite-schema.ts + + // schema.sql (fresh-install path) and the applyForwardReferenceBootstrap + // probe set in both engines. Renumbered 108→109 on the master merge that + // landed v0.41.31's pages_embedding_signature at v108. + idempotent: true, + sql: `ALTER TABLE sources ADD COLUMN IF NOT EXISTS newest_content_at TIMESTAMPTZ`, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 07c5c5714..37bd89613 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -49,6 +49,9 @@ CREATE TABLE IF NOT EXISTS sources ( -- FALSE for mounts by default; host is always trusted regardless. contextual_retrieval_mode TEXT, trust_frontmatter_overrides BOOLEAN NOT NULL DEFAULT false, + -- v0.41.32.0 (supersedes #1623): newest COMMIT timestamp at last sync + -- (mirrors src/schema.sql). REMOTE staleness reads this; NULL → wall-clock. + newest_content_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index b48847389..e0d7cdedd 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -53,6 +53,11 @@ CREATE TABLE IF NOT EXISTS sources ( -- (id='default') is always trusted regardless of this column. contextual_retrieval_mode TEXT, trust_frontmatter_overrides BOOLEAN NOT NULL DEFAULT false, + -- v0.41.32.0 (supersedes #1623): newest COMMIT timestamp (HEAD committer + -- time) recorded at last sync. The REMOTE staleness path reads this instead + -- of shelling out to git on a DB-supplied local_path, preserving the + -- v0.41.27.0 trust boundary. NULL → reader falls back to wall-clock. + newest_content_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); diff --git a/src/core/source-health.ts b/src/core/source-health.ts index 8553a8358..b721a3d3d 100644 --- a/src/core/source-health.ts +++ b/src/core/source-health.ts @@ -12,11 +12,19 @@ * D9: resolvePriority(config) — accepts 'high'|'normal'|'low', falls back * to 0 with once-per-source-per-process stderr warn on unknown values. * - * D17: isSourceStale helper — autopilot calls this to decide per-source - * sync dispatch independent of the brain_score gate. + * v0.41.32.0: commit-relative staleness. `lag_seconds` is no longer raw + * wall-clock `now - last_sync_at` (which false-flagged quiet, caught-up + * repos as SEVERE). Local callers pass `probeContent: true` and lag + * becomes 0 when the source is caught up by COMMIT HASH (HEAD == + * last_commit, untracked ignored, via `isSourceUnchangedSinceSync`). + * Remote callers (federation_health on the HTTP MCP path) read the stored + * `newest_content_at` column instead — NO git subprocess on a DB-supplied + * local_path (preserves the v0.41.27.0 trust boundary). */ +import { execFileSync } from 'child_process'; import type { BrainEngine } from './engine.ts'; import { parseSourceConfig, type SourceRow } from './sources-load.ts'; +import { isSourceUnchangedSinceSync } from './git-head.ts'; export interface SourceMetrics { source_id: string; @@ -97,15 +105,60 @@ export function resolvePriority(sourceId: string, config: unknown): number { } /** - * True iff the source's last_sync_at is older than `intervalMs`, OR it has - * never synced. Sources without a local_path are NOT considered stale (no - * way to sync them). Used by autopilot D17 freshness gate. + * Newest COMMIT timestamp for a source's checkout, in epoch ms, or `null` when + * not determinable cheaply (non-git path, git unavailable, timeout). This is + * the HEAD committer time (`git log -1 --format=%ct`) — NOT working-tree mtimes + * (untracked/tracked-uncommitted files are not "committed content," and parsing + * `git status --porcelain` for mtimes is fragile). Used at sync time to populate + * the durable `sources.newest_content_at` column that the REMOTE staleness path + * reads (so the HTTP MCP doctor never shells out to git). + * + * Fail-open: every error path returns `null`; the caller stores NULL and the + * remote reader falls back to wall-clock. Shell-injection-safe (execFileSync + * array args), matching the git-head.ts posture. */ -export function isSourceStale(src: SourceRow, intervalMs: number): boolean { - if (!src.local_path) return false; - if (!src.last_sync_at) return true; - const lastMs = new Date(src.last_sync_at).getTime(); - return Date.now() - lastMs >= intervalMs; +export function newestCommitMs(localPath: string | null): number | null { + if (!localPath) return null; + try { + const out = execFileSync('git', ['-C', localPath, 'log', '-1', '--format=%ct'], { + encoding: 'utf8', + timeout: 10_000, + stdio: ['ignore', 'pipe', 'ignore'], + }).trim(); + if (!out) return null; + const ms = Number(out) * 1000; + return Number.isFinite(ms) ? ms : null; + } catch { + return null; // not a git repo / git unavailable / timeout + } +} + +/** + * Commit-relative lag in seconds from a STORED content timestamp (the + * `newest_content_at` column), for REMOTE consumers that cannot shell out: + * - `null` when `lastSyncMs` is unknown. + * - Negative wall-clock (future `last_sync_at`) is surfaced as-is so upstream + * clock-skew detection still fires. + * - `0` when the stored content is at or before the last sync (caught up). + * - Wall-clock `now - lastSync` when content is newer, or when `contentMs` is + * null (no column value / pre-migration) — detection never regresses. + * + * Pure. The LOCAL path does NOT use this — it keys off the live commit hash via + * `isSourceUnchangedSinceSync` (robust against HEAD moving to an old-dated + * commit, which a timestamp comparison would miss). + */ +export function lagFromContentMs( + contentMs: number | null, + lastSyncMs: number | null, + nowMs: number, +): number | null { + if (lastSyncMs === null || !Number.isFinite(lastSyncMs)) return null; + const wallClockSeconds = Math.floor((nowMs - lastSyncMs) / 1000); + if (wallClockSeconds < 0) return wallClockSeconds; // clock skew passthrough + if (contentMs !== null && Number.isFinite(contentMs)) { + return contentMs <= lastSyncMs ? 0 : wallClockSeconds; + } + return wallClockSeconds; // no stored content signal — wall-clock fallback } /** @@ -123,6 +176,7 @@ export function isSourceStale(src: SourceRow, intervalMs: number): boolean { export async function computeAllSourceMetrics( engine: BrainEngine, sources: SourceRow[], + opts?: { probeContent?: boolean }, ): Promise { if (sources.length === 0) return []; @@ -130,6 +184,10 @@ export async function computeAllSourceMetrics( const chunkCounts = await chunkCountsBySource(engine); const jobCounts = await jobCountsBySource(engine); const now = Date.now(); + // v0.41.32.0: LOCAL callers (gbrain sources status/audit) opt into a live + // commit-hash probe; the REMOTE federation_health path leaves it off and + // reads the stored column (no subprocess on a DB-supplied local_path). + const probeContent = opts?.probeContent === true; return sources.map((src) => { const cfg = parseSourceConfig(src.config); @@ -142,9 +200,27 @@ export async function computeAllSourceMetrics( : Math.round((chunkStats.embedded / chunkStats.total) * 1000) / 10; const lastMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : null; - const lagSeconds = lastMs === null - ? null - : Math.max(0, Math.floor((now - lastMs) / 1000)); + // v0.41.32.0: commit-relative lag. + // LOCAL (probeContent): caught up iff HEAD == last_commit AND no tracked + // working-tree changes (untracked ignored) → lag 0; else wall-clock. + // Uses the live commit hash so a HEAD that moved to an old-dated commit + // is correctly NOT caught up. NULL last_commit → not caught up → wall-clock. + // REMOTE (default): read the stored newest_content_at column via + // lagFromContentMs — no git subprocess (v0.41.27.0 trust boundary). + let lagSeconds: number | null; + if (lastMs === null) { + lagSeconds = null; + } else if (probeContent) { + const caughtUp = isSourceUnchangedSinceSync(src.local_path, src.last_commit, { + requireCleanWorkingTree: 'ignore-untracked', + }); + lagSeconds = caughtUp ? 0 : Math.max(0, Math.floor((now - lastMs) / 1000)); + } else { + const contentMs = src.newest_content_at + ? new Date(src.newest_content_at).getTime() + : null; + lagSeconds = lagFromContentMs(contentMs, lastMs, now); + } return { source_id: src.id, diff --git a/src/core/sources-load.ts b/src/core/sources-load.ts index eb564135c..a72219f03 100644 --- a/src/core/sources-load.ts +++ b/src/core/sources-load.ts @@ -28,6 +28,14 @@ export interface SourceRow { config: Record | string; created_at: Date; archived?: boolean; + /** + * v0.41.32.0: newest COMMIT timestamp observed at last sync (HEAD committer + * time). The REMOTE staleness path reads this column so it never shells out + * to git on a DB-supplied local_path. Optional because the forward-reference + * fallback SELECT below omits it on pre-v109 brains; null/undefined → the + * reader falls back to wall-clock. + */ + newest_content_at?: Date | null; } export interface LoadAllSourcesOpts { @@ -67,13 +75,14 @@ export async function loadAllSources( let rows: SourceRow[]; try { rows = await engine.executeRaw( - `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived, newest_content_at FROM sources ORDER BY (id = 'default') DESC, id`, ); } catch (err) { - // Forward-reference safety: pre-v0.26.5 brains have no `archived` column. - // Re-issue without it; archived defaults to false. + // Forward-reference safety: pre-v0.26.5 brains lack `archived`; pre-v109 + // brains lack `newest_content_at`. Re-issue with the historical minimal + // set; archived defaults false, newest_content_at undefined → wall-clock. if (isUndefinedColumnError(err)) { rows = await engine.executeRaw( `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at @@ -102,7 +111,7 @@ export async function fetchSource( ): Promise { try { const rows = await engine.executeRaw( - `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived, newest_content_at FROM sources WHERE id = $1`, [id], ); diff --git a/src/schema.sql b/src/schema.sql index b5f874854..1eb53512b 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -49,6 +49,11 @@ CREATE TABLE IF NOT EXISTS sources ( -- (id='default') is always trusted regardless of this column. contextual_retrieval_mode TEXT, trust_frontmatter_overrides BOOLEAN NOT NULL DEFAULT false, + -- v0.41.32.0 (supersedes #1623): newest COMMIT timestamp (HEAD committer + -- time) recorded at last sync. The REMOTE staleness path reads this instead + -- of shelling out to git on a DB-supplied local_path, preserving the + -- v0.41.27.0 trust boundary. NULL → reader falls back to wall-clock. + newest_content_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); diff --git a/test/doctor.test.ts b/test/doctor.test.ts index a67c0a093..02684b98e 100644 --- a/test/doctor.test.ts +++ b/test/doctor.test.ts @@ -842,6 +842,121 @@ describe('v0.41.27.0 — sync_freshness git short-circuit', () => { }); }); +// ============================================================================ +// v0.41.32.0 — commit-relative staleness (supersedes #1623) +// ============================================================================ +// Two contracts: +// T1 (headline bug): a quiet repo whose only "dirt" is untracked files +// (`?? companies/`, `?? media/`) is now caught up on the LOCAL path — +// the short-circuit's clean check ignores untracked. Pre-v0.41.30 the +// strict clean check counted those as dirty → fell through to wall-clock +// → false SEVERE. +// T2 (trust boundary): the REMOTE path (no localOnly) computes lag from the +// stored newest_content_at column and NEVER shells out to git on a +// DB-supplied local_path (preserves the v0.41.27.0 boundary). +// ============================================================================ +describe('v0.41.32.0 — commit-relative staleness', () => { + function makeStubEngine(rows: any[]): any { + return { executeRaw: async () => rows }; + } + function agoMs(ms: number): Date { return new Date(Date.now() - ms); } + let currentChunkerVersion: string; + + beforeEach(async () => { + const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } = + await import('../src/core/git-head.ts'); + const { CHUNKER_VERSION } = await import('../src/core/chunkers/code.ts'); + currentChunkerVersion = String(CHUNKER_VERSION); + _setGitHeadProbeForTests(null); + _setGitCleanProbeForTests(null); + }); + afterAll(async () => { + const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } = + await import('../src/core/git-head.ts'); + _setGitHeadProbeForTests(null); + _setGitCleanProbeForTests(null); + }); + + test('T1: stale + HEAD match + DIRTY-by-untracked-only + localOnly → ok (untracked ignored)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } = + await import('../src/core/git-head.ts'); + _setGitHeadProbeForTests(() => 'abc123'); + // Clean ONLY when untracked is ignored (the bug scenario: `?? companies/`). + let sawIgnoreUntracked = false; + _setGitCleanProbeForTests((_path, ignoreUntracked) => { + if (ignoreUntracked) { sawIgnoreUntracked = true; return true; } + return false; // strict mode would call it dirty + }); + + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'media-corpus', name: '', local_path: '/tmp/media', + last_sync_at: agoMs(86 * 60 * 60 * 1000), // 86h — would be SEVERE on wall-clock + last_commit: 'abc123', chunker_version: currentChunkerVersion, + newest_content_at: null }, + ]), { localOnly: true }); + + expect(sawIgnoreUntracked).toBe(true); // the short-circuit asked to ignore untracked + expect(result.status).toBe('ok'); + expect(result.details).toEqual({ unchanged_count: 1, synced_recently_count: 0, stale_count: 0 }); + }); + + test('T2: REMOTE (no localOnly) reads column, quiet repo → ok, NO git subprocess', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } = + await import('../src/core/git-head.ts'); + let headCalls = 0, cleanCalls = 0; + _setGitHeadProbeForTests(() => { headCalls++; return 'x'; }); + _setGitCleanProbeForTests(() => { cleanCalls++; return true; }); + + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'remote', name: '', local_path: '/tmp/remote', + last_sync_at: agoMs(100 * 60 * 60 * 1000), + last_commit: 'x', chunker_version: currentChunkerVersion, + // Content committed BEFORE the last sync → caught up. + newest_content_at: agoMs(200 * 60 * 60 * 1000) }, + ])); // NOTE: no { localOnly: true } → remote path + + expect(headCalls).toBe(0); // trust boundary: no git probe on remote path + expect(cleanCalls).toBe(0); + expect(result.status).toBe('ok'); + expect(result.details).toEqual({ unchanged_count: 0, synced_recently_count: 1, stale_count: 0 }); + }); + + test('T2b: REMOTE + NULL column → wall-clock fallback → stale (no git subprocess)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } = + await import('../src/core/git-head.ts'); + let headCalls = 0; + _setGitHeadProbeForTests(() => { headCalls++; return 'x'; }); + _setGitCleanProbeForTests(() => true); + + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'remote', name: '', local_path: '/tmp/remote', + last_sync_at: agoMs(100 * 60 * 60 * 1000), + last_commit: 'x', chunker_version: currentChunkerVersion, + newest_content_at: null }, + ])); + + expect(headCalls).toBe(0); // still no git probe even on the fallback path + expect(result.status).toBe('fail'); // 100h wall-clock > 72h + expect(result.details?.stale_count).toBe(1); + }); + + test('T2c: REMOTE + content NEWER than last sync → wall-clock (genuinely behind)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'remote', name: '', local_path: '/tmp/remote', + last_sync_at: agoMs(100 * 60 * 60 * 1000), + last_commit: 'x', chunker_version: currentChunkerVersion, + // committed 10h ago, synced 100h ago → behind. + newest_content_at: agoMs(10 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('fail'); + expect(result.details?.stale_count).toBe(1); + }); +}); + // Supervisor crash classifier wiring. Pre-fix, doctor.ts:1013 counted every // `worker_exited` event as a crash regardless of `likely_cause`, inflating // `crashes_24h` to 120+/day from RSS-watchdog drains and SIGTERM stops. diff --git a/test/source-health.test.ts b/test/source-health.test.ts index e59d237dd..3e367de7e 100644 --- a/test/source-health.test.ts +++ b/test/source-health.test.ts @@ -1,22 +1,59 @@ /** - * Tests for src/core/source-health.ts (v0.40 D12 + D9 + D17). + * Tests for src/core/source-health.ts (v0.40 D12 + D9 + v0.41.32.0). * * Validates: - * - computeAllSourceMetrics: batched GROUP BY shape, vacuous truth for zero pages + * - computeAllSourceMetrics: batched GROUP BY shape, vacuous truth for zero + * pages, and v0.41.32.0 commit-relative lag (probeContent local path + + * stored-column remote path). * - resolvePriorityLabel: high/normal/low, unknown → normal + warn-once - * - isSourceStale: never-synced + lag-exceeded + fresh + missing local_path + * - newestCommitMs: HEAD committer time; null for non-git/missing. + * - lagFromContentMs: null/skew/null-content→wall-clock/caught-up→0/behind. + * - isSourceUnchangedSinceSync ignore-untracked: the commit-hash caught-up + * contract the local path relies on (incl. the codex old-dated-commit case). */ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { execFileSync } from 'child_process'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { computeAllSourceMetrics, resolvePriorityLabel, resolvePriority, - isSourceStale, + newestCommitMs, + lagFromContentMs, _resetPriorityWarningsForTest, } from '../src/core/source-health.ts'; +import { isSourceUnchangedSinceSync } from '../src/core/git-head.ts'; import { loadAllSources } from '../src/core/sources-load.ts'; +const HOUR = 3600_000; + +/** + * Create a throwaway git repo with one commit dated `commitDate`. Returns the + * dir + its HEAD sha so tests can seed `sources.last_commit` to match. + */ +function makeGitRepo(commitDate: Date, registry: string[]): { dir: string; head: string } { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-srchealth-')); + registry.push(dir); + const iso = commitDate.toISOString(); + const env = { + ...process.env, + GIT_AUTHOR_DATE: iso, GIT_COMMITTER_DATE: iso, + GIT_AUTHOR_NAME: 't', GIT_AUTHOR_EMAIL: 't@t', + GIT_COMMITTER_NAME: 't', GIT_COMMITTER_EMAIL: 't@t', + }; + const run = (args: string[]) => + execFileSync('git', ['-C', dir, ...args], { stdio: ['ignore', 'pipe', 'ignore'], env }); + run(['init', '-q']); + writeFileSync(join(dir, 'a.md'), '# a\n'); + run(['add', '-A']); + run(['commit', '-q', '-m', 'seed']); + const head = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(); + return { dir, head }; +} + let engine: PGLiteEngine; beforeAll(async () => { @@ -49,7 +86,6 @@ describe('resolvePriorityLabel', () => { expect(resolvePriorityLabel('s', null)).toBe('normal'); }); test('unknown values → normal with warn', () => { - // Reroute stderr to capture const orig = process.stderr.write.bind(process.stderr); let captured = ''; process.stderr.write = ((chunk: string | Uint8Array) => { @@ -75,8 +111,8 @@ describe('resolvePriorityLabel', () => { }) as never; try { resolvePriorityLabel('s1', { priority: 'urgent' }); - resolvePriorityLabel('s1', { priority: 'urgent' }); // same source - resolvePriorityLabel('s1', { priority: 42 }); // different bad value, same source + resolvePriorityLabel('s1', { priority: 'urgent' }); + resolvePriorityLabel('s1', { priority: 42 }); expect(count).toBe(1); } finally { process.stderr.write = orig; @@ -93,22 +129,95 @@ describe('resolvePriority (numeric)', () => { }); }); -describe('isSourceStale', () => { - test('never-synced (last_sync_at null) → true', () => { - const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: null, config: {}, created_at: new Date() }; - expect(isSourceStale(src, 60_000)).toBe(true); +// ── v0.41.32.0 commit-relative staleness ────────────────────────────── +describe('newestCommitMs', () => { + const repos: string[] = []; + afterAll(() => { + for (const d of repos) { try { rmSync(d, { recursive: true, force: true }); } catch { /* best-effort */ } } }); - test('no local_path → false (nothing to sync)', () => { - const src = { id: 's', name: 's', local_path: null, last_commit: null, last_sync_at: null, config: {}, created_at: new Date() }; - expect(isSourceStale(src, 60_000)).toBe(false); + + test('null for null / non-git / missing paths', () => { + expect(newestCommitMs(null)).toBeNull(); + expect(newestCommitMs('/tmp/gbrain-does-not-exist-' + Date.now())).toBeNull(); }); - test('synced within interval → false', () => { - const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 1000), config: {}, created_at: new Date() }; - expect(isSourceStale(src, 60_000)).toBe(false); + + test('returns the HEAD committer time in ms', () => { + const when = new Date(Date.now() - 50 * HOUR); + const { dir } = makeGitRepo(when, repos); + const ms = newestCommitMs(dir); + expect(ms).not.toBeNull(); + expect(Math.abs((ms as number) - when.getTime())).toBeLessThan(2000); }); - test('synced beyond interval → true', () => { - const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 120_000), config: {}, created_at: new Date() }; - expect(isSourceStale(src, 60_000)).toBe(true); +}); + +describe('lagFromContentMs (pure remote/column comparator)', () => { + const now = 1_000_000_000_000; + test('null last sync → null', () => { + expect(lagFromContentMs(now - HOUR, null, now)).toBeNull(); + }); + test('future last sync (skew) → negative passthrough', () => { + expect(lagFromContentMs(now, now + 10_000, now)).toBe(-10); + }); + test('null content → wall-clock fallback', () => { + expect(lagFromContentMs(null, now - 100 * HOUR, now)).toBe(360_000); // 100h in s + }); + test('content at/before last sync → caught up (0)', () => { + expect(lagFromContentMs(now - 200 * HOUR, now - 100 * HOUR, now)).toBe(0); + }); + test('content after last sync → wall-clock since sync', () => { + expect(lagFromContentMs(now - 10 * HOUR, now - 100 * HOUR, now)).toBe(360_000); + }); +}); + +describe('isSourceUnchangedSinceSync (ignore-untracked) — local caught-up contract', () => { + const repos: string[] = []; + afterAll(() => { + for (const d of repos) { try { rmSync(d, { recursive: true, force: true }); } catch { /* best-effort */ } } + }); + + test('HEAD == last_commit, clean → caught up', () => { + const { dir, head } = makeGitRepo(new Date(Date.now() - 200 * HOUR), repos); + expect(isSourceUnchangedSinceSync(dir, head, { requireCleanWorkingTree: 'ignore-untracked' })).toBe(true); + }); + + test('HEAD == last_commit WITH untracked dirs → still caught up (the headline bug)', () => { + const { dir, head } = makeGitRepo(new Date(Date.now() - 200 * HOUR), repos); + // Stray untracked dirs (the `?? companies/`, `?? media/` shape). + writeFileSync(join(dir, 'companies'), 'x'); // file is fine; untracked either way + execFileSync('git', ['-C', dir, 'status', '--porcelain'], { encoding: 'utf8' }); // sanity: dirty by default + expect(isSourceUnchangedSinceSync(dir, head, { requireCleanWorkingTree: 'ignore-untracked' })).toBe(true); + // Strict mode (pre-v0.41.30 behavior) would have called it dirty: + expect(isSourceUnchangedSinceSync(dir, head, { requireCleanWorkingTree: true })).toBe(false); + }); + + test('tracked uncommitted edit → NOT caught up (sync would re-walk the commit)', () => { + const { dir, head } = makeGitRepo(new Date(Date.now() - 200 * HOUR), repos); + writeFileSync(join(dir, 'a.md'), '# a edited\n'); // a.md is TRACKED + expect(isSourceUnchangedSinceSync(dir, head, { requireCleanWorkingTree: 'ignore-untracked' })).toBe(false); + }); + + test('HEAD moved to an OLD-dated commit → NOT caught up (codex: hash, not timestamp)', () => { + const { dir, head } = makeGitRepo(new Date(Date.now() - 200 * HOUR), repos); + // Add a SECOND commit with an even OLDER committer date. A timestamp + // comparison (newest content <= last sync) would falsely say "caught up"; + // the hash check correctly sees HEAD != last_commit. + const olderIso = new Date(Date.now() - 500 * HOUR).toISOString(); + const env = { + ...process.env, + GIT_AUTHOR_DATE: olderIso, GIT_COMMITTER_DATE: olderIso, + GIT_AUTHOR_NAME: 't', GIT_AUTHOR_EMAIL: 't@t', + GIT_COMMITTER_NAME: 't', GIT_COMMITTER_EMAIL: 't@t', + }; + writeFileSync(join(dir, 'b.md'), '# b\n'); + execFileSync('git', ['-C', dir, 'add', '-A'], { stdio: ['ignore', 'pipe', 'ignore'], env }); + execFileSync('git', ['-C', dir, 'commit', '-q', '-m', 'old-dated'], { stdio: ['ignore', 'pipe', 'ignore'], env }); + // `head` is still the FIRST commit's sha (the recorded last_commit). + expect(isSourceUnchangedSinceSync(dir, head, { requireCleanWorkingTree: 'ignore-untracked' })).toBe(false); + }); + + test('NULL last_commit → not provably caught up (false)', () => { + const { dir } = makeGitRepo(new Date(Date.now() - 10 * HOUR), repos); + expect(isSourceUnchangedSinceSync(dir, null, { requireCleanWorkingTree: 'ignore-untracked' })).toBe(false); }); }); @@ -128,7 +237,6 @@ describe('computeAllSourceMetrics', () => { }); test('aggregates pages + chunks + embedding coverage per source', async () => { - // Two pages with chunks, half embedded await engine.putPage('a', { type: 'note', title: 'a', compiled_truth: 'a' }); await engine.putPage('b', { type: 'note', title: 'b', compiled_truth: 'b' }); await engine.upsertChunks('a', [ @@ -145,7 +253,6 @@ describe('computeAllSourceMetrics', () => { expect(dflt.total_pages).toBe(2); expect(dflt.total_chunks).toBe(3); expect(dflt.embedded_chunks).toBe(1); - // 1/3 = 33.3% expect(dflt.embed_coverage_pct).toBeCloseTo(33.3, 1); }); @@ -196,4 +303,65 @@ describe('computeAllSourceMetrics', () => { const d = result.find((m) => m.source_id === 'default')!; expect(d.webhook_configured).toBe(false); }); + + // v0.41.32.0: commit-relative lag — local (probeContent) vs remote (column). + describe('commit-relative lag', () => { + const repos: string[] = []; + afterAll(() => { + for (const d of repos) { try { rmSync(d, { recursive: true, force: true }); } catch { /* best-effort */ } } + }); + + test('LOCAL (probeContent): caught-up repo synced 100h ago → lag 0', async () => { + const { dir, head } = makeGitRepo(new Date(Date.now() - 200 * HOUR), repos); + const syncIso = new Date(Date.now() - 100 * HOUR).toISOString(); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, config) + VALUES ('quiet', 'quiet', $1, $2, $3, '{"federated":true}'::jsonb)`, + [dir, head, syncIso], + ); + const sources = await loadAllSources(engine); + const metrics = await computeAllSourceMetrics(engine, sources, { probeContent: true }); + expect(metrics.find((m) => m.source_id === 'quiet')!.lag_seconds).toBe(0); + }); + + test('LOCAL (probeContent): HEAD moved (behind) → wall-clock lag', async () => { + const { dir } = makeGitRepo(new Date(Date.now() - 100 * HOUR), repos); + const staleCommit = 'b'.repeat(40); // last_commit no longer matches HEAD + const syncIso = new Date(Date.now() - 200 * HOUR).toISOString(); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, config) + VALUES ('behind', 'behind', $1, $2, $3, '{"federated":true}'::jsonb)`, + [dir, staleCommit, syncIso], + ); + const sources = await loadAllSources(engine); + const metrics = await computeAllSourceMetrics(engine, sources, { probeContent: true }); + expect(metrics.find((m) => m.source_id === 'behind')!.lag_seconds!).toBeGreaterThan(72 * 3600); + }); + + test('REMOTE (default): reads newest_content_at column, NO git probe → quiet repo lag 0', async () => { + const contentIso = new Date(Date.now() - 200 * HOUR).toISOString(); // content predates sync + const syncIso = new Date(Date.now() - 100 * HOUR).toISOString(); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, newest_content_at, config) + VALUES ('remote', 'remote', '/nonexistent/not-a-repo', 'x', $1, $2, '{"federated":true}'::jsonb)`, + [syncIso, contentIso], + ); + const sources = await loadAllSources(engine); + // probeContent OFF (remote): even though local_path is bogus, no git runs. + const metrics = await computeAllSourceMetrics(engine, sources); + expect(metrics.find((m) => m.source_id === 'remote')!.lag_seconds).toBe(0); + }); + + test('REMOTE (default): NULL column → wall-clock fallback', async () => { + const syncIso = new Date(Date.now() - 100 * HOUR).toISOString(); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, config) + VALUES ('nocol', 'nocol', '/nonexistent', 'x', $1, '{"federated":true}'::jsonb)`, + [syncIso], + ); + const sources = await loadAllSources(engine); + const metrics = await computeAllSourceMetrics(engine, sources); + expect(metrics.find((m) => m.source_id === 'nocol')!.lag_seconds!).toBeGreaterThan(99 * 3600); + }); + }); }); diff --git a/test/sync-all-parallel.test.ts b/test/sync-all-parallel.test.ts index 187a91ced..7ea24f2a7 100644 --- a/test/sync-all-parallel.test.ts +++ b/test/sync-all-parallel.test.ts @@ -135,7 +135,7 @@ describe('buildSyncStatusReport', () => { // `page_id` shape. The IRON RULE regression case lives in // test/e2e/sync-status-pglite.test.ts and exercises real SQL. function makeEngine(scripts: { - sourceRows?: Array<{ id: string; last_commit: string | null; last_sync_at: string | null }>; + sourceRows?: Array<{ id: string; last_commit: string | null; last_sync_at: string | null; newest_content_at?: string | null }>; countRows?: Array<{ source_id: string; pages: number; chunks_total: number; chunks_unembedded: number }>; }): BrainEngine { return { @@ -192,6 +192,48 @@ describe('buildSyncStatusReport', () => { expect(byId.get('never')!.staleness_hours).toBeNull(); }); + // v0.41.32.0 (supersedes #1623): buildSyncStatusReport backs the REMOTE + // get_status_snapshot MCP op, so staleness reads the stored newest_content_at + // column (NO git subprocess on a DB-supplied local_path). The makeEngine stub + // never runs git — if buildSyncStatusReport shelled out it would hit the real + // filesystem; these cases prove it reads the column instead. + test('content-relative staleness reads newest_content_at column (remote path)', async () => { + const now = Date.now(); + const syncIso = new Date(now - 100 * 60 * 60 * 1000).toISOString(); // synced 100h ago + const sources = [ + { id: 'quiet', name: 'quiet', local_path: '/tmp/quiet', config: { syncEnabled: true } }, + { id: 'behind', name: 'behind', local_path: '/tmp/behind', config: { syncEnabled: true } }, + { id: 'nocol', name: 'nocol', local_path: '/tmp/nocol', config: { syncEnabled: true } }, + ]; + const engine = makeEngine({ + sourceRows: [ + // Newest commit 200h ago, synced 100h ago → caught up → lag 0 → fresh. + { id: 'quiet', last_commit: 'a'.repeat(40), last_sync_at: syncIso, + newest_content_at: new Date(now - 200 * 60 * 60 * 1000).toISOString() }, + // Newest commit 10h ago, synced 100h ago → behind → wall-clock → severe. + { id: 'behind', last_commit: 'b'.repeat(40), last_sync_at: syncIso, + newest_content_at: new Date(now - 10 * 60 * 60 * 1000).toISOString() }, + // NULL column → wall-clock fallback → 100h → severe. + { id: 'nocol', last_commit: 'c'.repeat(40), last_sync_at: syncIso, newest_content_at: null }, + ], + countRows: [ + { source_id: 'quiet', pages: 10, chunks_total: 20, chunks_unembedded: 0 }, + { source_id: 'behind', pages: 10, chunks_total: 20, chunks_unembedded: 0 }, + { source_id: 'nocol', pages: 10, chunks_total: 20, chunks_unembedded: 0 }, + ], + }); + + const report = await buildSyncStatusReport(engine, sources); + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + // Legacy wall-clock would have called 'quiet' severe (100h). Content-relative + // correctly reports caught-up. + expect(byId.get('quiet')!.staleness_hours).toBe(0); + expect(byId.get('quiet')!.staleness_class).toBe('fresh'); + expect(byId.get('behind')!.staleness_class).toBe('severe'); + expect(byId.get('nocol')!.staleness_hours).toBeGreaterThan(72); + expect(byId.get('nocol')!.staleness_class).toBe('severe'); + }); + test('embedding_coverage_pct computed from chunks_total vs chunks_unembedded', async () => { const sources = [ { id: 'a', name: 'a', local_path: '/tmp/a', config: {} },