diff --git a/AGENTS.md b/AGENTS.md index 846a04ce4..5ae614780 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,8 +4,16 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL): start here. +> **Becoming someone's persistent personal agent** (identity + memory + private repo)? +> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap` +> flow — instead of the plain install below, then come back here for the operating +> protocol. Connecting to an EXISTING remote brain from a laptop agent? +> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP +> table in [`README.md`](./README.md)). + ## Install (5 min) + 1. Install gbrain via Bun (the canonical path): ```bash curl -fsSL https://bun.sh/install | bash @@ -26,8 +34,8 @@ start here. [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade` for existing users (search modes were added in v0.32.3). -4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow - (API keys, identity, cron, verification). +4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step + flow (API keys, identity, cron, verification). ## Read this order @@ -69,10 +77,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont `GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson` and `gbrain eval replay --against base.ndjson`. For public benchmark coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval - ` (v0.28.8) runs against an isolated in-memory PGLite + ` runs against an isolated in-memory PGLite per question — your `~/.gbrain` is never opened. Full guide: [`docs/eval-bench.md`](./docs/eval-bench.md). -- **Drive the brain to a target health score (v0.36.4.0):** the one-command +- **Drive the brain to a target health score:** the one-command loop. `gbrain doctor --remediation-plan --json` previews what would be fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5` walks a dependency-ordered plan (sync before extract, embed after @@ -81,22 +89,20 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont keys hit a `max_reachable_score` ceiling and bail with what's missing. Three phase handlers (synthesize / patterns / consolidate) are PROTECTED — only trusted local callers can submit them; MCP cannot. - Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md) - and the CHANGELOG entry for v0.36.4.0. -- **Track a founder/company over time (v0.35.7):** when an entity has + Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md). +- **Track a founder/company over time:** when an entity has typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`, `unit: USD`, `period: monthly` columns), run `gbrain eval trajectory ` for the chronological history with regressions auto-flagged, or `gbrain founder scorecard ` for a four-signal JSON rollup (claim_accuracy / consistency / growth_trajectory / red_flags). MCP op `find_trajectory` exposes the - same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:** - `gbrain think` now uses this substrate automatically on temporal / + same data — read scope, visibility-filtered for remote callers. + `gbrain think` uses this substrate automatically on temporal / knowledge_update intent (default ON; flip `think.trajectory_enabled=false` - to opt out). Migration v82 added `facts.event_type` so non-metric event - rows (`meeting`, `job_change`, `location_change`) ride through the same - pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query - them. + to opt out). Non-metric event rows (`meeting`, `job_change`, + `location_change`) ride through the same pipeline via `facts.event_type`; + pass `kind: 'event'` or `'all'` to `find_trajectory` to query them. - **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map. [`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for single-fetch ingestion. diff --git a/CLAUDE.md b/CLAUDE.md index 9b2913337..3c206b6f2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP +Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -191,9 +191,10 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently expensive one. tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of -the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost -picker copy in `gbrain init` carries the same matrix verbatim** — update -both when refreshing. +the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix +has three verbatim homes: this section, the `gbrain init` picker copy +(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step +3.5** — update all three when refreshing. **Per-query math vs real-world spend.** The matrix above is what an isolated benchmark would measure. Real agent loops with disciplined @@ -273,8 +274,9 @@ audit trail lives in the source repo's git history. ## Skills -Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills -organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19): +Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills +(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md` +(`AGENTS.md` is also accepted as of v0.19): **Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich, briefing, migrate, setup, publish. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d6408cd20..fdf3b9857 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,7 +48,9 @@ src/ core/ operations.ts Contract-first operation definitions (the foundation) engine.ts BrainEngine interface - postgres-engine.ts Postgres implementation + engine-factory.ts Engine factory (dynamic import of the configured engine) + postgres-engine.ts Postgres + pgvector implementation + pglite-engine.ts PGLite (embedded Postgres via WASM) implementation db.ts Connection management + schema loader import-file.ts Import pipeline (chunk + embed + tags) types.ts TypeScript types @@ -59,12 +61,16 @@ src/ supabase-admin.ts Supabase admin API file-resolver.ts MIME detection + content hashing migrate.ts Migration helpers + bootstrap/ Agent-bootstrap flow (interview, hooks, repo, verify) yaml-lite.ts Lightweight YAML parser chunkers/ 3-tier chunking (recursive, semantic, llm) search/ Hybrid search (vector, keyword, hybrid, expansion, dedup) - embedding.ts OpenAI embedding service + embedding.ts Embedding service (provider-routed; ZeroEntropy default) mcp/ server.ts MCP stdio server (generated from operations) + http-transport.ts HTTP MCP transport (OAuth, body caps) + dispatch.ts Op dispatch + scope enforcement + param redaction + rate-limit.ts Rate limiting schema.sql Postgres DDL skills/ Fat markdown skills for AI agents test/ Unit tests (bun test, no DB required) @@ -77,15 +83,21 @@ test/e2e/ E2E tests (requires DATABASE_URL, real Postgres+pgvect docs/ Architecture docs ``` +Per-file invariants live in `docs/architecture/KEY_FILES.md` — read a file's entry +before editing it. + ## Running tests +The canonical reference for test tiers, isolation rules, timing, and the E2E +lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version: + ```bash -# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests) +# Inner edit loop (~85s on a Mac dev box) bun run test # parallel 8-shard fan-out + serial post-pass bun test test/markdown.test.ts # specific unit test -# Pre-push gate (matches what CI runs on shard 1 + typecheck) -bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck +# Pre-push gate (19+ parallel checks + typecheck) +bun run verify # Pre-merge sanity (everything CI runs) bun run test:full # verify + parallel unit + slow + smart e2e @@ -103,92 +115,51 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t DATABASE_URL=postgresql://... bun run test:e2e ``` -Use `bun run verify` before pushing. The guard chain catches: banned fork-name -leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation +Use `bun run verify` before pushing. It runs 19+ guard checks in parallel +(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks +(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout (`scripts/check-progress-to-stdout.sh`), test-isolation rule violations (`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel loop" below), silent fallback to recursive chunking in the compiled binary (`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts -(`scripts/check-admin-build.sh`), and resolver drift on bundled skills -(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any -warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift -before merge). `bun run check:all` runs the full historical sweep including the -trailing-newline and exports-count checks. +(`scripts/check-admin-build.sh`), resolver drift on bundled skills +(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full +historical sweep including the trailing-newline and exports-count checks. ### Writing tests that survive the parallel loop -`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the -same shard share a process, so process-global state leaks between them. Four -lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation: +`bun run test` shards the unit-test files (1000+) across 8 worker processes. +Files in the same shard share a process, so process-global state leaks between +them. Four lint rules (`scripts/check-test-isolation.sh`, R1–R4) enforce +isolation: no direct `process.env` mutation (use `withEnv()` from +`test/helpers/with-env.ts`), no `mock.module(...)` outside `*.serial.test.ts`, +and every `new PGLiteEngine(` goes inside the canonical `beforeAll` block with +a paired `afterAll(disconnect)`. -| Rule | What it bans | Fix | -|---|---|---| -| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` | -| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` | -| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) | -| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` | +**The full rules, the canonical PGLite block, the `withEnv` pattern, and the +`*.serial.test.ts` quarantine policy live in +[`docs/TESTING.md`](docs/TESTING.md#test-isolation-lint-and-helpers) +— read that before writing a new test file.** Files that predate the rules are +listed in `scripts/check-test-isolation.allowlist`; the allow-list MUST shrink +over time — never add new entries. -Canonical PGLite block (R3 + R4 compliant — paste this verbatim): - -```ts -import { PGLiteEngine } from '../src/core/pglite-engine.ts'; -import { resetPgliteState } from './helpers/reset-pglite.ts'; - -let engine: PGLiteEngine; - -beforeAll(async () => { - engine = new PGLiteEngine(); - await engine.connect({}); - await engine.initSchema(); -}); -afterAll(async () => { await engine.disconnect(); }); -beforeEach(async () => { await resetPgliteState(engine); }); -``` - -Env-touching tests: - -```ts -import { withEnv } from './helpers/with-env.ts'; - -test('reads OPENAI_API_KEY', async () => { - await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => { - expect(loadConfig().openai_key).toBe('sk-test'); - }); -}); -``` - -`withEnv` saves and restores keys via try/finally including when the callback -throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is -process-global). Files using `withEnv` stay outside the future -`test.concurrent()` codemod's eligibility filter. - -When to quarantine instead of fix: rename to `*.serial.test.ts` if the file -uses `mock.module(...)`, is genuinely env-coupled (module-load env readers + -ESM caching defeat dynamic-import-after-env tricks), or intentionally shares -state across `it()` boundaries. Quarantine count cap: 10 (informational). - -Files that violated these rules at the v0.26.7 baseline are listed in -`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over -time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep -+ codemod) remove entries as files get fixed. - -### Local CI gate (recommended before pushing, v0.23.1+) +### Local CI gate (recommended before pushing) ```bash -bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential) +bun run ci:local # full gate: gitleaks + guards/typecheck + 4-shard parallel unit + E2E bun run ci:local:diff # gate with diff-aware E2E selector bun run ci:select-e2e # print which E2E files the selector would run ``` -`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via -`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then -tears down. Named volumes keep the install warm across runs (~16-20 min sequential -E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or -Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres -host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. +`ci:local` spins up four pgvector services plus a transaction-mode PgBouncer via +`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite +sharded 4 ways in parallel, then tears down. Named volumes keep the install warm +across runs. Requires Docker (Docker Desktop, OrbStack, or Colima) and `gitleaks` +on host (`brew install gitleaks`). Override the postgres host port with +`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. -Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune +Fail-closed selector: an unmapped `src/` change runs ALL E2E files. Hand-tune narrower mappings via `scripts/e2e-test-map.ts`. ### PR-side security checks @@ -225,7 +196,7 @@ Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync. See `docs/ENGINES.md` for the full guide. In short: 1. Create `src/core/myengine-engine.ts` implementing `BrainEngine` -2. Add to engine factory in `src/core/engine.ts` +2. Add to the engine factory in `src/core/engine-factory.ts` 3. Run the test suite against your engine 4. Document in `docs/` @@ -310,7 +281,7 @@ NDJSON wire format is documented in [`docs/eval-capture.md`](./docs/eval-capture.md). For public benchmark coverage on top of replay, `gbrain eval longmemeval -` (v0.28.1) runs LongMemEval against gbrain's hybrid +` runs LongMemEval against gbrain's hybrid retrieval. One in-memory PGLite per question, runtime-enumerated `TRUNCATE` between questions, ground-truth scoring via LongMemEval's published `evaluate_qa.py`. Use it alongside replay when changes affect @@ -319,9 +290,18 @@ regressions on YOUR queries, LongMemEval catches them on a public set the benchmark community already cites. See the "Public benchmarks: LongMemEval" section in [`docs/eval-bench.md`](./docs/eval-bench.md). +## Shipping + +Releases go through the `/ship` skill, never hand-rolled. The full release + +contributor process (CHANGELOG voice, version-locations sync, PR conventions, +community-PR-wave workflow) lives in [`docs/RELEASING.md`](docs/RELEASING.md). +Community PRs are batched into release waves rather than merged one-by-one; +contributor attribution stays attached via `Co-Authored-By:` trailers and every +accepted contribution is credited in `CHANGELOG.md`. + ## Welcome PRs -- SQLite engine implementation +- Additional engine implementations (see [`docs/ENGINES.md`](docs/ENGINES.md)) - Docker Compose for self-hosted Postgres - Additional migration sources - New enrichment API integrations diff --git a/INSTALL_FOR_AGENTS.md b/INSTALL_FOR_AGENTS.md index 2e944d59c..7516541f6 100644 --- a/INSTALL_FOR_AGENTS.md +++ b/INSTALL_FOR_AGENTS.md @@ -21,10 +21,13 @@ If you fetched this file by URL without cloning yet, the companion files live at ## Step 1: Install GBrain + > **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm > package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or > `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only -> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below. +> supported sources are `github:garrytan/gbrain` (optionally pinned as +> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a +> git clone, exactly as shown below. > If an unrelated npm install is already present, remove it first > (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this. @@ -93,6 +96,7 @@ default. Stop and ask the operator. **Present this matrix verbatim:** + ``` Per-query cost @ 10K queries/mo (typical single-user volume): @@ -203,13 +207,13 @@ scaffold the bundled skills into it: ```bash cd /path/to/agent/workspace -gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md +gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md ``` Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold refuses to overwrite anything that exists. Use `gbrain skillpack reference ` to diff against gbrain's bundle when you want upstream improvements. (The legacy -`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run +`gbrain skillpack install` managed-block model was removed in v0.33 — run `gbrain skillpack migrate-fence` once if upgrading from an older release.) Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the @@ -266,8 +270,17 @@ Verify: `gbrain integrations doctor` (after at least one is configured) ## Step 9: Verify -Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync -actually works) is the most important. +Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4 +(live sync actually works) is the most important. + +Once verification passes and the brain has content, run the activation probe: + +```bash +gbrain onboard --check --json +``` + +See "The onboard surface" below for what the recommendations mean and the +consent gates around unattended remediation. ## Upgrade @@ -310,7 +323,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old `splitBody` bug, run `gbrain sync --full` after upgrading to rebuild `compiled_truth` from source markdown. -## v0.42.0+ onboard surface (NEW) +## The onboard surface `gbrain onboard` is the activation surface gbrain did not have before. Once your brain has any content, run `gbrain onboard --check --json` to diff --git a/README.md b/README.md index 6e0b7d777..e8eb8b351 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ Retrieve and follow the instructions at: https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md ``` -The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work. +The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work. > **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new. @@ -159,7 +159,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL. ### Connect GBrain to your AI client (MCP) -GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use: +GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side). The specific snippet depends on which client you use: - **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token). - **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config. @@ -281,7 +281,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec ## Capabilities -**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. +**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. **Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). @@ -305,7 +305,7 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. -**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. +**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). @@ -319,14 +319,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h - **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md). - **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md). -- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). -- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). +- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). +- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). - **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md). - **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup. ## Architecture -**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source. +**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source. **Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md). @@ -340,10 +340,9 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h **`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model :` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing. -**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships -two flags + a recommended pattern. Switch your cron to a per-source loop -with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating -gracefully half-a-minute earlier: +**Hourly cron sync keeps timing out on a federated brain?** Switch your +cron to a per-source loop with shell `timeout(1)` doing the OS-level kill +and gbrain self-terminating gracefully half-a-minute earlier: ```bash gbrain sync --break-lock --all --max-age 1800 @@ -356,19 +355,17 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status `partial` and `last_commit` UNCHANGED — the next run re-walks the same diff and `content_hash` short-circuits already-imported files. The `--max-age 1800` first command self-heals any wedged-but-alive locks -left by a hung previous run, using the v98 `last_refreshed_at` semantic -(NOT `acquired_at`) so healthy long-running holders are safe by -construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md) -for the honest scope notes (extract + embed phases run to completion; -30-min rollout window for `--max-age` post-migration v98; full-sync -triggers deferred to v0.42+). +left by a hung previous run, keyed on the lock's last refresh time +(NOT when it was acquired) so healthy long-running holders are safe by +construction. Scope note: the extract + embed phases still run to +completion once started; `--timeout` interrupts the import walk only. -**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes -the bug class structurally. The engine now self-retries every bulk batch -write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on -Supavisor pooler blips, with a 12s worst-case wait that covers the full -5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents -via the new `batch_retry_health` check (reads the last 24h of +**Dream cycle silently losing wiki links on Supabase?** The engine +self-retries every bulk batch write (`addLinksBatch` / +`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips, +with a 12s worst-case wait that covers the full 5-10s circuit-breaker +recovery window. `gbrain doctor` surfaces incidents via the +`batch_retry_health` check (reads the last 24h of `~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually slow pooler: @@ -386,34 +383,33 @@ retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice. **Dream cycle losing ~150 link rows per run with `'No database -connection: connect() has not been called'` errors in the log?** v0.41.27.0 -makes the retry layer self-heal on a nulled-out database singleton. A -new `reconnect` callback on `withRetry` rebuilds the connection between -attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()` +connection: connect() has not been called'` errors in the log?** The +retry layer self-heals on a nulled-out database singleton: a +`reconnect` callback on `withRetry` rebuilds the connection between +attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()` so engine-level batch writes survive a mid-cycle disconnect by something -else in the same process. Same release: `gbrain capture` no longer trails -a `'No database connection'` stderr line from a background facts:absorb -worker firing after CLI exit — the op-dispatch finally block awaits +else in the same process. `gbrain capture` also no longer trails a +`'No database connection'` stderr line from a background facts:absorb +worker firing after CLI exit — op dispatch awaits `getFactsQueue().drainPending({timeout: 1000})` before `engine.disconnect()`. To find which code path is still calling disconnect mid-process, run `gbrain doctor --json | jq '.checks[] | -select(.id=="batch_retry_health")'`; the extended check now surfaces -24h disconnect-call count and the most-recent caller frame from a new -`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.) +select(.id=="batch_retry_health")'`; the check surfaces the +24h disconnect-call count and the most-recent caller frame from the +`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. **`gbrain brainstorm` returning `judge_failed: true` with 0 scored -ideas?** v0.41.21.0 closes the two bugs that caused it. The judge -hard-coded a 4K-token output cap; for any run past ~40 ideas the call -truncated mid-JSON and the parser threw. Same release closes a slash- -form pricing miss: `gbrain brainstorm --judge-model -anthropic/claude-sonnet-4-6 --max-cost 5` failed with -`BudgetExhausted reason=no_pricing` because every pricing site only -matched the colon form. Both shapes work now. No config change, no -schema migration — `gbrain upgrade` is the whole fix. +ideas?** Two historical bugs caused it, both fixed: the judge +hard-coded a 4K-token output cap (any run past ~40 ideas truncated +mid-JSON and the parser threw), and slash-form model ids +(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6 +--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because +pricing lookups only matched the colon form. Both shapes work now. No +config change, no schema migration — `gbrain upgrade` is the whole fix. **`gbrain reindex --markdown` wiped your auto/dream/signal-detector -tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and -`reindex --markdown` now ADD current frontmatter tags and never delete, +tags?** Upgrade — tag reconciliation is add-only now. Re-import and +`reindex --markdown` ADD current frontmatter tags and never delete, so enrichment tags written to the DB (auto-tag, dream synthesize, signal-detector) survive a re-chunk. The reindex DB-only fallback also reconstructs the full markdown (frontmatter + body + timeline) before @@ -421,10 +417,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter, title, and timeline instead of getting overwritten with empty frontmatter. Trade-off: removing a tag from a page's frontmatter no longer removes it from the DB on the next sync (frontmatter-tag removal -needs a provenance column, deferred). (Closes #1621.) +needs a provenance column, deferred). **`gbrain sync` wedges on a large brain (no progress, high CPU)?** -v0.41.37.0 ships three things. First, name the stalling file: +Three tools. First, name the stalling file: ```bash GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes @@ -439,28 +435,28 @@ the sync with the pack disabled and re-run extraction later: gbrain sync --no-schema-pack --no-pull --no-embed --yes ``` -`gbrain schema lint` now warns on the classic nested-quantifier ReDoS +`gbrain schema lint` warns on the classic nested-quantifier ReDoS shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`). Third, on a PGLite brain, stop `gbrain serve` before a large sync — PGLite is single-writer and a live MCP server contends for the write lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md) -for the full triage. (Closes #1569.) +for the full triage. **`gbrain init --migrate-only` / a schema migration fails on Windows -with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up +with `getaddrinfo ENOTFOUND`?** Upgrade — schema bring-up now runs its phases in-process instead of spawning a child `gbrain init --migrate-only` per phase. The spawned child died on Windows + bun + Supabase pooler with a DNS-resolution failure even though the parent connected fine; running in-process removes the spawn -entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an -82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL -pass (keyed on the page PK, soft-delete-filtered, source-safe) that -completes in ~1-2 seconds. (Closes #1605, #1581.) +entirely. The grandfather migration that used to hang 70+ minutes on an +80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on +the page PK, soft-delete-filtered, source-safe) and completes in seconds. ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end +- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall) - [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0) - [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows` - [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 6636ea9b2..b5b2f9d7a 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -4,7 +4,7 @@ Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else. -v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code. +Two engines ship today: `PGLiteEngine` — embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code. ## Why this matters @@ -12,7 +12,7 @@ Different users have different constraints: | User | Needs | Best engine | |------|-------|-------------| -| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) | +| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) | | Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase | | Open source hacker | Single file, no server, git-friendly | PGLiteEngine | | Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted | @@ -23,72 +23,30 @@ The engine interface means we don't have to choose. PGLite is the zero-friction ## The interface -```typescript -// src/core/engine.ts +**The single source of truth is `export interface BrainEngine` in +`src/core/engine.ts`.** It is large (100+ methods) and grows with every +feature wave — do NOT work from any snapshot of it, including an old copy of +this doc. Read the interface itself, and let +`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you +whether both engines agree. -export interface BrainEngine { - // Lifecycle - connect(config: EngineConfig): Promise; - disconnect(): Promise; - initSchema(): Promise; - transaction(fn: (engine: BrainEngine) => Promise): Promise; +The method families, to orient you before opening the file: - // Pages CRUD - getPage(slug: string): Promise; - putPage(slug: string, page: PageInput): Promise; - deletePage(slug: string): Promise; - listPages(filters: PageFilters): Promise; - - // Search - searchKeyword(query: string, opts?: SearchOpts): Promise; - searchVector(embedding: Float32Array, opts?: SearchOpts): Promise; - - // Chunks - upsertChunks(slug: string, chunks: ChunkInput[]): Promise; - getChunks(slug: string): Promise; - - // Links - addLink(from: string, to: string, context?: string, linkType?: string): Promise; - removeLink(from: string, to: string): Promise; - getLinks(slug: string): Promise; - getBacklinks(slug: string): Promise; - traverseGraph(slug: string, depth?: number): Promise; - - // Tags - addTag(slug: string, tag: string): Promise; - removeTag(slug: string, tag: string): Promise; - getTags(slug: string): Promise; - - // Timeline - addTimelineEntry(slug: string, entry: TimelineInput): Promise; - getTimeline(slug: string, opts?: TimelineOpts): Promise; - - // Raw data - putRawData(slug: string, source: string, data: object): Promise; - getRawData(slug: string, source?: string): Promise; - - // Versions - createVersion(slug: string): Promise; - getVersions(slug: string): Promise; - revertToVersion(slug: string, versionId: number): Promise; - - // Stats + health - getStats(): Promise; - getHealth(): Promise; - - // Ingest log - logIngest(entry: IngestLogInput): Promise; - getIngestLog(opts?: IngestLogOpts): Promise; - - // Config - getConfig(key: string): Promise; - setConfig(key: string, value: string): Promise; - - // Migration + advanced (added v0.7) - runMigration(sql: string): Promise; - getChunksWithEmbeddings(slug: string): Promise; -} -``` +- **Lifecycle + identity** — `connect` / `disconnect` / `reconnect`, + `initSchema`, `transaction`, `withReservedConnection`, and the `kind` + discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch. +- **Pages CRUD** — `getPage`, `putPage`, `deletePage`, `listPages`, slug + resolution. +- **Search** — `searchKeyword`, `searchVector`, chunk-level variants, takes + search (keyword + vector), and `relationalFanout` (the typed-edge recall + arm). +- **Chunks + embeddings** — upsert/get, embedding-bearing variants. +- **Graph** — links (single + batch writers), backlinks, `traverseGraph`, + `traversePaths`. +- **Tags, timeline (single + batch), raw data, versions.** +- **Takes / facts / eval / salience** — the epistemological layer and the + instruments over it. +- **Stats, health, ingest log, config, migrations.** ### Key design choices @@ -131,7 +89,7 @@ export interface BrainEngine { RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific. -## PostgresEngine (v0, ships) +## PostgresEngine **Dependencies:** `postgres` (porsager/postgres), `pgvector` @@ -144,9 +102,7 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o - JSONB for frontmatter with GIN index - Connection pooling via Supabase Supavisor (port 6543) -**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in. - -**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops. +**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same. ### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`) @@ -193,17 +149,17 @@ run under the role default and are not backstopped per caller. This is layer 2; the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins live in `test/postgres-engine-rls-scope.test.ts`. -## PGLiteEngine (v0.7, ships) +## PGLiteEngine -**Dependencies:** `@electric-sql/pglite` (v0.4.4+) +**Dependencies:** `@electric-sql/pglite` -**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented. +**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep. **PGLite-specific details:** - Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes) - Parameterized queries throughout (shared utilities in `src/core/utils.ts`) - `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set -- Data stored at `~/.gbrain/brain.db` (configurable) +- Data stored at `~/.gbrain/brain.pglite` (configurable) - pgvector HNSW index for cosine similarity vector search (same as Postgres) - tsvector + ts_rank for full-text search (same as Postgres) - pg_trgm for fuzzy slug resolution (same as Postgres) @@ -319,16 +275,22 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make. 1. Create `src/core/-engine.ts` implementing `BrainEngine` 2. Add to engine factory in `src/core/engine-factory.ts`: ```typescript - export function createEngine(type: string): BrainEngine { - switch (type) { - case 'pglite': return new PGLiteEngine(); - case 'postgres': return new PostgresEngine(); - case 'myengine': return new MyEngine(); - default: throw new Error(`Unknown engine: ${type}`); + export async function createEngine(config: EngineConfig): Promise { + switch (config.engine || 'postgres') { + case 'pglite': { + const { PGLiteEngine } = await import('./pglite-engine.ts'); + return new PGLiteEngine(); + } + case 'myengine': { + const { MyEngine } = await import('./my-engine.ts'); + return new MyEngine(); + } + // ... } } ``` - The factory uses dynamic imports so engines are only loaded when selected. + The factory uses dynamic imports so an engine's dependencies (e.g. the + PGLite WASM blob) are only loaded when that engine is selected. 3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }` 4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor. 5. Document in this file + add a design doc in `docs/` @@ -359,7 +321,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu | JSONB queries | GIN index | GIN index | Identical | | Concurrent access | Connection pooling | Single process | PGLite limitation | | Hosting | Supabase, self-hosted, Docker | Local file | | -| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 | +| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical | ## Future engine ideas diff --git a/docs/GBRAIN_RECOMMENDED_SCHEMA.md b/docs/GBRAIN_RECOMMENDED_SCHEMA.md index 16ca4c66c..f0549e048 100644 --- a/docs/GBRAIN_RECOMMENDED_SCHEMA.md +++ b/docs/GBRAIN_RECOMMENDED_SCHEMA.md @@ -6,6 +6,14 @@ A system prompt for any AI agent that wants to build and maintain a personal kno Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest. +> **Relationship to schema packs:** this document is the prose, paste-in +> version of the schema pattern. gbrain also ships a machine-enforced +> counterpart — schema packs (`gbrain schema`, typed pages, extraction, +> aliases, lint) — documented in `docs/architecture/schema-packs.md` and +> `docs/schema-author-tutorial.md`. The prose schema here and the active +> schema pack should describe the same brain; when you evolve one, evolve +> the other. + --- ## What this is diff --git a/docs/GBRAIN_SKILLPACK.md b/docs/GBRAIN_SKILLPACK.md index e1ad2ae44..402f52460 100644 --- a/docs/GBRAIN_SKILLPACK.md +++ b/docs/GBRAIN_SKILLPACK.md @@ -1,10 +1,10 @@ - # GBrain Skillpack: Reference Architecture for AI Agents This is a reference architecture for how a production AI agent uses gbrain as its knowledge backbone. Based on patterns from a real deployment with 14,700+ brain -files, 40+ skills, and 20+ cron jobs running continuously. +files, the 50+ bundled skills (`skills/manifest.json`), and 20+ cron jobs running +continuously. **The memex vision, realized.** Vannevar Bush imagined a device where an individual stores everything, mechanized so it may be consulted with exceeding speed. GBrain is @@ -25,6 +25,7 @@ The foundational read-write loop and data model. | [Entity Detection](guides/entity-detection.md) | Run it on every message. Capture original thinking + entity mentions | | [The Originals Folder](guides/originals-folder.md) | Capturing WHAT YOU THINK, not just what you found | | [Brain-First Lookup](guides/brain-first-lookup.md) | Check the brain before calling any external API | +| [Push-Based Context](guides/push-context.md) | volunteer_context: the brain volunteers relevant pages instead of waiting to be asked | | [Compiled Truth + Timeline](guides/compiled-truth.md) | Above the line: current synthesis. Below: append-only evidence | | [Source Attribution](guides/source-attribution.md) | Every fact needs a citation. Format and hierarchy | @@ -99,6 +100,7 @@ Keeping it running and up to date. | Guide | What It Covers | |-------|---------------| +| [Agent Bootstrap](guides/bootstrap.md) | The paste-in install: `gbrain bootstrap`, hooks, `bootstrap verify`, keyless mode | | [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files | | [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches | diff --git a/docs/GBRAIN_VERIFY.md b/docs/GBRAIN_VERIFY.md index 5bda19125..e5f049b04 100644 --- a/docs/GBRAIN_VERIFY.md +++ b/docs/GBRAIN_VERIFY.md @@ -1,5 +1,13 @@ # GBrain Installation Verification Runbook +> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install +> contract (round-trip, graph floor, and more) automatically and exits non-zero +> on failure — it is the modern first thing to run after any install. See +> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the +> **manual, deep-verification** companion: use it when `bootstrap verify` fails +> and you need to isolate which layer broke, or when you want to understand +> what "healthy" looks like check by check. + Run these checks after install to confirm every part of GBrain is working. Each check includes the command, expected output, and what to do if it fails. @@ -20,7 +28,8 @@ gbrain doctor --json **Expected:** All checks return `"ok"`: - `connection`: connected, N pages - `pgvector`: extension installed -- `rls`: enabled on all tables +- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains + skip this check; the embedded engine has no remote surface) - `schema_version`: current - `embeddings`: coverage percentage @@ -33,12 +42,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table. **Check:** Ask the agent: "What is the brain-agent loop?" -**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes -the read-write cycle: detect entities, read brain, respond with context, write -brain, sync. +**Expected:** The agent describes the read-write cycle documented in +[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities, +read brain, respond with context, write brain, sync. -**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the -install paste (read `docs/GBRAIN_SKILLPACK.md`). +**If it fails:** The agent hasn't loaded the skillpack. Have it read +`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links. --- @@ -53,8 +62,8 @@ gbrain check-update --json **Expected:** Returns JSON with `current_version`, `latest_version`, `update_available` (boolean). The cron `gbrain-update-check` is registered. -**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md -Section 17. +**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md) +for how to register the update-check cron. --- @@ -88,8 +97,9 @@ find /data/brain -name '*.md' \ Some difference is normal (files added since last sync), but if page count is less than half the file count, sync is silently skipping pages. -**If page count is way too low:** The #1 cause is an unreachable direct -connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543) +**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an +unreachable direct connection on an IPv4-only host. (PGLite brains have no +network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543) for reads, but routes migrations, DDL, and sync transactions to a derived direct connection (`db..supabase.co:5432`), which is IPv6-only. - On an IPv4-only host, reads work but sync transactions fail and silently skip @@ -122,7 +132,7 @@ This is the real test. Edit a brain page, push, wait, search. 1. Edit a page in the brain repo (e.g., correct a fact on a person's page): ```bash -# Example: fix a line in Gustaf's page +# Example: fix a line in alice-example's page cd /data/brain # Make a small edit to any .md file git add -A && git commit -m "test: verify live sync" && git push @@ -253,19 +263,23 @@ gbrain repair-jsonb Idempotent. PGLite brains always report 0 (unaffected by the original bug). -**Bonus check** — frontmatter-keyed queries actually resolve: +**Bonus check** — the doctor's dedicated JSONB scan agrees: ```bash -gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}' +gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*' ``` -If this returns rows on a brain with person pages, the JSONB path is healthy. +**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store +objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`. --- ## Quick Verification (all checks in one pass) ```bash +# 0. The one-command contract check (exits non-zero on failure) +gbrain bootstrap verify + # 1. Schema gbrain doctor --json diff --git a/docs/INSTALL.md b/docs/INSTALL.md index b70e1611f..71368d6e4 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -1,15 +1,24 @@ # Install -Three install paths. Pick one. Mix later if needed. +**Recommended door: the agent bootstrap.** Open your agent (Codex, Claude Code, +or any harness) in the folder that will become its home and paste the block +from the [README's install section](../README.md) — the agent fetches +`BOOTSTRAP_FOR_AGENTS.md` from the `latest-stable` tag, installs the CLI, +initializes a local PGLite brain, wires MCP, and isn't done until +`gbrain bootstrap verify` exits 0. Full contract, security posture, and +uninstall: [docs/guides/bootstrap.md](guides/bootstrap.md). -## 1. Run with an agent platform (recommended) +The paths below are the manual equivalents and deep-dive detail. Pick one. +Mix later if needed. + +## 1. Run with an agent platform Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)? ```bash -bun install -g github:garrytan/gbrain +bun install -g github:garrytan/gbrain#latest-stable gbrain init --pglite # 2 seconds; no server -gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace +gbrain skillpack scaffold --all # scaffolds every bundled skill (skills/manifest.json) into your agent workspace gbrain doctor # green checks all the way down ``` @@ -24,7 +33,7 @@ To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts No agent platform, just shell + MCP-aware editor. ```bash -bun install -g github:garrytan/gbrain +bun install -g github:garrytan/gbrain#latest-stable gbrain init --pglite ``` @@ -106,55 +115,31 @@ Useful for: team mounts, brain-as-a-service deployments, dev machines without di ## Verifying the install ```bash +gbrain bootstrap verify # the whole install contract; exits non-zero on failure gbrain doctor --json # full health check gbrain models # which AI models are configured for what gbrain models doctor # 1-token probe per configured model ``` -If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). +If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). For the manual check-by-check runbook, see [docs/GBRAIN_VERIFY.md](GBRAIN_VERIFY.md). ## Troubleshooting -### PGLite crashes on macOS 26.x (Tahoe) +### PGLite crashes at startup (`RuntimeError: Aborted()`) -This crash (`RuntimeError: Aborted()` at engine startup, typically first seen -after a macOS upgrade) is **not** a macOS/WASM incompatibility. The upgrade -reboot kills gbrain mid-write and tears the data dir's write-ahead log; every -subsequent open then fails WAL replay. Recovery ladder: +This crash (typically first seen after a macOS upgrade) is **not** a +macOS/WASM incompatibility — an unclean shutdown tore the data dir's +write-ahead log, and every subsequent open fails WAL replay. The short +version of the recovery ladder: -1. **Auto-repair (default):** just run any gbrain command — gbrain detects the - abort, resets the WAL in place (data preserved; a backup of the pre-repair - state is kept next to the data dir), and continues. Then run `gbrain doctor`. -2. **Manual repair:** `gbrain pglite-repair --dry-run` to diagnose, - `gbrain pglite-repair --yes` to repair in place. -3. **Rebuild:** `gbrain reinit-pglite` (wipes and re-creates the brain from - your brain repo; embedding settings default from your config). -4. **Switch engines** — if you prefer a server database anyway, native - Homebrew PostgreSQL works great and supports multiple concurrent agents: +1. **Auto-repair (default):** run any gbrain command — gbrain detects the + abort, resets the WAL in place (data preserved, backup kept), and + continues. Then run `gbrain doctor`. +2. **Manual repair:** `gbrain pglite-repair --dry-run`, then + `gbrain pglite-repair --yes`. +3. **Rebuild:** `gbrain reinit-pglite`. +4. **Switch engines:** Supabase or native Homebrew Postgres + pgvector. -```bash -# Install PostgreSQL + pgvector -brew install postgresql@17 -brew services start postgresql@17 -createdb gbrain - -# Build pgvector from source (required for vector search) -cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git -cd pgvector && make && make install -psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;" - -# Point gbrain at your local Postgres -cat > ~/.gbrain/config.json << 'EOF' -{ - "engine": "postgres", - "database_url": "postgresql://localhost:5432/gbrain", - "schema_pack": "gbrain-base-v2" -} -EOF - -# Run migrations and verify -gbrain apply-migrations --yes -gbrain doctor -``` - -Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend (plus multi-connection support: several agents can share one Postgres brain, which PGLite's single-process lock doesn't allow). +The full ladder — safety bounds, kill-switches, when WAL repair can't help, +and the Homebrew Postgres recipe — lives in +[docs/ENGINES.md](ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted). diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 2168f47ad..7d33fe541 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -26,19 +26,21 @@ Two equivalent paths: **Path B — manual lifecycle (still supported):** - `bun test` — unit tests (no database required) -- Follow the "E2E test DB lifecycle" steps above to spin up the test DB, - run `bun run test:e2e`, then tear it down. +- Follow the "E2E test DB lifecycle" steps in + [docs/TESTING.md](TESTING.md) to spin up the test DB, run + `bun run test:e2e`, then tear it down. Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests. -**Always run typecheck before pushing.** `bun test` (the bun runner) -skips TypeScript type checking — it only enforces runtime behavior. +**Always run typecheck before pushing.** Neither `bun test` (the bun runner) +nor `bun run test` gates on types — `bun run test` is just +`bash scripts/run-unit-parallel.sh` (the sharded unit runner; no typecheck, +no shell pre-checks — see the test-tier table in [docs/TESTING.md](TESTING.md)). Three ways to actually gate on types: -1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck` - plus the four shell pre-checks (`check-jsonb-pattern.sh`, - `check-progress-to-stdout.sh`, `check-trailing-newline.sh`, - `check-wasm-embedded.sh`) before the runner. Use this mid-branch. +1. `bun run verify` — runs the shell guard checks (privacy, jsonb, source-id, + progress-to-stdout, …) plus `bun run typecheck` in parallel + (`scripts/run-verify-parallel.sh`). Use this mid-branch. 2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo). 3. `bun run ci:local` — the full local CI gate from Path A. @@ -290,7 +292,8 @@ matter" with BrainBench-style before/after table, "what this means" closer, then Create a migration file at `skills/migrations/v[version].md` when a release includes changes that existing users need to act on. The auto-update agent -reads these files post-upgrade (Section 17, Step 4) and executes them. +reads these files post-upgrade (see `docs/guides/upgrades-auto-update.md`) +and executes them. **You need a migration file when:** - New setup step that existing installs don't have (e.g., v0.5.0 added live sync, @@ -343,8 +346,8 @@ canonical. ## Schema state tracking `~/.gbrain/update-state.json` tracks which recommended schema directories the user -adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17) -reads this during upgrades to suggest new schema additions without re-suggesting +adopted, declined, or added custom. The auto-update agent +(`docs/guides/upgrades-auto-update.md`) reads this during upgrades to suggest new schema additions without re-suggesting things the user already declined. The setup skill writes the initial state during Phase C/E. Never modify a user's custom directories or re-suggest declined ones. @@ -373,6 +376,46 @@ the release the same, uses that version's `CHANGELOG.md` entry as the notes (`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is missing), and attaches the compiled binaries. +### The `latest-stable` tag + +The **final step of the release job** force-advances the `latest-stable` tag to +the release commit (`git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"`). +`latest-stable` is the single sanctioned distribution ref: the README paste +block, the `BOOTSTRAP_FOR_AGENTS.md` fetch URL, and +`bun install -g github:garrytan/gbrain#latest-stable` all reference it +permanently, so paste blocks copied into the wild never rot and there is no 404 +window between VERSION landing and assets publishing. +`scripts/check-bootstrap-tag.sh` keeps the entry docs pinned to this ref. + +Because it moves ONLY after binaries + provenance attestation have fully +published, a half-built release never advances it. If the tag-advance step +alone fails, re-advance by hand (a full workflow re-run would skip — the +release already exists with all assets): + +```bash +git push origin "+refs/tags/v^{commit}:refs/tags/latest-stable" +``` + +### The `publish-template` job + +After the release job, a `publish-template` job force-pushes the rendered +agent-workspace template repo (the GitHub "Use this template" door, +`vars.TEMPLATE_REPO`, default `garrytan/gbrain-agent-template`) from CI only — +no human pushes it by hand, so what adopters clone is exactly what this repo +reviewed. It is guarded three ways: the release above fully published; the +vendored tree `templates/bootstrap/template-repo/` exists (skip, never fail, +if not); and the `TEMPLATE_REPO_PAT` secret is configured (skip if not). +Before pushing, it regenerates the template tree +(`bun run scripts/generate-template-repo.ts`) and byte-diffs it against the +vendored copy — a mismatch fails the job; regenerate + commit the vendored +tree (`scripts/check-bootstrap-templates.sh` runs the same diff offline in +`bun run verify`). + +**`TEMPLATE_REPO_PAT` scope:** a fine-grained PAT with `contents: write` on +the template repository ONLY — no other repositories, no other permissions. +Configure it as a repo secret; when absent, template publishing is disabled +and the job skips cleanly. + Why every bump, not selective: `gbrain check-update` resolves the latest version from `VERSION` on master, while binary self-update (`src/core/binary-self-update.ts`) downloads assets from `releases/latest`. @@ -395,6 +438,9 @@ Invariants: history; every new 4-segment `VERSION` mints a fresh tag. - **Permissions stay scoped.** `contents: write` lives on the release job only; everything else runs read-only. +- **Never advance `latest-stable` on a partial release.** The tag moves only + as the final release-job step, after every asset has published. Manual + re-advances must point at a fully published `v` release. ## PR descriptions cover the whole branch diff --git a/docs/TESTING.md b/docs/TESTING.md index e97f36a46..e6b4c25c0 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -11,13 +11,13 @@ Seven test command tiers, each with a clear scope: | Command | What it runs | Wallclock | When to use | |---|---|---|---| -| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. | -| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. | +| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8, and defaulted down to 4 when there's no `--shards`/`SHARDS` override; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | a few minutes on a Mac dev box | Inner edit loop. Default. | +| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. | | `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. | | `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. | | `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. | | `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. | -| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. | +| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. | ### Shell dispatch and Windows @@ -61,11 +61,16 @@ When `bun run test` finds any failure, the wrapper: 3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`). 4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so. -If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results. +If a shard hits the per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap (default 2400s — sized so the heaviest count-balanced shard finishes under 4-way contention), the wrapper classifies the kill one of two ways: + +- **EXIT-HANG → warn-pass.** If the shard's log had been silent for ≥300s at kill time AND shows zero `(fail)` markers, the shard finished all its work, leaked a handle, and never exited (a pre-existing, master-reproducible PGLite-adjacent leak — see TODOS.md "unit-shard exit hang"). The wrapper prints a `⚠️ shard N/M: EXIT-HANG ... Treating as pass-with-warning` banner, writes `EXIT-HANG (idle Ns, 0 fails) ... warn-pass` to the summary, and does NOT fail the run. Its pass counts are undercounted (bun never printed its final summary). Bun's per-test `--timeout` turns a genuinely hung TEST into a printed `(fail)` — new output — so this classification cannot mask a hung test; the residual maskable case is a file-level import hang in the very last file, which the banner keeps visible. +- **WEDGED → hard failure.** Anything else (failures present, or the log was still growing) writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log with the last 50 lines of the shard log, marks the run failed, and proceeds with other shards' results. + +Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT a test failure — don't burn time bisecting it; a `WEDGED` line is. ### File taxonomy -- `*.test.ts` → fast loop (parallel 8-shard fan-out). +- `*.test.ts` → fast loop (parallel sharded fan-out, default 4 shards). - `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock). - `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake). - `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset. @@ -83,6 +88,8 @@ Any change under `skills/` must regenerate it: `bun run scripts/generate-skills- ### Test-isolation lint and helpers +**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules. + The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped): | Rule | What it bans | Fix | diff --git a/docs/UPGRADING_DOWNSTREAM_AGENTS.md b/docs/UPGRADING_DOWNSTREAM_AGENTS.md index 053eb22c3..4da58eeb2 100644 --- a/docs/UPGRADING_DOWNSTREAM_AGENTS.md +++ b/docs/UPGRADING_DOWNSTREAM_AGENTS.md @@ -1,5 +1,13 @@ # Upgrading Downstream Agents +> **Currency note:** this file is an append-only historical log and stopped +> receiving new sections after v0.36.5.0. **The canonical, maintained upgrade +> channel is `skills/migrations/v*.md`** (the agent-executed migration files +> that `gbrain upgrade` / `gbrain post-upgrade` route through), plus +> `CHANGELOG.md` for what each release changed. Use this file only to catch a +> long-diverged fork up through the versions it covers; for anything after +> v0.36.5.0, walk the migration files and CHANGELOG instead. + GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments, agent forks of any kind) often **copy** these skill files into their own workspace and diverge over time — adding agent-specific phases, removing irrelevant ones, tightening diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 3a433fa0f..b6b29e02a 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -10,36 +10,35 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `docs/operations/conversation-parser-llm-fallback.md` — operator and maintainer contract for the default-off LLM parse fallback: exact config key, deterministic-first dispatch boundary, sampled data surface, untrusted-content prompt handling, page-date/cache-key coupling, timestamp validation, cache/checkpoint behavior, observability, limitations, and focused test commands. -- `src/commands/serve-http.ts` confidential revoke extension — a pre-router `/revoke` handler validates the RFC 7009 body, verifies hash-only secrets for both `client_secret_post` and `client_secret_basic`, rejects mixed authentication, preserves the SDK path for public clients, and separates opaque client-auth failures from retryable/backend failures. OAuth metadata advertises both confidential methods. Pinned by `test/e2e/serve-http-oauth.test.ts`. -- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. OAuth `whoami` exposes the authenticated `AuthInfo.sourceId` and `AuthInfo.allowedSources` grants as `source_id` and `federated_read`; absent grants serialize fail-closed as `null` and `[]`, while local, legacy, and stdio response shapes stay unchanged. `enforceSubagentSlugFence(ctx, slug, opName)` is the shared fail-closed subagent write fence: when `viaSubagent` and `allowedSlugPrefixes` is set, the slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Both `put_page` and `add_timeline_entry` (subagent-allowlisted) route through it. Auto-link skipped only when `remote=true && !trustedWorkspace`. `enforceClientSlugFence(ctx, slug, opName)` is the OAuth-client write fence: when `ctx.auth.boundSlugPrefixes` is present (threaded from `oauth_clients.bound_slug_prefixes` at token-verification time), every direct slug-mutating write op — `put_page`, `delete_page`, `restore_page`, `add_tag`, `remove_tag`, `add_link`/`remove_link` (`from` endpoint only; linking TO a readable page is a reference), `add_timeline_entry`, `revert_version`, `put_raw_data` — rejects out-of-prefix slugs with `permission_denied`, BEFORE each op's dry-run short-circuit. Plain-startsWith semantics matching `submit_agent`'s check for the same column (NOT the glob grammar of the subagent allow-list); empty-array binding is deny-all (fail-closed); no auth / no binding = no fence. The match rule itself lives in the exported `slugUnderBoundPrefixes(prefixes, slug)` so non-op write surfaces reuse it verbatim. It is BOUNDARY-AWARE (a prefix matches whole segments, so `emp-alice` does not admit `emp-alice-2/…`), lowercases both sides (stored slugs are lowercased by `validateSlug`, so comparing the caller's raw string let a mixed-case slug commit and only then trip the resolved-slug re-check), accepts BOTH the trailing-slash and the v85 `/*` glob spelling via `normalizeSlugPrefix` (the column predates this fence as submit_agent's binding, so one stored value must mean one span of slugs on both paths), and ignores empty-string prefixes. `assertValidSlugPrefixes` (`oauth-provider.ts`) rejects empty, whitespace-bearing, non-lowercase, and boundary-less entries at registration and rescope. `submit_agent` applies the same boundary-aware rule when validating a requested prefix against the binding, normalizes trailing-slash prefixes to the glob form `matchesSlugAllowList` expects before handing them to the child job, and collapses an EXPLICIT empty `allowed_tools`/`allowed_slug_prefixes` to the binding (the worker reads empty as "full registry" / "legacy `wiki/agents//` namespace", so `??` — which only substitutes null/undefined — left a vacuous-subset bypass). `put_page` additionally fences the RESOLVED slug when importFromContent's dedup pre-check redirects the write to a different page (same content_hash / `frontmatter.id`), since the disk write-through runs against that slug. That re-check applies whichever confinement the CALLER is under — OAuth binding and/or subagent allow-list/legacy namespace — via `slugOutsideCallerFence(ctx, slug)`, which composes `slugUnderBoundPrefixes` with the subagent fence's own match rule: the delegated `submit_agent` → subagent context carries `viaSubagent` + `allowedSlugPrefixes` but NO `auth`, so an auth-only test let a slug-bound client holding `agent` scope reach an out-of-fence page simply by delegating the write. Denials never name the resolved slug (it would be a slug-enumeration oracle). Pinned by `test/put-page-dedup-fence.test.ts`. `CLIENT_FENCED_WRITE_OPS` + `enforceBoundClientOpAllowList(auth, op)` are the fail-closed companion, applied once in `src/mcp/dispatch.ts` (the choke point both MCP transports share): a slug-bound client calling ANY `write`/`admin` op not on the allow-list gets `permission_denied`. This covers the ops that write by a key other than a slug and therefore cannot be fenced — `extract_entities`/`extract_facts` (mutate `people/*`, `companies/*`), `forget_fact` (numeric fact id, crosses sources), `ontology_propose` — and makes a write op added later denied-by-default instead of silently unfenced. `think` is on the allow-list because remote callers cannot persist from it. Pinned by `test/client-slug-fence.test.ts` and over-the-wire by `test/e2e/qm-provisioning.test.ts`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path, plus the by-slug reads `get_tags`/`get_links`/`get_backlinks`/`get_timeline` (and `get_page`'s tag fetch, which resolves against the concrete page's own `source_id`). `linkReadScopeOpts(ctx)` is the link-read sibling for `get_links`/`get_backlinks`: a link row references three pages (from/to/origin), and the engine's federated (`sourceIds[]`) branch scopes ALL THREE while its scalar (`sourceId`) branch scopes only the near endpoint (by design — trusted internal callers like `reconcileLinks` and back-link validators call the engine with a scalar scope and need the cross-source view). For an UNTRUSTED remote caller (`ctx.remote !== false`) carrying only a scalar scope it promotes that scope to a single-element `sourceIds:[id]`, routing them through the all-endpoint branch so a far/origin slug from outside the grant isn't disclosed; a federated array passes through unchanged and trusted local CLI (`ctx.remote === false`) keeps the scalar cross-source view. `thinkSourceScopeOpts(ctx)` maps the same precedence ladder onto `runThink`'s public options (`allowedSources`/`sourceId`) so the `think` op's gather and trajectory stages inherit the caller's source grant. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. The by-slug read methods carry the same federated axis: `getTags`/`getLinks`/`getBacklinks` opts and `TimelineOpts` (consumed by `getTimeline`) accept `sourceIds?: string[]` taking precedence over the scalar `sourceId` (`source_id = ANY($::text[])` scoping the slug→page-id lookup); the link reads (`getLinks`/`getBacklinks`) scope ALL THREE endpoints (from/to/origin) on the federated branch while the scalar branch scopes only the near endpoint for trusted internal cross-source callers. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. +- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. OAuth `whoami` exposes the authenticated `AuthInfo.sourceId` and `AuthInfo.allowedSources` grants as `source_id` and `federated_read`; absent grants serialize fail-closed as `null` and `[]`, while local, legacy, and stdio response shapes stay unchanged. `enforceSubagentSlugFence(ctx, slug, opName)` is the shared fail-closed subagent write fence: when `viaSubagent` and `allowedSlugPrefixes` is set, the slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Both `put_page` and `add_timeline_entry` (subagent-allowlisted) route through it. Auto-link skipped only when `remote=true && !trustedWorkspace`. `enforceClientSlugFence(ctx, slug, opName)` is the OAuth-client write fence: when `ctx.auth.boundSlugPrefixes` is present (threaded from `oauth_clients.bound_slug_prefixes` at token-verification time), every direct slug-mutating write op — `put_page`, `delete_page`, `restore_page`, `add_tag`, `remove_tag`, `add_link`/`remove_link` (`from` endpoint only; linking TO a readable page is a reference), `add_timeline_entry`, `revert_version`, `put_raw_data` — rejects out-of-prefix slugs with `permission_denied`, BEFORE each op's dry-run short-circuit. Plain-startsWith semantics matching `submit_agent`'s check for the same column (NOT the glob grammar of the subagent allow-list); empty-array binding is deny-all (fail-closed); no auth / no binding = no fence. The match rule itself lives in the exported `slugUnderBoundPrefixes(prefixes, slug)` so non-op write surfaces reuse it verbatim. It is BOUNDARY-AWARE (a prefix matches whole segments, so `emp-alice` does not admit `emp-alice-2/…`), lowercases both sides (stored slugs are lowercased by `validateSlug`, so comparing the caller's raw string let a mixed-case slug commit and only then trip the resolved-slug re-check), accepts BOTH the trailing-slash and the v85 `/*` glob spelling via `normalizeSlugPrefix` (the column predates this fence as submit_agent's binding, so one stored value must mean one span of slugs on both paths), and ignores empty-string prefixes. `assertValidSlugPrefixes` (`oauth-provider.ts`) rejects empty, whitespace-bearing, non-lowercase, and boundary-less entries at registration and rescope. `submit_agent` applies the same boundary-aware rule when validating a requested prefix against the binding, normalizes trailing-slash prefixes to the glob form `matchesSlugAllowList` expects before handing them to the child job, and collapses an EXPLICIT empty `allowed_tools`/`allowed_slug_prefixes` to the binding (the worker reads empty as "full registry" / "legacy `wiki/agents//` namespace", so `??` — which only substitutes null/undefined — left a vacuous-subset bypass). `put_page` additionally fences the RESOLVED slug when importFromContent's dedup pre-check redirects the write to a different page (same content_hash / `frontmatter.id`), since the disk write-through runs against that slug. That re-check applies whichever confinement the CALLER is under — OAuth binding and/or subagent allow-list/legacy namespace — via `slugOutsideCallerFence(ctx, slug)`, which composes `slugUnderBoundPrefixes` with the subagent fence's own match rule: the delegated `submit_agent` → subagent context carries `viaSubagent` + `allowedSlugPrefixes` but NO `auth`, so an auth-only test let a slug-bound client holding `agent` scope reach an out-of-fence page simply by delegating the write. Denials never name the resolved slug (it would be a slug-enumeration oracle). Pinned by `test/put-page-dedup-fence.test.ts`. `CLIENT_FENCED_WRITE_OPS` + `enforceBoundClientOpAllowList(auth, op)` are the fail-closed companion, applied once in `src/mcp/dispatch.ts` (the choke point both MCP transports share): a slug-bound client calling ANY `write`/`admin` op not on the allow-list gets `permission_denied`. This covers the ops that write by a key other than a slug and therefore cannot be fenced — `extract_entities`/`extract_facts` (mutate `people/*`, `companies/*`), `forget_fact` (numeric fact id, crosses sources), `ontology_propose` — and makes a write op added later denied-by-default instead of silently unfenced. `think` is on the allow-list because remote callers cannot persist from it. Pinned by `test/client-slug-fence.test.ts` and over-the-wire by `test/e2e/qm-provisioning.test.ts`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path, plus the by-slug reads `get_tags`/`get_links`/`get_backlinks`/`get_timeline` (and `get_page`'s tag fetch, which resolves against the concrete page's own `source_id`). `linkReadScopeOpts(ctx)` is the link-read sibling for `get_links`/`get_backlinks`: a link row references three pages (from/to/origin), and the engine's federated (`sourceIds[]`) branch scopes ALL THREE while its scalar (`sourceId`) branch scopes only the near endpoint (by design — trusted internal callers like `reconcileLinks` and back-link validators call the engine with a scalar scope and need the cross-source view). For an UNTRUSTED remote caller (`ctx.remote !== false`) carrying only a scalar scope it promotes that scope to a single-element `sourceIds:[id]`, routing them through the all-endpoint branch so a far/origin slug from outside the grant isn't disclosed; a federated array passes through unchanged and trusted local CLI (`ctx.remote === false`) keeps the scalar cross-source view. `thinkSourceScopeOpts(ctx)` maps the same precedence ladder onto `runThink`'s public options (`allowedSources`/`sourceId`) so the `think` op's gather and trajectory stages inherit the caller's source grant. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). (orphans fix): `findOrphanPages` (both engines) filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side, so soft-deleted pages don't appear as orphans AND links from soft-deleted source pages don't suppress live pages from orphan results. Pinned by `test/orphans.test.ts`'s soft-delete cases. 9 MCP ops: `get_active_schema_pack`, `list_schema_packs`, `schema_stats`, `schema_lint`, `schema_graph`, `schema_explain_type`, `schema_review_orphans` (all read-scope, NOT localOnly), plus `schema_apply_mutations` (admin scope, NOT localOnly so remote agents can author packs over HTTPS MCP — batched, one MCP tool taking a `mutations[]` array, delegating to `applyMutationsAtomic` for a single lock + single read + single write across the whole batch; a mid-batch failure reports `mutations_applied: 0` + `pack_unchanged: true` (never a `partial_results` list — nothing is written until every mutation validates), audit log captures `actor: mcp:`) and `reload_schema_pack` (admin, NOT localOnly). Trust posture: per-call `schema_pack` opt STAYS rejected for remote callers via `op-trust-gate.ts`. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. The by-slug read methods carry the same federated axis: `getTags`/`getLinks`/`getBacklinks` opts and `TimelineOpts` (consumed by `getTimeline`) accept `sourceIds?: string[]` taking precedence over the scalar `sourceId` (`source_id = ANY($::text[])` scoping the slug→page-id lookup); the link reads (`getLinks`/`getBacklinks`) scope ALL THREE endpoints (from/to/origin) on the federated branch while the scalar branch scopes only the near endpoint for trusted internal cross-source callers. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. two interface members: (1) optional `findDuplicatePage?(sourceId, {hash, frontmatterId?}): Promise<{slug, id} | null>` (identity precedence is content_hash OR frontmatter->>'id', both with `deleted_at IS NULL`); (2) `resolveSlugs(partial, opts?)` extended with `{sourceId?, sourceIds?}` so the MCP fuzzy `get_page` path scopes by source (field names match `sourceScopeOpts(ctx)` output so handlers spread directly; back-compatible — no opts gives prior behavior). Plus a stable tiebreaker `ORDER BY score DESC, page_id ASC, chunk_id ASC` in `searchVector` in both engines: on a score tie (basis-vector eval fixtures) older `page_id` wins, closing the planner-non-determinism class where a new index on `pages` could flip ranking on tied scores. - `src/core/engine-constants.ts` — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FIVE sinks register at module import: `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`), `context/volunteer-events.ts` (order 4, #2095 — batched volunteer-event INSERTs, drained like the rest). Every cli.ts teardown site reaches it through `finishCliTeardown` (`src/core/cli-force-exit.ts`), which drains the registry before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). Exports `backgroundWorkSinkCount()` so the teardown helper computes its backstop deadline from the registered sink count. CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the teardown backstop honors an errored op's exit code so a hung disconnect can't mask failure as success (see `cli-force-exit.ts`); `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of stalling the whole op (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. - `src/core/search/graph-signals.ts` — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring at full score, demote the rest). All three inherit the floor-ratio gate preventing weak pages from being boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width`. `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (incl. the IRON-RULE floor-gate regression). -- `src/core/search/hybrid.ts` extension — `runPostFusionStages` has a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Each post-fusion stage stamps its multiplier: `applyBacklinkBoost`→`backlink_boost`, `applySalienceBoost`→`salience_boost`, `applyRecencyBoost`→`recency_boost`. `applyReranker` (earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution powers `gbrain search --explain` — every boost surface carries its own field so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts`. -- `src/core/search/mode.ts` extension — `graph_signals: boolean` knob in `ModeBundle` (defaults: `conservative=false`, `balanced=true`, `tokenmax=true`). `KNOBS_HASH_VERSION` appends a `gs=` parts entry per the cache-key contamination convention so a graph-on cache write can't be served to a graph-off lookup. `SearchKeyOverrides` + `SearchPerCallOpts` + `loadOverridesFromConfig` + `SEARCH_MODE_CONFIG_KEYS` + `resolveSearchMode` + `attributeKnob` all carry the field. Opt-out: `gbrain config set search.graph_signals false`. Mid-deploy `query_cache` rows from before the upgrade hash differently — natural row segregation, clears within `cache.ttl_seconds` (3600s default). +- `src/core/search/mode.ts` — Named search-mode bundles + the search cache key. `MODE_BUNDLES` (conservative/balanced/tokenmax) and the resolution chain (per-call `SearchOpts` → per-key `search.*` config → bundle → balanced fallback) resolve every search knob; `knobsHash` folds every result-shaping knob into the `query_cache` key, and `KNOBS_HASH_VERSION` (exported from this file — the single source of truth for the current cache-key version) is bumped whenever a new knob shapes results so stale cache rows become unreachable. `graph_signals: boolean` knob in `ModeBundle` (defaults: `conservative=false`, `balanced=true`, `tokenmax=true`). `KNOBS_HASH_VERSION` appends a `gs=` parts entry per the cache-key contamination convention so a graph-on cache write can't be served to a graph-off lookup. `SearchKeyOverrides` + `SearchPerCallOpts` + `loadOverridesFromConfig` + `SEARCH_MODE_CONFIG_KEYS` + `resolveSearchMode` + `attributeKnob` all carry the field. Opt-out: `gbrain config set search.graph_signals false`. Mid-deploy `query_cache` rows from before the upgrade hash differently — natural row segregation, clears within `cache.ttl_seconds` (3600s default). `title_boost: number | undefined` knob in `ModeBundle` (default `1.25` for all three modes; multiplier for the post-fusion title-phrase boost). Override chain: per-call `SearchOpts` → `search.title_boost` config (clamped `[1.0, 5.0]`) → bundle. `KNOBS_HASH_VERSION` appends a `tib=` parts entry so a title-boost-on cache write can't be served to a title-boost-off lookup. `SEARCH_MODE_CONFIG_KEYS` gains `search.title_boost`. Cross-modal knobs in `ModeBundle`: `cross_modal_both_text_weight`/`cross_modal_both_image_weight` (weighted RRF for 'both' modality, defaults 0.6/0.4), `image_query_text_refinement_weight`/`image_query_image_refinement_weight` (hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (unified-column routing flags), `cross_modal_llm_intent` (opt-in LLM escalation). `SEARCH_MODE_CONFIG_KEYS` carries the corresponding config keys, and the modality knobs participate in `knobsHash` so a cached text-mode result can't be served to an image-mode caller. - `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), passes the rolling window (`getWindowTurns`, last 12 user/assistant turns; the reflex slices to its configured `retrieval_reflex_window_turns`), and appends the pointer block. `warmReflex()` fires at construction. - `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped) + `extractCandidatesFromWindow(turns)` (#2095: merges per-turn extraction across the last N turns by normalizeAlias form with occurrence/newest-turn/user-mention metadata; salience-ordered — recency > frequency > user-role — so the cap drops stale assistant chatter first). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); pointers carry `source_id`/`arm`/`confidence`/`matchedNorm` (#2095 — `ARM_CONFIDENCE` alias 0.9 / title 0.8 / slug-suffix 0.6 lives next to the arm definitions; arm-2 provenance classified in JS since the combined OR can't report which predicate matched); opts: `sourceIds?` federated scope (alias arm loops per source, arm 2 uses `source_id = ANY`), `suppression?` ('slug-and-title' legacy default; 'slug-only' REQUIRED under windowing — the title rule would suppress every entity merely mentioned in a prior window turn), ambient-channel event logging is DELIVERY-side, not in-resolver — `logDeliveredReflexPointers(engine, pointers)` fires only once a block is actually handed to the consumer (serve's resolve-IPC `onDelivered` hook post-write; `buildReflexAddition` post-timeout on the direct rung), so abandoned/timed-out blocks never pollute the volunteered-vs-used stats; synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync); windowed extraction when `windowTurns` present and `retrieval_reflex_window_turns` (default 4; 1 = exact legacy behavior) > 1 — switches suppression to slug-only; accept-side reflex-channel logging fires after the per-turn timeout admits the block (direct-Postgres rung only — IPC logs server-side at delivery; host-injected resolvers are a documented gap). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` + `retrieval_reflex_window_turns` in `src/core/config.ts` (env `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`). `volunteer.ts` (#2095): `parseWindow` (lenient `user:`/`assistant:` prefixes, unprefixed → one user turn), `volunteerContext` (extract → resolve → +0.05 multi-turn/newest-turn boost → `min_confidence` 0.7 gate → cap 3/5; deterministic rationale strings, never raw conversation text; slug-only suppression), `volunteerUsageStats` (per-arm/channel precision from the `pages.last_retrieved_at > volunteered_at` join — APPROXIMATE: the 5-min last-retrieved throttle causes false negatives, unrelated reads false positives). `volunteer-events.ts` (#2095): `insertVolunteerEvents` (ONE multi-row parameterized INSERT), `logVolunteerEventsFireAndForget` + bounded drain registered as the `volunteer-events` background-work sink (order 4), `purgeStaleVolunteerEvents` (90-day GC, called from the dream cycle's purge phase). Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`, `test/volunteer-context.test.ts`, `test/e2e/volunteer-context-postgres.test.ts`. - `src/commands/watch.ts` — `gbrain watch` (#2095): the push transport. Reads turns from stdin as they arrive (`user:`/`assistant:` prefixes; unprefixed = user turn), keeps a rolling in-process window (`--window-turns`, default 4), calls `volunteerContext` per turn, streams pointers to stdout (`--json` for JSONL with turn attribution), logs `channel: 'watch'` events with a per-session id. Session dedupe feeds already-pushed slugs back as priorContext so the core's slug-only suppression dedupes. Blocks in the stdin iteration (interactive alive until Ctrl-C/Ctrl-D; piped ends at EOF) — deliberately NOT in DAEMON_COMMANDS; SIGINT closes the stream so teardown flows through finishCliTeardown. Per-turn resolution failures are fail-open. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS (thin clients use the `volunteer_context` MCP op). Pinned by `test/watch-command.test.ts`. -- `src/commands/integrations.ts` — recipe install. The resolver-row install fence is keyed by `manifest.recipe` (`gbrain::resolver-rows`), so a second `copy-into-host-repo` recipe no longer writes a block mislabeled with the first recipe's name. Pinned by `test/integrations-install.test.ts`. Health-check DSL includes the staleness-aware `heartbeat_max_age` type (#2787): declares the sense's expected cadence (`max_age: 48h`), and `integrations doctor` FAILS when the newest heartbeat event is older — the only check type that catches a green-but-dead sense (all others are point-in-time). Not embedded-gated (reads only the local heartbeat file). Recipe frontmatter carries `output_paths` (repo-relative dirs the collector writes, e.g. calendar-to-brain → `daily/calendar/`); `getConfiguredCollectorOutputs()` surfaces them for the #2788 db_only-collision check/warning. Pinned by `test/integrations-heartbeat-max-age.test.ts`. +- `src/commands/integrations.ts` — recipe install. The resolver-row install fence is keyed by `manifest.recipe` (`gbrain::resolver-rows`), so a second `copy-into-host-repo` recipe no longer writes a block mislabeled with the first recipe's name. Pinned by `test/integrations-install.test.ts`. Health-check DSL includes the staleness-aware `heartbeat_max_age` type (#2787): declares the sense's expected cadence (`max_age: 48h`), and `integrations doctor` FAILS when the newest heartbeat event is older — the only check type that catches a green-but-dead sense (all others are point-in-time). Not embedded-gated (reads only the local heartbeat file). Recipe frontmatter carries `output_paths` (repo-relative dirs the collector writes, e.g. calendar-to-brain → `daily/calendar/`); `getConfiguredCollectorOutputs()` surfaces them for the #2788 db_only-collision check/warning. Pinned by `test/integrations-heartbeat-max-age.test.ts`. Standalone integration recipe management (no DB needed). Exports `getRecipeDirs()` (trust-tagged recipe sources), SSRF helpers (`isInternalUrl`, `parseOctet`, `hostnameToOctets`, `isPrivateIpv4`). Only package-bundled recipes are `embedded=true`; `$GBRAIN_RECIPES_DIR` and cwd `./recipes/` are untrusted and cannot run `command`/`http`/string health checks. - `src/core/audit/audit-writer.ts` — shared JSONL audit primitive consolidating the hand-rolled audit modules. Exports `createAuditWriter({kind, recordSchema})` returning `{log, readRecent}` plus shared helpers `computeIsoWeekFilename(kind, now?)` and `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). ISO-week file rotation; best-effort writes (stderr warn on failure, never throws); read-path scans current-week + previous-week files for boundary spans. Refactored onto it for parity: `src/core/rerank-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/minions/handlers/shell-audit.ts`, `src/core/minions/handlers/supervisor-audit.ts`, `src/core/facts/phantom-audit.ts` (each module's public API preserved bit-for-bit). The `graph-signals-failures` audit (`logGraphSignalsFailure`) uses the same primitive. One hand-rolled audit remains at `src/core/skillpack/audit.ts`. Pinned by `test/audit/audit-writer.test.ts`. - `src/core/cli-force-exit.ts` (#2084) — single owner of one-shot CLI exit + teardown, designed as a PAIR with the `import.meta.main` seam at the bottom of `src/cli.ts`. `finishCliTeardown({engine, drainTimeoutMs?})` is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into `process.exitCode`) whose deadline is COMPUTED from the bounds it guards (`computeTeardownDeadlineMs` = sinks × drainTimeoutMs + facts-abort grace + 2 × pool-end bound + slack, floor 10s; `GBRAIN_TEARDOWN_DEADLINE_MS` env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (`setCliExitVerdict`/`currentExitCode`; mirror-writes `process.exitCode` but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into `process.exitCode` at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot, doctor's FAIL verdict, extract, and cli.ts's swept inner exits — friction, claw-test, smoke-test, the no-DB eval runners, status/status-thin, whoknows-thin) calls `setCliExitVerdict`; `test/cli-exit-verdict-pin.test.ts` greps src/ so the next raw `process.exitCode =` write fails CI instead of silently reporting success on failure. The deadline arms at TEARDOWN start, never before the op handler (the pre-#2084 placement measured handler + teardown combined, so PgBouncer deployments paid a flat 10s force-exit tax on every query and any >10s op was killed mid-run with exit 0). All nine cli.ts disconnect sites route through it; the ONE process exit happens in cli.ts's `main().then/catch` via `flushThenExit(currentExitCode())`, gated by `shouldForceExitAfterMain()` (daemon list: `serve`) — the CLI never waits for Bun's event loop to drain, because `endPoolBounded` deliberately races past stuck PgBouncer sockets that would keep it alive. `flushThenExit(code)` fences stdout+stderr (`write('', cb)` raced with an unref'd guard, EPIPE-safe both sync and async) then holds a REF'D aliveness grace for non-TTY stdio before `process.exit` — Bun delivers queued pipe writes only while the process is alive (no flush API reaches `process.stdout`'s native queue; write callbacks fire on accept, not delivery), so the grace IS the flush (#1959 truncation class). Scope claim is deliberately cli.ts-only: command modules' mid-run engine lifecycles stay local (process-exit semantics inside them would be wrong) and are absorbed by the final explicit exit. Pinned by `test/cli-finish-teardown.test.ts`, `test/flush-then-exit-harness.test.ts` (real spawned-Bun pipe semantics), `test/cli-should-force-exit.test.ts`, `test/cli-pipe-truncation.test.ts` (real-CLI piped --tools-json byte-stable), `test/cli-exit-verdict-pin.test.ts`, the `#2084` describes in `test/fix-wave-structural.test.ts` + `test/e2e/pglite-cli-exit.serial.test.ts`, and `test/e2e/pgbouncer-teardown.test.ts` (CI transaction-mode pooler — the #1972/#2015/#2084 class, finally reproducible in CI). -- `src/core/cli-options.ts` extension — `CliOptions` gains `explain: boolean`. `parseGlobalFlags` recognizes `--explain` anywhere in argv (stripped before command dispatch). `src/cli.ts` `formatResult` for `search` + `query` cases routes to `formatResultsExplain` from `src/core/search/explain-formatter.ts` when `CliOptions.explain` is set; falls through to the existing JSON / human formatters otherwise. -- `src/commands/search.ts:gbrain search stats` extension — `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property; `_meta.metric_glossary` adds `graph_signals.enabled` + `graph_signals.failures_by_reason`. Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts`. -- `src/commands/doctor.ts` extension — silent-failure batch (#2250/#2784/#2788): `content_hash_duplicates` (single GROUP BY over `(source_id, content_hash)` with FILTER aggregates — never N² — flagging hash groups that hold BOTH a bare and a path-prefixed slug, the wrong-import-root pattern; warn carries sample pairs + the `pages delete` → `purge-deleted --older-than 0` remediation); `undeclared_db_only_pages` (per source with a local repo: markdown pages with no backing file outside every declared + derive-phase-default db_only prefix — the one check deliberately allowed to stat the repo); `db_only_collector_collision` (configured recipe `output_paths` inside a declared db_only dir — auto-gitignore means sync AND import silently skip the collector's files; same warning fires in sync's `manageGitignore` at config-write time). All warn-level, engine-parity pinned by `test/e2e/doctor-silent-death-parity.test.ts`; units in `test/doctor-silent-death-checks.test.ts`. -- `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. +- `src/commands/search.ts:gbrain search stats` — `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property; `_meta.metric_glossary` adds `graph_signals.enabled` + `graph_signals.failures_by_reason`. Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`). - `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that classifies the failure and, for the `wasm-abort` verdict on a persistent data dir (torn WAL/checkpoint state after an unclean shutdown — the #223/#1670/#2575 class, historically misdiagnosed as a macOS WASM bug), runs in-place auto-repair via `attemptWalRepairAndRetry` (static import from `pglite-repair.ts`, #3596 engine-live rule; the retry create is `preservingProcessExitCode`-wrapped; success sets the public `walRepairReceipt` field + prints `buildWalRepairNotice` to stderr and returns with the lock held). The seam never throws, so every non-repaired path funnels through the single lock-release-then-throw site; repair refuses when the lock was acquired by reaping (`LockHandle.reaped` → `'possibly-live-writer'`), when disabled (`GBRAIN_PGLITE_WAL_REPAIR=off`), on layout-validation failure, or inside the post-failure cooldown. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `searchTakes`/`searchTakesVector` take full `SearchOpts` and apply the standard source-scope predicates (federated `sourceIds[]` wins over scalar `sourceId`, via the joined page's `source_id`) alongside the holder allow-list — parity SQL in postgres-engine.ts; pinned by `test/e2e/think-source-isolation-pglite.test.ts`. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`, `timeline_entries.event_page_id` — column-only, migration v121 stays the source of truth for its FK + indexes) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. `PGlite.create()` runs inside `preservingProcessExitCode` (#2084): PGLite's Emscripten runtime writes its own status into `process.exitCode` (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close — and assigning `undefined` cannot clear a prior value), which would otherwise silently clobber an errored op's exit 1 back to 0. The wrapper keeps the global tidy for external readers; `db.close()` stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI's exit verdict is immune either way — it lives in the gbrain-owned channel in `cli-force-exit.ts` and never reads `process.exitCode` back. Exports `classifyPgliteInitError(message): 'bunfs' | 'wasm-abort' | 'corrupt' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original, platform?, ctx?)` + `stringifyPgliteInitError(err)` + `buildWalRepairNotice(receipt)` + the `PgliteInitRepairContext` type, routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `corrupt` — 58P01/`internal_load_library`/missing vector type, catalog corruption WAL repair can't fix — stays matched BEFORE the wasm arm and routes to `reinit-pglite`; `wasm-abort` matches the real production shapes `Aborted()`/`RuntimeError`/`unreachable` plus legacy signatures, names the corrupt-WAL root cause + the recovery ladder (`pglite-repair` → rebuild → engine switch) + what auto-repair did per `ctx` incl. the honesty-critical `failed-not-restored` arm, and keeps the #223 link; `unknown` is platform-gated per #2674). `stringifyPgliteInitError` also surfaces message-less Emscripten objects (`ErrnoError (errno N)`) instead of `[object Object]`. Pinned by `test/pglite-init-classifier.test.ts` + `test/pglite-wal-repair.serial.test.ts` + `test/fix-wave-structural.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers). Engine-path helper dependencies (`retry`, ontology, recency decay) avoid dynamic `import()`; the only lazy dynamic imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass. - `src/core/pglite-lock.ts` — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at, command, subcommand}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer; informational). A waiting acquirer reaps a holder ONLY when its PID is dead — a LIVE holder is NEVER stolen, regardless of how stale its heartbeat is (#2348). A live `gbrain serve` holder is identified from the parsed `subcommand` and reported immediately with separate CLI-retry and MCP-tool choices; other live holders keep the bounded wait. The heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/CHECKPOINTs, so a working `dream`/`embed` holder can look stale while alive; the old steal-on-stale-heartbeat grace let a second OS process open the same data dir and corrupt the catalog + pgvector extension (58P01 / `internal_load_library` / `type "vector" does not exist`), recoverable only by wipe+restore. A wedged-but-alive or PID-reused holder is never stolen: serve-tagged holders report immediately, while other holders time out with a message naming the PID. Each holder carries an ownership token (`:`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it. In-memory engines take no lock. There is deliberately NO same-process reentrancy or same-PID special case: a second `acquireLock` from the process that already holds the lock waits out the timeout like any other live holder (#1963 was this shape — a command double-connecting a second engine on the same data dir; the fix is to reuse the connected engine at the dispatch layer, never to soften the lock). `LockHandle.reaped` marks an acquisition that reaped a prior holder's lock (dead-PID reap or corrupt-lock-file removal — the only reaps that exist post-#2348); the WAL auto-repair gate refuses to run surgery on a reaped acquisition since a corrupt lock file cannot prove its holder is dead. A corrupt-lock reap ALSO writes a persisted marker (`.lock-reap.json`, read via exported `msSinceLastReap`) so the NEXT process's clean acquisition is still repair-quarantined for 10 minutes — the in-process flag alone let the reaper's successor run surgery under a possibly-live writer; dead-PID reaps (affirmative ESRCH verdict; EPERM reads as ALIVE) deliberately skip the marker so dead-holder recovery stays one-failed-command-plus-one-re-run. Heartbeat refreshes write via tmp+rename (a torn in-place write could be read mid-flight by a polling acquirer and misclassify a HEALTHY live holder as a corrupt lock). Pinned by `test/pglite-lock.test.ts`. A corrupted store surfaces a `reinit-pglite` recovery hint via `classifyPgliteInitError`'s `corrupt` verdict in `pglite-engine.ts`. - `src/core/pglite-resetwal.ts` — pg_resetwal for PGLite NodeFS data dirs, in TypeScript (ported from electric-sql/pglite PR #994 by @yestheboxer, Apache-2.0, rejected upstream as "should be a separate tool" — gbrain is that tool). Validates the PG17 pg_control layout fail-closed (`WalResetUnsupportedError` on any unsupported shape — PG_VERSION ≠ 17, control ≠ 8192 bytes, control version ≠ 1700, bad seg/block size), removes stale postmaster.pid + old WAL segments + archive_status/summaries entries, writes a replacement shutdown-checkpoint WAL segment + CRC32C'd pg_control. Both file writes are atomic + durable (tmp cleared then opened `'wx'` so a pre-planted symlink at the predictable tmp name can never redirect the write, + fsync(tmp) + rename + fsync(parent dir)); write order is segment-first/control-last so a mid-write kill leaves a state that still fails startup and the next attempt re-runs (idempotent — a torn pair can never claim success). WAL segment size is capped at 64MB (pglite ships 16MB; the Postgres-general 1GB bound would let a corrupt-but-plausible control field drive a 1GB allocation on the repair path). Exports the shared PG17 layout literals (`PG_CONTROL_FILE_SIZE`, `isWalSegmentName`) consumed by pglite-repair.ts. LAYOUT COUPLING: any pglite bump past PG17 must revisit this file together with the `./vector` export blocker (TODOS.md "pglite upgrade blocker" entry). Pinned by `test/pglite-resetwal.test.ts`. - `src/core/pglite-repair.ts` — WAL-repair orchestrator wrapping the resetWal port with the safety layers that make it runnable automatically from `connect()`: `validateWalRepairTarget` (read-only, fail-closed; refuses symlinked dataDir/pg_wal/`global`/pg_control — lstat follows INTERMEDIATE symlinks, so `global/` itself must be checked or surgery would write pg_control through it into a foreign dir; tolerates the in-dir `.gbrain-lock`), rename-based backup (the ENTIRE `pg_wal/` dir + postmaster.pid renamed into a sibling `.wal-repair-backup-/`, only the 8KB pg_control copied — zero transient disk cost), `restoreWalBackup` (overwrite order: control first via atomic tmp+rename, then a pg_wal dir swap with the reset dir set ASIDE inside the backup — nothing is ever deleted during restore; mtime guard refuses when a foreign segment is newer than the backup; a missing/empty backup NEVER reports `restored:true`), `WalRepairError` (thrown when resetWal fails AFTER the backup — carries the receipt + the best-effort restore's REAL result so the seam's `restored` flag and the `failed-restored`/`failed-not-restored` message arms never lie), a cooldown sidecar `.wal-repair-attempt.json` (skip `'recently-failed'` inside `GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600 — bounds the autopilot/supervisor reconnect loops) with episode-scoped backups (attempts within one corruption episode REUSE the episode's first backup — the pre-damage forensic state, honored only when the sidecar's path is a real non-symlink `.wal-repair-backup-*` sibling since the sidecar is user-writable JSON; retention keeps the newest 3 episodes, never pruning the open episode's), and `attemptWalRepairAndRetry` — the engine seam that NEVER throws (gates: kill-switch → reaped-lock `'possibly-live-writer'` → 10-minute reap-marker quarantine (`msSinceLastReap`, cross-process) → validation → cooldown; then repair → retry create once → restore-and-record on failure; prints a repair-start stderr line so a timeout-killed attempt is self-explaining). `inspectPgliteDataDir` is the read-only diagnosis for `gbrain doctor` + `pglite-repair --dry-run`. Imports runtime values only from `pglite-lock.ts`/`pglite-resetwal.ts`/node:fs — never from `pglite-engine.ts` (no cycle; the engine statically imports THIS file per the #3596 engine-live rule). Pinned by `test/pglite-repair.test.ts` + `test/pglite-wal-repair.serial.test.ts`. - `src/commands/pglite-repair.ts` — `gbrain pglite-repair`: the manual surface for WAL repair (`--dry-run | --yes | --json | --path `; CLI_ONLY + SELF_HELP; returns an exit code via `setCliExitVerdict`, never `process.exit`). Never connects an engine — works when the DB won't open and when auto-repair is disabled. `--dry-run` is strictly read-only. The real run validates BEFORE locking (`acquireLock` mkdirs the data dir — a typo'd `--path` must not create directories), refuses a live lock holder (pre-lock diagnosis names the PID; a live `gbrain serve` is called out), refuses a reaped acquisition (`refused_reaped_lock` — no `--force` by design: force-removing `.gbrain-lock` would reopen the #2348 concurrent-writer hole), re-validates under the lock, repairs with episode-backup reuse, and records the attempt in the sidecar. Pinned by `test/pglite-repair-command.serial.test.ts`. -- `src/commands/doctor.ts` `pglite_data_dir` check — fs-only check that runs when a PGLite brain FAILS to connect (`!fastMode && !engine && config.engine === 'pglite'`, placed after `orphan_clones`, before the DB-checks gate): `computePgliteDataDirCheck(dataDir, diagnosis)` (exported pure fn, `computeWorkerOomLoopCheck` convention) maps the `inspectPgliteDataDir` verdict to a Check — corruption-likely/looks-healthy-but-unopenable/unsupported-layout → `fail` naming `gbrain pglite-repair --dry-run`/`--yes` or the rebuild path, live-lock/missing-dir → `warn`; all `remediation_status: 'human_only'` (Minion remediation needs the DB that is down). Escalates when ≥2 repair attempts failed inside 7 days (unclean-shutdown genesis still active → engine-switch pointer) and reports retained backup-dir inventory (orphan_clones disk-visibility class). Registered in `doctor-categories.ts` OPS_CHECK_NAMES. Pinned by `test/doctor-pglite-datadir.test.ts`. -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`, `timeline_entries.event_page_id`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. A source holding a LIVE, non-expired per-source sync lock (`inspectLock(engine, syncLockId(source.id))` from `src/core/db-lock.ts`) is reported as actively syncing (the message names the holder pid + host) and counted in `synced_recently_count`, NOT flagged stale — the live lock is the only honest in-progress signal (checkpoint banking can't distinguish in-progress from wedged: a blocked sync banks its files but writes no anchor). A blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either falls through to the stale path and is never masked; the dynamic `db-lock` import is swallowed to a no-op on a stub engine or pre-lock-table brain, so this can only ADD an in-progress verdict, never suppress a real stale one. The in-progress note is appended to whatever verdict the buckets produce and is empty when nothing is syncing, so steady-state messages stay byte-for-byte unchanged. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). `pglite_data_dir` check: fs-only check that runs when a PGLite brain FAILS to connect (`!fastMode && !engine && config.engine === 'pglite'`, placed after `orphan_clones`, before the DB-checks gate): `computePgliteDataDirCheck(dataDir, diagnosis)` (exported pure fn, `computeWorkerOomLoopCheck` convention) maps the `inspectPgliteDataDir` verdict to a Check — corruption-likely/looks-healthy-but-unopenable/unsupported-layout → `fail` naming `gbrain pglite-repair --dry-run`/`--yes` or the rebuild path, live-lock/missing-dir → `warn`; all `remediation_status: 'human_only'` (Minion remediation needs the DB that is down). Escalates when ≥2 repair attempts failed inside 7 days (unclean-shutdown genesis still active → engine-switch pointer) and reports retained backup-dir inventory (orphan_clones disk-visibility class). Registered in `doctor-categories.ts` OPS_CHECK_NAMES. Pinned by `test/doctor-pglite-datadir.test.ts`. silent-failure batch (#2250/#2784/#2788): `content_hash_duplicates` (single GROUP BY over `(source_id, content_hash)` with FILTER aggregates — never N² — flagging hash groups that hold BOTH a bare and a path-prefixed slug, the wrong-import-root pattern; warn carries sample pairs + the `pages delete` → `purge-deleted --older-than 0` remediation); `undeclared_db_only_pages` (per source with a local repo: markdown pages with no backing file outside every declared + derive-phase-default db_only prefix — the one check deliberately allowed to stat the repo); `db_only_collector_collision` (configured recipe `output_paths` inside a declared db_only dir — auto-gitignore means sync AND import silently skip the collector's files; same warning fires in sync's `manageGitignore` at config-write time). All warn-level, engine-parity pinned by `test/e2e/doctor-silent-death-parity.test.ts`; units in `test/doctor-silent-death-checks.test.ts`. `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. `subagent_provider` check (layer 3 of 3). Resolves subagent model config in runtime order (`models.subagent` > `models.default` > `models.tier.subagent` > built-in default) and warns when the selected model lacks native tool-loop capability (message names the bad value + paste-ready fix `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK when subagent tier resolves to Anthropic. Tests in `test/doctor.test.ts`. `computeWorkerOomLoopCheck(engine)` is the single authoritative OOM-loop signal, unioning supervised `summarizeCrashes(readRecentSupervisorEvents(24)).by_cause.rss_watchdog` (cross-week read via `readRecentSupervisorEvents` so a Monday window can't lose Sunday) + bare-worker `minion_jobs error_text='aborted: watchdog'` count (Postgres-only; the same source `queue_health` subcheck 3 reads). Cap comes from the latest `rss_watchdog_loop` breaker alert's `max_rss_mb`, else `resolveDefaultMaxRssMb()` fallback. fail at breaker-tripped or oomKills≥5, warn at ≥1, null otherwise. `computePoolReapHealthCheck(engine)` is the Postgres-only `pool_reap_health` check reading `readRecentPoolRecoveries(1)` — fail when reconnect failures>0 (reconnect throwing is the actionable signal), warn at ≥10 reaps/hr (pooler thrash), null otherwise. Both registered in `buildChecks` after the `supervisor` block. The `supervisor` causeStr carries `rss=N (see worker_oom_loop)` and `queue_health`'s watchdog message cross-references `worker_oom_loop`. `DoctorReport.top_issues` + the cause-ranked render header. `worker_oom_loop` + `pool_reap_health` registered under ops in `doctor-categories.ts`. Pinned by `test/doctor-worker-oom-loop.test.ts`, `test/doctor-pool-reap-health.test.ts`. `supervisor_singleton` check (#1849), a SEPARATE check from `supervisor` (same split precedent as the niceness check) so a singleton-divergence warn can't clobber the crash/liveness precedence. Runs only when a `started` supervisor event was seen in the last 24h and a live engine is available. Reads the queue-scoped DB lock row (`gbrain_cycle_locks WHERE id = supervisorLockId(queue)`) and compares the lock holder (`holder_host:holder_pid`) against the local pidfile holder via the pure `classifySupervisorSingleton`. `mismatch` → warn (a second supervisor may be running with a different `--max-rss`; message names both holders, the effective cap from the `started` event's `max_rss_mb`, and the fix `gbrain jobs supervisor stop`); `single` → ok (names holder + cap); `no_lock` → no check emitted. Best-effort try/catch (silent skip on brains without the lock table). Registered under ops in `doctor-categories.ts` as `supervisor_singleton`. Pinned by `test/supervisor-db-lock.test.ts` + `test/doctor.test.ts`. `checkBatchRetryHealth`: `batch_retry_health` check surfacing Supavisor circuit-breaker incidents. Wired into both `runDoctor` (local) and `doctorReportRemote` (thin-client). Reads last 24h. States: `ok` (zero exhausted in 24h OR <3 from a single site), `warn` (>=3 same-site OR >=5 cross-site), `fail` (>=20 sustained breaker). Surfaces bad `GBRAIN_BULK_*` env at doctor startup. Corrupt-JSONL tolerant. Paste-ready fix hints in every warn/fail message. Also reads `readRecentDbDisconnects(24)` and appends `Disconnect-call audit: N call(s) in 24h (most recent caller: ).` to ALL three message paths so connection-incident signal is greppable from one `gbrain doctor --json` call (module-import wrapped in try/catch so older brains without the audit file degrade silently). Pinned by `test/doctor-batch-retry.test.ts` (10 cases). three checks wired into `runDoctor()` and the JSON envelope, all warn-only with paste-ready fix hints. (1) `checkSourceRoutingHealth(engine)` scans up to 200 pages on federated brains and flags pages whose `source_id` doesn't match what `resolveSourceWithTier()` would have picked for their `source_path`; single-source brains short-circuit to `ok`; the 200-page cap is total across the brain so doctor stays under 5s. (2) `checkOauthConfidentialHealth(engine)` probes registered confidential clients for `/token` reachability. (3) `checkAutopilotLockScope()` (pure, no engine) compares the resolved lock path to `$GBRAIN_HOME`; warns when set but the lock lives elsewhere, with a PID-safe inspection hint (`kill -0 ` before deletion). Pinned by `test/doctor-v0_37_7_checks.test.ts`. `buildChecks(engine, args, dbSource): Promise` exported as a test seam. `runDoctor` is a thin wrapper: `buildChecks → computeDoctorReport → render + process.exit`. All 10 `process.exit` sites stay in the wrapper; the two early-return paths (no engine, connection failure) return partial check lists instead of inline exits (observable output identical). Pinned by `test/doctor-behavioral.test.ts` (13 cases: pure aggregation math over `computeDoctorReport`, orchestrator cases for `--fast` skip set + `--json` flag + no-engine partial path + snapshot of load-bearing check names) and `test/doctor-cli-smoke.serial.test.ts` (1 subprocess case spawning `bun run src/cli.ts doctor --json` against a fresh PGLite tempdir, asserting schema_version=2 envelope, status enum, non-empty checks array — the render-path coverage buildChecks-only tests miss; quarantined `.serial` because PGLite write-locks don't play with parallel runners). three checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on live DB pages matching any junk pattern that escaped ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; `--content-audit` opts into a full scan. All three warn-only with paste-ready fix hints (junk → `gbrain sources audit ` + `git rm` source-of-truth, oversize → split or accept). two checks wired into `runDoctor()` + the JSON envelope: `quarantined_pages` (counts pages carrying the `quarantine` marker via `engine.executeRaw` JSONB `?` existence, works on PGLite + Postgres; warn-only with a `gbrain quarantine list` hint) and `flagged_pages` (counts `content_flag` pages — searchable but odd; warn-only). Both skip gracefully (status ok, "Skipped") on engines/brains where the probe errors. Pinned by `test/doctor.test.ts`. `home_dir_in_worktree`: filesystem check walking up from `gbrainPath()` toward `$HOME` looking for a `.git` directory (main repo) or `.git` file (linked worktree pointer; Conductor + git-worktrees topology). Walk terminates at `$HOME` so a `.git` above the user's home doesn't false-positive. Honors `GBRAIN_HOME` (appends `.gbrain` to the override). Warn (not fail) with worktree-root path + paste-ready fix pointing at `GBRAIN_HOME` override or moving the brain. `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`. 4 checks: `abandoned_threads`, `calibration_freshness`, `grade_confidence_drift` (mitigation surface; math ships later), `voice_gate_health`. +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`, `timeline_entries.event_page_id`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite. `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. +- `src/core/chunkers/recursive.ts` — base chunker: 300-word chunks, 50-word sentence-aware overlap, 5-level delimiter hierarchy. Lossless invariant: non-overlapping portions reassemble to the original. Also strips the facts fence via `stripFactsFence({keepVisibility:['world']})` so private fact visibility tiers never reach embeddings. +- `src/core/chunkers/semantic.ts` — embedding-based topic-boundary detection: embeds sentences, computes cosine-similarity valleys, smooths with a Savitzky-Golay filter (5-window, 3rd-order polynomial) to find chunk boundaries. +- `src/core/chunkers/llm.ts` — LLM-guided chunking: pre-splits into 128-word candidates via the recursive chunker, then asks a Haiku-class model "where does the FIRST topic shift occur?" per window. +- `src/core/search/dedup.ts` — 4-layer result dedup + compiled-truth guarantee: (1) top 3 chunks per page by score, (2) drop chunks >0.85 Jaccard-similar to kept results, (3) no page type exceeds 60% of results, (4) max 2 chunks per page (default; the two-pass structural expansion in `hybrid.ts` widens it), (5) ensure at least 1 compiled_truth chunk per page. Page identity is the composite `pageKey()` (source_id, slug) — the one canonical key helper every layer uses, so slug collisions across sources can't collapse recall. - `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`; EMBEDDINGS only — chat/completion pricing lives in `model-pricing.ts` (different unit) and is never mixed in. Every entry carries its official source URL + the date it was last read. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M); Voyage 4-large ($0.12/1M), 4 ($0.06/1M), 4-lite ($0.02/1M), legacy 3-large ($0.18/1M), 3 ($0.06/1M); ZeroEntropy zembed-1 ($0.05/1M), zerank-2 ($0.025/1M); Mistral mistral-embed ($0.10/1M); Perplexity pplx-embed-v1-4b ($0.03/1M), 0.6b ($0.004/1M). `voyage-4-nano` is deliberately unpriced (open-weight variant, no published hosted rate) so it degrades to "estimate unavailable" rather than a fabricated 0. `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token. Unknown providers degrade to "estimate unavailable" instead of fabricating numbers. - `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait. @@ -47,13 +46,13 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`. - `src/core/fts-language.ts` — Single source for the Postgres text-search configuration name used by FTS. `getFtsLanguage()` resolves `GBRAIN_FTS_LANGUAGE` (default `english`), validates against `/^[a-z][a-z0-9_]*$/` (tsvector config names can't be bound as parameters, so the value is interpolated into raw SQL — the allowlist regex is the injection guard; invalid values warn once and fall back to `english`), and caches on first read (`resetFtsLanguageCache()` is test-only). Consumed by both engines' `searchKeyword`/`searchKeywordChunks` (`websearch_to_tsquery` query side), the `configurable_fts_language` migration, and `reindex-search-vector` (write-side trigger functions). Pinned by `test/fts-language.serial.test.ts` + `test/fts-language-migration.serial.test.ts` (includes the `'; DROP TABLE pages; --` injection cases). - `src/commands/reindex-search-vector.ts` — `gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`. -- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count, skipped_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok (sources skipped by `--missing-path skip` count as ok), 1 any error. `--missing-path ` (default fail) handles sources whose `local_path` does not exist on this machine — machine-specific state in a brain-wide table, so a brain registered from several machines fails every foreign source on every run; `skip` classifies them `skipped_missing_path` (⊘ line, envelope entry with `local_path`, excluded from `error_count` and the rc gate) via the exported pure helpers `parseMissingPathMode` + `partitionMissingPathSources`, pinned by `test/sync-all-missing-path.test.ts`; default `fail` stays loud because on a single-machine brain a missing path usually means an unmounted volume. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). A warn-and-continue internal `git pull` failure (non-timeout class — e.g. a local-path origin rejected by `protocol.file.allow=never`) still falls through to sync the local working tree, but a ZERO-import run after a failed pull returns `partial` with `reason: 'pull_failed'` instead of `up_to_date`: `last_commit` AND the `last_sync_at` heartbeat stay frozen (so doctor `sync_freshness` / `sources status` staleness fires), the single-source CLI exits non-zero, `sync --all` exits non-zero if any source hit it (JSON envelope carries the per-source `reason`), and the autopilot cycle's sync phase maps it to `warn`. Timeout-class partials keep their pre-existing exit-0 / phase-`ok` semantics (they converge on retry; a failing pull does not). Pinned by `test/sync-pull-failed-anchor.serial.test.ts`. `resolveSlugByPathOrSourcePath`: Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path. - `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToSearchResult` projects email `message_id` / `thread_id` metadata and exposes `source_subject` only when a non-empty Message-ID proves the page is an email, so generated page titles never become authoritative email subjects. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` (#1972) — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the #2084 contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack. - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`). Copies the complete source catalog FIRST (`copyMigrationSources` — every `sources` row incl. archived rows and sync/routing metadata, `ON CONFLICT (id) DO UPDATE`, `default` ordered first) so every page write has a valid `pages.source_id` FK parent and the target preserves per-source behavior; pages copy afterward, tracked in the resume manifest by composite `(source_id, slug)` key. The resume manifest is target-aware: `migrationTargetId(config)` hashes `(engine, locator)` (`database_url` for Postgres, resolved `database_path` for PGLite) and `manifestMatchesTarget` requires `schema_version === 2` plus a matching `target_id` — a legacy engine-only manifest, or one from a DIFFERENT target of the same engine kind, starts fresh instead of skipping "completed" pages the new target never received. Pinned by `test/migrate-engine-resume.test.ts` (manifest identity) + `test/e2e/migrate-engine-sources-postgres.test.ts` (source catalog lands before overlapping-slug pages, PGLite → real Postgres). -- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`. -- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), `vendor`/`dist`/`build`/`venv`, dot-prefix dirs, and `*.raw` sidecars — NOT `ops/`, which is ordinary user content (#2404; the bundled daily-task-manager stores `ops/tasks` there); `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. +- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`. identity-based dedup pre-check at `:427-490`. Calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId})` (optional `?` so test doubles compile). Posture: SKIP when `frontmatter.id` matches (true external duplicate from overlapping ingest roots), WARN-ALWAYS on content_hash collision with different/missing `frontmatter.id` (templates and daily logs may legitimately share text), FAIL CLOSED on lookup error, bypass via `--force-rechunk`. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Pinned by `test/import-dedup-frontmatter-id.test.ts` (11 cases). `importFromContent` is the narrow waist every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It runs a three-tier content-quality disposition via `assessContentSanity` from `src/core/content-sanity.ts` BEFORE chunking: (1) high-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE (stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search) OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`; (2) fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker (page stays fully searchable, marker rides search results + get_page to warn the agent); (3) oversize → `embed_skip` soft-block via `buildEmbedSkipMarker()` PLUS a `content_flag:oversized` marker, AND deletes any pre-existing chunks in the same transaction so search can't surface stale chunks. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. `gbrain import` honors `errors > 0` for non-zero exit. `classifyErrorCode` in `src/core/sync.ts` recognizes the `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these. `extractEntityRefs` (canonical; matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks), `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled` config helper. Link candidates match any dir-shaped path (#2576; existence-checked at persist). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. Pinned by `test/import-file-content-sanity.test.ts`. +- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), `vendor`/`dist`/`build`/`venv`, dot-prefix dirs, and `*.raw` sidecars — NOT `ops/`, which is ordinary user content (#2404; the bundled daily-task-manager stores `ops/tasks` there); `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. The `commands/sync.ts` cleanup loop guards on `unsyncableReason(path)` being `'metafile'` OR `'pruned-dir'` (#2404) so previously-indexed metafile pages AND deliberately put-created pages under pruned dirs survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). `pruneDir`: `pruneDir(name, parentDir?)` extended with optional `parentDir`. When provided, additionally rejects directories containing `.git` as a FILE — the git submodule gitfile pattern (regular repos have `.git` as a DIRECTORY; submodules as a file pointing into the parent's `.git/modules/`). Sync + extract walkers thread `parentDir` so the gitfile-as-FILE check fires per descend step. Best-effort: `statSync` failures fall through and treat as a normal dir. Closes the phantom-import bug class where syncing a worktree-with-submodules walked into submodule trees. Pinned by `test/sync-walker-submodule.test.ts`. - `src/core/sync-failure-ledger.ts` — the bounded auto-skip sync failure ledger (issue #1939; formerly inline "Bug 9" in `sync.ts`). A LEAF module (imports only fs/path/crypto/config) so `sync.ts` can re-export it without a circular dependency. State lives in `~/.gbrain/sync-failures.jsonl`, one JSON object per line, keyed by `(source_id, path)` with a per-key `attempts` count and a 3-state machine: `open` (fresh/blocking) → `acknowledged` (human resolved via `gbrain sync --skip-failed`) or `auto_skipped` (chronic). `classifyErrorCode(errorMsg)` regex classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` (also recognizes `PAGE_JUNK_PATTERN` from the content-sanity gate); `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`; `MISSING_OPEN`/`MISSING_CLOSE`/`EMPTY_FRONTMATTER` regexes match the `markdown.ts` validator strings, `FILE_TOO_LARGE` covers `import-file.ts:199, 352, 401`, `SYMLINK_NOT_ALLOWED` covers `:347`. All mutations run under `withLedgerLock` (cross-process file lock) with an atomic rename write. The auto-skip threshold resolves via `resolveAutoSkipThreshold()` from `GBRAIN_SYNC_AUTOSKIP_AFTER` (default `DEFAULT_AUTOSKIP_AFTER = 3`; `0` disables the valve = pure fail-closed). Two pure decision functions are the unit-test surface: `decideGateAction({fileFailures, sentinels, attemptsByPath, threshold, skipFailed})` returns `hard_block | block | advance | advance_then_autoskip` (sentinels like `` ALWAYS hard-block, even with `--skip-failed`, so a history rewrite can't auto-skip; any FRESH failure with `attempts < threshold` blocks fail-closed; only when ALL failures are chronic does it `advance_then_autoskip`), and `decideSyncFailureSeverity({entries, nowMs, failHours})` returns the `sync_failures` doctor status (`ok` when zero unresolved; `fail` when ≥10 OPEN-blocking or the oldest OPEN failure has blocked the bookmark past `failHours`; otherwise `warn` — `auto_skipped`-only rows stay WARN-visible regardless of count because the bookmark already advanced). `applySyncFailureGate(input)` is the one orchestrator BOTH sync paths (incremental + full/`runImport`) call: it records/clears ledger rows, runs `decideGateAction`, then executes effects in the crash-safe order (advance the bookmark FIRST via the injected `advance()` callback, THEN auto-skip the chronic set) so a crash can never mark a file skipped while leaving sync wedged. `isSkippablePath` rejects `<…>` sentinels. Pinned by `test/sync-failure-ledger.serial.test.ts` + `test/sync-failures.test.ts`. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local). - `src/core/storage-config.ts` — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked`/`supabase_only`) to canonical (`db_tracked`/`db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Uses a dedicated parser for the `gbrain.yml` shape rather than gray-matter (broken on delimiter-less YAML). Also carries `DERIVE_PHASE_DB_ONLY_DEFAULTS` (`life/events/`, `atoms/`, `extracts/`, `dream-cycle-summaries/`) + `effectiveDbOnlyDirs` — the engine's derive-phase output prefixes treated as implicitly-declared db_only by the `undeclared_db_only_pages` doctor check but deliberately NOT merged into `loadStorageConfig` (a global merge would auto-gitignore those dirs and silently kill ingestion for brains that file-back them, the #2788 class) — and `findDbOnlyCollisions` (pure collector-output vs db_only overlap detector shared by the `db_only_collector_collision` doctor check and sync's `manageGitignore` warning). Pinned by `test/storage-config.test.ts` + `test/doctor-silent-death-checks.test.ts`. @@ -73,19 +72,25 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/code-def.ts` + `src/commands/code-refs.ts` — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface. The JSON envelope (CLI + the `code_def`/`code_refs` MCP ops) carries `status` + `ready` from `src/core/code-graph-readiness.ts` so a `count:0` result is distinguishable as `not_built` (no code indexed) vs `ready` (genuinely no match); human output prints a one-line hint when not ready. - `src/core/code-graph-readiness.ts` — typed readiness signal shared by the four code-* surfaces (`code-def`/`code-refs`/`code-callers`/`code-callees`). `resolveCodeReadiness(engine, {kind:'symbol'|'edge', count, sourceId?, allSources?})` returns `{status:'not_built'|'indexing'|'ready'|'unknown', ready, has_code, pending_edges}`. `count>0` short-circuits to `ready` with no query; on empty it runs `EXISTS` probes against `content_chunks` JOIN `pages` (`page_kind='code'`) — no `page_kind` index needed, and the pending probe rides the partial `idx_content_chunks_edges_backfill`. `kind:'symbol'` (code-def/refs) is 2-state + brain-wide because symbol metadata is set at chunk time; `kind:'edge'` (code-callers/callees) is 3-state + source-scoped, with the pending predicate mirroring the resolver (`edges_backfilled_at IS NULL OR < EDGE_EXTRACTOR_VERSION_TS` from `src/core/chunkers/symbol-resolver.ts`) so a resolver-version bump never falsely reports `ready`. Probe scope matches each command's result-query `deleted_at` posture (def/refs don't filter `deleted_at`, so neither do the probes). Any DB error returns `status:'unknown'` (fail-open; never breaks the command). `readinessHint(r)` renders the human one-liner. Wired into `code-def.ts`/`code-refs.ts` (brain-wide), `code-callers.ts`/`code-callees.ts` (resolved `sourceId`/`allSources`), and all four `code_*` MCP op handlers in `src/core/operations.ts`. Pinned by `test/code-graph-readiness.test.ts` + readiness-envelope cases in `test/e2e/code-intel-mcp-ops-pglite.test.ts`. - `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally. -- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level). +- `src/core/search/query-intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level). Also the modality suggester: `QuerySuggestions.suggestedModality: 'text' | 'image' | 'both'`, driven by the module-scope `CROSS_MODAL_PATTERNS` regex array (compiled once at module load); `isAmbiguousModalityQuery(query)` is the heuristic gate that bounds the opt-in LLM modality tie-break (`search/llm-intent.ts`) to a small fraction of queries. +- `src/core/search/llm-intent.ts` — opt-in LLM modality tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt; 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser (tolerates trailing punctuation + casing). Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns the fallback so a misbehaving LLM can never break search. Cost-bounded by `isAmbiguousModalityQuery` in `query-intent.ts` so the LLM call fires on only a small fraction of queries when on. +- `src/core/search/image-loader.ts` — `loadImageInput(input, opts)` accepts a local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). URLs route through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated; pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND). +- `src/core/search/by-image.ts` — `searchByImage(engine, input, opts)`. Always runs the image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). Hybrid intersect: when the caller provides an optional `query`, runs a parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the resolved mode's refinement weights. Widens to the unified column when `search.unified_multimodal=true` (transparently upgrades retrieval quality post-reindex). +- `src/core/ssrf-validate.ts` — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A and AAAA record against the internal-IP deny list, and returns the resolved IP so callers fetch by IP (validation IP === fetch IP defeats DNS rebinding). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation (max 3 hops by default). Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests. +- `src/core/spend-log.ts` — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table. `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate; brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed. +- `src/commands/reindex-multimodal.ts` — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (partial-failure-aware), persists. Lock via `tryAcquireDbLock` (360min) so a concurrent autopilot embed phase can't race it. Cost prompt + Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. Auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with a paste-ready command). +- `src/core/backfill-registry.ts` — registry of idempotent data backfills. The `modality` backfill flips `modality` to `'image'` on image-asset chunks the ingest path missed; its SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')` — the `chunk_source` guard ensures a non-image chunk that happens to have `embedding_image` populated is never flagged. A second run finds zero rows. - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator. - `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, /chat/ 0.5, archive/ 0.5, extracts/ 0.3) and `DEFAULT_HARD_EXCLUDES` (test/, attachments/, .raw/). `archive/` is DEMOTED (findable, ranked below curated), not hard-excluded — archive holds high-signal history users expect to retrieve; the demote is a prior at the SQL/fusion layer and the cross-encoder reranker can still promote a strongly-matching archive page. `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. The surviving exclude policy is auditable via the `hidden_by_search_policy` doctor check (`src/commands/doctor.ts`, local + remote paths) which counts chunked pages withheld per active exclude prefix, reusing `resolveHardExcludes` + `buildVisibilityClause` + the exported `escapeLikePattern`. - `src/core/search/sql-ranking.ts` — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash is Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. `buildBestPerPagePoolCte(...)` is the shared per-page max-pool CTE both engines' `searchVector` inject — instead of returning the single best chunk per page from an inner `ORDER BY embedding <=> vec LIMIT N` (which let a page lose to a neighbor on ONE weak chunk while its strong chunk sat just below the inner cut), the CTE pools the BEST chunk score per `(source_id, slug)` composite key so a page surfaces on its strongest evidence; composite key (not bare slug) keeps multi-source brains correct; single source of truth so the two engines can't drift. -- `src/core/search/title-match.ts` — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Mingtang"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (guards against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals. +- `src/core/search/title-match.ts` — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Helios"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (guards against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals. - `src/core/search/alias-normalize.ts` — ONE normalizer shared by the WRITE path (ingest projects frontmatter `aliases:` into `page_aliases`) and the READ path (search matches query against `page_aliases`), so stored aliases can't silently fail to match queries via divergent normalization (same single-source posture as `cjk.ts`). `normalizeAlias(raw)` does NFKC + lowercase + whitespace-collapse + trim + strip one layer of wrapping quotes/brackets; returns `''` for empty (callers MUST skip empty aliases). `normalizeAliasList(value)` coerces a frontmatter scalar / array / comma-list / garbage into a deduped list of normalized non-empty aliases — used by both the ingest projection and the `reindex --aliases` backfill. - `src/core/search/evidence.ts` — the agent-facing why-it-matched contract (closes the root behavior where an agent read one blended score, decided "no strong match, safe to create", and wrote a duplicate over a fully-developed page). `classifyEvidence(r)` names the strongest signal (precedence: `alias_hit` > `exact_title_match` > `high_vector_match` (base ≥ `HIGH_MATCH_FLOOR=0.85`) > `keyword_exact` (base ≥ `SOLID_MATCH_FLOOR=0.6`) > `weak_semantic`). `createSafetyFor(evidence)` derives the don't-duplicate hint (`exists`/`probable`/`unknown`) the agent keys off INSTEAD of a raw threshold (a blended RRF/cosine score is not a calibrated probability). `stampEvidence(results)` stamps `evidence` + `create_safety` in place once at pipeline end (after the alias hop, before slice); idempotent. -- `src/core/search/mode.ts` extension — `title_boost: number | undefined` knob in `ModeBundle` (default `1.25` for all three modes; multiplier for the post-fusion title-phrase boost). Override chain: per-call `SearchOpts` → `search.title_boost` config (clamped `[1.0, 5.0]`) → bundle. `KNOBS_HASH_VERSION` appends a `tib=` parts entry so a title-boost-on cache write can't be served to a title-boost-off lookup. `SEARCH_MODE_CONFIG_KEYS` gains `search.title_boost`. - `src/commands/search-diagnose.ts` — `gbrain search diagnose "" --target [--json] [--source ]`: Phase-0 retrieval diagnostic. Traces WHERE a target page surfaces (or fails to) across keyword / vector (per-page max-pool) / alias / hybrid layers and names the layer responsible for an incident, so an operator can pin whether the fix is max-pool/innerLimit (vector) vs title/alias. The verdict names the layer that DOES surface the target (or "none"). Pinned by `test/search/search-diagnose.test.ts`. - `src/commands/reindex-aliases.ts` — `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source ]`: backfills the free-text alias layer for EXISTING pages whose frontmatter `aliases:` predate the alias table (the import-time projection covers new + changed pages). Reads each page's frontmatter `aliases:`, writes via `engine.setPageAliases`. Idempotent + convergent (setPageAliases replaces a page's alias set) so no op-checkpoint needed; walks `listAllPageRefs` (cheap cross-source enumeration), `--source` narrows. Pinned by `test/search/reindex-aliases.test.ts`. - `src/eval/retrieval-quality/harness.ts` + `src/commands/eval-retrieval-quality.ts` + `test/fixtures/retrieval-quality/namedthing.jsonl` — NamedThingBench, the retrieval-quality eval that makes the named-thing-miss incident impossible to reintroduce silently. Seven query families, each a distinct failure class: `title-substring` (the direct regression), `generic-to-named` (tourist label → named thing), `alias-synonym` (declared alias / romanization → canonical), `multi-chunk-dilution` (one strong chunk among many weak — stresses max-pool), `short-vs-rich`, `graph-relationship` (guardrail), `hard-negative` (precision guard, must NOT return a page). `gbrain eval retrieval-quality ` runs it with hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects a `SearchFn` (CLI uses `hybridSearch`, tests stub) so it's engine-agnostic. Metric glossary entries (`hit@1`/`hit@3`) added to `src/core/eval/metric-glossary.ts`. Pinned by `test/eval-retrieval-quality.test.ts` + `test/retrieval-quality-harness.test.ts`. - `docs/architecture/RETRIEVAL.md` + `docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md` — retrieval-pipeline architecture reference + the named-thing-miss incident write-up (root cause, the five-layer fix, the eval that pins it). -- `src/core/types.ts` extension + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` — the wiring layer for the retrieval cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op uses a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query), surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`. +- `src/core/types.ts` + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` — the wiring layer for the retrieval cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op uses a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query), surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`. - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. Sub-subcommand dispatch on `args[0]` routes `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. `gbrain eval cross-modal` is in the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine). - `src/commands/eval-cross-modal.ts` — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (partial cost guardrail) via the shared `resolveCycleDefault(explicit, isTty)` in `src/core/eval/cycle-default.ts`; the cost-estimate banner appends `cycleDefaultSuffix(...)` (`for 1 cycle(s) (non-interactive default; --cycles N for more)`) when the value is the silent non-TTY fallback, so the 1-vs-3 difference isn't hidden. Receipts land at `gbrainPath('eval-receipts')/-.json`. `--batch [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`. - `src/core/eval/cycle-default.ts` — single source of truth for the eval cycle-count default. Exports `DEFAULT_CYCLES_TTY = 3`, `DEFAULT_CYCLES_NONTTY = 1`, `resolveCycleDefault(explicit, isTty): {cycles, usedNonTtyDefault}`, and `cycleDefaultSuffix(r)` (returns ` (non-interactive default; --cycles N for more)` only when the non-TTY default was applied, else `''`). Consumed by `eval-cross-modal.ts`, `eval-takes-quality.ts` (run + regress), and `takes-quality-eval/runner.ts` (core uses only the constant — library stays TTY-agnostic; the CLI owns the TTY=3 upgrade + banner annotation). `eval-suspected-contradictions.ts` applies the same transparency to its `$5`/`$1` budget default via a `budgetUsdExplicit` flag (the budget is overwritten in-place so explicitness can't be inferred post-hoc). Not shared with `resolveWorkersWithClamp` (different domain, no engine, no dedup). Pinned by `test/eval/cycle-default.test.ts`, `test/eval-suspected-contradictions-budget-default.test.ts`. @@ -99,7 +104,6 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/eval-replay.ts` — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real. See `docs/eval-bench.md`. `parseNdjson` skips lines where `_kind === 'baseline_metadata'` so `gbrain bench publish` baselines parse cleanly without the metadata header polluting row counts. Exports `replayCore(engine, opts): Promise<{summary, results}>` + `ReplaySummary` type so `gbrain eval gate` calls replay in-process (NOT subprocess — avoids gbrain-version-drift for source-tree CI). CLI `runEvalReplay` wraps `replayCore`. - `src/core/bench/baseline-file.ts` + `src/core/bench/qrels-file.ts` + `src/core/bench/correctness-gate.ts` + `src/commands/bench-publish.ts` + `src/commands/eval-gate.ts` — the eval-loop wave. `gbrain bench publish --from --to ` writes a baseline (stamps stable `query_hash` per row; metadata header carries `_kind: 'baseline_metadata'` + thresholds + `source_hash` + `baseline_mean_latency_ms`; deterministic sort by `(tool_name, query_hash)`; strict: empty=fail, dupes=fail with paste-ready hint, `--to` exists=refuse without `--force`). `gbrain eval gate [--baseline X] [--qrels Y]` is the two-gate dispatcher (regression gate via in-process `replayCore`, correctness gate via bare `hybridSearch` for determinism, both must pass when both flags set, exit 0 PASS / 1 FAIL / 2 USAGE). Source-id-aware: `bench publish` dedup key is `(tool_name, source_ids, query_hash)`; qrels compare keys are `${source_id}::${slug}` everywhere (closes the multi-source bug class at the file-shape layer). Latency math: `(baseline + delta) / baseline <= multiplier`. Fail-closed: ANY in-process throw flips verdict to fail with named breach in `breaches[]` — never silently exit 0. `.qrels.json` preserves the 12-row `test/fixtures/eval-baselines/qrels-search.json` fixture (slug-only `relevant_slugs` + `first_relevant_slug` auto-promote to `source_id='default'`) AND supports the federated shape (explicit `relevant: [{source_id, slug}]` + `expected_top1`). `correctness-gate.ts` runs each qrels query via bare `hybridSearch`; per-query throw recorded as `errored: true` and flagged as gate failure. Audit JSONL at `~/.gbrain/audit/bench-publish-YYYY-Www.jsonl`. Pinned by `test/bench/baseline-file.test.ts`, `test/bench/qrels-file.test.ts`, `test/bench/correctness-gate.test.ts`, `test/bench-publish.test.ts`, `test/eval-gate.test.ts`, `test/eval-replay-metadata-skip.test.ts`, `test/cycle/nightly-probe-adapters.test.ts`, `test/autopilot-nightly-probe-wiring.test.ts`, `test/e2e/eval-loop.test.ts`. - `src/core/cycle/nightly-probe-adapters.ts` — bridges the autopilot's object-shape `NightlyProbeDeps` to the argv-shape `runEvalLongMemEval` + `runEvalCrossModal` CLI functions. Cross-modal adapter argv MUST include `--output summaryPath` (without it the summary lands at the default receipt path and the adapter reads nothing from `summaryPath`). In-process invocation (NOT subprocess) — avoids gbrain-version-drift for source-tree CI. Pinned by `test/cycle/nightly-probe-adapters.test.ts` (incl. argv-shape regression for the `--output` requirement). -- `src/commands/autopilot.ts` extension — tick body invokes `runNightlyQualityProbe` when `cfg.autopilot.nightly_quality_probe.enabled === true` (default OFF — opt-in to protect API spend). NO scheduler-side rate-limit check — `runNightlyQualityProbe`'s internal `shouldRunNightly` (reading the audit JSONL) is the single source of truth. Probe call wrapped in try/catch that logs via `logError` and does NOT bump `consecutiveErrors` (probe failure is informational, never crashes the loop). Default `max_usd` cap = 5. Pinned by `test/autopilot-nightly-probe-wiring.test.ts`. - `test/eval-replay-gate.test.ts` + `test/fixtures/eval-baselines/qrels-search.json` — hermetic retrieval qrels gate running in the standard PR unit-shard CI matrix (`.github/workflows/test.yml`, NOT the fixed-file E2E workflow). Uses the canonical PGLite block (test-isolation R3+R4) and the basis-vector embedding pattern from `test/e2e/search-quality.test.ts:23-28` for fully hermetic retrieval. The qrels fixture (12 queries) uses PLACEHOLDER names only (alice-example, widget-co-example, etc. — privacy rule) and embeds each query at a deterministic basis dimension so retrieval is reproducible. Each query lists `relevant_slugs[]` + `first_relevant_slug`; the test computes `top1_match_rate` (top-1 == first_relevant) and `recall@10` (fraction of relevant_slugs in top-10), asserting both meet floors (defaults `>= 0.80` and `>= 0.85`). Env-overridable floors `GBRAIN_REPLAY_GATE_TOP1_FLOOR` / `GBRAIN_REPLAY_GATE_RECALL_FLOOR` (via `withEnv()` per R1). Refresh discipline: when ranking changes intentionally move expected slugs, edit `qrels-search.json` directly with a `Why:` line in the commit body or the gate degrades to rubber-stamp. Pinned by `test/eval-replay-gate.test.ts` (incl. a privacy-grep regression guard against real-name reintroduction). - `src/core/cycle/nightly-quality-probe.ts` + `src/core/audit-quality-probe.ts` + `test/fixtures/longmemeval-nightly.jsonl` + `test/nightly-quality-probe.test.ts` — opt-in nightly cross-modal quality probe. The phase runs `gbrain eval longmemeval --by-type` against the committed 10-question placeholder fixture, pipes output through `gbrain eval cross-modal --batch --max-usd 5 --yes`, and writes one event per run to `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` (ISO-week-rotated, mirrors `audit-slug-fallback.ts`; honors `GBRAIN_AUDIT_DIR`). Default DISABLED — opt-in via `gbrain config set autopilot.nightly_quality_probe.enabled true` (prevents surprise API spend). 24h rate limit (pure `shouldRunNightly(now, recentEvents, windowMs?)`) skips with audit row `outcome: rate_limited`. Embedding-key short-circuit: longmemeval needs `gateway.embedQuery()`, so the phase exits early with `outcome: no_embedding_key` + stderr warn when no provider configured. Full DI surface via `NightlyProbeDeps` (`isEnabled`, `hasEmbeddingProvider`, `resolveMaxUsd`, `resolveRepoRoot`, `runLongMemEval`, `runCrossModalBatch`, `now`) so the unit test stubs every external effect. Cost ceiling: $5/run × 30 nights ≈ $150/month worst-case; expected ~$10.50/month. New `nightly_quality_probe_health` doctor check (`src/commands/doctor.ts`, right after `slug_fallback_audit`) reads last 7 days: SKIPPED when flag off (with enable command); OK when enabled + all PASS; WARN on any FAIL / ERROR / BUDGET_EXCEEDED with per-outcome counts. Pinned by `test/nightly-quality-probe.test.ts`. - `src/commands/eval-trajectory.ts` + `src/commands/founder-scorecard.ts` + `src/core/trajectory.ts` — temporal trajectory + founder scorecard. `gbrain eval trajectory ` shows the chronological typed-claim history (mrr/arr/team_size/etc) with regressions auto-flagged inline; `gbrain founder scorecard ` rolls up claim_accuracy / consistency / growth_trajectory / red_flags into one JSON. Pure-function math in `trajectory.ts`: `detectRegressions(points, threshold)` walks consecutive metric-value pairs per metric (10% drop default, env override `GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD`); `computeDriftScore(points)` returns `1 - mean(cosine(emb[i], emb[i-1]))` over existing embeddings (null when <3 embedded points). Backed by `BrainEngine.findTrajectory(opts)` — both Postgres and PGLite, single SQL query, deterministic `ORDER BY valid_from ASC, id ASC`. Source-scoped via the `sourceId` scalar / `sourceIds` array dual pattern; visibility-filtered for remote callers. MCP op `find_trajectory` (read scope, NOT localOnly) registered after `find_experts`. Migration v67 adds optional typed-claim columns (`claim_metric`, `claim_value`, `claim_unit`, `claim_period`) + a partial index on `(entity_slug, claim_metric, valid_from) WHERE claim_metric IS NOT NULL`. Fence widens from 10 to 14 cells when any row has typed data; renderer stays at 10 cells when none do (no churn diff on existing fences). Metric labels normalize to lowercase snake_case via `normalizeMetricLabel` (15-entry seed map). The `consolidate` cycle phase does semantic upsert keyed on `(page_id, claim, since_date)` (fixes the duplicate-takes bug where re-running the cycle after `extract_facts` cleared `consolidated_at` appended duplicates via `MAX(row_num)+1`) and writes chronological `valid_until` on each cluster's older facts. The `extract_facts` cycle phase batch-embeds via `gateway.embed()` before insert AND threads `pages.effective_date` as the `pageEffectiveDate` fallback for `valid_from` (precedence: fence-row > pageEffectiveDate > now()). The contradiction probe MUST NOT write `valid_until` — grep guard at `test/eval-contradictions/no-valid-until-write.test.ts`. Haiku extraction lives in `src/core/facts/extract.ts` (not the `extract-facts.ts` cycle phase); its output cap is config `facts.extraction_max_tokens` (default 4000), a `stopReason: 'length'` response retries once at 2× the cap, and persistent truncation warns loudly on stderr instead of silently extracting zero facts; `pageEffectiveDate` is OPTIONAL because `fence-write.ts` callers have no Page object. Migration v89 adds a nullable `event_type TEXT` column on `facts` so the substrate carries event-shaped rows (`event_type='meeting'` / `'job_change'` / `'location_change'`) alongside metric rows. `TrajectoryPoint.event_type: string | null` projected by both engines. `TrajectoryOpts.kind?: 'metric' | 'event' | 'all'` filter (default `'all'`); `founder-scorecard` + `eval-trajectory` pass `kind: 'metric'` explicitly. Back-compat pinned by `test/regressions/v0_40_2_0-trajectory-backcompat.test.ts` (byte-identical `computeFounderScorecard` + `computeTrajectoryStats` with and without event rows); engine parity in `test/engine-parity-event-type.test.ts`. @@ -107,12 +111,11 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/think/intent.ts` + `src/core/think/entity-extract.ts` — pure `classifyIntent(question)` returns `'temporal' | 'knowledge_update' | 'other'` (regex-first, no LLM, `'other'` fast path short-circuits with zero SQL). `extractCandidateEntities(question, retrievedSlugs)` pulls high-precision candidates from retrieved entity-prefix slugs (`people/`, `companies/`, `organizations/`) and medium-precision noun phrases. Stop-word boundaries + leading-verb stripper handle "When did I last meet Marco" → `marco`. Both consumed by `runThink` and the LongMemEval harness so the two paths cannot drift. Pinned by `test/think-intent.test.ts` and `test/think-entity-extract.test.ts`. - `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned; UTF-8-safe truncation; confidence-floor double-enforcement; resolution_kind output drives paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — avoids bias from silent skip), trend writes to `eval_contradictions_runs`, source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker for stable cache hit-rate). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (batched), `writeContradictionsRun` + `loadContradictionsTrend`, `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache`. Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). Doctor check surfaces high-severity findings with paste-ready resolution commands; synthesize phase pre-fetches the latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. Architecture doc: `docs/contradictions.md`. - `src/core/think/index.ts` — `runThink` builds its internal `LLMClient` via a small adapter wrapping `gateway.chat()` from `src/core/ai/gateway.ts` (not `new Anthropic()` directly) so stdio MCP launches (Claude Desktop, Cursor) that don't inherit shell env still find a key set via `gbrain config set anthropic_api_key` (the gateway reads `~/.gbrain/config.json` AND env). Test seam: `opts.client?: ThinkLLMClient` injection works (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`); `opts.stubResponse` short-circuits before any LLM call. When neither key nor client is available, the "no LLM available" stub fires with `NO_ANTHROPIC_API_KEY`. Trajectory injection (default ON): `runThink` orchestrates `classifyIntent(question)` → `extractCandidateEntities(question, retrievedSlugs)` → `findTrajectory` (5s `Promise.race` timeout per candidate, concurrency cap 3) → `formatTrajectoryBlock`. `buildThinkUserMessage` (in `src/core/think/prompt.ts`) has a `trajectory?: ThinkTrajectoryBlockOpts` slot honoring BOTH prompt orderings (calibration mode: retrieval → calibration → trajectory → question; default mode: question → retrieval → trajectory → instruction). The MCP `think` op handler maps `sourceScopeOpts(ctx)` onto `RunThinkOpts` via `thinkSourceScopeOpts(ctx)` (operations.ts), and `runThink` threads the scope into `runGather` (`src/core/think/gather.ts`) — so every gather stream (hybrid retrieval, takes keyword + vector via the engines' scoped `searchTakes`/`searchTakesVector`, graph walk via `traversePaths`) AND trajectory resolution stay within the caller's source grant (federated `sourceIds[]` wins over scalar `sourceId`); pinned by `test/e2e/think-source-isolation-pglite.test.ts`. Config key `think.trajectory_enabled` (default `true`). Any error in the trajectory path degrades to "no block injected" + `TRAJECTORY_INJECTION_FAILED` warning — the think call never crashes from trajectory. Production path skips `fallback_slugify` resolutions (avoid querying invented slugs); the LongMemEval harness accepts them. Pinned by `test/think-trajectory-injection.test.ts`. Debug: `GBRAIN_THINK_DEBUG=1 gbrain think "..."` prints the spliced prompt to stderr. -- `src/core/operations.ts` extension (orphans fix) — `findOrphanPages` (both engines) filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side, so soft-deleted pages don't appear as orphans AND links from soft-deleted source pages don't suppress live pages from orphan results. Pinned by `test/orphans.test.ts`'s soft-delete cases. - `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` — `gbrain eval longmemeval ` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. One in-memory PGLite per run via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` (schema-migration-safe); infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) preserved. `cli.ts` pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults OFF (deterministic, no per-query Haiku); pass `--expansion` to opt in. Default model via `resolveModel()` 6-tier chain with `models.eval.longmemeval` config key. Sanitization parity: `harness.ts` reuses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` so adding a pattern covers takes AND benchmarks. Retrieved chat content wrapped in ``; the answer-gen system prompt declares content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client without an API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (`test/eval-longmemeval.test.ts` perf gate). Hand the JSONL to LongMemEval's `evaluate_qa.py` to score (not bundled — needs OpenAI gpt-4o). Per-question JSONL row carries `question: string` (additive; `evaluate_qa.py` ignores unknown fields) so `gbrain eval cross-modal --batch` has the `task` text without joining; also `question_type: string` and `recall_hit?: boolean` so a `--resume-from` run rebuilds cumulative `recallByType` from the file alone. `--by-type` flag emits a `{schema_version:1, kind:"by_type_summary", recall_by_type:{...}, aggregate:{...}}` line as the FINAL line; resume-replace strips any prior summary at the tail so 5 resumed runs produce 1 summary. Empty-bucket guard: `aggregate.rate` is `null` (not NaN) when no questions had ground truth. Optional `--by-type-floor F` (0..1) exits non-zero with a stderr line per breached `question_type` (default informational). Pure `buildByTypeSummary(buckets)` + `emitByTypeSummary(path, summary)` + `seedRecallByTypeFromFile(path, bucket)` exported for unit tests. Inline Haiku extractor + trajectory routing (methodology change): `src/eval/longmemeval/extract.ts` runs `extractAndInsertClaims()` over each haystack session before retrieval, populating the benchmark brain's `facts` table inline at import. Single Haiku call per session with content-hash cache (cuts a 3-iteration run from $1.50 to $0.50 when sessions repeat). Per-question alias map (fresh per question, never leaks) collapses `"Marco"` + `"Marco Smith"` + `"marco"` to one canonical slug via first-mention-wins. Fail-open on every error path (malformed JSON, Haiku throw, insert collision, empty array → `inserted: 0`). `getCacheStats()` writes empirical hit rate to stderr. `src/eval/longmemeval/intent.ts` prefers the dataset's `question_type` label before falling back to the SHARED regex set from `src/core/think/intent.ts` — single source of truth means think and longmemeval cannot drift. `runOneQuestion` routes temporal/knowledge_update intents through shared `extractCandidateEntities` → `findTrajectory` → splice into the answer-gen prompt before the retrieved-sessions block. `--no-trajectory` bypasses BOTH extractor and intent routing (baseline default-on vs no-trajectory across 3 seeds with paired-bootstrap CI). JSON envelope adds 5 per-question fields when trajectory routing is on: `intent`, `trajectory_points`, `entity_resolved`, `resolution_source`, `methodology_note`. The `methodology_note` writes to stderr at run completion (`extractor=haiku-preprocess-full-haystack-v1`) — honest disclosure that the published number is "gbrain + Haiku-preprocess pipeline" vs "gbrain alone", NOT directly comparable to baseline LongMemEval scores without that note. Pinned by `test/longmemeval-extract.test.ts`, `test/longmemeval-intent.test.ts`, `test/longmemeval-trajectory-routing.test.ts` (end-to-end through `runEvalLongMemEval` with both clients stubbed). - `docs/eval-bench.md` — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers (catches MCP + CLI + subagent tool-bridge from one site). Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. Capture is off by default — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Contributors set `export GBRAIN_CONTRIBUTOR_MODE=1`. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. -- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape. `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture records what hybridSearch actually did; existing callers leave it undefined. `HybridSearchOpts.types?: PageType[]` (on `SearchOpts`) threads a multi-type filter into per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks` as `AND p.type = ANY($N::text[])` (primary consumer `gbrain whoknows`, filters to `['person','company']`); AND-applies alongside the single-value `type` filter. `hybridSearch` resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor (not a raw string) into per-engine `searchVector`, and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision so a repointed `embedding` builtin doesn't leak across vector spaces. `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B; `search` (keyword-only) rejects it. Two post-fusion stages + evidence stamp: `applyTitleBoost(results, query, titleBoost, floorThreshold)` multiplies a result's score by the resolved `title_boost` when `isTitlePhraseMatch` fires, stamps `title_match_boost`, inherits the floor-ratio gate so a title match can't shove a much-stronger page below it; `applyAliasHop(engine, results, query, opts)` normalizes the query, calls `engine.resolveAliases`, and on exact normalized-alias match surfaces that page at top-of-organic + epsilon with `alias_hit=true`; `stampEvidence(...)` runs LAST (after the alias hop, before slice) on every path — keyword-only, no-embed, and full hybrid — so MCP callers and `--explain` read the same `evidence` + `create_safety` contract. `title_boost` resolved from the mode bundle and threaded in. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape. `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture records what hybridSearch actually did; existing callers leave it undefined. `HybridSearchOpts.types?: PageType[]` (on `SearchOpts`) threads a multi-type filter into per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks` as `AND p.type = ANY($N::text[])` (primary consumer `gbrain whoknows`, filters to `['person','company']`); AND-applies alongside the single-value `type` filter. `hybridSearch` resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor (not a raw string) into per-engine `searchVector`, and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision so a repointed `embedding` builtin doesn't leak across vector spaces. `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B; `search` (keyword-only) rejects it. Two post-fusion stages + evidence stamp: `applyTitleBoost(results, query, titleBoost, floorThreshold)` multiplies a result's score by the resolved `title_boost` when `isTitlePhraseMatch` fires, stamps `title_match_boost`, inherits the floor-ratio gate so a title match can't shove a much-stronger page below it; `applyAliasHop(engine, results, query, opts)` normalizes the query, calls `engine.resolveAliases`, and on exact normalized-alias match surfaces that page at top-of-organic + epsilon with `alias_hit=true`; `stampEvidence(...)` runs LAST (after the alias hop, before slice) on every path — keyword-only, no-embed, and full hybrid — so MCP callers and `--explain` read the same `evidence` + `create_safety` contract. `title_boost` resolved from the mode bundle and threaded in. `runPostFusionStages` has a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Each post-fusion stage stamps its multiplier: `applyBacklinkBoost`→`backlink_boost`, `applySalienceBoost`→`salience_boost`, `applyRecencyBoost`→`recency_boost`. `applyReranker` (earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution powers `gbrain search --explain` — every boost surface carries its own field so `formatResultsExplain` reads them all without coupling to internal stage ordering. with `src/core/search/sql-ranking.ts` + `src/core/operations.ts` + `src/core/types.ts`: agent-warning channel. `SearchResult.content_flag?: {reason, detail}` (new optional field in `types.ts`) is stamped post-fusion by `stampContentFlags` (the `stampEvidence` precedent) in `hybridSearch` AND in the keyword-only `search` MCP op so both retrieval paths surface the marker. `get_page` returns a top-level `content_flag` parallel field via `getContentFlag(page.frontmatter)`. `buildVisibilityClause` (sql-ranking.ts) ANDs in `QUARANTINE_FILTER_FRAGMENT` so quarantined pages are excluded from all six search call sites (alongside soft-delete + archived-source filters). Pinned by `test/sql-ranking.test.ts` + `test/e2e/quarantine-search-exclusion.test.ts`. Cross-modal routing at the embed step: `effectiveModality` resolves per-call `opts.crossModal` (literal `'auto'` → undefined) → `suggestions.suggestedModality` → `'text'`. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skipping expansion + keyword. `'both'` route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Unified routing fires when `search.unified_multimodal` is true — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, fail-open on zero rows (non-strict falls through to dual-column). LLM modality escalation fires only when no explicit per-call opt is set AND the regex returned `'text'` AND `search.cross_modal.llm_intent` is on AND `isAmbiguousModalityQuery` fires; fail-open on every error. - `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` — runtime contract test (R2). Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. `BATCH_SIZE=100` (per-recipe pre-split + recursive halving + adaptive shrink-on-miss live in the gateway; the outer paginator is for progress-callback granularity, not batch protection). `estimateEmbeddingCostUsd(tokens)` prices against the currently-configured model's rate via `currentEmbeddingPricePerMTok()` (resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate 0.13 only when the gateway is unconfigured or the model is unknown to the pricing table). `EMBEDDING_COST_PER_1K_TOKENS` retained for back-compat with direct importers/tests. `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from current and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. (See `src/core/sync-delta.ts` + `src/core/spend-posture.ts` for the #2139 cost-gate supporting modules.) `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so cost gate and embed decision can't drift. `shouldBlockSync(costUsd, floorUsd, mode, posture='gated'): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate), and `posture === 'tokenmax'` never blocks (the operator declared cost isn't the constraint; an `off`/`unlimited` floor is `Infinity` and so is never exceeded). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. @@ -120,7 +123,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/spend-posture.ts` — spend-control surface (#2139). `resolveSpendPosture(engine): 'gated'|'tokenmax'` (DB-plane `spend.posture`, fail-open `gated`); `tokenmax` makes every cost gate informational across sync/reindex/enrich/onboard (spend still ledgered — removes the ceiling, not the accounting). `parseUsdLimit(raw, def, {allowZero?})` accepts `off`/`unlimited`/`none` → `Infinity`; `formatUsdLimit(n)` renders `Infinity` as the string `'unlimited'` (never raw — `JSON.stringify(Infinity)` is `null`); `usdLimitToCap(n)` maps `Infinity` → `undefined` at the BudgetTracker boundary so ledger rows never serialize null. `normalizeSpendPosture`/`isValidSpendPosture` back the `config set` validation. Doc: `docs/operations/spend-controls.md`. Pinned by `test/sync-cost-preview.test.ts` + `test/spend-off-switch.test.ts`. - `src/core/ai/dims.ts` — per-provider `providerOptions` resolver for embed-time dimension passthrough; the single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, `isValidVoyageOutputDim(dims)`. Voyage path uses the SDK-supported `dimensions` field (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built (the AI SDK's openai-compatible adapter doesn't recognize the wire-key, so sending it from here would be silently dropped and Voyage would return its default 1024-dim). Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary (most common trigger: `embedding_model: voyage:voyage-4-large` without `embedding_dimensions`, falling back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). - `src/core/ai/types.ts` — provider/recipe types. `EmbeddingTouchpoint` has optional `chars_per_token` (default 4, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling), both consulted only when `max_batch_tokens` is also set; Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64). Pre-split budget = `max_batch_tokens × safety_factor / chars_per_token`. `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes mixing text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`); when omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` (OpenAI text + Voyage images without flipping the primary pipeline). `EmbeddingTouchpoint.trust_custom_dims?: true` — passthrough tier for a user-declared `--embedding-dimensions` on local / bring-your-own-backend recipes (ollama, llama-server, litellm) where the model catalog can't be enumerated; consumed by `isCustomDimValidForProvider` in `src/core/embedding-dim-check.ts` AFTER Tier 1 (recipe `dims_options`) and Tier 2 (provider Matryoshka allowlists), so a recipe that declares fixed options (openrouter) is still governed by those and fixed-dim hosted providers (openai/voyage/zeroentropy) stay fail-closed; the provider's `/embeddings` response-dim validation catches a genuine mismatch pre-storage. `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers riding alongside auth on every openai-compat touchpoint; mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) rejected at `applyResolveAuth` call time so defaults can't shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple. -- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). Provider-agnostic plumbing: `resolveNativeBaseUrl(provider, cfg)` normalizes a configured `ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL` to carry the `/v1` suffix and is passed explicitly at every native `createAnthropic` / `createOpenAI` site (chat/expansion/embedding), so an env-injected bare host doesn't 404; returns `undefined` when unset so the SDK default is preserved (Google deferred until its native suffix is verified). `diagnoseEmbedding` fails closed with `user_provided_dims_unset` when a user-provided / zero-default recipe (litellm/llama-server) has no configured `embedding_dimensions` — this REPLACED the old `user_provided_model_unset` guard, which was structurally unreachable (parseModelId throws on a bare provider) and only ever false-positived for `litellm:`, silently disabling vector search. `configureGateway` no longer backfills `embedding_dimensions` (readers default it themselves), keeping the "no dims set" signal honest for that guard and the multimodal skip. +- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). Provider-agnostic plumbing: `resolveNativeBaseUrl(provider, cfg)` normalizes a configured `ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL` to carry the `/v1` suffix and is passed explicitly at every native `createAnthropic` / `createOpenAI` site (chat/expansion/embedding), so an env-injected bare host doesn't 404; returns `undefined` when unset so the SDK default is preserved (Google deferred until its native suffix is verified). `diagnoseEmbedding` fails closed with `user_provided_dims_unset` when a user-provided / zero-default recipe (litellm/llama-server) has no configured `embedding_dimensions` — this REPLACED the old `user_provided_model_unset` guard, which was structurally unreachable (parseModelId throws on a bare provider) and only ever false-positived for `litellm:`, silently disabling vector search. `configureGateway` no longer backfills `embedding_dimensions` (readers default it themselves), keeping the "no dims set" signal honest for that guard and the multimodal skip. `withBudgetTracker`: gateway-layer enforcement via `AsyncLocalStorage`. `withBudgetTracker(tracker, fn)` installs the tracker on the module-internal store; every `gateway.chat / embed / rerank` call inside the scope auto-composes (reserve before, record in try/finally). Outside-scope calls are budget no-ops. Nested scopes restore the outer tracker on exit. `getCurrentBudgetTracker()` is the test seam. The chat path uses the pessimistic fallback on error paths; the embed path estimates input tokens from char count × recipe's `chars_per_token` because the AI SDK doesn't surface per-batch embed token usage; the rerank path estimates char count of query+docs. Pinned by 6 unit cases. module-scoped `_extendedModels: Map>>` registry feeds `assertTouchpoint`'s extended-model path without broadening unrelated surfaces. `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `registerConfigSelectedChatModel(model)` is the narrow runtime seam for a dedicated contextual-synopsis model: the ID joins the chat allowlist but remains rejected for embedding, expansion, and reranking. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. - `src/core/ai/recipes/zeroentropyai.ts` — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (pinned by regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/ai/recipes/llama-server-reranker.ts` — sibling of `llama-server` (the embedding recipe) for llama.cpp in `--reranking` mode. Distinct recipe rather than dual-touchpoint extension because `--reranking` and `--embeddings` are mutually exclusive at server-launch time, so the two backends need independent base URLs (default 8081 here vs 8080 there). Declares `reranker` touchpoint with `models: []` (user-provided id matching the `--alias` the user launched with), `path: '/rerank'` (leaf-only; consumes `RerankerTouchpoint.path` override; gateway concatenates with `base_url_default` which ends in `/v1`, producing `…/v1/rerank`), `default_timeout_ms: 30_000` (consumed by `src/core/search/mode.ts`'s reranker timeout chain — CPU-only first-call warmup headroom; the 5s mode-bundle default would fail-open as `timeout`), `cost_per_1m_tokens_usd: 0` (recognized by `FREE_LOCAL_RERANK_PROVIDERS` in `src/core/budget/budget-tracker.ts` so `--max-cost` callers don't hard-fail on local rerank). Setup hint emphasizes `--alias` because llama-server's `/v1/models` defaults model id to the gguf file path without it. Covers Qwen3-Reranker via llama.cpp AND self-hosted ZE weights via llama.cpp — same recipe, different `--model` at launch. Pinned by `test/ai/recipe-llama-server-reranker.test.ts`. Voyage / Cohere / vLLM rerankers stay out of scope (different wire shapes). Same wave adds: `path?: string` + `default_timeout_ms?: number` on `RerankerTouchpoint` in `src/core/ai/types.ts`; consumed by the URL build at `src/core/ai/gateway.ts:rerank()` and by mode-resolution at `src/core/search/mode.ts:resolveSearchMode` (precedence: per-call > config-key > recipe touchpoint default > mode bundle); `LLAMA_SERVER_RERANKER_BASE_URL` env passthrough in `src/cli.ts:buildGatewayConfig`; `FREE_LOCAL_RERANK_PROVIDERS` set in `src/core/budget/budget-tracker.ts:lookupPricing` (rerank-kind-only zero-pricing for the local provider prefix); doctor-fix at `src/commands/models.ts:probeRerankerConfig` reads `search.reranker.model` via `loadSearchModeConfig` + `resolveSearchMode` (closes file-plane / DB-plane divergence where doctor said "not configured" while live search was actively reranking — the field-plane `getRerankerModel()` read nothing writes); `probeRerankerReachability` reads the recipe's `default_timeout_ms` so CPU-only cold-start doesn't false-fail. - `src/core/ai/recipes/openrouter.ts` — OpenRouter openai-compatible recipe: single key, many providers via `openrouter:/` strings. `base_url_default: 'https://openrouter.ai/api/v1'`. Embedding touchpoint: `openai/text-embedding-3-small` at 1536 dims with Matryoshka `dims_options: [512, 768, 1024, 1536]`; `max_batch_tokens: 300_000` = OpenAI's aggregate-per-request token cap (NOT per-input). Chat touchpoint declares 8 curated entry points (gpt-5.2, gpt-5.2-chat, gpt-5.5, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-4.7, gemini-3-flash-preview, deepseek-chat) but openai-compat tier accepts any model ID; deliberately no `max_context_tokens` because OR's catalog spans 128K to 1M+. `supports_subagent_loop: false` is INFORMATIONAL — the real gate is `isAnthropicProvider()` in `src/core/model-config.ts` which hard-pins gbrain's subagent infra to Anthropic-direct. Declares `resolveDefaultHeaders(env)` returning OR's three attribution headers: `HTTP-Referer` (required for OR app-attribution), `X-OpenRouter-Title` (preferred), `X-Title` (back-compat alias); defaults to `https://gbrain.ai` / `gbrain`; forks override via `OPENROUTER_REFERER` / `OPENROUTER_TITLE` env vars. Smoke-tested by `test/ai/recipe-openrouter.test.ts` (incl. the shape-test regression guard: every model in the chat list matches `^[a-z0-9-]+\/[a-z0-9._-]+$`). @@ -136,18 +139,14 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/takes-quality-eval/pricing.ts` — fail-closed budget pricing for `eval takes-quality run --budget-usd N`. `MODEL_PRICING` is a curated `provider:model` allowlist (default panel + likely overrides) whose VALUES are derived from `model-pricing.ts` via `canonicalLookup`; an allowlisted id missing from canonical throws at module load. Schema is `{input_per_1m, output_per_1m}`. A model NOT on the allowlist aborts the run with an actionable error rather than guessing (distinct from `cross-modal-eval/runner.ts`, which silently estimates zero on unknown models — both now source numbers from canonical). - `src/core/budget/budget-tracker.ts` — keystone primitive for the brainstorm cost-cathedral wave. One typed error (`BudgetExhausted` with `reason: 'cost' | 'runtime' | 'no_pricing'`), one schema-stable audit JSONL at `~/.gbrain/audit/budget-YYYY-Www.jsonl`. Contracts: `record()` throws when cumulative spend exceeds cap (the cap is a real ceiling, not a suggestion); `reserve()` hard-fails with `reason: 'no_pricing'` when `maxCostUsd` is set AND the model is missing from pricing maps (warn-once preserved when cap is unset); `extractUsageFromError(err, fallback)` returns `err.usage` when the SDK provides it, else the pessimistic fallback (caller passes `maxOutputTokens`, not the optimistic pre-call estimate). `onExhausted(cb)` fires once synchronously BEFORE the throw propagates so callers can persist checkpoints. Replaces three parallel copies (inline brainstorm class, cycle/budget-meter, eval-contradictions). Adapts the old `BudgetMeter` (public shape preserved + `schema_version: 1` stamped on every dream-budget audit line). Pinned by 18 unit cases. - `src/core/audit-week-file.ts` — single source of truth for ISO-week audit JSONL filename math. Exports `isoWeek(d)`, `isoWeekFilename(prefix, now?)`, `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). Year-boundary correctness pinned by tests at 2020-W53 (the 53-week year), 2025-W01 rolling in from 2024-12-30 (Monday), 2026-W01. Four call sites migrated: `src/core/minions/handlers/shell-audit.ts`, `src/core/facts/phantom-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/cycle/budget-meter.ts`. Each keeps its `computeAuditFilename` thin wrapper for back-compat with existing tests. -- `src/core/ai/gateway.ts:withBudgetTracker` — gateway-layer enforcement via `AsyncLocalStorage`. `withBudgetTracker(tracker, fn)` installs the tracker on the module-internal store; every `gateway.chat / embed / rerank` call inside the scope auto-composes (reserve before, record in try/finally). Outside-scope calls are budget no-ops. Nested scopes restore the outer tracker on exit. `getCurrentBudgetTracker()` is the test seam. The chat path uses the pessimistic fallback on error paths; the embed path estimates input tokens from char count × recipe's `chars_per_token` because the AI SDK doesn't surface per-batch embed token usage; the rerank path estimates char count of query+docs. Pinned by 6 unit cases. - `src/core/diarize/payload-fitter.ts` — generic fit-arbitrarily-large-items-into-per-call-token-budget utility. `'batch'` strategy is deterministic token-budgeted chunking with no LLM calls. `'summarize'` strategy embed-clusters into ceil(items/4) groups via cheap deterministic nearest-neighbor on cosine, Haiku-summarizes each cluster via `Promise.allSettled` at parallelism=4. Each Haiku call composes the active BudgetTracker via the AsyncLocalStorage. Quality gate: when `success_ratio < min_success_ratio` (default 0.75), result is flagged `degraded: true` — the fitter preserves the successful subset; the caller decides whether to surface a partial result or abort. - `src/core/brainstorm/checkpoint.ts` — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB/run) so resume MERGES pre-crash ideas with post-resume ideas before the judge runs (a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume ` covers both failed AND never-attempted crosses); `--list-runs` prints run_ids mtime-newest-first; `--force-resume` bypasses the 7-day staleness gate. Cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by `test/e2e/brainstorm-resume.test.ts` (20 unit + 3 E2E cases incl. the merge contract). - `src/core/remediation-checkpoint.ts` — `doctor --remediate` checkpoint at `~/.gbrain/remediation/.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned, atomic `.tmp + rename`. `gbrain doctor --remediate --resume ` (no arg picks newest matching) loads it and skips completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases. - `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). Four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. 8-step resolution chain: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern (routes through `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids like `anthropic/claude-sonnet-4-6` classify correctly). `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves non-Anthropic, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` (the Anthropic Messages API tool-loop can't run on OpenAI/Gemini). `_resetDeprecationWarningsForTest()` also clears `_subagentTierWarningsEmitted`. Pinned by `test/model-config.serial.test.ts`. -- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` takes an optional 4th `extendedModels: ReadonlySet`: when the modelId is in that set the native-recipe allowlist throw is bypassed (user explicitly opted in via config, so provider rejection surfaces as `model_not_found` at HTTP call time and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — source typos still fail fast (the fail-fast contract for chat + expand + embed stays intact). -- `src/core/ai/gateway.ts` extension — module-scoped `_extendedModels: Map>>` registry feeds `assertTouchpoint`'s extended-model path without broadening unrelated surfaces. `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `registerConfigSelectedChatModel(model)` is the narrow runtime seam for a dedicated contextual-synopsis model: the ID joins the chat allowlist but remains rejected for embedding, expansion, and reranking. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. -- `src/core/minions/queue.ts` extension — `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`. +- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` takes an optional 4th `extendedModels: ReadonlySet`: when the modelId is in that set the native-recipe allowlist throw is bypassed (user explicitly opted in via config, so provider rejection surfaces as `model_not_found` at HTTP call time and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — source typos still fail fast (the fail-fast contract for chat + expand + embed stays intact). `parseModelId`: gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/models.ts` — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain), every per-task override (13 `PER_TASK_KEYS`, now including provider-neutral `models.contextual_synopsis` with legacy-key/env attribution), the alias map, and a source-of-truth column (`default` / `config: ` / `env: `). `gbrain models doctor [--skip=] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. Wired into `cli.ts` dispatch + `CLI_ONLY` set. A zero-token `embedding_config` probe runs FIRST, before any chat/expansion probes spend money: `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. `ProbeStatus` variant `'config'` + optional `fix?: string` on `ProbeResult` surface a paste-ready `gbrain config set ...` line in human + JSON output; touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`. - `src/core/init-embed-check.ts` — embedding-key validation at `gbrain init`. `runInitEmbedCheck(opts)` runs a config-only `diagnoseEmbedding` (catches a missing key for ANY provider) plus a best-effort `liveTestEmbed` (1-token `gateway.embed(['probe'], {inputType:'query', abortSignal})`, 5s `AbortController` timeout, never throws — catches an invalid/expired key). Loud warning to stderr; init still exits 0 (`--no-embedding` is the deferred-setup escape; `--skip-embed-check` / `GBRAIN_INIT_SKIP_EMBED_CHECK=1` skip the check). Builds the effective env (`process.env` + file-plane `openai/anthropic/zeroentropy_api_key` from `loadConfigFileOnly()` + `opts.apiKey`) and configures the gateway via `buildGatewayConfig` before diagnose/probe, so the check sees the same keys AND provider base URLs runtime will (no false "missing key" for config.json-keyed users; the probe hits the right endpoint). Init-specific warning text names `--no-embedding` / `--skip-embed-check`, not the sync-flavored `--no-embed`. Wired into `initPGLite` + `initPostgres` in `src/commands/init.ts`, with the result added to the `--json` envelope as `embedding_check {ok, reason?, live_ok?}`. Pinned by `test/init-embed-check.test.ts` (hermetic via the gateway embed-transport seam + `withEnv`). - `src/core/ai/build-gateway-config.ts` — `buildGatewayConfig(c: GBrainConfig): AIGatewayConfig`, extracted from `src/cli.ts` (which re-exports it for back-compat). Lets core modules (`init-embed-check.ts`) reuse it without importing the CLI entrypoint. Single owner of folding file-plane API keys (openai/anthropic/zeroentropy) into the gateway env and threading local-server `*_BASE_URL` env vars into base_urls. `process.env` wins EXCEPT empty-string / undefined values are dropped before the merge, so an injected empty `ANTHROPIC_API_KEY=''` (Claude Code neuters subprocess LLM calls this way) can't clobber a valid config-plane key; `'0'` / `'false'` are preserved. Pinned by `test/ai/build-gateway-config.test.ts`. -- `src/commands/doctor.ts` extension — `subagent_provider` check (layer 3 of 3). Resolves subagent model config in runtime order (`models.subagent` > `models.default` > `models.tier.subagent` > built-in default) and warns when the selected model lacks native tool-loop capability (message names the bad value + paste-ready fix `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK when subagent tier resolves to Anthropic. Tests in `test/doctor.test.ts`. - `src/core/skill-trigger-index.ts` — Shared loader that unions per-skill SKILL.md frontmatter `triggers:` with curated RESOLVER.md / AGENTS.md rows from `skillsDir` AND the parent dir (preserves the OpenClaw workspace-root layout). UNION semantics: explicit RESOLVER.md rows ADD to frontmatter triggers (don't replace). Dedup keyed on `(skillPath, trigger.trim().toLowerCase())`. Three consumers fold through this primitive — `checkResolvable`, `runRoutingEvalCli`, `mounts-cache.composeResolvers` — so fixing frontmatter reaches all of them. Exports `loadSkillTriggerIndex(skillsDir): SkillTriggerEntry[]`, `entriesToResolverContent(entries): string` (synthesizes a markdown-table resolver string for `runRoutingEval`'s string-content API), `findPrimaryResolverPath(skillsDir): string | null`, the `FRONTMATTER_SECTION` constant, and `_resetWarnedSkillsForTests`. Skip rules: non-directory entries, `_*`/`.*` prefixes, `conventions/`+`migrations/` subdirs, skills with no `SKILL.md` (deprecated `install/` graceful-skipped), no `triggers:` array, or malformed YAML (warn-once + skip). Reuses `parseSkillFrontmatter` from `src/core/skill-frontmatter.ts` (regex-based, not full YAML). Pinned by `test/skill-trigger-index.test.ts` (18 hermetic cases). CI gate `bun run check:resolver` (= `bun src/cli.ts check-resolvable --strict --skills-dir skills/`) wired into `bun run verify`. - `src/core/skill-catalog.ts` — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, so "using" one = fetching its body then calling the gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack): (1) **publish gate** — `assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement** — `assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist** — `GetSkillResult.frontmatter` projects a safe subset; private `writes_to` + `sources` dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`) over the file plane (`ctx.config.mcp`). Tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` vs `unavailable_tools`; `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS`) carries the "these are prose, follow-then-call-tools" protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply. `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws). Config keys in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills`/`mcp.publish_skills_prompted`/`mcp.skills_dir` + `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new installs (existing config wins on re-init). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until owner opts in). Two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints:{name:'skills'}`; `get_skill` taking `name` + `cliHints:{name:'skill', positional:['name']}`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array). Descriptions in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI: `gbrain skills` / `gbrain skill `. Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local) over `test/fixtures/skill-catalog/`. - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts `conventions/quality.md` and `_brain-filing-rules.md`). `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. `parseResolverEntries` accepts BOTH the markdown table AND a compact list format (`- **skill-name**: trigger1 | trigger2 | trigger3` or `- skill-name: trigger1 | trigger2`); shapes can mix in one file, folded by the multi-resolver merge. Skill name MUST be kebab-lowercase (regex `[a-z][a-z0-9-]+`) so prose bullets like `- **Note**:`/`- **Convention**:`/`- **TODO**:` don't false-match as skill rows. `skillPath` is ALWAYS derived as `skills//SKILL.md`: an optional `→ \`skills/path\`` (or ASCII `->`) suffix is stripped from the trigger but NOT honored as the path — two consumers (`routing-eval.ts:skillSlugFromPath`, the manifest lookup) assume the convention; use the table format for non-conventional paths. Multi-trigger rows fan out to one entry per trigger sharing the same `skillPath`; `checkResolvable` dedupes so the reachability count counts each skill once. Pinned by `test/check-resolvable.test.ts` (11 cases: bold+plain forms, Unicode+ASCII suffix strip, ellipsis filter, empty pipe segments, mixed shapes, prose-bullet rejection) + `test/check-resolvable-openclaw-compact.test.ts` (8 cases over `test/fixtures/openclaw-compact-resolver/` and `test/fixtures/openclaw-mixed-merge/`). Tutorial: `docs/guides/scaling-skills.md` (three-tier scaling: ~300-skill agent to ~4K tokens/turn from ~25K). @@ -159,7 +158,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/skillify-check.ts` — `gbrain skillpack-check` agent-readable health report. Exit 0/1/2 for CI gating; JSON for debugging. Wraps `check-resolvable --json`, `doctor --json`, and migration ledger into one payload. Required item 12 (`brain_first_compliance`) calls `analyzeSkillBrainFirst()` on the candidate SKILL.md; exits 1 when the verdict is `missing_brain_first` (external-lookup pattern present, no callout, no `brain_first: exempt`). The scaffold path in `src/core/skillify/templates.ts` pre-inserts the canonical Convention callout into new SKILL.md files so freshly-scaffolded skills pass item 12. - `src/commands/book-mirror.ts` — `gbrain book-mirror --chapters-dir --slug [flags]`. Submits N read-only subagent jobs (one per chapter; `allowed_tools: ['get_page', 'search']`), waits for all via `waitForCompletion`, reads each child's `job.result`, assembles two-column markdown CLI-side, writes a single operator-trust `put_page` to `media/books/-personalized.md`. Trust narrowing happens at the tool-allowlist layer (subagents can't call put_page) so untrusted EPUB content can't prompt-inject any people page. Cost-estimate prompt before launching; refuses to spend in non-TTY without `--yes`. Per-chapter idempotency keys (`book-mirror::ch-`) for retry-friendly re-runs. Partial-failure: assembles completed chapters + a `## Failed chapters` section. Pinned by `test/book-mirror.test.ts` (9 cases). - `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,scaffold,reference,migrate-fence,scrub-legacy,harvest,harvest-lint,copy,apply-hunks,diff-text,installer}.ts` — managed-block install model retired; `install`/`uninstall` removed (exit non-zero with a hint to the replacement). Surface: `scaffold` (one-time additive copy via `copyArtifacts` in `copy.ts`; refuses to overwrite; partial-state fills missing paired sources declared in SKILL.md frontmatter `sources:`), `reference` (read-only diff lens + `--apply-clean-hunks` two-way auto-apply via pure-JS unified-diff parser/applier in `apply-hunks.ts` + `diff-text.ts`), `migrate-fence` (one-shot strip of legacy fence; cumulative-slugs receipt → row-parsing fallback; preserves rows verbatim as user-owned routing), `scrub-legacy-fence-rows` (opt-in row cleanup with skill-present + non-empty-triggers gate), `harvest` (host→gbrain inverse with symlink-reject + canonical-path containment via a `validateUploadPath`-style gate + default-on privacy linter in `harvest-lint.ts` against `~/.gbrain/harvest-private-patterns.txt` plus built-in a built-in fork-name pattern + email + Slack-channel patterns; rollback on match). Paired-source declarations live in each SKILL.md's frontmatter `sources:` array (validated by `loadSkillSources` in `bundle.ts`). `autoDetectSkillsDir` (in `src/core/repo-root.ts`) has a `cwd_walk_up` tier ahead of `~/.openclaw/workspace` (`$OPENCLAW_WORKSPACE` precedence preserved). `gbrain skillpack check --strict` exits non-zero on drift (CI gate); top-level `gbrain skillpack-check` keeps exit-1-on-issues for cron. Companion editorial skill `skills/skillpack-harvest/SKILL.md` drives the genericization checklist. Doc: `docs/guides/skillpacks-as-scaffolding.md`. Test coverage across `test/skillpack-{copy,scaffold,reference,reference-apply,apply-hunks,migrate-fence,scrub-legacy,harvest,harvest-lint,frontmatter-sources}.test.ts` + 9-case E2E in `test/e2e/skillpack-flow.test.ts`. `installer.ts` + `test/skillpack-install.test.ts` survive — `gbrain skillpack diff` still uses `diffSkill` from there. -- `src/core/skillpack/{manifest-v1,tarball,state,remote-source,trust-prompt,bootstrap-display,scaffold-third-party,registry-schema,registry-client,rubric,doctor,init-scaffold,pack-publish,endorse,audit}.ts` + `examples/skillpack-reference/` + `docs/skillpack-anatomy.md` + `scripts/build-skillpack-anatomy.ts` — third-party skillpack ecosystem. `gbrain skillpack scaffold ` resolves the spec via `classifySpec`, fetches through SSRF-hardened `git-remote.ts` (git) or extracts the tarball into `~/.gbrain/skillpack-cache/////`, validates `skillpack.json` (api_version `gbrain-skillpack-v1`), checks `gbrain_min_version`, surfaces a TOFU first-install identity-confirm prompt (author + source + pinned commit + tarball SHA + tier; non-TTY requires `--trust`), records the pin in machine-owned `~/.gbrain/skillpack-state.json` (schema `gbrain-skillpack-state-v1`, atomic `.tmp + rename`, `isAlreadyTrusted` skips re-prompt on author+pin match), runs through `enumerateScaffoldEntries` → `copyArtifacts` (one-time additive, refuses to overwrite), then DISPLAYS `runbooks/bootstrap.md` WITHOUT executing (deliberately does not auto-execute). Registry catalog at `garrytan/gbrain-skillpack-registry` split into `registry.json` (PR-able, `gbrain-registry-v1`) + `endorsements.json` (Garry-only overlay, `gbrain-endorsements-v1`); `effectiveTier` merges. `registry-client.ts` fetches both via `If-None-Match` etag with 1h soft-TTL + stale-fallback (origins `fresh_fetch | cache_warm | cache_soft_stale | cache_hard_stale`); hard-fail only on no-cache + no-network. CLI: `gbrain skillpack {search,info,registry,doctor,init,pack,endorse}`. Doctor walks `SKILLPACK_RUBRIC_V1` (10 binary dimensions: 5 required CORE — manifest_valid, skills_have_skill_md, routing_evals_present ≥5 intents, skills_have_unique_triggers MECE, changelog_present_and_current — and 5 quality BADGES — unit_tests_present, e2e_tests_present, llm_eval_present ≥3 cases, bootstrap_runbook_present, license_present); tier eligibility: `endorsed` needs all 10, `community` needs core + ≥3 badges, `experimental` needs core only, `blocked` when any core fails. `--quick` ~5s structural sweep; `--fix --yes` auto-scaffolds `auto_fixable: true` dimensions and refuses to overwrite files whose mtime is newer than `skillpack.json`. `gbrain skillpack init ` lands 11 files (skillpack.json, SKILL.md, routing-eval.jsonl, test/example.test.ts, e2e/example.e2e.test.ts, evals/example.judge.json, runbooks/{bootstrap,uninstall,upgrade-template}.md, CHANGELOG, README, LICENSE); freshly-init'd scores 10/10; `--minimal` skips test/e2e/evals. `gbrain skillpack pack` packs a deterministic tarball via GNU tar (`--sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner` + `GZIP=-n` + `TZ=UTC`); refuses on `tier_eligibility === 'blocked'`. Extract caps (5000 files / 100MB total / 1MB per file / 255-char paths / 100:1 compression ratio); rejects symlinks/hardlinks/devices/FIFOs. `gbrain skillpack endorse [--tier ...] [--push] [--dry-run]` runs in a clone of the registry repo: validates the pack in `registry.json`, mutates `endorsements.json` via pure `applyEndorsement`, stable-key-orders the write, commits `endorse: -> `, optionally pushes. JSONL audit at `~/.gbrain/audit/skillpack-YYYY-Www.jsonl` (ISO-week rotated, honors `GBRAIN_AUDIT_DIR`). `examples/skillpack-reference/` is a 10/10 reference pack pinned by `test/e2e/skillpack-third-party.test.ts`. `docs/skillpack-anatomy.md` auto-generated via `scripts/build-skillpack-anatomy.ts` (`--check` for CI drift). CLI dispatch in `src/commands/skillpack.ts` disambiguates third-party (contains `/`, `://`, `.tgz`) from bundled-skill kebab; kebab routes bundled-first, registry-fallback. Tests: `test/skillpack-{manifest-v1,tarball,state,remote-source,trust-prompt,registry-schema,registry-client,rubric,doctor,init-scaffold,pack-publish,endorse,audit,scaffold-third-party}.test.ts` + `test/e2e/skillpack-third-party.test.ts`. Spec at `docs/designs/SKILLPACK_REGISTRY_V1_SPEC.md`. +- `src/core/skillpack/{manifest-v1,tarball,state,remote-source,trust-prompt,bootstrap-display,scaffold-third-party,registry-schema,registry-client,rubric,doctor,init-scaffold,pack-publish,endorse,audit}.ts` + `examples/skillpack-reference/` + `docs/skillpack-anatomy.md` + `scripts/build-skillpack-anatomy.ts` — third-party skillpack ecosystem. `gbrain skillpack scaffold ` resolves the spec via `classifySpec`, fetches through SSRF-hardened `git-remote.ts` (git) or extracts the tarball into `~/.gbrain/skillpack-cache/////`, validates `skillpack.json` (api_version `gbrain-skillpack-v1`), checks `gbrain_min_version`, surfaces a TOFU first-install identity-confirm prompt (author + source + pinned commit + tarball SHA + tier; non-TTY requires `--trust`), records the pin in machine-owned `~/.gbrain/skillpack-state.json` (schema `gbrain-skillpack-state-v1`, atomic `.tmp + rename`, `isAlreadyTrusted` skips re-prompt on author+pin match), runs through `enumerateScaffoldEntries` → `copyArtifacts` (one-time additive, refuses to overwrite), then DISPLAYS `runbooks/bootstrap.md` WITHOUT executing (deliberately does not auto-execute). Registry catalog at `garrytan/gbrain-skillpack-registry` split into `registry.json` (PR-able, `gbrain-registry-v1`) + `endorsements.json` (maintainer-only overlay, `gbrain-endorsements-v1`); `effectiveTier` merges. `registry-client.ts` fetches both via `If-None-Match` etag with 1h soft-TTL + stale-fallback (origins `fresh_fetch | cache_warm | cache_soft_stale | cache_hard_stale`); hard-fail only on no-cache + no-network. CLI: `gbrain skillpack {search,info,registry,doctor,init,pack,endorse}`. Doctor walks `SKILLPACK_RUBRIC_V1` (10 binary dimensions: 5 required CORE — manifest_valid, skills_have_skill_md, routing_evals_present ≥5 intents, skills_have_unique_triggers MECE, changelog_present_and_current — and 5 quality BADGES — unit_tests_present, e2e_tests_present, llm_eval_present ≥3 cases, bootstrap_runbook_present, license_present); tier eligibility: `endorsed` needs all 10, `community` needs core + ≥3 badges, `experimental` needs core only, `blocked` when any core fails. `--quick` ~5s structural sweep; `--fix --yes` auto-scaffolds `auto_fixable: true` dimensions and refuses to overwrite files whose mtime is newer than `skillpack.json`. `gbrain skillpack init ` lands 11 files (skillpack.json, SKILL.md, routing-eval.jsonl, test/example.test.ts, e2e/example.e2e.test.ts, evals/example.judge.json, runbooks/{bootstrap,uninstall,upgrade-template}.md, CHANGELOG, README, LICENSE); freshly-init'd scores 10/10; `--minimal` skips test/e2e/evals. `gbrain skillpack pack` packs a deterministic tarball via GNU tar (`--sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner` + `GZIP=-n` + `TZ=UTC`); refuses on `tier_eligibility === 'blocked'`. Extract caps (5000 files / 100MB total / 1MB per file / 255-char paths / 100:1 compression ratio); rejects symlinks/hardlinks/devices/FIFOs. `gbrain skillpack endorse [--tier ...] [--push] [--dry-run]` runs in a clone of the registry repo: validates the pack in `registry.json`, mutates `endorsements.json` via pure `applyEndorsement`, stable-key-orders the write, commits `endorse: -> `, optionally pushes. JSONL audit at `~/.gbrain/audit/skillpack-YYYY-Www.jsonl` (ISO-week rotated, honors `GBRAIN_AUDIT_DIR`). `examples/skillpack-reference/` is a 10/10 reference pack pinned by `test/e2e/skillpack-third-party.test.ts`. `docs/skillpack-anatomy.md` auto-generated via `scripts/build-skillpack-anatomy.ts` (`--check` for CI drift). CLI dispatch in `src/commands/skillpack.ts` disambiguates third-party (contains `/`, `://`, `.tgz`) from bundled-skill kebab; kebab routes bundled-first, registry-fallback. Tests: `test/skillpack-{manifest-v1,tarball,state,remote-source,trust-prompt,registry-schema,registry-client,rubric,doctor,init-scaffold,pack-publish,endorse,audit,scaffold-third-party}.test.ts` + `test/e2e/skillpack-third-party.test.ts`. Spec at `docs/designs/SKILLPACK_REGISTRY_V1_SPEC.md`. - `src/core/archive-crawler-config.ts` — safety gate for the `archive-crawler` skill. Refuses to run unless `archive-crawler.scan_paths:` is explicitly set in the brain repo's `gbrain.yml`. Mirrors the storage-config.ts parsing pattern (sibling file, separate concern from storage tiering). `loadArchiveCrawlerConfig(repoPath)` throws `ArchiveCrawlerConfigError(missing_section | empty_scan_paths | invalid_path | parse_error)`. `normalizeAndValidateArchiveCrawlerConfig` rejects relative paths and `..` traversal; `~` is expanded; paths are stored resolved and terminated with the PLATFORM separator (`path.sep`) so error output reads natively on each OS. `isPathAllowed(candidate, config)` is the runtime per-file gate (scan_paths prefix-match with directory-boundary correctness; deny_paths overrides). Candidate, scan_paths and deny_paths all funnel through the private `toComparablePrefix()` before the prefix test — on Windows it folds `\`→`/` and lowercases (NTFS is case-insensitive, so a deny_path spelled `Private` must still match `private`, else the gate fails OPEN); on POSIX it is identity apart from the trailing separator, deliberately NOT folding, since `\` is a legal filename character and paths are case-sensitive. Storing a native separator while appending a hardcoded `/` is the mixed-separator bug that made `isPathAllowed` deny every real path on Windows; the two functions must stay symmetric or the prefix test is meaningless. Pinned by `test/archive-crawler-config.test.ts` (26 cases, platform-selected fixtures + `it.if`-gated win32/POSIX comparison semantics). - `test/helpers/cli-pty-runner.ts` — generic real-PTY harness (~470 lines) using pure `Bun.spawn({terminal:})` (Bun 1.3.10+; engines.bun pin in package.json). Generic primitives only, no plan-mode orchestrators. Exports `launchPty`, `resolveBinary`, `stripAnsi`, `parseNumberedOptions`, `optionsSignature`, `isNumberedOptionListVisible`, `isTrustDialogVisible`. Self-tests in `test/cli-pty-runner.test.ts` (24 cases). - `src/core/skillpack/{init-brain-pack,brain-pack-advisory,brain-pack-lint,brain-resident-locate,nag-state}.ts` (#2180) — brain-resident skillpacks. `manifest-v1.ts` gains optional `brain_resident` + `schema_pack` (additive). `runInitBrainPack` scaffolds a pack beside brain content (`brain_resident:true`, exact `gbrain_min_version`, 5-section machine-parseable README); `applyWritePlan` is factored out of `init-scaffold.ts` for the shared refuse-overwrite loop. `brain-pack-lint.lintBrainPackTools` validates each skill's `tools:` against the serving op set (E6 version-skew). Topology A: `src/commands/sources.ts` `runAdd` prints `brain-pack-advisory` to stderr after `opsAddSource`, fail-open; `nag-state.ts` (`~/.gbrain/skillpack-nag-state.json`) keys declines by `(source-repo brain_id, source_id, pack_name)` with pure `decideNagAction` (first/reminder/version-bump/ceiling) — declines count ONLY on CLI-interactive displays. Topology B: `brain-resident-locate.loadResidentPacksForServer` (source-scoped via `sourceScopeOpts`) backs the `list_brain_skillpack` op; `getResidentSkillDetail` backs `get_skill` `source_id`; `scaffold_spec` is the git source, never a server FS path. Tests: `test/skillpack-{init-brain-pack,nag-state,brain-resident-locate}.test.ts` + the brain-resident cases in `test/skillpack-manifest-v1.test.ts`. @@ -185,11 +184,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/cycle/extract-atoms-drain.ts` — pure single-hold bounded drain for the silent lens-phase backlog. `runExtractAtomsDrain(deps, opts)` over injected deps (`withLock`, `runBatch`, `countRemaining`, `now`, optional `onBatch`) loops bounded batches under ONE continuous lock hold, **rediscovering eligibility each batch** (idempotent NOT-EXISTS-on-`source_hash`, so content mutated by a concurrent process simply doesn't match — no cross-window stale cursor), until the backlog is empty OR the time window elapses. Returns `{phase, status, extracted, skipped, remaining, batches, stopped}`. Backs `gbrain dream --phase extract_atoms --drain`. Takes the SAME `cycleLockIdFor(sourceId)` the routine cycle takes (a concurrent autopilot tick genuinely defers with `cycle_already_running`); NO release/reacquire-between-windows primitive. The shared wiring helper `runExtractAtomsDrainForSource(engine, {sourceId, windowSeconds, brainDir?, maxBatches?, onBatch?})` owns the lock+batch+count+defer wiring (dynamic imports of db-lock/cycle/extract-atoms keep the pure loop cheap to unit-test) and is the ONE drain path for three callers — `gbrain dream --drain` (which calls it), the `extract-atoms-drain` Minion handler, and autopilot auto-drain — so lock id / window / defer-on-busy can't drift. `sourceId: undefined` → legacy `gbrain-cycle` lock + `'default'` extraction; a real id → `gbrain-cycle:`. `LockUnavailableError` propagates to the caller (each reports the busy case its own way). Pinned by `test/extract-atoms-drain.test.ts`. - `scripts/check-worker-lock-renewal-shape.sh` — CI guard wired into `bun run verify`. Two invariants on `src/core/minions/worker.ts`: (1) the bug pattern `lockTimer = setInterval(async ...)` must NOT appear (narrowed via `lockTimer =` prefix so unrelated `setInterval(async)` calls — like the stall detector — don't false-fire), (2) `runLockRenewalTick` must remain referenced so the pure-function test seam survives refactors. Bug-pattern-specific by design — a future refactor to `setTimeout`-recursion or `AbortController`-based scheduling passes as long as the bug pattern stays absent. POSIX ERE + `[[:space:]]` for BSD-grep portability. Honors `GBRAIN_LOCK_RENEWAL_SHAPE_TARGET` env override for fixture-based meta-tests. Pinned by `test/scripts/check-worker-lock-renewal-shape.test.ts` (5 cases). - `src/core/doctor-cause-rank.ts` — pure cause-ranking for `gbrain doctor`. `rankIssues(checks)` returns non-ok checks ordered fail-before-warn then root-before-symptom then name (deterministic). `ROOT_CAUSE_CHECKS` / `SYMPTOM_CHECKS` are ORDERING ONLY — tier membership asserts no causality. `downstream_of` is set ONLY from a small map of KNOWN grounded edges (`queue_health` / `supervisor` → `worker_oom_loop`, since they read the same `aborted: watchdog` / `rss_watchdog` source) AND only when the named root is itself failing — never a root×symptom cartesian (co-occurrence never implies causality). `fix` prefers `details.fix_hint` else the message. `CAUSE_GRAPH_NAMES` + `allKnownCheckNames()` back a drift guard asserting every graphed name is a real check. Consumed by `computeDoctorReport` (`top_issues` field, additive, schema_version stays 2) + the "Top issues (ranked by cause)" header in `outputResults`. Pinned by `test/doctor-cause-rank.test.ts`. -- `src/commands/doctor.ts` extension — `computeWorkerOomLoopCheck(engine)` is the single authoritative OOM-loop signal, unioning supervised `summarizeCrashes(readRecentSupervisorEvents(24)).by_cause.rss_watchdog` (cross-week read via `readRecentSupervisorEvents` so a Monday window can't lose Sunday) + bare-worker `minion_jobs error_text='aborted: watchdog'` count (Postgres-only; the same source `queue_health` subcheck 3 reads). Cap comes from the latest `rss_watchdog_loop` breaker alert's `max_rss_mb`, else `resolveDefaultMaxRssMb()` fallback. fail at breaker-tripped or oomKills≥5, warn at ≥1, null otherwise. `computePoolReapHealthCheck(engine)` is the Postgres-only `pool_reap_health` check reading `readRecentPoolRecoveries(1)` — fail when reconnect failures>0 (reconnect throwing is the actionable signal), warn at ≥10 reaps/hr (pooler thrash), null otherwise. Both registered in `buildChecks` after the `supervisor` block. The `supervisor` causeStr carries `rss=N (see worker_oom_loop)` and `queue_health`'s watchdog message cross-references `worker_oom_loop`. `DoctorReport.top_issues` + the cause-ranked render header. `worker_oom_loop` + `pool_reap_health` registered under ops in `doctor-categories.ts`. Pinned by `test/doctor-worker-oom-loop.test.ts`, `test/doctor-pool-reap-health.test.ts`. -- `src/commands/doctor.ts` extension — `supervisor_singleton` check (#1849), a SEPARATE check from `supervisor` (same split precedent as the niceness check) so a singleton-divergence warn can't clobber the crash/liveness precedence. Runs only when a `started` supervisor event was seen in the last 24h and a live engine is available. Reads the queue-scoped DB lock row (`gbrain_cycle_locks WHERE id = supervisorLockId(queue)`) and compares the lock holder (`holder_host:holder_pid`) against the local pidfile holder via the pure `classifySupervisorSingleton`. `mismatch` → warn (a second supervisor may be running with a different `--max-rss`; message names both holders, the effective cap from the `started` event's `max_rss_mb`, and the fix `gbrain jobs supervisor stop`); `single` → ok (names holder + cap); `no_lock` → no check emitted. Best-effort try/catch (silent skip on brains without the lock table). Registered under ops in `doctor-categories.ts` as `supervisor_singleton`. Pinned by `test/supervisor-db-lock.test.ts` + `test/doctor.test.ts`. - `src/core/audit/pool-recovery-audit.ts` — reap/reconnect audit on the shared `audit-writer` cathedral. Events: `reap_detected` (CONNECTION_ENDED), `reconnect_other` (network/auth/health-check), `reconnect_succeeded`, `reconnect_failed`. `readRecentPoolRecoveries(hours=1)` returns `{reaps, recoveries, failures, others, events}`. Error summaries route through `redactConnectionInfo` before truncation (DSN/host/IP safe). Emitted ONLY from `PostgresEngine.reconnect(ctx?)` (the rare reap-retry path, near-zero hot-path cost); `reconnect()` classifies the threaded error via `isConnectionEndedError` (in retry-matcher.ts) so only true pooler reaps are labeled `reap_detected`. The retry callback in retry.ts threads the triggering error as `(ctx?: {error?}) => Promise`. Pinned by `test/audit/pool-recovery-audit.test.ts`. -- `src/commands/autopilot.ts` extension — per-source `extract_atoms` auto-drain. Postgres-only block after the freshness fan-out: gated on `autopilot.auto_drain.enabled` (default true) AND `!packDeclaresPhase(engine,'extract_atoms')` (the silent-backlog condition) AND per-source `countExtractAtomsBacklog > threshold` (default 25) AND a daily cap `floor(max_usd_per_day / ~$0.30)`. Enumerates `loadAllSources`. Submits the PROTECTED `extract-atoms-drain` job (`{allowProtectedSubmit:true}`) with a UTC-day time-sloted idempotency key `autopilot-extract-atoms-drain::` (a static key would block the source after the first job completed). `src/core/minions/protected-names.ts` adds `extract-atoms-drain`; `src/commands/jobs.ts` registers the handler (thin wrapper over `runExtractAtomsDrainForSource`, `LockUnavailableError` → `{deferred:true}`); `src/core/config.ts` adds the `autopilot.auto_drain.*` config keys + the `autopilot.` key prefix. Pinned by `test/extract-atoms-drain-handler.test.ts`, `test/autopilot-auto-drain-wiring.test.ts`. -- `src/commands/doctor.ts:checkBatchRetryHealth` — `batch_retry_health` check surfacing Supavisor circuit-breaker incidents. Wired into both `runDoctor` (local) and `doctorReportRemote` (thin-client). Reads last 24h. States: `ok` (zero exhausted in 24h OR <3 from a single site), `warn` (>=3 same-site OR >=5 cross-site), `fail` (>=20 sustained breaker). Surfaces bad `GBRAIN_BULK_*` env at doctor startup. Corrupt-JSONL tolerant. Paste-ready fix hints in every warn/fail message. Also reads `readRecentDbDisconnects(24)` and appends `Disconnect-call audit: N call(s) in 24h (most recent caller: ).` to ALL three message paths so connection-incident signal is greppable from one `gbrain doctor --json` call (module-import wrapped in try/catch so older brains without the audit file degrade silently). Pinned by `test/doctor-batch-retry.test.ts` (10 cases). - `src/core/audit/db-disconnect-audit.ts` — JSONL audit for every call to `db.disconnect()` and `PostgresEngine.disconnect()`. Built on `audit-writer.ts`. Schema: `{ts, engine_kind: 'postgres'|'pglite'|'unknown', connection_style: 'module'|'instance'|'unknown', caller_stack, command, pid}`. `caller_stack` captured via `new Error().stack` truncated to ~20 frames so operators identify the offending caller without inflating JSONL. Privacy: stack frames carry file paths but NO SQL content / row data / user strings. File: `~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` (honors `GBRAIN_AUDIT_DIR`). `readRecentDbDisconnects(hours=24)` walks current + previous ISO week and returns `{count, most_recent_caller, files_scanned}`. Wired into `src/core/db.ts:disconnect` and `src/core/postgres-engine.ts:disconnect`, logging BEFORE the early-return branches so even no-op disconnects on never-connected engines are recorded (that case may itself be a caller-side bug). Pinned by `test/db-disconnect-audit.test.ts` (6 cases: round-trip, stack truncation, sort order, empty-dir nulls, stable feature name, EROFS best-effort). - `src/core/facts/queue.ts:FactsQueue.drainPending` — method `drainPending({timeout?: number}): Promise<{drained, unfinished}>`. Semantically distinct from `shutdown()` (which calls `this.internalAbort.abort()` and would abort the very facts:absorb worker trying to log its post-completion event). Drain lets in-flight finish; only the wait is bounded. Default timeout `1000ms` so commands that don't enqueue facts pay one fast 0ms check before exit. `src/cli.ts` op-dispatch finally block awaits `getFactsQueue().drainPending({timeout: 1000})` BEFORE `engine.disconnect()`. Lazy-import keeps the facts-queue module off the hot path for ops that never touch it. Closes the trailing `'No database connection'` line after `gbrain capture` (post-page-write facts:absorb outlived the CLI process). Pinned by `test/facts-queue-drain-pending.test.ts` (4 cases: empty fast-path, in-flight settled without abort, unfinished count on timeout, default timeout = 1000ms). - `scripts/check-no-double-retry.sh` + `scripts/check-batch-audit-site.sh` — CI lint guards wired into `bun run verify`. The former greps src/ for `withRetry(...engine.{addLinksBatch|addTimelineEntriesBatch|upsertChunks})` patterns and fails the build on hit (prevents 3×3=9 retry amplification on incomplete reverts). The latter extracts every string-literal `auditSite: '...'` from src/ and validates each appears in the `BATCH_AUDIT_SITES` const in `src/core/retry.ts` (typo guard — prevents fragmented doctor output). @@ -199,54 +194,36 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/extraction-review.ts` — Extraction quarantine lane markers (issue #160), sibling of `src/core/quarantine.ts` / `embed-skip.ts` (frontmatter-key pattern, no schema migration). Auto-extracted stubs from untrusted input carry the PAIR `provenance: 'auto-extracted'` + `status: 'unverified'` (both required — user pages with their own `status`/`provenance` never match). Exports `quarantineMarkers()`, `isUnverifiedExtraction()` (JS predicate) and `unverifiedExtractionFragment(alias)` — the single SQL source of truth consumed by `buildSourceFactorCase` (namespace source-boost guard), both engines' `getUnverifiedExtractionPageIds`, the `extraction_pending` op, and the `unverified_extractions` doctor check, so filter and marker keys can never drift. Consequences: unverified stubs are excluded from the compiled-truth fusion boost + the `people/`/`companies/` source-boost (rank as ordinary content), stamped `unverified: true` in search results (`stampUnverifiedExtractions`, hybrid.ts), listed by `extraction_pending`, promoted (status → `verified`, provenance kept for audit) or rejected (soft-delete) by the owner-only `extraction_review` op. Pinned by `test/extraction-review.test.ts` (PGLite) + `test/e2e/extraction-review-postgres.test.ts` (live Postgres parity). - `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich::')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `…` data-envelope delimiters (injection escape, mirrors the `` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity). - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping. -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp per page when every chunk embedded cleanly. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. Embed failures are never silent (#3037): all three page paths embed via `embedPageTexts`, which tries the page's chunks in one batch and, on a PERMANENT request-shaped failure (non-429, non-`AITransientError`, non-auth), retries once per chunk so one bad chunk costs one chunk instead of darkening the whole page (failed chunks stay `embedding IS NULL` for the next `--stale` pass; a partially-failed page is never signature-stamped). Rate-limit/outage/auth failures do NOT fan out (cost bounding — `embedBatchWithBackoff` already owns 429 backoff). Failed chunk counts land on `EmbedResult.failures` + capped `failure_samples`, and `src/cli.ts`'s embed case sets a non-zero exit verdict on `failures > 0` (mirror of the `import` errors>0 guard). Pinned by `test/embed-partial-failure-3037.serial.test.ts` + `test/embed-exit-code-3037.serial.test.ts` (real spawned CLI). +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp per page when every chunk embedded cleanly. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. Embed failures are never silent (#3037): all three page paths embed via `embedPageTexts`, which tries the page's chunks in one batch and, on a PERMANENT request-shaped failure (non-429, non-`AITransientError`, non-auth), retries once per chunk so one bad chunk costs one chunk instead of darkening the whole page (failed chunks stay `embedding IS NULL` for the next `--stale` pass; a partially-failed page is never signature-stamped). Rate-limit/outage/auth failures do NOT fan out (cost bounding — `embedBatchWithBackoff` already owns 429 backoff). Failed chunk counts land on `EmbedResult.failures` + capped `failure_samples`, and `src/cli.ts`'s embed case sets a non-zero exit verdict on `failures > 0` (mirror of the `import` errors>0 guard). Pinned by `test/embed-partial-failure-3037.serial.test.ts` + `test/embed-exit-code-3037.serial.test.ts` (real spawned CLI). applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title/slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter allowed. Pinned by `test/embed-skip.test.ts`. both inline sliding-pool sites (`embedAll` simple at `:458-467` and `embedAllStale` paginated + AbortSignal at `:586-632`) call `runSlidingPool` from the shared worker-pool helper. Invariant-level contract preserved (counts + cost + AbortSignal propagation + per-batch rate-limit retry via `embedBatchWithBackoff`); byte-equality on progress-event ORDERING is NOT promised. The `GBRAIN_EMBED_CONCURRENCY || 20` default is preserved and embed bypasses `resolveWorkersWithClamp` because the 20-worker default would otherwise silently change every brain's embed hot path. Pinned by `test/embed-helper-migration.test.ts` (asserts the helper is wired in AND the pre-migration `let nextIdx = 0` + `Promise.all(Array.from({length: numWorkers}, ...))` shapes are gone). wires `--background` as the reference integration for the `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job, prints `job_id=N` to stdout, exits 0. Composable: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same pattern in a follow-up wave. #1737: `runEmbedCore` accepts an optional `signal` threaded down both the `--stale` and `--all` paths (`embedAllStale`/`embedAll`/`embedPage`); each composes it with the internal wall-clock budget via `anySignal` and checks `isAborted`/`effectiveSignal.aborted` in every per-slug loop, page-claim pool, and `embedBatch` call, so a worker abort (wall-clock timeout / lock loss / SIGTERM) stops embedding within a batch. Pinned by `test/embed.serial.test.ts`. - `src/core/retrieval-upgrade-planner.ts` — `runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL THREE dim-pinned text-embedding-space columns at `targetDim` — `content_chunks.embedding`, `query_cache.embedding`, `facts.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache + facts are created at brain-birth width by `migrate.ts` and NO migration ever ALTERs them, so omitting either leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all three widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`. - `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state` → `runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions` → `persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})` → `SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector). - `src/commands/migrate-embeddings.ts` — `gbrain migrate embeddings --to [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite). - `src/core/conversation-parser/` — 17-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (17 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md). - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). -- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. +- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). `deleteOrphanFactsForPage(engine, sourceId, slug)` provides delete-orphans-first replay safety — wipes facts from a prior crashed run for this (sourceId, slug) before re-extracting, closing the "terminal audit row written after partial insertFacts failure" class. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Checkpoint state is a shared `cpMap: Map` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-workers.test.ts` + the existing extract-conversation-facts behavioral tests. with `src/commands/doctor.ts` durable outcome authority: page completion survives operation-checkpoint GC through versioned terminal audit rows (`cli:extract-conversation-facts:terminal:v2`), while recognized pages with no eligible segment use the separate `cli:extract-conversation-facts:non-extractable:v2` source. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed delete-first. `no_match`, errors, cancellation, and dry runs never become durable negatives. Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. - `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. #2576: markdown links, bare-slug prose refs, and slash-shaped wikilinks match ANY dir-shaped path (`ANY_DIR_SEGMENT`), not a directory whitelist — nonexistent targets are dropped by the persist paths' page-existence checks (`resolveCandidateSources`, put_page's allSlugs filter, `addLinksBatch` INNER JOINs) and counted as `skippedMissingTarget` in the extract summaries; the `DIR_PATTERN` whitelist survives only as the typed fast-path for pass-2b wikilinks (non-whitelisted `[[dir/...]]` get an equivalent direct typed candidate in pass 2c, plus the flag-gated suffix rescue for non-exact matches). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). No in-memory dedup pre-load — candidates buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, `created` counter returns real rows inserted. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id ` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id ] [--catch-up] [--dry-run] [--json]` branch (`extractStaleFromDB`) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three new `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush: link/timeline flush throws propagate and abort the batch; stamp LAST so a crash leaves pages unstamped and they re-extract idempotently (`addLinksBatch` ON CONFLICT DO NOTHING + timeline dedup). Race fix: `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Schema parity in schema.sql + pglite-schema.ts + schema-embedded.ts + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` gains `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` for `pagesAffected` at the inline-extract call site, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` adds `'extract.stale'` to `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` adds `links_extraction_lag` to `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp regression + crash-contract), `test/sync-inline-extract-stamps.serial.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite regression in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. - Extract CLI help — `EXTRACT_HELP` in `src/commands/extract.ts` is the canonical detailed usage shared by `--help` and invalid-subcommand errors. `src/cli.ts` routes `extract --help` before engine connection so help works on unconfigured installs; the top-level TOOLS block advertises every mode-specific flag. Pinned by `test/cli-help-discoverability.test.ts`. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` — unified extract operator surface. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts`. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. `extract_health` doctor check reads last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok`. CLI: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted halt_rate desc + cost desc, top-5 + "more rows" hint, stable `schema_version: 1`); `gbrain extract --explain ` (resolution chain pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)`, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; ships as a stub-reporter). `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` — parses but refuses at runtime); `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Pinned by `test/extractable-spec-widening.test.ts` (22), `test/extract/receipt-writer.test.ts` (12, canonical PGLite block R3+R4), `test/extract/benchmark.test.ts` (17), `test/extract/status.test.ts` (15), `test/schema-pack/scaffold-extractable.test.ts` (15, privacy guards), `test/doctor-extract-health.test.ts` (8). -- `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the path-set checkpoint. `--source-id ` routes pages to the named source (resolved via `resolveSourceWithTier()` at the boundary; consistent across `import`, `extract`, `graph-query`, `sources current`). Pinned by `test/import-source-id.test.ts`. +- `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the path-set checkpoint. `--source-id ` routes pages to the named source (resolved via `resolveSourceWithTier()` at the boundary; consistent across `import`, `extract`, `graph-query`, `sources current`). Pinned by `test/import-source-id.test.ts`. `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. `runImport` opts also carry `exclude` (glob filter over dir-relative paths, threaded by `performFullSync` for `sync --exclude`; warns when every file is excluded — NAV-4) and `slugRoot` (slug/`source_path` base for monorepo subdir syncs, #753/#774; the resume checkpoint stays dir-relative per `resumeFilter`'s contract).- `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). Foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. - `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. `current [--json]` calls `resolveSourceWithTier()` and prints `source_id`, `tier` (`flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail` (decision table in `skills/conventions/brain-routing.md`). `status [--json]` — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures); thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` from `src/commands/sync.ts`; `--json` emits stable `{schema_version: 1, sources, ...}` on stdout; filters input to `local_path IS NOT NULL AND archived IS NOT TRUE`. `audit [--json]` — read-only dry-run disk scan for size distribution + would-blocks + junk-pattern hits WITHOUT touching the DB; walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). The live `runStatus` health table gains a `BACKFILL` column between `EMBED` and `FAILS` (`active(N)` beats `queued(N)` beats `idle`, from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`) so operators see deferred `embed-backfill` minion work after `sync --all` exits 0; `jobCountsBySource` in `source-health.ts` widens its `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (best-effort, all-0 on pre-minions brains). Pinned by `test/content-sanity.test.ts`, `test/import-file-content-sanity.test.ts`, `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. `reindexFrontmatterCli(engine, args)` takes the ALREADY-CONNECTED engine from cli.ts's dispatch (#1963); it must never build/connect its own engine — a second connect on the same PGLite data dir self-deadlocks on the data-dir lock (this process already holds it) and timed out 100% of the time on PGLite. Same rule applies to `runBackfillCommand(engine, args)` in `src/commands/backfill.ts` and any future command dispatched from cli.ts's engine-connected switch. Pinned by `test/reindex-frontmatter-connect.test.ts` (library path) and `test/reindex-frontmatter-pglite-spawn.serial.test.ts` (CLI dispatch seam, both commands). - `src/core/source-config-sql.ts` + `src/core/sources-load.ts` — canonical recovery for historical non-object `sources.config` values. The application reader unwraps nested JSON strings and merges recoverable array fragments left-to-right; the shared SQL expression mirrors that policy atomically for both engines, source config updates, archive/restore, and the paste-ready `source_config_shape` doctor repair. `localFederatedSourceIds` reads config through the same parser so stdio/CLI federation cannot silently disagree with `sources list`. `sourceConfigHasRemoteUrl` uses that parser for autopilot pull policy, including PGLite's JSON-string config shape. Invalid fragments degrade to `{}` rather than throwing. Pinned by `test/sources-load.test.ts`, `test/job-pull-policy.test.ts`, `test/list-all-sources.test.ts`, `test/local-federated-search-scope.test.ts`, `test/destructive-guard.test.ts`, and `test/doctor-source-config-shape.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (14 cases), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`). -- `src/core/sync.ts` extension — `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. The `commands/sync.ts` cleanup loop guards on `unsyncableReason(path)` being `'metafile'` OR `'pruned-dir'` (#2404) so previously-indexed metafile pages AND deliberately put-created pages under pruned dirs survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). -- `src/core/import-file.ts` extension — identity-based dedup pre-check at `:427-490`. Calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId})` (optional `?` so test doubles compile). Posture: SKIP when `frontmatter.id` matches (true external duplicate from overlapping ingest roots), WARN-ALWAYS on content_hash collision with different/missing `frontmatter.id` (templates and daily logs may legitimately share text), FAIL CLOSED on lookup error, bypass via `--force-rechunk`. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Pinned by `test/import-dedup-frontmatter-id.test.ts` (11 cases). -- `src/core/engine.ts` extension — two interface members: (1) optional `findDuplicatePage?(sourceId, {hash, frontmatterId?}): Promise<{slug, id} | null>` (identity precedence is content_hash OR frontmatter->>'id', both with `deleted_at IS NULL`); (2) `resolveSlugs(partial, opts?)` extended with `{sourceId?, sourceIds?}` so the MCP fuzzy `get_page` path scopes by source (field names match `sourceScopeOpts(ctx)` output so handlers spread directly; back-compatible — no opts gives prior behavior). Plus a stable tiebreaker `ORDER BY score DESC, page_id ASC, chunk_id ASC` in `searchVector` in both engines: on a score tie (basis-vector eval fixtures) older `page_id` wins, closing the planner-non-determinism class where a new index on `pages` could flip ranking on tied scores. -- `src/core/migrate.ts` v95 — `pages_dedup_partial_index` adds `CREATE INDEX pages_dedup_idx ON pages (source_id, content_hash) WHERE deleted_at IS NULL`. Postgres uses `CREATE INDEX CONCURRENTLY` with `transaction: false` + pre-drops any invalid remnant; PGLite uses plain `CREATE INDEX`. Powers `findDuplicatePage` hot path (O(log n) instead of O(n)). -- `src/cli.ts` extension — `dream` dispatch at lines 1063-1080 binds the caught engine-connect error and emits `[dream] WARNING: could not connect to DB (...)` to stderr before falling through to filesystem-only phases; the `runDream(null, ...)` no-DB fallback is preserved. Pinned by `test/cli-dream-engine-warn.test.ts` (2 subprocess cases against good + bad DATABASE_URL). -- `src/commands/autopilot.ts` extension — federated-brain co-existence + launchd hygiene. (1) `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME` (two brains can run autopilot simultaneously without lock-stealing); lock file stores PID, startup checks `kill -0 ` before refusing to start (stale lock from a crashed process no longer blocks). (2) exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`; unrecoverable causes `process.exit(0)` so launchd backs off instead of looping `config.database_url undefined`. (3) exported pure `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`. -- `src/core/oauth-provider.ts` + `src/commands/serve-http.ts` extension — custom `/token` middleware that runs BEFORE the MCP SDK's `clientAuth`. The SDK does plaintext compare against the request's `client_secret`; gbrain stores SHA-256 hashes only, so every confidential-client `/token` request would fail. The middleware detects confidential auth via `Authorization: Basic` header OR `client_secret_post` form body (both shapes per RFC 6749 §2.3.1), verifies via `verifyClient(client_id, presented_secret)` (SHA-256 hash compare), and falls through to the SDK for public PKCE clients (which the SDK's clientAuth still accepts via NULL-`client_secret_hash` normalization). Pinned by `test/oauth-confidential-client.test.ts` (both `client_secret_basic` and `client_secret_post`). -- `src/core/sync.ts:pruneDir` extension — `pruneDir(name, parentDir?)` extended with optional `parentDir`. When provided, additionally rejects directories containing `.git` as a FILE — the git submodule gitfile pattern (regular repos have `.git` as a DIRECTORY; submodules as a file pointing into the parent's `.git/modules/`). Sync + extract walkers thread `parentDir` so the gitfile-as-FILE check fires per descend step. Best-effort: `statSync` failures fall through and treat as a normal dir. Closes the phantom-import bug class where syncing a worktree-with-submodules walked into submodule trees. Pinned by `test/sync-walker-submodule.test.ts`. -- `src/core/minions/handlers/subagent.ts` extension — terminal-state short-circuit on resume. When a stored message thread already ends in `stop_reason: 'end_turn'`, the handler returns `{ ok: true }` immediately instead of issuing another `messages.create` call (re-prompting past `end_turn` would get a 400 and dead-letter an already-successful job). Pinned by `test/subagent-handler.test.ts`. -- `src/commands/doctor.ts` extension — three checks wired into `runDoctor()` and the JSON envelope, all warn-only with paste-ready fix hints. (1) `checkSourceRoutingHealth(engine)` scans up to 200 pages on federated brains and flags pages whose `source_id` doesn't match what `resolveSourceWithTier()` would have picked for their `source_path`; single-source brains short-circuit to `ok`; the 200-page cap is total across the brain so doctor stays under 5s. (2) `checkOauthConfidentialHealth(engine)` probes registered confidential clients for `/token` reachability. (3) `checkAutopilotLockScope()` (pure, no engine) compares the resolved lock path to `$GBRAIN_HOME`; warns when set but the lock lives elsewhere, with a PID-safe inspection hint (`kill -0 ` before deletion). Pinned by `test/doctor-v0_37_7_checks.test.ts`. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape. `retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations. v95: `pages_dedup_partial_index` adds `CREATE INDEX pages_dedup_idx ON pages (source_id, content_hash) WHERE deleted_at IS NULL`. Postgres uses `CREATE INDEX CONCURRENTLY` with `transaction: false` + pre-drops any invalid remnant; PGLite uses plain `CREATE INDEX`. Powers `findDuplicatePage` hot path (O(log n) instead of O(n)). v74 `mcp_spend_log` uses BTREE on `(client_id, created_at)` + `(token_name, created_at)` — `date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; a `created_at` range scan covers the per-day rollup. v75 `embedding_multimodal_column` is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). +- `src/cli.ts` — `dream` dispatch at lines 1063-1080 binds the caught engine-connect error and emits `[dream] WARNING: could not connect to DB (...)` to stderr before falling through to filesystem-only phases; the `runDream(null, ...)` no-DB fallback is preserved. Pinned by `test/cli-dream-engine-warn.test.ts` (2 subprocess cases against good + bad DATABASE_URL). - `skills/conventions/brain-routing.md` — agent-facing convention skill documenting the canonical 6-tier source resolution chain (flag → env → dotfile → local_path → brain_default → seed_default) with paste-ready decision tables. Linked from CLAUDE.md's "Two organizational axes" section and from `gbrain sources current`'s hint output. -- `src/commands/doctor.ts` extension — `buildChecks(engine, args, dbSource): Promise` exported as a test seam. `runDoctor` is a thin wrapper: `buildChecks → computeDoctorReport → render + process.exit`. All 10 `process.exit` sites stay in the wrapper; the two early-return paths (no engine, connection failure) return partial check lists instead of inline exits (observable output identical). Pinned by `test/doctor-behavioral.test.ts` (13 cases: pure aggregation math over `computeDoctorReport`, orchestrator cases for `--fast` skip set + `--json` flag + no-engine partial path + snapshot of load-bearing check names) and `test/doctor-cli-smoke.serial.test.ts` (1 subprocess case spawning `bun run src/cli.ts doctor --json` against a fresh PGLite tempdir, asserting schema_version=2 envelope, status enum, non-empty checks array — the render-path coverage buildChecks-only tests miss; quarantined `.serial` because PGLite write-locks don't play with parallel runners). -- `src/core/cycle.ts` extension — `runPhaseLint` + `runPhaseBacklinks` carry the `export` keyword so behavioral tests can drive them directly (internal helpers exposed for test-only consumption; downstream code should NOT depend on them). Pinned by `test/cycle-legacy-phases.test.ts` (11 cases across both phases: clean run → status='ok', partial fix → status='warn' with `dryRun` in details, dry-run path doesn't write, throw-from-lib → status='fail' with the wrapper's try/catch envelope populated). Future phase wrappers (sync, extract, embed, orphans, extract_facts, resolve_symbol_edges, recompute_emotional_weight) land as additional describes in the same file. - `test/operations-trust-boundary.test.ts` + `scripts/check-operations-filter-bypass.sh` — operations trust-boundary contract coverage. Pure assertions over all 74 ops (every op has a scope annotation; every mutating op has a non-read scope; `localOnly: true` ops are excluded from `operations.filter(op => !op.localOnly)`; the seven historically-sensitive localOnly ops snapshot-pinned by name) plus targeted handler-invocation regressions for the two historically-broken HTTP-callable classes: `submit_job` with `name='shell'` + `ctx.remote=true` MUST reject (the HTTP MCP shell-job RCE class), and `search_by_image` with `image_path` + `ctx.remote=true` MUST reject (the P0 image-leak class). `file_upload` and `sync_brain` omitted from handler-invocation tests because they're `localOnly: true` (that path would test an impossible production scenario). The shell guard greps `src/` for any module importing the `operations` value outside the canonical filter site at `src/commands/serve-http.ts` (three import shapes: destructured, aliased, namespace), with an explicit 10-entry allow-list + a literal-string check that `serve-http.ts` still contains `operations.filter(op => !op.localOnly)`. Wired into `bun run verify`. -- `src/core/content-sanity.ts` — pure assessor for the content-sanity defense. `assessContent(content, opts): SanityVerdict` returns one of `ok | warn_oversize | hard_block_junk_pattern | soft_block_oversize` with `{reason, bytes, matched_pattern_name?}`. Six built-in junk patterns (Cloudflare challenge dumps, CAPTCHAs, 403 dumps, bare error-page titles) compiled at module load; operator literal substrings via `loadOperatorLiterals()` from `src/core/content-sanity-literals.ts`. `ContentSanityBlockError` tagged class is the typed throw shape every wrapper site (`gbrain import`, `put_page` MCP op, `gbrain sync`, `/ingest` webhook) catches via the existing exception flow. The bytes-parity contract pins `Buffer.byteLength(content, 'utf8')` against the embedder's actual byte count so a 499K-byte page can't be soft-blocked on assessment then overflow on embed. Knob resolution chain env > file (`~/.gbrain/config.json`) > DB > defaults. Four knobs: `content_sanity.bytes_warn` (50_000), `content_sanity.bytes_block` (500_000), `content_sanity.junk_patterns_enabled` (true), `content_sanity.disabled` (false; `GBRAIN_NO_SANITY=1` is the loud-stderr kill-switch). New `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`; adds the fuzzy prose-vs-markup ratio pass (markup chars / total above `max_markup_ratio`; code pages exempt; gated by `prose_check_enabled`) on top of the byte + junk-pattern passes. Three more knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (true). Pinned by `test/content-sanity.test.ts`. +- `src/core/content-sanity.ts` — pure assessor for the content-sanity defense. `assessContent(content, opts): SanityVerdict` returns one of `ok | warn_oversize | hard_block_junk_pattern | soft_block_oversize` with `{reason, bytes, matched_pattern_name?}`. Six built-in junk patterns (Cloudflare challenge dumps, CAPTCHAs, 403 dumps, bare error-page titles) compiled at module load; operator literal substrings via `loadOperatorLiterals()` from `src/core/content-sanity-literals.ts`. `ContentSanityBlockError` tagged class is the typed throw shape every wrapper site (`gbrain import`, `put_page` MCP op, `gbrain sync`, `/ingest` webhook) catches via the existing exception flow. The bytes-parity contract pins `Buffer.byteLength(content, 'utf8')` against the embedder's actual byte count so a 499K-byte page can't be soft-blocked on assessment then overflow on embed. Knob resolution chain env > file (`~/.gbrain/config.json`) > DB > defaults. Four knobs: `content_sanity.bytes_warn` (50_000), `content_sanity.bytes_block` (500_000), `content_sanity.junk_patterns_enabled` (true), `content_sanity.disabled` (false; `GBRAIN_NO_SANITY=1` is the loud-stderr kill-switch). New `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`; adds the fuzzy prose-vs-markup ratio pass (markup chars / total above `max_markup_ratio`; code pages exempt; gated by `prose_check_enabled`) on top of the byte + junk-pattern passes. Three more knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (true). Pinned by `test/content-sanity.test.ts`. new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the byte + junk-pattern passes. Three config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override), `content_sanity.max_markup_ratio` (0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (true). Same env > file > DB > defaults resolution chain. Pinned by `test/content-sanity.test.ts`. - `src/core/content-sanity-literals.ts` — operator literal-substring loader. Reads `~/.gbrain/junk-substrings.txt`, one literal per non-comment non-blank line; optional `# name=` header pairs an identifier with the following literal so audit JSONL groups by site (`linkedin_auth_wall`, `reddit_blocked`, etc.). Fail-soft on ENOENT (missing file = empty array). Loaded on every ingest. Deliberately literal substrings (NOT regex) to defeat ReDoS. Pinned by `test/content-sanity-literals.test.ts`. - `src/core/embed-skip.ts` — 5-site shared predicate for the soft-block embed-skip filter. Exports `shouldSkipEmbedding(frontmatter): boolean` (JS predicate for callers holding the page in memory), `EMBED_SKIP_SQL_FRAGMENT` (parameterized SQL clause shared by Postgres + PGLite via `executeRaw`), and `buildEmbedSkipMarker(reason: string)` (writes `frontmatter.embed_skip = {at: ISO_TIMESTAMP, reason}` so the JSONB shape stays uniform). The 5 sites: `embed.ts --stale`, `embed.ts --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks`. Single source of truth so the filter cannot drift. Pinned by `test/embed-skip.test.ts` (cross-site invariant + JSONB shape). - `src/core/audit/content-sanity-audit.ts` — ISO-week JSONL audit at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl` built on the `audit-writer.ts` primitive. Records every hard-block, soft-block, warn-trip, and quarantine/flag event with `{kind, source_id, slug, bytes, matched_pattern_name?, reason, ts}`. Doctor reads the last 7 days, aggregates by `(matched_pattern_name, source_id)` so operators see which scraper is the problem. Honors `GBRAIN_AUDIT_DIR` for shared-filesystem multi-host setups. Pinned by `test/audit/content-sanity-audit.test.ts`. -- `src/commands/doctor.ts` extension — three checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on live DB pages matching any junk pattern that escaped ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; `--content-audit` opts into a full scan. All three warn-only with paste-ready fix hints (junk → `gbrain sources audit ` + `git rm` source-of-truth, oversize → split or accept). -- `src/commands/lint.ts` extension — lint rules `huge-page` (flags pages exceeding `content_sanity.bytes_warn`) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor. `lint.ts` lifts DB config when `~/.gbrain/` is reachable; falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts`. -- `src/commands/embed.ts` extension — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title/slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter allowed. Pinned by `test/embed-skip.test.ts`. -- `src/core/import-file.ts` extension — `importFromContent` is the narrow waist every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It runs a three-tier content-quality disposition via `assessContentSanity` from `src/core/content-sanity.ts` BEFORE chunking: (1) high-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE (stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search) OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`; (2) fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker (page stays fully searchable, marker rides search results + get_page to warn the agent); (3) oversize → `embed_skip` soft-block via `buildEmbedSkipMarker()` PLUS a `content_flag:oversized` marker, AND deletes any pre-existing chunks in the same transaction so search can't surface stale chunks. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. `gbrain import` honors `errors > 0` for non-zero exit. `classifyErrorCode` in `src/core/sync.ts` recognizes the `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these. `extractEntityRefs` (canonical; matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks), `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled` config helper. Link candidates match any dir-shaped path (#2576; existence-checked at persist). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. Pinned by `test/import-file-content-sanity.test.ts`. - `src/core/quarantine.ts` — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, page stays searchable, marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`. -- `src/core/content-sanity.ts` extension — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the byte + junk-pattern passes. Three config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override), `content_sanity.max_markup_ratio` (0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (true). Same env > file > DB > defaults resolution chain. Pinned by `test/content-sanity.test.ts`. - `src/commands/quarantine.ts` — `gbrain quarantine ` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages so junk predating the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`. -- `src/core/search/hybrid.ts` + `src/core/search/sql-ranking.ts` + `src/core/operations.ts` + `src/core/types.ts` extensions — agent-warning channel. `SearchResult.content_flag?: {reason, detail}` (new optional field in `types.ts`) is stamped post-fusion by `stampContentFlags` (the `stampEvidence` precedent) in `hybridSearch` AND in the keyword-only `search` MCP op so both retrieval paths surface the marker. `get_page` returns a top-level `content_flag` parallel field via `getContentFlag(page.frontmatter)`. `buildVisibilityClause` (sql-ranking.ts) ANDs in `QUARANTINE_FILTER_FRAGMENT` so quarantined pages are excluded from all six search call sites (alongside soft-delete + archived-source filters). Pinned by `test/sql-ranking.test.ts` + `test/e2e/quarantine-search-exclusion.test.ts`. -- `src/commands/doctor.ts` extension — two checks wired into `runDoctor()` + the JSON envelope: `quarantined_pages` (counts pages carrying the `quarantine` marker via `engine.executeRaw` JSONB `?` existence, works on PGLite + Postgres; warn-only with a `gbrain quarantine list` hint) and `flagged_pages` (counts `content_flag` pages — searchable but odd; warn-only). Both skip gracefully (status ok, "Skipped") on engines/brains where the probe errors. Pinned by `test/doctor.test.ts`. -- `src/commands/lint.ts` + `src/commands/sources.ts` extensions — `gbrain lint` gains a `markup-heavy` rule (flags pages whose prose-vs-markup ratio exceeds `content_sanity.max_markup_ratio`, reusing `assessContentSanity` so lint/gate/scan share one assessor); pinned by `test/lint-content-sanity.test.ts`. `gbrain sources audit ` becomes disposition-aware: its dry-run disk scan reports would-quarantine / would-reject / would-flag counts driven by the effective `content_sanity.junk_disposition` + markup config, so an operator previews the gate's verdict before sync. The `content-sanity-audit` JSONL (`src/core/audit/content-sanity-audit.ts`) records the new quarantine/flag dispositions. - `src/core/zombie-reap.ts` — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()`. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes. - `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell). -- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). The `maxWaiting` coalesce path uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other. `claim` and `renewLock` issue their UPDATE via `engine.executeRawDirect` (not `executeRaw`) so the lock heartbeat runs on the direct session-mode pool that the transaction pooler won't recycle mid-hold; on PGLite this is identical to `executeRaw`. The two terminal dead-letter paths (`handleWallClockTimeouts` wall-clock kill and the stall dead-letter CTE in the stall sweep) BOTH increment `attempts_made` so a long job killed there reads as an honest attempt instead of `attempts 0 / started N`; the stall path also bumps `stalled_counter`, surfaced by `gbrain jobs get` as `Attempts: M/N (started: X, stalled: S/MaxS)`. At submit, `add()` stamps a default `timeout_ms` via `defaultTimeoutMsFor(jobName)` (from `handler-timeouts.ts`) when the caller passed none, so long handlers aren't wall-clock-killed mid-progress by the short null-default; an explicit `opts.timeout_ms` always wins. Guarded by `test/queue-lock-retry.test.ts` (claim never falls back to `executeRaw`), `test/postgres-execute-raw-direct.test.ts` (routing decision matrix), and `test/minions.test.ts` (attempt accounting + default-timeout stamping). +- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). The `maxWaiting` coalesce path uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other. `claim` and `renewLock` issue their UPDATE via `engine.executeRawDirect` (not `executeRaw`) so the lock heartbeat runs on the direct session-mode pool that the transaction pooler won't recycle mid-hold; on PGLite this is identical to `executeRaw`. The two terminal dead-letter paths (`handleWallClockTimeouts` wall-clock kill and the stall dead-letter CTE in the stall sweep) BOTH increment `attempts_made` so a long job killed there reads as an honest attempt instead of `attempts 0 / started N`; the stall path also bumps `stalled_counter`, surfaced by `gbrain jobs get` as `Attempts: M/N (started: X, stalled: S/MaxS)`. At submit, `add()` stamps a default `timeout_ms` via `defaultTimeoutMsFor(jobName)` (from `handler-timeouts.ts`) when the caller passed none, so long handlers aren't wall-clock-killed mid-progress by the short null-default; an explicit `opts.timeout_ms` always wins. Guarded by `test/queue-lock-retry.test.ts` (claim never falls back to `executeRaw`), `test/postgres-execute-raw-direct.test.ts` (routing decision matrix), and `test/minions.test.ts` (attempt accounting + default-timeout stamping). `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`. - `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the moved-out generic abort listener now fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape was the unhandledRejection crash class during PgBouncer rotation). Gaps closed: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` + per-call `Promise.race` timeout (`GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`, default `lockDuration/3`); (3) time-based abort (`Date.now() - lastSuccessfulRenewalAt >= lockDuration - GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`) so we release the lock BEFORE another worker can reclaim; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims). CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. The self-health DB-liveness probe now runs EVEN under a supervisor (`GBRAIN_SUPERVISED=1`): the outer guard is `if (this.opts.healthCheckInterval > 0)` and only the STALL-detection block is wrapped in `if (!isSupervisedChild)` — so a supervised worker whose own pool dies self-exits `unhealthy(db_dead)` after `dbFailExitAfter` probes (the supervisor watches a different connection and can't see this worker's dead pool), while the supervisor's progress watchdog owns forward-progress (#1801). Pinned by `test/worker-lock-renewal.test.ts` (18), `test/audit/lock-renewal-audit.test.ts` (11), `test/scripts/check-worker-lock-renewal-shape.test.ts` (5), `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11), `test/worker-supervised-db-probe.test.ts` (3). - `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets. Worker exit classifier emits `likely_cause` on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. Consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping the in-process SIGCHLD reaper can't reach); exposes `isTiniDetected` read-only accessor. The spawn-and-respawn loop is the shared `ChildWorkerSupervisor` core: MinionSupervisor composes it via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` back through `emit()` SupervisorEvent (JSONL audit consumers see byte-compatible output). PID lock, signal handlers, health check, and `process.exit` on the HARD crash ceiling stay in MinionSupervisor. Crossing the SOFT budget (`maxCrashes`) no longer permanently gives up (#1994/#2227): the core drops into degraded retry (capped backoff + a `crash_budget_degraded` health_warn) and self-heals when a respawn runs stably; permanent `process.exit(MAX_CRASHES)` fires only at the hard ceiling `resolveHardStopMaxCrashes(maxCrashes)` (default `maxCrashes × 10`, env `GBRAIN_SUPERVISOR_HARD_STOP_CRASHES`, `0` = never). Separately, `gbrain jobs supervisor status` + `gbrain doctor` detect a live supervisor through this queue lock (`inspectLock` + `isLockHolderLive`, freshness-keyed so PID reuse can't false-positive) when the `$HOME`-derived pidfile is absent, so a split-`$HOME` deployment no longer reads a healthy supervisor as "not running" (#2227). `code=0` leaves `crashCount` untouched (so a worker alternating real crashes + watchdog drains still trips `max_crashes`); `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop via `health_warn { reason: 'clean_restart_budget_exceeded' }` + backoff. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)`. Progress watchdog (#1801): `healthCheck()` restarts an alive-but-wedged child via `childSupervisor.restartCurrentChild(35_000)` when a queue has claimable work, 0 live-lock active jobs, and stale completions across `wedgeRestartChecks` (default 3) consecutive checks past `wedgeRestartMinutes` (default 15, 0 disables) + a `startupGraceMs` window; bounded by `wedgeRestartLoopBudget` (default 3 / `wedgeRestartLoopWindowMs`) which switches to a one-shot `wedge_restart_loop` alert. The wedge query is the exported `queryWedgeSignals(engine, queue, handlerNames)` — name+queue-scoped, `active_healthy` = live-lock only (an expired-lock active row does NOT mask the wedge), due-delayed counted. Claimable names are derived at start via a throwaway `registerBuiltinHandlers` worker (its new `quiet` opt). Flags `--wedge-restart-minutes` / `--wedge-restart-checks` + env `GBRAIN_WEDGE_RESTART_MINUTES` / `GBRAIN_WEDGE_RESTART_CHECKS`. The worker argv is built by the exported pure `buildWorkerArgs(opts)` (appends `--nice N` when `opts.nice_requested` is set, alongside `--concurrency`/`--queue`/`--max-rss`); the niceness apply RESULT (`nice_requested`/`nice_effective`/`nice_error`, computed by the CLI in jobs.ts — the supervisor doesn't call setPriority) rides on the `started`/`worker_spawned` audit emissions (#1815). Queue-scoped singleton (#1849): the real authority is a DB lock (`tryAcquireDbLock` from `src/core/db-lock.ts`) keyed on `supervisorLockId(queue)` = `gbrain-supervisor:` — keyed on the QUEUE ALONE because the lock row lives inside the target database, so the (database) half of the mutex is physical, not part of the key (an earlier revision mixed in a config-derived DB identity, which let two supervisors on the same physical DB via different-but-equivalent URLs compute different ids and both acquire — fixed). Two supervisors with different `$HOME`/`--pid-file` against the same `(database, queue)` no longer both run with conflicting `--max-rss`; the second exits `LOCK_HELD`. The pidfile-cleanup `process.on('exit')` listener is installed BEFORE the DB-lock acquisition so the `LOCK_HELD` early-exit can't strand the pidfile this process just created. The default pidfile is now brain-scoped (`supervisor-.pid`) so different brains under one HOME don't false-block. The lock refreshes on its own `setInterval` (TTL 5min, refresh 60s, max 3 failures = 180s < TTL); a refresh that fails past the threshold exits `LOCK_LOST` (code 4) rather than risk a split-brain. `shutdown()` releases the lock so a clean restart re-acquires immediately. The `started` audit now records `max_rss_mb` so `gbrain doctor`'s `supervisor_singleton` check can surface the effective cap. Exports `supervisorLockId()` and the pure `classifySupervisorSingleton({lockLive, lockHolderHost, lockHolderPid, localHost, localPid}) → 'no_lock'|'single'|'mismatch'` (host+pid compare, bare pid meaningless cross-host) that doctor consumes. Pinned by `test/supervisor.test.ts` (16 cases), `test/supervisor-tini.test.ts`, `test/supervisor-wedge.test.ts`, `test/supervisor-build-worker-args.test.ts`, and `test/supervisor-db-lock.test.ts`. - `src/core/minions/child-worker-supervisor.ts` — shared spawn-and-respawn core reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon), so the two consumers can't drift into parallel-loop bugs. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void`. Exit classifier: `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences); `code != 0` follows `runDuration > stableRunResetMs ? 1 : ++crashCount`. Clean-restart budget: sliding window of code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s). The exit classifier special-cases `WORKER_EXIT_RSS_WATCHDOG` — `likely_cause='rss_watchdog'`, and that exit does NOT bump `crashCount` (routes to its own breaker); a dedicated `_watchdogExitTimestamps` sliding window trips a loud `rss_watchdog_loop` health_warn naming the cap when N watchdog exits land inside the window, INDEPENDENT of the stable-run reset (which would otherwise hide a >5-min-run watchdog drain loop). New opts `watchdogLoopBudget` (3), `watchdogLoopWindowMs` (600000), `watchdogBackoffMs` (30000); `ChildSupervisorEvent` extended. Public read-only accessors `childAlive`, `inBackoff`, `crashCount`; `killChild(signal)` gates on liveness (`exitCode === null && signalCode === null`), NOT `.killed` — `.killed` flips true once a signal is *sent*, so the old `!this._child.killed` guard made a follow-up SIGKILL after an ignored SIGTERM a silent no-op (#1801; the bug also lived in the `shutdown()` drain). `restartCurrentChild(graceMs)` (#1801 wedge self-heal) captures the CURRENT child ref, SIGTERM→grace→SIGKILLs THAT ref (never the respawn — closes the timer-kills-fresh-worker race), and flags `_intentionalRestart` so the exit is `likelyCause='wedge_restart'`, leaves `crashCount` UNTOUCHED (never trips `max_crashes`; like `rss_watchdog`), and respawns immediately (`backoff ms:0 reason='wedge_restart'`). `awaitChildExit(timeoutMs)` short-circuits when `child.exitCode !== null || child.signalCode !== null` so fast-SIGTERM responders don't cause a 35s shutdown hang. Degraded-retry (#1994/#2227): the `run()` loop no longer fires `onMaxCrashesExceeded` at the soft `maxCrashes`; it announces `health_warn { reason: 'crash_budget_degraded' }` once per episode and keeps respawning with capped backoff (the 60s cap makes it a paced retry, not a hot loop), re-arming after a stable-run reset drops the count. Permanent give-up fires only at `hardStopMaxCrashes` (default `maxCrashes × HARD_STOP_CRASH_MULTIPLIER` = 10×; `0` disables). Test hooks `_backoffFloorMs`, `_now`. `supervisor-audit.ts` adds `rss_watchdog` as a non-clean cause + its own `CrashSummary.by_cause` bucket, and `wedge_restart` to `CLEAN_EXIT_CAUSES` (a self-heal, not a crash; denylist preserved so future causes route to `legacy`). Pinned by `test/child-worker-supervisor.test.ts` (12 cases). @@ -256,17 +233,16 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/minions/supervisor-pid.ts` — `readSupervisorPid(pidFile) → {pid, running}`: the shared `existsSync → readFileSync → parseInt → process.kill(pid,0)` PID-file + liveness reader extracted from the three copies in `jobs.ts` (supervisor status), `jobs.ts` (stats), and `doctor.ts`. EPERM from the liveness probe counts as running. Pinned by `test/supervisor-pid.test.ts`. - `src/core/minions/handler-timeouts.ts` (#1737) — per-handler default wall-clock budgets. `HANDLER_DEFAULT_TIMEOUT_MS` maps the long handlers (`subagent`, `subagent_aggregator`, `embed-backfill`, `autopilot-cycle`, `autopilot-global-maintenance`) to a 30-min default (the value `cycle/patterns.ts` already passed for subagents, generalized). `defaultTimeoutMsFor(jobName)` returns that default or `null` for short handlers (keep the tight null-default wall-clock). `MinionQueue.add()` stamps this onto `minion_jobs.timeout_ms` at submit time when the caller passed no `timeout_ms`, so a long job submitted without one isn't wall-clock-killed mid-progress; an explicit value always wins and already-queued jobs are NOT backfilled. Pinned by `test/minions.test.ts`. - `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`. -- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules. +- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules. `PROTECTED_JOB_NAMES` includes `synthesize`, `patterns`, `consolidate`. These phases internally submit `subagent` children with `allowProtectedSubmit=true` and can spend Anthropic credits. Only trusted local callers (CLI, autopilot, `doctor --remediate`) can submit them; MCP requests are rejected by `submit_job`'s protected-name guard. - `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides + `inherit:`-resolved keys. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). `ShellJobParams.inherit?: string[]` is a free-form list of snake_case config-key names; the worker resolves each via `loadConfig()` and injects the value under the derived env key (`database_url` → `GBRAIN_DATABASE_URL`; else uppercased). Names persist in `minion_jobs.data` (and the shell-audit JSONL); values never do. The canonical validator `validateShellJobParams` (sibling `shell-validate.ts`) runs PRE-ENQUEUE in both submit surfaces — `gbrain jobs submit shell` (jobs.ts:271) AND the `submit_job` op for `name='shell'` (operations.ts:2085); the handler-entry re-validation here is defense-in-depth (closes the bug class where validation ran AFTER `queue.add()` persisted the row). The validator does NOT police which config keys the agent inherits — same-uid trust model treats the agent as a peer of the worker. - `src/core/minions/handlers/shell-inherit.ts` — three helpers. `INHERIT_NAME_RE` (`/^[a-z][a-z0-9_]*$/`) is the snake_case shape guard used by the validator; rejects `__proto__`, leading-underscore, uppercase, and path-traversal shapes so audit logs stay readable and prototype-pollution lookups can't smuggle through. `deriveEnvKey(name)` maps config-key → child-env-key (`name.toUpperCase()` with one override: `database_url` → `GBRAIN_DATABASE_URL` because plain `DATABASE_URL` is ambiguous). `resolveInheritValue(cfg, name)` is the value lookup; uses `Object.hasOwn` to defeat prototype-pollution lookups, returns undefined for missing / non-string / empty-string values. No closed enum — agent and worker share a uid, so refusing arbitrary config keys defends nothing in that trust model. - `src/core/minions/handlers/shell-validate.ts` — `validateShellJobParams(data, opts?)` shared pre-enqueue validator. Throws `UnrecoverableError` with paste-ready operator hints on every failure. Three rules: (1) cmd/argv/cwd/env shape, (2) inherit array shape + snake_case regex per element (prototype-pollution defense), (3) fail-fast on missing config value with `gbrain config set ` hint. Optional `redact_secrets?: boolean` for output-side scrubbing. Deliberately does NOT police WHICH secrets the agent passes — single-uid trust model. Test seam: `opts.config` drives the validator hermetically without mocking. Re-called at `shell.ts` handler entry for defense-in-depth (catches rows submitted before the pre-enqueue validator existed). - `src/core/minions/handlers/shell-redact.ts` — opt-in output-side scrubbing for shell-job stdout/stderr. Pure `redactSecretsInText(text, secrets)`: string-mode `replaceAll` so regex metacharacters in values stay literal. When the caller passes `redact_secrets: true` (or `--redact-secrets`), the handler builds a Map of inherit-name → resolved-value and post-processes both tails before throw/return so persisted `result.stdout_tail` / `result.stderr_tail` / `error_text` carry ``. Only `inherit:`-resolved values are scrubbed; caller-supplied `env:` values pass through. Heuristic — defeats `echo "$GBRAIN_DATABASE_URL"`, not adversarial encode-then-print. Default `false`. - `src/core/config.ts:ensureGitignore` — idempotent retroactive writer of `~/.gbrain/.gitignore` (single line `*`). Called from `saveConfig()` so every config-writing path lays it down, AND from `runPostUpgrade()` so existing users pick it up on `gbrain upgrade`. Never clobbers a user-customized `.gitignore` (checks file exists + content non-empty before writing). Scope: blocks casual `git add ~/.gbrain` from inside an enclosing worktree, but does NOT cover already-tracked files, screenshots, backups (Time Machine / iCloud / Dropbox), or `git add -f`. The doctor check `home_dir_in_worktree` surfaces what `.gitignore` can't. -- `src/commands/doctor.ts:home_dir_in_worktree` — filesystem check walking up from `gbrainPath()` toward `$HOME` looking for a `.git` directory (main repo) or `.git` file (linked worktree pointer; Conductor + git-worktrees topology). Walk terminates at `$HOME` so a `.git` above the user's home doesn't false-positive. Honors `GBRAIN_HOME` (appends `.gbrain` to the override). Warn (not fail) with worktree-root path + paste-ready fix pointing at `GBRAIN_HOME` override or moving the brain. - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. - `src/core/minions/handlers/supervisor-audit.ts` — supervisor lifecycle JSONL audit at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` (ISO-week rotation; shares `computeIsoWeekName()` with `shell-audit.ts`). `writeSupervisorEvent(emission, supervisorPid)` appends one line per event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `health_error`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`). `readSupervisorEvents({sinceMs})` is the readback for `gbrain doctor`. Exports `isCrashExit(event)`, `summarizeCrashes(events)`, `CrashSummary` type, and `CLEAN_EXIT_CAUSES` denylist (`'clean_exit' | 'graceful_shutdown'`). Single regression point — both `gbrain doctor` (supervisor check at `doctor.ts:1011-1043`) and `gbrain jobs supervisor status` (`jobs.ts:803-826`) import from here so the two surfaces can't drift. `isCrashExit` classifies a single `worker_exited` against the denylist: clean/graceful are NON-crashes; everything else (incl. any future `likely_cause` from `child-worker-supervisor.ts`) is a crash; audit lines lacking `likely_cause` fall back to `code !== 0`. `summarizeCrashes` returns `{total, by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}, clean_exits}` — the `legacy` bucket catches both old fallback entries AND unrecognized future causes (fail-loud, not silent underreport); denylist-over-allowlist is deliberate. Pinned by `test/supervisor-audit.test.ts` (14 cases) and 4 source-grep wiring assertions in `test/doctor.test.ts`. - `src/core/minions/backpressure-audit.ts` — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. One line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the maxWaiting guard introduced. -- `src/core/minions/handlers/subagent.ts` — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Per-turn output cap resolves via `resolveMaxOutputTokens` (`data.max_tokens` → `agent.max_output_tokens` config → 8192 default); a `stop_reason: 'max_tokens'` final turn surfaces as `SubagentStopReason 'max_tokens'` (not a silent `end_turn`), and a max_tokens stop mid-tool-round injects a truncation note into the tool-result turn so the model re-issues the dropped call. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times. Catches both initial-prompt overflow and turn-N tool-loop accumulation that `synthesize.ts`'s chunker can't bound ahead of time. +- `src/core/minions/handlers/subagent.ts` — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Per-turn output cap resolves via `resolveMaxOutputTokens` (`data.max_tokens` → `agent.max_output_tokens` config → 8192 default); a `stop_reason: 'max_tokens'` final turn surfaces as `SubagentStopReason 'max_tokens'` (not a silent `end_turn`), and a max_tokens stop mid-tool-round injects a truncation note into the tool-result turn so the model re-issues the dropped call. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times. Catches both initial-prompt overflow and turn-N tool-loop accumulation that `synthesize.ts`'s chunker can't bound ahead of time. terminal-state short-circuit on resume. When a stored message thread already ends in `stop_reason: 'end_turn'`, the handler returns `{ ok: true }` immediately instead of issuing another `messages.create` call (re-prompting past `end_turn` would get a 400 and dead-letter an already-successful job). Pinned by `test/subagent-handler.test.ts`. - `src/core/minions/handlers/subagent-aggregator.ts` — `subagent_aggregator` handler. Claims AFTER all children resolve (queue guarantees every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds a deterministic mixed-outcome markdown summary. No LLM call. - `src/core/minions/handlers/subagent-audit.ts` — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback for `gbrain agent logs`. - `src/core/minions/rate-leases.ts` — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms). @@ -279,18 +255,19 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection). - `src/commands/agent.ts` — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. - `src/commands/agent-logs.ts` — `gbrain agent logs [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs. -- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. `case 'work'` wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging (the worker must not disconnect an engine it doesn't own; pool slots free immediately on shutdown rather than waiting for TCP keepalive). `jobs submit` surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as flags: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the SIGKILL-rescue regression guard. `registerBuiltinHandlers` always registers `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at startup with a loud per-plugin line; `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface). The `autopilot-cycle` handler forwards `job.data.phases` to `runCycle`, validated against `ALL_PHASES` from `src/core/cycle.ts` (invalid names filtered; empty/missing falls back to the default cycle); when `source_id` is set it binds `brainDir` to that source's `local_path` (null for a pure-DB source, never the global repo — the #2194/#2227 mixed-scope fix) and checks `isSourceInCooldown` before `runCycle`, returning a no-op `skipped` (not a failure) for a source still in its failure cooldown. The sibling `autopilot-global-maintenance` handler runs the brain-wide `GLOBAL_PHASES` once (no `sourceId`, `pull:false`) and stamps `autopilot.last_global_at` on success. `resolveJobPull` gives both cycle and standalone sync jobs one positive-polarity `pull` contract while preserving queued payloads that still carry the inverse legacy `noPull` key; explicit `pull` wins. The `sync` handler resolves `sourceId` at entry from `sources.local_path` (mirrors `cycle.ts:480`) so multi-source brains read the per-source `last_commit` anchor; concurrency routes through `autoConcurrency()` in `src/core/sync-concurrency.ts` (PGLite stays serial); `noEmbed` default is `true`. `gbrain jobs supervisor status` at `jobs.ts:803-826` consumes `summarizeCrashes()` from `src/core/minions/handlers/supervisor-audit.ts` for parity with `gbrain doctor`: JSON adds `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` + `clean_exits_24h`; human output gains per-cause + clean-exits lines. Pinned by `test/job-pull-policy.test.ts` and 4 source-grep wiring assertions in `test/doctor.test.ts` requiring `crashes_by_cause` + `clean_exits_24h=` in both `doctor.ts` and `jobs.ts`. `gbrain jobs watch` decouples its two output axes: `--json` picks FORMAT (human default, never gated on isTTY), `--follow` picks LOOP (default `isTTY && !json`). Non-TTY with no flags prints ONE human snapshot then exits (clean for subagent/pipe/cron); `--follow` opts into a continuous stream (human plain per tick, or JSONL with `--json`); a TTY with no flags keeps the live ANSI dashboard. Resolution is the pure `resolveWatchMode(opts, isTTY): {json, follow, useAnsiDashboard}` in `src/commands/jobs-watch.ts`; the dispatch wires `--follow`. Pinned by `test/jobs-watch-mode.test.ts` (format×loop matrix incl. the TTY+`--json`-one-shot case) + `test/e2e/non-tty-output.serial.test.ts` (the `cmd threshold` (default 25) AND a daily cap `floor(max_usd_per_day / ~$0.30)`. Enumerates `loadAllSources`. Submits the PROTECTED `extract-atoms-drain` job (`{allowProtectedSubmit:true}`) with a UTC-day time-sloted idempotency key `autopilot-extract-atoms-drain::` (a static key would block the source after the first job completed). `src/core/minions/protected-names.ts` adds `extract-atoms-drain`; `src/commands/jobs.ts` registers the handler (thin wrapper over `runExtractAtomsDrainForSource`, `LockUnavailableError` → `{deferred:true}`); `src/core/config.ts` adds the `autopilot.auto_drain.*` config keys + the `autopilot.` key prefix. Pinned by `test/extract-atoms-drain-handler.test.ts`, `test/autopilot-auto-drain-wiring.test.ts`. federated-brain co-existence + launchd hygiene. (1) `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME` (two brains can run autopilot simultaneously without lock-stealing); lock file stores PID, startup checks `kill -0 ` before refusing to start (stale lock from a crashed process no longer blocks). (2) exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`; unrecoverable causes `process.exit(0)` so launchd backs off instead of looping `config.database_url undefined`. (3) exported pure `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`. targeted-submit loop instead of blanket `autopilot-cycle` dispatch. Each tick: cheap `engine.getHealth()` (single SQL count) + `computeRecommendations()`, then route by shape — `score >= 95 AND no plan AND <60min since last full` → sleep; `score >= 95 AND >=60min` → submit `autopilot-cycle` (60-min floor exercises phase-coupling invariants on healthy brains); `plan <= 3 steps AND est <5min` → submit individual handlers; `plan large OR score < 70` → submit full `autopilot-cycle`. The `gbrain-cycle` lock ensures targeted submissions and the full cycle can't run concurrently. `maxWaiting: 1` per submit closes the queue-fan-out vector. - `src/mcp/server.ts` — MCP stdio server (generated from operations). Tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. Stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'` — gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`) pipe the handshake then close their stdin half, which would otherwise kill the server before the first tool call; signal handlers (SIGTERM/SIGINT/SIGHUP) + the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`. - `src/mcp/dispatch.ts` — shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults `remote: true` (untrusted); local CLI callers pass `remote: false`. Also exports `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed, returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved sorted; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes by probing. Raw payload visibility is opt-in via `gbrain serve --http --log-full-params` (loud stderr warning). New logging paths route through this helper, not `JSON.stringify(params)`. - `src/mcp/rate-limit.ts` — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth. -- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and `express-rate-limit` at 50 req / 15 min on `/token`. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, and the admin bootstrap token line — but the generated token's raw value only prints when stderr is an interactive TTY (`shouldSuppressBootstrapPrint`): a non-TTY (containerized/piped) start hides it so the secret never lands in centralized log storage, env-sourced tokens (`$GBRAIN_ADMIN_BOOTSTRAP_TOKEN`) are always hidden, `--print-admin-token` forces the raw value on a trusted terminal, and `--suppress-bootstrap-token` hides everything. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard at `operations.ts:1391` saw a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs — RCE). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(sql, engineName, version, timeoutMs)` racing `sql\`SELECT 1\`` against the exported `HEALTH_TIMEOUT_MS = 3000` (returns the same `ProbeHealthResult` tagged-union as `probeHealth`, single timer-cleanup site, single 503 envelope); body shape `{status, version, engine}` only. Full stats moved to admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. Every OAuth/admin/audit SQL call routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` + `AuthInfo.takesHoldersAllowList` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` (source scope from the `oauth_clients` row; takes-holders from `access_tokens.permissions.takes_holders` for legacy bearer tokens). The `/mcp` dispatch site reads `authInfo.takesHoldersAllowList ?? ['world']` — absent grants (OAuth-client tokens, pre-v29 brains) fail closed to world-only takes visibility, while an explicit `[]` grant is preserved as deny-all; pinned end-to-end by `test/e2e/serve-http-takes-holders.test.ts`. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). `POST /ingest` enforces the slug-prefix write fence at the ROUTE, not the op layer: the route hands its payload to the `ingest_capture` minion handler, which deliberately bypasses `put_page`, so no `OperationContext` exists and `enforceClientSlugFence` never runs — a slug-bound client must therefore supply `X-Gbrain-Slug` and it must satisfy `slugUnderBoundPrefixes`, else 403 (without the check a bound client could overwrite any page, in the `default` source, since untrusted payloads carry no source grant). +- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and `express-rate-limit` at 50 req / 15 min on `/token`. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, and the admin bootstrap token line — but the generated token's raw value only prints when stderr is an interactive TTY (`shouldSuppressBootstrapPrint`): a non-TTY (containerized/piped) start hides it so the secret never lands in centralized log storage, env-sourced tokens (`$GBRAIN_ADMIN_BOOTSTRAP_TOKEN`) are always hidden, `--print-admin-token` forces the raw value on a trusted terminal, and `--suppress-bootstrap-token` hides everything. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard at `operations.ts:1391` saw a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs — RCE). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(sql, engineName, version, timeoutMs)` racing `sql\`SELECT 1\`` against the exported `HEALTH_TIMEOUT_MS = 3000` (returns the same `ProbeHealthResult` tagged-union as `probeHealth`, single timer-cleanup site, single 503 envelope); body shape `{status, version, engine}` only. Full stats moved to admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. Every OAuth/admin/audit SQL call routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` + `AuthInfo.takesHoldersAllowList` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` (source scope from the `oauth_clients` row; takes-holders from `access_tokens.permissions.takes_holders` for legacy bearer tokens). The `/mcp` dispatch site reads `authInfo.takesHoldersAllowList ?? ['world']` — absent grants (OAuth-client tokens, pre-v29 brains) fail closed to world-only takes visibility, while an explicit `[]` grant is preserved as deny-all; pinned end-to-end by `test/e2e/serve-http-takes-holders.test.ts`. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). `POST /ingest` enforces the slug-prefix write fence at the ROUTE, not the op layer: the route hands its payload to the `ingest_capture` minion handler, which deliberately bypasses `put_page`, so no `OperationContext` exists and `enforceClientSlugFence` never runs — a slug-bound client must therefore supply `X-Gbrain-Slug` and it must satisfy `slugUnderBoundPrefixes`, else 403 (without the check a bound client could overwrite any page, in the `default` source, since untrusted payloads carry no source grant). confidential revoke: a pre-router `/revoke` handler validates the RFC 7009 body, verifies hash-only secrets for both `client_secret_post` and `client_secret_basic`, rejects mixed authentication, preserves the SDK path for public clients, and separates opaque client-auth failures from retryable/backend failures. OAuth metadata advertises both confidential methods. Pinned by `test/e2e/serve-http-oauth.test.ts`. three admin routes: `/admin/api/calibration/profile`, `/admin/api/calibration/charts/:type` (image/svg+xml; type in {brier-trend, domain-bars, pattern-statements, abandoned-threads}), `/admin/api/calibration/pattern/:id` (drill-down). - `src/core/sql-query.ts` — engine-aware tagged-template SQL adapter for OAuth/admin/auth infrastructure. `sqlQueryForEngine(engine)` returns a `SqlQuery` (`(strings, ...values) => Promise`) that walks the template, builds `$N` positional SQL, asserts every value is a `SqlValue` (string | number | bigint | boolean | Date | null), and routes through `engine.executeRaw(sql, params)` (Postgres via postgres.js `unsafe(sql, params)`, PGLite via `db.query(sql, params)`). Deliberately narrower than postgres.js's `sql` tag: no nested fragments, `sql.json()`, `sql.unsafe()`, `sql.begin()`, or array binding — the narrow scalar-only surface is the feature (keeps it from drifting into a partial postgres.js clone). JSONB writes go through `executeRawJsonb(engine, sql, scalarParams, jsonbParams)` which composes positional `$N::jsonb` casts and passes JS **objects** through; an object reaches the wire with the correct type oid, so executeRawJsonb is safe (verified by `test/sql-query.test.ts` on PGLite, `test/e2e/auth-permissions.test.ts:67` on Postgres). Positional binding is NOT universally immune, though: binding a `JSON.stringify(x)` **string** to a bare `$N::jsonb` via `unsafe()` double-encodes it into a jsonb string scalar on real Postgres (the #2339 class; PGLite hides it). Fixes: pass a raw object (executeRawJsonb / `sql.json`), or cast through `$N::text::jsonb`. `scripts/check-jsonb-pattern.sh` (template grep) doesn't fire on `executeRawJsonb(...)` because it passes objects; the positional `$N::jsonb` + `JSON.stringify` form is caught by `scripts/check-jsonb-params.mjs`. Consumed by `src/commands/auth.ts`, `src/commands/serve-http.ts`, `src/core/oauth-provider.ts`, `src/commands/files.ts`, `src/mcp/http-transport.ts` so all five work uniformly against PGLite and Postgres. - `src/commands/serve.ts` — `gbrain serve` stdio MCP entrypoint with idempotent shutdown across every parent-disconnect signal. Stdio EOF, SIGTERM, SIGINT, SIGHUP, and parent-process death (every reparent case — PID 1, launchd subreaper, systemd, tmux, or a parent shell with `PR_SET_CHILD_SUBREAPER`) all funnel into one `cleanup(reason)` that releases the engine and the PGLite write-lock dir within 5 seconds (otherwise the lock is held indefinitely after Claude Desktop / Cursor / launchd-managed gateways disconnect, forcing a 5-minute stale-lock wait on next start). Watchdog reparent check is `getParentPid() !== initialParentPid` (the `=== 1` check missed the subreaper case under launchd/systemd). Bun's `process.ppid` cache is stale across reparenting ([oven-sh/bun#30305](https://github.com/oven-sh/bun/issues/30305)) so `getParentPid()` runs `spawnSync('ps', ['-o', 'ppid=', '-p', PID])` per tick. Startup probe verifies `ps` is on PATH; if not (stripped containers, busybox), the watchdog skips installing AND emits a loud `[gbrain serve] watchdog disabled: ps unavailable ...` stderr line so operators see the degraded mode. Pinned by `test/serve-stdio-lifecycle.test.ts` (22 cases). Credit @Aragorn2046 + @seungsu-kr. -- `src/core/oauth-provider.ts` — `GBrainOAuthProvider` implementing the MCP SDK's `OAuthServerProvider` + `OAuthRegisteredClientsStore`. Backed by raw SQL (works on both PGLite and Postgres — OAuth is infrastructure, not a BrainEngine concern). Full OAuth 2.1: `authorize` + `exchangeAuthorizationCode` with PKCE, `client_credentials`, `refresh_token` with rotation, `revokeToken`, `registerClient` (DCR validates redirect_uri is `https://` or loopback per RFC 6749 §3.1.2.1). All tokens + client secrets SHA-256 hashed before storage. Auth codes single-use with 10-minute TTL via atomic `DELETE...RETURNING` (closes RFC 6749 §10.5 TOCTOU); refresh rotation also `DELETE...RETURNING` (§10.4 stolen-token detection). `pgArray()` escapes commas/quotes/braces so a comma-bearing redirect_uri can't smuggle a second array element. Legacy `access_tokens` fallback in `verifyAccessToken` grandfathers pre-v0.26 bearer tokens as `read+write+admin`, and threads BOTH stored grants off the token's `permissions` JSONB: `source_id` via `parseLegacyTokenScope` and `takes_holders` via `parseTakesHoldersAllowList` (both in `src/core/legacy-token-scope.ts`, shared with the legacy HTTP transport so the two transports cannot drift; `[]` takes-holders preserved as explicit deny-all, missing/non-array → undefined → the `/mcp` dispatch site's fail-closed `['world']`; OAuth-client tokens carry no takes-holders grant pending per-client storage — TODOS.md). `sweepExpiredTokens()` runs on startup in try/catch and returns the count via `RETURNING 1` + array length. RFC hardening: `client_id` folded atomically into the `DELETE WHERE` for both auth-code exchange and refresh rotation (wrong-client paths don't burn the row); refresh-scope-subset enforced against the original grant on the row (RFC 6749 §6, so revoking a scope shrinks existing refresh tokens); `client_id` bound on `revokeToken` (RFC 7009 §2.1); `/token` `redirect_uri` validated against the `/authorize` value (RFC 6749 §4.1.3, empty-string treated as missing not wildcard); bare `catch {}` in `verifyAccessToken`/`getClient` replaced by `isUndefinedColumnError` from `src/core/utils.ts` (only SQLSTATE 42703 falls through to legacy; lock timeouts/network blips throw); `dcrDisabled` constructor option lets `serve-http.ts` disable `/register` without monkey-patching the router. Module-private `coerceTimestamp()` normalizes postgres-driver-as-string BIGINT columns to JS numbers at 5 read sites (`getClient` for RFC 7591 §3.2.1 numeric timestamps, `exchangeRefreshToken` + `verifyAccessToken` for the SDK's `typeof === 'number'` check); throws on NaN/Infinity (fail loud at boundary), returns undefined for SQL NULL (callers treat NULL as expired). Not promoted to `utils.ts` — generic BIGINT precision-loss risk. `registerClient` honors `token_endpoint_auth_method: "none"` (RFC 7591 §3.2.1): public PKCE clients store `client_secret_hash = NULL` and the response omits `client_secret`; confidential clients (`client_secret_post` / `client_secret_basic`) keep their one-time-reveal shape; `getClient` normalizes NULL `client_secret_hash` to JS `undefined` so the SDK's clientAuth path accepts public clients. `verifyAccessToken` JOINs `oauth_clients.source_id` (write scope, scalar) + `oauth_clients.federated_read` (read scope, TEXT[]) + `oauth_clients.bound_slug_prefixes` (write fence, TEXT[] — consumed by `enforceClientSlugFence` in `operations.ts`) onto the returned `AuthInfo`; legacy brains degrade via `isUndefinedColumnError` fallback, dropping the newest projection first. `rescopeClient(clientId, {sourceId?, federatedRead?, boundSlugPrefixes?})` is the trusted-operator rescope (CLI `gbrain auth rescope-client`, admin `POST /admin/api/rescope-client`); `boundSlugPrefixes` is tri-state — undefined leaves the binding untouched, `null` clears it, a non-empty array replaces it (explicit empty array rejected as ambiguous deny-all) — so roster churn updates the write fence in place without rotating secrets. +- `src/core/oauth-provider.ts` — `GBrainOAuthProvider` implementing the MCP SDK's `OAuthServerProvider` + `OAuthRegisteredClientsStore`. Backed by raw SQL (works on both PGLite and Postgres — OAuth is infrastructure, not a BrainEngine concern). Full OAuth 2.1: `authorize` + `exchangeAuthorizationCode` with PKCE, `client_credentials`, `refresh_token` with rotation, `revokeToken`, `registerClient` (DCR validates redirect_uri is `https://` or loopback per RFC 6749 §3.1.2.1). All tokens + client secrets SHA-256 hashed before storage. Auth codes single-use with 10-minute TTL via atomic `DELETE...RETURNING` (closes RFC 6749 §10.5 TOCTOU); refresh rotation also `DELETE...RETURNING` (§10.4 stolen-token detection). `pgArray()` escapes commas/quotes/braces so a comma-bearing redirect_uri can't smuggle a second array element. Legacy `access_tokens` fallback in `verifyAccessToken` grandfathers pre-v0.26 bearer tokens as `read+write+admin`, and threads BOTH stored grants off the token's `permissions` JSONB: `source_id` via `parseLegacyTokenScope` and `takes_holders` via `parseTakesHoldersAllowList` (both in `src/core/legacy-token-scope.ts`, shared with the legacy HTTP transport so the two transports cannot drift; `[]` takes-holders preserved as explicit deny-all, missing/non-array → undefined → the `/mcp` dispatch site's fail-closed `['world']`; OAuth-client tokens carry no takes-holders grant pending per-client storage — TODOS.md). `sweepExpiredTokens()` runs on startup in try/catch and returns the count via `RETURNING 1` + array length. RFC hardening: `client_id` folded atomically into the `DELETE WHERE` for both auth-code exchange and refresh rotation (wrong-client paths don't burn the row); refresh-scope-subset enforced against the original grant on the row (RFC 6749 §6, so revoking a scope shrinks existing refresh tokens); `client_id` bound on `revokeToken` (RFC 7009 §2.1); `/token` `redirect_uri` validated against the `/authorize` value (RFC 6749 §4.1.3, empty-string treated as missing not wildcard); bare `catch {}` in `verifyAccessToken`/`getClient` replaced by `isUndefinedColumnError` from `src/core/utils.ts` (only SQLSTATE 42703 falls through to legacy; lock timeouts/network blips throw); `dcrDisabled` constructor option lets `serve-http.ts` disable `/register` without monkey-patching the router. Module-private `coerceTimestamp()` normalizes postgres-driver-as-string BIGINT columns to JS numbers at 5 read sites (`getClient` for RFC 7591 §3.2.1 numeric timestamps, `exchangeRefreshToken` + `verifyAccessToken` for the SDK's `typeof === 'number'` check); throws on NaN/Infinity (fail loud at boundary), returns undefined for SQL NULL (callers treat NULL as expired). Not promoted to `utils.ts` — generic BIGINT precision-loss risk. `registerClient` honors `token_endpoint_auth_method: "none"` (RFC 7591 §3.2.1): public PKCE clients store `client_secret_hash = NULL` and the response omits `client_secret`; confidential clients (`client_secret_post` / `client_secret_basic`) keep their one-time-reveal shape; `getClient` normalizes NULL `client_secret_hash` to JS `undefined` so the SDK's clientAuth path accepts public clients. `verifyAccessToken` JOINs `oauth_clients.source_id` (write scope, scalar) + `oauth_clients.federated_read` (read scope, TEXT[]) + `oauth_clients.bound_slug_prefixes` (write fence, TEXT[] — consumed by `enforceClientSlugFence` in `operations.ts`) onto the returned `AuthInfo`; legacy brains degrade via `isUndefinedColumnError` fallback, dropping the newest projection first. `rescopeClient(clientId, {sourceId?, federatedRead?, boundSlugPrefixes?})` is the trusted-operator rescope (CLI `gbrain auth rescope-client`, admin `POST /admin/api/rescope-client`); `boundSlugPrefixes` is tri-state — undefined leaves the binding untouched, `null` clears it, a non-empty array replaces it (explicit empty array rejected as ambiguous deny-all) — so roster churn updates the write fence in place without rotating secrets. with `src/commands/serve-http.ts`: custom `/token` middleware that runs BEFORE the MCP SDK's `clientAuth`. The SDK does plaintext compare against the request's `client_secret`; gbrain stores SHA-256 hashes only, so every confidential-client `/token` request would fail. The middleware detects confidential auth via `Authorization: Basic` header OR `client_secret_post` form body (both shapes per RFC 6749 §2.3.1), verifies via `verifyClient(client_id, presented_secret)` (SHA-256 hash compare), and falls through to the SDK for public PKCE clients (which the SDK's clientAuth still accepts via NULL-`client_secret_hash` normalization). Pinned by `test/oauth-confidential-client.test.ts` (both `client_secret_basic` and `client_secret_post`). - `admin/` — React 19 + Vite + TypeScript admin SPA embedded in the binary via `admin/dist/` served by `serve-http.ts`. 7 screens: Login (bootstrap token → session cookie), Dashboard (metrics + SSE feed + token health), Agents (sortable table + sparklines + Register), Register (modal with scope checkboxes + grant type selector), Credentials reveal (Copy + Download JSON + one-time-only warning), Request Log (filterable paginated), Agent Detail drawer (Details / Activity / Config Export tabs + Revoke). Design tokens: `#0a0a0f` bg, Inter for UI, JetBrains Mono for data, 4-32px spacing scale, rounded pill badges. HTTP-only SameSite=Strict cookie auth. 65KB gzip. Build: `cd admin && bun install && bun run build`; output at `admin/dist/` is committed for self-contained binaries. - `src/commands/auth.ts` — token management. `gbrain auth create/list/revoke/test` for legacy bearer tokens, plus `gbrain auth register-client` and `gbrain auth revoke-client ` for OAuth 2.1 client lifecycle. `revoke-client` runs an atomic `DELETE...RETURNING` on `oauth_clients`; FK `ON DELETE CASCADE` on `oauth_tokens.client_id` and `oauth_codes.client_id` purges every active token + auth code in one transaction; `process.exit(1)` on no-such-client (idempotent). Legacy tokens stored as SHA-256 hashes in `access_tokens`; OAuth clients in `oauth_clients`; legacy tokens grandfather to `read+write+admin` scopes on the OAuth HTTP server (no migration). Every SQL site routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` (and `executeRawJsonb` for the takes-holders `permissions` JSONB column) so `gbrain auth` works against PGLite; the takes-holders write goes through `executeRawJsonb(engine, sql, [name, hash], [{takes_holders:[...]}])` which round-trips with `jsonb_typeof = 'object'`. `register-client` accepts `--source ` (write authority, scalar) and `--federated-read ` (read scope, array) and prints the resolved `Write source` + `Federated reads`; pre-v0.34 clients backfill to `source_id='default'` via migration v60. The bare `gbrain auth create ` form (no `--takes-holders`) mints a token via the exported pure `parseAuthCreateArgs(rest)` (the inline version used `rest[takesIdx + 1]` resolving to `rest[0]` when `takesIdx === -1`, excluding the name from the positional search). Pinned by `test/auth-create-args.test.ts`. +- `src/core/mcp-client.ts` — the thin-client transport (trust boundary). `callRemoteTool(config, toolName, args, opts)` with `CallRemoteToolOptions {timeoutMs, signal}`; `buildAbortController` composes an external signal with the timeout. All transport errors normalize to `RemoteMcpError` via the `toRemoteMcpError` funnel: stable `RemoteMcpErrorReason` union, `RemoteMcpErrorDetail.kind` (`'timeout' | 'aborted' | 'unreachable'`) sub-tag, `RemoteMcpErrorDetail.code` carrying server-supplied error codes (e.g. `missing_scope`). `extractToolErrorCode` parses JSON error envelopes first, falls back to substring detection for legacy server messages. `unpackToolResult(res)` parses tool-call JSON content. `_clearMcpClientTokenCache()` test escape. The CLI routing seam that consumes this lives in `src/cli.ts` (`runThinClientRouted`); see `docs/architecture/thin-client.md`. - `src/commands/connect.ts` + `src/core/connect-probe.ts` — `gbrain connect [--token ]` one-command coding-agent onboarding from a bearer token. Turns an MCP URL + token into a paste-ready `claude mcp add ... -H "Authorization: Bearer ..."` block (default) or, with `--install`, runs it directly and smoke-tests the token. Direct HTTP MCP — Claude Code talks straight to a remote `gbrain serve --http`, no local install needed. Token resolution: `--token` > `$GBRAIN_REMOTE_TOKEN` > placeholder (print) / error (install). The generated block tells the agent to call `get_brain_identity` + `list_skills` (the `LEARN_INSTRUCTION` export, which names `put_page` not `capture` since `capture` is CLI-only, not an MCP tool) with a core-tools fallback for hosts without skill publishing. URL normalization appends `/mcp` to a bare host but REJECTS a scheme-less host; pure helpers (`isLinkLocalOrMetadata`, URL parse, render) are unit-tested. Flags: `--token`, `--name ` (default `gbrain`, validated against `NAME_RE`), `--agent claude-code|codex|perplexity|generic`, `--install`, `--yes` (required for `--install` in non-TTY), `--force`, `--json` (token redacted unless `--show-token`), `--timeout-ms`. `connect` is in `CLI_ONLY` + `CLI_ONLY_SELF_HELP`; dispatched in `cli.ts:handleCliOnly` with no local DB connect. `AGENT_SPECS` drives per-agent rendering + `--install`: `claude-code` → `buildClaudeMcpAddArgv` (literal `-H "Authorization: Bearer "`); `codex` → `buildCodexMcpAddArgv` = `codex mcp add --url --bearer-token-env-var GBRAIN_REMOTE_TOKEN` (Codex reads the token from the env var at runtime, never written to config; `--install` runs it and prints an `export GBRAIN_REMOTE_TOKEN` hint when missing); `perplexity` + `generic` are `installable:false` and reject `--install`. `--oauth` (`supportsOAuth:true` = perplexity/generic only) emits an OAuth 2.1 client-credentials connector block (Issuer URL via `issuerFromMcpUrl` = mcp-url minus `/mcp`, Client ID, Client Secret) — least-privilege scopes + short-lived rotating tokens vs a long-lived full-access secret. Creds from `--client-id`/`--client-secret` (BYO) or `--register` (`deps.registerOAuthClient` shells `gbrain auth register-client --grant-types client_credentials --scopes --token-endpoint-auth-method client_secret_post` and parses `Client ID:`/`Client Secret:`); `--oauth` rejected for claude-code/codex and incompatible with `--install`. `buildJson` is a generic shape (`agent`, `command`/`command_argv` null for perplexity/generic, `header`, `env_var`, oauth fields with redaction); the codex `command` carries only the env-var name, never the token. `cmdString(binary, argv)` POSIX-single-quotes args. `ConnectDeps` = `{isTTY, promptYesNo, hasBinary(bin), runBinary(bin, argv), probe, env(name)}` — binary-generic so `claude` and `codex` share the path; `env` injectable for tests. Security: rendered command single-quotes the token so shell metacharacters can't run code when pasted; token validated before it lands in an HTTP header; link-local / cloud-metadata addresses (incl. IPv4-mapped IPv6 `::ffff:169.254.x.x` and AWS IMDSv2-over-IPv6 `fd00:ec2::254`) refused as a token-exfil guard while localhost/RFC1918/LAN stay allowed; token redacted from all error output. `src/core/connect-probe.ts` is the raw-bearer MCP smoke probe backing `--install`: connects the official MCP SDK `Client` over `StreamableHTTPClientTransport` with a STATIC `Authorization` header (no OAuth/discovery — distinct from `mcp-client.ts:callRemoteTool` which is OAuth-only and `remote-mcp-probe.ts:smokeTestMcp` which only sends `initialize`), runs the full `initialize` handshake via `client.connect()`, then calls `get_brain_identity` (read-scope, non-localOnly) to prove a tool call round-trips. Never throws — every failure maps to `{ ok: false, reason: 'auth' | 'unreachable' | 'timeout' | 'tool_error' | 'unknown', message }` so a wrong/expired token fails at setup, not on the agent's first request. `DEFAULT_PROBE_TIMEOUT_MS = 15_000` shared with `connect.ts`. `serve-http.ts` adds exported pure `skillPublishStatus(publishSkills)` for the startup banner `Skills: published / not published` line + a one-line `gbrain config set mcp.publish_skills true` stderr nudge when publishing is OFF. Docs: `docs/mcp/CODEX.md`, `docs/mcp/PERPLEXITY.md`, `docs/mcp/CLAUDE_CODE.md`, `docs/tutorials/connect-coding-agent.md`. Pinned by `test/connect.test.ts` (pure-helper + render, all four agents) + `test/e2e/connect-bearer.test.ts` (raw-bearer probe + full OAuth chain register→connect→discovery→`/token` mint→`get_brain_identity`, client registered in `beforeAll` before serve takes the PGLite single-writer lock; drives real `claude` + `codex` binaries through `connect --install` with sandboxed `HOME`/`CODEX_HOME`, asserts registration + token never in Codex config, skips when a binary is absent) + `test/e2e/serve-stdio-roundtrip.test.ts` (spawns real `gbrain serve` stdio against a fresh `init --pglite` brain, drives the SDK client through `initialize`→`tools/list`→`tools/call`, asserts the advertised core-tool set and that `capture` is NOT advertised) + `test/serve-skills-publish-nudge.test.ts` (the `test/audit/batch-retry-audit.test.ts` ENOENT case was made hermetic — it had read the real `~/.gbrain/audit`). - `src/commands/upgrade.ts` — self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (`src/commands/migrations/index.ts`) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no runtime walk of `skills/migrations/*.md`). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify); `phaseASchema` has a 600s timeout for duplicate-heavy brains. `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. The RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape (orchestrators no longer call `appendCompletedMigration`). `statusForVersion` prefers `complete` over `partial` (never regresses); 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. Schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) live in the `MIGRATIONS` array in `src/core/migrate.ts`. `in-process.ts` exports `runMigrateOnlyCore({timeoutMs?})` — single source of truth for "bring schema to head" (`configureGateway` → `createEngine` → `connect` → `initSchema` → `disconnect`, idempotent, 600s `MIGRATE_ONLY_TIMEOUT_MS` guard, throws `MigrateOnlyError` on no-config / timeout); the orchestrators' 9 schema phases AND `init.ts:initMigrateOnly` both delegate to it so schema bring-up can't drift (running in-process removes the spawn that died with `getaddrinfo ENOTFOUND` on Windows + bun + Supabase pooler). `runGbrainSubprocess` is the diagnostic wrapper for the remaining non-schema spawns (extract/repair/stats): captures child stderr (64MB buffer) into the thrown error. `v0_13_1.ts:phaseCGrandfather` is a CHUNKED bulk SQL pass keyed on `pages.id` (globally unique PK, NOT slug — slug uniqueness is `(source_id, slug)`), filters `deleted_at IS NULL` (no tombstones), chunked in `CHUNK_SIZE` batches (DELETE_BATCH_SIZE convention) for bounded lock-hold; the rollback log carries `{id, slug, source_id, pre_frontmatter}` so rollback is unambiguous across sources; idempotent + resumable (each UPDATE flips its rows out of `GRANDFATHER_WHERE`). Pinned by `test/migration-in-process.serial.test.ts` and `test/migrations-v0_13_1-grandfather.test.ts`. @@ -298,35 +275,25 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo] [--source ]`: surfaces pages with zero inbound wikilinks, grouped by domain (auto-generated/raw/pseudo filtered by default). Also exposed as `find_orphans` MCP op. `findOrphans`/`getOrphansData` (the canonical pure fn shared with doctor's `orphan_ratio`) takes `{ sourceId?, sourceIds? }`; `BrainEngine.findOrphanPages(opts?)` (both engines) scopes ONLY the candidate side (`p.source_id = $1` scalar, or `= ANY($1::text[])` federated) while still counting inbound links from ANY source — a page in source X linked FROM source Y is reachable, so NOT an orphan of X (deliberate definition; the stricter intra-source-only reading is rejected). `--source` is an explicit raw-flag parse (NOT `resolveSourceWithTier`, which would scope bare invocations to a default). The `total_linkable` denominator enumerates ALL live pages (scoped) and subtracts every excluded-by-slug page (templates/, scratch/, etc.) so excluded NON-orphan pages with inbound links don't inflate it and suppress warnings. The `find_orphans` MCP op threads `sourceScopeOpts(ctx)` so a source-bound OAuth client doesn't see brain-wide orphans. `gbrain doctor --source ` scopes `orphan_ratio` and, under explicit `--source` below 100 entity pages, reports the ratio with a low-scale caveat (thin-client `doctor --source` orphan_ratio remains brain-wide — TODO). Pinned by `test/orphans-source-scope.test.ts` (PGLite) + `test/e2e/engine-parity.test.ts` (Postgres↔PGLite scalar + federated parity). Contributed by @knee5. - `src/commands/salience.ts` — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`. - `src/commands/anomalies.ts` — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds: tag, type. -- `src/commands/whoknows.ts` — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false). Ranking formula: `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via `SearchOpts.types: ['person', 'company']` (no post-filter waste); hybridSearch's internal salience+recency boosts are intentionally disabled so the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. +- `src/commands/whoknows.ts` — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false). Ranking formula: `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via `SearchOpts.types: ['person', 'company']` (no post-filter waste); hybridSearch's internal salience+recency boosts are intentionally disabled so the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. with `src/core/operations.ts:find_experts`: T1.5 wiring sites. Pack-aware via `expertTypesFromPack(pack.manifest)` from `best-effort.ts`. Pack-load failure → EMPTY filter (NOT hardcoded `['person', 'company']` defaults). A `researcher` type declared `--expert` now surfaces in `whoknows` results. - `src/commands/eval-whoknows.ts` — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate. Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`; exit 0/1/2 for pass/fail/usage. `WhoknowsFn` callable abstraction makes the gates impl-agnostic; `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})`, local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` (dispatch shape under `src/commands/eval.ts`); the regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants pinned by `test/eval-whoknows.test.ts` (25 cases incl. null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries; placeholder uses obviously-example slugs (`wiki/people/example-alice`). Drives `test/e2e/whoknows.test.ts` (seeds a matching synthetic brain, asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904). `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent: validation-gated (median-of-3 + epsilon=0.05), budget-capped (preflight estimator), per-skill DB-locked (`tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (history-intent-first 5-step commit), body-only mutations (frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload`. Two reflect calls per step; rejected-edit buffer LRU-bounded to 100; bundled-skill gate; bootstrap workflow (sentinel + `--bootstrap-reviewed`); D_sel floor (>=5 with `--split` override); audit JSONL via `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Surface: dream-cycle phase wrapper; `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`); `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`); MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly, validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers); Minion `skillopt` handler + `--background` with `allowProtectedSubmit: true`; write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag); held-out real-user test set via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`). `--bootstrap-from-skill` → `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts`: reads `SKILL.md` directly (no `routing-eval.jsonl`), makes ONE LLM call emitting a full starter benchmark (tasks + rule judges) as JSONL, parsed line-by-line with skip-bad-line salvage and a min-2-valid-checks-per-task drop; provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`). `--bootstrap-tasks N` (default 15, capped 50); `maxTokens` scales `min(8000, max(4000, N*220))`. The stderr REVIEW line prints the literal `gbrain skillopt --bootstrap-reviewed --split 1:1:1` — load-bearing because the default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark needs `--split 1:1:1`. Both bootstrap generators share `assertBenchmarkAbsent` + `readSkillBodyOrThrow`; `--bootstrap-from-skill` is mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume`. Generated rule judges are explicitly WEAK DRAFTS to be strengthened during the review gate. The F11 held-out gate is wired: `--held-out ` is parsed and threaded through every caller (CLI main + `--background` `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` `held_out_path` param), running at CHECKPOINT ACCEPTANCE so no-mutate/fleet paths can't promote a held-out-failing candidate. `assertBundledMutationHeldOut` in bundled-skill-gate.ts: bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5, derived so they can't desync) or hard-refuses (exit 2), for ALL callers (they funnel through `runSkillOpt`); held-out must be task_id-DISJOINT from the benchmark (overlap rejected — can't catch overfitting). `receipt.baseline_sel_score` populated + a real final-test eval (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`; shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs baseline/final-test/held-out scoring. `--no-mutate` writes proposed.md via `writeProposed` in version-store.ts. `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'`), recorded in `RunReceipt` + audit `run_start` for replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the partition; one-shot fence-strip is anchored (`^```...```$`) so an embedded code sample isn't truncated. Budget no-pricing fix: Claude Haiku 4.5's dateless canonical id `claude-haiku-4-5` is in `src/core/anthropic-pricing.ts` (a `BudgetTracker`-capped run on Haiku otherwise threw `no_pricing` on the FIRST `chat()` of every rollout); `runValidationGate` (validate-gate.ts) scans settled results for `isMustAbortError(error)` (from `worker-pool.ts`; `BUDGET_EXHAUSTED` is in `MUST_ABORT_ERROR_TAGS`) and re-throws so the caller aborts loudly instead of recording a hollow `selScore:0` — ordinary non-abort rollout errors still fail-open to `score:0` (judge-hiccup posture preserved). Pinned by 152 tests across 18 files (foundation + adversarial + v2 surface + E2E PGLite serial), `test/skillopt/bootstrap-from-skill.test.ts` (20 cases), `test/skillopt/rollout.test.ts`, `test/skillopt/validate-gate-abort.test.ts` (3 cases), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, no-DB-pollution). Drives the Track B SkillOpt benchmark suite in the sibling `gbrain-evals` repo. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge rejecting ideas with resistance >4.5 "too obvious", stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2`. `judges.ts` exports `runJudge(config, ideas)` + two configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` cognitive_load 0.50 + inversion rule). Calibration cold-start fallback: when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back in `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch); internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. The fire-and-forget IIFE is tracked in a module-scoped `Set>`; `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns the pending count. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then a fallback `process.exit(0)` fires ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive) — closes the PGLite CLI search/query/get-hang class where the IIFE raced disconnect and PGLite's WASM kept Bun's event loop alive. `pages.last_retrieved_at TIMESTAMPTZ NULL` has a full (NOT partial) B-tree index covering both NULL and range branches; full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output via `isLsdOutput()` in `src/core/cycle/transcript-discovery.ts` short-circuiting `isDreamOutput()`. `gbrain eval brainstorm ` is a three-axis conjunctive gate (distance + usefulness + grounding — distance alone is gameable). `gbrain doctor` has a `brainstorm_health` check (migration applied, `search.track_retrieval` setting, calibration cold-start status). `judges.ts` computes the judge token budget via `computeJudgeMaxTokens(ideaCount, modelId)` (named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`; `ANTHROPIC_OUTPUT_CAPS` map: Opus 4.7 32K, Sonnet 4.6 / Haiku 4.5 64K, legacy Claude 3.5 8K) so a large multi-call judge doesn't truncate mid-JSON; with no `modelOverride` the cap routes through the gateway's actual configured chat model via `getChatModel()`. `--save` for both commands persists through the canonical ingestion path: `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so search finds it, no embedding cost at save) THEN renders the saved row to disk via the shared `writePageThrough` helper (file rendered FROM the row so the two sinks can't diverge and `gbrain sync` doesn't churn it). `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message (both-sinks, DB-only when no `sync.repo_path`/repo-not-a-dir, DB-saved-but-file-errored, total-failure → loud `save FAILED … NOT persisted` on stderr + nonzero exit) — closes the silent-false-success class where `--save` printed "Saved" unconditionally even when the DB write failed. `buildIdeaSlug(question, label, nonce?)` adds a random nonce suffix (injectable for tests) so two same-day runs sharing the first 60 slug chars don't clobber. `--json` callers stay DB-only. `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (string `buildBrainstormFrontmatter` untouched). Pinned by `test/last-retrieved.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir asserts search/get/query exit 0 in <15s + daemon-survival), `test/fix-wave-structural.test.ts` (asserts the drain `await` is textually BEFORE `engine.disconnect`), `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm,judges-maxtokens,save}.test.ts`. Open Collider source: `github.com/CL-ML/open-collider`. - `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. On a durability-hardened repo (`isDurabilityHardened` — the gbrain post-commit hook is installed, i.e. the user ran `gbrain sources harden`), a successful write is best-effort COMMITTED via `commitWriteThroughFile` (path-limited `git commit -- `, never sweeps unrelated edits; the hook then background-pushes) so write-through content reaches git instead of accumulating uncommitted forever (#2426); result carries `committed?: boolean`. Unhardened repos keep write-only behavior. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` + `test/write-through-commit.serial.test.ts`. - `src/core/model-id.ts` — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface has no parallel re-implementations of `provider:model` splitting — slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly instead of falling through to "unknown model". Distinct from the gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`, which throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by `test/model-id.test.ts`. -- `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). Batch-load fast path on Postgres uses a single SQL query (fixes the PgBouncer round-trip timeout, ~60s → ~6s), gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1`. Batch projection is `SELECT ... ORDER BY source_id, slug` (NOT `SELECT DISTINCT ON (slug)`, which collapsed same-slug-different-source pages into one scan) so multi-source brains scan each `(source, slug)` row independently. Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`; batch + sequential paths report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. A source holding a LIVE, non-expired per-source sync lock (`inspectLock(engine, syncLockId(source.id))` from `src/core/db-lock.ts`) is reported as actively syncing (the message names the holder pid + host) and counted in `synced_recently_count`, NOT flagged stale — the live lock is the only honest in-progress signal (checkpoint banking can't distinguish in-progress from wedged: a blocked sync banks its files but writes no anchor). A blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either falls through to the stale path and is never masked; the dynamic `db-lock` import is swallowed to a no-op on a stub engine or pre-lock-table brain, so this can only ADD an in-progress verdict, never suppress a real stale one. The in-progress note is appended to whatever verdict the buckets produce and is empty when nothing is syncing, so steady-state messages stay byte-for-byte unchanged. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape. `retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations. - `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as active prefix; nested wraps replace then restore), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log`/`console.error`). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog`/`serr` fall through to bare `console.log`/`console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form) to defeat log-injection through newline/control-character names. Coverage: `src/commands/sync.ts` performSync + callees, `src/commands/embed.ts` runEmbedCore + helpers, `src/core/progress.ts` emitHumanLine. -- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` / `--brain ` stripped. `--brain` is the brain-axis (which database) selector: exact-match only (`--brain-*` per-command flags pass through), value validated against the mount-id regex at parse time, missing/malformed value THROWS — never a silent host fallback. `connectEngine` in `src/cli.ts` feeds it (plus the ambient `GBRAIN_BRAIN_ID` / `.gbrain-mount` / mount-path tiers) through `resolveBrainId` → `BrainRegistry.getBrain`, which throws `UnknownBrainError` for an unregistered id; mounts get no auto-migrations and keep the host-config AI gateway. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators (propagates `--brain=` so children stay on the parent's brain). `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. +- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` / `--brain ` stripped. `--brain` is the brain-axis (which database) selector: exact-match only (`--brain-*` per-command flags pass through), value validated against the mount-id regex at parse time, missing/malformed value THROWS — never a silent host fallback. `connectEngine` in `src/cli.ts` feeds it (plus the ambient `GBRAIN_BRAIN_ID` / `.gbrain-mount` / mount-path tiers) through `resolveBrainId` → `BrainRegistry.getBrain`, which throws `UnknownBrainError` for an unregistered id; mounts get no auto-migrations and keep the host-config AI gateway. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators (propagates `--brain=` so children stay on the parent's brain). `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. `CliOptions` gains `explain: boolean`. `parseGlobalFlags` recognizes `--explain` anywhere in argv (stripped before command dispatch). `src/cli.ts` `formatResult` for `search` + `query` cases routes to `formatResultsExplain` from `src/core/search/explain-formatter.ts` when `CliOptions.explain` is set; falls through to the existing JSON / human formatters otherwise. `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow ` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on. - `src/core/db-lock.ts` — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the `gbrain_cycle_locks` table. Parameterized lock id so scopes nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID`) for `performSync`'s narrower writer window. UPSERT-with-TTL semantics survive PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. It also does automatic same-host dead-pid takeover: when the upsert finds a held, NOT-TTL-expired lock whose holder is on this host and provably dead, it reclaims via a guarded `DELETE WHERE id=$1 AND holder_pid=$2` + one normal-upsert retry returning the standard handle (refresh/release intact). The liveness check is the exported `classifyHolderLiveness(pid, host, ageMs, opts?)` / `isHolderDeadLocally(...)` (injectable `process.kill` seam; `HOLDER_TAKEOVER_GRACE_MS = 60_000` PID-reuse guard; EPERM classified as `alive` so a live process you don't own is never stolen). TTL-expired locks stay the upsert's job; cross-host stays TTL-only. `runBreakLock` (`src/commands/sync.ts`) consumes the same predicate. Background reaper (#1972): `reapDeadHolderLocks(engine)` is the periodic sweep the contention path lacked — it deletes locks whose holder is `isHolderDeadLocally`, scoped to the `gbrain-sync:*` / `gbrain-cycle`/`gbrain-cycle:*` namespaces ONLY (election/supervisor/reindex locks keep TTL-only behavior, untouched), via `deleteLockRowExact(engine, id, pid, acquiredAt)` — a snapshot-matched delete (`date_trunc('milliseconds', acquired_at) = $3`, so the ms a JS Date keeps survives) that's TOCTOU-safe against a reused PID taking the lock between SELECT and DELETE. `cycle.ts` runs it at cycle start (before the sync phase); `gbrain doctor --fix` runs it for no-autopilot brains. `selectLockRows(engine, opts?)` + a shared row→`LockSnapshot` mapper are the single canonical reader now backing `inspectLock` + `listStaleLocks` + the reaper (was triplicated). `isLockHolderLive(snap, ttlMinutes)` (#2227) is the observability liveness predicate — freshness-keyed (`ttl_expired` plus the heartbeat steal-grace), never `process.kill`, so `gbrain jobs supervisor status` / `gbrain doctor` can report a live supervisor via its queue lock without a PID-reuse false-positive. Pinned by `test/db-lock-auto-takeover.test.ts` + `test/db-lock-reap.test.ts`. - `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. `resolveMaxConnections()` (reads `GBRAIN_MAX_CONNECTIONS`, undefined when unset) + `clampWorkersForConnectionBudget(workers, perWorkerPool, maxConnections, parentPool)` back the opt-in single-sync connection-footprint clamp so a big sync stays under a low pooler cap (`parent_pool + workers×perWorkerPool ≤ budget`); `gbrain doctor`'s `pool_budget` check (`computePoolBudgetCheck` / `checkPoolBudget` in `src/commands/doctor.ts`) warns when the budget leaves no room for a worker, pointing at `GBRAIN_POOL_SIZE=2`. Pinned by `test/pglite-workers-clamp.test.ts`. - `src/core/worker-pool.ts` — Canonical sliding-pool + bounded-semaphore primitive (extracted from `src/commands/embed.ts` sliding-pool sites and `src/commands/eval-cross-modal.ts` `runWithLimit` semaphore). Two exports: `runSlidingPool({items, workers, onItem, signal?, onError?, failureLabel?, onProgress?})` + `runWithLimit({items, limit, fn, signal?})`. Atomicity invariant: `const idx = nextIdx++` is one synchronous JS statement (no `await` between read and write — guaranteed by the single-threaded event loop), documented in the module header AND enforced by `scripts/check-worker-pool-atomicity.sh` (wired into `bun run verify`), which rejects importing `worker_threads` in any consuming file and inserting `await` between the `nextIdx` read and write. `MUST_ABORT_ERROR_TAGS` set is seeded with `BUDGET_EXHAUSTED` from `src/core/budget/budget-tracker.ts`; tagged errors (matched via `err.tag === 'BUDGET_EXHAUSTED'` to avoid cross-module import) bypass `onError` and hard-abort the pool via `AbortController.abort()` to in-flight `onItem` — the budget cap is a structural ceiling under concurrency. `failures[]` shape is `{idx, label, error}` records (NOT full items; callers supply `failureLabel(item) => string`) for bounded memory under huge brains. Pinned by `test/worker-pool.test.ts` + `test/scripts/check-worker-pool-atomicity.test.ts`. Drives every `--workers N` bulk command. -- `src/commands/embed.ts` extension — both inline sliding-pool sites (`embedAll` simple at `:458-467` and `embedAllStale` paginated + AbortSignal at `:586-632`) call `runSlidingPool` from the shared worker-pool helper. Invariant-level contract preserved (counts + cost + AbortSignal propagation + per-batch rate-limit retry via `embedBatchWithBackoff`); byte-equality on progress-event ORDERING is NOT promised. The `GBRAIN_EMBED_CONCURRENCY || 20` default is preserved and embed bypasses `resolveWorkersWithClamp` because the 20-worker default would otherwise silently change every brain's embed hot path. Pinned by `test/embed-helper-migration.test.ts` (asserts the helper is wired in AND the pre-migration `let nextIdx = 0` + `Promise.all(Array.from({length: numWorkers}, ...))` shapes are gone). -- `src/commands/extract-conversation-facts.ts` extension — `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). `deleteOrphanFactsForPage(engine, sourceId, slug)` provides delete-orphans-first replay safety — wipes facts from a prior crashed run for this (sourceId, slug) before re-extracting, closing the "terminal audit row written after partial insertFacts failure" class. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Checkpoint state is a shared `cpMap: Map` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-workers.test.ts` + the existing extract-conversation-facts behavioral tests. -- `src/commands/extract-conversation-facts.ts` + `src/commands/doctor.ts` durable outcome authority — page completion survives operation-checkpoint GC through versioned terminal audit rows (`cli:extract-conversation-facts:terminal:v2`), while recognized pages with no eligible segment use the separate `cli:extract-conversation-facts:non-extractable:v2` source. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed delete-first. `no_match`, errors, cancellation, and dry runs never become durable negatives. Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. -- `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. -- `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. -- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. `synthesize_concepts` writes concept pages through `importFromContent` (#2163: the same parse→chunk→embed pipeline put_page uses, with put_page's `isAvailable('embedding')` → `noEmbed` gate) so `concepts/` pages carry `content_chunks` + embeddings and are reachable by retrieval (where `source-boost.ts` weights them 1.3×). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count, skipped_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok (sources skipped by `--missing-path skip` count as ok), 1 any error. `--missing-path ` (default fail) handles sources whose `local_path` does not exist on this machine — machine-specific state in a brain-wide table, so a brain registered from several machines fails every foreign source on every run; `skip` classifies them `skipped_missing_path` (⊘ line, envelope entry with `local_path`, excluded from `error_count` and the rc gate) via the exported pure helpers `parseMissingPathMode` + `partitionMissingPathSources`, pinned by `test/sync-all-missing-path.test.ts`; default `fail` stays loud because on a single-machine brain a missing path usually means an unmounted volume. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). A warn-and-continue internal `git pull` failure (non-timeout class — e.g. a local-path origin rejected by `protocol.file.allow=never`) still falls through to sync the local working tree, but a ZERO-import run after a failed pull returns `partial` with `reason: 'pull_failed'` instead of `up_to_date`: `last_commit` AND the `last_sync_at` heartbeat stay frozen (so doctor `sync_freshness` / `sources status` staleness fires), the single-source CLI exits non-zero, `sync --all` exits non-zero if any source hit it (JSON envelope carries the per-source `reason`), and the autopilot cycle's sync phase maps it to `warn`. Timeout-class partials keep their pre-existing exit-0 / phase-`ok` semantics (they converge on retry; a failing pull does not). Pinned by `test/sync-pull-failed-anchor.serial.test.ts`. -- `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. `runImport` opts also carry `exclude` (glob filter over dir-relative paths, threaded by `performFullSync` for `sync --exclude`; warns when every file is excluded — NAV-4) and `slugRoot` (slug/`source_path` base for monorepo subdir syncs, #753/#774; the resume checkpoint stays dir-relative per `resumeFilter`'s contract).- `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. +- `src/core/embedding-dim-check.ts` — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). -- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration — and for `synthesize` (#1586: threaded as `SynthesizePhaseOpts.sourceId` so synthesized pages land in the cycle's resolved source, not `'default'`) — so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. +- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration — and for `synthesize` (#1586: threaded as `SynthesizePhaseOpts.sourceId` so synthesized pages land in the cycle's resolved source, not `'default'`) — so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. `runPhaseLint` + `runPhaseBacklinks` carry the `export` keyword so behavioral tests can drive them directly (internal helpers exposed for test-only consumption; downstream code should NOT depend on them). Pinned by `test/cycle-legacy-phases.test.ts` (11 cases across both phases: clean run → status='ok', partial fix → status='warn' with `dryRun` in details, dry-run path doesn't write, throw-from-lib → status='fail' with the wrapper's try/catch envelope populated). Future phase wrappers (sync, extract, embed, orphans, extract_facts, resolve_symbol_edges, recompute_emotional_weight) land as additional describes in the same file. with `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts`: six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. `synthesize_concepts` writes concept pages through `importFromContent` (#2163: the same parse→chunk→embed pipeline put_page uses, with put_page's `isAvailable('embedding')` → `noEmbed` gate) so `concepts/` pages carry `content_chunks` + embeddings and are reachable by retrieval (where `source-boost.ts` weights them 1.3×). `purge` phase (the cycle's 9th phase, for soft-delete TTLs) also GCs stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE). #1737: the cycle threads its abort signal into the embed phase (`runPhaseEmbed(engine, dryRun, signal)`) so a timed-out cycle's long embed phase honors cancellation and releases `gbrain_cycle_locks` right away instead of after a full backlog run. - `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`; when `dream.synthesize.output_root` is set, `loadAllowedSlugPrefixes(outputRoot)` remaps the `wiki/`-rooted globs to the configured namespace — #2415 — and the same root drives the prompt slug templates; default 'wiki', validated against the slug grammar via the exported `loadOutputRoot`). The phase is source-scoped (#1586): cycle.ts threads `cycleSourceId` as `opts.sourceId` → each child's `SubagentHandlerData.source_id` → the subagent tool registry's `OperationContext.sourceId`, so put_page writes, collected refs, the summary page, and reverse-writes all target the cycle's resolved source ('default' when unscoped; reverse-writes for the cycle's own source land at `brainDir/.md`, foreign sources under `brainDir/.sources//`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `stampDreamProvenance` (#2569) additionally persists the same marker into the `pages.frontmatter` JSONB row (merge via `executeRawJsonb`, raw object bound to `$N::jsonb`) for every child-written page BEFORE reverse-rendering, so generated pages are DB-queryable and a later put_page write-through (which re-renders from the DB row) can't erase the stamp. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24); per-chunk subagent job/wait timeouts are `dream.synthesize.subagent_timeout_ms` / `dream.synthesize.subagent_wait_timeout_ms` (defaults 30/35 min). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. - `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize (imports `loadAllowedSlugPrefixes` + `loadOutputRoot` from synthesize.ts — #2415: the reflections lookup, prompt slug templates, and allow-list all honor `dream.synthesize.output_root`, default 'wiki'). Subagent job/wait timeouts are config keys `dream.patterns.subagent_timeout_ms` / `dream.patterns.subagent_wait_timeout_ms` (defaults 30/35 min, mirroring the `dream.synthesize.*` pair). The phase status reflects the child outcome: non-`complete` outcome with zero writes → `fail` (error code `PATTERNS_CHILD_`); non-`complete` with partial writes → `warn`. Runs AFTER `extract` so the graph is fresh. @@ -359,7 +326,6 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section; includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich. - `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`) - `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts) -- `src/commands/integrations.ts` — Standalone integration recipe management (no DB needed). Exports `getRecipeDirs()` (trust-tagged recipe sources), SSRF helpers (`isInternalUrl`, `parseOctet`, `hostnameToOctets`, `isPrivateIpv4`). Only package-bundled recipes are `embedded=true`; `$GBRAIN_RECIPES_DIR` and cwd `./recipes/` are untrusted and cannot run `command`/`http`/string health checks. - `src/core/search/expansion.ts` — Multi-query expansion via Haiku. Exports `sanitizeQueryForPrompt` + `sanitizeExpansionOutput` (prompt-injection defense-in-depth). Sanitized query is only used for the LLM channel; the original query still drives search. - `recipes/` — Integration recipe files (YAML frontmatter + markdown setup instructions) - `docs/guides/` — Individual SKILLPACK guides (broken out from monolith) @@ -377,7 +343,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `docs/mcp/` — Per-client setup guides (Claude Desktop, Code, Cowork, Perplexity) - BrainBench (benchmark suite + corpus): lives in the separate [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. Not installed alongside gbrain. - `skills/_brain-filing-rules.md` — Cross-cutting brain filing rules (referenced by all brain-writing skills) -- `skills/RESOLVER.md` — Skill routing table (based on the agent-fork AGENTS.md pattern) +- `skills/RESOLVER.md` — Skill routing table (based on the agent-fork AGENTS.md pattern) with `skills/manifest.json`: schema-author wired into the dispatcher with the full functional-area trigger list (compressed routing pattern per the dispatcher convention). - `skills/conventions/` — Cross-cutting rules (quality, brain-first, model-routing, test-before-bulk, cross-modal) - `skills/_output-rules.md` — Output quality standards (deterministic links, no slop, exact phrasing) - `skills/signal-detector/SKILL.md` — Always-on idea+entity capture on every message @@ -402,24 +368,17 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter - `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls) - `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law) -- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates) +- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates) lint rules `huge-page` (flags pages exceeding `content_sanity.bytes_warn`) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor. `lint.ts` lifts DB config when `~/.gbrain/` is reachable; falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts`. with `src/commands/sources.ts`: `gbrain lint` gains a `markup-heavy` rule (flags pages whose prose-vs-markup ratio exceeds `content_sanity.max_markup_ratio`, reusing `assessContentSanity` so lint/gate/scan share one assessor); pinned by `test/lint-content-sanity.test.ts`. `gbrain sources audit ` becomes disposition-aware: its dry-run disk scan reports would-quarantine / would-reject / would-flag counts driven by the effective `content_sanity.junk_disposition` + markup config, so an operator previews the gate's verdict before sync. The `content-sanity-audit` JSONL (`src/core/audit/content-sanity-audit.ts`) records the new quarantine/flag dispositions. - `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment) - `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real. - `src/commands/pages.ts` — `gbrain purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations. - `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `recordCompleted` binds its array through `$3::text::jsonb` (NOT a bare `$3::jsonb`) so postgres.js `.unsafe()` doesn't double-encode `JSON.stringify(sorted)` into the scalar string that CHECK rejects — the #2339 bug that aborted every multi-source sync at the first pin write (PGLite parsed it silently, so it shipped). A DATABASE_URL-gated `test/e2e/op-checkpoint-jsonb-parity.test.ts` (its own CI job) asserts the array shape on real Postgres. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred. - `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage). -- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`. -- `src/commands/jobs.ts` extension — registers 11 Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` stays the single source of truth for phase semantics. The standalone `sync` handler passes `noExtract: true` to match `runPhaseSync`'s contract (doctor's remediation plan emitting `[sync, extract]` would otherwise double-extract). -- `src/core/minions/protected-names.ts` extension — `PROTECTED_JOB_NAMES` includes `synthesize`, `patterns`, `consolidate`. These phases internally submit `subagent` children with `allowProtectedSubmit=true` and can spend Anthropic credits. Only trusted local callers (CLI, autopilot, `doctor --remediate`) can submit them; MCP requests are rejected by `submit_job`'s protected-name guard. -- `src/commands/autopilot.ts` extension — targeted-submit loop instead of blanket `autopilot-cycle` dispatch. Each tick: cheap `engine.getHealth()` (single SQL count) + `computeRecommendations()`, then route by shape — `score >= 95 AND no plan AND <60min since last full` → sleep; `score >= 95 AND >=60min` → submit `autopilot-cycle` (60-min floor exercises phase-coupling invariants on healthy brains); `plan <= 3 steps AND est <5min` → submit individual handlers; `plan large OR score < 70` → submit full `autopilot-cycle`. The `gbrain-cycle` lock ensures targeted submissions and the full cycle can't run concurrently. `maxWaiting: 1` per submit closes the queue-fan-out vector. - `src/core/abort-check.ts` (#1737) — one canonical place for cooperative-abort checks across gbrain's long loops. `isAborted(signal?)` → boolean (for loops that `break` and return partial progress). `throwIfAborted(signal?, label?)` throws an `AbortError` (`name === 'AbortError'`) at phase boundaries, preferring the signal's `reason` ('wall-clock'/'lock-lost'/'shutdown') so the unwind self-describes. `anySignal(internal, external?)` composes two signals into one that fires when EITHER does (platform `AbortSignal.any` with a manual-relay fallback), returning the internal unchanged when there's no external so non-aborting callers pay nothing. The fix for the #1737 cycle-wedge: the embed phase ignored its abort signal and ran to completion, so `gbrain_cycle_locks` stayed held and later autopilot cycles skipped with `cycle_already_running`; threading these checks through `runPhaseEmbed → runEmbedCore → embedAll(Stale)/embedPage` lets the phase bail and release the lock immediately. Coverage now spans every long cycle-reachable phase (#1972), not just embed: `extract` (incremental `extractForSlugs` + the full-walk `extractLinksFromDir`/`extractTimelineFromDir`, all via `runSlidingPool`'s signal), `extract_facts` (per-page loop + the per-page `embed` signal + `runPhantomRedirectPass`'s 30s lock-retry), `consolidate`'s bucket loop, and `lint` (which is synchronous, so it `await`s a periodic yield to let the signal land). `runCycle` adds a terminal abort check before stamping `last_full_cycle_at` so a cancelled cycle never reports a completed full run, plus a per-phase `duration_ms` warning that names any phase overrunning the worker's 30s force-evict deadline. Pinned by `test/abort-check.test.ts` + `test/cycle-abort.test.ts`. -- `src/core/cycle.ts` extension — `purge` phase (the cycle's 9th phase, for soft-delete TTLs) also GCs stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE). #1737: the cycle threads its abort signal into the embed phase (`runPhaseEmbed(engine, dryRun, signal)`) so a timed-out cycle's long embed phase honors cancellation and releases `gbrain_cycle_locks` right away instead of after a full backlog run. -- `src/commands/embed.ts` extension — wires `--background` as the reference integration for the `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job, prints `job_id=N` to stdout, exits 0. Composable: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same pattern in a follow-up wave. #1737: `runEmbedCore` accepts an optional `signal` threaded down both the `--stale` and `--all` paths (`embedAllStale`/`embedAll`/`embedPage`); each composes it with the internal wall-clock budget via `anySignal` and checks `isAborted`/`effectiveSignal.aborted` in every per-slug loop, page-claim pool, and `embedBatch` call, so a worker abort (wall-clock timeout / lock loss / SIGTERM) stops embedding within a batch. Pinned by `test/embed.serial.test.ts`. -- `src/core/cli-options.ts` extension — `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow ` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on. - `openclaw.plugin.json` — ClawHub bundle plugin manifest -- `src/commands/capture.ts` + `src/commands/serve-http.ts` + `src/core/{operations,import-file,types,utils,facts/absorb-log,brainstorm/{orchestrator,error-classify},scope,postgres-engine,pglite-engine}.ts` extensions — ingestion-cathedral productionization after a smoke test against Supabase+PgBouncer. Capture frontmatter merge via `mergeCaptureFrontmatter` (uses gray-matter directly, NOT the lossy `parseMarkdown`); `/ingest` null-guard + outer try/catch envelope with `!res.headersSent` guard; dedup via separate normalize-for-hash (`normalizeForHash` strips BOM/CRLF/whitespace/NFKC) + body-after-frontmatter-strip on the DB hash (excludes `captured_at` + `ingested_at` so capture-cli timestamp variations don't invalidate the chunk cache); friendly `pages_source_id_fk` rewrite via `maybeRewriteSourceFkError` on BOTH local + thin-client `callRemoteTool` catch blocks; `facts:absorb` 'No database connection' suppression via typed `instanceof GBrainError && e.problem` check + first-occurrence stack-trace info log (module-scoped `_hasLoggedDisconnectedFactsAbsorb` flag, test seam `_resetFactsAbsorbDisconnectedFlagForTests`); CLI help discoverability (`capture` added to `CLI_ONLY_SELF_HELP` + pre-engine-bind `--help` short-circuit in `handleCliOnly` + a `BRAIN` section in `printHelp`); binary-file guard via `detectBinaryNullByte(buf)` first-8KB NUL scan on `--file` (Buffer-read, no encoding) and `--stdin` (`readStdinBuffer` accumulator); provenance write-through — put_page accepts 3 optional params (source_kind, source_uri, ingested_via; `ingested_at` server-stamped) + trust gate (when `ctx.remote !== false` IGNORE client params, server stamps `mcp:put_page`, fail-closed) + COALESCE-preserve UPDATE semantics (omitting params on a later put_page preserves prior values; first-write-wins); `/admin/api/register-client` scopes normalization via `normalizeScopesInput(raw: unknown)` in `src/core/scope.ts` (accepts string/string[]/missing; rejects `['read write']` space-in-element shape, non-string elements, empty array, unknown scopes; deduped + sorted); brainstorm timeout surfacing via an orchestrator-level try/catch at `runBrainstorm` entry (single-point wrap covers every internal SQL site, classifies SQLSTATE 57014 via postgres.js `.code` / `.sqlState` / message fallback into `StructuredAgentError` code `brainstorm_timeout` with a hint covering all 3 PG cancel sub-causes); read-path surfaces all 4 provenance columns via `getPage` projection + `rowToPage` 3-state optional read + `Page` interface; canonical source resolver routes capture through `resolveSourceWithTier(engine, parsed.source, cwd)`; thin-client `--source` rejection (server-side OAuth client registration owns source scope); the `source_kind` taxonomy is closed (`capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler`), `--source` maps to source_id only. Tests: `test/capture-build-content.test.ts`, `test/capture-runcapture.test.ts`, `test/put-page-provenance.test.ts`, `test/scope-normalize.test.ts`, `test/cli-help-discoverability.test.ts`, `test/brainstorm-timeout.test.ts`; extended `test/facts-absorb-log.test.ts`, `test/import-file.test.ts`, `test/e2e/engine-parity.test.ts`, `test/e2e/serve-http-ingest-webhook.test.ts`. Report at `docs/v0.38-smoke-test-report.md`. Follow-ups in TODOS.md: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer, magic-byte allowlist for binary detection, `--source-kind` override flag, ingest_capture handler migration, provenance-history table, facts:absorb root-cause trace. +- `src/commands/capture.ts` + `src/commands/serve-http.ts` + `src/core/{operations,import-file,types,utils,facts/absorb-log,brainstorm/{orchestrator,error-classify},scope,postgres-engine,pglite-engine}.ts` — ingestion-cathedral productionization after a smoke test against Supabase+PgBouncer. Capture frontmatter merge via `mergeCaptureFrontmatter` (uses gray-matter directly, NOT the lossy `parseMarkdown`); `/ingest` null-guard + outer try/catch envelope with `!res.headersSent` guard; dedup via separate normalize-for-hash (`normalizeForHash` strips BOM/CRLF/whitespace/NFKC) + body-after-frontmatter-strip on the DB hash (excludes `captured_at` + `ingested_at` so capture-cli timestamp variations don't invalidate the chunk cache); friendly `pages_source_id_fk` rewrite via `maybeRewriteSourceFkError` on BOTH local + thin-client `callRemoteTool` catch blocks; `facts:absorb` 'No database connection' suppression via typed `instanceof GBrainError && e.problem` check + first-occurrence stack-trace info log (module-scoped `_hasLoggedDisconnectedFactsAbsorb` flag, test seam `_resetFactsAbsorbDisconnectedFlagForTests`); CLI help discoverability (`capture` added to `CLI_ONLY_SELF_HELP` + pre-engine-bind `--help` short-circuit in `handleCliOnly` + a `BRAIN` section in `printHelp`); binary-file guard via `detectBinaryNullByte(buf)` first-8KB NUL scan on `--file` (Buffer-read, no encoding) and `--stdin` (`readStdinBuffer` accumulator); provenance write-through — put_page accepts 3 optional params (source_kind, source_uri, ingested_via; `ingested_at` server-stamped) + trust gate (when `ctx.remote !== false` IGNORE client params, server stamps `mcp:put_page`, fail-closed) + COALESCE-preserve UPDATE semantics (omitting params on a later put_page preserves prior values; first-write-wins); `/admin/api/register-client` scopes normalization via `normalizeScopesInput(raw: unknown)` in `src/core/scope.ts` (accepts string/string[]/missing; rejects `['read write']` space-in-element shape, non-string elements, empty array, unknown scopes; deduped + sorted); brainstorm timeout surfacing via an orchestrator-level try/catch at `runBrainstorm` entry (single-point wrap covers every internal SQL site, classifies SQLSTATE 57014 via postgres.js `.code` / `.sqlState` / message fallback into `StructuredAgentError` code `brainstorm_timeout` with a hint covering all 3 PG cancel sub-causes); read-path surfaces all 4 provenance columns via `getPage` projection + `rowToPage` 3-state optional read + `Page` interface; canonical source resolver routes capture through `resolveSourceWithTier(engine, parsed.source, cwd)`; thin-client `--source` rejection (server-side OAuth client registration owns source scope); the `source_kind` taxonomy is closed (`capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler`), `--source` maps to source_id only. Tests: `test/capture-build-content.test.ts`, `test/capture-runcapture.test.ts`, `test/put-page-provenance.test.ts`, `test/scope-normalize.test.ts`, `test/cli-help-discoverability.test.ts`, `test/brainstorm-timeout.test.ts`; extended `test/facts-absorb-log.test.ts`, `test/import-file.test.ts`, `test/e2e/engine-parity.test.ts`, `test/e2e/serve-http-ingest-webhook.test.ts`. Report at `docs/v0.38-smoke-test-report.md`. Follow-ups in TODOS.md: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer, magic-byte allowlist for binary detection, `--source-kind` override flag, ingest_capture handler migration, provenance-history table, facts:absorb root-cause trace. -### BrainBench — in a sibling repo (v0.20+) +### BrainBench — in a sibling repo BrainBench — the public benchmark for personal-knowledge agent stacks — lives in [github.com/garrytan/gbrain-evals](https://github.com/garrytan/gbrain-evals). It @@ -434,12 +393,11 @@ gbrain-evals consumes: `gbrain/engine`, `gbrain/types`, `gbrain/operations`, `gbrain/extract`. Removing any of these is a breaking change for the gbrain-evals consumer. -## v0.36.1.0 Hindsight calibration wave (key files cluster) +## Hindsight calibration wave (key files cluster) The wave that taught gbrain to know how the user tends to be wrong + use that knowledge at every advice surface. Six-migration schema (v67-v72), -three new cycle phases, eight expansions, one admin tab. Plan persisted -at `~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md`. +three new cycle phases, eight expansions, one admin tab. Convention skill at `skills/conventions/calibration.md` has the agent- facing rules. @@ -471,31 +429,27 @@ unresolvable+true|false, pre-v80 NULL/NULL rows survive). - `src/core/calibration/think-ab.ts` — A/B harness. `runAbTrial` calls thinkRunner twice (baseline + with-calibration), records preference to `think_ab_results`. `buildAbReport` aggregates over a 30-day window; flags `calibration_net_negative` when n>=20 + win rate < 45% on decisive trials. - `src/core/calibration/recall-footer.ts` — formatter for the morning-pulse calibration block. Cold-brain branch when <5 resolved. Opt-in via the wiring layer. - `src/core/eval-contradictions/calibration-join.ts` — cross-reference. `tagFindingWithCalibration(finding, profile)` returns bias-tag context for contradictions matching active patterns. Returns null when profile missing (output byte-identical to the pre-calibration baseline). -- `src/core/think/prompt.ts` extension — anti-bias rewrite. `withCalibration` option on `buildThinkSystemPrompt` adds anti-bias rules. `buildCalibrationBlock()` emits the `` XML. `buildThinkUserMessage` has TWO shapes: default (question first), and with-calibration (retrieval → calibration → question) when opt-in. Wired into `runThink` via `opts.withCalibration` + `opts.calibrationHolder`. +- `src/core/think/prompt.ts` — anti-bias rewrite. `withCalibration` option on `buildThinkSystemPrompt` adds anti-bias rules. `buildCalibrationBlock()` emits the `` XML. `buildThinkUserMessage` has TWO shapes: default (question first), and with-calibration (retrieval → calibration → question) when opt-in. Wired into `runThink` via `opts.withCalibration` + `opts.calibrationHolder`. - `src/commands/calibration.ts` — CLI: `gbrain calibration` (read + print), `--regenerate`, `--undo-wave `, `ab-report`. MCP op `get_calibration_profile` (scope: read) backs the same data path. Source-scoped via `sourceScopeOpts(ctx)`. -- `src/commands/serve-http.ts` extension — three admin routes: `/admin/api/calibration/profile`, `/admin/api/calibration/charts/:type` (image/svg+xml; type in {brier-trend, domain-bars, pattern-statements, abandoned-threads}), `/admin/api/calibration/pattern/:id` (drill-down). - `src/core/owner-holder.ts` — single source of truth for "the brain owner" holder string. `DEFAULT_OWNER_HOLDER = 'self'` (matches the consolidate facts→takes writer + `docs/takes-vs-facts.md`); `resolveOwnerHolder({override, configValue})` returns override > `emotional_weight.user_holder` config > `'self'`. Consumed by the calibration_profile cycle phase, `gbrain calibration` CLI, the `get_calibration_profile` op, `think`'s calibration block, `emotional-weight`'s `DEFAULT_USER_HOLDER`, and doctor's `calibration_freshness`. Pure; unit-tested in `test/owner-holder.test.ts`. Does NOT unify owner-identity fragmentation (`self`/`brain`/`people-`) — tracked separately. -- `src/commands/takes.ts` extension — `gbrain takes revisit ` opens $EDITOR on the source page with a `` cursor marker. -- `src/commands/doctor.ts` extension — 4 checks: `abandoned_threads`, `calibration_freshness`, `grade_confidence_drift` (mitigation surface; math ships later), `voice_gate_health`. +- `src/commands/takes.ts` — `gbrain takes revisit ` opens $EDITOR on the source page with a `` cursor marker. - `admin/src/pages/Calibration.tsx` — Calibration tab. Single-column layout. `` wrapper handles `dangerouslySetInnerHTML` for the server-rendered SVG. -- `admin/src/index.css` extension — `--text-muted: #777` (WCAG AA contrast bump to ~5.5 on the #0a0a0f bg). +- `admin/src/index.css` — `--text-muted: #777` (WCAG AA contrast bump to ~5.5 on the #0a0a0f bg). - `test/fixtures/calibration/extract-takes-corpus/` — synthetic prompt-tuning corpus. Ships 5 representative pages; full 50-page + 10-page holdout generated by `gbrain calibration build-corpus`. All anonymized per CLAUDE.md placeholder list. - `scripts/check-synthetic-corpus-privacy.sh` — CI guard in `bun run verify`. Greps for explicit dollar amounts + verifies non-essay fixtures reference at least one placeholder name. - `test/regressions/v0.36.1.0-iron-rule.test.ts` — R1-R5 regression inventory; pins all 5 IRON-RULE regressions in one place for future bisects. - `DESIGN.md` — repo-root design system. Formalizes the de facto admin tokens. Calibration target for future `/plan-design-review` and `/design-review`. -## Schema Cathedral v3 (v0.40.7.0) +## Schema Cathedral v3 -The schema-pack mutation surface shipped in v0.40.7.0 as the production rebuild of -closed community PR #1321 (`@garrytan-agents`). Six new foundation modules + a -mutate skeleton + stats/sync data plane + 14 CLI verbs + 9 MCP ops + a first-class -agent skill. See `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` -for the full plan + 21 captured design decisions. +The schema-pack mutation surface (the production rebuild of community PR #1321, +credit `@garrytan-agents`): six foundation modules + a mutate skeleton + +stats/sync data plane + CLI verbs + MCP ops + a first-class agent skill. Key files (v0.40.7.0 additions): - `src/core/schema-pack/pack-lock.ts` — Atomic `O_CREAT|O_EXCL` per-pack lock. DELIBERATELY NOT the `existsSync + writeFileSync` TOCTOU shape from `src/core/page-lock.ts`. Default 60s TTL, refresh every 10s while `withPackLock(fn)` runs, `--force` semantics = "steal stale lock" NOT "skip locking." Lock path per-pack so two packs never block each other. - `src/core/schema-pack/mutate-audit.ts` — ISO-week JSONL at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Privacy-redacted: type names → sha8, prefixes → first slug segment only, matches `candidate-audit.ts` privacy posture. Logs BOTH success AND failure events so the `schema_pack_writability` doctor check has signal. `summarizeMutations()` is the cross-surface parity primitive. -- `src/core/schema-pack/registry.ts` extensions — `resolvePack` walks the `extends` chain (depth cap via `EXTENDS_DEPTH_WARN` / `EXTENDS_DEPTH_HARD_CAP`), RETAINS each ancestor manifest, materializes `borrow_from`, and composes all of it into `resolved.manifest` through `mergeInheritedManifest`. Every downstream consumer reads `resolved.manifest`, so doing the merge here is what makes inheritance visible without per-consumer wiring. `borrow_from` is selective (only the named `types` / `link_types`, and only from the target's OWN declarations), non-transitive, and fail-closed — a missing target throws `UnknownPackError` via `loadByName`, matching the extends path; an omitted category borrows none of it. The alias graph + closure hash are computed on the MERGED manifest, so a cross-pack alias cycle surfaces as `AliasCycleError` at resolve. `manifest_sha8` / `packIdentity` stay the CHILD's own bytes — a parent edit does not move the child's identity, so the invalidation path is what keeps a child honest. `invalidatePackCache(name?)` walks the extends-chain reverse-graph (editing a parent pack must not leave children stale). `tryCachedPack(name)` TTL-gated fast path: inside `STAT_TTL_MS` (default 1000ms, env `GBRAIN_PACK_STAT_TTL_MS`) returns cached without statting; outside the window it stats every TRACKED file — the extends chain PLUS every borrowed pack — and cascade-invalidates on mtime change (cross-process detection), so editing a borrowed pack invalidates its borrowers. Pinned by `test/schema-pack-registry.test.ts` + `test/schema-pack-merge.test.ts`. +- `src/core/schema-pack/registry.ts` — `resolvePack` walks the `extends` chain (depth cap via `EXTENDS_DEPTH_WARN` / `EXTENDS_DEPTH_HARD_CAP`), RETAINS each ancestor manifest, materializes `borrow_from`, and composes all of it into `resolved.manifest` through `mergeInheritedManifest`. Every downstream consumer reads `resolved.manifest`, so doing the merge here is what makes inheritance visible without per-consumer wiring. `borrow_from` is selective (only the named `types` / `link_types`, and only from the target's OWN declarations), non-transitive, and fail-closed — a missing target throws `UnknownPackError` via `loadByName`, matching the extends path; an omitted category borrows none of it. The alias graph + closure hash are computed on the MERGED manifest, so a cross-pack alias cycle surfaces as `AliasCycleError` at resolve. `manifest_sha8` / `packIdentity` stay the CHILD's own bytes — a parent edit does not move the child's identity, so the invalidation path is what keeps a child honest. `invalidatePackCache(name?)` walks the extends-chain reverse-graph (editing a parent pack must not leave children stale). `tryCachedPack(name)` TTL-gated fast path: inside `STAT_TTL_MS` (default 1000ms, env `GBRAIN_PACK_STAT_TTL_MS`) returns cached without statting; outside the window it stats every TRACKED file — the extends chain PLUS every borrowed pack — and cascade-invalidates on mtime change (cross-process detection), so editing a borrowed pack invalidates its borrowers. Pinned by `test/schema-pack-registry.test.ts` + `test/schema-pack-merge.test.ts`. - `src/core/schema-pack/merge.ts` — the pure child-wins composition helper behind `resolvePack`. `mergeInheritedManifest(ancestorsBaseFirst, child, borrowed)` returns the fully-composed manifest; precedence is child → borrowed → nearest parent … → base. SIX ingest/query-shaping fields inherit: `page_types`, `link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`, `takes_kinds`. `phases` + `calibration_domains` are DELIBERATELY child-only — they gate real cycle execution (`cycle.ts` `packDeclaresPhase`), so inheriting them would silently run phases a pack never declared; `mapping_rules`, `migration_from`, `extends`, `borrow_from`, and the identity fields are child-only too (all ride the `...child` spread). `mergePageTypes` carries the ordering contract `inferTypeFromPack` depends on (first-`path_prefix`-match-wins, array order): the BASE (root, `extends: null`) pack is the ordered foundation/tail; an override of a base type keeps the base POSITION (`Map.set` updates the value, keeps insertion order) so base's curated priority survives; a genuinely-new type from ANY non-base layer — child, borrowed, or a middle pack — is PREPENDED nearest-first, so a more-derived prefix wins regardless of chain depth. `mergeByKey` keeps the first occurrence per key walking highest-precedence-first (the order-insensitive keyed fields); `frontmatter_links` keys on `page_type\x00link_type` — a NUL, not a space, because both are unconstrained strings and a space-join would collide `{"a b","c"}` with `{"a","b c"}`. `mergeUnion` backs `takes_kinds`: UNION not replace, because the Zod default makes an omitted field indistinguishable from an explicit one — so a child can ADD kinds but CANNOT narrow below base ∪ parent. Pure + deterministic: no disk, no engine. Pinned by `test/schema-pack-merge.test.ts`. - `src/core/schema-pack/best-effort.ts` — `loadActivePackBestEffort(ctx)` returns `ResolvedPack | null`. Single source of truth for the T1.5 wiring sites. `null` means EMPTY FILTER (NOT hardcoded defaults — closes the silent-violation bug class). - `src/core/schema-pack/lint-rules.ts` — 12 pure rule functions. `withMutation`'s pre-write validation gate composes the 10 file-plane rules; the 2 DB-aware rules (`extractable_empty_corpus`, `mutation_count_anomaly`) need an engine. Single source of truth consumed by CLI lint + MCP `schema_lint` + the pre-write validation gate. New file-plane rule `link_regex_catastrophic_backtrack` — advisory ReDoS pre-screen flagging the classic nested-quantifier shapes (`(a+)+`, `(a*)*`, `(a+)*`, `(\w+)+`) in a link_type's `inference.regex` via `NESTED_QUANTIFIER_RE`. WARNING not error: a hard reject would disable the whole pack on upgrade (pages fall back to legacy typing). The runtime input-length cap in `redos-guard.ts` is the actual safety net; this rule tells the pack author to fix the pattern. @@ -504,12 +458,9 @@ Key files (v0.40.7.0 additions): - `src/core/schema-pack/mutate.ts` — 8-step `withMutation` skeleton (bundled-guard → lock → read → mutator → validate → atomic write → audit → invalidate) backs the 11 single-mutation primitives: `addTypeToPack`, `removeTypeFromPack` (with reference check), `updateTypeOnPack`, `addAliasToType`, `removeAliasFromType`, `addPrefixToType`, `removePrefixFromType`, `addLinkTypeToPack`, `removeLinkTypeFromPack`, `setExtractableOnType`, `setExpertRoutingOnType`. Each primitive's business-rule validation + transform is factored into a `build*Mutator(...)` pure `(manifest) => manifest` function shared with `applyMutationsAtomic` (the `schema_apply_mutations` batch entry point) so single-call and batched mutations can never validate differently. `applyMutationsAtomic` locks + reads the pack file ONCE, applies + lint-validates every mutation in the batch against an in-memory manifest, and calls `writePackManifest` at MOST ONCE — only after the whole batch checks out — so a batch that fails partway leaves the pack file byte-identical to its pre-batch state. Atomic single write via `.tmp + fsync + rename` — the pack file on disk is NEVER partial, for either a single mutation or a batch. Inline minimal JSON→YAML emitter so YAML packs stay YAML (does NOT preserve comments — pin pack.json if you care about layout). - `src/core/schema-pack/stats.ts` — `runStatsCore(engine, opts)` returns per-source + aggregate page counts + coverage % + `dead_prefixes` (declared prefixes with zero matching pages — agent drilldown signal). Multi-source aware (`sourceIds[]` federated, `sourceId` single, or whole-brain). PGLite + Postgres parity via `executeRaw`. Empty brain → coverage:1.0 (vacuous truth). - `src/core/schema-pack/sync.ts` — `runSyncCore(engine, opts)` chunked UPDATE in 1000-row batches per declared prefix. Concurrent writers never block on a single row >100ms. Write-side scoping via `ctx.sourceId` directly (NOT `sourceScopeOpts`, which inherits OAuth read federation). Idempotent on `--apply` re-run. -- `src/commands/schema.ts` extension — 14 CLI verbs in the dispatch table: `add-type`, `remove-type`, `update-type`, `add-alias`, `remove-alias`, `add-prefix`, `remove-prefix`, `add-link-type`, `remove-link-type`, `set-extractable`, `set-expert-routing`, `stats`, `sync`, `reload`. `withConnectedEngine` routes `loadConfig()` through the canonical `toEngineConfig()` helper and passes the complete result (`database_url` and `database_path`) to factory construction and connect, so PGLite schema commands open the configured brain. Lifecycle-grouped help text (Inspection / Activation / Authoring / Discovery+repair). Pinned by `test/schema-cli-database-path.serial.test.ts`. -- `src/core/operations.ts` extension — 9 MCP ops: `get_active_schema_pack`, `list_schema_packs`, `schema_stats`, `schema_lint`, `schema_graph`, `schema_explain_type`, `schema_review_orphans` (all read-scope, NOT localOnly), plus `schema_apply_mutations` (admin scope, NOT localOnly so remote agents can author packs over HTTPS MCP — batched, one MCP tool taking a `mutations[]` array, delegating to `applyMutationsAtomic` for a single lock + single read + single write across the whole batch; a mid-batch failure reports `mutations_applied: 0` + `pack_unchanged: true` (never a `partial_results` list — nothing is written until every mutation validates), audit log captures `actor: mcp:`) and `reload_schema_pack` (admin, NOT localOnly). Trust posture: per-call `schema_pack` opt STAYS rejected for remote callers via `op-trust-gate.ts`. -- `src/commands/whoknows.ts` + `src/core/operations.ts:find_experts` — T1.5 wiring sites. Pack-aware via `expertTypesFromPack(pack.manifest)` from `best-effort.ts`. Pack-load failure → EMPTY filter (NOT hardcoded `['person', 'company']` defaults). A `researcher` type declared `--expert` now surfaces in `whoknows` results. +- `src/commands/schema.ts` — 14 CLI verbs in the dispatch table: `add-type`, `remove-type`, `update-type`, `add-alias`, `remove-alias`, `add-prefix`, `remove-prefix`, `add-link-type`, `remove-link-type`, `set-extractable`, `set-expert-routing`, `stats`, `sync`, `reload`. `withConnectedEngine` routes `loadConfig()` through the canonical `toEngineConfig()` helper and passes the complete result (`database_url` and `database_path`) to factory construction and connect, so PGLite schema commands open the configured brain. Lifecycle-grouped help text (Inspection / Activation / Authoring / Discovery+repair). Pinned by `test/schema-cli-database-path.serial.test.ts`. - `skills/schema-author/SKILL.md` — Agent dispatcher for "evolve the schema pack." Triggers: 15+ phrasings incl. "add a page type", "my brain has untyped pages", "propose new types from my corpus", "backfill page types". Explicit Non-goals callout to `brain-taxonomist` (files one page) and `eiirp` (schema-check during iteration) so agents pick the right surface. 7-phase workflow: brain → assess → propose → apply → sync → verify → commit. Lists every gbrain schema CLI verb + every MCP op the skill uses. `brain_first: exempt` frontmatter. Required conformance sections: Contract, Anti-Patterns, Output Format. - `skills/conventions/schema-evolution.md` — Canonical convention: "when to add a type vs alias vs prefix." Decision tree: <20 pages → don't pack-codify; 20-100 → alias or narrow prefix on existing type; 100+ → first-class type. Don'ts section + "when to remove a type" + "when to commit the pack" all answered in one place. -- `skills/RESOLVER.md` + `skills/manifest.json` — schema-author wired into the dispatcher with the full functional-area trigger list (compressed routing pattern per the dispatcher convention). T1.5 wiring is partial in v0.40.7.0. Three follow-ups filed in TODOS.md under "v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+)" — enrichment-service.ts diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.md index 2595e45c5..a5853888f 100644 --- a/docs/architecture/RETRIEVAL.md +++ b/docs/architecture/RETRIEVAL.md @@ -4,14 +4,14 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e ## The four strategies in concert -1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC". +1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at acme-example?" → pages mentioning "alice-example + retrieval" even when the user never typed "acme". 2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors. 3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote. 4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can. ## Why each one alone fails -**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages. +**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in alice-example's portfolio" returns essays about portfolios, not company pages. **Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases. @@ -36,8 +36,8 @@ BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/ga Every `put_page` runs `extractEntityRefs` on the markdown body. It matches: -- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)` -- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]` +- Standard markdown links: `[Alice Example](wiki/people/alice-example)` +- Obsidian wikilinks: `[[wiki/people/alice-example|Alice Example]]` - Typed-link blockquotes: `> **Convention:** see [path](path).` Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') JOIN pages ON CONFLICT DO NOTHING RETURNING 1` (free-text-safe; the prior `unnest(${arr}::text[])` form crashed on calendar/Zoom context per gbrain#1861). The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds. @@ -46,7 +46,7 @@ Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`, ## ZeroEntropy as reranker: 60% top-1 reshuffle -v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number. +ZeroEntropy's `zerank-2` is the default reranker (on for the `balanced` and `tokenmax` mode bundles, off for `conservative`). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number. The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong. @@ -62,8 +62,9 @@ The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `Sea ## Named-thing retrieval (per-page pool + title + alias + evidence) -A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than -embedding proximity. Four layers, added after the incident in +A brain organized around *chosen names* (project codenames, place nicknames — +say a project named "Helios" whose page is also known as "the Sun Room") needs +more than embedding proximity. Four layers, added after the incident in [`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md): - **Per-page max-pool** — `searchVector` (both engines) collapses chunk-grain @@ -79,7 +80,7 @@ embedding proximity. Four layers, added after the incident in `page_aliases` table (separate from the `slug_aliases` wikilink redirect) and consulted at query time: a full normalized-query match injects/boosts the canonical page (`applyAliasHop`). The only layer that bridges true synonyms - with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill + with zero surface overlap ("the Sun Room" → the Helios page). Backfill existing pages with `gbrain reindex --aliases`. - **Evidence contract** — every result carries `evidence` (`alias_hit | exact_title_match | high_vector_match | keyword_exact | @@ -103,7 +104,7 @@ specific miss with `gbrain search diagnose "" --target `. ## Intent-aware query rewriting -`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs: +`src/core/search/query-intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs: - **Entity** queries ("who works at X?") apply a higher graph-traversal weight. - **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface. @@ -123,35 +124,61 @@ Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `con The full pipeline for a `query` op: ``` -intent classify +intent classify (query-intent.ts — deterministic, no LLM) │ ▼ -expansion (if enabled) +expansion (if enabled — tokenmax only by default) │ ▼ -hybrid search: - ├── vector (HNSW on chunk embeddings) +hybrid recall + fusion: + ├── vector (HNSW on chunk embeddings, per-page max-pool) ├── keyword (BM25 via tsvector) - ├── relational (v0.42.34.0: typed-edge recall arm — relational queries only) + ├── title-phrase arm + ├── relational (typed-edge recall arm — relational queries only) ├── source-aware re-rank (CASE in SQL) - └── RRF fusion → top 30 + └── RRF fusion → cosine re-score → post-fusion boosts + (backlink / salience / recency / graph signals / exact-match) │ ▼ -graph augment (typed-edge traversal from any seed) +graph augment (optional two-pass structural expansion — walkDepth > 0) │ ▼ -reranker (zerank-2 cross-encoder, top 30 → reordered) +deduplication (4-layer: per-page cap, Jaccard, type diversity) │ ▼ -token-budget enforcement (per mode bundle) +reranker (zerank-2 cross-encoder — balanced/tokenmax; fail-open) │ ▼ -deduplication (same slug, different chunks → keep best) +alias hop (exact alias match injects/boosts the canonical page) + │ + ▼ +evidence stamp → adaptive return (opt-in) → autocut (reranked modes) + │ + ▼ +limit slice → token-budget enforcement (per mode bundle) │ ▼ results ``` +The stage order is pinned by `hybridSearch` in `src/core/search/hybrid.ts`: +dedup runs BEFORE the reranker (so the reranker sees a diverse candidate pool, +capped by its own `topNIn`), the alias hop runs AFTER the reranker (so a query +that is a page's declared name reliably surfaces that page regardless of how +the reranker scored body chunks), and the token budget is enforced last, on +the final slice. + +### Autocut: score-discontinuity result-sizing + +Default-on for `balanced` and `tokenmax` (off for `conservative`, which has no +reranker and therefore no trustworthy cliff signal). `applyAutocut` +(`src/core/search/autocut.ts`) cuts the ranked set at the largest +cross-encoder rerank-score cliff, before the limit slice, first page only. +Never-empty failsafe (`minKeep`), no-op when fewer than 2 results carry a +finite rerank score (covers the fail-open reranker path), and alias-hop exact +matches are preserved through the cut. Knobs: per-call `SearchOpts.autocut` → +`search.autocut` / `search.autocut_jump` config → mode bundle. + Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans. ## How to verify on your own brain diff --git a/docs/architecture/brains-and-sources.md b/docs/architecture/brains-and-sources.md index 00534127f..50eaa4873 100644 --- a/docs/architecture/brains-and-sources.md +++ b/docs/architecture/brains-and-sources.md @@ -19,18 +19,18 @@ need to understand both of them, or queries misroute silently. A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase. Each brain has: - Its own `pages` table, `chunks` table, `embeddings`, etc. -- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2). +- Its own OAuth surface if served over HTTP MCP. - Its own separate lifecycle, backup, access control. Brains are enumerated by: - **host** — your default brain, configured in `~/.gbrain/config.json`. - **mounts** — additional brains registered in `~/.gbrain/mounts.json` via - `gbrain mounts add ` (v0.19+). + `gbrain mounts add `. Routing: `--brain `, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or longest-path match against registered mount paths. Falls back to `host`. -### Sources (the repo axis, v0.18.0+) +### Sources (the repo axis) A **source** is a named content repo *inside* one brain. Every `pages` row carries a `source_id`. Slugs are unique per source, not globally. @@ -142,7 +142,7 @@ Use this topology when: You're senior enough to sit across multiple teams. You maintain your personal brain (with N sources inside) AND mount several work team brains. Each team -brain is itself a multi-source brain in the v0.18.0 sense — organized +brain is itself a multi-source brain — organized internally however the team owner chose. ``` @@ -181,7 +181,7 @@ Use this topology when: - You need latent-space federation (agent decides when to query across brains), not SQL federation. -Cross-brain queries are **not deterministic** in v0.19. The agent sees the +Cross-brain queries are **not deterministic**. The agent sees the brain list and re-queries as needed. That's the feature — it keeps debugging sane and access control clean. @@ -202,6 +202,13 @@ WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)? Both axes follow the same layered pattern on purpose. If you know one, you know the other. +One addition on the source axis for remote (MCP/OAuth) callers: a client +registered with federated reads carries `ctx.auth.allowedSources` — an +ARRAY of readable sources that takes precedence over the scalar +`ctx.sourceId` on every read path (`sourceScopeOpts(ctx)` in the +operations layer). Local CLI callers never set it; the scalar chain above +is the whole story for them. + --- ## For agents reading this @@ -236,7 +243,7 @@ know the other. ## Further reading -- v0.18.0 CHANGELOG — introduced `sources` primitive. -- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`. -- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain - publisher, not just the subscriber. +- [`topologies.md`](./topologies.md) — where the DB lives (operator recipes + for each deployment shape). +- `skills/conventions/brain-routing.md` — the agent-facing decision table. +- `CHANGELOG.md` — release history for the `sources` and `mounts` primitives. diff --git a/docs/architecture/calibration-quality-gate-spec.md b/docs/architecture/calibration-quality-gate-spec.md index 54a03e414..b57489b1f 100644 --- a/docs/architecture/calibration-quality-gate-spec.md +++ b/docs/architecture/calibration-quality-gate-spec.md @@ -10,14 +10,16 @@ > v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80 > during successive master merges — v0.37.0.0's autonomous-remediation wave > claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79. -> - **Follow-up minor** (forthcoming): falsifiability + category extraction at -> `propose_takes`, SQL-side grade gate, per-category calibration scorecards, -> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation -> against the v0.36.1.0 fixtures. +> - **Follow-up minor — NEVER IMPLEMENTED.** The falsifiability + category +> extraction at `propose_takes`, SQL-side grade gate, per-category +> calibration scorecards, and pg_trgm-based proposal dedup described in the +> sections below remain UNSHIPPED design. Do not read §§1–4 as current +> behavior; only the `unresolvable` hotfix above landed. > > Preserved here per the hotfix plan's PR #1191 close protocol so the -> production context (96K-page brain, 6.8% falsifiability rate, category -> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation. +> production context (falsifiability rate + category breakdown observed on a +> large real brain) doesn't get lost in the CHANGELOG → release-notes +> condensation. ## Problem diff --git a/docs/architecture/frontmatter-scan-incremental.md b/docs/architecture/frontmatter-scan-incremental.md index 9c3c92479..de989247e 100644 --- a/docs/architecture/frontmatter-scan-incremental.md +++ b/docs/architecture/frontmatter-scan-incremental.md @@ -68,11 +68,13 @@ existing. ## Migration shape ```ts -// src/core/migrate.ts — append after the v80 entry +// src/core/migrate.ts — append after the CURRENT last entry in the +// MIGRATIONS array (take the next unused version number at implementation +// time; the numbers below are placeholders, not a reserved slot) const migrations = [ - // ...existing v1-v80... + // ...existing entries... { - version: 81, + version: NEXT_VERSION, // next unused number in the MIGRATIONS array name: 'frontmatter_scan_state', sql: ` CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...); @@ -194,7 +196,7 @@ stale data as authoritative. ``` - [ ] Implement Phase 2: DB-backed frontmatter scan state. Design lives at docs/architecture/frontmatter-scan-incremental.md. - Schema migration v81 + sync-side UPSERT + incremental scan command + New schema migration + sync-side UPSERT + incremental scan command + autopilot cycle phase + doctor reader. Two-phase rollout: ship table + writes first; flip the reader one release later. ``` diff --git a/docs/architecture/infra-layer.md b/docs/architecture/infra-layer.md index 87be52376..44a843b15 100644 --- a/docs/architecture/infra-layer.md +++ b/docs/architecture/infra-layer.md @@ -1,105 +1,33 @@ -# GBrain Infrastructure Layer +# GBrain Infrastructure Layer (orientation pointer) The shared foundation that all skills, recipes, and integrations build on. +This page is a router — the detailed, current-state references live in the +docs below (this file once carried its own copies of the pipeline and schema; +those rotted, so each concept now has exactly one home). -## Data Pipeline +## Where things live -``` -INPUT (markdown files, git repo) - ↓ -FILE RESOLUTION (local → .redirect → .supabase → error) - ↓ -MARKDOWN PARSER (gray-matter frontmatter + body) - → compiled_truth + timeline separation - ↓ -CONTENT HASH (SHA-256 idempotency check — skip if unchanged) - ↓ -CHUNKING (3 strategies, configurable) - ├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy - ├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing - └── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates - ↓ -EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions) - → batch 100, exponential backoff, non-fatal if fails - ↓ -DATABASE TRANSACTION (atomic: page + chunks + tags + version) - ↓ -SEARCH (hybrid, available immediately) -``` - -## Search Architecture - -GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search: - -``` -User Query - ↓ -EXPANSION (optional: Claude Haiku generates 2 alternative phrasings) - ↓ - ├── VECTOR SEARCH (pgvector HNSW, cosine distance) - │ → 2x limit results per query variant - │ - └── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank) - → 2x limit results - ↓ -RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly) - ↓ -4-LAYER DEDUP - ├── Best 3 chunks per page (source dedup) - ├── Jaccard similarity > 0.85 (text dedup) - ├── No type exceeds 60% (diversity) - └── Max 2 chunks per page (page cap) - ↓ -TOP N RESULTS (default 20) -``` - -## Key Components - -| File | Purpose | -|------|---------| -| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) | -| `src/core/postgres-engine.ts` | Postgres + pgvector implementation | -| `src/core/import-file.ts` | importFromFile + importFromContent pipeline | -| `src/core/sync.ts` | Git-based incremental change detection | -| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing | -| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff | -| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) | -| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection | -| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking | -| `src/core/search/hybrid.ts` | RRF merge of vector + keyword | -| `src/core/search/dedup.ts` | 4-layer result deduplication | -| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku | -| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) | -| `src/core/operations.ts` | Contract-first operation definitions (31 ops) | -| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) | - -## Schema Overview - -10 tables in Postgres: - -- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB) -- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline) -- **links** — typed edges (knows, works_at, invested_in, founded, etc.) -- **tags** — many-to-many page tagging -- **timeline_entries** — structured events (date, source, summary, detail) -- **page_versions** — snapshot history for diff/revert -- **raw_data** — sidecar JSON from external APIs (preserves provenance) -- **files** — binary attachments in storage backend -- **ingest_log** — audit trail of import operations -- **config** — brain-level settings (version, embedding model, chunk strategy) - -Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C). -Vector search uses HNSW index with cosine distance on content_chunks.embedding. +| Topic | Home | +|---|---| +| Ingest pipeline (file resolution → frontmatter parse → content-hash idempotency → chunking → embedding → atomic write) | per-file entries in [`KEY_FILES.md`](./KEY_FILES.md): `src/core/import-file.ts`, `src/core/sync.ts`, `src/core/markdown.ts`, `src/core/embedding.ts`, `src/core/chunkers/*` | +| Chunking strategies (recursive / semantic / LLM-guided) | `src/core/chunkers/{recursive,semantic,llm}.ts` entries in [`KEY_FILES.md`](./KEY_FILES.md) | +| Search pipeline (hybrid RRF, graph, reranker, autocut, dedup, budgets) | [`RETRIEVAL.md`](./RETRIEVAL.md) | +| Search modes + cost knobs | `docs/guides/search-modes.md` + the CLAUDE.md Search Mode table | +| Per-file index of `src/` (what each file does + its invariants) | [`KEY_FILES.md`](./KEY_FILES.md) | +| Schema DDL | the `MIGRATIONS` array in `src/core/migrate.ts` (source of truth) + `src/schema.sql`; per-table classification in [`system-of-record.md`](./system-of-record.md) | +| Engines (PGLite vs Postgres, parity rules) | `docs/ENGINES.md` + the engine entries in [`KEY_FILES.md`](./KEY_FILES.md) | +| Operations contract (CLI + MCP generated from one source) | `src/core/operations.ts` (100+ operations; run `gbrain --tools-json` for the live list) | +| Brains vs sources (which database vs which repo inside it) | [`brains-and-sources.md`](./brains-and-sources.md) | ## The Thin Harness Principle -GBrain is the deterministic layer. Skills and recipes are the latent space layer. +GBrain is the deterministic layer. Skills and recipes are the latent-space layer. See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full architecture philosophy. - **GBrain CLI** = thin harness (same input → same output) -- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills +- **Skills** (the bundled set routed by `skills/RESOLVER.md`) = fat skills - **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work. diff --git a/docs/architecture/lens-packs.md b/docs/architecture/lens-packs.md index 0d486f185..96406a191 100644 --- a/docs/architecture/lens-packs.md +++ b/docs/architecture/lens-packs.md @@ -1,4 +1,4 @@ -# Lens packs (v0.41.2.0) +# Lens packs Four bundled schema packs that turn the gbrain dream cycle into a multi-lens brain. Activate one with `gbrain config set schema_pack ` and the cycle @@ -7,7 +7,7 @@ picks up the pack's declared phases on the next `gbrain dream` run. ## The four packs ``` - gbrain-base (shipped v0.38) + gbrain-base ▲ │ extends ┌──────────────┼──────────────────────┐ @@ -60,37 +60,33 @@ conviction so high-stakes misses cost more). ### gbrain-engineer Bridge-only pack. Declares `learning` page type + reuses base `code`. -No new cycle phases — the daemon-side `gstack-learnings` IngestionSource -(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits +No new cycle phases — the daemon-side `gstack-learnings` IngestionSource watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits each JSONL line as a `learning` page when this pack is active. Three calibration domains: `architecture_calls` (scalar_brier), `effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier). -Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred -to v0.42+ — they'll ship when a real user authors the first one (D8). +Speculative ADR/postmortem/refactor_thesis/tech_debt types are +deferred — they'll ship when a real user authors the first one. ### gbrain-everything -Meta-pack stacking creator + investor + engineer via the v0.38 +Meta-pack stacking creator + investor + engineer via the `extends` + `borrow_from` chain. Single-active-pack constraint preserved — this IS the active pack; the registry walks extends + borrow to materialize the merged view. -**Merge contract (T20 / #1749).** `resolvePack` merges parent → child -(child-wins) for the six ingest/query-shaping fields: `page_types`, -`link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`, -and `takes_kinds` (unioned — a child cannot narrow it). `phases` and -`calibration_domains` are **NOT** inherited: they gate cycle execution, -so each pack must declare its own participation explicitly. That is why -`gbrain-everything` re-declares all its phases and all 7 -`calibration_domains` — inheritance does not carry them. +**Merge contract.** The full `extends` + `borrow_from` merge rules live in +[`schema-packs.md` § Merge contract](./schema-packs.md#merge-contract-extends--borrow_from). +The one rule that matters here: `phases` and `calibration_domains` are +**NOT** inherited (they gate cycle execution, so each pack must declare +its own participation explicitly) — which is why `gbrain-everything` +re-declares all its phases and all 7 `calibration_domains`. Activate via `gbrain config set schema_pack gbrain-everything` and calibration_profile produces all 7 domain scorecards in one JSONB. -## Calibration profile widening (T10) +## Calibration profile domains -Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a -`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared +Each declared domain produces a `{n, brier, accuracy, aggregator, page_types, extras}` entry. Four aggregator algorithms (closed enum): @@ -109,20 +105,20 @@ Domain names are OPEN (third-party packs can declare new domain labels without a gbrain release). Aggregator algorithms are CLOSED (safe SQL stays in code, validated at pack-load). -## take_domain_assignments table (T1) +## take_domain_assignments table -New JOIN table (migration v94): +JOIN table (migration v94): `take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT, source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id, -domain))`. Multi-domain assignment honest — a take about "Sequoia's -investment in Anthropic" can land in BOTH `deal_success` AND +domain))`. Multi-domain assignment honest — a take about "fund-a's +investment in acme-example" can land in BOTH `deal_success` AND `market_call` rather than being force-bucketed. ## What this enables for the user - **Atoms + concepts ship in the binary.** Your OpenClaw's parallel atom-pipeline-coordinator + atom-backfill-coordinator + concept- - synthesis crons can retire (T12 follow-up). One `gbrain dream` cron + synthesis crons can retire. One `gbrain dream` cron covers everything. - **gstack learnings reach gbrain.** Engineer-pack-active brains surface every gstack-logged learning as a queryable page within @@ -131,22 +127,22 @@ investment in Anthropic" can land in BOTH `deal_success` AND often you're wrong on deals AND market calls AND architecture AND effort estimates in one `gbrain calibration --json` call. - **Lossless OpenClaw migration.** The `markdown-greenfield` - importer (T7, mode='migration') re-ingests existing OpenClaw + importer (mode='migration') re-ingests existing OpenClaw pages with permanent slug-keyed idempotency + per-row JSONL audit + the `imported_from` marker so extract_atoms + synthesize_concepts don't re-extract already-atomized material. -## v0.41.2.1 follow-ups (filed in plan) +## Known gaps / deferred follow-ups - Per-page-type `frontmatter_validators` on PageTypeSchema so the atom_type enum (currently hardcoded in extract_atoms.ts) reads from - the active pack manifest at runtime per D11. + the active pack manifest at runtime. - 3-check quality gate (truism / punchline / entity-page reject) as a multi-pass extract_atoms refinement. - Embedding-similarity dedup in synthesize_concepts (currently exact-string concept ref match only). -- Voice gate integration for T1 Canon narratives. +- Voice gate integration for concept narratives. - op_checkpoint resumability for cross-cycle continuation in both phases. -- Parity-baseline eval gates against your OpenClaw's existing 13K atoms - + 11K concepts on a 500-page sample subset. +- Parity-baseline eval gates against a pre-existing downstream + atom/concept corpus on a sample subset. diff --git a/docs/architecture/pack-upgrade-mechanism.md b/docs/architecture/pack-upgrade-mechanism.md index fce352a21..a1dd4ca1b 100644 --- a/docs/architecture/pack-upgrade-mechanism.md +++ b/docs/architecture/pack-upgrade-mechanism.md @@ -127,10 +127,10 @@ candidate ≠ the active pack name, loads the manifest via migration_from.version)`. Returns matching packs sorted by version descending. -v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed -packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the -filesystem-scan cost needs the cache invalidation strategy from -`registry.ts`). +Successor detection covers bundled packs only. Future work: enumerate +user-installed packs at `~/.gbrain/schema-packs/*/pack.yaml` (deferred +because the filesystem-scan cost needs the cache invalidation strategy +from `registry.ts`). ## The manual_only apply policy @@ -173,8 +173,8 @@ migration_from: version: "1.x" page_types: - # Inherit gbrain-base-v2's 15 types here (or use extends to merge - # automatically once v0.43+ extends-chain composition lands) + # Inherit gbrain-base-v2's 15 types here (or declare `extends: + # gbrain-base-v2` and let the merge contract in schema-packs.md merge them) - { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true } - { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true } # ... all 13 other v2 canonicals ... @@ -222,17 +222,17 @@ Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl` identities (before + after), per-phase counts (would_apply + applied), warnings, completion timestamp. Privacy: page slugs are NOT logged in bulk (only the per-rule sample_slugs[≤10]); for forensic debugging -add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired). +a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired. ## What's NOT yet supported -- Subprocess sandbox for the publish-gate (v0.43+ TODO) +- Subprocess sandbox for the publish-gate - Per-source pack-upgrade (the handler accepts `sourceId` but `findPackSuccessors` doesn't yet pass it through) - Cross-brain federated mounts that disagree on canonical packs - Automatic rollback (today: manual SQL or `gbrain restore`) -- LLM-assisted mapping_rules codegen from production data (`gbrain - schema detect-mappings`; deferred to v0.43+) +- LLM-assisted mapping_rules codegen from production data (a proposed + `gbrain schema detect-mappings`) ## Reference @@ -242,6 +242,6 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired). - Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable` - Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS` - Handler: `src/core/schema-pack/unify-types-handler.ts` -- Migration: `src/core/migrate.ts:105` (slug_aliases table) +- Migration: the `slug_aliases` entry in `src/core/migrate.ts`'s `MIGRATIONS` array - Type taxonomy doc: `docs/architecture/type-taxonomy.md` - Skill: `skills/schema-unify/SKILL.md` diff --git a/docs/architecture/schema-packs.md b/docs/architecture/schema-packs.md index e8bf8c7ec..0dff6cac8 100644 --- a/docs/architecture/schema-packs.md +++ b/docs/architecture/schema-packs.md @@ -7,20 +7,25 @@ paths, and which link verbs connect what to what. The schema pack is the querying, or routing experts. It is the single source of truth for "what's in your brain." -The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the -user-facing reference; for implementation details see -`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in -`src/core/schema-pack/`. +This doc is the user-facing reference; for implementation details see +`docs/designs/V038_SCHEMA_PACKS.md` (the original design) and the engine +layer in `src/core/schema-pack/`. ## What ships in the box -Two bundled packs: +Seven bundled packs (`src/core/schema-pack/base/`): -- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior - byte-for-byte. Existing brains see zero behavior change after upgrade. - Covers: person, company, deal, meeting, project, place, concept, writing, - analysis, guide, hardware, architecture, etc. (the original - `ALL_PAGE_TYPES` list). +- **`gbrain-base-v2`** — the 15-type canonical taxonomy. Fresh installs + (`gbrain init`) activate this by default. See + [`type-taxonomy.md`](./type-taxonomy.md) for the full type list and the + upgrade path from `gbrain-base`. + +- **`gbrain-base`** — the original hardcoded behavior, byte-for-byte + (person, company, deal, meeting, project, place, concept, writing, + analysis, guide, hardware, architecture, etc. — the original + `ALL_PAGE_TYPES` list). Still the resolution-chain fallback (tier 7) + for brains with no pack configured anywhere, so pre-existing brains see + zero behavior change until they opt in to something newer. - **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal, @@ -32,12 +37,17 @@ Two bundled packs: gbrain schema use gbrain-recommended ``` +- **`gbrain-creator`**, **`gbrain-investor`**, **`gbrain-engineer`**, + **`gbrain-everything`** — the lens packs, which add cycle phases and + calibration domains on top of the base taxonomy. See + [`lens-packs.md`](./lens-packs.md). + Plus user-installed packs at `~/.gbrain/schema-packs//pack.yaml` that you author with `gbrain schema init` or `gbrain schema fork`. ## CLI surface -Five inspection verbs (shipped in v0.38): +Inspection verbs: ```bash gbrain schema active # show resolved pack + which tier set it @@ -47,7 +57,7 @@ gbrain schema validate # validate a manifest's shape gbrain schema use # activate a pack (writes ~/.gbrain/config.json) ``` -Eight authoring + discovery verbs (shipped in v0.39): +Authoring + discovery verbs: ```bash gbrain schema detect # propose types matching brain shape @@ -62,12 +72,12 @@ gbrain schema graph # ASCII type listing (experimental) gbrain schema lint # flag duplicates + missing prefixes gbrain schema explain # plain-English type description (experimental) gbrain schema downgrade --to

# restore previous pack (recovery) -gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry) +gbrain schema usage --since 30d # per-verb invocation counts (telemetry) ``` -The verbs marked `experimental` are demand-gated per D14: their usage is -tracked via T15's schema-events audit, and v0.40+ retro decides whether -to deprecate any that stay <5% usage. +The verbs marked `experimental` are demand-gated: usage is tracked via the +schema-events audit (`gbrain schema usage`), which informs whether +rarely-used verbs get deprecated. ## Resolution chain (7 tiers) @@ -78,10 +88,10 @@ this chain top-down. First match wins. |------|--------|-------| | 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. | | 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. | -| 3 | Per-source DB config key `schema_pack:source:` | New in v0.38. | +| 3 | Per-source DB config key `schema_pack:source:` | | | 4 | Brain-wide DB config key `schema_pack` | | | 5 | `gbrain.yml schema:` section | Repo-checked. | -| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. | +| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` (and `gbrain init`, which sets `gbrain-base-v2`) writes. | | 7 | Default: `gbrain-base` | Always present. | ## How the agent uses the active pack @@ -97,18 +107,18 @@ Every read + write path consults the active pack at runtime: - **`extract_facts`** runs only on `extractable: true` types. - **`enrichment-service`** routes person/company enrichment based on the pack's primitive declarations. -- **Search hybrid cache** (`knobsHash`) folds in pack name + version - (v0.39 T21). A cache row written under pack A is unreachable when pack +- **Search hybrid cache** (`knobsHash`) folds in pack name + version. + A cache row written under pack A is unreachable when pack B is active. Cross-pack contamination is structurally impossible. -## The magical moment (T2-T4 + T10) +## The magical moment Persona A (Notion refugee) installs gbrain, imports her exports, and the brain looks unfamiliar — the default `gbrain-base` pack expects `people/`, `companies/`, etc., but her files live under `Projects/`, `Reading/`, `Daily Notes/`. The friction signal fires in two places: -1. **Import warn (T7):** the end of `gbrain import` prints +1. **Import warn:** the end of `gbrain import` prints `[schema] X of Y pages (Z%) have no type matching the active schema pack. Run gbrain schema detect to propose a pack matching your content shape.` @@ -124,7 +134,7 @@ gbrain schema review-candidates # human gate on promotion gbrain schema review-candidates --apply Projects/ # accept ``` -The agent (via the new EIIRP skill) automates phases 1-3 of this for any +The agent (via the EIIRP skill, `skills/eiirp/SKILL.md`) automates phases 1-3 of this for any significant work session. The brain's schema becomes a living artifact the agent maintains, not a hardcoded ceremony the user authors. @@ -172,9 +182,10 @@ filing_rules: [] ## Merge contract (`extends` + `borrow_from`) +This section is the single home for the merge rules (other docs link here). `resolvePack` composes a pack against its `extends` chain (and any -`borrow_from` targets) into the `resolved.manifest` every consumer reads -(T20 / #1749). The rules: +`borrow_from` targets) into the `resolved.manifest` every consumer reads. +The rules: - **Six fields inherit, child-wins:** `page_types`, `link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`. @@ -200,9 +211,8 @@ filing_rules: [] ## Recovery + revert -The single-PR cathedral is hard to revert atomically. Per codex finding -#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore -the active-pack config field: +A pack activation is config, not code, so reverting code alone doesn't +undo it. `gbrain schema downgrade` restores the active-pack config field: ```bash gbrain schema downgrade --to gbrain-base @@ -214,19 +224,19 @@ gbrain schema downgrade 1. `git revert ` — restores the code. 2. `gbrain schema downgrade --to gbrain-base` — restores config. -3. (Optional) `gbrain purge-deleted --older-than 0h` — drops - v0.39-typed pages that no longer have a matching type in the active +3. (Optional) `gbrain pages purge-deleted --older-than 0h` — hard-deletes + soft-deleted pages that no longer have a matching type in the active pack. The cache + eval rows that pack-aware code wrote are isolated by the -`knobsHash` pack-folding (T21) — they become unreachable under the +`knobsHash` pack-folding — they become unreachable under the restored pack so no eviction is needed. ## Distribution -`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as -`.gbrain-skillpack` tarballs (T14 artifact abstraction). The -discriminator is `api_version` in the manifest: +`.gbrain-schema` tarballs ride the same distribution pipeline as +`.gbrain-skillpack` tarballs. The discriminator is `api_version` in the +manifest: - `gbrain-schema-pack-v1` → schemapack - `gbrain-skillpack-v1` → skillpack @@ -237,22 +247,17 @@ respectively. Publication to the public registries (`garrytan/gbrain-schema-registry`, `garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR -workflow as v0.37 skillpack publishing. +workflow as skillpack publishing. -## What's deferred to v0.40+ +## Known limits / deferred work - **Per-source pack federation across mounts.** A query crossing multiple - sources currently rejects with `permission_denied` when those sources - have divergent active packs (T19 + codex finding #2). The v0.40+ work - computes a true per-source closure via the existing - `buildSourceClosureCte` engine surface. -- **`extends` chain semver compatibility checks** between pack versions. -- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack - can declare "I work best with these primitives present in your pack." -- **Live schema migration helpers** — when you add a type, auto-suggest - backfill of existing pages. -- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the - full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+ - retro reads T23 usage telemetry to decide which to deprecate. + sources rejects with `permission_denied` when those sources have + divergent active packs (`src/core/schema-pack/op-trust-gate.ts`). A true + per-source closure via the existing `buildSourceClosureCte` engine + surface remains future work. +- **Pack version upgrades** (e.g. `gbrain-base` → `gbrain-base-v2`) are + handled by the successor-detection + unify-types mechanism — see + [`pack-upgrade-mechanism.md`](./pack-upgrade-mechanism.md). -See `TODOS.md` v0.40+ section for the full deferred list. +The live deferred list is in `TODOS.md`. diff --git a/docs/architecture/serve-sync-concurrency.md b/docs/architecture/serve-sync-concurrency.md index 567fa6449..13c21bf60 100644 --- a/docs/architecture/serve-sync-concurrency.md +++ b/docs/architecture/serve-sync-concurrency.md @@ -52,3 +52,13 @@ gbrain sync --no-schema-pack --no-pull --no-embed --yes `gbrain schema lint` flags the classic nested-quantifier ReDoS shapes (`(a+)+`, `(a*)*`, …) in pack regexes as warnings. + +The manual diagnosis above has an automated cousin: the progress-aware stall +watchdog. If the import drain makes no forward progress for +`GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900; keyed on file-import +progress, not the lock heartbeat), the run aborts with +`reason: 'stall_timeout'` and releases the per-source lock so the next +`gbrain sync` resumes from the checkpoint. It fires BETWEEN files — a hang +inside one file's import runs until the wall-clock hard deadline. `0` +disables it. The full sync-resumability knob table lives in CLAUDE.md +("Sync resumability + lock tuning"). diff --git a/docs/architecture/system-of-record.md b/docs/architecture/system-of-record.md index 283c7eb4f..d10bfc053 100644 --- a/docs/architecture/system-of-record.md +++ b/docs/architecture/system-of-record.md @@ -88,6 +88,7 @@ the repo. The architectural rule still holds — these aren't | `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. | | `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. | | `gbrain_cycle_locks` / migration ledger | Infrastructure. | +| `op_checkpoint_paths` | Sync-resume checkpoint. Append-only progress banking; a completed sync makes it irrelevant. | | `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). | A new derived table that holds user-knowledge MUST land FS-first. @@ -189,9 +190,6 @@ reconciler / migration layer without the explicit allow-list comment. ## Related -- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md` - — the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1 - and round 2 finds) - `skills/migrations/v0.32.2.md` — the agent-facing migration guide - `CHANGELOG.md` v0.32.2 entry — the release manifesto - `scripts/check-system-of-record.sh` — the CI gate that enforces diff --git a/docs/architecture/thin-client.md b/docs/architecture/thin-client.md index 781e80c25..1656ced16 100644 --- a/docs/architecture/thin-client.md +++ b/docs/architecture/thin-client.md @@ -3,68 +3,62 @@ On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants only; release history lives in `CHANGELOG.md` + git. -`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local -brain content, just an OAuth client pointing at a remote `gbrain serve --http`. -v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25 -silently fell through to `connectEngine()` and opened the empty local PGLite, -returning "No results." against a populated remote brain. v0.31.1 fixes the -silent-empty-results bug class for every operation surface. +`gbrain init --mcp-only` sets up a thin-client install: no local brain content, +just an OAuth client pointing at a remote `gbrain serve --http`. Every operation +surface routes through the remote brain — a thin-client install never opens the +empty local PGLite, so a populated remote brain can't silently return +"No results." Local-only commands refuse with a pinpoint hint instead of +falling through. -Key files: +Key files (per-file detail lives in each file's `KEY_FILES.md` entry; this doc +carries the routing-seam picture): -- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no +- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (no parallel `src/core/thin-client/` module; routing is a ~80-line conditional in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine` so thin-client installs never open the empty PGLite. localOnly ops on thin-client refuse via `refuseThinClient` (with pinpoint hint table - `THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort` + `THIN_CLIENT_REFUSE_HINTS`, which covers the full DB-bound command surface — + sync, embed, extract, migrate, enrich, dream, jobs, sources, pages, files, + eval, code-*, and more). Banner via `printIdentityBannerBestEffort` before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`, non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason` - for canned, actionable error messages. ENG-2 renderer parity: local-engine + for canned, actionable error messages. Renderer parity: the local-engine path runs `JSON.parse(JSON.stringify(result))` so renderers see the same - shape on both paths (kills Date/bigint/Buffer drift class). -- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`. - Hardened in v0.31.1 (CDX-4): all transport errors normalized to - `RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions - {timeoutMs, signal}`; `buildAbortController` composes external signal with - timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind` - ('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code` - field carrying server-supplied error codes (e.g. `missing_scope`). - `extractToolErrorCode` parses JSON envelopes first, falls back to substring - detection for legacy server messages. `unpackToolResult(res)` unchanged - (parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape. -- `src/core/cli-options.ts` — `parseGlobalFlags` adds `--timeout=Ns` (accepts - `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s - for most ops, 180s for `think`). `parseTimeout(s)` exported helper. -- `src/core/doctor-remote.ts` — `gbrain remote doctor` adds the - `oauth_client_scopes_probe` check (CDX-5). Probes the read tier via - `get_brain_identity` and admin tier via `get_health`; reports per-tier + shape on both paths (kills the Date/bigint/Buffer drift class). +- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`, + the transport under the routing seam. All transport errors normalize to + `RemoteMcpError` via the `toRemoteMcpError` funnel, with a stable + `RemoteMcpErrorReason` union the dispatcher's `never` switch keys off. + Full symbol-level detail: the `src/core/mcp-client.ts` entry in + [`KEY_FILES.md`](./KEY_FILES.md). +- `src/core/cli-options.ts` — `parseGlobalFlags` supports `--timeout=Ns` + (accepts `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command + default (30s for most ops, 180s for `think`). `parseTimeout(s)` exported + helper. +- `src/core/doctor-remote.ts` — `gbrain remote doctor` includes the + `oauth_client_scopes_probe` check. Probes the read tier via + `get_brain_identity` and the admin tier via `get_health`; reports per-tier status with pinpoint remediation when admin is missing. `buildScopeCheck` + `ScopeProbeResult` exported for test access. Skippable via `GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC initialize level only (MCP SDK Client hangs on shape mismatch). -- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests. -- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries. -- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller). -- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality` → `'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error. -- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND). -- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex). -- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed. -- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on). -- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command). -- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows. -- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)` — `date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query. - `src/core/operations.ts` — `get_brain_identity` op (read scope, no params, banner-only): cheap counter packet `{version, engine, page_count, chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses - `engine.getStats()`; banner's 60s client-side TTL bounds frequency to - ≤1/60s per CLI process (well below the Fly.io health-check cadence that - motivated the original `getStats` cost warning). + `engine.getStats()`; the banner's 60s client-side TTL bounds frequency to + ≤1/60s per CLI process. - `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command thin-client routing branches. These commands bypass the operation-layer dispatch in cli.ts (call `engine.foo()` directly), so each gets its own `if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags to op params. `think` is a special case: the server's `think` op - intentionally disables `--save`/`--take` for remote callers - (operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns - loudly when those flags are set. + intentionally disables `--save`/`--take` for remote callers (the + `safeSave`/`safeTake` trust-boundary gate in the `think` handler in + `operations.ts`); thin-client `think` warns loudly when those flags are set. + +Cross-modal search files (image query, SSRF-guarded image loading, spend +tracking, multimodal reindex) are indexed per-file in +[`KEY_FILES.md`](./KEY_FILES.md) and described behaviorally in +[`RETRIEVAL.md`](./RETRIEVAL.md) — they are not part of the thin-client +routing seam. diff --git a/docs/architecture/topologies.md b/docs/architecture/topologies.md index 9c564b458..0c9389daa 100644 --- a/docs/architecture/topologies.md +++ b/docs/architecture/topologies.md @@ -108,11 +108,13 @@ instead of a local DB connection: } ``` -The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`, -`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`, -`integrity`, `serve`) on a thin-client install with a clear error pointing -at the remote host. `gbrain doctor` runs a dedicated thin-client check set -(OAuth discovery, token round-trip, MCP smoke). +The CLI dispatch guard refuses every DB-bound command (`sync`, `embed`, +`extract`, `migrate`, `serve`, `enrich`, `jobs`, `sources`, `pages`, +`files`, `eval`, and the rest of the local-only surface — the full hint +table is `THIN_CLIENT_REFUSE_HINTS` in `src/cli.ts`) on a thin-client +install with a clear error pointing at the remote host. `gbrain doctor` +runs a dedicated thin-client check set (OAuth discovery, token round-trip, +MCP smoke). See [`thin-client.md`](./thin-client.md) for the routing seam. ### Setup @@ -394,6 +396,9 @@ simultaneously — that's by design. ## See also +- `docs/guides/bootstrap.md` — `gbrain bootstrap`, the paved-road paste-in + install for Topology 1 with a desktop coding agent (interview, hooks, + MCP registration, verify). - `docs/architecture/brains-and-sources.md` — in-brain organization (brains vs sources axes). - `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup. diff --git a/docs/architecture/type-taxonomy.md b/docs/architecture/type-taxonomy.md index 138d7b52f..2ea9e20ac 100644 --- a/docs/architecture/type-taxonomy.md +++ b/docs/architecture/type-taxonomy.md @@ -1,7 +1,7 @@ -# Type Taxonomy (v0.41.22: gbrain-base-v2) +# Type Taxonomy (gbrain-base-v2) -> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor -> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs +> The 14-canonical-type DRY/MECE taxonomy. Predecessor +> `gbrain-base` (24 types) stays bundled for back-compat; fresh installs > default to `gbrain-base-v2`. ## Why @@ -79,7 +79,7 @@ gbrain jobs submit unify-types \ # PROTECTED + manual_only --params '{"target_pack":"gbrain-base-v2","apply":true}' # omit "apply":true → dry-run (default) ↓ -Handler runs 4 phases: +Handler runs 8 phases: ┌─────────────────────────────────────┐ │ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL) ├─────────────────────────────────────┤ @@ -165,8 +165,10 @@ explicitly disambiguated this as canonical, so it should outrank fuzzy matches that hit aliases by accident." `SearchResult.alias_resolved_boost` is stamped on touched results for -`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to -invalidate pre-v0.42 cache rows that don't reflect the new stage. +`--explain` formatter visibility. The stage participates in the search +cache key (`KNOBS_HASH_VERSION` in `src/core/search/mode.ts` is the +single source of truth for the current cache-key version), so cache rows +written before the stage existed are unreachable. ## Reference @@ -176,4 +178,3 @@ invalidate pre-v0.42 cache rows that don't reflect the new stage. - Migration handler: `src/core/schema-pack/unify-types-handler.ts` - Onboard checks: `src/core/onboard/checks.ts` - Skill: `skills/schema-unify/SKILL.md` -- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md` diff --git a/docs/contradictions.md b/docs/contradictions.md index af02d7771..46ca5012f 100644 --- a/docs/contradictions.md +++ b/docs/contradictions.md @@ -1,4 +1,4 @@ -# gbrain eval suspected-contradictions (v0.32.6) +# gbrain eval suspected-contradictions The contradiction probe samples retrieval results, asks an LLM judge whether any pair contradicts on a factual claim relevant to the user's query, and @@ -142,25 +142,27 @@ pay near-zero on re-runs (until you bump PROMPT_VERSION). gate makes accidental private-data commits hard, but the operator MUST inspect every redaction before commit. +## Temporal axis + +The judge distinguishes real contradictions from legitimate change-over-time. +The verdict enum has six members (`no_contradiction | contradiction | +temporal_supersession | temporal_regression | temporal_evolution | +negation_artifact`), and `pages.effective_date` is threaded into the judge +prompt so the probe doesn't cry wolf on facts that simply changed. + +The trajectory substrate builds on the same signal: +`gbrain eval trajectory ` shows the chronological typed-claim +history with regressions flagged inline; `gbrain founder scorecard +` rolls up four signals (accuracy, consistency, growth +direction, red flags) into a stable JSON contract. MCP op +`find_trajectory` (read scope, visibility-filtered for remote callers) +exposes the same data to agents. The probe's `temporal_supersession` +verdict and the consolidate phase's `valid_until` writeback both +preserve the `auto-supersession.ts` "NEVER auto-applies" invariant +— the probe only emits paste-ready commands; only `consolidate` +writes `valid_until` (a grep guard pins this). + ## See also -- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md` -- CHANGELOG: `## [0.32.6]` entry covers the whole release. - Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence + trend-tracking workflow. -- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a - six-member verdict enum (`no_contradiction | contradiction | - temporal_supersession | temporal_regression | temporal_evolution | - negation_artifact`) and threaded `pages.effective_date` into the judge - prompt so the probe stops crying wolf on legitimate change-over-time. - v0.35.7 lands the trajectory substrate the probe pointed at: - `gbrain eval trajectory ` shows the chronological typed-claim - history with regressions flagged inline; `gbrain founder scorecard - ` rolls up four signals (accuracy, consistency, growth - direction, red flags) into a stable JSON contract. MCP op - `find_trajectory` (read scope, visibility-filtered for remote callers) - exposes the same data to agents. The probe's `temporal_supersession` - verdict and the consolidate phase's `valid_until` writeback both - preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant - — the probe still emits paste-ready commands, only `consolidate` - writes `valid_until` (R1+R8 grep guard pins this). diff --git a/docs/designs/AGENT_BOOTSTRAP_DESIGN.md b/docs/designs/AGENT_BOOTSTRAP_DESIGN.md index dbae2b8d7..d1cf7ae78 100644 --- a/docs/designs/AGENT_BOOTSTRAP_DESIGN.md +++ b/docs/designs/AGENT_BOOTSTRAP_DESIGN.md @@ -68,7 +68,8 @@ open >10 days from first code commit) 0. **Spike + quota gate** (manual, gates door-1 ship; per-harness quota measured; a p90 day must fit ≤10% of weekly subscription quota or schedule scope is cut). 1. **Shared body + engine machinery:** `gbrain bootstrap` family, templates, format - spec, secret-scan-gated persistence, verify. + spec, secret-scan-gated persistence, verify, uninstall (v1 via CEO-review + expansion; receipt-keyed scope per the PLAN's CX2-12). 2. **Codex door ships first** (runbook variant + approvals preflight + capability probe; CLI path not spike-gated). 3. **Claude Code door:** hooks, IPC turn_context, transcript ingestion, greeting @@ -111,8 +112,11 @@ Paste block + tag-pinned runbook (`BOOTSTRAP_FOR_AGENTS.md`, fetched at the `latest-stable` ref — advanced by the release job only after assets publish, so published copies never rot); optional GitHub template repo (generated at release from the same renderer); binary via `bun install -g github:garrytan/gbrain#latest-stable` -(never npm). The paste block lives in the README's "Quick start: Claude Code or -Codex" section; `INSTALL_FOR_AGENTS.md` remains the paste path for agent platforms. +(never npm). The paste block lives in the README's `## Install` section, as +per-harness subsections ordered "For Codex — the recommended first step" → "For +Claude Code" → "For OpenClaw or Hermes" (the 2026-08-09 ordering decision, recorded +in the PLAN's artifact table). `INSTALL_FOR_AGENTS.md` remains the paste path for +agent platforms and lives inside the OpenClaw/Hermes subsection. ## Threat model (v1 summary) diff --git a/docs/designs/AGENT_BOOTSTRAP_PLAN.md b/docs/designs/AGENT_BOOTSTRAP_PLAN.md index a2d4d3722..2df125dc5 100644 --- a/docs/designs/AGENT_BOOTSTRAP_PLAN.md +++ b/docs/designs/AGENT_BOOTSTRAP_PLAN.md @@ -14,6 +14,36 @@ below with their finding IDs. 0 unresolved decisions. --- +## As-shipped deltas (read this first — where the code moved after the plan froze) + +This plan is layered: later absorption sections (the post-design-review deltas, the +CX2 series) override earlier prose, and THIS section overrides everything below it. +The shipped implementation matches the plan except for these deltas: + +1. **Verify runs LAST, not before host registration.** [CX2-5]'s determinism goal + survived, but the shipped phase order (single TS source: + `src/core/bootstrap/status.ts` `PHASES`) is + preflight → engine → interview → render → skills → wire → repo → **verify**, + and verify runs in-process on the caller-held engine, calling + `runMaintenanceSweep` directly — no transient serve. It works pre-registration + AND as the weekly re-run (`src/core/bootstrap/verify.ts`). +2. **Uninstall scope: [CX2-12] wins over the CEO-expansion bullet.** `~/.gbrain` is + NEVER deleted wholesale — only receipt-enumerated bootstrap-created state + (`src/core/bootstrap/uninstall.ts`). +3. **Module naming/layout:** `private-repo.ts` shipped as `repo.ts`; additional + shipped modules the artifact table doesn't list: `attach.ts, assets.ts, + format.ts, host-specs.ts, hooks.ts, lock.ts, status.ts, template-repo.ts, + uninstall.ts`. +4. **Templates layout:** all bootstrap templates live under `templates/bootstrap/` + (not at `templates/` root). +5. **Test filenames:** `test/hook-command.serial.test.ts` and + `test/e2e/bootstrap-*.serial.test.ts` — the `.serial` variants the plan's own + [A7] mandated; the artifact table predates that. +6. **README ordering:** the D5 placement was superseded by the 2026-08-09 user + decision — per-harness `## Install` sections ordered Codex → Claude Code → + OpenClaw/Hermes, with `INSTALL_FOR_AGENTS.md` living inside the OpenClaw/Hermes + section (annotated in the artifact table; the D5 prose at the bottom is stale). + ## Post-design-review deltas (2026-08-07, /office-hours APPROVED — these override below) Product: **"GBrain for Codex" + "GBrain for Claude Code"** (names contingent on @@ -90,7 +120,9 @@ ChatGPT-app user). CLIs come along via shared machinery. kept in sync with templates/ by extending scripts/check-bootstrap-templates.sh to diff the template repo content. Build order 2. - **`gbrain bootstrap uninstall`** in v1 (was fast-follow): removes MCP registration + - hooks + `~/.gbrain` (confirm-gated), leaves the repo ("the body remains yours"). + hooks + bootstrap-created state (confirm-gated), leaves the repo ("the body remains + yours"). [Scope superseded by CX2-12 + as-shipped delta 2: `~/.gbrain` is never + deleted wholesale — only receipt-enumerated bootstrap-created state.] - **Docker cold-machine e2e (offline parts) in CI** in v1: networkless read-only container running interview → render → verify with fake gh (codex-as-agent tests/docker shape). The full networked paste flow stays a fast-follow (flake). @@ -269,6 +301,8 @@ ChatGPT-app user). CLIs come along via shared machinery. local-only CLI entry (`gbrain sweep --once`, CLI_ONLY, never over MCP), and `bootstrap verify` runs BEFORE host registration on its own transient serve/engine: write via op → `sweep --once` → edge query. No timing nondeterminism. + [Sequencing superseded by as-shipped delta 1: verify shipped as the LAST phase, + in-process on the caller-held engine; the determinism goal is unchanged.] - [CX2-6 P1] **Cross-platform lock replaces flock dependence:** flock(1) absent ⇒ locking silently disabled (brain-repo-durability.ts:137) — macOS is the v1 target. One cross-platform lock (atomic mkdir/lockfile with PID+age+token semantics) spans @@ -767,7 +801,8 @@ settings.local.json + config.toml writers (single module owns each host format). your data). The routing-table-size concern is mitigated by frontmatter-trigger routing (authoritative since v0.36) and noted for a future curated-profile fast-follow if dispatch accuracy suffers in practice. -- **D5 = Codex/Claude-Code-scoped placement.** This is NOT the new headline install — +- **D5 = Codex/Claude-Code-scoped placement.** [Superseded by the 2026-08-09 user + decision — see as-shipped delta 6 and the artifact table's README row.] This is NOT the new headline install — most users still use GBrain with OpenClaw/Hermes, so `INSTALL_FOR_AGENTS.md` remains the primary paste path at the top of the README. The bootstrap paste block becomes the flagship "For Codex" / "For Claude Code" README sections, ahead of the OpenClaw/Hermes path at equal weight (and diff --git a/docs/designs/AGENT_BOOTSTRAP_SPIKE.md b/docs/designs/AGENT_BOOTSTRAP_SPIKE.md index 05d10b742..07662c03c 100644 --- a/docs/designs/AGENT_BOOTSTRAP_SPIKE.md +++ b/docs/designs/AGENT_BOOTSTRAP_SPIKE.md @@ -57,3 +57,7 @@ A filled copy of this doc committed as `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` (scrubbed: no real names beyond the maintainer, no account identifiers), plus the gate decision recorded in the design doc: door-1 ships full / ships as documented beta / schedule scope cut per quota. + +**Gate status:** not yet run — no `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` is committed, +so no gate decision is recorded and door 1 has not been promoted past the +documented-beta bar by this instrument. Update this line when the results land. diff --git a/docs/eval-bench.md b/docs/eval-bench.md index 85cf9f8ab..f1cd627e6 100644 --- a/docs/eval-bench.md +++ b/docs/eval-bench.md @@ -8,11 +8,10 @@ For the **NDJSON wire format** consumed by gbrain-evals, see [`eval-capture.md`](./eval-capture.md). This doc is the human dev loop that lives on top of that format. -## v0.41 update — the LOOP is now real +## The eval gate loop -Before v0.41, you could capture eval rows and replay them but nothing -stitched them into a gate. `gbrain bench publish` + `gbrain eval gate` -close the loop. Two gates: +`gbrain bench publish` + `gbrain eval gate` stitch captured eval rows into +a pass/fail gate. Two gates: - **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline you captured against your current brain. Catches: "did my refactor break @@ -39,7 +38,7 @@ gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/person gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson ``` -### Privacy posture (D9) +### Privacy posture **Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real user captures stay local in `~/.gbrain/baselines/`. The boundary is @@ -131,14 +130,9 @@ gbrain query "anything" >/dev/null psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0 ``` -To override (force on/off regardless of env var), edit `~/.gbrain/config.json`: - -```json -{"eval": {"capture": true}} // force on -{"eval": {"capture": false}} // force off -``` - -Explicit config beats the env var both directions. +The full on/off resolution order (config beats env var, both directions) is +documented once in [`eval-capture.md`](./eval-capture.md) — that file is the +capture contract. ## The 4-command loop @@ -205,7 +199,7 @@ retrieval, and which queries did it move most?" For a third evaluation axis — public benchmark, ground-truth labels, full question-answer pipeline (not just retrieval) — `gbrain eval longmemeval -` (v0.28.8) runs the LongMemEval benchmark against gbrain's +` runs the LongMemEval benchmark against gbrain's hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack imported, the question asked, the hypothesis emitted as JSONL — exactly the shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is @@ -337,7 +331,7 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune | `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields | | Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured | -## Public benchmarks: LongMemEval (v0.28.8) +## Public benchmarks: LongMemEval `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark directly against gbrain's hybrid retrieval. Different evaluation @@ -398,7 +392,7 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the 500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and LLM latency. -## Measuring brain consistency over time (v0.32.6) +## Measuring brain consistency over time `gbrain eval suspected-contradictions` is a complementary measurement instrument: it samples retrieval results for unmarked semantic @@ -435,20 +429,19 @@ commands per high-severity finding. - CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing decision criteria gated on Wilson CI lower-bound. -## v0.40.1.0 Track D — Eval infrastructure +## Eval infrastructure: by-type breakdowns, the hermetic gate, batch scoring -Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section -covers the dev loop that uses them and the gates they enforce. +Three further eval surfaces, and the dev loop that uses them. ### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown -LongMemEval has always computed per-question-type recall internally; v0.40.1.0 -surfaces it in machine-readable form. Two additive changes: +LongMemEval computes per-question-type recall internally, and surfaces it in +machine-readable form: -1. Every per-question JSONL row now includes a `question: string` field so the +1. Every per-question JSONL row includes a `question: string` field so the `gbrain eval cross-modal --batch` consumer (below) can read it without joining back against the source dataset. -2. New `--by-type` flag emits a final aggregate line keyed by `question_type`: +2. The `--by-type` flag emits a final aggregate line keyed by `question_type`: ```json {"schema_version": 1, "kind": "by_type_summary", @@ -480,11 +473,11 @@ echo "exit=$?" # 1 if any type fell below 0.80 ### Hermetic retrieval gate — `test/eval-replay-gate.test.ts` -The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/` -silently regress retrieval." Replaces the original "replay against captured -eval_candidates" design (which Codex caught as non-functional in CI — see -the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred -real-query version). +The structural fix for "PRs touching `src/core/search/` silently regress +retrieval." A "replay against captured eval_candidates" design can't work in +CI (CI has no captured production queries), so the gate is hermetic; see the +`contributor-mode CI capture` TODO in `TODOS.md` for the deferred +real-query version. How it works: - Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json` @@ -499,7 +492,7 @@ How it works: - Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it runs on every PR via `bun test`, NOT in the E2E fixed-file workflow. -#### Refreshing the qrels fixture (the `Why:` discipline, D4) +#### Refreshing the qrels fixture (the `Why:` discipline) When CI fails because a legitimate ranking change moved expected slugs, the fix is to edit `qrels-search.json` directly. **Always include a `Why:` line diff --git a/docs/eval-takes-quality.md b/docs/eval-takes-quality.md index 214b9d46e..215c394cd 100644 --- a/docs/eval-takes-quality.md +++ b/docs/eval-takes-quality.md @@ -28,7 +28,7 @@ receipt file from disk and re-renders it. The other modes need the brain. |---|---|---| | `--limit N` | 100 | Random sample of N takes from the brain. | | `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. | -| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). | +| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud rather than silently blowing the budget. | | `--source db|fs` | `db` | `fs` is reserved for v0.33+. | | `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. | | `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. | @@ -73,8 +73,8 @@ receipt file from disk and re-renders it. The other modes need the brain. - `schema_version` — locks the contract. Adding optional fields is additive and compatible. Renaming, removing, or changing semantics bumps the version. -- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch - (codex review #3). When the rubric definition changes, both fields update, +- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch. + When the rubric definition changes, both fields update, and trend mode groups runs accordingly so a stricter rubric doesn't silently look like a quality drop. - `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge @@ -83,7 +83,7 @@ receipt file from disk and re-renders it. The other modes need the brain. models in `--models` doesn't change the sha (sort is stable). - `successes_per_cycle` — count of contributing models per cycle. A model contributes when (a) its JSON parsed AND (b) every declared rubric dim - has a finite score (codex review #5 — missing-dim drops the contribution). + has a finite score (a missing dim drops the whole contribution). - `verdict` — `pass` if every dim mean >= 7 AND every dim min across contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than 2/3 models contributed complete scores. @@ -93,11 +93,11 @@ receipt file from disk and re-renders it. The other modes need the brain. ## Receipt persistence -Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per -codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality----.json` +Receipts persist to **`eval_takes_quality_runs`** (the DB is authoritative) +AND to disk at `~/.gbrain/eval-receipts/takes-quality----.json` as a best-effort artifact. The DB row carries the full receipt JSON in the `receipt_json` JSONB column, so when the disk artifact is gone, `replay` -can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring). +can still reconstruct via `loadReceiptFromDb`. The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent. diff --git a/docs/eval/SEARCH_MODE_METHODOLOGY.md b/docs/eval/SEARCH_MODE_METHODOLOGY.md index 6093a4770..238b7da5b 100644 --- a/docs/eval/SEARCH_MODE_METHODOLOGY.md +++ b/docs/eval/SEARCH_MODE_METHODOLOGY.md @@ -1,6 +1,6 @@ # Search Mode Evaluation Methodology -_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._ +_How gbrain measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible — pinned datasets, recorded seeds, and the exact run commands below._ ## 1. What this measures and what it doesn't @@ -21,14 +21,15 @@ If you want to know how a mode behaves on YOUR brain, run `gbrain search stats - - **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run. - **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run. -No private brain content is used in any reported result. The committed NDJSON dumps under `/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs. +No private brain content is used in any reported result. The NDJSON run records under `/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs. ## 3. Sample selection - **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record. - **No per-question curation.** Splits are taken whole; no question is filtered for reporting. -- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable. -- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps. +- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode bundle is the only independent variable. A mode Δ therefore measures the joint effect of every knob the bundles differ on — today that's `tokenBudget`, `expansion`, `relationalRetrieval` (the typed-edge fourth recall arm, ON for balanced/tokenmax, OFF for conservative), and `searchLimit`; the canonical diff is `MODE_BUNDLES` in `src/core/search/mode.ts`. +- **Cache comparability across upgrades.** The query cache keys on a versioned knobs hash (`KNOBS_HASH_VERSION` in `mode.ts`) that folds in the active knob set + embedding column/provider, so one mode's cached results can't be served to another mode's queries — and a version bump makes prior rows unreachable (one-time miss spike). Cross-run comparisons that straddle a knobs-hash bump see a cold cache on the first re-run. +- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from a run's `--output` dumps. ## 4. Run procedure @@ -46,14 +47,14 @@ gbrain eval run-all \ --limit 500 \ --budget-usd-retrieval 5 \ --budget-usd-answer 20 \ - --output docs/eval/results/v0.32.3/ + --output docs/eval/results// # Render the comparison. -gbrain eval compare --md > docs/eval/results/v0.32.3/README.md -gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json +gbrain eval compare --md > docs/eval/results//README.md +gbrain eval compare --json > docs/eval/results//comparison.json ``` -The orchestrator writes per-run records to `/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. +The orchestrator writes per-run records to `/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. When a release publishes eval numbers, the `--output` dumps under `docs/eval/results//` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. **No dumps are committed in the repo right now** — reproduce by running the commands above; determinism (§3) means your re-run matches the reported orderings. ## 5. Threats to validity @@ -68,7 +69,7 @@ Honest list. We name what would let a critic dismiss the numbers. ## 6. Per-question raw outputs -Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version. +Every reported metric is reproducible from the NDJSON dumps a run writes to its `--output` directory (`docs/eval/results//` when a release publishes numbers; none are committed right now — see §4). The commit SHA in the methodology footer pins the code version. **Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule: diff --git a/docs/guides/agent-to-gbrain.md b/docs/guides/agent-to-gbrain.md index 8454e1638..679944fe8 100644 --- a/docs/guides/agent-to-gbrain.md +++ b/docs/guides/agent-to-gbrain.md @@ -12,13 +12,13 @@ surfaces**, and which one you pick depends on the operation. │ gbrain process │ │ │ Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │ - openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │ - │ │ (HTTP + OAuth) │ │ admin ops │ │ + openclaw, fork) ────┼──▶ MCP ops surface │ │ local-only │ │ + │ │ (HTTP + OAuth) │ │ commands │ │ │ │ │ │ │ │ │ │ search, query, │ │ sync, embed, │ │ - │ │ put_page, │ │ dream, doctor,│ │ - │ │ get_page, │ │ autopilot, │ │ - │ │ find_experts, │ │ init, secrets │ │ + │ │ put_page, │ │ extract, │ │ + │ │ get_page, │ │ dream, │ │ + │ │ find_experts, │ │ enrich, ... │ │ │ │ ... │ │ │ │ │ └──────────────────┘ └────────────────┘ │ │ ▲ ▲ │ @@ -26,7 +26,7 @@ surfaces**, and which one you pick depends on the operation. │ │ │ │ │ thin-client OAuth shell-job `inherit:`│ │ (preferred for (only path for │ - │ MCP-equivalent ops) localOnly ops) │ + │ MCP-equivalent ops) local-only work) │ └─────────────────────────────────────────────┘ ``` @@ -45,7 +45,7 @@ the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset The host runs gbrain as a long-lived HTTP server: ```bash -GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131 +gbrain serve --http --port 3131 ``` The agent registers as an OAuth client (one-time): @@ -75,22 +75,27 @@ commands through the configured remote MCP. The agent can call agent to a specific source within a federated brain. - One audit surface (`mcp_request_log`) covers every op call uniformly. -## Surface 2 — localOnly admin ops via shell-job `inherit:` +## Surface 2 — local-only work via shell-job `inherit:` -Some operations are flagged `localOnly: true` in `src/core/operations.ts` and -are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full -list (as of v0.36.5.0) includes: +Two mechanisms keep local-only work off the remote surface, and they operate +at different layers: -- `sync` (filesystem walks need local FS access) -- `embed` (orchestrates the embed pipeline) -- `extract` (walks markdown files) -- `dream` (synthesis cycle) -- `doctor` (filesystem hygiene checks) -- `autopilot` (background daemon orchestration) -- `init` (creates `~/.gbrain/`) -- `secrets` (config management) +- **Op layer:** operations flagged `localOnly: true` in + `src/core/operations.ts` are filtered out of the HTTP MCP surface entirely + — a remote caller never sees them. +- **CLI layer:** on a thin-client install (remote MCP configured, no local + engine), commands that require a local engine or the local filesystem are + refused at dispatch with a pinpoint hint naming the closest alternative. + The authoritative set is `THIN_CLIENT_REFUSED_COMMANDS` in `src/cli.ts` — + read it there rather than trusting any list copied into a doc; it covers + `sync`, `embed`, `extract`, `dream`, `enrich`, `serve`, `config`, and a + couple dozen more. -For these, the agent cannot route through HTTP MCP. The only path is to run +Notable non-members: `doctor` is NOT refused on a thin client — it reroutes +to an outbound-HTTP probe set (`src/core/doctor-remote.ts`); `bootstrap` and +`hook` are engine-free and work on any install shape. + +For refused commands, the agent cannot route through HTTP MCP. The path is to run `gbrain` as a CLI subprocess. The recommended pattern is to submit the subprocess as a shell job to the gbrain Minions worker so retry / backoff / DLQ / audit trail all come for free. @@ -114,12 +119,11 @@ full validation rules and error catalog. ### Why this is preferred over writing secrets into `env:` per-job -- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` - per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit - JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain - via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue - validation. The error message names `inherit: ["database_url"]` as the - replacement. +- Passing `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` per job would + land the URL plaintext in `minion_jobs.data` and the shell-audit JSONL — + visible to anyone with brain-DB read access (or a brain dump, or a shared + brain via mounts). Pre-enqueue validation rejects it; the error message + names `inherit: ["database_url"]` as the replacement. ### Worker setup (one-time, per host) @@ -146,11 +150,11 @@ proxy for worker env. | `get_page` / `list_pages` | HTTP MCP | Same. | | `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. | | `find_experts` / `find_orphans` | HTTP MCP | Same. | -| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. | -| `dream` | Shell job + `inherit:` | `localOnly: true`. | -| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. | +| `sync` / `embed` / `extract` | Shell job + `inherit:` | Thin-client refused; needs local engine + FS. | +| `dream` | Shell job + `inherit:` | Thin-client refused; synthesis runs on the host. | +| `doctor` | Run directly (any install) | Not refused: thin clients get the remote probe set. | | `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. | -| `init` / `secrets` | One-time host setup | Operator action, not agent action. | +| `init` / `config` | One-time host setup | Operator action, not agent action. | ## Recommended patterns @@ -166,16 +170,19 @@ proxy for worker env. - **`env:` still works** for non-secret values, or for cases where you WANT the value in the row (e.g. an opaque correlation token your audit flow needs to read back later). The validator doesn't second-guess you. -- **Never try to route a `localOnly` op through thin-client MCP.** It will - fail with `localOnly op refused in thin-client mode`. Use shell-job + - `inherit:` (for secrets) or `env:` (for non-secrets). +- **Never try to route a refused command through a thin client.** The CLI + refuses it at dispatch with a hint. Use shell-job + `inherit:` (for + secrets) or `env:` (for non-secrets) on the host instead. +- **Push-based context.** Beyond request/response ops, MCP clients can + receive volunteered context via the `volunteer_context` op — see + [push-context.md](./push-context.md). -## Migration: from pre-v0.36.5.0 +## Migration: from `env:`-passed secrets If your agent submits shell jobs that pass secrets via `env:`: ```jsonc -// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext. +// Rejected at submit: the URL would persist in minion_jobs.data plaintext. { "cmd": "gbrain sync --skip-failed", "cwd": "/data/gbrain", @@ -186,7 +193,7 @@ If your agent submits shell jobs that pass secrets via `env:`: Switch to (recommended): ```jsonc -// v0.36.5.0+: name in row, value resolved at child-spawn from worker config. +// Name in row, value resolved at child-spawn from worker config. { "cmd": "gbrain sync --skip-failed", "cwd": "/data/gbrain", diff --git a/docs/guides/bootstrap.md b/docs/guides/bootstrap.md index bf3ae6144..dc201c3e8 100644 --- a/docs/guides/bootstrap.md +++ b/docs/guides/bootstrap.md @@ -75,7 +75,8 @@ zero in keyless mode; with a key, the standard spend gates apply the cause. - **Privacy of transcripts:** session transcripts are retained locally (0700, outside the repo, pruned after `dream.synthesize.corpus_retention_days`, default - 30) and secret-redacted at write time. They never enter the repo. The extraction + 30 — set it in the config file, `~/.gbrain/config.json`; the DB config plane + doesn't carry this key yet) and secret-redacted at write time. They never enter the repo. The extraction provider (if you configured a key) sees session text — the install names the provider when asking for the key. diff --git a/docs/guides/brain-agent-loop.md b/docs/guides/brain-agent-loop.md index 0957b8e11..ef67da5a6 100644 --- a/docs/guides/brain-agent-loop.md +++ b/docs/guides/brain-agent-loop.md @@ -27,6 +27,7 @@ READ: check brain FIRST (before responding) │ → gbrain search "{entity name}" │ → gbrain get {slug} (if you know it) │ → gbrain query "what do we know about {topic}" + │ → full protocol: brain-first-lookup.md │ ▼ RESPOND with brain context (every answer is better with context) @@ -100,12 +101,14 @@ on_message(text): Write immediately after the conversation, while the context is fresh. 3. **Sync after every write batch.** Without sync, the brain search index is - stale. The next query won't find what you just wrote. + stale. The next query won't find what you just wrote. On installs set up + via `gbrain bootstrap`, per-turn context injection and session-end + persistence hooks automate parts of this loop — see + [bootstrap.md](bootstrap.md) and [push-context.md](push-context.md). -4. **External APIs are fallback, not primary.** `gbrain search` before - Brave Search. `gbrain get` before Crustdata. The brain has relationship - history, your own assessments, meeting transcripts, cross-references. - No external API can provide that. +4. **External APIs are fallback, not primary.** `gbrain search` before any + web or enrichment API. The full brain-before-external protocol (and why) + lives in [brain-first-lookup.md](brain-first-lookup.md). ## How to Verify It Works diff --git a/docs/guides/brain-first-lookup.md b/docs/guides/brain-first-lookup.md index 5fbdc4d6b..6dc759e4e 100644 --- a/docs/guides/brain-first-lookup.md +++ b/docs/guides/brain-first-lookup.md @@ -51,22 +51,25 @@ The brain has context no external API can provide: - Timeline (what changed recently, what's trending) A LinkedIn scrape gives you their job title. The brain gives you: "co-founded -Brex, you had coffee with him 3 times, last discussed the payments infrastructure -thesis, he's interested in your take on AI agents." +widget-co, you had coffee with her 3 times, last discussed the payments +infrastructure thesis, she's interested in your take on AI agents." ## Tricky Spots 1. **Try keyword first, then hybrid.** Keyword search works without embeddings - (day one). Hybrid search needs embeddings but finds semantic matches. Try - both in sequence. + (day one — and it's ALL you get in keyless mode, see + [bootstrap.md](bootstrap.md)). Hybrid search needs embeddings but finds + semantic matches. Try both in sequence. -2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact - slug doesn't exist, it suggests alternatives. Use this for name variants - ("Pedro" → "pedro-franceschi"). +2. **Fuzzy slug matching is opt-in.** Pass `--fuzzy` (the `fuzzy: true` param + on `get_page`) and a near-miss slug resolves to the unique candidate, or + returns an `ambiguous_slug` error listing the candidates. WITHOUT the flag + a miss just throws `page_not_found` (with a hint to retry with + `fuzzy: true`). Use it for name variants ("Alice" → "alice-example"). 3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?" - should check the brain first. The brain might have it, and the lookup adds - no latency (< 100ms for keyword search). + should check the brain first. The brain might have it, and a keyword + lookup is fast enough to be effectively free. 4. **Load compiled truth + recent timeline.** The compiled truth gives you the state of play in 30 seconds. The timeline gives you what changed recently. diff --git a/docs/guides/brain-vs-memory.md b/docs/guides/brain-vs-memory.md index 6d127dbd4..c65f4f3e9 100644 --- a/docs/guides/brain-vs-memory.md +++ b/docs/guides/brain-vs-memory.md @@ -17,8 +17,8 @@ on new_information(info): # This is world knowledge -- facts about entities external to the agent gbrain put --content "..." # Examples: - # "Pedro is CEO of Brex" -> gbrain (person page) - # "Brex raised Series D at $12B" -> gbrain (company page) + # "alice-example is CEO of widget-co" -> gbrain (person page) + # "widget-co raised Series D at $12B" -> gbrain (company page) # "Tuesday's meeting covered Q2" -> gbrain (meeting page) # "The meatsuit maintenance tax" -> gbrain (originals page) @@ -57,15 +57,16 @@ on user_asks(question): ## Tricky Spots -1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world. +1. **Don't store people in agent memory.** "alice-example prefers email over Slack" feels like a preference, but it's a fact about Alice -- it goes in GBrain on her page. Agent memory is for the agent's own operational state, not facts about people in the world. 2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration. 3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory. 4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything. + On installs set up via `gbrain bootstrap`, "agent memory" has a concrete file form: MEMORY.md and the other identity files in the agent repo (see [bootstrap.md](bootstrap.md)). The routing rule is unchanged -- those files hold operational state and identity, not world knowledge. 5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session. ## How to Verify -1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain. +1. Ask the agent "Who is alice-example?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain. 2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state. 3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers. 4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory). diff --git a/docs/guides/compiled-truth.md b/docs/guides/compiled-truth.md index 43a9329b4..8c28aea48 100644 --- a/docs/guides/compiled-truth.md +++ b/docs/guides/compiled-truth.md @@ -44,12 +44,12 @@ Sharp technical leader. Under-appreciated internally. Watch for signs of burnout Ascending. Likely CTO track if the migration succeeds. ## Relationship -Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis. +Met through alice-example. Had coffee 3x. Last: discussed API architecture thesis. ## Contact sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen ---- + ## Timeline @@ -58,7 +58,7 @@ sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen [Source: Meeting notes, 2026-04-07 2:00 PM PT] - **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops. [Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT] -- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background. +- **2026-03-15** | First meeting. Intro from alice-example. Strong technical background. [Source: User, direct conversation, 2026-03-15 3:00 PM PT] ``` @@ -113,9 +113,19 @@ support that claim. truth chunks with higher relevance than timeline chunks. This means the freshest synthesis surfaces first in search results. -4. **The --- separator matters.** GBrain uses the first standalone `---` after - frontmatter to split compiled_truth from timeline. Everything above is compiled - truth, everything below is timeline. +4. **The timeline sentinel matters — and a bare `---` is NOT one.** GBrain + splits compiled_truth from timeline at the first recognized sentinel, in + order of precedence: + 1. `` — preferred; unambiguous, and what GBrain itself + emits when it writes a page. + 2. `--- timeline ---` — decorated separator. + 3. `---` ONLY when the next non-empty line is `## Timeline` or + `## History` (backward-compat for older gbrain-written files). + + A plain `---` line anywhere else is a markdown horizontal rule, not a + separator. Author new pages with `` (as in the example + above); everything above it is compiled truth, everything below is + timeline. 5. **Don't skip the Assessment section.** The assessment is the value. "Strong technical leader" is something no API can provide. It's YOUR read on this diff --git a/docs/guides/content-media.md b/docs/guides/content-media.md index 9fa257bd1..16bd94709 100644 --- a/docs/guides/content-media.md +++ b/docs/guides/content-media.md @@ -8,6 +8,12 @@ Without this: media links are bookmarks that decay -- you remember watching a vi ## Implementation +gbrain's own media surfaces complement this pattern: the bundled +`media-ingest` skill (`skills/media-ingest/`) ships the ingestion workflow, +and `gbrain files` handles binary/file upload for attachments that should +live alongside pages. For meeting recordings specifically, see +[meeting-ingestion.md](meeting-ingestion.md). + ``` on user_shares_media(url_or_file): @@ -119,7 +125,7 @@ on user_shares_media(url_or_file): ## Tricky Spots 1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it. -2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain. +2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "The speaker made a specific claim about compute scaling that contradicts what another researcher said in their NeurIPS talk -- see media/youtube/a-researcher-neurips-2025" is useful. The analysis connects the new media to the existing brain. 3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page. 4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry. 5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power. diff --git a/docs/guides/cron-schedule.md b/docs/guides/cron-schedule.md index a96609f2d..24c9b8d0c 100644 --- a/docs/guides/cron-schedule.md +++ b/docs/guides/cron-schedule.md @@ -26,6 +26,27 @@ fixed. You wake up and the brain is smarter than when you went to sleep. | Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) | | Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below | +### Prefer gbrain's native schedulers where they fit + +System cron is the lowest common denominator, but gbrain ships its own +scheduling surfaces — reach for these first: + +- **`gbrain dream`** — the shipped nightly maintenance cycle (lint, + backlinks, extract, sync, embed, synthesize). Schedule THIS instead of + hand-rolling the dream cycle below. +- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry, + backoff, and an audit trail. See the `minion-orchestrator` skill. +- **`gbrain autopilot`** — the long-lived background daemon that runs cycles + on its own cadence. +- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to + manage its harness's scheduler. +- **Bootstrap session-triggered schedules** — `gbrain bootstrap` installs + HEARTBEAT.md-driven schedules that fire on session activity; see + [bootstrap.md](bootstrap.md). + +For scheduling `sync` + `embed --stale` specifically, the home doc is +[live-sync.md](live-sync.md). + ## Implementation: Setting Up Cron Jobs ```bash @@ -50,18 +71,12 @@ fixed. You wake up and the brain is smarter than when you went to sleep. ### Quiet Hours Gate (MANDATORY) -Every cron job that sends notifications MUST check quiet hours first. -See [Quiet Hours](quiet-hours.md) for the full pattern. - -```bash -# In every cron script: -if ! bash scripts/quiet-hours-gate.sh; then - mkdir -p /tmp/cron-held - echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md - exit 0 -fi -# Not quiet hours — send normally -``` +Every cron job that sends notifications MUST check quiet hours first. The +gate is a small script YOU create (it doesn't ship with gbrain) and call at +the top of every notification-sending cron script; held output goes to a +holding directory that the morning briefing drains. See +[Quiet Hours](quiet-hours.md) for the gate script and the full pattern — +don't copy a snippet from here, that page is the single home. ### Travel-Aware Timezone Handling @@ -88,6 +103,12 @@ morning briefing. Zero config change needed. The most important cron job. Runs while you sleep. +**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle +(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule +it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is +covered. The pseudocode that follows is the harness-side variant for agents +that also do LLM-driven entity sweeps and memory consolidation on top. + ### What It Does ``` @@ -150,11 +171,11 @@ echo "Dream cycle starting at $(date)" # Phase 1: Entity sweep (spawn sub-agent) # Read today's conversation logs, extract entities, update brain -# Phase 2: Citation hygiene -gbrain doctor --json | jq '.checks[] | select(.status=="warn")' +# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize) +gbrain dream -# Phase 3: Embed any stale content -gbrain embed --stale +# Phase 3: Surface anything the cycle flagged +gbrain doctor --json | jq '.checks[] | select(.status=="warn")' echo "Dream cycle complete at $(date)" ``` diff --git a/docs/guides/deterministic-collectors.md b/docs/guides/deterministic-collectors.md index 9f59b8ed8..bd172cb30 100644 --- a/docs/guides/deterministic-collectors.md +++ b/docs/guides/deterministic-collectors.md @@ -143,4 +143,7 @@ the same pass. --- -*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* +*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The +[email-to-brain recipe](../../recipes/email-to-brain.md) implements this +collector pattern end-to-end; [cron-schedule.md](cron-schedule.md) covers +scheduling it.* diff --git a/docs/guides/diligence-ingestion.md b/docs/guides/diligence-ingestion.md index 1cb9b4286..10fc3eee2 100644 --- a/docs/guides/diligence-ingestion.md +++ b/docs/guides/diligence-ingestion.md @@ -53,6 +53,8 @@ Write extracted content to `brain/diligence/{company}/{doc-name}.md`: **Step 5: Save Raw Files.** Copy original PDFs/files to `brain/diligence/{company}/.raw/` Preserve originals for reference. The diarized version is for search. +This is safe by design: `.raw/` directories are excluded from sync, so the +originals never enter the search index — only your extracted markdown does. **Step 6: Create or Update index.md.** Every diligence directory needs an `index.md`: @@ -103,7 +105,8 @@ cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} in **Step 9: Publish (if asked).** When the user wants a shareable brief, create a password-protected -published version. Strip internal notes and raw assessment language. +published version via the `publish` skill (`skills/publish/`). Strip +internal notes and raw assessment language. ### Quality Bar diff --git a/docs/guides/embedding-migration.md b/docs/guides/embedding-migration.md index ac748609e..ee8c6b4e8 100644 --- a/docs/guides/embedding-migration.md +++ b/docs/guides/embedding-migration.md @@ -7,8 +7,8 @@ sunsetting provider (for example ZeroEntropy's hosted API, which shuts down but it is provider-agnostic: any configured `provider:model` works as a target. -Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the -README reference). +Also reachable as `gbrain retrieval-upgrade` — the alias that `gbrain doctor` +repair hints and the README point at. ## Quick start diff --git a/docs/guides/enrichment-pipeline.md b/docs/guides/enrichment-pipeline.md index a9caad98d..e48872f34 100644 --- a/docs/guides/enrichment-pipeline.md +++ b/docs/guides/enrichment-pipeline.md @@ -8,6 +8,11 @@ Without this: brain pages are thin shells with only what the user manually typed ## Implementation +gbrain ships both halves of this: `gbrain enrich` is the batch enrichment +primitive (finds thin pages and enriches at scale), and the `enrich` skill +(`skills/enrich/`) is the agent-driven page-at-a-time workflow. The pipeline +below is the pattern they implement — use it to customize or extend. + ``` on enrich(entity, trigger): # trigger: meeting mention, email thread, social interaction, user request @@ -68,9 +73,10 @@ on enrich(entity, trigger): gbrain link # person -> deal # Every entity page links to every other entity page that references it -# People page sections (not a LinkedIn profile -- a living portrait): -# Executive Summary, State, What They Believe, What They're Building, -# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline +# People page sections: use the person-page structure from compiled-truth.md +# (Executive Summary, State, What They Believe, ... Timeline) -- that doc is +# the single home for the section taxonomy. Enrichment can add texture +# sections on top (What Motivates Them, Hobby Horses, Open Threads). # Facts are table stakes. TEXTURE is the value. # Extract texture, not just facts: @@ -100,4 +106,4 @@ on enrich(entity, trigger): 5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old. --- -*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* +*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Compiled Truth](compiled-truth.md) for the person-page section taxonomy, [Spend Controls](../operations/spend-controls.md) for gbrain's own embedding/LLM spend gates.* diff --git a/docs/guides/entity-detection.md b/docs/guides/entity-detection.md index eba654882..4004d8935 100644 --- a/docs/guides/entity-detection.md +++ b/docs/guides/entity-detection.md @@ -7,10 +7,10 @@ Every inbound message gets scanned for original thinking AND entity mentions so ## What the User Gets Without this: the agent answers questions but forgets everything. You mention -Pedro in a meeting, and next week the agent doesn't know who Pedro is. +Alice in a meeting, and next week the agent doesn't know who Alice is. With this: every person, company, and idea mentioned in conversation gets a -brain page. Next time Pedro comes up, the agent already has context. The +brain page. Next time Alice comes up, the agent already has context. The brain compounds. ## Implementation @@ -18,6 +18,11 @@ brain compounds. Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it to finish before responding. It runs in parallel. +This pattern is harness-side by design, but gbrain ships help on both ends: +the `signal-detector` skill (`skills/signal-detector/`) is the bundled +version of this detection loop, and `gbrain extract` runs gbrain's own +extraction machinery (entities, facts) over already-synced content. + ``` on_every_message(message_text, source_context): @@ -109,6 +114,7 @@ is_notable(entity): | Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions | | Hot takes with reasoning | Routine operational messages | | Metaphors that reveal new angles | Requests without embedded insight | +| Emotional/psychological insights about self or others | | ### Filing Rules @@ -116,11 +122,17 @@ is_notable(entity): |--------|-------------| | User generated the idea | `brain/originals/{slug}.md` | | User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) | +| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) | | World concept someone else coined | `brain/concepts/{slug}.md` | | Product or business idea | `brain/ideas/{slug}.md` | | Person mentioned | `brain/people/{slug}.md` | | Company mentioned | `brain/companies/{slug}.md` | | Media referenced | `brain/media/{type}/{slug}.md` | +| Article ABOUT the user | `brain/media/writings/{slug}.md` | + +This table is the single home for the capture/filing taxonomy. Other guides +([idea-capture](idea-capture.md) especially) link here rather than carrying +their own copy. ### The Iron Law of Back-Linking @@ -128,21 +140,21 @@ Every entity mention MUST create a back-link FROM the entity page TO the source. This is not optional. ``` -// When message mentions "Pedro" and creates a meeting page: +// When message mentions "Alice" and creates a meeting page: // 1. Update the meeting page (normal) brain/meetings/2026-04-10-board-sync.md: - - Pedro presented Q1 numbers + - Alice presented Q1 numbers -// 2. ALSO update Pedro's page (back-link) -brain/people/pedro-franceschi.md: +// 2. ALSO update Alice's page (back-link) +brain/people/alice-example.md: ## Timeline - **2026-04-10** | Presented Q1 numbers at board sync [Source: User, board meeting, 2026-04-10] ``` Without back-links, you can't traverse the graph. "Show me everything related -to Pedro" only works if Pedro's page links back to every mention. +to Alice" only works if Alice's page links back to every mention. ## Tricky Spots @@ -163,7 +175,7 @@ to Pedro" only works if Pedro's page links back to every mention. 5. **Dedup before creating.** Always `gbrain search` before creating a page. Variant spellings, nicknames, and company abbreviations cause duplicates. - "Pedro Franceschi" and "Pedro" might be the same person. + "Alice Example" and "Alice" might be the same person. ## How to Verify @@ -182,7 +194,7 @@ to Pedro" only works if Pedro's page links back to every mention. 4. **Send a boring message.** Say "ok sounds good." Verify: nothing was created. The detector should report "No signals detected." -5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi." +5. **Check for duplicates.** Mention "Alice" then later "Alice Example." Verify: one page, not two. --- diff --git a/docs/guides/executive-assistant.md b/docs/guides/executive-assistant.md index 06739f9d1..4be22e486 100644 --- a/docs/guides/executive-assistant.md +++ b/docs/guides/executive-assistant.md @@ -8,6 +8,11 @@ Without this: the agent triages email mechanically ("you have 12 unread"), preps ## Implementation +Before hand-rolling these: gbrain bundles the morning-briefing half of this +pattern as the `briefing` skill (`skills/briefing/`) and the task-prep half +as `daily-task-prep` (`skills/daily-task-prep/`). Use the workflows below to +extend or customize what those skills already ship. + ``` # WORKFLOW 1: Email Triage on email_batch(emails): @@ -59,8 +64,8 @@ on upcoming_meeting(meeting): briefing[attendee] = "No brain page -- consider enriching" # Surface: shared history, what to follow up on, what to watch for - # "Last time you discussed the Series B timeline. Pedro was concerned - # about burn rate. Here's the latest from his company page." + # "Last time you discussed the Series B timeline. alice-example was + # concerned about burn rate. Here's the latest from her company page." # WORKFLOW 3: Post-Inbox Brain Updates on inbox_cleared(): @@ -93,9 +98,9 @@ on schedule_request(meeting): 1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical. 2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise. -3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate." +3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with alice-example -- last time you discussed the Series B, she was concerned about burn rate." 4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip. -5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide). +5. **Scheduling nudges require timeline data.** "You haven't met with charlie-example in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide). ## How to Verify diff --git a/docs/guides/idea-capture.md b/docs/guides/idea-capture.md index 9a26ce2f4..e82466fe8 100644 --- a/docs/guides/idea-capture.md +++ b/docs/guides/idea-capture.md @@ -61,14 +61,13 @@ capture_idea(message_text, source_context): ### The Authorship Test -| Signal | Destination | -|--------|-------------| -| User generated the idea | `brain/originals/{slug}.md` | -| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) | -| World concept someone else coined | `brain/concepts/{slug}.md` | -| Product or business idea | `brain/ideas/{slug}.md` | -| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) | -| Article ABOUT user | `brain/media/writings/` | +Who authored the idea determines where it files: user-generated ideas, +syntheses, and ghostwritten work go to `brain/originals/`; borrowed world +concepts to `brain/concepts/`; product ideas to `brain/ideas/`; articles +ABOUT the user to `brain/media/writings/`. The full filing-rules table (and +the what-counts-as-original-thinking criteria) lives in +[entity-detection.md](entity-detection.md) — the single home for the capture +taxonomy. ### Capture Standards @@ -78,21 +77,6 @@ capture_idea(message_text, source_context): "tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase. The vivid version is the real version. -**What counts as worth capturing:** -- Original observations about how the world works -- Novel connections between disparate things -- Frameworks and mental models -- Pattern recognition moments ("I keep seeing X in every Y") -- Hot takes with reasoning behind them -- Metaphors that reveal new angles -- Emotional/psychological insights about self or others - -**What does NOT count:** -- Routine operational messages ("ok", "do it") -- Pure questions without embedded observations -- Echoing back something the agent said -- Acknowledgments and reactions - ### The Depth Test **Could someone unfamiliar with the user read this page and understand not @@ -137,21 +121,9 @@ Every original MUST link to: ### Notability Filtering -Before creating any entity page, check notability: - -**Create a page for:** -- People you know or discuss with specificity -- Companies you're evaluating, working with, or investing in -- Media you mention with personal reaction -- Anyone you've explicitly engaged with - -**Don't create pages for:** -- Generic references or passing examples -- Low-engagement accounts who mentioned you once -- Pure metaphors ("like the Roman Empire...") -- One-off encounters with no follow-up - -**Decision:** If notable AND no page exists, create a full page with web +Before creating any entity page, check notability — the full create/skip +criteria live in [entity-detection.md](entity-detection.md#notability-filtering). +The decision rule: if notable AND no page exists, create a FULL page with web search enrichment. No stubs. If you make a page, make it good. ## Tricky Spots @@ -187,4 +159,6 @@ search enrichment. No stubs. If you make a page, make it good. --- -*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* +*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The bundled +`idea-ingest` skill (`skills/idea-ingest/`) ships this workflow. See also: +[Entity Detection](entity-detection.md).* diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.md index 66d5e7ccf..4405c959e 100644 --- a/docs/guides/live-sync.md +++ b/docs/guides/live-sync.md @@ -43,7 +43,12 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale - `gbrain sync --repo ` -- one-shot incremental sync. Detects changes via `git diff`, imports only what changed. For small changesets (<= 100 files), - embeddings are generated inline during import. + embeddings are generated inline during import — unless the inline cost gate + intervenes: when the estimated embedding spend crosses the configured floor + in a non-interactive session (cron, `--json`), sync auto-defers embeds to a + capped `embed-backfill` job instead of spending silently. Either way the + chunks get embedded; a deferred run just finishes asynchronously. See + [spend controls](../operations/spend-controls.md). - `gbrain embed --stale` -- backfill embeddings for any chunks that don't have them. Safety net for large syncs (>100 files) or prior `--no-embed` runs. - `gbrain sync --watch --repo ` -- foreground polling loop, every 60s @@ -97,15 +102,27 @@ Triggers sync on push events for instant sync (<5s). ### What Gets Synced Sync only indexes "syncable" markdown files. These are excluded by design: -- Hidden paths (`.git/`, `.raw/`, etc.) -- The `ops/` directory -- Meta files: `README.md`, `index.md`, `schema.md`, `log.md` +- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees + (`node_modules/`, `dist/`, `build/`, `venv/`) +- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md` -### Sync is Idempotent +Everything else is ordinary synced content — including `ops/` (the bundled +daily-task-manager skill files its canonical page under `ops/tasks`). + +### Sync is Idempotent — and Resumable Concurrent runs are safe. Two syncs on the same commit no-op because content hashes match. If both a cron and `--watch` fire simultaneously, no conflict. +Long syncs also survive being killed: progress checkpoints into the database +as files drain, so a killed or aborted run resumes from where it stopped, and +the sync bookmark only advances on true completion. A progress-aware stall +watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts +a run that stops making forward progress and releases the per-source lock so +the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence +and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env +vars — incident-time escape hatches, not everyday knobs. + ## Tricky Spots 1. **Always chain sync + embed.** Running `gbrain sync` without diff --git a/docs/guides/meeting-ingestion.md b/docs/guides/meeting-ingestion.md index 81ad159ba..43be8a3f1 100644 --- a/docs/guides/meeting-ingestion.md +++ b/docs/guides/meeting-ingestion.md @@ -58,15 +58,20 @@ on new_meeting_transcript(meeting): # Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings # Source: Circleback (https://circleback.ai) or any service with # speaker diarization + API/webhook access + +# Automation: the built-in `extract-timeline-from-meetings` Minion job +# automates step 3 (entity timeline propagation) for already-ingested +# meeting pages: gbrain jobs submit extract-timeline-from-meetings --follow ``` ## Tricky Spots 1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth. 2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation. -3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room. -4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful. +3. **Mentioned people are not just attendees.** If the meeting discussed "Alice's team at widget-co," then Alice's page AND widget-co's page need updates -- even though Alice wasn't in the room. +4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Alice pushed back on the burn rate, Charlie didn't commit to the timeline, and nobody addressed the pricing gap" is useful. 5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always. +6. **`--source` on `timeline-add` is the citation text, not source routing.** Because the op declares its own `source` parameter, the CLI binds `--source` to it. To write into a different registered source, use the `.gbrain-source` dotfile or `GBRAIN_SOURCE` env for routing instead. ## How to Verify @@ -76,5 +81,11 @@ on new_meeting_transcript(meeting): 4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages. 5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran). +## Related + +- `skills/meeting-ingestion/SKILL.md` — the bundled, agent-executable skill + for this workflow (the canonical step-by-step home; this guide is the + pattern overview). + --- *Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* diff --git a/docs/guides/minions-deployment.md b/docs/guides/minions-deployment.md index d7115e5d8..4e2f3087a 100644 --- a/docs/guides/minions-deployment.md +++ b/docs/guides/minions-deployment.md @@ -40,6 +40,10 @@ gbrain jobs supervisor status --json # Graceful stop (SIGTERM + drain wait + SIGKILL fallback). gbrain jobs supervisor stop + +# Optional: cap worker memory in MB (--max-rss). Without the flag the RSS +# watchdog is still on, at a RAM-relative auto-sized cap. +gbrain jobs supervisor --concurrency 4 --max-rss 4096 ``` **Exit codes:** @@ -50,9 +54,11 @@ gbrain jobs supervisor stop | 1 | Max crashes exceeded (worker kept dying) | | 2 | Another supervisor holds the PID lock | | 3 | PID file unwritable (permission / path error) | +| 4 | Queue-scoped DB lock lost mid-run (`LOCK_LOST` — exited rather than risk a split-brain) | An agent seeing exit=2 can safely treat it as "one is already running"; -exit=1 should page a human. +exit=4 as "restart me — the DB lock refresh failed"; exit=1 should page +a human. ### Lowering scheduling priority (`--nice`) @@ -233,7 +239,7 @@ use a dedicated queue name like `nightly-enrich` above. ## Upgrading from an older deployment -### From `minion-watchdog.sh` (pre-v0.20) +### From `minion-watchdog.sh` Earlier versions of this guide shipped a 68-line bash watchdog (`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor` @@ -270,10 +276,10 @@ Regardless of which deployment path you're upgrading from: in-flight job landing partial schema. 2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if `gbrain doctor` reports any migration as `partial` or `pending`. -3. **If you run shell jobs:** from v0.14 onward, pass - `--allow-shell-jobs` to the supervisor (or keep - `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't - need the flag; only the worker does. +3. **If you run shell jobs:** pass `--allow-shell-jobs` to the + supervisor (or keep `GBRAIN_ALLOW_SHELL_JOBS=1` in + `/etc/gbrain.env`). Submitters don't need the flag; only the worker + does. 4. **Verify.** `gbrain doctor` should report zero `pending` or `partial` migrations plus a healthy `supervisor` check. `gbrain jobs stats` should show no unexplained growth in `dead` between pre- and @@ -283,29 +289,30 @@ Regardless of which deployment path you're upgrading from: ### Supabase connection drops -The worker uses a single Postgres connection. If Supabase drops it -(maintenance, connection limits, network blip), lock renewal fails -silently. The stall detector then dead-letters the job after -`max_stalled` misses. +If Supabase drops the worker's Postgres connection (maintenance, +connection limits, network blip), this now self-heals under the +supervisor: the worker's DB-liveness probe self-exits (`db_dead`) on a +dead pool and the supervisor respawns it with a fresh pool, and the +supervisor also restarts a worker that stops making progress while +claimable work waits. The escalation commands and thresholds live in the +[queue operations runbook](queue-operations-runbook.md) — that's the +canonical home for wedge recovery. -**Current defaults that make this worse:** +What can still bite: a *brief* blip during a long-running job can make +lock renewal miss, and the stall detector dead-letters the job after +`max_stalled` misses (schema column default 5; lock duration and stall +check interval are both 30 s). -- `lockDuration: 30000` (30 s) — too short for long jobs during - connection blips. -- `max_stalled: 5` (schema column default — see `src/schema.sql` and - `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. -- `stalledInterval: 30000` (30 s) — checks too aggressively. - -**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, +**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`, `--backoff-type fixed|exponential`, `--backoff-delay `, -`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags -(since v0.13.1). These write onto the job row at submit time — which is -what `handleStalled()` reads — so per-job tuning is the real knob today. +`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags. +These write onto the job row at submit time — which is what +`handleStalled()` reads — so per-job tuning is the real knob. ### DO NOT pass `maxStalledCount` to `MinionWorker` It's a no-op. The stall detector reads the row's `max_stalled` column -(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`. +(set at submit time), not the worker opt in `src/core/minions/worker.ts`. Use `gbrain jobs submit --max-stalled N` per-job instead. ### Zombie shell children diff --git a/docs/guides/minions-fix.md b/docs/guides/minions-fix.md index 17142d638..c2507e8e3 100644 --- a/docs/guides/minions-fix.md +++ b/docs/guides/minions-fix.md @@ -1,5 +1,9 @@ # Minions fix — repairing a half-migrated install +> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're +> on any recent release, the canonical fix below (`gbrain apply-migrations +> --yes`) is all you need; the stopgap sections exist for archaeology. + **tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs still on `agentTurn`), run: @@ -34,17 +38,16 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`. gbrain doctor ``` -If the install is half-migrated, you'll see: +If the install is half-migrated, you'll see the `minions_migration` check +fail: ``` [FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes ``` -or - -``` -[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes -``` +(Missing `~/.gbrain/preferences.json` on a fresh install is a valid +pre-`apply-migrations` state — doctor deliberately does NOT fail on that +alone; the partial-migration record is the canonical half-migration signal.) For a machine-readable report (cron-friendly): diff --git a/docs/guides/minions-shell-jobs.md b/docs/guides/minions-shell-jobs.md index a5cd26b4b..62875ccf9 100644 --- a/docs/guides/minions-shell-jobs.md +++ b/docs/guides/minions-shell-jobs.md @@ -129,11 +129,10 @@ JSONL records the same. Pre-enqueue validation rejects the submission if the worker can't resolve the requested key, with a paste-ready `gbrain config set database_url ` hint. -**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers -wrote things like: +**Why not just write the URL into `env:` directly?** You *can*: ```jsonc -// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time. +// ❌ Works, but plants the secret in the job row. Prefer inherit:. { "cmd": "gbrain stats", "cwd": "/data/gbrain", @@ -141,14 +140,14 @@ wrote things like: } ``` -This planted plaintext secrets in `minion_jobs.data` (DB row) and in the +This plants plaintext secrets in `minion_jobs.data` (DB row) and in the shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump, -or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't -forbid that pattern — the validator trusts the agent — but **prefer +or a shared brain via the mounts feature) sees the URL. The validator +doesn't forbid the pattern — it trusts the agent — but **prefer `inherit:`** for any secret you want kept out of the row. Names land in the row; values resolve at child-spawn from the worker's config. -**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case +**Scope:** `inherit:` is **free-form**. Pass any snake_case config-key name and the worker resolves the value from `loadConfig()` at child-spawn time: diff --git a/docs/guides/multi-source-brains.md b/docs/guides/multi-source-brains.md index 03702f758..9e7cd1413 100644 --- a/docs/guides/multi-source-brains.md +++ b/docs/guides/multi-source-brains.md @@ -5,6 +5,11 @@ is a `source`: a logical brain-within-the-brain with its own slug namespace, its own sync state, and its own federation policy. The rest of this guide walks the three canonical scenarios. +(Sources are the *within-one-database* axis. If you want to connect a +whole separate database — a team-published brain with its own access +policy — that's the *brain* axis: `gbrain mounts add`. See +`docs/architecture/brains-and-sources.md` for the two-axis topology.) + ## The three scenarios ### 1. Unified knowledge recall (wiki + gstack) @@ -66,14 +71,14 @@ gbrain search "tech layoffs" --source yc-media,garrys-list ### 3. Mixed (wiki federated + sessions isolated) Your main wiki is federated with a few trusted sources. Your session -transcripts (coming in v0.18) land in a separate isolated source so -they don't dominate every search result. +transcripts (`gbrain transcripts` ingests them) land in a separate +isolated source so they don't dominate every search result. ```bash # Federated sources gbrain sources add gstack --path ~/.gstack --federated -# Isolated source (future v0.18 — sessions use this shape today for ingest) +# Isolated source for session transcripts gbrain sources add sessions --path ~/.claude/sessions --no-federated ``` @@ -104,14 +109,16 @@ Every source row stores `config.federated: boolean` in its JSONB config. | `true` | Source participates in unqualified `gbrain search "X"` results. | | `false` (default for new sources) | Source only searched when explicitly named via `--source ` or qualified citation. | -The seeded `default` source is `federated=true` so pre-v0.17 brains -behave exactly as before — every page appears in search. +The seeded `default` source is `federated=true` so single-source brains +behave as you'd expect — every page appears in search. Flip later with `gbrain sources federate ` / `unfederate `. ## Commands -Full subcommand reference: +The most-used subcommands (run `gbrain sources --help` for the full, +always-current reference — it also covers `status`, `current`, +`set-cr-mode`, and the `push`/`pull` durability surface): ``` gbrain sources add --path

[--name ] [--federated|--no-federated] [--force] @@ -119,9 +126,17 @@ gbrain sources add --path

[--name ] [--federated|--no-federated] [-- --path must be a git repo (or a subdirectory of one) — see "The git requirement for --path sources" below. --force skips that check to register before git-init exists. +gbrain sources add --url [--pat-file

] [--clone-dir ] [--no-harden] + Clone + register a remote repo in one step; auto-hardens + for durability when a PAT is provided (see "Durability" below). gbrain sources list [--json] List all sources with page counts + federation state. -gbrain sources remove [--yes] [--dry-run] [--keep-storage] - Cascade-delete a source (pages, chunks, timeline). +gbrain sources archive Soft-delete: hide from search, keep data for a TTL + grace window. Prefer this over `remove`. +gbrain sources restore Un-archive. `gbrain sources archived` lists expiries; + `gbrain sources purge` permanently deletes expired archives. +gbrain sources remove [--confirm-destructive] [--dry-run] + Permanently cascade-delete a source (pages, chunks, + timeline). Shows an impact preview first. gbrain sources rename Change display name only; id is immutable. gbrain sources default Set the brain-level default. @@ -253,8 +268,8 @@ reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1` ## Upgrading an existing brain -`gbrain upgrade` runs the v16 + v17 migrations automatically. Your -existing pages all move under `source_id='default'`. Behavior is +`gbrain upgrade` runs the needed schema migrations automatically. Your +existing pages all live under `source_id='default'`. Behavior is unchanged until you add a second source. To add one: @@ -266,13 +281,13 @@ cd ~/.gstack && gbrain sources attach gstack && gbrain sync Two commands. The existing default source is untouched. -## Not in v0.18.0 +## Related features that build on sources -- Session transcript ingest (`.jsonl`, raised size cap, session - PageType) — v0.18. -- Per-source retention/TTL (`gbrain sources prune`) — v0.18. -- ACL enforcement via caller-identity — v0.17.1. -- `gbrain sources import-from-github ` one-shot bootstrap — patch - release after the core plumbing stabilizes. - -All of these build on the `sources` primitive shipped here. +- **Session transcript ingest** — `gbrain transcripts` (server-private: + raw chat exports stay on the host machine). +- **Per-source retention** — `gbrain sources archive` / `archived` / + `purge` (soft-delete with a TTL grace window). +- **One-shot remote bootstrap** — `gbrain sources add --url ` + (clone + register + auto-harden). +- **Access control across brains** — the *brain* axis (`gbrain mounts`); + see `docs/architecture/brains-and-sources.md`. diff --git a/docs/guides/operational-disciplines.md b/docs/guides/operational-disciplines.md index 17d08db50..bcc8a713e 100644 --- a/docs/guides/operational-disciplines.md +++ b/docs/guides/operational-disciplines.md @@ -43,13 +43,14 @@ on information_needed(topic): # An agent that reaches for the web before checking its own brain # is wasting money and giving worse answers. -# DISCIPLINE 3: Sync After Every Write (MANDATORY) -on brain_write_complete(): +# DISCIPLINE 3: Sync After Every Repo Write (MANDATORY) +on brain_repo_files_changed(): gbrain sync - # Without this, search results are stale. - # The page you just wrote won't appear in gbrain search or gbrain query - # until sync runs. Skipping this means the next lookup misses the - # most recent data. + # `gbrain put` indexes immediately -- pages written through the CLI/MCP + # are searchable the moment the command returns. No sync needed there. + # But files written DIRECTLY to the brain repo (an editor, a script, + # another agent committing markdown) are invisible to search until + # `gbrain sync` imports them. If anything touched repo files, sync. # DISCIPLINE 4: Daily Heartbeat Check on daily_schedule("09:00"): @@ -62,6 +63,8 @@ on daily_schedule("09:00"): on nightly_schedule("02:00"): # The dream cycle is the most important discipline. # The brain COMPOUNDS overnight. + # gbrain ships this as a first-class command: `gbrain dream`. + # The pseudocode below is the shape of the work it does. # 5a: Entity sweep -- find unlinked mentions pages = gbrain list @@ -103,16 +106,16 @@ on nightly_schedule("02:00"): ## Tricky Spots 1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots. -2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes. -3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken. -4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context. +2. **Skipping Discipline 3 (sync after repo writes) means stale search results.** A file lands in the brain repo, then you search for it -- and get nothing back. The file exists but isn't imported. Always sync after repo-file writes. (`gbrain put` is exempt: it indexes on write.) +3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Alice yesterday about the board seat" in passing -- that's a timeline entry on Alice's page, a potential update to her State section, and a signal about the board. If the agent doesn't catch it, the system is broken. +4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Alice Example" returns a LinkedIn profile. The brain returns the full picture including private context. 5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss. ## How to Verify 1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline `). 2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order). -3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran). +3. Write a markdown file directly into the brain repo (not via `gbrain put`), run `gbrain sync`, then `gbrain search` for it. Confirm it appears in results (verifies the sync discipline). A `gbrain put` page should appear in search immediately, with no sync. 4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues. 5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": ""}'`). diff --git a/docs/guides/plugin-authors.md b/docs/guides/plugin-authors.md index 9f803bcd0..4444b2772 100644 --- a/docs/guides/plugin-authors.md +++ b/docs/guides/plugin-authors.md @@ -1,4 +1,4 @@ -# Plugin authors guide (v0.15) +# Plugin authors guide — subagent definitions `gbrain` discovers subagent definitions from outside this repo via `GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw @@ -7,6 +7,11 @@ subagents alongside it, drop a plugin directory on that env path. This guide is for plugin authors. The CLI user doesn't need to read it. +> **Two plugin systems.** This doc covers *subagent definitions* +> (markdown prompts the `subagent` job handler runs). Custom *job +> handlers* (code the Minion worker executes) are a separate system — +> see [plugin-handlers.md](plugin-handlers.md). + ## Minimum viable plugin ``` @@ -65,7 +70,7 @@ You control where your plugin lives on disk; `gbrain` doesn't guess. the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped with a warning naming both sources. -**Trust policy.** Plugins ship subagent definitions ONLY in v0.15: +**Trust policy.** Plugins ship subagent definitions ONLY: - You **cannot** declare new tools. - You **cannot** extend the brain tool allow-list. @@ -76,8 +81,8 @@ with a warning naming both sources. your plugin gives you a loud startup error, not a silent "tool never fires" at 3am. -v0.16+ may open up plugin-declared tools with a separate contract. Don't -expect it. +Plugin-declared tools would require a new `plugin_version` contract; +nothing under `gbrain-plugin-v1` opens that up. ## `gbrain.plugin.json` @@ -85,9 +90,9 @@ expect it. |------------------|--------|----------|--------------------------------------------------------------------| | `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. | | `version` | string | yes | Your plugin's semver. Informational. | -| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. | +| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"`. | | `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. | -| `description` | string | no | Shown in a future plugin-listing command. | +| `description` | string | no | Informational; appears in load/collision warnings. | ## Subagent definition files @@ -103,8 +108,7 @@ Recognized frontmatter fields: | `max_turns` | number | no | Cap on assistant turns. Defaults to 20. | | `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. | -Unknown frontmatter fields are preserved but ignored by the handler. v0.16 -may consume more of them. +Unknown frontmatter fields are preserved but ignored by the handler. ## Caveats that will bite you @@ -115,8 +119,8 @@ may consume more of them. 2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your worker runs on a different host than the `gbrain agent logs` caller, - the CLI won't see heartbeats from that worker. v0.16 will unify this; - for now assume worker + CLI share a filesystem. + the CLI won't see heartbeats from that worker. Assume worker + CLI + share a filesystem. 3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI invocation. Tools that gate on `remote=true` (file_upload's strict diff --git a/docs/guides/plugin-handlers.md b/docs/guides/plugin-handlers.md index a2292784f..89985901e 100644 --- a/docs/guides/plugin-handlers.md +++ b/docs/guides/plugin-handlers.md @@ -1,7 +1,12 @@ # Plugin handlers — registering host-specific Minion handlers -GBrain's Minion worker ships with seven built-in handlers: `sync`, -`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`. +GBrain's Minion worker ships with a full set of built-in handlers, +registered by `registerBuiltinHandlers` in `src/commands/jobs.ts` — +that registry is the source of truth. Examples: `sync`, `embed`, +`lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`, `shell`, +`subagent`, `orphans`, `integrity`, plus dream-cycle phases and other +maintenance jobs. Submitting an unknown job name with +`gbrain jobs submit --follow` prints the full registered list. These cover every background operation the gbrain CLI itself performs. Host platforms (OpenClaw deployments, future hosts) register their own @@ -10,6 +15,11 @@ handlers via a plugin bootstrap that imports code, loaded by the worker, with the same trust model as any other code in the host's repo. +> **Two plugin systems.** This doc covers *job handlers* (code the Minion +> worker runs). Custom *subagent definitions* (markdown prompts loaded via +> `GBRAIN_PLUGIN_PATH`) are a separate system — see +> [plugin-authors.md](plugin-authors.md). + ## Why code, not data An earlier design draft shipped `~/.claude/gbrain-handlers.json` where @@ -64,14 +74,23 @@ auto-loads on startup (configurable via a host-provided entry point). ## Handler contract -Every handler receives a `MinionJobContext`: +Every handler receives a `MinionJobContext` (canonical definition: +`src/core/minions/types.ts`). The load-bearing fields: ```ts interface MinionJobContext { - data: Record; // job params (whatever the cron submit passed) - job: MinionJob; // full job row (id, queue, attempts, etc.) - signal: AbortSignal; // set to aborted when the worker is shutting down - inbox: MinionInbox; // read messages sent to this job while it runs + id: number; // job id + name: string; // job type + data: Record; // job params (whatever the cron submit passed) + attempts_made: number; + signal: AbortSignal; // fires on timeout, cancel, pause, or lock loss + shutdownSignal: AbortSignal; // fires only on worker SIGTERM/SIGINT + deadlineAtMs: number | null; // wall-clock deadline from timeout_at, if set + updateProgress(progress: unknown): Promise; + updateTokens(tokens: TokenUpdate): Promise; + log(message: string | TranscriptEntry): Promise; + isActive(): Promise; // is the job lock still held? + readInbox(): Promise; // unread messages sent to this job } ``` diff --git a/docs/guides/queue-operations-runbook.md b/docs/guides/queue-operations-runbook.md index 639cadee8..a46527f84 100644 --- a/docs/guides/queue-operations-runbook.md +++ b/docs/guides/queue-operations-runbook.md @@ -1,8 +1,8 @@ # Queue operations runbook "My queue looks wedged — what do I run?" The commands below are in the order -you probably want them. Shipped with v0.19.1 after a production incident -where the queue held for 90+ minutes before the operator noticed. +you probably want them. Born from a production incident where the queue held +for 90+ minutes before the operator noticed. ## First signal: jobs aren't running @@ -23,16 +23,18 @@ container health), but its DB connection died (common behind a transaction pooler) and never came back, so it claims no jobs and finishes nothing. Jobs pile up with **0 active**. Liveness checks all pass; nothing crashes. -As of v0.42.22.0 this self-heals — you usually won't have to do anything: +This self-heals — you usually won't have to do anything: - **The worker exits on its own dead pool.** Under a supervisor, the worker's DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the supervisor respawns it with a fresh pool. - **The supervisor restarts a worker that stops making progress.** If a queue has claimable work, **0 live-lock active jobs**, and no completions for 15 - minutes while the child is alive, the supervisor restarts it (covers stuck - handlers too, not just dead pools). Tune with `--wedge-restart-minutes` / - `--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables). + minutes across 3 consecutive health checks while the child is alive, the + supervisor restarts it (covers stuck handlers too, not just dead pools). + These thresholds are built in — there are no CLI flags to tune them. A + restart-loop breaker caps wedge restarts at 3 per 30-minute window, then + switches to a one-shot `wedge_restart_loop` alert in the audit log. The signal is loud now — check either: @@ -45,8 +47,12 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")' stale completions). Manual fix if you ever need it: ```bash -gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool -gbrain jobs retry # dead-lettered jobs +# Restart the supervisor with a fresh pool. `start` alone runs in the +# FOREGROUND (blocks); use --detach to get your shell back. +gbrain jobs supervisor stop && gbrain jobs supervisor start --detach --json + +# Re-queue any jobs that were dead-lettered during the wedge. +gbrain jobs retry ``` ## Triage commands @@ -101,9 +107,9 @@ claiming. Start one: GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4 ``` -## Follow-ups tracked for v0.20+ +## Related -- B7 — `minion_workers` heartbeat table for ground-truth liveness (the - `--no-worker` probe and the dropped `queue_health` worker-heartbeat - subcheck both need this). -- B3 — `gbrain doctor --fix` learns to rescue queue wedges. +- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle, + exit codes, and per-platform deployment (systemd / Fly / Render). +- [Minions shell jobs](minions-shell-jobs.md) — the `shell` job type's + security model and error table. diff --git a/docs/guides/quiet-hours.md b/docs/guides/quiet-hours.md index 00b0b655e..2bd97ee18 100644 --- a/docs/guides/quiet-hours.md +++ b/docs/guides/quiet-hours.md @@ -113,6 +113,22 @@ fi send_notification "$OUTPUT" ``` +### GBrain-native hooks + +Two places gbrain already understands quiet hours natively — use these +before rolling your own gate for the same job: + +- **Self-upgrade** — `auto` mode only applies upgrades during quiet hours, + configured via `gbrain config set self_upgrade.quiet_hours + '{"start":23,"end":8,"tz":"US/Pacific"}'`. See + [upgrades-auto-update.md](upgrades-auto-update.md). +- **Cron prompts** — schedule-driven notification jobs should carry the + gate described in this doc; [cron-schedule.md](cron-schedule.md) covers + the scheduling side. + +The shell pattern below is for everything else: your own cron jobs, +collectors, and notification paths that gbrain doesn't gate for you. + ### Configurable Hours Some users want different quiet hours. Store the config: @@ -140,7 +156,11 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring) skill reads and clears the held directory. Orphaned held files mean the pickup integration is broken. -3. **Timezone auto-detection is fragile.** Calendar-based timezone detection +3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a + held message must not be lost across a restart, use a durable held + directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`. + +4. **Timezone auto-detection is fragile.** Calendar-based timezone detection relies on the user having airline/hotel events with location data. If the user books travel without calendar entries, the system won't detect the move. Fall back to activity-hour analysis (responding at 3 AM PT = probably diff --git a/docs/guides/repo-architecture.md b/docs/guides/repo-architecture.md index 278ca31fd..c976b1d82 100644 --- a/docs/guides/repo-architecture.md +++ b/docs/guides/repo-architecture.md @@ -9,8 +9,9 @@ Separate agent behavior (replaceable) from world knowledge (permanent) into two Without this: agent config and world knowledge are mixed together. Switch agents and you lose your knowledge. Switch knowledge tools and you lose your agent setup. -With this: your brain (14,700+ files of people, companies, meetings, ideas) -survives any agent swap. Your agent config survives any knowledge tool swap. +With this: your brain (thousands of files of people, companies, meetings, +ideas) survives any agent swap. Your agent config survives any knowledge +tool swap. ## Implementation @@ -119,7 +120,15 @@ without losing your agent setup. notes). The agent repo contains operational config. Different access controls. **GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep -the search index current. The agent repo is never indexed by GBrain. +the search index current. The agent repo is not indexed by default. + +**Multi-source nuance.** With multi-source brains +([multi-source-brains.md](multi-source-brains.md)), "the brain repo" means +"each registered source." You CAN deliberately register a non-brain repo +(e.g. `~/.gstack`) as its own isolated or federated source — that's a +conscious registration with its own slug namespace, not a violation of the +boundary. The rule below is about *unregistered, accidental* indexing of +agent config. ## Tricky Spots @@ -133,9 +142,12 @@ the search index current. The agent repo is never indexed by GBrain. belongs in the brain. Agent configs, skills, cron jobs, and operational state are replaceable. People, companies, ideas, and meetings are not. -3. **Don't index the agent repo.** GBrain indexes the brain repo only. - Running `gbrain sync` against the agent repo pollutes search results - with operational config instead of world knowledge. +3. **Don't casually index the agent repo.** Running `gbrain sync` against + the agent repo pollutes search results with operational config instead + of world knowledge. (Registering it deliberately as an isolated source + is different — see the multi-source nuance above. Pin the working + directory to the right source with a `.gbrain-source` dotfile via + `gbrain sources attach `.) ## How to Verify diff --git a/docs/guides/rls-and-you.md b/docs/guides/rls-and-you.md index 3131a3f50..448267189 100644 --- a/docs/guides/rls-and-you.md +++ b/docs/guides/rls-and-you.md @@ -34,35 +34,35 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch. 99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done. -## v0.26.7 — auto-RLS event trigger and one-time backfill +## Auto-RLS: the event trigger and the one-time backfill -Starting in v0.26.7 (migration v35), gbrain ships two changes that close the -gap where a table could exist in your `public` schema without RLS for any -amount of time at all. +gbrain ships two mechanisms (schema migration v35) that close the gap where a +table could exist in your `public` schema without RLS for any amount of time +at all. **1. The event trigger.** A Postgres DDL event trigger named `auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every newly created `public.*` table. It covers `CREATE TABLE`, `CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres reports as a table-creation command. Tables created by gbrain itself, by -your other apps sharing the same Supabase project (Baku, Hermes, anything), -or by a human running raw SQL all get RLS enabled the moment they exist. +any other app sharing the same Supabase project, or by a human running raw +SQL all get RLS enabled the moment they exist. Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly ignored — Supabase manages those, and we should not touch them. -**2. The one-time backfill.** When you upgrade to v0.26.7, the migration +**2. The one-time backfill.** The first upgrade that applies migration v35 walks every existing `public.*` base table whose RLS is off and whose comment doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op on every brain. -### Breaking change: read this before upgrading +### Read this before upgrading a pre-auto-RLS brain If you have public tables that are intentionally RLS-off and you want them to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before** -running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public -table that doesn't carry the exact comment contract documented below. There -is no `--dry-run` flag on the migration. +the upgrade that applies migration v35. The backfill flips RLS on for any +public table that doesn't carry the exact comment contract documented below. +There is no `--dry-run` flag on the migration. The minimum cost of getting this wrong is one round-trip: the operator runs the SQL to enable RLS on a table that should have been exempt, then @@ -71,7 +71,7 @@ prevent a re-flip on a later doctor run. No data is lost. ### Cross-app implications -If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates +If a non-gbrain app (a side project, a script you wrote, anything) creates tables in the same Supabase project, the trigger will enable RLS on those tables too. Two ways to handle that: @@ -90,17 +90,40 @@ ship a policy. ### What if the trigger gets dropped? -`gbrain doctor` includes a new `rls_event_trigger` check that verifies the +`gbrain doctor` includes an `rls_event_trigger` check that verifies the trigger is installed and enabled. If you drop it manually for any reason -(debugging, migration testing, anything), doctor warns and gives you the -recovery command: +(debugging, migration testing, anything), doctor warns and points you here. -``` -gbrain apply-migrations --force-retry 35 +Recreate it by re-running the trigger DDL from migration v35 — idempotent +(`CREATE OR REPLACE` + `DROP EVENT TRIGGER IF EXISTS`), safe to paste into +psql as a BYPASSRLS role (e.g. `postgres`): + +```sql +CREATE OR REPLACE FUNCTION auto_enable_rls() +RETURNS event_trigger AS $$ +DECLARE + obj record; +BEGIN + FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands() + WHERE object_type = 'table' + AND schema_name = 'public' + LOOP + EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', obj.object_identity); + END LOOP; +END; +$$ LANGUAGE plpgsql; + +DROP EVENT TRIGGER IF EXISTS auto_rls_on_create_table; +CREATE EVENT TRIGGER auto_rls_on_create_table + ON ddl_command_end + WHEN TAG IN ('CREATE TABLE', 'CREATE TABLE AS', 'SELECT INTO') + EXECUTE FUNCTION auto_enable_rls(); ``` -Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS` -and recreates cleanly. +(This is the same DDL migration v35 runs — the canonical copy lives in the +`MIGRATIONS` array in `src/core/migrate.ts`. There's no CLI shortcut: +`gbrain apply-migrations --force-retry` targets the vX.Y.Z orchestrator +registry, not numeric schema migrations like v35.) ### Why no FORCE ROW LEVEL SECURITY? @@ -147,7 +170,7 @@ Rules: ```sql ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY; COMMENT ON TABLE public.expenses_ramp IS - 'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22'; + 'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=you, 2026-04-22'; ``` After that, `gbrain doctor` reports: diff --git a/docs/guides/scaling-skills.md b/docs/guides/scaling-skills.md index b765ad6ad..0e758b82d 100644 --- a/docs/guides/scaling-skills.md +++ b/docs/guides/scaling-skills.md @@ -127,9 +127,9 @@ you can use as a reference shape. The skillpack story for distributing your own resolvers across machines is covered in [skillpacks as scaffolding](skillpacks-as-scaffolding.md). -## The compact list format (v0.41.7.0) +## The compact list format -GBrain's resolver parser used to require markdown tables: +GBrain's resolver parser originally required markdown tables: ```markdown | Trigger | Skill | @@ -146,14 +146,14 @@ format that scales better: - **flight-tracker**: track my flight | flight status | when does my flight land ``` -Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a -306-skill compact-format resolver, the doctor reported every skill as -unreachable: **238 FAIL errors on every doctor run**. The parser was -silently treating the compact dialect as zero skills. +When `gbrain doctor` only spoke the table dialect, a 306-skill +compact-format resolver reported every skill as unreachable: **238 FAIL +errors on every doctor run**. The parser was silently treating the compact +dialect as zero skills. -v0.41.7.0 ships dual-format support. The same `parseResolverEntries` -function reads both table rows and list rows in the same file, with the -v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace +Today the parser supports both. The same `parseResolverEntries` +function reads table rows and list rows in the same file, with the +multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace `../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor` and the 238 FAILs collapse to 0. @@ -267,8 +267,7 @@ I initially converted my resolver from a clean list format to a table format because the validator only spoke tables. That was wrong. When a tool fails against valid data, the right move is to fix the tool, not reshape the data. The list format was correct, compact, readable, easy -to maintain. The parser needed to support both shapes. v0.41.7.0 is -that fix. +to maintain. The parser needed to support both shapes — and now it does. The same principle applies everywhere in agent systems. Your SKILL.md is the source of truth. Your AGENTS.md is the source of truth. Your resolver diff --git a/docs/guides/search-modes.md b/docs/guides/search-modes.md index 45a9c0d6b..1e776a132 100644 --- a/docs/guides/search-modes.md +++ b/docs/guides/search-modes.md @@ -1,25 +1,115 @@ # Search Modes -## Goal -Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result. +Two decisions shape every gbrain lookup, and this guide covers both: -## What the User Gets -Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls. +1. **Which mode bundle** your brain runs — `conservative` / `balanced` / + `tokenmax`, the named cost-knob presets that control cache, token budget, + query expansion, and result count. This is the config-level decision you + make once (at `gbrain init` or via `gbrain config set search.mode`). +2. **Which lookup verb** to use per call — `gbrain search` (keyword), + `gbrain query` (hybrid), or `gbrain get` (direct). This is the + per-lookup decision an agent makes on every question. -## Implementation +## The three mode bundles + +A search mode is a named preset that sets every search-cost knob at once. +The bundles are frozen in `src/core/search/mode.ts` (`MODE_BUNDLES`): + +| Knob | `conservative` | `balanced` | `tokenmax` | +|-------------------------------|----------------|------------|----------------| +| `cache.enabled` | true | true | true | +| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 | +| `cache.ttl_seconds` | 3600 | 3600 | 3600 | +| `intentWeighting` | true | true | true | +| `tokenBudget` | **4000** | **12000** | **off** | +| `expansion` (LLM multi-query) | false | false | **true** | +| `relationalRetrieval` | false | **true** | **true** | +| `searchLimit` default | 10 | 25 | 50 | + +- **`conservative`** — smallest payloads. Pairs naturally with a cheap + downstream model (Haiku-class) or a high query volume. +- **`balanced`** — the default and the fallback when no mode is set. +- **`tokenmax`** — no token budget, LLM query expansion on, 50 results. + Pairs with an expensive downstream model you want fully fed. + +Two of the knobs deserve a sentence: + +- **`expansion`** rewrites your query into multiple variants via a cheap + LLM call per search (adds roughly $1.50 per 1K queries) — better recall, + small extra cost. +- **`relationalRetrieval`** adds a graph-walk recall arm for relational + questions ("who invested in X", "what connects A and B"); it's a pure + no-op for non-relational queries. The `query` op's `relational` flag + forces it on/off per call. + +### Setting and resolving the mode + +```bash +gbrain config set search.mode tokenmax +``` + +Per-knob resolution (highest first): + + per-call SearchOpts → per-key config override (search.cache.enabled, …) → + MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback) + +Mode resolution lives in bare `hybridSearch`, not just the cached wrapper, +so eval replays test the same mode-affected behavior as the production +`query` op. The query cache folds the active knobs into its key +(`knobs_hash`), so switching modes never serves you a stale result set +from a different configuration. + +### Cost intuition + +gbrain's own cost is rounding error; what the mode really controls is how +many tokens your *downstream agent* pays to read per query. The +corner-to-corner spread is ~25x once you pair mode with downstream model. +Rough anchors at 10K queries/month, full payload, no cache savings: + +| Mode \ Downstream | Haiku-class (\$1/M in) | Sonnet-class (\$3/M in) | Opus-class (\$5/M in) | +|---|---|---|---| +| conservative (~4K tok) | **\$40/mo** | \$120/mo | \$200/mo | +| balanced (~10K tok) | \$100/mo | \$300/mo | \$500/mo | +| tokenmax (~20K tok) | \$200/mo | \$600/mo | **\$1,000/mo** | + +Scales linearly with volume. Cache hits cut all numbers ~50%; disciplined +prompt caching in the agent loop cuts further. Mismatched pairings waste +capacity in both directions — a tokenmax payload overwhelms a cheap model, +a conservative payload starves an expensive one. The full methodology and +realistic-scale walkthrough live in +[`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md). + +### CLI surfaces + +```bash +gbrain search modes # what is running, with per-knob attribution +gbrain search modes --reset # clear search.* overrides (mode bundle wins) +gbrain search stats [--days N] # cache hit rate, intent mix, budget drops +gbrain search tune [--apply] # data-driven recommendations +gbrain search diagnose "" --target + # trace where a page surfaces (or fails to) + # across the keyword/vector/alias/hybrid layers +``` + +The mode picker runs inside `gbrain init` (non-TTY auto-selects `balanced`). + +## Choosing a lookup verb (search vs query vs get) + +Independent of which bundle is active, every individual lookup should use +the cheapest verb that answers the question. ``` on user_asks_about(topic): - # Decision tree: pick the right search mode + # Decision tree: pick the right lookup verb if know_exact_slug(topic): - # MODE 3: Direct get -- instant, no search overhead + # Direct get -- instant, no search overhead result = gbrain get - # e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi + # e.g., "Tell me about Alice" -> gbrain get alice-example # Returns the FULL page -- compiled truth + timeline elif topic.is_exact_name or topic.is_keyword: - # MODE 1: Keyword search -- fast, no embeddings needed, day-one ready + # Keyword search -- fast, no embeddings needed, day-one ready results = gbrain search "{name_or_keyword}" # e.g., "Find anything about Series A" -> gbrain search "Series A" # Returns CHUNKS, not full pages @@ -30,7 +120,7 @@ on user_asks_about(topic): full_page = gbrain get elif topic.is_semantic_question: - # MODE 2: Hybrid search -- semantic + keyword, needs embeddings + # Hybrid search -- semantic + keyword, needs embeddings results = gbrain query "{natural language question}" # e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts" # Returns ranked chunks via vector + keyword + RRF @@ -40,7 +130,7 @@ on user_asks_about(topic): full_page = gbrain get # Quick reference: -# | Mode | Command | Needs Embeddings | Speed | Best For | +# | Verb | Command | Needs Embeddings | Speed | Best For | # |---------|----------------------|------------------|---------|---------------------------------| # | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches | # | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match | @@ -58,21 +148,22 @@ on user_asks_about(topic): # 4. External sources (web search, APIs) ``` -## Tricky Spots +### Tricky Spots 1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get ` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters. 2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available. -3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug. -4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page. +3. **Don't use hybrid search for known names.** `gbrain query "Alice Example"` wastes embedding compute. Use `gbrain search "Alice Example"` or better yet `gbrain get alice-example` if you know the slug. +4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Alice" -- get the full page. 5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first. -## How to Verify +### How to Verify -1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references. +1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references. 2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech"). -3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline. -4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page. +3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline. +4. Compare: search for the same entity using all three verbs. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page. 5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone. +6. Run `gbrain search modes` -- confirm the active mode bundle and any per-key overrides are what you expect. --- *Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* diff --git a/docs/guides/skill-development.md b/docs/guides/skill-development.md index 2d1a0384e..03f2948e9 100644 --- a/docs/guides/skill-development.md +++ b/docs/guides/skill-development.md @@ -40,7 +40,9 @@ Show the user the results. Get feedback. - Revise the process based on what you learned. **Step 4: Codify into a Skill.** -Write the SKILL.md. Either: +Write the SKILL.md (`gbrain skillify scaffold ` scaffolds the tree for +you; `gbrain skillopt` optimizes an existing skill against a benchmark). +Either: - **New skill** -- genuinely new capability - **Add to existing skill** -- variation of something that exists (parameterize it) @@ -63,7 +65,8 @@ Skills should be **Mutually Exclusive, Collectively Exhaustive**: - Each signal source has exactly ONE owner skill - Two skills creating the same brain page = MECE violation -**Example ownership (no overlap):** +**Example ownership (no overlap — illustrative; your skill roster will +differ):** | Signal Source | Owner Skill | Creates | |--------------|-------------|---------| diff --git a/docs/guides/skillopt.md b/docs/guides/skillopt.md index 2ce805ab7..3ac008bba 100644 --- a/docs/guides/skillopt.md +++ b/docs/guides/skillopt.md @@ -65,14 +65,14 @@ For each step: more than 0.05. 6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe. -After each epoch with no improvement: D6 slow-update fires one meta-edit -proposal (this lives in v0.42 follow-up; v1 emits the audit event). +After each epoch with no improvement: D6 slow-update fires. Today it emits +the audit event only; the full meta-edit proposal is a tracked follow-up. ## Flags | Flag | Default | Purpose | |---|---|---| -| `--benchmark ` | `skills//skillopt-benchmark.jsonl` | Path to benchmark JSONL | +| `--benchmark ` | `skills//skillopt-benchmark.jsonl` | Path to benchmark JSONL | | `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) | | `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) | | `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl | @@ -136,7 +136,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`. - **No benchmark.** Optimizing against guesses is worse than not optimizing. - **Write-flavored skills.** Skills whose job is to `put_page` heavily can't - use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up. + use the read-only sandbox; mocked-write capture is a tracked follow-up. - **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful validation needs ≥20 tasks total per the paper. diff --git a/docs/guides/skillpacks-as-scaffolding.md b/docs/guides/skillpacks-as-scaffolding.md index db33a8a1d..06886ecbb 100644 --- a/docs/guides/skillpacks-as-scaffolding.md +++ b/docs/guides/skillpacks-as-scaffolding.md @@ -1,12 +1,11 @@ # Skillpacks as scaffolding, not amber -GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a -scaffold + reference library. This guide explains the model and the -workflow. +`gbrain skillpack` is a scaffold + reference library, not a package +manager. This guide explains the model and the workflow. -## Why we changed it +## Why it works this way -Pre-v0.33 (the "amber" model): +An earlier design (the "amber" model): - `gbrain skillpack install ` copied bundled skills into your workspace AND wrote a managed-block fence into your `RESOLVER.md` / @@ -26,7 +25,16 @@ repo. You scaffold once, you own them, you fork and edit freely. When gbrain ships a new version, you ask "what changed?" — the agent reads the diff and decides what (if anything) to integrate. -## The five commands +## The core workflow commands + +The five commands below are the scaffold-and-own workflow. The full +`gbrain skillpack` surface is larger (`list`, `diff`, `check`, `search`, +`info`, `registry`, `doctor`, `init`, `pack`, `endorse`, …) — run +`gbrain skillpack --help` for the always-current list. One worth calling +out here: **`gbrain skillpack init-brain-pack `** scaffolds a +*brain-resident* pack inside a brain/source repo (`brain_resident: true` +plus a machine-parseable README) that connecting harnesses discover on +`gbrain sources add`. ### `gbrain skillpack scaffold [--workspace PATH]` @@ -77,15 +85,15 @@ gbrain skillpack reference book-mirror `reference --apply-clean-hunks` is the auto-apply path. It parses the diff between gbrain's bundle and your local copy, applies every hunk whose pre-change context matches uniquely. **Two-way merge -limitation**: without scaffold-time base tracking (intentionally -out-of-scope for v0.33), this cannot distinguish "gbrain changed X" +limitation**: without scaffold-time base tracking (intentionally out of +scope), this cannot distinguish "gbrain changed X" from "you changed X." Applied hunks align everything to gbrain. Use `--dry-run` first to preview, or run plain `reference` to inspect the diff before letting auto-apply touch anything. ### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]` -One-shot conversion for workspaces on the pre-v0.33 managed-block +One-shot conversion for workspaces still on the legacy managed-block model. Strips the `` / `end -->` markers and the manifest receipt comment from your resolver file. @@ -157,7 +165,7 @@ Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the frontmatter, and match the user's intent against every skill's `triggers:` array. When a match scores high enough, invoke that skill. -This replaces the v0.32 model where `gbrain skillpack install` wrote +This replaces the legacy model where `gbrain skillpack install` wrote table rows into your `RESOLVER.md`. Rows are gone (or, for users migrating from the old model, preserved transitionally by `migrate-fence` until they run `scrub-legacy-fence-rows`). @@ -173,7 +181,7 @@ If you're a downstream agent author updating to this model: ## Removing a scaffolded skill -There's no `gbrain skillpack uninstall` command in v0.33. The files +There's no `gbrain skillpack uninstall` command. The files in your `skills//` are first-class members of your repo — delete them like any other code: @@ -195,16 +203,17 @@ You own the files. There's no manifest to update, no fence to rebuild. ## When to use which command (quick decision tree) - **New host repo, want a gbrain skill** → `scaffold` +- **Shipping a pack from inside a brain/source repo** → `init-brain-pack` - **gbrain shipped a new version, want to see what's changed** → `reference` (read-only) or `reference --apply-clean-hunks` (auto) -- **Upgrading from v0.32 or earlier** → `migrate-fence` (one-shot) +- **Upgrading from the legacy managed-block model** → `migrate-fence` (one-shot) - **Cleanup after `migrate-fence`** → `scrub-legacy-fence-rows` - **Lift your fork's skill back into gbrain** → `harvest` + the `skillpack-harvest` editorial skill ## What about `install` and `uninstall`? -Both are removed in v0.33. Running either prints an error pointing at -the replacement command. No deprecated alias — this is a clean break. +Both are removed. Running either prints an error pointing at the +replacement command. No deprecated alias — this is a clean break. If you have existing scripts referencing the old names, update them once and move on. diff --git a/docs/guides/source-attribution.md b/docs/guides/source-attribution.md index 1443c0719..13d9f074c 100644 --- a/docs/guides/source-attribution.md +++ b/docs/guides/source-attribution.md @@ -4,7 +4,7 @@ Every fact in the brain traces to where it came from -- who said it, in what context, and when. ## What the User Gets -Without this: six months from now, someone reads a brain page and has no idea if "Pedro co-founded Brex" came from Pedro himself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality. +Without this: six months from now, someone reads a brain page and has no idea if "Alice co-founded widget-co" came from Alice herself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality. ## Implementation @@ -23,11 +23,11 @@ on brain_write(page, fact): # [Source: Crustdata LinkedIn enrichment, 2026-04-07 12:35 PM PT] elif source.type == "social_media": # MUST include full URL -- not just @handle - # [Source: X/@pedroh96 tweet, product launch, 2026-04-07](https://x.com/pedroh96/status/...) + # [Source: X/@alice_example tweet, product launch, 2026-04-07](https://x.com/alice_example/status/...) elif source.type == "email": - # [Source: email from Sarah Chen re Q2 board deck, 2026-04-05 2:30 PM PT] + # [Source: email from Alice Example re Q2 board deck, 2026-04-05 2:30 PM PT] elif source.type == "workspace": - # [Source: Slack #engineering, Keith re deploy schedule, 2026-04-06 11:45 AM PT] + # [Source: Slack #engineering, Charlie re deploy schedule, 2026-04-06 11:45 AM PT] elif source.type == "web": # [Source: Happenstance research, 2026-04-07 12:35 PM PT] elif source.type == "published": @@ -57,7 +57,7 @@ SOURCE_PRIORITY = [ ## Tricky Spots -1. **Compiled truth is NOT exempt from citations.** "Pedro co-founded Brex" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar. +1. **Compiled truth is NOT exempt from citations.** "Alice co-founded widget-co" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar. 2. **Tweet URLs are mandatory.** `[Source: X/@handle tweet, topic, date]` without a URL is a broken citation. Hundreds of brain pages end up with unreachable tweet references when the URL is omitted. Always: `[Source: X/@handle tweet, topic, date](https://x.com/handle/status/ID)`. 3. **"User said it" isn't enough.** WHERE, ABOUT WHAT, WHEN. `[Source: User, direct message, 2026-04-07 12:33 PM PT]` -- not just `[Source: User]`. 4. **Don't silently resolve conflicts.** When the user says one thing and an API says another, note the contradiction in compiled truth with both citations. Let the reader decide. @@ -71,5 +71,13 @@ SOURCE_PRIORITY = [ 4. Check timeline entries on 3 random pages. Each entry should have a source citation with date and context. 5. Look for a page where the user stated something that contradicts an API result. Confirm the contradiction is noted, not silently resolved. +## Related + +- `skills/_brain-filing-rules.md` — the canonical citation-format rules every + brain write follows (this guide is the narrative walkthrough of the same + contract). +- `skills/citation-fixer/SKILL.md` — audits and repairs existing pages + against that format. + --- *Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).* diff --git a/docs/guides/sub-agent-routing.md b/docs/guides/sub-agent-routing.md index 67907851b..27eb3c34b 100644 --- a/docs/guides/sub-agent-routing.md +++ b/docs/guides/sub-agent-routing.md @@ -2,19 +2,42 @@ ## Goal -Route sub-agents to the cheapest model that can do the job, saving 10-40x on costs without sacrificing quality. +Route sub-agents to the cheapest model that can do the job, saving large +multiples on cost without sacrificing quality. ## What the User Gets -Without this: every sub-agent runs on Opus ($15/MTok). Entity detection on -every message costs $3-5/day. Research tasks cost $10+ each. +Without this: every sub-agent runs on your most expensive model. Entity +detection fires on every message at top-tier rates; research tasks cost +several dollars each. -With this: entity detection runs on Sonnet ($3/MTok, 5x cheaper). Research -runs on DeepSeek ($0.50/MTok, 30x cheaper). Main session stays on Opus for -quality. Total cost drops 70-80%. +With this: entity detection runs on a cheap fast model, research execution +runs on a budget model, and only planning/synthesis touch the expensive +model. Total cost drops 70-80%. + +(Illustrative input-token anchors from gbrain's canonical pricing table, +`src/core/model-pricing.ts`: Opus-class $5/MTok, Sonnet-class $3/MTok, +Haiku-class $1/MTok. Budget providers run well under $1/MTok. Prices +drift — the pricing table is the source of truth, not this doc.) ## Implementation +### GBrain's native mechanism: model tiers + +Before hardcoding vendors, use gbrain's tier routing. Every gbrain +subagent/LLM call resolves through a named tier +(`utility` / `reasoning` / `deep` / `subagent`), and you point each tier +at whatever model you want once: + +```bash +gbrain config set models.tier.subagent anthropic:claude-haiku-4-5 +gbrain config set models.tier.deep anthropic:claude-opus-4-7 +``` + +Per-call override: `gbrain agent run --model `. The +conventions file `skills/conventions/model-routing.md` is the canonical +routing policy; this guide is the cost rationale behind it. + ### Routing Table | Task Type | Recommended Model | Why | @@ -44,14 +67,16 @@ on_every_message(text): 3. FACTS: New info about existing entities -> update timeline 4. CITATIONS: Every fact needs [Source: ...] attribution 5. Sync changes to brain repo`, - model: "sonnet-class", // fast + cheap + model: "sonnet-class", // fast + cheap; haiku-class is cheaper still timeout: 120s }) ``` -**Why Sonnet-class for detection:** Entity detection is pattern matching, not -deep reasoning. Sonnet is 5-10x cheaper than Opus and fast enough for async -detection. The main session continues on Opus while detection runs in parallel. +**Why a cheaper class for detection:** Entity detection is pattern matching, +not deep reasoning. Sonnet-class runs at a fraction of Opus-class cost, and +Haiku-class at a fraction of that — both fast enough for async detection. +The main session continues on your best model while detection runs in +parallel. ### Research Pipeline Pattern @@ -64,9 +89,9 @@ For research-heavy tasks, use a multi-model pipeline: ``` **Why this works:** The planning and synthesis steps need taste and judgment -(Opus). The execution step is mechanical data gathering (DeepSeek at 25-40x -lower cost). You get Opus-quality output at DeepSeek-level cost for 80% of -the work. +(Opus-class). The execution step is mechanical data gathering (a budget +model at a small fraction of the cost). You get top-tier output at +budget-model cost for 80% of the work. ### When to Spawn Sub-Agents @@ -82,15 +107,16 @@ the work. The main session runs on your best model. Everything else runs on the cheapest model that can do the job. In practice, 60-70% of sub-agent -work is entity detection (Sonnet) and research execution (DeepSeek), -which are 10-40x cheaper than the main session model. +work is entity detection and research execution, which run at a small +fraction of the main session model's cost. ## Tricky Spots -1. **Sonnet, not Opus, for detection.** The most common mistake is running - entity detection on Opus. Detection is pattern matching, not deep reasoning. - Sonnet is 5-10x cheaper and fast enough. Reserve Opus for the main session - where reasoning quality matters. +1. **A cheap class, not Opus, for detection.** The most common mistake is + running entity detection on Opus-class. Detection is pattern matching, not + deep reasoning. Sonnet- or Haiku-class is several times cheaper and fast + enough. Reserve Opus-class for the main session where reasoning quality + matters. 2. **Don't block the main thread.** Sub-agents must run asynchronously. If the signal detector runs synchronously, the user waits 30-120 seconds for every @@ -98,10 +124,11 @@ which are 10-40x cheaper than the main session model. a response immediately. 3. **Cost optimization is multiplicative.** Entity detection runs on every - single message. If you use Opus at $15/MTok for detection across 50 - messages/day, that's $3-5/day just for detection. Sonnet at $3/MTok brings - that to $0.60-1.00/day. Over a month, the wrong model choice costs $100+ - more than necessary. + single message, so the per-call price difference compounds across 50+ + messages/day. Routing detection from Opus-class ($5/MTok in) to + Haiku-class ($1/MTok in) is a flat 5x cut on your highest-frequency LLM + call — over a month, the wrong model choice for detection alone costs + real money. (Current per-model rates: `src/core/model-pricing.ts`.) ## How to Verify diff --git a/docs/guides/upgrades-auto-update.md b/docs/guides/upgrades-auto-update.md index 40ead2b4a..8d6b13de6 100644 --- a/docs/guides/upgrades-auto-update.md +++ b/docs/guides/upgrades-auto-update.md @@ -16,9 +16,9 @@ benefit-focused bullets, waits for explicit permission, then runs the full upgrade flow including re-reading skills, running migrations, and syncing schema. The user gets new capabilities automatically. -## Self-upgrade modes (v0.42) +## Self-upgrade modes -gbrain now stays current the way gstack does: it rides invocation frequency. A +gbrain stays current the way gstack does: it rides invocation frequency. A throttled, cache-read-only check runs at the start of every `gbrain` invocation (CLI and MCP) and emits an `UPGRADE_AVAILABLE ` marker on stderr. No host cron required — every agent kind (Claude Code, Codex, OpenClaw, Hermes, the @@ -44,6 +44,14 @@ execution. The trust model is TLS + GitHub (same as `gbrain upgrade`); signature verification is a tracked follow-up. Apply manually any time with `gbrain self-upgrade`. +The `auto` quiet-hours window is configured via the +`self_upgrade.quiet_hours` config key +(`gbrain config set self_upgrade.quiet_hours '{"start":23,"end":8,"tz":"US/Pacific"}'`). +The quiet-hours *pattern* itself — gating any notification or background +action on the user's local sleep window — is owned by +[quiet-hours.md](quiet-hours.md); this doc only covers the self-upgrade +hook into it. + ## Implementation ### The Check (cron-initiated) @@ -163,15 +171,18 @@ Also persist in `~/.gbrain/update-state.json` so it survives agent context reset If you loaded this SKILLPACK directly (copied or read from GitHub) without installing gbrain, you can still stay current. Both GBRAIN_SKILLPACK.md and -GBRAIN_RECOMMENDED_SCHEMA.md have version markers: +GBRAIN_RECOMMENDED_SCHEMA.md carry a `` header pointing +at their canonical copies, and GBRAIN_RECOMMENDED_SCHEMA.md also carries a +version marker: ```bash -curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md | head -1 -# Returns: +curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md | head -1 +# Returns: ``` -If the remote version is newer, fetch the full file and replace your local -copy. Set up a weekly cron to check automatically. +If the remote version is newer (or the remote SKILLPACK content differs from +your local copy), fetch the full file and replace your local copy. Set up a +weekly cron to check automatically. ## Tricky Spots diff --git a/docs/mcp/ALTERNATIVES.md b/docs/mcp/ALTERNATIVES.md index 49c68345e..2605c6943 100644 --- a/docs/mcp/ALTERNATIVES.md +++ b/docs/mcp/ALTERNATIVES.md @@ -2,8 +2,9 @@ GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it accessible from other devices and AI clients, run `gbrain serve --http` -(built-in HTTP transport with bearer auth, Postgres-only ... see -[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options. +(built-in HTTP transport with OAuth 2.1 + bearer auth, works on both PGLite +and Postgres brains — see [DEPLOY.md](DEPLOY.md)) behind a public tunnel. +Here are your tunnel options. ## ngrok (recommended) @@ -58,10 +59,11 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits. | Works when laptop is off | No | No | Yes | | Cold start | None | None | None | | Timeout limits | None | None | None | -| All 30 operations | Yes | Yes | Yes | +| Full remote operation surface (100+ ops, minus `localOnly`) | Yes | Yes | Yes | | Setup time | 5 min | 10 min | 15 min | -**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth -against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap, -per-request audit log. Postgres-only by design (PGLite is local-only). See -[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables. +**Note:** `gbrain serve --http` is the built-in HTTP transport. OAuth 2.1 plus +bearer auth against the `access_tokens` table, default-deny CORS, two-bucket rate +limit, body cap, per-request audit log. Works on both PGLite and Postgres brains. +See [DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and +tunables. diff --git a/docs/mcp/CHATGPT.md b/docs/mcp/CHATGPT.md index 0da8d6549..47b2974e9 100644 --- a/docs/mcp/CHATGPT.md +++ b/docs/mcp/CHATGPT.md @@ -1,35 +1,39 @@ # Connect GBrain to ChatGPT -**Status (v0.26.0):** Unblocked. GBrain's `gbrain serve --http` ships OAuth 2.1 -with PKCE, which is the ChatGPT MCP connector's hard requirement. Before v1.0, -this was a P0 TODO — the only major AI client that could not connect. +ChatGPT's MCP connector requires OAuth 2.1 with PKCE — it does not support +bearer-token MCP servers. GBrain's `gbrain serve --http` speaks exactly that, +so ChatGPT connects natively. -ChatGPT does not support bearer-token MCP servers. You must use the OAuth 2.1 -HTTP server. +This page covers only the ChatGPT-specific parts. The full server setup — +starting `gbrain serve --http`, the admin bootstrap token, the `/admin` +dashboard, tunnels, and `--bind` / `--public-url` — lives in +[DEPLOY.md](DEPLOY.md). Do steps 1 (start the server) and 3 (expose it) +from there, then come back for the ChatGPT client. ## Setup -### 1. Start the HTTP server +### 1. Start and expose the server (DEPLOY.md steps 1 + 3) -```bash -gbrain serve --http --port 3131 -``` - -Save the admin bootstrap token printed on stderr. Open -`http://localhost:3131/admin` and paste it to access the dashboard. +Follow [DEPLOY.md — OAuth 2.1 Setup](DEPLOY.md#oauth-21-setup) to start +`gbrain serve --http`, save the admin bootstrap token, and expose the server +at a public HTTPS URL (e.g. `https://your-brain.ngrok.app`). ChatGPT's +connector auto-discovers the spec-compliant endpoint at +`/.well-known/oauth-authorization-server`. ### 2. Register a ChatGPT client -ChatGPT uses the authorization code flow with PKCE (browser-based OAuth). -Register from the `/admin` dashboard: +The ChatGPT-specific delta: ChatGPT uses the **authorization code flow with +PKCE** (browser-based OAuth), so the client needs the `authorization_code` +grant type and a redirect URI. Register from the `/admin` dashboard: 1. Click **Register client**. 2. Name: `chatgpt`. 3. Grant type: `authorization_code`. 4. Scopes: `read`, `write` (leave `admin` unchecked for ChatGPT). -5. Redirect URI: ChatGPT's OAuth redirect (copy it from the ChatGPT - connector setup screen — something like - `https://chat.openai.com/connector_platform_oauth_redirect`). +5. Redirect URI: ChatGPT's OAuth redirect — **always copy the exact value + from the ChatGPT connector setup screen** (it looks like + `https://chatgpt.com/connector_platform_oauth_redirect`, but the domain + has changed before; trust the setup screen, not this doc). 6. Hit **Register**. The credential-reveal modal shows the `client_id` once with Copy and Download JSON buttons. There is no client secret for PKCE-based public clients. @@ -41,22 +45,11 @@ await oauthProvider.registerClientManual( 'chatgpt', ['authorization_code'], 'read write', - ['https://chat.openai.com/connector_platform_oauth_redirect'], + [''], ); ``` -### 3. Expose the server publicly - -```bash -brew install ngrok -ngrok http 3131 --url your-brain.ngrok.app -``` - -Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. ChatGPT's -connector auto-discovers the spec-compliant endpoint at -`/.well-known/oauth-authorization-server`. - -### 4. Add the connector in ChatGPT +### 3. Add the connector in ChatGPT 1. Open ChatGPT > Settings > Connectors. 2. Click **Add connector**. @@ -71,10 +64,11 @@ calls show up in the admin dashboard's live SSE feed in real time. ## Scopes ChatGPT clients can request any combination of `read`, `write`, `admin`. The -scopes granted at consent time are enforced on every tool call. Four -operations are `localOnly` and rejected over HTTP regardless of scope: -`sync_brain`, `file_upload`, `file_list`, `file_url`. The HTTP server fails -closed for any attempt to reach local filesystem surface area. +scopes granted at consent time are enforced on every tool call. Operations +flagged `localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain` +and the `file_*` ops among them) are rejected over HTTP regardless of scope. +The HTTP server fails closed for any attempt to reach local filesystem +surface area. Recommended ChatGPT scope: `read write`. Leave `admin` for your local CLI and the admin dashboard. diff --git a/docs/mcp/CLAUDE_CODE.md b/docs/mcp/CLAUDE_CODE.md index 171a75eb5..4eec8a3b2 100644 --- a/docs/mcp/CLAUDE_CODE.md +++ b/docs/mcp/CLAUDE_CODE.md @@ -81,11 +81,10 @@ search for [any topic in your brain] You should see results from your GBrain knowledge base. > **`list_skills` returns nothing?** Skill discovery is gated by `mcp.publish_skills` -> on the host. New brains from `gbrain init` default it ON; brains upgraded from an -> older release stay OFF until you opt in. Enable it on the host with -> `gbrain config set mcp.publish_skills true`. The core tools (search, query, -> get_page, put_page, think, find_experts) work regardless. Note: `capture` is a -> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`. +> on the host — enable it with `gbrain config set mcp.publish_skills true`. The core +> tools (search, query, get_page, put_page, think, find_experts) work regardless; +> `capture` is CLI-only, so agents write over MCP with `put_page`. Why brains differ +> on the default: [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http). ## Remove diff --git a/docs/mcp/CLAUDE_COWORK.md b/docs/mcp/CLAUDE_COWORK.md index 35e5601bc..41a4e94e8 100644 --- a/docs/mcp/CLAUDE_COWORK.md +++ b/docs/mcp/CLAUDE_COWORK.md @@ -27,6 +27,19 @@ Desktop bridges local MCP servers into Cowork via its SDK layer. This means: if `gbrain serve` is running and configured in Claude Desktop, you don't need a separate server for Cowork. +## Verify + +In a Cowork session, try: + +``` +Call get_brain_identity, then search my brain for [any topic] +``` + +You should get pages from your brain back. If `list_skills` returns nothing, +skill publishing is off on the host — enable it with +`gbrain config set mcp.publish_skills true` (see +[CLAUDE_CODE.md](CLAUDE_CODE.md) for the full gotcha). + ## Which to use? - **Remote server:** works even when your laptop is closed, available to all org members diff --git a/docs/mcp/CLAUDE_DESKTOP.md b/docs/mcp/CLAUDE_DESKTOP.md index 5df65cb25..921e85d5e 100644 --- a/docs/mcp/CLAUDE_DESKTOP.md +++ b/docs/mcp/CLAUDE_DESKTOP.md @@ -1,5 +1,10 @@ # Connect GBrain to Claude Desktop +This page covers connecting Claude Desktop to a **remote** brain. For a brain +on the same machine as Claude Desktop, a local stdio entry in +`claude_desktop_config.json` with `"command": "gbrain", "args": ["serve"]` +works too — but only against a full local install, never a thin-client one. + **Important:** Claude Desktop does NOT connect to remote MCP servers via `claude_desktop_config.json`. That file only works for local stdio servers. Remote HTTP servers must be added through the GUI. diff --git a/docs/mcp/CODEX.md b/docs/mcp/CODEX.md index 4e8d4410d..e96a46f83 100644 --- a/docs/mcp/CODEX.md +++ b/docs/mcp/CODEX.md @@ -9,9 +9,9 @@ > durable body — not just a connection? That's `gbrain bootstrap`: see the paste > block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md). -Codex CLI (`@openai/codex`, v0.130+) supports remote streamable-HTTP MCP servers -with a bearer token read from an environment variable. The token lives in your -shell env, not in Codex's config file. +Recent versions of the Codex CLI (`@openai/codex`) support remote +streamable-HTTP MCP servers with a bearer token read from an environment +variable. The token lives in your shell env, not in Codex's config file. ## Fastest path: `gbrain connect` @@ -55,11 +55,11 @@ Call get_brain_identity, then search my brain for [topic]. `get_brain_identity` confirms whose brain you're connected to; `list_skills` shows everything it can do. -> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host (default -> ON for `gbrain init` brains, OFF for brains upgraded from older releases). Enable -> it on the host: `gbrain config set mcp.publish_skills true`. The core tools -> (search, query, get_page, put_page, think, find_experts) work regardless. -> `capture` is CLI-only, not an MCP tool — write over MCP with `put_page`. +> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host — enable +> it with `gbrain config set mcp.publish_skills true`. The core tools (search, +> query, get_page, put_page, think, find_experts) work regardless; `capture` is +> CLI-only, so write over MCP with `put_page`. Why brains differ on the default: +> [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http). ## Remove diff --git a/docs/mcp/DEPLOY.md b/docs/mcp/DEPLOY.md index d7fab9013..42330f1d2 100644 --- a/docs/mcp/DEPLOY.md +++ b/docs/mcp/DEPLOY.md @@ -1,17 +1,16 @@ # Deploy GBrain Remote MCP Server -> **v0.26.0+:** `gbrain serve --http` ships full OAuth 2.1 (client credentials, -> auth code + PKCE, refresh rotation, optional DCR), an embedded React admin -> dashboard at `/admin`, scoped operations, and a live SSE activity feed. -> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back -> to the `access_tokens` table and grandfathers tokens to `read+write+admin`. -> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only); -> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md) -> for env vars and tunable defaults. +> `gbrain serve --http` ships full OAuth 2.1 (client credentials, auth code + +> PKCE, refresh rotation, optional DCR), an embedded React admin dashboard at +> `/admin`, scoped operations, and a live SSE activity feed. Legacy bearer +> tokens still work — `verifyAccessToken` falls back to the `access_tokens` +> table and grandfathers tokens to `read+write+admin`. Both the OAuth surface +> and the bearer fallback work on both engines (PGLite and Postgres). See +> [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults. Access your brain from any device, any AI client. GBrain ships two transports: -`gbrain serve` (stdio) for local agents, and `gbrain serve --http` (v0.26.0+) -for remote clients over OAuth 2.1. +`gbrain serve` (stdio) for local agents, and `gbrain serve --http` for remote +clients over OAuth 2.1. ## Three Paths @@ -24,7 +23,7 @@ gbrain serve Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio. No server, no tunnel, no token needed. Works on both PGLite and Postgres engines. -### Remote over OAuth 2.1 (recommended, v0.26.0+) +### Remote over OAuth 2.1 (recommended) ```bash gbrain serve --http --port 3131 @@ -45,28 +44,27 @@ Supported clients: - **Perplexity** — OAuth 2.1 client credentials grant. - **Claude Code, Cursor, Windsurf** — can use OAuth or legacy bearer. -See the [OAuth 2.1 setup](#oauth-21-setup-v100) section below. +See the [OAuth 2.1 setup](#oauth-21-setup) section below. -### Remote with legacy bearer tokens (pre-v0.26 deployments) — Postgres only +### Remote with legacy bearer tokens (simplest) ``` Your AI client (Claude Desktop, Perplexity, etc.) → ngrok tunnel (https://YOUR-DOMAIN.ngrok.app) → gbrain serve --http (built-in transport with bearer auth) - → Postgres (pooler connection or self-hosted) + → Postgres or PGLite ``` This requires: -1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres; - running `gbrain serve --http` against a PGLite install fails fast at startup) -2. A machine running `gbrain serve --http` -3. A public tunnel (ngrok, Tailscale, or cloud host) -4. A bearer token created via `gbrain auth create ` +1. A machine running `gbrain serve --http` (works on both PGLite and Postgres + brains) +2. A public tunnel (ngrok, Tailscale, or cloud host) +3. A bearer token created via `gbrain auth create ` -Pre-v1.0 tokens are grandfathered as `read+write+admin` scopes when you upgrade -to the HTTP server, so no migration is required. +Existing bearer tokens are grandfathered as `read+write+admin` scopes on the +OAuth-capable HTTP server, so no migration is required. -## OAuth 2.1 Setup (v0.26.0+) +## OAuth 2.1 Setup ### 1. Start the HTTP server @@ -92,8 +90,8 @@ Save this token. Open `http://localhost:3131/admin` and paste it to access the dashboard. The dashboard shows live activity, registered clients, request logs, and per-client config export. -> **v0.26.9+:** `mcp_request_log.params` and the live SSE activity feed default -> to a redacted summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. +> `mcp_request_log.params` and the live SSE activity feed default to a redacted +> summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. > Declared param keys are kept (intersected against the operation's spec); unknown > keys are counted but never named, and byte sizes round up to 1KB so size-probe > attacks can't binary-search secret content. Operators on a personal laptop who @@ -124,9 +122,9 @@ gbrain auth register-client perplexity \ --scopes "read write" ``` -**v0.34 — source-scoped clients.** Multi-source brains can scope a client's -write authority to one source and its read scope to a curated set with the -new `--source` and `--federated-read` flags: +**Source-scoped clients.** Multi-source brains can scope a client's write +authority to one source and its read scope to a curated set with the +`--source` and `--federated-read` flags: ```bash gbrain auth register-client dept-x-agent \ @@ -138,9 +136,12 @@ gbrain auth register-client dept-x-agent \ `--source` controls the write authority — `put_page` / `add_link` / etc only land in `dept-x`. `--federated-read` controls the read axis independently; -queries return rows from any of the listed sources. Omit both flags for the -v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to -`source_id='default'` on `gbrain upgrade`. +queries return rows from any of the listed sources. Omit both flags for an +unscoped super-client. Clients registered before source scoping existed are +backfilled to `source_id='default'` on `gbrain upgrade`. Within a source, +slug-level write fencing is also available: `--bound-slug-prefixes p1/,p2/` +rejects slug-mutating writes outside the listed prefixes (update later with +`gbrain auth rescope-client --bound-slug-prefixes `). Host-repo wrappers can register programmatically: @@ -158,7 +159,7 @@ start the server with `--enable-dcr`. DCR is off by default. ### 3. Expose the server -**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`. +**Bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`. To accept connections from the ngrok tunnel (or any non-loopback source), restart with `--bind`: @@ -182,10 +183,10 @@ router exposes the spec-compliant discovery endpoint at ### 4. Scopes and localOnly -Every operation is tagged `read | write | admin`. Four operations are -`localOnly` and rejected over HTTP regardless of scope: `sync_brain`, -`file_upload`, `file_list`, `file_url`. Remote agents cannot reach local -filesystem surface area. +Every operation is tagged `read | write | admin`. Operations flagged +`localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain` and +the `file_*` ops among them) are rejected over HTTP regardless of scope. +Remote agents cannot reach local filesystem surface area. | Scope | What it allows | |-------|---------------| @@ -193,10 +194,13 @@ filesystem surface area. | `write` | `put_page`, `delete_page`, `add_link`, `add_timeline_entry` | | `admin` | Client management, token revocation, sweep, local-only ops | +Write ops can additionally be fenced per client with `--bound-slug-prefixes` +(see [Register OAuth clients](#2-register-oauth-clients) above). + ## Legacy Bearer Token Setup -Keep using pre-v0.26 bearer tokens if you aren't ready to migrate. They -grandfather to `read+write+admin` scopes on the HTTP server. +Bearer tokens are the simple path when you don't need per-client scoping. +They grandfather to `read+write+admin` scopes on the HTTP server. ### 1. Set up the tunnel @@ -243,15 +247,20 @@ gbrain auth test \ ## Operations -All 30 GBrain operations are available remotely, including `sync_brain` and -`file_upload` (no timeout limits with self-hosted server). +GBrain's full operation catalog (100+ operations in `src/core/operations.ts`) +is available remotely, with no timeout limits on a self-hosted server. The +only exceptions are the operations flagged `localOnly: true` — `sync_brain` +and the `file_*` ops among them — which are rejected over HTTP regardless of +scope (see [Scopes and localOnly](#4-scopes-and-localonly) above). -**Security note on `file_upload`:** remote MCP callers are confined to the working -directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute -paths outside cwd are rejected. Page slugs and filenames are allowlist-validated -(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local -CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since -the user owns the machine. +**Security note on file access:** the `file_*` operations being localOnly is +the first line of defense; as defense-in-depth, `file_upload` also confines +any caller that isn't verifiably the trusted local CLI to the working +directory where `gbrain serve` was launched. Symlinks, `..` traversal, and +absolute paths outside cwd are rejected, and page slugs and filenames are +allowlist-validated (alphanumeric + hyphens; no control chars, RTL overrides, +or backslashes). Local CLI callers (`gbrain files upload ...`) keep +unrestricted filesystem access since the user owns the machine. ## Deployment Options @@ -321,8 +330,8 @@ Remote servers must be added via Settings > Integrations, NOT | put_page | 100-500ms | Write + trigger search_vector update | | get_stats | < 100ms | Aggregate query | -**Note:** `gbrain serve --http` shipped in v0.26.0 with OAuth 2.1 + admin -dashboard baked into the binary. The custom HTTP wrapper pattern (see +**Note:** `gbrain serve --http` has OAuth 2.1 + the admin dashboard baked +into the binary. The custom HTTP wrapper pattern (see [voice recipe](../../recipes/twilio-voice-brain.md)) is still supported for teams that need bespoke middleware, but for most remote deployments the built-in server is the recommended path. diff --git a/docs/mcp/PERPLEXITY.md b/docs/mcp/PERPLEXITY.md index fb6595f89..d0dd4cf5f 100644 --- a/docs/mcp/PERPLEXITY.md +++ b/docs/mcp/PERPLEXITY.md @@ -18,13 +18,16 @@ gbrain serve --http --port 3131 --bind 0.0.0.0 \ --public-url https://YOUR-DOMAIN.ngrok.app ``` -- **`--bind 0.0.0.0` is required.** Since v0.34, `--http` defaults to - `127.0.0.1`, so without it the tunnel reaches the server but the connection is - refused (`ECONNREFUSED`). +- **`--bind 0.0.0.0` is required.** `--http` defaults to `127.0.0.1`, so + without it the tunnel reaches the server but the connection is refused + (`ECONNREFUSED`). - **`--public-url` must match the tunnel.** The OAuth issuer in the discovery metadata has to line up with the URL Perplexity actually hits (RFC 8414 §3.3), or OAuth client-credentials auth fails. +Full detail on both flags (and the rest of the server setup) lives in +[DEPLOY.md — Expose the server](DEPLOY.md#3-expose-the-server). + ## 2. Expose it with a tunnel ```bash @@ -36,9 +39,12 @@ tunnel. ## 3. Create credentials -Two supported auth paths. +Two supported auth paths. (Full client-registration mechanics — the `/admin` +dashboard flow, grant types, scope format — live in +[DEPLOY.md — Register OAuth clients](DEPLOY.md#2-register-oauth-clients); +below is the Perplexity-specific shape.) -**OAuth 2.1 client credentials (recommended, v0.26.0+).** Perplexity is a cloud +**OAuth 2.1 client credentials (recommended).** Perplexity is a cloud service, so it holds whatever credential you give it. OAuth is the correct choice: least-privilege scopes + short-lived rotating access tokens instead of a long-lived full-access secret. Mint a client and print the connector fields in diff --git a/docs/operations/headless-install.md b/docs/operations/headless-install.md index b0ba50fb1..d14c04af6 100644 --- a/docs/operations/headless-install.md +++ b/docs/operations/headless-install.md @@ -1,8 +1,8 @@ # Headless install: Docker, CI, postinstall -As of v0.37, `gbrain init --pglite` in a non-TTY context (Docker `RUN`, CI step, postinstall hook) exits 1 when no embedding-provider API key is present in the environment. This is a deliberate fail-loud — the alternative was the v0.36 silent-broken-state class where init succeeded with a default that didn't match any real key. +`gbrain init --pglite` in a non-TTY context (Docker `RUN`, CI step, postinstall hook) exits 1 when no embedding-provider API key is present in the environment. This is a deliberate fail-loud — the alternative is a silent-broken state where init succeeds with a default that doesn't match any real key. -Two patterns work for headless installs. Pick whichever fits your image lifecycle. +Three patterns work for headless installs. Pick whichever fits your image lifecycle. ## Pattern 1: Provider key available at image build time @@ -16,7 +16,7 @@ FROM oven/bun:1 AS builder ARG OPENAI_API_KEY ENV OPENAI_API_KEY=$OPENAI_API_KEY -RUN bun install -g github:garrytan/gbrain +RUN bun install -g github:garrytan/gbrain#latest-stable RUN gbrain init --pglite # auto-picks OpenAI, persists config ``` @@ -26,7 +26,7 @@ RUN gbrain init --pglite # auto-picks OpenAI, persists config env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} run: | - bun install -g github:garrytan/gbrain + bun install -g github:garrytan/gbrain#latest-stable gbrain init --pglite ``` @@ -38,7 +38,7 @@ If the API key is a runtime secret (Kubernetes secret, runtime env injection, en ```dockerfile FROM oven/bun:1 -RUN bun install -g github:garrytan/gbrain +RUN bun install -g github:garrytan/gbrain#latest-stable # Build the brain shape without a provider — schema lands at the default # width, but no embed callsite will actually run until runtime config. @@ -59,6 +59,20 @@ The runtime `gbrain init --force` re-runs the init flow against the now-populate - Resolves the provider via env detection. - Re-templates the PGLite schema if dim differs from the build-time default. +## Pattern 3: No key, ever (keyless mode) + +`--no-embedding` isn't only a deferral — it's also the install shape for **keyless mode**, a first-class supported end state (not a broken one). With zero provider keys, gbrain runs keyword-only (BM25) search and takes memory from agent-authored `## Facts` fences and write ops; embedding and extraction paths refuse cleanly instead of failing silently. + +```dockerfile +FROM oven/bun:1 +RUN bun install -g github:garrytan/gbrain#latest-stable +RUN gbrain init --pglite --no-embedding # keyless install — done; no runtime re-init needed +``` + +`gbrain bootstrap verify` (and the agent-bootstrap flow generally) prints an honest capability report for this posture — `gbrain capabilities: keyless mode`, per-touchpoint lines, and the one-key upsell (`src/core/capability.ts`). Keyless installs for the agent-bootstrap path are covered in `docs/guides/bootstrap.md`; this doc covers the Docker/CI shape. Adding a single provider key later upgrades in place via Pattern 2's runtime `gbrain init --force`. + +Since every embedding cost gate is structurally moot with no key, none of `docs/operations/spend-controls.md` applies until you add one. + ## What WON'T work ```dockerfile @@ -67,7 +81,7 @@ The runtime `gbrain init --force` re-runs the init flow against the now-populate RUN gbrain init --pglite ``` -If you upgrade from a pre-v0.37 image that used this pattern, `gbrain doctor` will surface the mismatch on first run after upgrade and print a paste-ready repair command (`gbrain init --force --embedding-model …` for empty brains, `gbrain retrieval-upgrade --reindex` for non-empty). +If an older image used this pattern, `gbrain doctor` will surface the mismatch on first run after upgrade and print a paste-ready repair command — `gbrain init --force --pglite --embedding-model --embedding-dimensions ` for brains with no embeddings yet, `gbrain migrate embeddings --to --dim ` for non-empty brains. ## Verifying a headless install diff --git a/docs/operations/spend-controls.md b/docs/operations/spend-controls.md index 0af1f2c92..967f8c2f1 100644 --- a/docs/operations/spend-controls.md +++ b/docs/operations/spend-controls.md @@ -8,6 +8,11 @@ The orienting idea: **GBrain itself is rounding error; the spend that matters is downstream embedding.** These gates exist so a routine sync or enrich can't run up an unexpected embedding bill, while never wedging an unattended cron. +**Keyless mode:** if you run with zero provider keys (`gbrain init --no-embedding`, +the keyless bootstrap posture — see `docs/guides/bootstrap.md` and +`docs/operations/headless-install.md`), nothing here can spend and none of these +gates ever fire. This doc applies once you add a key. + ## `spend.posture` — one switch for "cost is not my constraint" ```bash @@ -18,7 +23,7 @@ gbrain config set spend.posture gated # default — gates enforce | Value | Effect | |-------|--------| | `gated` (default) | Every cost gate enforces its limit as documented below. | -| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. | +| `tokenmax` | Every embedding-spend gate in the table below prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. (Commands with their own LLM cost caps outside this doc's embedding scope — e.g. `extract-conversation-facts --max-cost-usd` — don't resolve posture; their per-call flags govern.) | `spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs retrieval payload size, not embedding spend). When a gate fires and diff --git a/docs/schema-author-tutorial.md b/docs/schema-author-tutorial.md index e871703c5..6924e4939 100644 --- a/docs/schema-author-tutorial.md +++ b/docs/schema-author-tutorial.md @@ -212,13 +212,12 @@ gbrain schema lint --with-db **Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal). -**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. +**For agents (MCP):** the same operations are reachable over HTTPS MCP as schema ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. **Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`. ## Related docs -- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture. +- **Reference:** `gbrain schema --help` for the full CLI surface (30+ subcommands); the "Schema Cathedral v3" section of `docs/architecture/KEY_FILES.md` for the module-by-module architecture. - **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit). - **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix. -- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private). diff --git a/docs/skillpack-anatomy.md b/docs/skillpack-anatomy.md index b925a5030..70aa5a1a1 100644 --- a/docs/skillpack-anatomy.md +++ b/docs/skillpack-anatomy.md @@ -105,6 +105,21 @@ gbrain skillpack scaffold # owner/repo, https, ./dir, ./*.tgz gbrain skillpack registry --url X # point at a custom registry ``` +## Brain-resident packs + +A brain/source repo can carry its own publishable skillpack (`brain_resident: true` +in `skillpack.json`, plus a `schema_pack` declaration). Scaffold one with: + +```bash +gbrain skillpack init-brain-pack # inside the brain repo; --dry-run to preview +``` + +Connecting harnesses discover the pack on `gbrain sources add`, and remote +agents reach it over MCP via the source-scoped `list_brain_skillpack` op + +`get_skill --source_id` (gated by the `mcp.publish_skills` config key). The +anatomy above applies unchanged — a brain-resident pack is a normal pack that +happens to live inside a brain repo. + ## See also - `examples/skillpack-reference/` — the live 10/10 reference pack diff --git a/docs/storage-tiering.md b/docs/storage-tiering.md index e5da8be7d..8514c8ea8 100644 --- a/docs/storage-tiering.md +++ b/docs/storage-tiering.md @@ -4,7 +4,7 @@ GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database. -> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands. +> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning; rename them in your config by hand to silence it. ## Configuration diff --git a/docs/takes-vs-facts.md b/docs/takes-vs-facts.md index f11568c69..30d7b4bf6 100644 --- a/docs/takes-vs-facts.md +++ b/docs/takes-vs-facts.md @@ -14,10 +14,10 @@ The epistemological layer. WHO believes WHAT, with confidence weight and time. - **Scale:** 100K+ rows across thousands of holders in a mature brain **Example takes:** -- `holder=people/garry-tan kind=bet` "AI will replace 50% of coding by 2030" (w=0.75) -- `holder=people/jared-friedman kind=take` "Momo has strong retention" (w=0.80) -- `holder=world kind=fact` "Clipboard raised $100M Series C" (w=1.0) -- `holder=brain kind=hunch` "Garry has a hero/rescuer pattern" (w=0.70) +- `holder=people/alice-example kind=bet` "AI will replace 50% of coding by 2030" (w=0.75) +- `holder=people/bob-example kind=take` "widget-co has strong retention" (w=0.80) +- `holder=world kind=fact` "acme-example raised a Series C" (w=1.0) +- `holder=brain kind=hunch` "alice-example has a hero/rescuer pattern" (w=0.70) **Query surface:** `gbrain takes list`, `gbrain takes search`, `gbrain think` @@ -32,9 +32,9 @@ Personal knowledge from the brain owner's conversations. Real-time capture. - **Bridge:** Dream cycle `consolidate` phase promotes hot facts → cold takes nightly **Example facts:** -- `kind=event` "I have a meeting with Brian tomorrow" +- `kind=event` "I have a meeting with alice-example tomorrow" - `kind=preference` "I don't drink coffee" -- `kind=commitment` "We decided on nesting custody" +- `kind=commitment` "We decided to move the offsite to March" - `kind=belief` "I think the market is overheated" **Query surface:** `gbrain recall`, MCP `_meta.brain_hot_memory` @@ -42,8 +42,8 @@ Personal knowledge from the brain owner's conversations. Real-time capture. ## The Category Error **Never dump takes into the facts table.** Takes include other people's attributed -beliefs (Jared's assessment of a company, PG's view on schools, a founder's -revenue claims). These are NOT the brain owner's personal facts. +beliefs (a partner's assessment of a company, an investor's view on markets, a +founder's revenue claims). These are NOT the brain owner's personal facts. **Never dump facts into the takes table without transformation.** Facts are scoped to what the owner said in conversation. They become takes only through @@ -85,7 +85,7 @@ First full takes extraction run on a ~100K-page brain: ### Key Learnings for Extraction Prompts -1. **Holder ≠ subject.** "Garry has a hero/rescuer pattern" → holder=brain, NOT people/garry-tan +1. **Holder ≠ subject.** "alice-example has a hero/rescuer pattern" → holder=brain, NOT people/alice-example 2. **Atomic claims.** Split compound claims into separate rows 3. **Amplification ≠ endorsement.** Retweet-only → max weight 0.55 4. **Self-reported ≠ verified.** "Reports 7 figures" → holder=person, weight=0.75, NOT world/1.0 diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index daf3332b0..d5fe8f167 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -32,5 +32,6 @@ Tutorials follow the [Diataxis](https://diataxis.fr/) tutorial pattern: learning - **Reference:** [`docs/architecture/`](../architecture/) — system design, topologies, retrieval theory - **How-to:** [`docs/guides/`](../guides/) — task-oriented runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion). Highlight: [scaling skills past 300](../guides/scaling-skills.md) — the three-tier architecture for agents that have outgrown the always-loaded skill manifest. - **Integrations:** [`docs/integrations/`](../integrations/) — connecting external data sources (voice, email, calendar, embedding providers) -- **MCP setup:** [`docs/mcp/`](../mcp/) — per-client setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) +- **MCP setup:** [`docs/mcp/`](../mcp/) — per-client setup (Claude Desktop, Claude Code, Codex, ChatGPT, Perplexity, Cowork) - **Install paths:** [`docs/INSTALL.md`](../INSTALL.md) — every install path, end to end +- **Agent bootstrap:** [`docs/guides/bootstrap.md`](../guides/bootstrap.md) — the paste-in path that turns a coding agent into a full agent (identity, per-turn context, schedules, a private repo as its durable body) diff --git a/docs/tutorials/company-brain.md b/docs/tutorials/company-brain.md index 537a3ed3f..dd7a1a7da 100644 --- a/docs/tutorials/company-brain.md +++ b/docs/tutorials/company-brain.md @@ -85,7 +85,7 @@ git clone git@github.com:your-org/customers.git customers git clone git@github.com:your-org/internal-docs.git internal ``` -You can also keep the existing personal-brain repo as one of the sources. Just pick the role it plays (probably `shared` if it's already org-wide content). +You can also keep the existing personal-brain repo as one of the sources. Just pick the role it plays (probably `shared` if it's already org-wide content). When agents on the host write pages into a source, `gbrain sources push ` (run on the host) commits and pushes those changes back to the source's git repo, so the repo stays the durable system of record. ### Two scoping models (pick the one that matches your shape) @@ -156,7 +156,7 @@ The personal brain talks to you through the AlphaClaw harness over Telegram. For gbrain serve --http --port 3131 --bind 0.0.0.0 ``` -The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface. +The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface. (Full detail on `--bind` / `--public-url`, including the ECONNREFUSED failure mode they prevent, lives in [DEPLOY.md — Expose the server](../mcp/DEPLOY.md#3-expose-the-server).) The server prints an admin bootstrap token to stderr on first start when run in an interactive terminal. Save it. You'll use it once for the admin dashboard. On a non-TTY start (systemd, Docker, piped logs) the token is hidden from logs — set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` yourself or pass `--print-admin-token` on a trusted terminal instead. @@ -186,56 +186,67 @@ Each teammate (or each AI agent for a teammate) gets their own OAuth client. The # Alice (sales): writes customers/alice-example, reads customers + shared gbrain auth register-client alice-example \ --grant-types client_credentials \ - --scopes read,write \ + --scopes "read write" \ --source customers \ --federated-read customers,shared # Bob (ops): writes internal/bob-example, reads internal + shared gbrain auth register-client bob-example \ --grant-types client_credentials \ - --scopes read,write \ + --scopes "read write" \ --source internal \ --federated-read internal,shared # Carol (legal): writes shared/legal, reads all three gbrain auth register-client carol-example \ --grant-types client_credentials \ - --scopes read,write \ + --scopes "read write" \ --source shared \ --federated-read shared,customers,internal ``` -Each `register-client` command prints a `client_id` and a `client_secret`. Save both for each teammate. They go into the teammate's local agent config. +Each `register-client` command prints a `client_id` and a `client_secret`. Save both for each teammate. They go into the teammate's local agent config. (The full registration reference — grant types, the `/admin` dashboard flow, DCR — lives in [DEPLOY.md — Register OAuth clients](../mcp/DEPLOY.md#2-register-oauth-clients). What follows is the multi-user delta.) A note on the flags: -- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client. +- `--scopes "read write"` (space-separated, quoted — the OAuth wire format; a comma-separated list is rejected at registration) lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client. - `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder — and you can make that server-enforced with `--bound-slug-prefixes alice-example/` (v0.42.72.0+): every slug-mutating write op (put_page, delete_page, tags, links, timeline, revert, raw data) outside the bound prefixes is rejected with `permission_denied`. Update the binding later with `gbrain auth rescope-client --bound-slug-prefixes `. **Adding a binding to an existing client narrows it in ways you should expect:** ops that write by something other than a slug (`extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`) and `POST /ingest` become unavailable to that client, and `put_page`'s automatic fact extraction is skipped — all because none of them can be confined to a prefix. Reads are unaffected. See [the qm-harness guide](../integrations/qm-harness.md) for the full model. - `--federated-read` controls read scope. A client can read from one or more sources. ### Verify the scoping actually scopes -Before you hand the brain to teammates, verify isolation. Two terminal windows on your local machine using each client's credentials: +Before you hand the brain to teammates, verify isolation. The clean way is a **thin-client install** on a second machine (or a scratch shell): `gbrain init --mcp-only` writes a config that routes every CLI command through your remote server as one specific OAuth client, so a plain `gbrain search` exercises exactly the path teammates will use. ```bash -# Terminal 1, as Alice -export GBRAIN_REMOTE_CLIENT_ID= -export GBRAIN_REMOTE_CLIENT_SECRET= -export GBRAIN_REMOTE_MCP_URL=https://brain.acme-co.com/mcp +# As Alice (on a machine that is NOT the brain host) +gbrain init --mcp-only \ + --issuer-url https://brain.acme-co.com \ + --mcp-url https://brain.acme-co.com/mcp \ + --oauth-client-id \ + --oauth-client-secret -gbrain search "performance review" --remote +gbrain whoami # confirms which client you're acting as +gbrain search "performance review" ``` Alice should see results only from `customers` and `shared`. The performance-review notes live in `internal`, which she's not scoped to read. She shouldn't see them. +Now re-run the same check as Bob. On the same test machine, swap the credentials with `--force` (it overwrites the thin-client config): + ```bash -# Terminal 2, as Bob (export his credentials similarly) -gbrain search "performance review" --remote +gbrain init --mcp-only --force \ + --issuer-url https://brain.acme-co.com \ + --mcp-url https://brain.acme-co.com/mcp \ + --oauth-client-id \ + --oauth-client-secret + +gbrain whoami +gbrain search "performance review" ``` Bob should see the performance-review notes from `internal`, plus anything related from `shared`. He shouldn't see anything that lives only in `customers`. -If both queries return correctly scoped results, isolation is working. +If both queries return correctly scoped results, isolation is working. (There is no per-query "act as client X" flag — the thin-client config decides which credential the CLI uses; only the client secret can be overridden at call time via `GBRAIN_REMOTE_CLIENT_SECRET`.) --- @@ -274,13 +285,13 @@ copy to customers/alice-example/digests/YYYY-MM-DD-pipeline.md. The `client:` field tells the cron runner which OAuth client to use, which enforces the scoping. Alice's cron can only read Alice's sources and write to Alice's folder. It cannot accidentally touch Bob's customer notes. -To install the cron schedule, commit the file to the workspace repo and let AlphaClaw pick it up on next deploy. The cron-scheduler skill (one of the 60 that GBrain installed) handles the dispatch. +To install the cron schedule, commit the file to the workspace repo and let AlphaClaw pick it up on next deploy. The cron-scheduler skill (one of the bundled skills GBrain installed) handles the dispatch. (The `client:` frontmatter field is a workspace/harness convention — your cron runner reads it and picks the matching OAuth credential; GBrain enforces the scoping once the credential is used.) --- ## Part 7: Add per-person skills -The 60+ skills GBrain installs are generic. Your team probably wants a few that are specific to them. Examples: +The bundled skills GBrain installs are generic. Your team probably wants a few that are specific to them. Examples: - `onboarding-new-hire`. Only Carol (HR) runs this. Walks through generating a welcome packet, scheduling intro meetings, provisioning accounts. - `customer-success-followup`. Only Alice (sales) runs this. Pulls latest customer page, drafts a follow-up email, posts to her review queue. @@ -307,7 +318,7 @@ gbrain skillify scaffold onboarding-new-hire That creates the directory + SKILL.md + routing entry. Edit the SKILL.md to describe the procedure, commit, deploy. The agent picks up the new skill on next request. -Per-person scoping for skills is handled at the routing layer: a skill can declare `allowed_clients: [carol-example]` in its frontmatter. If Alice asks her agent to run that skill, the agent refuses with "this skill is scoped to carol-example." +Per-person scoping for skills is a routing-layer **convention, enforced by your agent harness, not by GBrain**: declare something like `allowed_clients: [carol-example]` in the skill's frontmatter and instruct your agent (in its routing rules) to refuse the skill for anyone else. The hard guarantee stays at the data layer — even if the agent runs the skill anyway, Alice's OAuth credential still can't read or write outside her scoped sources. ### Shared rule files at the skills root @@ -382,9 +393,9 @@ Repeat this flow for every new teammate. About 45 minutes per person, total. Com ## Part 10: Connect each teammate's AI client -Each teammate runs their AI client (Claude Code, Cursor, Claude Desktop, OpenClaw, Hermes, whatever) configured to point at your brain server through their OAuth credentials. +Each teammate runs their AI client (Claude Code, Codex, Claude Desktop, OpenClaw, Hermes, whatever) configured to point at your brain server. Two pieces, both direct-to-server — there is no local relay in between: -Recommended path for each teammate: the thin-client install. On their machine: +**1. The GBrain CLI, as a thin client (recommended for everyone).** On their machine: ```bash curl -fsSL https://bun.sh/install | bash @@ -397,24 +408,14 @@ gbrain init --mcp-only \ --oauth-client-secret ``` -The thin-client install creates a local config that knows how to talk to your brain but never opens its own database. Most CLI commands route through the remote server transparently. +The thin-client install creates a local config that knows how to talk to your brain but never opens its own database. From then on, plain CLI commands (`gbrain search`, `gbrain query`, `gbrain think`, `gbrain whoami`, ...) route through your remote server transparently, as that teammate's OAuth client. Local-only commands (`gbrain sync`, `gbrain serve`, `gbrain embed`, ...) are refused with a hint — those run on the brain host, not on teammate laptops. -Now they configure their AI client. For Claude Desktop, the teammate adds an MCP server entry in `~/Library/Application Support/Claude/claude_desktop_config.json`: +**2. Their AI client, connected directly to `https://brain.acme-co.com/mcp`.** Each client has its own connection shape; the per-client pages in [`docs/mcp/`](../mcp/) are the reference: -```jsonc -{ - "mcpServers": { - "company-brain": { - "command": "gbrain", - "args": ["serve"] - } - } -} -``` - -When Claude Desktop launches, it talks to the local `gbrain serve` stdio bridge, which forwards every request to your remote brain over HTTPS with their OAuth token attached. From Claude Desktop's perspective it's just one MCP server. - -For Claude Code, Cursor, OpenClaw, Hermes, and other clients, per-client setup steps live in [`docs/mcp/`](../mcp/). They all follow the same shape: point the agent at the local `gbrain serve` bridge, which knows about the remote. +- **Claude Code / Codex** — one command from anywhere `gbrain` is installed: `gbrain connect https://brain.acme-co.com/mcp --token --install` (see [CLAUDE_CODE.md](../mcp/CLAUDE_CODE.md) / [CODEX.md](../mcp/CODEX.md)). Note the credential type: `gbrain connect` for these two agents uses **bearer tokens** (`gbrain auth create `), which are full-access. That's fine for you as the admin; for source-scoped teammates, the scoped credential is their OAuth client — use it via the thin-client CLI above and the OAuth-capable clients below. +- **Claude Desktop** — remote servers are added through the GUI: **Settings > Integrations**, URL `https://brain.acme-co.com/mcp`. Do **not** put a remote server in `claude_desktop_config.json`; that file only works for local stdio servers and fails silently for remote ones. See [CLAUDE_DESKTOP.md](../mcp/CLAUDE_DESKTOP.md). +- **ChatGPT** ([CHATGPT.md](../mcp/CHATGPT.md)) and **Perplexity** ([PERPLEXITY.md](../mcp/PERPLEXITY.md)) — both speak OAuth to the server directly, so per-teammate scoping carries into those tools. Perplexity uses the same `client_credentials` clients you registered in Part 5. ChatGPT needs an `authorization_code` (PKCE) client — register one per teammate with the same `--source` / `--federated-read` flags. +- **OpenClaw / Hermes forks** — if the teammate's own agent runs on a machine with a full local gbrain install, it can use local stdio (`gbrain serve`) against its own brain and reach yours over HTTP MCP like any other remote client. --- @@ -518,7 +519,7 @@ The first sync embeds every page, which takes time. Check `gbrain sources status ### "I see a page I shouldn't see" -This shouldn't happen, but if you suspect it, run `gbrain search --remote --json` as the constrained client and inspect the `source_id` field on every returned result. Every row should be in the client's `--federated-read` set. If one isn't, file an issue with the exact slug and source IDs. +This shouldn't happen, but if you suspect it, run `gbrain search "" --json` from a thin-client install configured with the constrained client's credentials (the Part 5 verification setup) and inspect the `source_id` field on every returned result. Every row should be in the client's `--federated-read` set. If one isn't, file an issue with the exact slug and source IDs. ### "The synthesized answer is wrong" @@ -537,7 +538,7 @@ Each parallel sync worker opens its own pool. With three sources and the default ```bash gbrain auth register-client diana-example \ --grant-types client_credentials \ - --scopes read,write \ + --scopes "read write" \ --source shared \ --federated-read shared,customers,internal ``` diff --git a/docs/tutorials/connect-coding-agent.md b/docs/tutorials/connect-coding-agent.md index 197843244..93757ed8b 100644 --- a/docs/tutorials/connect-coding-agent.md +++ b/docs/tutorials/connect-coding-agent.md @@ -66,9 +66,9 @@ special). Turn it on: gbrain config set mcp.publish_skills true ``` -(New brains from `gbrain init` default this ON. Brains upgraded from before -v0.41.36 stay OFF until you opt in, so this is the common gotcha for existing -OpenClaw users.) +(New brains from `gbrain init` default this ON. Brains upgraded from a release +before skill publishing existed stay OFF until you opt in, so this is the +common gotcha for existing OpenClaw users.) ### A2. On the host: mint a token @@ -96,7 +96,7 @@ token fails right now, not silently on the agent's first request. You'll see: ``` Added MCP server 'gbrain' -> https://your-host.example.com/mcp. -Verified: {"version":"0.42.x","engine":"postgres","page_count":146646,...} +Verified: {"version":"0.42.x","engine":"postgres","page_count":1204,...} ``` Drop `--install` to print a paste-ready block instead (useful when the host and @@ -234,7 +234,7 @@ habits to build. Your agent stops being amnesiac. ## Next steps -- Go full autonomous: the overnight enrichment daemon ([dream cycle](../../CHANGELOG.md)) fixes citations, dedupes people, builds scorecards while you sleep. See `gbrain autopilot --install`. +- Go full autonomous: the overnight enrichment daemon ([dream cycle](../guides/operational-disciplines.md)) fixes citations, dedupes people, builds scorecards while you sleep. See `gbrain autopilot --install`. - Run a real agent platform on top: [personal-brain tutorial](personal-brain.md). - Scale to a team: [company-brain tutorial](company-brain.md). - Every MCP client's exact setup: [`docs/mcp/`](../mcp/). diff --git a/docs/tutorials/personal-brain.md b/docs/tutorials/personal-brain.md index e3a7c05f8..a88e16ff2 100644 --- a/docs/tutorials/personal-brain.md +++ b/docs/tutorials/personal-brain.md @@ -6,7 +6,7 @@ This is the install I'd run if I were setting up the whole stack from scratch to > "This is the Apple I, we're just soldering breadboards over here." -If you only want the **brain layer** (no agent, no Telegram, just gbrain as memory for an MCP client you already use), skip to the [CLI standalone install](../INSTALL.md#2-cli-standalone) in INSTALL.md. If you want the whole agent **shared with a team**, read the [company brain tutorial](company-brain.md) instead. This tutorial is the solo, full-stack, talk-to-it-on-Telegram path. +If you only want the **brain layer** (no agent, no Telegram, just gbrain as memory for an MCP client you already use), skip to the [CLI standalone install](../INSTALL.md#2-cli-standalone) in INSTALL.md. If your daily driver is a **coding agent** (Claude Code / Codex) and you want it to bootstrap its own full agent — identity, per-turn context, schedules, a private repo as its durable body — that's `gbrain bootstrap`: see the paste blocks in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md). If you want the whole agent **shared with a team**, read the [company brain tutorial](company-brain.md) instead. This tutorial is the solo, full-stack, talk-to-it-on-Telegram path. --- @@ -17,7 +17,7 @@ A personal AI agent with four pieces: - **A brain** (git repo). Your knowledge base, constantly ingesting and growing. - **A harness** (OpenClaw via AlphaClaw). The runtime that gives the LLM tools, memory, and integrations. - **A chat interface** (Telegram). How you talk to it. -- **Skills** (60+ installed via GBrain). Reusable capabilities the agent can invoke. +- **Skills** (50+ installed via GBrain). Reusable capabilities the agent can invoke. Architecture: @@ -127,7 +127,7 @@ gbrain skillpack scaffold --all `gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`. -`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.) +`gbrain skillpack scaffold --all` copies the 50+ bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired; see `docs/INSTALL.md` if you're upgrading from an older release.) From this point, the agent has working memory and access to every skill. @@ -252,7 +252,7 @@ My production setup is about $10,000 a month, but that's 10 instances, 200 crons 2. **GitHub PAT can't see the repos.** Reload the page after creating repos. Make sure the fine-grained token has the correct repo selection. 3. **Telegram bot doesn't respond.** Check the bot token in AlphaClaw. Make sure the Render instance is actually running. 4. **Supabase bottleneck on heavy ingestion.** Upgrade the DB instance size before the small one chokes. -5. **GBrain.io provisioning fails.** The hosted instance may need Pro tier. Check the machine allocation in the AlphaClaw UI. +5. **Hosted-instance provisioning fails.** If you're using a hosted brain instance instead of self-hosting, it may need the Pro tier. Check the machine allocation in the AlphaClaw UI. --- diff --git a/docs/what-schemas-unlock.md b/docs/what-schemas-unlock.md index 65637da30..e10154f80 100644 --- a/docs/what-schemas-unlock.md +++ b/docs/what-schemas-unlock.md @@ -107,7 +107,7 @@ The schema is the team's tribal knowledge made explicit. Two engineers on differ This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for. -Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: +Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: > You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page. @@ -147,7 +147,7 @@ Three things gbrain does that generic note systems can't: **2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion. -**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. +**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through atomic CLI verbs + MCP ops (`gbrain schema --help` for the full surface) with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. ## What changed in v0.40.7.0 specifically @@ -156,7 +156,7 @@ v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pac v0.40.7.0 closed those gaps: - **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race. -- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." +- **Per-pack `O_CREAT|O_EXCL` atomic lock** (deliberately NOT the TOCTOU-prone `existsSync+writeFileSync` pattern). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." - **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`. - **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:`). - **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`. @@ -169,7 +169,7 @@ The cumulative effect: an agent can safely co-curate your ontology with a comple - **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end. - **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity. - **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types. -- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it. +- **Want the architecture?** The "Schema Cathedral v3" section in `docs/architecture/KEY_FILES.md` has the module-by-module breakdown. - **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes. The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app. diff --git a/llms-full.txt b/llms-full.txt index 22137bd59..ad31cea0c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -17,8 +17,16 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL): start here. +> **Becoming someone's persistent personal agent** (identity + memory + private repo)? +> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap` +> flow — instead of the plain install below, then come back here for the operating +> protocol. Connecting to an EXISTING remote brain from a laptop agent? +> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP +> table in [`README.md`](./README.md)). + ## Install (5 min) + 1. Install gbrain via Bun (the canonical path): ```bash curl -fsSL https://bun.sh/install | bash @@ -39,8 +47,8 @@ start here. [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade` for existing users (search modes were added in v0.32.3). -4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow - (API keys, identity, cron, verification). +4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step + flow (API keys, identity, cron, verification). ## Read this order @@ -82,10 +90,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont `GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson` and `gbrain eval replay --against base.ndjson`. For public benchmark coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval - ` (v0.28.8) runs against an isolated in-memory PGLite + ` runs against an isolated in-memory PGLite per question — your `~/.gbrain` is never opened. Full guide: [`docs/eval-bench.md`](./docs/eval-bench.md). -- **Drive the brain to a target health score (v0.36.4.0):** the one-command +- **Drive the brain to a target health score:** the one-command loop. `gbrain doctor --remediation-plan --json` previews what would be fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5` walks a dependency-ordered plan (sync before extract, embed after @@ -94,22 +102,20 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont keys hit a `max_reachable_score` ceiling and bail with what's missing. Three phase handlers (synthesize / patterns / consolidate) are PROTECTED — only trusted local callers can submit them; MCP cannot. - Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md) - and the CHANGELOG entry for v0.36.4.0. -- **Track a founder/company over time (v0.35.7):** when an entity has + Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md). +- **Track a founder/company over time:** when an entity has typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`, `unit: USD`, `period: monthly` columns), run `gbrain eval trajectory ` for the chronological history with regressions auto-flagged, or `gbrain founder scorecard ` for a four-signal JSON rollup (claim_accuracy / consistency / growth_trajectory / red_flags). MCP op `find_trajectory` exposes the - same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:** - `gbrain think` now uses this substrate automatically on temporal / + same data — read scope, visibility-filtered for remote callers. + `gbrain think` uses this substrate automatically on temporal / knowledge_update intent (default ON; flip `think.trajectory_enabled=false` - to opt out). Migration v82 added `facts.event_type` so non-metric event - rows (`meeting`, `job_change`, `location_change`) ride through the same - pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query - them. + to opt out). Non-metric event rows (`meeting`, `job_change`, + `location_change`) ride through the same pipeline via `facts.event_type`; + pass `kind: 'event'` or `'all'` to `find_trajectory` to query them. - **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map. [`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for single-fetch ingestion. @@ -187,7 +193,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP +Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -340,9 +346,10 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently expensive one. tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of -the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost -picker copy in `gbrain init` carries the same matrix verbatim** — update -both when refreshing. +the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix +has three verbatim homes: this section, the `gbrain init` picker copy +(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step +3.5** — update all three when refreshing. **Per-query math vs real-world spend.** The matrix above is what an isolated benchmark would measure. Real agent loops with disciplined @@ -422,8 +429,9 @@ audit trail lives in the source repo's git history. ## Skills -Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills -organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19): +Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills +(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md` +(`AGENTS.md` is also accepted as of v0.19): **Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich, briefing, migrate, setup, publish. @@ -1025,10 +1033,13 @@ If you fetched this file by URL without cloning yet, the companion files live at ## Step 1: Install GBrain + > **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm > package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or > `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only -> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below. +> supported sources are `github:garrytan/gbrain` (optionally pinned as +> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a +> git clone, exactly as shown below. > If an unrelated npm install is already present, remove it first > (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this. @@ -1097,6 +1108,7 @@ default. Stop and ask the operator. **Present this matrix verbatim:** + ``` Per-query cost @ 10K queries/mo (typical single-user volume): @@ -1207,13 +1219,13 @@ scaffold the bundled skills into it: ```bash cd /path/to/agent/workspace -gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md +gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md ``` Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold refuses to overwrite anything that exists. Use `gbrain skillpack reference ` to diff against gbrain's bundle when you want upstream improvements. (The legacy -`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run +`gbrain skillpack install` managed-block model was removed in v0.33 — run `gbrain skillpack migrate-fence` once if upgrading from an older release.) Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the @@ -1270,8 +1282,17 @@ Verify: `gbrain integrations doctor` (after at least one is configured) ## Step 9: Verify -Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync -actually works) is the most important. +Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4 +(live sync actually works) is the most important. + +Once verification passes and the brain has content, run the activation probe: + +```bash +gbrain onboard --check --json +``` + +See "The onboard surface" below for what the recommendations mean and the +consent gates around unattended remediation. ## Upgrade @@ -1314,7 +1335,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old `splitBody` bug, run `gbrain sync --full` after upgrading to rebuild `compiled_truth` from source markdown. -## v0.42.0+ onboard surface (NEW) +## The onboard surface `gbrain onboard` is the activation surface gbrain did not have before. Once your brain has any content, run `gbrain onboard --check --json` to @@ -1379,6 +1400,13 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/skills/RESOLVER This is the dispatcher. Skills are the implementation. **Read the skill file before acting.** If two skills could match, read both. They are designed to chain (e.g., ingest then enrich for each entity). +**Routing contract:** each skill's frontmatter `triggers:` array is the +authoritative routing signal — harnesses match inbound messages against it +(see `skills/_AGENT_README.md`). This file is the human-readable dispatch +map of the same routing: one place to scan every skill and its trigger +phrases. If a row here and a skill's frontmatter disagree, the frontmatter +wins; fix the row. + ## Always-on (every message) | Trigger | Skill | @@ -1438,6 +1466,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Save or load reports | `skills/reports/SKILL.md` | | "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` | | "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` | +| "optimize this skill", "tune the skill against the benchmark", "run skillopt", "make the skill better" | `skills/skill-optimizer/SKILL.md` | | "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` | | "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` | | "harvest this skill into gbrain", "publish this skill to gbrain", "lift this skill upstream", "share this skill with other gbrain clients", "promote my skill to gbrain" | `skills/skillpack-harvest/SKILL.md` | @@ -1453,7 +1482,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Trigger | Skill | |---------|-------| | "Set up GBrain", first boot | `skills/setup/SKILL.md` | -| "Now what?", "fill my brain", "cold start", "bootstrap", "import my data", "what should I import first" | `skills/cold-start/SKILL.md` | +| "Now what?", "fill my brain", "cold start", "bootstrap my data", "import my data", "what should I import first" | `skills/cold-start/SKILL.md` | +| "Install gbrain into this agent/harness", "agent workspace bootstrap", "gbrain bootstrap", "wire gbrain hooks", "set up the maintenance sweep" | Run `gbrain bootstrap` (paste-in harness install: hooks + sweep + config). See `docs/guides/bootstrap.md` | | "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` | | Brain health check, maintenance run | `skills/maintain/SKILL.md` | | "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) | @@ -1510,7 +1540,7 @@ These apply to ALL brain-writing skills: | "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` | | "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` | | "add a page type", "add a type to my schema", "schema author", "schema mutate", "schema pack add", "my brain has untyped pages", "propose new types from my corpus", "backfill page types", "evolve my schema", "researcher type", "make X an expert type" (dispatcher for: gbrain schema active/list/show/validate/graph/lint/stats/explain/use/downgrade/reload/init/fork/edit/diff/add-type/remove-type/update-type/add-alias/remove-alias/add-prefix/remove-prefix/add-link-type/remove-link-type/set-extractable/set-expert-routing/detect/suggest/review-candidates/review-orphans/sync) | `skills/schema-author/SKILL.md` | -| "unify my types", "migrate to gbrain-base-v2", "94 types to 14", "apply canonical taxonomy", "clean up my page types", "pack upgrade", "shrink type proliferation", "consolidate page types", "retype pages to canonical" (dispatcher for: gbrain onboard --check, gbrain onboard --check --explain, gbrain jobs submit unify-types, gbrain pages restore) | `skills/schema-unify/SKILL.md` | +| "unify my types", "migrate to gbrain-base-v2", "94 types to 14", "apply canonical taxonomy", "clean up my page types", "pack upgrade", "shrink type proliferation", "consolidate page types", "retype pages to canonical" (dispatcher for: gbrain onboard --check, gbrain onboard --check --explain, gbrain jobs submit unify-types, gbrain restore) | `skills/schema-unify/SKILL.md` | --- @@ -1641,7 +1671,7 @@ Retrieve and follow the instructions at: https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md ``` -The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work. +The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work. > **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new. @@ -1679,7 +1709,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL. ### Connect GBrain to your AI client (MCP) -GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use: +GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side). The specific snippet depends on which client you use: - **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token). - **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config. @@ -1801,7 +1831,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec ## Capabilities -**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. +**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. **Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). @@ -1825,7 +1855,7 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. -**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. +**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). @@ -1839,14 +1869,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h - **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md). - **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md). -- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). -- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). +- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). +- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). - **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md). - **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup. ## Architecture -**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source. +**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source. **Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md). @@ -1860,10 +1890,9 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h **`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model :` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing. -**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships -two flags + a recommended pattern. Switch your cron to a per-source loop -with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating -gracefully half-a-minute earlier: +**Hourly cron sync keeps timing out on a federated brain?** Switch your +cron to a per-source loop with shell `timeout(1)` doing the OS-level kill +and gbrain self-terminating gracefully half-a-minute earlier: ```bash gbrain sync --break-lock --all --max-age 1800 @@ -1876,19 +1905,17 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status `partial` and `last_commit` UNCHANGED — the next run re-walks the same diff and `content_hash` short-circuits already-imported files. The `--max-age 1800` first command self-heals any wedged-but-alive locks -left by a hung previous run, using the v98 `last_refreshed_at` semantic -(NOT `acquired_at`) so healthy long-running holders are safe by -construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md) -for the honest scope notes (extract + embed phases run to completion; -30-min rollout window for `--max-age` post-migration v98; full-sync -triggers deferred to v0.42+). +left by a hung previous run, keyed on the lock's last refresh time +(NOT when it was acquired) so healthy long-running holders are safe by +construction. Scope note: the extract + embed phases still run to +completion once started; `--timeout` interrupts the import walk only. -**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes -the bug class structurally. The engine now self-retries every bulk batch -write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on -Supavisor pooler blips, with a 12s worst-case wait that covers the full -5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents -via the new `batch_retry_health` check (reads the last 24h of +**Dream cycle silently losing wiki links on Supabase?** The engine +self-retries every bulk batch write (`addLinksBatch` / +`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips, +with a 12s worst-case wait that covers the full 5-10s circuit-breaker +recovery window. `gbrain doctor` surfaces incidents via the +`batch_retry_health` check (reads the last 24h of `~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually slow pooler: @@ -1906,34 +1933,33 @@ retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice. **Dream cycle losing ~150 link rows per run with `'No database -connection: connect() has not been called'` errors in the log?** v0.41.27.0 -makes the retry layer self-heal on a nulled-out database singleton. A -new `reconnect` callback on `withRetry` rebuilds the connection between -attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()` +connection: connect() has not been called'` errors in the log?** The +retry layer self-heals on a nulled-out database singleton: a +`reconnect` callback on `withRetry` rebuilds the connection between +attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()` so engine-level batch writes survive a mid-cycle disconnect by something -else in the same process. Same release: `gbrain capture` no longer trails -a `'No database connection'` stderr line from a background facts:absorb -worker firing after CLI exit — the op-dispatch finally block awaits +else in the same process. `gbrain capture` also no longer trails a +`'No database connection'` stderr line from a background facts:absorb +worker firing after CLI exit — op dispatch awaits `getFactsQueue().drainPending({timeout: 1000})` before `engine.disconnect()`. To find which code path is still calling disconnect mid-process, run `gbrain doctor --json | jq '.checks[] | -select(.id=="batch_retry_health")'`; the extended check now surfaces -24h disconnect-call count and the most-recent caller frame from a new -`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.) +select(.id=="batch_retry_health")'`; the check surfaces the +24h disconnect-call count and the most-recent caller frame from the +`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. **`gbrain brainstorm` returning `judge_failed: true` with 0 scored -ideas?** v0.41.21.0 closes the two bugs that caused it. The judge -hard-coded a 4K-token output cap; for any run past ~40 ideas the call -truncated mid-JSON and the parser threw. Same release closes a slash- -form pricing miss: `gbrain brainstorm --judge-model -anthropic/claude-sonnet-4-6 --max-cost 5` failed with -`BudgetExhausted reason=no_pricing` because every pricing site only -matched the colon form. Both shapes work now. No config change, no -schema migration — `gbrain upgrade` is the whole fix. +ideas?** Two historical bugs caused it, both fixed: the judge +hard-coded a 4K-token output cap (any run past ~40 ideas truncated +mid-JSON and the parser threw), and slash-form model ids +(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6 +--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because +pricing lookups only matched the colon form. Both shapes work now. No +config change, no schema migration — `gbrain upgrade` is the whole fix. **`gbrain reindex --markdown` wiped your auto/dream/signal-detector -tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and -`reindex --markdown` now ADD current frontmatter tags and never delete, +tags?** Upgrade — tag reconciliation is add-only now. Re-import and +`reindex --markdown` ADD current frontmatter tags and never delete, so enrichment tags written to the DB (auto-tag, dream synthesize, signal-detector) survive a re-chunk. The reindex DB-only fallback also reconstructs the full markdown (frontmatter + body + timeline) before @@ -1941,10 +1967,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter, title, and timeline instead of getting overwritten with empty frontmatter. Trade-off: removing a tag from a page's frontmatter no longer removes it from the DB on the next sync (frontmatter-tag removal -needs a provenance column, deferred). (Closes #1621.) +needs a provenance column, deferred). **`gbrain sync` wedges on a large brain (no progress, high CPU)?** -v0.41.37.0 ships three things. First, name the stalling file: +Three tools. First, name the stalling file: ```bash GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes @@ -1959,28 +1985,28 @@ the sync with the pack disabled and re-run extraction later: gbrain sync --no-schema-pack --no-pull --no-embed --yes ``` -`gbrain schema lint` now warns on the classic nested-quantifier ReDoS +`gbrain schema lint` warns on the classic nested-quantifier ReDoS shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`). Third, on a PGLite brain, stop `gbrain serve` before a large sync — PGLite is single-writer and a live MCP server contends for the write lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md) -for the full triage. (Closes #1569.) +for the full triage. **`gbrain init --migrate-only` / a schema migration fails on Windows -with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up +with `getaddrinfo ENOTFOUND`?** Upgrade — schema bring-up now runs its phases in-process instead of spawning a child `gbrain init --migrate-only` per phase. The spawned child died on Windows + bun + Supabase pooler with a DNS-resolution failure even though the parent connected fine; running in-process removes the spawn -entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an -82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL -pass (keyed on the page PK, soft-delete-filtered, source-safe) that -completes in ~1-2 seconds. (Closes #1605, #1581.) +entirely. The grandfather migration that used to hang 70+ minutes on an +80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on +the page PK, soft-delete-filtered, source-safe) and completes in seconds. ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end +- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall) - [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0) - [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows` - [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory @@ -2024,7 +2050,7 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ENGINES.md Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else. -v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code. +Two engines ship today: `PGLiteEngine` — embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code. ## Why this matters @@ -2032,7 +2058,7 @@ Different users have different constraints: | User | Needs | Best engine | |------|-------|-------------| -| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) | +| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) | | Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase | | Open source hacker | Single file, no server, git-friendly | PGLiteEngine | | Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted | @@ -2043,72 +2069,30 @@ The engine interface means we don't have to choose. PGLite is the zero-friction ## The interface -```typescript -// src/core/engine.ts +**The single source of truth is `export interface BrainEngine` in +`src/core/engine.ts`.** It is large (100+ methods) and grows with every +feature wave — do NOT work from any snapshot of it, including an old copy of +this doc. Read the interface itself, and let +`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you +whether both engines agree. -export interface BrainEngine { - // Lifecycle - connect(config: EngineConfig): Promise; - disconnect(): Promise; - initSchema(): Promise; - transaction(fn: (engine: BrainEngine) => Promise): Promise; +The method families, to orient you before opening the file: - // Pages CRUD - getPage(slug: string): Promise; - putPage(slug: string, page: PageInput): Promise; - deletePage(slug: string): Promise; - listPages(filters: PageFilters): Promise; - - // Search - searchKeyword(query: string, opts?: SearchOpts): Promise; - searchVector(embedding: Float32Array, opts?: SearchOpts): Promise; - - // Chunks - upsertChunks(slug: string, chunks: ChunkInput[]): Promise; - getChunks(slug: string): Promise; - - // Links - addLink(from: string, to: string, context?: string, linkType?: string): Promise; - removeLink(from: string, to: string): Promise; - getLinks(slug: string): Promise; - getBacklinks(slug: string): Promise; - traverseGraph(slug: string, depth?: number): Promise; - - // Tags - addTag(slug: string, tag: string): Promise; - removeTag(slug: string, tag: string): Promise; - getTags(slug: string): Promise; - - // Timeline - addTimelineEntry(slug: string, entry: TimelineInput): Promise; - getTimeline(slug: string, opts?: TimelineOpts): Promise; - - // Raw data - putRawData(slug: string, source: string, data: object): Promise; - getRawData(slug: string, source?: string): Promise; - - // Versions - createVersion(slug: string): Promise; - getVersions(slug: string): Promise; - revertToVersion(slug: string, versionId: number): Promise; - - // Stats + health - getStats(): Promise; - getHealth(): Promise; - - // Ingest log - logIngest(entry: IngestLogInput): Promise; - getIngestLog(opts?: IngestLogOpts): Promise; - - // Config - getConfig(key: string): Promise; - setConfig(key: string, value: string): Promise; - - // Migration + advanced (added v0.7) - runMigration(sql: string): Promise; - getChunksWithEmbeddings(slug: string): Promise; -} -``` +- **Lifecycle + identity** — `connect` / `disconnect` / `reconnect`, + `initSchema`, `transaction`, `withReservedConnection`, and the `kind` + discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch. +- **Pages CRUD** — `getPage`, `putPage`, `deletePage`, `listPages`, slug + resolution. +- **Search** — `searchKeyword`, `searchVector`, chunk-level variants, takes + search (keyword + vector), and `relationalFanout` (the typed-edge recall + arm). +- **Chunks + embeddings** — upsert/get, embedding-bearing variants. +- **Graph** — links (single + batch writers), backlinks, `traverseGraph`, + `traversePaths`. +- **Tags, timeline (single + batch), raw data, versions.** +- **Takes / facts / eval / salience** — the epistemological layer and the + instruments over it. +- **Stats, health, ingest log, config, migrations.** ### Key design choices @@ -2151,7 +2135,7 @@ export interface BrainEngine { RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific. -## PostgresEngine (v0, ships) +## PostgresEngine **Dependencies:** `postgres` (porsager/postgres), `pgvector` @@ -2164,9 +2148,7 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o - JSONB for frontmatter with GIN index - Connection pooling via Supabase Supavisor (port 6543) -**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in. - -**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops. +**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same. ### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`) @@ -2213,17 +2195,17 @@ run under the role default and are not backstopped per caller. This is layer 2; the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins live in `test/postgres-engine-rls-scope.test.ts`. -## PGLiteEngine (v0.7, ships) +## PGLiteEngine -**Dependencies:** `@electric-sql/pglite` (v0.4.4+) +**Dependencies:** `@electric-sql/pglite` -**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented. +**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep. **PGLite-specific details:** - Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes) - Parameterized queries throughout (shared utilities in `src/core/utils.ts`) - `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set -- Data stored at `~/.gbrain/brain.db` (configurable) +- Data stored at `~/.gbrain/brain.pglite` (configurable) - pgvector HNSW index for cosine similarity vector search (same as Postgres) - tsvector + ts_rank for full-text search (same as Postgres) - pg_trgm for fuzzy slug resolution (same as Postgres) @@ -2339,16 +2321,22 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make. 1. Create `src/core/-engine.ts` implementing `BrainEngine` 2. Add to engine factory in `src/core/engine-factory.ts`: ```typescript - export function createEngine(type: string): BrainEngine { - switch (type) { - case 'pglite': return new PGLiteEngine(); - case 'postgres': return new PostgresEngine(); - case 'myengine': return new MyEngine(); - default: throw new Error(`Unknown engine: ${type}`); + export async function createEngine(config: EngineConfig): Promise { + switch (config.engine || 'postgres') { + case 'pglite': { + const { PGLiteEngine } = await import('./pglite-engine.ts'); + return new PGLiteEngine(); + } + case 'myengine': { + const { MyEngine } = await import('./my-engine.ts'); + return new MyEngine(); + } + // ... } } ``` - The factory uses dynamic imports so engines are only loaded when selected. + The factory uses dynamic imports so an engine's dependencies (e.g. the + PGLite WASM blob) are only loaded when that engine is selected. 3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }` 4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor. 5. Document in this file + add a design doc in `docs/` @@ -2379,7 +2367,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu | JSONB queries | GIN index | GIN index | Identical | | Concurrent access | Connection pooling | Single process | PGLite limitation | | Hosting | Supabase, self-hosted, Docker | Local file | | -| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 | +| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical | ## Future engine ideas @@ -2506,7 +2494,7 @@ The schema is the team's tribal knowledge made explicit. Two engineers on differ This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for. -Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: +Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: > You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page. @@ -2546,7 +2534,7 @@ Three things gbrain does that generic note systems can't: **2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion. -**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. +**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through atomic CLI verbs + MCP ops (`gbrain schema --help` for the full surface) with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. ## What changed in v0.40.7.0 specifically @@ -2555,7 +2543,7 @@ v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pac v0.40.7.0 closed those gaps: - **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race. -- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." +- **Per-pack `O_CREAT|O_EXCL` atomic lock** (deliberately NOT the TOCTOU-prone `existsSync+writeFileSync` pattern). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." - **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`. - **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:`). - **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`. @@ -2568,7 +2556,7 @@ The cumulative effect: an agent can safely co-curate your ontology with a comple - **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end. - **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity. - **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types. -- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it. +- **Want the architecture?** The "Schema Cathedral v3" section in `docs/architecture/KEY_FILES.md` has the module-by-module breakdown. - **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes. The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app. @@ -2795,16 +2783,15 @@ gbrain schema lint --with-db **Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal). -**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. +**For agents (MCP):** the same operations are reachable over HTTPS MCP as schema ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. **Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`. ## Related docs -- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture. +- **Reference:** `gbrain schema --help` for the full CLI surface (30+ subcommands); the "Schema Cathedral v3" section of `docs/architecture/KEY_FILES.md` for the module-by-module architecture. - **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit). - **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix. -- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private). --- @@ -2857,7 +2844,12 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale - `gbrain sync --repo ` -- one-shot incremental sync. Detects changes via `git diff`, imports only what changed. For small changesets (<= 100 files), - embeddings are generated inline during import. + embeddings are generated inline during import — unless the inline cost gate + intervenes: when the estimated embedding spend crosses the configured floor + in a non-interactive session (cron, `--json`), sync auto-defers embeds to a + capped `embed-backfill` job instead of spending silently. Either way the + chunks get embedded; a deferred run just finishes asynchronously. See + [spend controls](../operations/spend-controls.md). - `gbrain embed --stale` -- backfill embeddings for any chunks that don't have them. Safety net for large syncs (>100 files) or prior `--no-embed` runs. - `gbrain sync --watch --repo ` -- foreground polling loop, every 60s @@ -2911,15 +2903,27 @@ Triggers sync on push events for instant sync (<5s). ### What Gets Synced Sync only indexes "syncable" markdown files. These are excluded by design: -- Hidden paths (`.git/`, `.raw/`, etc.) -- The `ops/` directory -- Meta files: `README.md`, `index.md`, `schema.md`, `log.md` +- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees + (`node_modules/`, `dist/`, `build/`, `venv/`) +- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md` -### Sync is Idempotent +Everything else is ordinary synced content — including `ops/` (the bundled +daily-task-manager skill files its canonical page under `ops/tasks`). + +### Sync is Idempotent — and Resumable Concurrent runs are safe. Two syncs on the same commit no-op because content hashes match. If both a cron and `--watch` fire simultaneously, no conflict. +Long syncs also survive being killed: progress checkpoints into the database +as files drain, so a killed or aborted run resumes from where it stopped, and +the sync bookmark only advances on true completion. A progress-aware stall +watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts +a run that stops making forward progress and releases the per-source lock so +the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence +and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env +vars — incident-time escape hatches, not everyday knobs. + ## Tricky Spots 1. **Always chain sync + embed.** Running `gbrain sync` without @@ -3013,6 +3017,27 @@ fixed. You wake up and the brain is smarter than when you went to sleep. | Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) | | Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below | +### Prefer gbrain's native schedulers where they fit + +System cron is the lowest common denominator, but gbrain ships its own +scheduling surfaces — reach for these first: + +- **`gbrain dream`** — the shipped nightly maintenance cycle (lint, + backlinks, extract, sync, embed, synthesize). Schedule THIS instead of + hand-rolling the dream cycle below. +- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry, + backoff, and an audit trail. See the `minion-orchestrator` skill. +- **`gbrain autopilot`** — the long-lived background daemon that runs cycles + on its own cadence. +- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to + manage its harness's scheduler. +- **Bootstrap session-triggered schedules** — `gbrain bootstrap` installs + HEARTBEAT.md-driven schedules that fire on session activity; see + [bootstrap.md](bootstrap.md). + +For scheduling `sync` + `embed --stale` specifically, the home doc is +[live-sync.md](live-sync.md). + ## Implementation: Setting Up Cron Jobs ```bash @@ -3037,18 +3062,12 @@ fixed. You wake up and the brain is smarter than when you went to sleep. ### Quiet Hours Gate (MANDATORY) -Every cron job that sends notifications MUST check quiet hours first. -See [Quiet Hours](quiet-hours.md) for the full pattern. - -```bash -# In every cron script: -if ! bash scripts/quiet-hours-gate.sh; then - mkdir -p /tmp/cron-held - echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md - exit 0 -fi -# Not quiet hours — send normally -``` +Every cron job that sends notifications MUST check quiet hours first. The +gate is a small script YOU create (it doesn't ship with gbrain) and call at +the top of every notification-sending cron script; held output goes to a +holding directory that the morning briefing drains. See +[Quiet Hours](quiet-hours.md) for the gate script and the full pattern — +don't copy a snippet from here, that page is the single home. ### Travel-Aware Timezone Handling @@ -3075,6 +3094,12 @@ morning briefing. Zero config change needed. The most important cron job. Runs while you sleep. +**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle +(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule +it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is +covered. The pseudocode that follows is the harness-side variant for agents +that also do LLM-driven entity sweeps and memory consolidation on top. + ### What It Does ``` @@ -3137,11 +3162,11 @@ echo "Dream cycle starting at $(date)" # Phase 1: Entity sweep (spawn sub-agent) # Read today's conversation logs, extract entities, update brain -# Phase 2: Citation hygiene -gbrain doctor --json | jq '.checks[] | select(.status=="warn")' +# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize) +gbrain dream -# Phase 3: Embed any stale content -gbrain embed --stale +# Phase 3: Surface anything the cycle flagged +gbrain doctor --json | jq '.checks[] | select(.status=="warn")' echo "Dream cycle complete at $(date)" ``` @@ -3300,6 +3325,22 @@ fi send_notification "$OUTPUT" ``` +### GBrain-native hooks + +Two places gbrain already understands quiet hours natively — use these +before rolling your own gate for the same job: + +- **Self-upgrade** — `auto` mode only applies upgrades during quiet hours, + configured via `gbrain config set self_upgrade.quiet_hours + '{"start":23,"end":8,"tz":"US/Pacific"}'`. See + [upgrades-auto-update.md](upgrades-auto-update.md). +- **Cron prompts** — schedule-driven notification jobs should carry the + gate described in this doc; [cron-schedule.md](cron-schedule.md) covers + the scheduling side. + +The shell pattern below is for everything else: your own cron jobs, +collectors, and notification paths that gbrain doesn't gate for you. + ### Configurable Hours Some users want different quiet hours. Store the config: @@ -3327,7 +3368,11 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring) skill reads and clears the held directory. Orphaned held files mean the pickup integration is broken. -3. **Timezone auto-detection is fragile.** Calendar-based timezone detection +3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a + held message must not be lost across a restart, use a durable held + directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`. + +4. **Timezone auto-detection is fragile.** Calendar-based timezone detection relies on the user having airline/hotel events with location data. If the user books travel without calendar entries, the system won't detect the move. Fall back to activity-hour analysis (responding at 3 AM PT = probably @@ -3486,9 +3531,9 @@ you can use as a reference shape. The skillpack story for distributing your own resolvers across machines is covered in [skillpacks as scaffolding](skillpacks-as-scaffolding.md). -## The compact list format (v0.41.7.0) +## The compact list format -GBrain's resolver parser used to require markdown tables: +GBrain's resolver parser originally required markdown tables: ```markdown | Trigger | Skill | @@ -3505,14 +3550,14 @@ format that scales better: - **flight-tracker**: track my flight | flight status | when does my flight land ``` -Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a -306-skill compact-format resolver, the doctor reported every skill as -unreachable: **238 FAIL errors on every doctor run**. The parser was -silently treating the compact dialect as zero skills. +When `gbrain doctor` only spoke the table dialect, a 306-skill +compact-format resolver reported every skill as unreachable: **238 FAIL +errors on every doctor run**. The parser was silently treating the compact +dialect as zero skills. -v0.41.7.0 ships dual-format support. The same `parseResolverEntries` -function reads both table rows and list rows in the same file, with the -v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace +Today the parser supports both. The same `parseResolverEntries` +function reads table rows and list rows in the same file, with the +multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace `../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor` and the 238 FAILs collapse to 0. @@ -3626,8 +3671,7 @@ I initially converted my resolver from a clean list format to a table format because the validator only spoke tables. That was wrong. When a tool fails against valid data, the right move is to fix the tool, not reshape the data. The list format was correct, compact, readable, easy -to maintain. The parser needed to support both shapes. v0.41.7.0 is -that fix. +to maintain. The parser needed to support both shapes — and now it does. The same principle applies everywhere in agent systems. Your SKILL.md is the source of truth. Your AGENTS.md is the source of truth. Your resolver @@ -3762,18 +3806,17 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY # Deploy GBrain Remote MCP Server -> **v0.26.0+:** `gbrain serve --http` ships full OAuth 2.1 (client credentials, -> auth code + PKCE, refresh rotation, optional DCR), an embedded React admin -> dashboard at `/admin`, scoped operations, and a live SSE activity feed. -> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back -> to the `access_tokens` table and grandfathers tokens to `read+write+admin`. -> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only); -> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md) -> for env vars and tunable defaults. +> `gbrain serve --http` ships full OAuth 2.1 (client credentials, auth code + +> PKCE, refresh rotation, optional DCR), an embedded React admin dashboard at +> `/admin`, scoped operations, and a live SSE activity feed. Legacy bearer +> tokens still work — `verifyAccessToken` falls back to the `access_tokens` +> table and grandfathers tokens to `read+write+admin`. Both the OAuth surface +> and the bearer fallback work on both engines (PGLite and Postgres). See +> [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults. Access your brain from any device, any AI client. GBrain ships two transports: -`gbrain serve` (stdio) for local agents, and `gbrain serve --http` (v0.26.0+) -for remote clients over OAuth 2.1. +`gbrain serve` (stdio) for local agents, and `gbrain serve --http` for remote +clients over OAuth 2.1. ## Three Paths @@ -3786,7 +3829,7 @@ gbrain serve Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio. No server, no tunnel, no token needed. Works on both PGLite and Postgres engines. -### Remote over OAuth 2.1 (recommended, v0.26.0+) +### Remote over OAuth 2.1 (recommended) ```bash gbrain serve --http --port 3131 @@ -3807,28 +3850,27 @@ Supported clients: - **Perplexity** — OAuth 2.1 client credentials grant. - **Claude Code, Cursor, Windsurf** — can use OAuth or legacy bearer. -See the [OAuth 2.1 setup](#oauth-21-setup-v100) section below. +See the [OAuth 2.1 setup](#oauth-21-setup) section below. -### Remote with legacy bearer tokens (pre-v0.26 deployments) — Postgres only +### Remote with legacy bearer tokens (simplest) ``` Your AI client (Claude Desktop, Perplexity, etc.) → ngrok tunnel (https://YOUR-DOMAIN.ngrok.app) → gbrain serve --http (built-in transport with bearer auth) - → Postgres (pooler connection or self-hosted) + → Postgres or PGLite ``` This requires: -1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres; - running `gbrain serve --http` against a PGLite install fails fast at startup) -2. A machine running `gbrain serve --http` -3. A public tunnel (ngrok, Tailscale, or cloud host) -4. A bearer token created via `gbrain auth create ` +1. A machine running `gbrain serve --http` (works on both PGLite and Postgres + brains) +2. A public tunnel (ngrok, Tailscale, or cloud host) +3. A bearer token created via `gbrain auth create ` -Pre-v1.0 tokens are grandfathered as `read+write+admin` scopes when you upgrade -to the HTTP server, so no migration is required. +Existing bearer tokens are grandfathered as `read+write+admin` scopes on the +OAuth-capable HTTP server, so no migration is required. -## OAuth 2.1 Setup (v0.26.0+) +## OAuth 2.1 Setup ### 1. Start the HTTP server @@ -3854,8 +3896,8 @@ Save this token. Open `http://localhost:3131/admin` and paste it to access the dashboard. The dashboard shows live activity, registered clients, request logs, and per-client config export. -> **v0.26.9+:** `mcp_request_log.params` and the live SSE activity feed default -> to a redacted summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. +> `mcp_request_log.params` and the live SSE activity feed default to a redacted +> summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. > Declared param keys are kept (intersected against the operation's spec); unknown > keys are counted but never named, and byte sizes round up to 1KB so size-probe > attacks can't binary-search secret content. Operators on a personal laptop who @@ -3886,9 +3928,9 @@ gbrain auth register-client perplexity \ --scopes "read write" ``` -**v0.34 — source-scoped clients.** Multi-source brains can scope a client's -write authority to one source and its read scope to a curated set with the -new `--source` and `--federated-read` flags: +**Source-scoped clients.** Multi-source brains can scope a client's write +authority to one source and its read scope to a curated set with the +`--source` and `--federated-read` flags: ```bash gbrain auth register-client dept-x-agent \ @@ -3900,9 +3942,12 @@ gbrain auth register-client dept-x-agent \ `--source` controls the write authority — `put_page` / `add_link` / etc only land in `dept-x`. `--federated-read` controls the read axis independently; -queries return rows from any of the listed sources. Omit both flags for the -v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to -`source_id='default'` on `gbrain upgrade`. +queries return rows from any of the listed sources. Omit both flags for an +unscoped super-client. Clients registered before source scoping existed are +backfilled to `source_id='default'` on `gbrain upgrade`. Within a source, +slug-level write fencing is also available: `--bound-slug-prefixes p1/,p2/` +rejects slug-mutating writes outside the listed prefixes (update later with +`gbrain auth rescope-client --bound-slug-prefixes `). Host-repo wrappers can register programmatically: @@ -3920,7 +3965,7 @@ start the server with `--enable-dcr`. DCR is off by default. ### 3. Expose the server -**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`. +**Bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`. To accept connections from the ngrok tunnel (or any non-loopback source), restart with `--bind`: @@ -3944,10 +3989,10 @@ router exposes the spec-compliant discovery endpoint at ### 4. Scopes and localOnly -Every operation is tagged `read | write | admin`. Four operations are -`localOnly` and rejected over HTTP regardless of scope: `sync_brain`, -`file_upload`, `file_list`, `file_url`. Remote agents cannot reach local -filesystem surface area. +Every operation is tagged `read | write | admin`. Operations flagged +`localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain` and +the `file_*` ops among them) are rejected over HTTP regardless of scope. +Remote agents cannot reach local filesystem surface area. | Scope | What it allows | |-------|---------------| @@ -3955,10 +4000,13 @@ filesystem surface area. | `write` | `put_page`, `delete_page`, `add_link`, `add_timeline_entry` | | `admin` | Client management, token revocation, sweep, local-only ops | +Write ops can additionally be fenced per client with `--bound-slug-prefixes` +(see [Register OAuth clients](#2-register-oauth-clients) above). + ## Legacy Bearer Token Setup -Keep using pre-v0.26 bearer tokens if you aren't ready to migrate. They -grandfather to `read+write+admin` scopes on the HTTP server. +Bearer tokens are the simple path when you don't need per-client scoping. +They grandfather to `read+write+admin` scopes on the HTTP server. ### 1. Set up the tunnel @@ -4005,15 +4053,20 @@ gbrain auth test \ ## Operations -All 30 GBrain operations are available remotely, including `sync_brain` and -`file_upload` (no timeout limits with self-hosted server). +GBrain's full operation catalog (100+ operations in `src/core/operations.ts`) +is available remotely, with no timeout limits on a self-hosted server. The +only exceptions are the operations flagged `localOnly: true` — `sync_brain` +and the `file_*` ops among them — which are rejected over HTTP regardless of +scope (see [Scopes and localOnly](#4-scopes-and-localonly) above). -**Security note on `file_upload`:** remote MCP callers are confined to the working -directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute -paths outside cwd are rejected. Page slugs and filenames are allowlist-validated -(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local -CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since -the user owns the machine. +**Security note on file access:** the `file_*` operations being localOnly is +the first line of defense; as defense-in-depth, `file_upload` also confines +any caller that isn't verifiably the trusted local CLI to the working +directory where `gbrain serve` was launched. Symlinks, `..` traversal, and +absolute paths outside cwd are rejected, and page slugs and filenames are +allowlist-validated (alphanumeric + hyphens; no control chars, RTL overrides, +or backslashes). Local CLI callers (`gbrain files upload ...`) keep +unrestricted filesystem access since the user owns the machine. ## Deployment Options @@ -4083,8 +4136,8 @@ Remote servers must be added via Settings > Integrations, NOT | put_page | 100-500ms | Write + trigger search_vector update | | get_stats | < 100ms | Aggregate query | -**Note:** `gbrain serve --http` shipped in v0.26.0 with OAuth 2.1 + admin -dashboard baked into the binary. The custom HTTP wrapper pattern (see +**Note:** `gbrain serve --http` has OAuth 2.1 + the admin dashboard baked +into the binary. The custom HTTP wrapper pattern (see [voice recipe](../../recipes/twilio-voice-brain.md)) is still supported for teams that need bespoke middleware, but for most remote deployments the built-in server is the recommended path. @@ -4101,6 +4154,14 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_VER # GBrain Installation Verification Runbook +> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install +> contract (round-trip, graph floor, and more) automatically and exits non-zero +> on failure — it is the modern first thing to run after any install. See +> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the +> **manual, deep-verification** companion: use it when `bootstrap verify` fails +> and you need to isolate which layer broke, or when you want to understand +> what "healthy" looks like check by check. + Run these checks after install to confirm every part of GBrain is working. Each check includes the command, expected output, and what to do if it fails. @@ -4121,7 +4182,8 @@ gbrain doctor --json **Expected:** All checks return `"ok"`: - `connection`: connected, N pages - `pgvector`: extension installed -- `rls`: enabled on all tables +- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains + skip this check; the embedded engine has no remote surface) - `schema_version`: current - `embeddings`: coverage percentage @@ -4134,12 +4196,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table. **Check:** Ask the agent: "What is the brain-agent loop?" -**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes -the read-write cycle: detect entities, read brain, respond with context, write -brain, sync. +**Expected:** The agent describes the read-write cycle documented in +[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities, +read brain, respond with context, write brain, sync. -**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the -install paste (read `docs/GBRAIN_SKILLPACK.md`). +**If it fails:** The agent hasn't loaded the skillpack. Have it read +`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links. --- @@ -4154,8 +4216,8 @@ gbrain check-update --json **Expected:** Returns JSON with `current_version`, `latest_version`, `update_available` (boolean). The cron `gbrain-update-check` is registered. -**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md -Section 17. +**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md) +for how to register the update-check cron. --- @@ -4189,8 +4251,9 @@ find /data/brain -name '*.md' \ Some difference is normal (files added since last sync), but if page count is less than half the file count, sync is silently skipping pages. -**If page count is way too low:** The #1 cause is an unreachable direct -connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543) +**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an +unreachable direct connection on an IPv4-only host. (PGLite brains have no +network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543) for reads, but routes migrations, DDL, and sync transactions to a derived direct connection (`db..supabase.co:5432`), which is IPv6-only. - On an IPv4-only host, reads work but sync transactions fail and silently skip @@ -4223,7 +4286,7 @@ This is the real test. Edit a brain page, push, wait, search. 1. Edit a page in the brain repo (e.g., correct a fact on a person's page): ```bash -# Example: fix a line in Gustaf's page +# Example: fix a line in alice-example's page cd /data/brain # Make a small edit to any .md file git add -A && git commit -m "test: verify live sync" && git push @@ -4354,19 +4417,23 @@ gbrain repair-jsonb Idempotent. PGLite brains always report 0 (unaffected by the original bug). -**Bonus check** — frontmatter-keyed queries actually resolve: +**Bonus check** — the doctor's dedicated JSONB scan agrees: ```bash -gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}' +gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*' ``` -If this returns rows on a brain with person pages, the JSONB path is healthy. +**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store +objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`. --- ## Quick Verification (all checks in one pass) ```bash +# 0. The one-command contract check (exits non-zero on failure) +gbrain bootstrap verify + # 1. Schema gbrain doctor --json @@ -4403,6 +4470,10 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/min # Minions fix — repairing a half-migrated install +> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're +> on any recent release, the canonical fix below (`gbrain apply-migrations +> --yes`) is all you need; the stopgap sections exist for archaeology. + **tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs still on `agentTurn`), run: @@ -4437,17 +4508,16 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`. gbrain doctor ``` -If the install is half-migrated, you'll see: +If the install is half-migrated, you'll see the `minions_migration` check +fail: ``` [FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes ``` -or - -``` -[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes -``` +(Missing `~/.gbrain/preferences.json` on a fresh install is a valid +pre-`apply-migrations` state — doctor deliberately does NOT fail on that +alone; the partial-migration record is the canonical half-migration signal.) For a machine-readable report (cron-friendly): diff --git a/recipes/agent-voice.md b/recipes/agent-voice.md index ae90385d0..58884ee7f 100644 --- a/recipes/agent-voice.md +++ b/recipes/agent-voice.md @@ -10,15 +10,9 @@ secrets: - name: OPENAI_API_KEY description: OpenAI API key with Realtime API access enabled where: https://platform.openai.com/api-keys — click "+ Create new secret key", copy immediately - - name: TWILIO_ACCOUNT_SID - description: (optional) Twilio Account SID — only if wiring inbound Twilio calls - where: https://www.twilio.com/console - - name: TWILIO_AUTH_TOKEN - description: (optional) Twilio auth token — only if wiring inbound Twilio calls - where: https://www.twilio.com/console health_checks: - type: env_exists - var: OPENAI_API_KEY + name: OPENAI_API_KEY label: OPENAI_API_KEY present setup_time: 10 min cost_estimate: "$0.06-0.24/min OpenAI Realtime, optional $1-2/mo Twilio number" @@ -34,20 +28,22 @@ A reference voice agent (WebRTC-first; OpenAI Realtime) shipped as **copy-into-y - **WebRTC browser client** at `/call?test=1` for the production-grade voice loop. Production load installs zero test instrumentation; `?test=1` enables Web Audio API tee → MediaRecorder capture for the E2E. - **Tool router** with a read-only allow-list by default (search, query, get_page, list_pages, find_experts, get_recent_salience, get_recent_transcripts, read_article). Write ops are denylisted; operators opt in to a bounded set via local override. - **Persona-aware prompt builder** with identity-first composition + Unicode sanitization for Realtime API safety. -- **Optional Twilio adapter** (`/voice` TwiML, WSS bridge) for phone inbound. Skip if you only want browser voice. +- **Optional Twilio adapter** (`/voice` TwiML, WSS bridge) for phone inbound. Skip if you only want browser voice. If you wire it, set `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` in `$TARGET_REPO/.env` (from https://www.twilio.com/console). They're deliberately NOT in this recipe's `secrets:` frontmatter — every listed secret must be set before the integration reports `configured`, and Twilio is genuinely optional. - **Three skills** for resolver routing: `voice-persona-mars`, `voice-persona-venus`, `voice-post-call`. -- **Unit + E2E tests** that ride with the copy. PII-shape regex guards every prompt, classifier triages upstream vs plumbing failures. +- **Unit tests** that ride with the copy (PII-shape regex guards every prompt; a classifier triages upstream vs plumbing failures). The E2E and eval suites stay gbrain-side under `recipes/agent-voice/tests/` — see Tests below. ## The skillpack-as-reference paradigm Earlier gbrain skillpacks installed to `~/.gbrain/skills//` as managed-block-canonical first-class skills. The user's local edits drifted from the canonical and updates were either "overwrite local" or "skip update" — neither is what an operator wants on code they've extended. -This recipe ships a different shape: gbrain holds the up-to-date REFERENCE, and `gbrain integrations install agent-voice --target ` COPIES it into the operator's repo. The code now lives in the host repo, on the operator's release cadence, with the operator's edits. Subsequent `--refresh` invocations diff host-side files against gbrain's reference and propose changes; the operator picks per-file (keep mine / take theirs / merge). +This recipe ships a different shape: gbrain holds the up-to-date REFERENCE, and `gbrain integrations install agent-voice --target ` COPIES it into the operator's repo. The code now lives in the host repo, on the operator's release cadence, with the operator's edits. Subsequent `--refresh` invocations diff host-side files against gbrain's reference and apply updates while preserving local edits by default (`--auto keep-mine|take-theirs` for CI lanes). The shipped reference does NOT contain personal names, hardcoded private paths, or upstream-agent codenames. A CI guard (`scripts/check-no-pii-in-agent-voice.sh`) blocks any drift back; a deterministic import script (`scripts/import-from-upstream.sh`) refreshes the gbrain reference from an upstream voice-agent source. ## Install +> Note: this recipe's category is `voice`, which the `gbrain integrations list` dashboard does not render (it shows infra / sense / reflex sections only). Install it directly by id, as below. + ```bash # 1. Detect target repo export TARGET_REPO=$OPENCLAW_WORKSPACE # or your agent repo path @@ -85,15 +81,9 @@ git -C $(which gbrain | xargs -I{} dirname {})/.. pull # or your gbrain update gbrain integrations install agent-voice --target $TARGET_REPO --refresh ``` -`--refresh` reads the `.gbrain-source.json` manifest written by the original install, re-computes per-file SHA-256 against gbrain's current reference, and classifies each file: +`--refresh` reads the `.gbrain-source.json` manifest written by the original install, re-computes per-file SHA-256 against gbrain's current reference, and classifies each file into one of six states (identical / stale / locally-modified / host-deleted / source-deleted / new-in-manifest). Stale and new files are updated automatically; **locally-modified files are preserved by default** (`--auto take-theirs` to overwrite; `--dry-run` to preview). An append-only audit journal is written to `/services/voice-agent/.gbrain-source.refresh.log`. -- **unchanged-identical** — host file matches gbrain reference; skip. -- **unchanged-stale** — host file matches the recorded SHA but reference moved; offer to update. -- **locally-modified** — host file diverges from the recorded SHA; show diff, offer three options (keep mine / take theirs / merge). -- **source-deleted** — gbrain reference removed a file; offer cleanup. -- **source-renamed** — detected via path-mapping; offer to follow. - -A transaction journal at `/services/voice-agent/.gbrain-source.refresh.log` allows partial-apply recovery if the refresh is interrupted. +The full state machine and per-state decisions live in [`recipes/agent-voice/install/refresh-algorithm.md`](agent-voice/install/refresh-algorithm.md) — the single home for refresh semantics. ## Architecture @@ -142,9 +132,15 @@ Reference code ships intentionally minimal. Before public deployment: ## Tests +The install copies the **unit suites only**; the E2E and eval suites stay gbrain-side under `recipes/agent-voice/tests/` (they carry puppeteer fixtures and live-API costs the host repo shouldn't inherit). + ```bash +# Host-side (rides with the copy) cd $TARGET_REPO/services/voice-agent -bun run test # host-side unit tests (5 suites, ~100 cases) +bun run test # unit tests + +# gbrain-side (from your gbrain checkout) +cd /recipes/agent-voice && bun install AGENT_VOICE_E2E=1 bun run test:e2e # WebRTC roundtrip (~$0.10/run) AGENT_VOICE_FULL_E2E=1 bun run test:full-flow # openclaw-driven install + roundtrip (~$1-2/run) ``` diff --git a/recipes/agent-voice/README.md b/recipes/agent-voice/README.md index 586de1120..8687f1167 100644 --- a/recipes/agent-voice/README.md +++ b/recipes/agent-voice/README.md @@ -25,10 +25,10 @@ recipes// ├── README.md # paradigm doc; gbrain-side only (not copied) ├── package.json # top-of-bundle; copied to /services//package.json ├── code/ # copied to /services//code/ -├── tests/ # copied to /services//tests/ -│ ├── unit/ -│ ├── e2e/ -│ └── evals/ +├── tests/ +│ ├── unit/ # copied to /services//tests/unit/ +│ ├── e2e/ # gbrain-side only (puppeteer + live-API costs) +│ └── evals/ # gbrain-side only (LLM-judge suites) ├── skills/ # copied to /skills// ├── install/ # gbrain-side only; install metadata │ ├── manifest.json # src → target map + per-file SHA-256 @@ -54,7 +54,7 @@ Three rules for new recipes following this shape: - `code/lib/personas/private-name-blocklist.json` — privacy guard source of truth (read by the shipped guard script and by host-side prompt-shape tests). - `code/lib/personas/context-builder.contract.md` — API the operator implements for live brain context. -## Files (in `bundle = code/ + tests/ + skills/ + package.json`) — copied to host repo +## Files (in `bundle = code/ + tests/unit/ + skills/ + package.json`) — copied to host repo The install subcommand reads `install/manifest.json` and copies each listed file to its target path under the host repo. SHA-256s computed at copy time get persisted into `/services//.gbrain-source.json` so `--refresh` can do three-way classification (unchanged-identical / unchanged-stale / locally-modified) without re-walking the entire bundle. diff --git a/recipes/agent-voice/install/post-install-hint.md b/recipes/agent-voice/install/post-install-hint.md index 66f5d716c..5dd9bfc58 100644 --- a/recipes/agent-voice/install/post-install-hint.md +++ b/recipes/agent-voice/install/post-install-hint.md @@ -67,9 +67,14 @@ bun run start # or `npm start` Open `http://localhost:8765/call` in a browser, click Connect, grant mic permission. You should be talking to Venus (or Mars if you set `DEFAULT_PERSONA=mars`). -### 6. (Optional) Run the WebRTC roundtrip E2E +### 6. (Optional) Run the WebRTC roundtrip E2E — from the gbrain checkout + +The install copies **unit tests only**. The E2E and eval suites stay gbrain-side +(under `recipes/agent-voice/tests/`), so run them from your gbrain checkout, +not from ``: ```bash +cd /recipes/agent-voice && bun install export AGENT_VOICE_E2E=1 OPENAI_API_KEY=sk-... bun run test:e2e # → ~$0.10/run; spawns server, drives puppeteer with a fake-audio WAV @@ -83,15 +88,15 @@ gbrain claw-test --scenario voice-agent-install --live --agent openclaw # → ~$1-2/run; friction-discovery test, NOT a ship gate ``` -### 7. (Optional) Run the LLM-judge persona evals +### 7. (Optional) Run the LLM-judge persona evals — from the gbrain checkout ```bash -cd /services/voice-agent +cd /recipes/agent-voice node tests/evals/mars-eval.mjs # ~$1-3 for the full 3-model judge sweep node tests/evals/venus-eval.mjs ``` -Synthetic canonical baselines are committed under `tests/evals/baseline-runs/canonical/`. Live receipts you generate go to `tests/evals/baseline-runs/` (gitignored — they may contain residual brain content from your live personas). +Live receipts you generate go to `tests/evals/baseline-runs/` (gitignored — they may contain residual brain content from your live personas). See `tests/evals/README.md` for the pass criteria and failure triage. ### 8. Update later @@ -101,4 +106,4 @@ When gbrain ships a new agent-voice reference, refresh your local copy: gbrain integrations install agent-voice --target --refresh ``` -The refresh classifies each file (identical / stale / locally-modified / source-deleted / host-deleted) and lets you decide per-file. See `/services/voice-agent/code/install/refresh-algorithm.md` (copied from gbrain) for the contract. +Refresh classifies each file (six states — identical / stale / locally-modified / host-deleted / source-deleted / new-in-manifest) and applies a deterministic decision per state: local edits are preserved by default; pass `--auto take-theirs` to take upstream everywhere, or `--dry-run` to preview. The full contract lives gbrain-side at `recipes/agent-voice/install/refresh-algorithm.md` (not copied to the host repo). diff --git a/recipes/agent-voice/install/refresh-algorithm.md b/recipes/agent-voice/install/refresh-algorithm.md index 35fd406cc..29390861f 100644 --- a/recipes/agent-voice/install/refresh-algorithm.md +++ b/recipes/agent-voice/install/refresh-algorithm.md @@ -1,74 +1,74 @@ # Refresh algorithm (diff-and-propose) -`gbrain integrations install agent-voice --refresh` re-walks the manifest, classifies every file into one of five states, and lets the operator decide per-file. The reference implementation is in `src/commands/integrations.ts` under the `install_kind: copy-into-host-repo` branch. +`gbrain integrations install agent-voice --refresh` re-walks the manifest, classifies every file into one of six states, and applies a deterministic decision per state. The implementation is in `src/commands/integrations.ts` under the `install_kind: copy-into-host-repo` branch (`refreshRecipeIntoHostRepo` / `classifyForRefresh`). + +This file is the single home for refresh semantics. `recipes/agent-voice.md` and `install/post-install-hint.md` summarize and link here. ## State machine -For each file declared in `install/manifest.json`: +For each file declared in `install/manifest.json` (plus each file in the prior install record): ``` -Let src_hash = SHA-256 of gbrain-side file at manifest.src -Let host_path = / -Let recorded = .gbrain-source.json.files[].sha256 for this entry (or absent if first refresh) -Let host_hash = SHA-256 of host_path (or absent if file deleted on host side) +Let src_hash = SHA-256 of gbrain-side file at manifest.src +Let host_path = / +Let recorded = .gbrain-source.json.files[].sha256 for this entry (absent if new) +Let host_hash = SHA-256 of host_path (absent if file missing on host side) -State: +State (and what refresh does about it): - "unchanged-identical" iff host_hash == src_hash → no-op - "unchanged-stale" iff host_hash == recorded AND host_hash != src_hash - → operator unmodified, source moved → offer update - - "locally-modified" iff host_hash != recorded AND host_hash != src_hash AND host_hash is defined - → operator edited locally; offer three options (see below) - - "host-deleted" iff host_hash is absent AND src exists - → operator removed the file; offer to restore or to remove from manifest - - "source-deleted" iff src is absent AND host_hash is defined - → gbrain reference removed the file; offer cleanup (remove from host) + → operator unmodified, source moved → auto-updated (copied over) + - "locally-modified" iff host_hash != recorded AND host_hash != src_hash AND host exists + → operator edited locally → default keep-mine; see below + - "host-deleted" iff host file absent AND src exists + → left deleted, UNLESS --auto take-theirs (restores the file) + - "source-deleted" iff entry in the prior record but not in the current manifest + → left in place ("orphan"), UNLESS --auto take-theirs (removes it) + - "new-in-manifest" iff entry in the manifest but not in the prior record + → auto-installed (copied in) ``` -A path-mapping renames table in the manifest (`renames: [{from, to}]`, not yet shipped) allows the refresh algorithm to detect a source-renamed file as a logical update rather than a delete+add. +There is no interactive per-file prompt: every run is non-interactive, and the only lever is `--auto keep-mine|take-theirs`. Without `--auto`, the defaults above apply (they match `--auto keep-mine`). Run `--dry-run` first to see the per-file classification before anything is written. -## "Locally-modified" decision +A path-mapping renames table in the manifest (`renames: [{from, to}]`, not yet shipped) would let refresh detect a source-renamed file as a logical update rather than a delete+add. -When a file shows `locally-modified`, the operator picks one of three options: +## The "locally-modified" decision -- **keep-mine** — leave host file untouched. The manifest entry's `sha256` is updated to the current host hash (the operator's edit becomes the new "recorded" baseline; future refreshes won't re-flag it until they edit it again OR the source changes). -- **take-theirs** — copy the gbrain reference over the host file. The recorded SHA becomes the new src_hash. -- **merge** — print a unified diff. Operator hand-merges in their editor; the refresh command exits without writing. Re-run `--refresh` after the merge to confirm. +- **keep-mine** (the default) — leave the host file untouched. The recorded `sha256` in `.gbrain-source.json` is re-baselined to the current host hash, so future refreshes won't re-flag this file until either side changes again. +- **take-theirs** (`--auto take-theirs`) — copy the gbrain reference over the host file. The recorded SHA becomes the new src_hash. -## Transaction journal +There is no `merge` option and no diff output. To hand-merge: run `--dry-run` to find locally-modified files, diff them yourself against the gbrain-side reference (the `src` path printed per file), merge in your editor, then re-run `--refresh`. -`/services/voice-agent/.gbrain-source.refresh.log` is a JSONL append-only file. Each line records: +## Transaction journal (audit log) + +`/services/voice-agent/.gbrain-source.refresh.log` is a JSONL append-only file. Each line records one refresh event: ```json -{"ts": "2026-05-17T12:34:56Z", "src": "code/server.mjs", "state": "locally-modified", "decision": "keep-mine"} +{"ts": "2026-05-17T12:34:56Z", "event": "preserved_local", "src": "code/server.mjs", "target": "services/voice-agent/code/server.mjs", "decision": "keep-mine"} ``` -The journal exists for two reasons: -1. **Partial-apply recovery.** If the refresh is interrupted mid-loop (Ctrl-C, crash, machine reboot), re-running `--refresh` reads the journal and resumes where it stopped. -2. **Audit.** Operators can grep the journal to see which files were touched and why. - -The journal is rotated by file size (>1MB triggers rename to `.gbrain-source.refresh.log.1`) and ignored by `--refresh`'s own scan (the journal is host-only metadata, not a managed file). - -## Concurrent refresh guard - -`--refresh` acquires an advisory file lock at `/services/voice-agent/.gbrain-source.refresh.lock` for the duration of the run. Concurrent `--refresh` invocations on the same host repo fail-fast with "refresh already in progress." +The journal is an **audit log only** — grep it to see which files were touched by which refresh and why. It is never read back by `--refresh` (every run re-classifies from scratch), it is not rotated, and it is ignored by the scan itself (host-only metadata, not a managed file). Delete or truncate it whenever you like. ## CLI surface ```bash gbrain integrations install agent-voice --target --refresh -gbrain integrations install agent-voice --target --refresh --dry-run # report-only -gbrain integrations install agent-voice --target --refresh --auto take-theirs # non-interactive -gbrain integrations install agent-voice --target --refresh --auto keep-mine # bias toward operator's edits +gbrain integrations install agent-voice --target --refresh --dry-run # report-only, per-file detail +gbrain integrations install agent-voice --target --refresh --auto take-theirs # always take upstream +gbrain integrations install agent-voice --target --refresh --auto keep-mine # explicit form of the default ``` -`--auto ` applies the named decision to ALL `locally-modified` files without prompting. Useful for CI lanes that want either "always take upstream" or "always preserve local" without operator interaction. +`--auto ` applies the named decision to ALL `locally-modified` files (and, for `take-theirs`, also restores host-deleted files and cleans up source-deleted orphans). Useful for CI lanes. ## What this v0 deliberately skips -- Conflict resolution for files that exist in both manifests but at different paths (treated as add+delete). -- Concurrent edits on the SAME file mid-refresh (the advisory lock + per-file atomic write covers this). -- Semantic merges (we offer file-level diff only; no per-hunk picking). -- Manifest schema migration (v0.1.0 → v0.2.0 changes are handled by the install command refusing to refresh old manifests and asking the operator to re-install). +- **Interactive per-file prompting and a merge option** — every run is batch; hand-merges happen in your editor between a `--dry-run` and a re-run. +- **Journal replay / partial-apply resume** — an interrupted refresh is simply re-run; classification is recomputed from scratch, and completed copies classify as `unchanged-identical` on the second pass. +- **Journal rotation** — the log grows unbounded (slowly); truncate it yourself if it bothers you. +- **A concurrent-refresh lock** — don't run two refreshes against the same host repo at once. +- Renamed-path detection (the `renames` table above). +- Semantic merges (file-level only; no per-hunk picking). +- Manifest schema migration (breaking manifest changes are handled by the install command refusing to refresh and asking the operator to re-install). Each of those is a follow-up TODO. diff --git a/recipes/agent-voice/skills/voice-persona-mars/SKILL.md b/recipes/agent-voice/skills/voice-persona-mars/SKILL.md index f5d158521..afb5d33ca 100644 --- a/recipes/agent-voice/skills/voice-persona-mars/SKILL.md +++ b/recipes/agent-voice/skills/voice-persona-mars/SKILL.md @@ -52,9 +52,9 @@ The persona prompt (`services/voice-agent/code/lib/personas/mars.mjs`) carries t ## Solo-mode tool posture -Mars uses tools SPARINGLY in solo mode. The right tools are: -- `search_brain` (find related concepts/people/meetings to deepen the reflection) -- `read_brain_page` (read a specific page aloud when the operator says "tell me about X") +Mars uses tools SPARINGLY in solo mode. The right tools (op names from the shipped `tools.mjs` allow-list) are: +- `search` (find related concepts/people/meetings to deepen the reflection) +- `get_page` (read a specific page aloud when the operator says "tell me about X") - `read_article` (summarize a link the operator shared) Calendar, tasks, email tools are DELIBERATELY ABSENT from Mars's solo-mode usage even though they're in the read-only allow-list. Mars redirects logistical questions to Venus. @@ -62,8 +62,8 @@ Calendar, tasks, email tools are DELIBERATELY ABSENT from Mars's solo-mode usage ## Demo-mode tool posture Mars uses tools AGGRESSIVELY in demo mode: -- Search the brain for people/companies the operator introduces -- Pull current events via `web_search` (when wired) +- Search the brain (`search` / `query`) for people/companies the operator introduces +- Summarize links via `read_article` (a general web-search tool is NOT in the shipped allow-list — wire one host-side if you want it) - Cross-reference what the operator is saying against the brain in near-real-time The goal: make the demo audience think "oh, this is what a personal AI can actually do." diff --git a/recipes/agent-voice/skills/voice-persona-venus/SKILL.md b/recipes/agent-voice/skills/voice-persona-venus/SKILL.md index 0feba009f..a065b69cf 100644 --- a/recipes/agent-voice/skills/voice-persona-venus/SKILL.md +++ b/recipes/agent-voice/skills/voice-persona-venus/SKILL.md @@ -45,16 +45,17 @@ Venus boots already knowing the topic's recent conversation. Only the `topicId` ## Tool posture -Venus uses the read-only allow-list from `services/voice-agent/code/tools.mjs`: +Venus uses the read-only allow-list from `services/voice-agent/code/tools.mjs` (op names as advertised to the model): -- `search_brain` (semantic + keyword search) -- `read_brain_page` (full page read aloud) +- `search` / `query` (semantic + keyword search) +- `get_page` (full page read aloud) / `list_pages` - `read_article` (URL fetch + summarize) -- `web_search` (when wired) - `get_recent_salience` (what's been emotionally active lately) - `get_recent_transcripts` (recent voice notes / meeting transcripts) - `find_experts` (who knows about a topic) +There is no general web-search tool in the shipped allow-list; wire one host-side if you want it. + Write tools (`put_page`, `submit_job`, `set_reminder` unless opted in, etc.) are NOT in Venus's tool surface. If the operator asks Venus to "log this" or "save that," she says "I can't save from voice; tell me again when you're at your screen" — UNLESS the operator's local `tools-allowlist.local.json` opts into the bounded write set. ## Language diff --git a/recipes/agent-voice/skills/voice-post-call/SKILL.md b/recipes/agent-voice/skills/voice-post-call/SKILL.md index cdad18c4a..7df716dab 100644 --- a/recipes/agent-voice/skills/voice-post-call/SKILL.md +++ b/recipes/agent-voice/skills/voice-post-call/SKILL.md @@ -1,7 +1,7 @@ --- name: voice-post-call version: 0.1.0 -description: Post-call handling for a voice session — turn the transcript into a brain page, post the summary to the operator's messaging surface, archive the audio. Belt-and-suspenders: fires both from a tool the voice persona can call mid-call AND from the automatic call-end handler in server.mjs. +description: Post-call handling for a voice session — turn the transcript into a brain page, post the summary to the operator's messaging surface, archive the audio. The pipeline is the contract; the firing paths are operator-wired (see "Two firing paths" below for what ships today). triggers: - "after the call" - "call ended" @@ -18,13 +18,11 @@ writes_to: # voice-post-call — Post-session transcript + summary handling -> **Convention:** see [conventions/quality.md](../conventions/quality.md) for citation rules + back-link enforcement. -> -> **Convention:** see [_brain-filing-rules.md](../_brain-filing-rules.md) for filing decision protocol. +> **Convention:** see gbrain's `skills/conventions/quality.md` for citation rules + back-link enforcement, and `skills/_brain-filing-rules.md` for the filing decision protocol. (These are not copied by the install; the relative paths resolve only if your host repo mirrors gbrain's skills layout.) ## Iron Law -**Every call gets processed, even on tool-call failure.** The voice persona MAY call a `log_call_summary` tool mid-session, OR the call may end without that tool firing (model forgot, WebRTC dropped, browser crashed). The automatic call-end handler in `services/voice-agent/code/server.mjs` posts a structured signal regardless so the brain still gets the transcript + audio reference. +**Every call gets processed, even on tool-call failure.** The voice persona MAY log mid-session via an opted-in write tool, OR the call may end without that tool firing (model forgot, WebRTC dropped, browser crashed). A call-end handler should post a structured signal regardless so the brain still gets the transcript + audio reference — see "Two firing paths" below for which of these ships today and which the operator implements. If both paths fire (the tool call AND the call-end handler), the second one is idempotent — it sees the brain page already exists and updates instead of duplicating. @@ -52,13 +50,13 @@ If both paths fire (the tool call AND the call-end handler), the second one is i Slack, Discord — whichever is wired in $TARGET_REPO/.env). ``` -## Two firing paths (belt + suspenders) +## Two firing paths (both operator-wired today) -**Path A — Persona-initiated mid-call:** -The voice persona calls `log_call_summary` via the WebRTC data channel. The host-repo `/tool` endpoint dispatches to `tools.mjs`. Note: `log_call_summary` is in `OPTIONAL_OPS`, not `READ_ONLY_OPS`, so this only works if the operator's `tools-allowlist.local.json` opts in. +**Path A — Persona-initiated mid-call (opt-in):** +The voice persona calls `log_to_brain` via the WebRTC data channel; the host-repo `/tool` endpoint dispatches through `tools.mjs`. `log_to_brain` is in `OPTIONAL_OPS`, not `READ_ONLY_OPS`, so this only works if the operator's `tools-allowlist.local.json` opts in (there is no `log_call_summary` tool — the override can only enable ops listed in `OPTIONAL_OPS`). -**Path B — Automatic call-end (default):** -When the WebSocket / WebRTC connection closes, `server.mjs` fires a `call_end` event. The host repo's post-call handler (operator-implemented; the recipe ships a stub) reads the captured audio + transcript, runs the pipeline above. This path requires NO operator opt-in to work — the call-end handler is part of the shipped server. +**Path B — Call-end handler (not yet shipped):** +The shipped `server.mjs` has **no automatic call-end handler** — nothing fires when the WebSocket / WebRTC connection closes. To get the safety-net behavior, implement a post-call handler in your host repo that reads the captured audio + transcript on connection close and runs the pipeline above. Until you do, Path A (opt-in) is the only firing path, and calls where the persona never logs are NOT processed. ## Brain page format @@ -116,10 +114,15 @@ created: 2026-05-17 ## Related skills +Ships with this bundle (sibling directories after install): + - [voice-persona-mars](../voice-persona-mars/SKILL.md) — the persona that may invoke this - [voice-persona-venus](../voice-persona-venus/SKILL.md) — the other persona that may invoke this -- [meeting-ingestion](../meeting-ingestion/SKILL.md) — analogous flow for multi-party meeting transcripts (different in that voice-call is typically 1:1) -- [voice-note-ingest](../voice-note-ingest/SKILL.md) — for recorded one-way voice memos (different from live voice calls) + +Lives in gbrain's `skills/` (present on the host only if your repo mirrors gbrain's skills layout): + +- `meeting-ingestion` — analogous flow for multi-party meeting transcripts (different in that voice-call is typically 1:1) +- `media-ingest` — for recorded one-way voice memos (different from live voice calls) ## Contract diff --git a/recipes/agent-voice/tests/evals/README.md b/recipes/agent-voice/tests/evals/README.md index 98bf2c4c1..035a62116 100644 --- a/recipes/agent-voice/tests/evals/README.md +++ b/recipes/agent-voice/tests/evals/README.md @@ -13,10 +13,10 @@ Pass criterion: every axis mean ≥ 7/10 AND no model scored any axis < 5 AND ## Running ```bash -# All four eval suites at the default judge tier (~$1-3/full run) -bun run gen:baselines # mars-eval + venus-eval + persona-routing + mars-multilingual +# Baseline receipts for the two persona evals (~$1-3/full run) +bun run gen:baselines # mars-eval --baseline + venus-eval --baseline -# Individually +# Individually (all four suites) node tests/evals/mars-eval.mjs node tests/evals/venus-eval.mjs node tests/evals/persona-routing-eval.mjs @@ -45,9 +45,9 @@ Capped well below the $1-3 budget. Cost stays low because the judge runs are sho ## Receipts -`baseline-runs/canonical/*.json` carries **agent-authored synthetic exemplars** — what a passing eval verdict looks like, with no real model output. Used for code-review and onboarding ("what does the harness produce?") without ever shipping residual private context. +`baseline-runs/canonical/` is **reserved for agent-authored synthetic exemplars** — what a passing eval verdict looks like, with no real model output — for code-review and onboarding ("what does the harness produce?") without ever shipping residual private context. No exemplars are committed yet; see `canonical/README.md` for the contract they must follow. (Note for whoever lands them: the sibling `.gitignore`'s `!canonical/` pattern does not unignore files inside the directory — it needs `!canonical/*.json`.) -`baseline-runs/*.json` (non-`canonical/`) is **gitignored**. Live receipts you generate against your own scrubbed personas live there; never commit them — they may carry response text that leaks operator-specific configuration. +`baseline-runs/*.json` is **gitignored**. Live receipts you generate against your own scrubbed personas live there; never commit them — they may carry response text that leaks operator-specific configuration. ## When evals fail diff --git a/recipes/agent-voice/tests/evals/baseline-runs/canonical/README.md b/recipes/agent-voice/tests/evals/baseline-runs/canonical/README.md index eb6249425..8bf7cd3d8 100644 --- a/recipes/agent-voice/tests/evals/baseline-runs/canonical/README.md +++ b/recipes/agent-voice/tests/evals/baseline-runs/canonical/README.md @@ -1,12 +1,12 @@ # Canonical baselines (synthetic exemplars) -These JSON files are **agent-authored synthetic exemplars** — what a passing eval verdict looks like. They contain NO real model output, NO real persona responses, NO operator-specific brain content. PII-impossible by construction. +This directory is reserved for **agent-authored synthetic exemplars** — what a passing eval verdict looks like. Exemplars must contain NO real model output, NO real persona responses, NO operator-specific brain content: PII-impossible by construction. **None are committed yet**; whoever lands the first ones must also fix the parent `.gitignore` (its `!canonical/` pattern doesn't unignore files inside the directory — it needs `!canonical/*.json`). -Use them as: +Once landed, use them as: 1. **Code-review reference** — when reviewing changes to `judge.mjs` or the persona prompts, eyeball these to see what the receipt schema looks like. 2. **Onboarding** — new contributors can read these to understand what the eval suite produces without spending API tokens. 3. **Schema documentation** — the field shape is the contract that live receipts must match. **Never commit live receipts here.** Live receipts go in `../` (gitignored). The canonical/ subdirectory is the ONLY committed eval output in the entire bundle. -If the eval harness changes its receipt schema, regenerate these by hand-editing the JSON to match — do NOT generate them by running the harness against the real personas. +If the eval harness changes its receipt schema, regenerate exemplars by authoring the JSON to match the new schema — do NOT generate them by running the harness against the real personas. diff --git a/recipes/calendar-to-brain.md b/recipes/calendar-to-brain.md index 558fd27e3..64bd1d613 100644 --- a/recipes/calendar-to-brain.md +++ b/recipes/calendar-to-brain.md @@ -90,7 +90,7 @@ Agent reads daily files - 09:00-09:30 **Team standup** (Work) — with Alice, Bob, Carol - 10:00-11:00 **Board meeting** (Work) 📍 Office — with Diana, Eduardo, Fiona -- 12:00-13:00 **Lunch with Pedro** (Personal) 📍 Chez Panisse — with Pedro Franceschi +- 12:00-13:00 **Lunch with Charlie** (Personal) 📍 A Restaurant — with charlie-example - 14:00-14:30 **1:1 with Jordan** (Work) — with Jordan Lee ``` @@ -113,79 +113,26 @@ This builds the full relationship graph from day one. ## Setup Flow -### Step 1: Choose and Configure Calendar Access +### Step 1: Configure Calendar Access (via credential-gateway) -Ask the user: "How do you want to connect to Google Calendar? +Credential setup (ClawVisor vs direct Google OAuth, consent screen, validation +commands) lives in ONE place: run the **[credential-gateway](credential-gateway.md)** +recipe first — this recipe declares `requires: [credential-gateway]` for exactly +that reason. Then apply the two calendar-specific details: -**Option A: ClawVisor (recommended)** -ClawVisor handles OAuth, token refresh, and encryption. You never touch Google -credentials directly. If you already use ClawVisor for email, this uses the same setup. +- **Option A (ClawVisor):** activate the **Google Calendar** service and use a + task purpose like: "Full calendar access for historical backfill and ongoing + sync. List events, read event details, search across all calendars." + (Be EXPANSIVE — narrow purposes block requests; see credential-gateway's + Tricky Spots.) +- **Option B (direct OAuth):** the scope is + `https://www.googleapis.com/auth/calendar.readonly`, and the sync script's + OAuth flow stores tokens in `~/.gbrain/google-tokens.json` (auto-refreshes + on expiry). Also enable the Calendar API at + https://console.cloud.google.com/apis/library/calendar-json.googleapis.com -**Option B: Google OAuth2 directly** -Connect to Google Calendar API directly. No extra service needed, but you manage -OAuth tokens yourself. Good if you don't want another dependency." - -#### Option A: ClawVisor Setup - -Tell the user: -"I need your ClawVisor URL and agent token. -1. Go to https://clawvisor.com -2. Create an agent (or use existing) -3. Activate the **Google Calendar** service -4. Create a standing task with purpose: 'Full calendar access for historical - backfill and ongoing sync. List events, read event details, search across - all calendars.' - IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes block requests. -5. Copy the gateway URL and agent token" - -Validate: -```bash -curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL" -``` - -**STOP until ClawVisor validates.** - -#### Option B: Google OAuth2 Setup - -Tell the user: -"I need Google OAuth2 credentials. Here's exactly how to set them up: - -1. Go to https://console.cloud.google.com/apis/credentials - (create a Google Cloud project if you don't have one) -2. Click **'+ CREATE CREDENTIALS'** at the top, select **'OAuth client ID'** -3. If prompted, configure the OAuth consent screen first: - - User type: **External** (or Internal if you have Google Workspace) - - App name: anything (e.g., 'GBrain Calendar') - - Scopes: add **'Google Calendar API .../auth/calendar.readonly'** - - Test users: add your own email -4. Back on Credentials, create the OAuth client ID: - - Application type: **Desktop app** - - Name: anything (e.g., 'GBrain') -5. Click **'Create'**. You'll see the Client ID and Client Secret. -6. Copy both and paste them to me. - -Also enable the Calendar API: -7. Go to https://console.cloud.google.com/apis/library/calendar-json.googleapis.com -8. Click **'Enable'**" - -Validate the credentials are set: -```bash -[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \ - && echo "PASS: Google OAuth credentials set" \ - || echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET" -``` - -Then run the OAuth flow to get an access token: -```bash -# The sync script should handle the OAuth flow: -# 1. Open browser to Google auth URL with calendar.readonly scope -# 2. User grants access -# 3. Script receives auth code, exchanges for access + refresh token -# 4. Stores tokens in ~/.gbrain/google-tokens.json -# 5. Auto-refreshes on expiry -``` - -**STOP until OAuth flow completes and tokens are stored.** +**STOP until credential-gateway's validation passes** (ClawVisor `/health` OK, +or OAuth tokens stored). ### Step 2: Identify Calendar Accounts @@ -281,7 +228,7 @@ gbrain sync --no-pull --no-embed && gbrain embed --stale ```bash mkdir -p ~/.gbrain/integrations/calendar-to-brain -echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"accounts":"ACCOUNT_COUNT","start_year":"YYYY"}}' >> ~/.gbrain/integrations/calendar-to-brain/heartbeat.jsonl +echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.0","status":"ok","details":{"accounts":"ACCOUNT_COUNT","start_year":"YYYY"}}' >> ~/.gbrain/integrations/calendar-to-brain/heartbeat.jsonl ``` Tell the user: "Calendar-to-brain is set up. You have [N] days of calendar history @@ -322,7 +269,7 @@ filter_attendees(attendees): return attendees.filter(a => !a.email?.includes('@resource.calendar.google.com') AND // conference rooms !a.email?.includes('@group.calendar.google.com') AND // mailing lists - !a.name?.startsWith('YC-SF-') // internal distros + !a.name?.startsWith('ORG-') // internal distros (use your org's prefix) ) ``` diff --git a/recipes/email-to-brain.md b/recipes/email-to-brain.md index 103304bfa..058f69273 100644 --- a/recipes/email-to-brain.md +++ b/recipes/email-to-brain.md @@ -104,76 +104,32 @@ Every email gets a baked-in Gmail link: `[Open in Gmail](https://mail.google.com 3. **Gmail access** via one of: - ClawVisor (recommended: E2E encrypted credential gateway) - Google OAuth credentials (direct API access) - - Hermes Gateway (built-in Gmail connector) + - Your harness's own Gmail connector, if it ships one (e.g. Hermes Gateway) — + this recipe carries no setup steps for that path; follow your harness's docs, + then continue at Step 2 ## Setup Flow -### Step 1: Validate Credential Gateway +### Step 1: Configure Gmail Access (via credential-gateway) -Ask the user: "How do you access Gmail programmatically? Options: -1. ClawVisor (recommended, handles OAuth and encryption) -2. Google OAuth credentials (you manage tokens yourself) -3. Hermes Gateway (if you're using Hermes Agent)" +Credential setup (ClawVisor vs direct Google OAuth, consent screen, validation +commands) lives in ONE place: run the **[credential-gateway](credential-gateway.md)** +recipe first — this recipe declares `requires: [credential-gateway]` for exactly +that reason. Then apply the two Gmail-specific details: -#### Option A: ClawVisor (recommended) +- **Option A (ClawVisor):** activate the **Gmail** service and use a task purpose + like: "Full executive assistant email management including inbox triage, + searching by any criteria, reading emails, tracking threads." + (Be EXPANSIVE — narrow purposes like "email triage" cause legitimate requests + to fail verification; see credential-gateway's Tricky Spots.) +- **Option B (direct OAuth):** the scope is + `https://www.googleapis.com/auth/gmail.readonly`, and the collector script's + OAuth flow stores tokens in `~/.gbrain/google-tokens.json` (auto-refreshes on + expiry). Also enable the Gmail API at + https://console.cloud.google.com/apis/library/gmail.googleapis.com -Tell the user: -"I need your ClawVisor URL and agent token. -1. Go to https://clawvisor.com -2. Create an agent (or use existing) -3. Activate the Gmail service -4. Create a standing task with purpose: 'Full executive assistant email management - including inbox triage, searching by any criteria, reading emails, tracking threads' - IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes like 'email triage' - will cause legitimate requests to fail verification. -5. Copy the gateway URL and agent token" - -Validate: -```bash -curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL" -``` - -**STOP until ClawVisor validates.** - -#### Option B: Google OAuth2 directly - -Tell the user: -"I need Google OAuth2 credentials for Gmail access. Here's how: - -1. Go to https://console.cloud.google.com/apis/credentials - (create a Google Cloud project if you don't have one) -2. Click **'+ CREATE CREDENTIALS'** > **'OAuth client ID'** -3. If prompted, configure the OAuth consent screen: - - User type: **External** (or Internal for Google Workspace) - - App name: 'GBrain Email' (anything works) - - Scopes: add **'Gmail API .../auth/gmail.readonly'** - - Test users: add your own email address -4. Create the OAuth client ID: - - Application type: **Desktop app** - - Name: 'GBrain' -5. Copy the **Client ID** and **Client Secret** -6. Also enable the Gmail API: - Go to https://console.cloud.google.com/apis/library/gmail.googleapis.com - Click **'Enable'**" - -Validate: -```bash -[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \ - && echo "PASS: Google OAuth credentials set" \ - || echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET" -``` - -Then run the OAuth flow to get tokens: -```bash -# The collector script handles the OAuth flow: -# 1. Opens browser to Google consent URL with gmail.readonly scope -# 2. User grants access -# 3. Script receives auth code, exchanges for access + refresh token -# 4. Stores tokens in ~/.gbrain/google-tokens.json -# 5. Auto-refreshes on expiry -``` - -**STOP until OAuth flow completes and tokens are stored.** +**STOP until credential-gateway's validation passes** (ClawVisor `/health` OK, +or OAuth tokens stored). ### Step 2: Set Up the Email Collector diff --git a/recipes/meeting-sync.md b/recipes/meeting-sync.md index 538962fc2..a34d17e42 100644 --- a/recipes/meeting-sync.md +++ b/recipes/meeting-sync.md @@ -18,6 +18,8 @@ health_checks: Content-Type: "application/json" body: '{"jsonrpc":"2.0","method":"tools/list","id":1}' label: "Circleback API" +output_paths: + - meetings/ setup_time: 15 min cost_estimate: "$0-17/mo (Circleback free for 10 meetings/mo, Pro $17/mo unlimited)" --- @@ -104,7 +106,7 @@ tags: [team, weekly, sync] ``` **Attendee filtering:** -- Skip calendar resources (e.g., "YC-SF Conference Room") +- Skip calendar resources (e.g., "HQ Conference Room") - Skip group addresses (e.g., "team@company.com") - Extract display names, not email addresses diff --git a/recipes/ngrok-tunnel.md b/recipes/ngrok-tunnel.md index 352e39adf..b8798b848 100644 --- a/recipes/ngrok-tunnel.md +++ b/recipes/ngrok-tunnel.md @@ -13,8 +13,10 @@ health_checks: - type: command argv: ["pgrep", "-f", "ngrok.*http"] label: "ngrok process" - - type: http - url: "http://localhost:4040/api/tunnels" + # NOTE: this must stay a `command` check. The `http` check type blocks + # internal/loopback URLs (SSRF guard), so `http: localhost:4040` can never pass. + - type: command + argv: ["curl", "-sf", "http://localhost:4040/api/tunnels"] label: "ngrok API" setup_time: 10 min cost_estimate: "$8/mo for Hobby tier (fixed domain). Free tier works but URLs change on restart." @@ -43,17 +45,21 @@ never changes. ``` Local services (your machine) - ├── GBrain MCP server (port 3000) gbrain serve + ├── GBrain MCP server (port 3131) gbrain serve --http [--port N] + │ (plain `gbrain serve` is stdio-only — not tunnelable) └── Voice agent (port 8765) node server.mjs │ ▼ ngrok tunnel (fixed domain) - └── https://your-brain.ngrok.app - │ - ├── /mcp → Claude Desktop, Claude Code, Perplexity - └── /voice → Twilio webhooks + └── https://your-brain.ngrok.app → ONE local port per tunnel ``` +A single `ngrok http ` forwards ALL paths to that one port. To serve both +`/mcp` (→ 3131) and `/voice` (→ 8765) on one domain you need either an ngrok +traffic policy (path-based routing, see ngrok's docs) or a local reverse proxy +(Caddy/nginx) in front of both services — or simply tunnel whichever single +service you need (most voice installs only tunnel 8765 for Twilio). + ## Setup Flow ### Step 1: Create ngrok Account + Get Hobby Tier @@ -221,8 +227,11 @@ to debug MCP connection issues (see request/response headers, latency, errors). watchdog handles Twilio, but Claude Desktop and Perplexity must be manually reconfigured. This is why Hobby ($8/mo) is worth it. -3. **One domain, multiple services.** Hobby gives 1 free domain. Route by path - (`/mcp`, `/voice`) on one domain, or pay $8/mo more for a second domain. +3. **One domain, multiple services.** Hobby gives 1 free domain, and a bare + `ngrok http ` sends every path to that single port. To route `/mcp` + and `/voice` to different local ports on one domain you need an ngrok + traffic policy or a local reverse proxy (see Architecture above) — or pay + $8/mo more for a second domain and run one tunnel per service. 4. **The watchdog must run on startup.** If the machine reboots, ngrok won't auto-start unless you have a watchdog cron or systemd service. @@ -231,7 +240,8 @@ to debug MCP connection issues (see request/response headers, latency, errors). 1. Start tunnel. Visit `https://your-brain.ngrok.app` in a browser. You should see a response (health check or default page). -2. From Claude Desktop, run `gbrain search "test"`. Results should come back. +2. From Claude Desktop, ask it to search your brain (it invokes the MCP `search` + tool over the tunnel). Results should come back. 3. Kill ngrok. Wait 2 minutes. Check the watchdog restarted it. 4. From a different device (phone), access the same URL. Verify it works. diff --git a/recipes/restart-sweep.md b/recipes/restart-sweep.md index 95efe59ca..e6fbdc5aa 100644 --- a/recipes/restart-sweep.md +++ b/recipes/restart-sweep.md @@ -569,8 +569,9 @@ tail -20 ~/.gbrain/integrations/restart-sweep/cron.log ## Step 6: Verification -1. `gbrain integrations doctor restart-sweep` — should pass all three - health checks +1. `gbrain integrations doctor` — restart-sweep's three health checks + should pass (the command runs checks for ALL configured integrations; + it takes no per-recipe filter, so look for the restart-sweep rows) 2. `~/.gbrain/integrations/restart-sweep/sweep.log.jsonl` exists and gets a new entry every 5 minutes 3. `~/.gbrain/integrations/restart-sweep/cron.log` shows successful diff --git a/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md b/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md index 3fec8b376..a2bdc804a 100644 --- a/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md +++ b/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md @@ -9,7 +9,8 @@ triggers: mutating: false writes_pages: false writes_to: [] -tools: [get_page, query, graph, backlinks] +# MCP op names; over the CLI, traverse_graph/get_backlinks are `gbrain graph` / `gbrain backlinks` +tools: [get_page, query, traverse_graph, get_backlinks] --- # Retrieval Reflex — retrieve on demand, when an entity is salient diff --git a/recipes/twilio-voice-brain.md b/recipes/twilio-voice-brain.md index e44429d79..5050349a6 100644 --- a/recipes/twilio-voice-brain.md +++ b/recipes/twilio-voice-brain.md @@ -2,7 +2,7 @@ id: twilio-voice-brain name: Voice-to-Brain (DEPRECATED — see agent-voice) version: 0.8.2 -description: "DEPRECATED in v0.40.0.0. New installs use `gbrain integrations install agent-voice` — the copy-into-host-repo paradigm with WebRTC-first browser client + Mars/Venus personas + read-only tool router. This recipe stays for one release as redirect; will be removed in v0.41." +description: "DEPRECATED. New installs use `gbrain integrations install agent-voice` — the copy-into-host-repo paradigm with WebRTC-first browser client + Mars/Venus personas + read-only tool router. This recipe stays as a redirect for existing Twilio installs; it is frozen (no longer updated) and will be removed in a future release." category: sense requires: [ngrok-tunnel] secrets: @@ -33,15 +33,24 @@ cost_estimate: "$15-25/mo (Twilio number $1-2 + voice $0.01/min, OpenAI Realtime # Voice-to-Brain: Phone Calls That Create Brain Pages -> **⚠️ DEPRECATED as of v0.40.0.0.** New installs should use the [agent-voice](agent-voice.md) +> **⚠️ DEPRECATED.** New installs should use the [agent-voice](agent-voice.md) > recipe — a WebRTC-first voice agent with Mars + Venus personas, copy-into-host-repo -> install paradigm, and read-only tool router. This recipe stays for one release as a -> redirect for operators with existing Twilio installs. It will be removed in v0.41. +> install paradigm, and read-only tool router. This recipe stays as a redirect for +> operators with existing Twilio installs. It is **frozen**: no longer updated, and +> will be removed in a future release once existing installs have migrated. > > **Migration:** `gbrain integrations install agent-voice --target ` copies a > working reference into your host agent repo where you own the edits. The new recipe > includes a Twilio bridge in `code/lib/twilio-bridge.mjs` for operators who still want > phone inbound, but the WebRTC `/call?test=1` flow is the headline experience. +> +> **Where this recipe's content now lives (canonical homes):** +> - ngrok tunnel setup + watchdog → [ngrok-tunnel.md](ngrok-tunnel.md) +> - post-call transcript pipeline → the `voice-post-call` skill in the [agent-voice](agent-voice.md) bundle +> - voice production patterns (unicode sanitize, PII scrub, identity-first prompt, conversation timing) → shipped as code in the agent-voice bundle +> +> The copies below are kept only so existing installs have a self-contained runbook; +> they may drift from current gbrain behavior. Treat the homes above as authoritative. Call a phone number. Talk. A structured brain page appears with entity detection, @@ -478,7 +487,7 @@ fi ```bash mkdir -p ~/.gbrain/integrations/twilio-voice-brain -echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.1","status":"ok","details":{"phone":"TWILIO_NUMBER","deployment":"local+ngrok"}}' >> ~/.gbrain/integrations/twilio-voice-brain/heartbeat.jsonl +echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.2","status":"ok","details":{"phone":"TWILIO_NUMBER","deployment":"local+ngrok"}}' >> ~/.gbrain/integrations/twilio-voice-brain/heartbeat.jsonl ``` Tell the user: "Voice-to-brain is fully set up. Your number is [NUMBER]. Here's diff --git a/scripts/run-unit-parallel.sh b/scripts/run-unit-parallel.sh index 370fb0cf3..6b8dbd07b 100755 --- a/scripts/run-unit-parallel.sh +++ b/scripts/run-unit-parallel.sh @@ -13,7 +13,7 @@ # # Env overrides: # SHARDS=N same as --shards -# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 1500) +# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 2400) # GBRAIN_TEST_SHARD_KILL_AFTER grace after TERM before KILL (default 30) # GBRAIN_TEST_MAX_CONCURRENCY passed through to bun test (default 4) # diff --git a/skills/RESOLVER.md b/skills/RESOLVER.md index 44cb82d3d..241184617 100644 --- a/skills/RESOLVER.md +++ b/skills/RESOLVER.md @@ -2,6 +2,13 @@ This is the dispatcher. Skills are the implementation. **Read the skill file before acting.** If two skills could match, read both. They are designed to chain (e.g., ingest then enrich for each entity). +**Routing contract:** each skill's frontmatter `triggers:` array is the +authoritative routing signal — harnesses match inbound messages against it +(see `skills/_AGENT_README.md`). This file is the human-readable dispatch +map of the same routing: one place to scan every skill and its trigger +phrases. If a row here and a skill's frontmatter disagree, the frontmatter +wins; fix the row. + ## Always-on (every message) | Trigger | Skill | @@ -61,6 +68,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Save or load reports | `skills/reports/SKILL.md` | | "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` | | "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` | +| "optimize this skill", "tune the skill against the benchmark", "run skillopt", "make the skill better" | `skills/skill-optimizer/SKILL.md` | | "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` | | "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` | | "harvest this skill into gbrain", "publish this skill to gbrain", "lift this skill upstream", "share this skill with other gbrain clients", "promote my skill to gbrain" | `skills/skillpack-harvest/SKILL.md` | @@ -76,7 +84,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Trigger | Skill | |---------|-------| | "Set up GBrain", first boot | `skills/setup/SKILL.md` | -| "Now what?", "fill my brain", "cold start", "bootstrap", "import my data", "what should I import first" | `skills/cold-start/SKILL.md` | +| "Now what?", "fill my brain", "cold start", "bootstrap my data", "import my data", "what should I import first" | `skills/cold-start/SKILL.md` | +| "Install gbrain into this agent/harness", "agent workspace bootstrap", "gbrain bootstrap", "wire gbrain hooks", "set up the maintenance sweep" | Run `gbrain bootstrap` (paste-in harness install: hooks + sweep + config). See `docs/guides/bootstrap.md` | | "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` | | Brain health check, maintenance run | `skills/maintain/SKILL.md` | | "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) | @@ -133,4 +142,4 @@ These apply to ALL brain-writing skills: | "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` | | "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` | | "add a page type", "add a type to my schema", "schema author", "schema mutate", "schema pack add", "my brain has untyped pages", "propose new types from my corpus", "backfill page types", "evolve my schema", "researcher type", "make X an expert type" (dispatcher for: gbrain schema active/list/show/validate/graph/lint/stats/explain/use/downgrade/reload/init/fork/edit/diff/add-type/remove-type/update-type/add-alias/remove-alias/add-prefix/remove-prefix/add-link-type/remove-link-type/set-extractable/set-expert-routing/detect/suggest/review-candidates/review-orphans/sync) | `skills/schema-author/SKILL.md` | -| "unify my types", "migrate to gbrain-base-v2", "94 types to 14", "apply canonical taxonomy", "clean up my page types", "pack upgrade", "shrink type proliferation", "consolidate page types", "retype pages to canonical" (dispatcher for: gbrain onboard --check, gbrain onboard --check --explain, gbrain jobs submit unify-types, gbrain pages restore) | `skills/schema-unify/SKILL.md` | +| "unify my types", "migrate to gbrain-base-v2", "94 types to 14", "apply canonical taxonomy", "clean up my page types", "pack upgrade", "shrink type proliferation", "consolidate page types", "retype pages to canonical" (dispatcher for: gbrain onboard --check, gbrain onboard --check --explain, gbrain jobs submit unify-types, gbrain restore) | `skills/schema-unify/SKILL.md` | diff --git a/skills/_AGENT_README.md b/skills/_AGENT_README.md index 82b101a65..1ae8f490d 100644 --- a/skills/_AGENT_README.md +++ b/skills/_AGENT_README.md @@ -43,8 +43,12 @@ array. Substring match is the baseline. Semantic similarity (embedding or keyword expansion) is fine on top. When a trigger matches strongly, invoke the skill — read its SKILL.md body in full and follow the workflow described there. -**Do NOT** look for a managed-block table inside `RESOLVER.md` or `AGENTS.md`. -That pattern was retired in gbrain v0.36. Routing lives in frontmatter now. +**The routing contract:** frontmatter `triggers:` are authoritative. +`skills/RESOLVER.md` is the human-readable dispatch map of the same routing — +useful for scanning every skill and its trigger phrases in one place, and it +carries the disambiguation rules for overlapping matches. If the two disagree, +frontmatter wins. (There is no machine-managed block inside `RESOLVER.md` or +`AGENTS.md`; that pattern was retired.) ## When the user invokes a skill @@ -105,7 +109,8 @@ accidental or you want to fully reset to gbrain's current bundle. ## Removing a scaffolded skill -There is no `uninstall` command in v0.36. The files are yours. +There is no `uninstall` command (`gbrain skillpack uninstall` exits with an +error pointing here). The files are yours. ```bash rm -rf skills/ diff --git a/skills/briefing/SKILL.md b/skills/briefing/SKILL.md index 8d65c6767..cb8111b8f 100644 --- a/skills/briefing/SKILL.md +++ b/skills/briefing/SKILL.md @@ -97,7 +97,7 @@ Run these queries to populate the briefing sections: - `gbrain query "active deals status"` -- deal pipeline snapshot - `gbrain query "meetings this week"` -- recent meeting pages with insights - `gbrain query "pending commitments follow-ups"` -- open threads and action items -- `gbrain search --type person --sort updated --limit 10` -- people in play +- `gbrain list --type person --sort updated_desc --limit 10` -- people in play ## Output Format diff --git a/skills/cold-start/SKILL.md b/skills/cold-start/SKILL.md index bc87f6b78..b1fa7931b 100644 --- a/skills/cold-start/SKILL.md +++ b/skills/cold-start/SKILL.md @@ -105,18 +105,20 @@ them at request time, enforces policies, and logs everything. **Setup (15 min):** 1. Sign up at [app.clawvisor.com](https://app.clawvisor.com) 2. Create an agent in the dashboard, copy the agent token -3. Set environment variables: +3. Set environment variables (in the host agent's environment — shell profile + or harness config; gbrain itself has no ClawVisor config keys, these are + consumed by the host's ClawVisor integration): ```bash - gbrain config set clawvisor_url "https://app.clawvisor.com" - gbrain config set clawvisor_agent_token "" + export CLAWVISOR_URL="https://app.clawvisor.com" + export CLAWVISOR_AGENT_TOKEN="" ``` 4. Activate Google services (Gmail, Calendar, Contacts) in the dashboard 5. Create a standing task with expansive scope: > "Full brain bootstrapping: read emails, calendar events, and contacts to > populate knowledge base. List, read, and search across all connected accounts." -6. Save the standing task ID: +6. Save the standing task ID the same way: ```bash - gbrain config set clawvisor_task_id "" + export CLAWVISOR_TASK_ID="" ``` **Critical scoping rule:** Be expansive in task purposes. "Email triage" gets @@ -168,8 +170,10 @@ done ### Import ```bash -# For Obsidian vaults, use the migrate skill for proper wikilink handling -gbrain migrate --from obsidian --path /path/to/vault +# Obsidian vaults are markdown directories — import directly, then wire wikilinks +# (full flow: skills/migrate/SKILL.md) +gbrain import /path/to/vault --no-embed --workers 4 +gbrain extract links --source db # parses [[wikilinks]] natively # For plain markdown directories gbrain import /path/to/dir --no-embed --workers 4 @@ -380,12 +384,11 @@ Delegate to the `archive-crawler` skill. It handles: - Text extraction from PDFs, images (OCR), documents - Entity extraction and brain page creation -> **Safety gate:** Archive crawling can be slow and create many pages. Always start -> with a scan-only pass: -> ```bash -> gbrain archive-crawler --scan-only --path /path/to/archive -> ``` -> Show the user the manifest before proceeding with full ingestion. +> **Safety gate:** Archive crawling can be slow and create many pages. +> archive-crawler is a skill, not a CLI command — it refuses to run without an +> explicit `archive-crawler.scan_paths:` allow-list in `gbrain.yml`. Add the +> archive path to the allow-list, run the skill's scan pass first, and show the +> user the manifest before proceeding with full ingestion. **Supported sources:** - Local directories (Dropbox sync folder, Google Drive, old hard drives) diff --git a/skills/conventions/salience-and-recency.md b/skills/conventions/salience-and-recency.md index aa451a610..295a0bae1 100644 --- a/skills/conventions/salience-and-recency.md +++ b/skills/conventions/salience-and-recency.md @@ -125,7 +125,7 @@ no boost. ## See also -- `docs/recency.md` — full reference +- `src/core/search/recency-decay.ts` — the decay implementation (config + env resolution) - `gbrain query --explain` — see resolved values + factor contributions - `get_recent_salience` op gains `recency_bias: 'flat' | 'on'` — opt into per-prefix decay on the dedicated salience query diff --git a/skills/conventions/search-modes.md b/skills/conventions/search-modes.md index cc5714056..5c0b4b8c2 100644 --- a/skills/conventions/search-modes.md +++ b/skills/conventions/search-modes.md @@ -26,6 +26,8 @@ Any agent doing search-adjacent work in a gbrain brain consults this convention: The 3 bundles live in `src/core/search/mode.ts` as `MODE_BUNDLES` (frozen). Don't redefine them per-install; that breaks the public methodology numbers. +The canonical knob table (with cost anchors) lives in +`docs/guides/search-modes.md` — update that first if the bundles change. | Knob | `conservative` | `balanced` | `tokenmax` | |-------------------------------|----------------|------------|----------------| @@ -35,6 +37,7 @@ Don't redefine them per-install; that breaks the public methodology numbers. | `intentWeighting` | true | true | true | | `tokenBudget` | **4000** | **12000** | **off** | | `expansion` (LLM multi-query) | false | false | **true** | +| `relationalRetrieval` | false | **true** | **true** | | `searchLimit` default | 10 | 25 | 50 | **Cache, intent weighting, and similarity threshold are constant across modes** diff --git a/skills/conventions/subagent-routing.md b/skills/conventions/subagent-routing.md index 05adc996d..66ac2fa2e 100644 --- a/skills/conventions/subagent-routing.md +++ b/skills/conventions/subagent-routing.md @@ -51,7 +51,7 @@ When ≥1 signal fires, pause and offer the switch: > agents. Want me to flip this task to Minions? (~10s, no extra setup.)" If the user says yes, submit the task as a Minion job with the same prompt. -Optionally propose flipping the default: `gbrain config set minion_mode always`. +Optionally propose flipping the default to `always` (see "Flipping modes" below). ### Mode C: `off` @@ -85,13 +85,18 @@ Before submitting batch jobs: ## Flipping modes -The user can change their mind at any time: +The user can change their mind at any time. `minion_mode` lives in +`~/.gbrain/preferences.json` (NOT DB config — `gbrain config set minion_mode` +is rejected as an unknown key). Edit the file directly: -```bash -gbrain config set minion_mode always # switch to always-on -gbrain config set minion_mode pain_triggered # back to default -gbrain config set minion_mode off # disable suggestions +```json +{ "minion_mode": "always" } ``` -Or edit `~/.gbrain/preferences.json` directly. The convention reads the file -on every decision, so changes take effect next tool call. +Valid values: `always` | `pain_triggered` | `off`. Keep any other keys the +file already has. `gbrain apply-migrations --mode ` +also writes it without prompting. The convention reads the file on every +decision, so changes take effect next tool call. + +`skills/conventions/cron-via-minions.md` documents the same key for +cron-scheduled work; both files use the preferences.json mechanism. diff --git a/skills/frontmatter-guard/SKILL.md b/skills/frontmatter-guard/SKILL.md index 6c0744945..f9cb0f8e2 100644 --- a/skills/frontmatter-guard/SKILL.md +++ b/skills/frontmatter-guard/SKILL.md @@ -124,7 +124,7 @@ When the user says any of these, route here: - `gbrain doctor` — the `frontmatter_integrity` subcheck reports the same counts as `audit`. - `skills/maintain/SKILL.md` — broader brain health audit; chain after this skill if other classes of issue are suspected. -- `skills/lint/SKILL.md` (via `gbrain lint`) — overlapping rules for skill-file lint; the `frontmatter-*` rule names in lint output come from this skill's validation surface. +- `gbrain lint` — overlapping rules for skill-file lint (a CLI command, not a skill); the `frontmatter-*` rule names in lint output come from this skill's validation surface. ## Output Format diff --git a/skills/schema-author/SKILL.md b/skills/schema-author/SKILL.md index 67de503af..f31d4b4a9 100644 --- a/skills/schema-author/SKILL.md +++ b/skills/schema-author/SKILL.md @@ -69,9 +69,8 @@ flags). For these adjacent jobs, route elsewhere: already has a schema-check phase. Don't duplicate. - **Just looking up a type's settings** → `gbrain schema explain ` directly. This skill is for CHANGING the pack, not READING from it. -- **Querying who knows about X** → `skills/expert-routing/SKILL.md` (or - `gbrain whoknows` directly). schema-author makes a type expert-routable; - it does not run the query. +- **Querying who knows about X** → `gbrain whoknows ` directly. + schema-author makes a type expert-routable; it does not run the query. ## Convention @@ -120,7 +119,7 @@ declared prefixes with zero matching pages — probable mis-declarations). If coverage < 90%, there's untyped content worth typing. ``` -gbrain schema review-orphans --limit 50 --json +gbrain schema review-orphans --json ``` Untyped pages drilldown. Look for shared path prefixes (e.g. "12 of these diff --git a/skills/schema-unify/SKILL.md b/skills/schema-unify/SKILL.md index aa1b65529..b622ac952 100644 --- a/skills/schema-unify/SKILL.md +++ b/skills/schema-unify/SKILL.md @@ -7,7 +7,7 @@ tools: - gbrain onboard --check --explain - gbrain onboard --check --json - gbrain jobs submit unify-types - - gbrain jobs follow + - gbrain jobs get - gbrain schema active - gbrain schema use - gbrain schema stats @@ -100,7 +100,8 @@ nothing and left the active pack unflipped. Omit it to preview. Watch progress per phase: ```bash -gbrain jobs follow +gbrain jobs get # one job: status, progress, result +gbrain jobs watch --follow # live dashboard of the whole queue ``` On a 186K-page brain expect ~10 minutes. The handler runs: @@ -245,7 +246,7 @@ Final celebration summary to stderr: ═══════════════════════════════════════════════════════════ ``` -JSON output (`gbrain jobs follow --json`) returns the structured `UnifyTypesResult` shape with `per_phase`, `pack_identity_after`, `active_pack_flipped`. +For structured JSON, `gbrain call get_job '{"id": }'` returns the job row; its `result` field carries the `UnifyTypesResult` shape with `per_phase`, `pack_identity_after`, `active_pack_flipped` (`gbrain jobs get ` prints the same result inline). ## Reference diff --git a/skills/setup/SKILL.md b/skills/setup/SKILL.md index d88f2e532..bdfb8eb68 100644 --- a/skills/setup/SKILL.md +++ b/skills/setup/SKILL.md @@ -17,6 +17,12 @@ mutating: true Set up GBrain from scratch. Target: working brain in under 5 minutes. +> **Installing into an agent harness?** (Claude Code, Codex, OpenClaw, etc.) +> `gbrain bootstrap` is the paste-in install path — it wires hooks, the +> maintenance sweep, and harness config in one command. See +> `docs/guides/bootstrap.md`. This skill covers the brain-side setup +> (database, sync, first import); the two are complementary. + ## Contract - Setup completes with a working brain verified by `gbrain doctor --json` (all checks OK). @@ -256,14 +262,24 @@ echo "=== Discovery Complete ===" > "You have N binary files (X GB) in your brain repo. Want to move them to cloud > storage? Your git repo will drop from X GB to Y MB. All links keep working." - If the user agrees, configure storage and run migration: - ```bash - # Configure storage backend (Supabase Storage recommended) - gbrain config set storage.backend supabase - gbrain config set storage.bucket brain-files - gbrain config set storage.projectUrl - gbrain config set storage.serviceRoleKey + If the user agrees, configure storage and run migration. The storage backend + is a **file-plane** config object — `gbrain config set` writes the DB plane, + which the files commands never read. Add a `storage` object to + `~/.gbrain/config.json` directly (shape matches `StorageConfig` in + `src/core/storage.ts`; Supabase Storage recommended): + ```json + { + "storage": { + "backend": "supabase", + "bucket": "brain-files", + "projectUrl": "https://.supabase.co", + "serviceRoleKey": "" + } + } + ``` + Then run the migration: + ```bash # Migrate binary files to cloud (3-step lifecycle) gbrain files mirror # Upload to cloud, keep local gbrain files redirect # Replace local with .redirect.yaml pointers diff --git a/skills/skillpack-check/SKILL.md b/skills/skillpack-check/SKILL.md index 612586cb2..2d1f0d05c 100644 --- a/skills/skillpack-check/SKILL.md +++ b/skills/skillpack-check/SKILL.md @@ -69,7 +69,7 @@ Common `actions[]` entries and what they mean: non-builtin cron handlers that need plugin registration — follow `skills/migrations/v0.11.0.md`. - `gbrain embed --stale` — Embeddings are stale. -- `gbrain check-backlinks --fix` — Dead links or missing back-links. +- `gbrain check-backlinks fix` — Dead links or missing back-links. - Free-text action (no `Run:` prefix in the source message) — agent judgment needed. Quote it in the report for the user. diff --git a/skills/skillpack-harvest/SKILL.md b/skills/skillpack-harvest/SKILL.md index bd6f2dfbb..6212398ab 100644 --- a/skills/skillpack-harvest/SKILL.md +++ b/skills/skillpack-harvest/SKILL.md @@ -91,8 +91,8 @@ a list of files written. JSON mode (`--json`) returns the full ## When to invoke -- The user developed a skill in their host fork (Wintermute, Neuromancer, - Zion, etc.) and wants other gbrain clients to be able to use it +- The user developed a skill in their host fork (your OpenClaw or any + private downstream fork) and wants other gbrain clients to be able to use it - A skill has proven itself in production and is ready to generalize - The user explicitly asks to "harvest" or "publish" a skill upstream @@ -123,7 +123,7 @@ Ask the user: - What slug should the harvested skill have? (Slugs must be kebab-case, globally unique in the gbrain bundle.) - Which host repo is the source? (Path to repo root, not to the skill - directory — e.g. `~/git/wintermute`, not `~/git/wintermute/skills/foo`.) + directory — e.g. `~/git/agent-fork`, not `~/git/agent-fork/skills/foo`.) - Should paired source files come along? (Check the host SKILL.md's frontmatter `sources:` array.) @@ -153,8 +153,8 @@ files and apply this checklist. If anything matches, edit the host file FIRST, then run harvest. 1. **Fork-specific names → generic phrasing** - - `Wintermute` → `your OpenClaw` (or `OpenClaw deployment`) - - `Neuromancer`, `Zion`, `` → same treatment + - `` → `your OpenClaw` (or `OpenClaw deployment`); + any pet name for the user's private agent gets the same treatment - Personal first names (`garry`, `jane`, etc.) → `the user` / `you` / a generic placeholder @@ -197,7 +197,8 @@ gbrain skillpack harvest --from Default behavior: - Path-confinement + symlink rejection at file copy - Privacy linter runs against `~/.gbrain/harvest-private-patterns.txt` - (plus built-in defaults: `\bWintermute\b`, email, Slack channels) + (plus built-in defaults — the canonical private fork name, email + addresses, Slack channels; see `src/core/skillpack/harvest-lint.ts`) - On any match → rollback (delete the harvested files) + exit non-zero - `openclaw.plugin.json` updated to add the slug, sorted @@ -240,7 +241,7 @@ gbrain skillpack harvest --from --no-lint ``` **Document the bypass in the commit message.** Future maintainers -should be able to see WHY the lint was bypassed (e.g. "Wintermute +should be able to see WHY the lint was bypassed (e.g. "the fork name appears in a citation, not a real reference — verified manually"). Never bypass the linter on a casual basis. The whole point of the diff --git a/skills/skills.lock.json b/skills/skills.lock.json index 18ff8f08e..5ffe05b48 100644 --- a/skills/skills.lock.json +++ b/skills/skills.lock.json @@ -1,6 +1,6 @@ { - "RESOLVER.md": "ad9f85d0f953ad0ec80c70091cb31c7227a89e4a08c924454be1d61d4df5b65c", - "_AGENT_README.md": "3dd82df125ceb87bdbc1ad16be4306fb3ca6e3a491c70122f6c7724a862f0b21", + "RESOLVER.md": "791d5c90a7f9ae1aaacfe45a975c6855027c82600c95f6b2526b7cd0f0802bb5", + "_AGENT_README.md": "d9b3bca4e427b707bd3b6058b74e09feeb869b732ac363b5580fc3012a87ec67", "_brain-filing-rules.json": "cf850df6a7425464c6d63b3ace71991cc93497fa0cc8cd21acd31883e17939c6", "_brain-filing-rules.md": "2d2d75b7c76081c56f41b2c0a5a978c355ce957300f9b0a5575dc4079ef1f877", "_friction-protocol.md": "1b6e7cfa58725a6a5dc2dc787242141bc33f5fde524540d85b14ec22266140f7", @@ -19,11 +19,11 @@ "brain-pdf/routing-eval.jsonl": "119e4fa113ea45783cee4499e63a729fdeecb4d9a45d47497754b4f5b21d0734", "brain-taxonomist/SKILL.md": "dea4557b540868ec2c56bf43ee7f63c5d03a22d4047cd0dfbeaf19adef334f60", "brain-taxonomist/routing-eval.jsonl": "8b485b3d735aace60be703854f0f2e9d97c52d52564efdaf7334a0c39e8d20ae", - "briefing/SKILL.md": "a661804c3eb2ce5bd4eb6f3e7106283046913945d9dba0b967bdc8483a58dd66", + "briefing/SKILL.md": "bbab7deb7bd59056bd98b3c2fac543efd6cdfe3d68cafdc02e24e5f8d3e245b5", "capture/SKILL.md": "98568ac96331f57397ea072749641d9748b1ce31e8b09d512b8db25c8fcda65f", "citation-fixer/SKILL.md": "abdadbf0740a529b9c4f86f05bba416417624503fdcbc6054402d5546afd08b4", "citation-fixer/routing-eval.jsonl": "52b23b71e66fdc18aee67d0576099b0c83997d648cf4ecf8fe7753b91b6c9c53", - "cold-start/SKILL.md": "a2c42dd7c4eceb7d3ce6449a414b417d55195aa445e723e6c798d90906cbf4e6", + "cold-start/SKILL.md": "33dfc46ad186322823f84efa58589daf0aaea8fffe03da77984e6f0c8494e334", "concept-synthesis/SKILL.md": "2bc060ae6d706c4e8e7d784cbe3e577b85e211a21c68cba094a1754d3f34436b", "concept-synthesis/routing-eval.jsonl": "51d1da894158503ce18b892a34edd203f40732e79ac1c0e85141fd37e0b9922f", "conventions/brain-first.md": "29d020470d0168f8f0b29dde0350a485a9b0472f7ac9962e34948f4897455590", @@ -33,10 +33,10 @@ "conventions/cross-modal.yaml": "c012c3d72614a87b1ee698173dce2a0fb0d057a54df7aab87993c4b07fff6280", "conventions/model-routing.md": "fb7ae8746a578500d6789b68ff40049037aa4d337b65b42f7c1745ae7080c2db", "conventions/quality.md": "8aa681001114689d34268ccadaf0e2ff07b8f68aa5987c093a8c4a7a744f12a6", - "conventions/salience-and-recency.md": "62b0b303bf48bef10adcf08d3b51f23d3b1f1187b3850476a4b5532c2ee88926", + "conventions/salience-and-recency.md": "0ee7f0216f36a5e321673740d148cb0f1113268df19301bb2473a48a728a500b", "conventions/schema-evolution.md": "b5cdd5a17e43b4f2d0546ebf8c421cc0378cbcf74cb2fe97199e115ccb05f9c2", - "conventions/search-modes.md": "2a920225d1c95ea978fb1c77c5170f6a598377ab86fc0024b70962d5a84d54d0", - "conventions/subagent-routing.md": "59afd362ff0cbaf3a63586e97c53f68feb837a258a2bd43f9177af4e57d2b20e", + "conventions/search-modes.md": "2862364874115d75816ba5d0e10f9073043a5fdaa44e9c7f530afc6c492e4a24", + "conventions/subagent-routing.md": "8b8830b815a9a8581a12b489f966c0b0a39eb9b5f66e905a691a03653eef348d", "conventions/test-before-bulk.md": "5073e5b93d570445f72f3c10ed3e6ec10c1fee7574ad73c2e2695993c850eca5", "cron-scheduler/SKILL.md": "e3f9745c4f8e2dacba5b055f408b1ffe541b450aee6bbfe4773f7b424d90e04c", "cross-modal-review/SKILL.md": "685233b1afd477e96697562c502233eea22eda5db8df116b81dd2fa78f01f80c", @@ -46,7 +46,7 @@ "eiirp/SKILL.md": "9d42d6a5f61bba30cb47400db58927c501cd89e5cc3241b5218d7962d1a68804", "eiirp/routing-eval.jsonl": "416459ff68da2e5f5eb216a0c24368c8eebdf4edbbc540a28fe730ceeb4c9700", "enrich/SKILL.md": "9988168348f6c3391d3aeec6621f9c99c8d75d1e1bfdd6c44d48e4c1067ab775", - "frontmatter-guard/SKILL.md": "5142ab53f5428ebc084ded78f1fb7eb4bfa386d276ee034bf4d57273256570c6", + "frontmatter-guard/SKILL.md": "a0112906c2dd7f021faa913da864fa11a25fc416962d992347a3b9c3236dbd60", "frontmatter-guard/routing-eval.jsonl": "243c28b04b557bac5360318c0f47bb6f1dd56d2f720aad3763b93fc0b2ea081e", "functional-area-resolver/SKILL.md": "52df04bc4f8e678f931c3b2078b2126524e6d2d72676ad46b6b710d13271b46c", "functional-area-resolver/routing-eval.jsonl": "f80674d915acdfe229046737a5b171da834be15ac6524b5a3fd18048e9b37028", @@ -107,17 +107,17 @@ "query/routing-eval.jsonl": "74f5a91e52fabc54e0e9403fa17db87ee26bb7ebb8ae8005148c51142abc62fe", "repo-architecture/SKILL.md": "4ec2b8f45d168aaa55f17ecd1ed404ab04217a75c2317f0710c71705846f5394", "reports/SKILL.md": "5dc190a0c3a2ee518254e8b596418dbe19ff389ea5ec8c8d30fcb0dfef4d0ed5", - "schema-author/SKILL.md": "4da9a472c966f8e4fb43d97a3a3608c67ec3d43da6b8c626ee0c30a4e70da26f", - "schema-unify/SKILL.md": "e1d50a54a6ff29434d38a841a572e815d236a8167d459609ae697e548770f500", - "setup/SKILL.md": "4a47a5f6ee99ac8a2649a304abb261fd34810cf6185fd7d6a53e1b6c4cbc0764", + "schema-author/SKILL.md": "09d69ee45970191bb2592a764685f67196cabf350db40b4c0c5ffc19ea9e2df3", + "schema-unify/SKILL.md": "14ddc0f8bc7d8b11eb03dc4eb35621d140fba71bd35779835d85e297acfb0177", + "setup/SKILL.md": "68dde0de48bb4c93b13d9e19c3a20ff82155985ad09f509669aff8ece0a6f4fb", "signal-detector/SKILL.md": "64e4547f5a8624c53d875001b423d240ec73ee9fd026a96c7b799d287c5fb6e4", "skill-creator/SKILL.md": "4a11f8935d4214b21b4664a5c0c03149733731020ec0d5d09dc9fd8c40bd92f6", "skill-optimizer/SKILL.md": "ba3028c7351dec3e644a7114e08e59cae7c60dc367dc4fd10560265a162baa90", "skill-optimizer/routing-eval.jsonl": "48f7fc04414b194ee8674577c3e58e03ebd7f74d836766bd16cb5abfd4effb76", "skill-optimizer/skillopt-benchmark.jsonl": "5552457d6eaa32486b79796d12fcbe2c078b0c53d7fbdaf582f0fd1a17e9a381", "skillify/SKILL.md": "a154e43409458136e1e1cea084aa973e4cf46cd8450c6be8e9891a4a3ddf6146", - "skillpack-check/SKILL.md": "3f347ec8b498530a662be212d05f4cd06b205bce5795c2231b6a4cecef149ea0", - "skillpack-harvest/SKILL.md": "3c4c591b33f03a5ccf11ca0ddde56b54fba541efef0d590b6d435687733182d7", + "skillpack-check/SKILL.md": "cb045e20a60b22774578e86f57872181a3cda73e5861add7e5750dc4de65c1fb", + "skillpack-harvest/SKILL.md": "775133021077f49e6b9da0cf1eafa8df87a9f311b946063db572273effb6dd15", "skillpack-harvest/routing-eval.jsonl": "cb4783288e95af3132b32ecb40a54cffc095f57b36240f96a25c2d2adf5e68c6", "smoke-test/SKILL.md": "f2f2172d41e63e288095451132a0c56848ccc34101d265b330b6cf00e5769f5d", "soul-audit/SKILL.md": "7f162dddcc511e97a24db3a46136295fcbda76023019ad8994546744d7b8eb0a",