Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 1134c27b0f fix(pglite): embed WASM + extension assets so the compiled binary can serve
A `bun build --compile` gbrain binary could not `serve` a PGLite brain: the
compile bundles JS but not PGLite's runtime payload (pglite.wasm, initdb.wasm,
pglite.data, vector/pg_trgm tarballs), so `serve` on PGLite died with a
bunfs/ENOENT. Now the assets ride inside the binary.

- src/core/pglite-embedded-assets.ts: embeds the five assets via
  `import … with { type: 'file' }` (the ENG-6 idiom) and exposes
  getEmbeddedPgliteOptions() → { pgliteWasmModule, initdbWasmModule, fsBundle,
  extensions:{vector,pg_trgm} }. WASM/fsBundle are consumed as bytes; the two
  extension tarballs are materialized to a content-addressed temp file (atomic,
  size-verified reuse) because PGLite reads them via fs.createReadStream, which
  cannot read a /$bunfs path. Unconditional (works in bun-run and compiled),
  so no fragile mode branch.
- src/core/pglite-engine.ts: static-import getEmbeddedPgliteOptions (engine
  path stays static per the engine-dynamic-import invariant); spread into both
  PGlite.create sites (initial + WAL-repair retry). The bunfs classifier stays
  as a backstop but no longer fires for a correct binary.
- scripts/check-pglite-embedded.sh (+ smoketest): compiles a focused binary and
  asserts it boots PGLite, CREATE EXTENSION vector/pg_trgm, and round-trips a
  page — wired into `bun run verify` (now 37 checks), check:all, and
  check:pglite-embedded. Fail-soft only when compile is unavailable.
- agent-harness.ts probeCompiledPglite now passes → the real-agent e2e uses the
  fast compiled MCP server. TODOS: the P2 "can't serve PGLite" item is closed.

Verified: fresh compiled binary ran `search`/`query` against a PGLite brain and
returned the seeded row (no bunfs/ENOENT); verify 37/37; pglite-engine 120/0
source-mode; typecheck clean; engine-dynamic-import + parity guards pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 15:52:19 -07:00
Garry TanandClaude Fable 5 ce781d3b95 fix(ci): real-agent-e2e job — bash array + --timeout (actionlint SC2086 + bun-test-timeout guard)
The real-agent-e2e job's file loop used an unquoted $FILES (SC2086) and ran
`bun test` without --timeout (check-bun-test-timeout guard). Switch to a bash
array and add --timeout=600000 (real-agent turns are slow; the tests self-skip
without authed binaries so it's a no-op elsewhere).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 15:34:03 -07:00
Garry TanandClaude Fable 5 2f6b817a28 test(bootstrap): real-agent e2e — drive the actual claude + codex binaries end to end
Closes the audit's biggest gap ("no real harness ever drives a turn"). Adapts
gstack's PTY/headless agent harness to prove the bootstrap install + smoke work
against the REAL binaries, not PATH shims. Test/CI/docs only — zero src changes.

- test/helpers/agent-harness.ts: hermetic clean-room child env (ported from
  gstack; drops CONDUCTOR_/CLAUDE_/GSTACK_/MCP_/GBRAIN_, promotes
  GSTACK_ANTHROPIC_API_KEY→ANTHROPIC_API_KEY), real-binary resolvers + auth
  probes, headless `claude -p --output-format stream-json` and `codex exec
  --json` turn runners, a gbrain stdio MCP-config writer, and a keyless brain
  seeder. + a fixture-parse unit test (no binary needed).
- test/e2e/bootstrap-real-claude.serial.test.ts: real `gbrain bootstrap`
  install → REAL `claude mcp add` (verified via `claude mcp get`) → verify
  exit 0 → a real `claude -p --mcp-config --strict-mcp-config` turn that
  invokes mcp__gbrain__search and answers from the brain (proven: toolCalls
  include mcp__gbrain__search, final text carries the seeded fact).
- test/e2e/bootstrap-real-codex.serial.test.ts: same install with REAL `codex
  mcp add` into a real ~/.codex/config.toml + Gate-3 pull-protocol assertion,
  then a real `codex exec --json` turn surfacing the fact (MCP or the pull-
  protocol shell path). Bounded retry absorbs codex's occasional MCP-call
  cancellation without softening the fact-requiring assertion.
- Everything hermetic (temp HOME/CLAUDE_CONFIG_DIR/CODEX_HOME/GBRAIN_HOME;
  real ~/.codex auth copied read-only) and skipIf-gated so it self-skips
  cleanly where the binaries/auth are absent.
- heavy-tests.yml: gated `real-agent-e2e` job (nightly/label, never the PR
  shard; no-op on a runner without authed binaries).
- TODOS: compiled `gbrain` binary can't serve a PGLite brain (bun compile
  omits the WASM/extension payloads); harness falls back to `bun run` serve.

Verified against live claude 4.6 + codex 0.147.0: 15 pass / 0 fail; verify
36/36; typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 14:56:52 -07:00
Garry Tan 99e15f0a10 Merge remote-tracking branch 'origin/master' into garrytan/codex-as-agent-default-install
# Conflicts:
#	CHANGELOG.md
#	VERSION
#	docs/architecture/KEY_FILES.md
#	package.json
2026-08-11 12:52:03 -07:00
Garry TanandClaude Fable 5 6fb6790b2a test(bootstrap): real end-to-end coverage — cross-session recall, per-turn content, Codex door, realistic corpus
Closes the seven e2e gaps a coverage audit surfaced: the plumbing was
well-unit-tested but the product claims ("Codex works, context shows up every
turn with real content, it remembers across restarts, machine two recovers,
Postgres works") were unproven end to end. Test-only wave — zero src changes.

- Hermetic synthetic corpus (test/fixtures/bootstrap-corpus/ + a loader helper):
  12 interlinked pages (52 edges, timelines), 12 world/private beliefs, 8 gold
  queries — curated from the gbrain-evals synthetic corpora, 100% placeholder
  names, so recall is asserted on a real multi-entity brain instead of a
  2-node self-planted probe.
- GAP1 magic moment: author a fact via the real write path, disconnect the
  engine, reopen against the same DB, recall it — a real session boundary, not
  verify.ts's same-connection SQL read-back. Plus a source-isolation assertion.
- GAP2 per-turn content: hook-under-serve Pin 1 now seeds a known fact and
  asserts its text lands in the injected block AND private beliefs never do
  (was: empty brain, empty_block accepted as a pass).
- GAP3 Codex door: assert the rendered AGENTS.md carries the Gate-3 brain-first
  pull protocol; make the fake codex shim implement `mcp get` so the [FIX7]
  target-verification can actually fail; the Docker cold-machine harness now
  exercises the hooks/MCP registration step instead of skipping it.
- GAP4 corpus recall: turn-context + verify graph-floor/qrels run on the real
  multi-entity brain with real edges.
- GAP5 attach: machine-two now re-ingests the cloned brain/ into a fresh DB and
  recalls a fact authored only on machine one — the multi-device payoff.
- GAP6 keyed + Postgres (env-gated): real embeddings prove semantic recall a
  paraphrase query can reach but keyless BM25 cannot; bootstrap verify drives a
  real Postgres engine (skipIf DATABASE_URL/keys absent).
- GAP7 persistence: session-end runs the REAL push (not the mocked seam) to a
  local bare remote and the remote receives the content; a planted secret is
  blocked at the gate; the 15-min cron installs and fires a scan-gated push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 11:30:07 -07:00
Garry TanandClaude Fable 5 5183e9f039 feat(bootstrap): default per-turn hooks + search mode ON without a prompt
Installing gbrain for your coding agent IS the consent for the behaviors that
make it work, so stop re-litigating them with install-time questions whose
"no" defeats the product:

- Per-turn hooks (Claude Code) install ON by default — no prompt. Off-ramps:
  `--no-hooks` at install, `GBRAIN_HOOKS=0` at runtime, `bootstrap uninstall`.
  The "hooks installed" line now surfaces the kill switch so default-on is
  never silent. A persisted HOOKS_CONSENT=no (interview --skip) still declines.
- Search mode defaults to `balanced` silently (nobody knows the modes at
  install; `gbrain search modes` changes it any time).
- MCP scope stays the ONE deliberate prompt — project vs user is a real
  cross-repo privacy choice, not friction.

Marks the two consents `silent: true` in the question bank (new QuestionSpec
field), rewrites the runbook phases so the agent no longer asks them, adds the
`--no-hooks` flag (+ registry regen), and adds default-on / opt-out tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 09:48:11 -07:00
Garry TanandClaude Fable 5 f55af9da7b fix(skills): declare 'bootstrap my data' trigger in cold-start frontmatter
The doc reform added 'bootstrap my data' to cold-start's RESOLVER.md row (to
disambiguate data-bootstrap from agent-bootstrap) but not to the skill's own
frontmatter triggers, tripping the RESOLVER↔frontmatter round-trip contract
(resolver.test.ts). Declare it; regenerate skills.lock.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 13:53:42 -07:00
Garry TanandClaude Fable 5 56dd1438e5 fix(ci): silence SC2016 on the intentional askpass literal in release.yml
The one-shot GIT_ASKPASS script must contain literal $1 and
$TEMPLATE_REPO_PAT so they expand when /bin/sh runs it at git's credential
prompt, not when the outer shell writes the file — single quotes are correct.
Add a scoped shellcheck disable so actionlint passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 13:46:37 -07:00
Garry TanandClaude Fable 5 d72047caff docs: sync KEY_FILES + bootstrap plan to security-fix behavior
Cross-referenced the security-fix round against the reference docs and
corrected the drift those commits introduced:

- workspace-push.ts entry: stage-FIRST-then-scan order (the TOCTOU fix),
  fail-closed blocked_unscannable, and the sources-push status -> exit-code map.
- hooks.ts entry: MCP registration pins `serve --surface full`.
- hook.ts entry: session-start/session-end pushes run in a detached child
  (non-blocking); atomic corpus write clears stale sidecars.
- bootstrap.ts entry: render hard-refuses a public origin (template door).
- verify.ts entry: source_id collision resolution (workspace-<path-hash>).
- AGENT_BOOTSTRAP_PLAN as-shipped delta note for the scan/stage reorder.

llms bundle unchanged (KEY_FILES is link-only); build:llms and
test/build-llms.test.ts green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 13:42:14 -07:00
Garry TanandClaude Fable 5 25b798d612 fix: regenerate flag registry for security-fix flags; drop fabricated gbrain capabilities doc ref
CI caught two real failures under the merged state:
- the flag registry lagged the blocked_unscannable/exit-5 flags the security
  round added, tripping the #2185 freshness guard.
- headless-install.md described the keyless capability report as a
  `gbrain capabilities` command, which the #3502 doc-command resolver
  rejects — reworded to prose (the real surface is bootstrap verify's report).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 12:09:19 -07:00
Garry TanandClaude Fable 5 7816981ac0 v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code
Turns a Codex or Claude Code session into a persistent personal agent:
interview-rendered identity files, a local PGLite brain, per-turn context via
serve IPC (Claude Code hooks / Codex pull protocol), session-triggered
persistence, and a private GitHub repo as the agent's portable body. Keyless-
first (the harness model is the LLM; one optional key adds embeddings +
extraction). New `gbrain bootstrap` command family + `gbrain hook` + `gbrain
sweep`; doctor bootstrap health checks; latest-stable distribution ref +
template-repo publish job. Opt-in, additive — existing installs untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 11:04:43 -07:00
Garry TanandClaude Fable 5 2437a60784 fix(security): close adversarial-review findings — scan fails closed, whole-PEM redaction, bound repo push
Cross-model adversarial pass (Claude + Codex) on the bootstrap wave:

- secret scan fails CLOSED: an unreadable, oversized, or binary staged blob
  now blocks the push (blocked_unscannable, exit 5) instead of committing
  unscanned; only a confirmed staged deletion is skipped. This was the
  headline "block secrets before they leave the machine" property failing open.
- private-key redaction spans the whole PEM block (header+body+footer), not
  just the header line — the base64 body no longer survives into the corpus
  the sweep sends to an extraction provider.
- bootstrap repo commits the workspace (secret-scan-gated) before the first
  push and verifies the remote actually received it, so a push-fail retry
  can't adopt an empty remote as success.
- privacy verify is re-bound to origin immediately before push (a concurrent
  origin rewrite between verify and push is refused).
- session-end corpus write is atomic and clears the stale ingested/in-progress
  sidecars so a resumed session's appended transcript is re-ingested.
- public-origin refusal enforced at render (not only status); MCP "already
  registered" is verified to target this workspace, not blessed blindly;
  verify probe cleanup scopes deletes to its own slugs, not a token substring;
  allowlist fingerprint floor raised 8→16 hex.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 11:03:18 -07:00
Garry Tan 004434e886 Merge remote-tracking branch 'origin/master' into garrytan/codex-as-agent-default-install
# Conflicts:
#	docs/architecture/KEY_FILES.md
#	docs/eval-bench.md
#	src/core/cli-flag-registry.generated.ts
2026-08-10 10:28:29 -07:00
Garry TanandClaude Fable 5 7c9e9fd331 fix(bootstrap): origin-ownership gate, world visibility, collision-safe source id, consent + templates
- repo adoption requires an exact receipt repo_url match or authed-owner
  check (undefined repo_url was a wildcard); create verifies privacy BEFORE
  the first push.
- verify sets facts.default_visibility=world if unset, so agent-authored
  facts surface in per-turn context (they defaulted private before).
- source_id derives a path-hash suffix when 'workspace' is taken by another
  checkout; every consumer reads manifest.source_id.
- skipped HOOKS_CONSENT now declines (was falling through to default yes);
  --minimal refuses on an initialized manifest; tilde fences escaped.
- MCP registration pins --surface full; status hard-fails a public origin
  (template door); receipt writers guard against newer/corrupt receipts;
  uninstall only claims brain-deleted after a real rm.
- templates ship jobs disabled + provider-consent + support-relay lines;
  soul-audit re-runs over the shared interview bank. TODOS: 11 follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 10:27:23 -07:00
Garry TanandClaude Fable 5 aeb14df35c fix(hook,sweep,ipc): non-blocking hook pushes, bounded sweep + cache, source-bound resolve
- session-start/session-end no longer run synchronous git + inline push
  inside their self-deadline; a detached child does the push and the hook
  returns immediately (blocked Claude Code startup for minutes on a dirty
  tree before).
- serve sweep drops the unbounded listAllPageRefs, resolves only candidate
  targets, claims corpus files atomically (no double-LLM-spend race), and
  caps the fence LIKE scan; heartbeat writes are O_APPEND with rare compaction.
- hot-memory cache evicts expired entries and bounds entry count (the key is
  caller-controlled via _meta.session_id).
- v1 resolve IPC honors boundSourceId like turn_context; turn-context runs
  its arms concurrently. doctor reads push/heartbeat thresholds from hook.ts.
- new tests: doctor bootstrap checks, hook push-gate + deadline, concurrent
  sweep claims, cache eviction, bound-source resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 10:27:23 -07:00
Garry TanandClaude Fable 5 3cf666e288 fix(ci): regenerate flag registry for new commands, harden shard classifier + release token
- cli-flag-registry.generated.ts regenerated: bootstrap/hook/sweep and
  sources push --message/--allow-unverified-remote were missing, so the
  strict #2185 validator rejected real invocations and skipped the new
  commands entirely.
- EXIT-HANG shard classifier now requires every assigned file to have
  started before warn-passing a watchdog kill (was fail-open).
- release.yml passes TEMPLATE_REPO_PAT via http.extraheader, off the argv.
- compiled-binary e2e fails loud in CI instead of a silent permanent skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 10:27:03 -07:00
Garry TanandClaude Fable 5 eacc98b6a3 fix(scan,push): modern OpenAI/Voyage key patterns, scan staged blobs not disk
- secret-scan matches sk-proj-/sk-svcacct-/sk-None- and pa- Voyage keys
  (the bare sk- pattern missed every current OpenAI key format).
- workspacePush stages first, then scans the staged index blobs via
  git cat-file, closing the scan-then-stage TOCTOU where a file changed
  between snapshot and commit shipped unscanned.
- shared binary-sniff helper, memoized glob regexes, atomic push-status
  write, and tests for pull_conflict + gitignored deny-match paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 10:27:02 -07:00
Garry Tan c1d7a61a6d Merge remote-tracking branch 'origin/master' into garrytan/codex-as-agent-default-install
# Conflicts:
#	CLAUDE.md
#	CONTRIBUTING.md
#	README.md
#	docs/INSTALL.md
#	docs/TESTING.md
#	docs/architecture/KEY_FILES.md
#	docs/architecture/thin-client.md
#	docs/guides/search-modes.md
#	docs/mcp/CLAUDE_CODE.md
#	docs/mcp/DEPLOY.md
#	llms-full.txt
#	scripts/run-unit-parallel.sh
#	src/cli.ts
2026-08-09 17:50:53 -07:00
Garry TanandClaude Fable 5 989794701e docs(readme): refresh production-brain stats to current brain-repo counts
155,795 pages / 24,589 people / 5,340 companies, counted from the brain
repo's current HEAD; the "100K-page brain" framing moves to 150K to match.
Cron-fleet count unchanged (its store lives on the deployment host, not in
the repos available for verification).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 17:29:54 -07:00
Garry TanandClaude Fable 5 f94b9ef316 docs: repo-wide accuracy + MECE reform from the 9-bucket markdown audit
A code-grounded audit of every markdown file (root, architecture, guides,
mcp, tutorials, docs-root, operations/eval/designs, skills, recipes) followed
by a fix wave with per-bucket ownership. Four classes of change:

Accuracy — every documented command/flag verified against src/ before writing:
dead commands replaced with working ones (pages purge-deleted, jobs watch
--follow, gbrain restore, import-based Obsidian flow, space-separated --scopes,
real thin-client recipes, working isolation verification, real supervisor
restart procedure, curl-based ngrok health check, real minion_mode setter);
count drift fixed with rot-proof phrasing (100+ ops, 50+ bundled skills via
skills/manifest.json, 140+ engine methods, KNOBS_HASH_VERSION pointer instead
of hardcoded versions); stale claims corrected (search-mode defaults, RETRIEVAL
pipeline order incl. autocut, sentinel rules, refusal-list mechanism, engine
snapshot, shard cap 2400s + EXIT-HANG classifier in TESTING.md, latest-stable +
publish-template documented in RELEASING.md as release.yml promises).

MECE — one home per concept, pointers elsewhere: test isolation → TESTING.md;
OAuth registration + --bind/--public-url lore → DEPLOY.md; mode bundles →
guides/search-modes.md (the home the CLAUDE.md dispatcher always promised);
merge contract → schema-packs.md; WAL ladder → ENGINES.md; quiet-hours →
quiet-hours.md; capture taxonomy → entity-detection.md; person-page taxonomy →
compiled-truth.md; brain-first protocol → brain-first-lookup.md; refresh
semantics → refresh-algorithm.md; KEY_FILES.md deduplicated (58 extension
entries merged, one entry per file); infra-layer.md rewritten as a pointer page.

Privacy — placeholder sweep across guides, docs, skills, and recipes per the
iron rule; per-release narration stripped from reference docs (current-state
prose only).

Bootstrap coverage — AGENTS.md pointer, RESOLVER routing row, INSTALL.md path,
tutorial cross-links, keyless-mode sections in spend-controls/headless-install.

skills.lock.json regenerated; llms.txt/llms-full.txt rebuilt. Gates: verify
36/36, typecheck clean, doctor 96/96, skills-integrity + resolver + build-llms
+ config-set + migrations all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 17:22:42 -07:00
Garry TanandClaude Fable 5 64e70c53a0 fix: doc-audit code chasers — broken recovery hints, stale op description, auto_link key
Four small code fixes surfaced by the markdown accuracy audit:
- doctor's auto-RLS recovery hint pointed at `apply-migrations --force-retry 35`,
  which cannot work (--force-retry targets the vX.Y.Z orchestrator registry, not
  the numeric schema MIGRATIONS array). Hint now points at the recreate SQL in
  docs/guides/rls-and-you.md; test pins against regression.
- v0_11_0 migration printed the same broken-mechanism class of hint
  (`config set minion_mode` writes DB config nothing reads); now names
  `apply-migrations --mode` + preferences.json, the real setter.
- submit_job's op description hardcoded a stale handler list; now points at
  registerBuiltinHandlers as the source plus the --follow discovery trick.
- `auto_link` added to KNOWN_CONFIG_KEYS: read by link-extraction, reconcile-links,
  and sweep, and documented as the off-switch in brain-ops/maintain, but the
  allowlist rejected `config set auto_link false`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 17:22:24 -07:00
Garry TanandClaude Fable 5 ecc2c378d1 docs(readme): Codex as the recommended first step; OpenClaw/Hermes framed as-intended, high-cost
The install section now routes newcomers explicitly: Codex first
(subscription-priced, nothing to deploy), OpenClaw/Hermes as GBrain used the
way it was designed — always on, at real server + API cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 09:49:54 -07:00
Garry TanandClaude Fable 5 4627c7a047 docs(readme): per-harness install sections — Codex, Claude Code, then OpenClaw/Hermes
Each harness gets its own complete paste-block section (desktop app first,
terminal noted — Claude Code CLI is the identical harness; Codex CLI works
pull-based today); the OpenClaw/Hermes platform path keeps equal weight with
its one-click deploys and INSTALL_FOR_AGENTS block intact; memory-only and
remote-connect tiers consolidated under 'Lighter ways in'. Supersedes the
review's D5 ordering by user direction; stale heading references updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 08:26:08 -07:00
Garry TanandClaude Fable 5 83817561e6 test(bootstrap): quarantine env-mutating suites to the serial lane
check-test-isolation R1: six new files mutate GBRAIN_HOME/env at module scope —
the serial lane (one process per file) is the guard's prescribed home for them.
All 114 tests pass post-rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 01:22:45 -07:00
Garry TanandClaude Fable 5 59ba03afcb fix(test): shard cap 2400s — the count-balanced heavy shard needs it under contention
Observed: the heavy shard still progressing 22s before an 1800s kill while
siblings finish at 1150-1550s (split balances file count, not weight). Filed
the load-sensitive WAL-repair flake (pre-existing, master's v0.42.75.0 wave).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 00:46:49 -07:00
Garry TanandClaude Fable 5 ce350250e2 fix(test): classify post-suite exit-hangs as warn-pass; file the leak forensics
A shard killed by the wedge watchdog with every assigned file started and zero
fail markers did all its work and leaked a handle at exit — pre-existing and
master-reproducible (P1 TODO carries the full bisect forensics). Bun's per-test
timeout turns a hung test into a (fail), so the classifier cannot mask one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 00:11:06 -07:00
Garry TanandClaude Fable 5 12cb232470 test(ci): cache-hash policy — README + runbook edits must invalidate [C2]
The old deny-list assertion predates the paste block; README.md and
BOOTSTRAP_FOR_AGENTS.md are policy-doc re-admissions now, so their edits must
change the hash (a paste-block edit shipping under a cached green was the C2
hole).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 22:25:21 -07:00
Garry TanandClaude Fable 5 bdab11e524 fix(test): shard wedge cap tracks suite growth (1500s -> 1800s)
At ~9000 tests a healthy shard finished at 1466s and two progressing shards
were false-killed at the old cap; 1800s restores ~25% headroom over the
slowest observed healthy shard. Real hangs still hit it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 21:50:43 -07:00
Garry TanandClaude Fable 5 9f28e66bfe fix(bootstrap): register doctor check categories + system-of-record allow comments
The six bootstrap doctor checks join OPS_CHECK_NAMES; the sweep's batch link/
timeline inserts carry the explicit extract-path allow comments (the sweep IS
the extraction path for workspace pages).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 20:49:13 -07:00
Garry TanandClaude Fable 5 5374fc8519 test(bootstrap): e2e pins — hook-under-live-serve, attach, degraded modes, compiled binary, Docker harness
The permanent pins: a real serve holds the PGLite lock while the engine-free
hook completes (and a direct engine open provably throws LiveServeLockError);
stale-socket fail-open; machine-2 attach with marker-keyed hook repair;
decline-everything installs verify green with every degradation named; the
compiled binary renders bundled templates in an empty cwd. Offline Docker
harness (networkless, read-only) gated into heavy-tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:29:30 -07:00
Garry TanandClaude Fable 5 ffb4af81a2 docs: KEY_FILES bootstrap cluster + CLAUDE.md dispatcher row (+ build:llms)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:08:52 -07:00
Garry TanandClaude Fable 5 15d8107b28 feat(bootstrap): dispatcher, verify, status — the command assembled
gbrain bootstrap {status,interview,render,repo,hooks,verify,uninstall,attach}:
engine-free except verify (owns its engine, in-process sweep — no live-serve
conflict); phase list is the TS source of truth with install.jsonl telemetry
and the support blob; verify's fail-soft check suite covers the real write path
(put_page → write-through file → sweep → graph floor → recall), passes keyless,
persists snapshots, and ends with the first-run tour. cli.ts wired per the
three-touchpoint rule; doctor gains the bootstrap check group (silent on
machines with no bootstrap state). 28 new tests; 353 adjacent green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 19:06:10 -07:00
Garry TanandClaude Fable 5 6ab29f3e31 docs(bootstrap): cross-link the full-agent path from the connection docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:29:24 -07:00
Garry TanandClaude Fable 5 3fb54d0d9e feat(hooks): engine-free gbrain hook command, settings writers, transcript parser
Four hook events (session-start digest + crashed-session recovery push,
user-prompt turn-context injection under an 800ms deadline and the 10KB cap,
stop buffers, session-end corpus write with redaction/retention/dedup +
best-effort push); structural JSON settings merger keyed by a _gbrain marker
(foreign hooks and permissions survive); dated host-spec registry; Claude Code
.jsonl parser as a spec-target with a scrubbed 7-shape fixture. Heartbeat is
counters-only by construction. 59 tests; zero engine modules in the import
graph.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:27:24 -07:00
Garry TanandClaude Fable 5 9e92c4b0c5 feat(serve): resident maintenance sweep + keyless capability probe
The lock-owning serve process now closes the persistence loop: startup (3s
post-connect, best-effort, unref'd) and idle (10-min quiet intervals through
the injectable timer seam) sweeps run facts-fence reconciliation, deterministic
link/timeline extraction over recent workspace pages, and spend-gated corpus
ingest (skipped keyless — agent-authored fences cover it). gbrain sweep --once
is the trusted CLI seam bootstrap verify uses. Capability probe renders the
honest keyless/keyed report. Full reuse of the cycle extractor + extract cores;
26 new tests, neighbors green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:26:59 -07:00
Garry TanandClaude Fable 5 503e58a9f5 fix(sources): harden/pull gates accept sources inside a parent git repo
The bootstrap workspace registers brain/ (a subdirectory) as the source; the
durability core already resolves the repo root, so the command gates now check
inside-a-repo rather than .git-right-here [CX2-3].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:07:03 -07:00
Garry TanandClaude Fable 5 0004f280c3 feat(persistence): secret-scan, gbrain sources push, durability unification
Pattern secret scanner (own runtime allowlist, redacted previews, corpus-write
redaction mode); sources push runs the whole scan→stage→commit→pull→push
sequence under one cross-platform lock (mkdir-atomic, pid+age+token) with a
deny-glob backstop, commit-first divergence-safe pull, refuse-unverifiable
visibility, and push-status telemetry; gbrain-home choke point unifies
GBRAIN_HOME semantics with config (0700); durability is parent-repo-aware and
rotates its push log at 0600. 35 new tests; 200 existing green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:05:28 -07:00
Garry TanandClaude Fable 5 3f64a6d467 feat(context): IPC v2 turn_context + 8KB assembly + visibility resolver + session identity
Discriminated-union IPC with handler map, protocol echo (stale-serve detection),
shared-secret gate, server-side source binding, per-kind budgets; turn-context
assembly (reflex pointers + volunteered pages + world-only hot facts) under a
data-not-instructions envelope trimmed to the harness's 10KB hook-output cap;
facts.default_visibility resolved through one helper at all four sites (explicit
caller wins, typos fail closed); typed sessionId threads _meta.session_id into
the hot-memory cache key. 50 new tests; 180 adjacent tests confirmed green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:02:27 -07:00
Garry TanandClaude Fable 5 1129fa062c feat(release): latest-stable ref, template-repo publish job, bootstrap CI guards
release.yml advances the latest-stable tag only after assets publish (the paste
block's permanent ref — copies in the wild never rot) and gains a PAT-gated
publish-template job verified against the vendored tree; two skip-graceful
guards (sanctioned-ref + runbook stamp; template/token bijection + placeholder
assertion + generator byte-diff) wired into verify; README + runbook re-admitted
to the CI cache hash; vendored deterministic template tree generated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 18:00:34 -07:00
Garry TanandClaude Fable 5 75e30015a7 feat(bootstrap): private-repo lifecycle — repo create, attach, uninstall, run lock
gh-gated private repo creation with API-verified privacy (rate-limit distinct
from public), refuse-foreign-origin with attach as the sanctioned path, atomic
bootstrap mutex (pid liveness + age + token), receipt-keyed uninstall that
never wholesale-deletes the gbrain home and only offers --delete-brain for a
brain it created; read-only PGLite lock probe (never opens the engine).
54 unit tests, injectable exec seam throughout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:58:40 -07:00
Garry TanandClaude Fable 5 b66bd8d155 feat(bootstrap): interview + render engines
Interview gate with read-back confirm-hash (any later answer change clears the
confirmation — the hostile single-batch case is structurally impossible),
per-answer provenance, caps + escaping at set time, config-sink routing for the
provider key; renderer with hard-fail token sweep, subordinate fencing of
principal input, never-clobber + backups, deterministic minimal mode for the
template repo, scaled byte floors. 58 unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:55:49 -07:00
Garry TanandClaude Fable 5 59a6ce091a docs(designs): spike instrument for the bootstrap wave (build order 0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:46:21 -07:00
Garry TanandClaude Fable 5 1b66e911f9 feat(bootstrap): runbook, README paste block, bootstrap guide, TODOS entries
BOOTSTRAP_FOR_AGENTS.md (agent-driven install runbook: CLI phase list is the
source of truth, never-invent rules, Codex approvals preflight, keyless posture,
failure-modes table, version stamp for the skew check); README gains the
full-agent paste block pinned to latest-stable inside the Claude Code/Codex
quick start (memory-only tier stays); docs/guides/bootstrap.md carries the full
install/security/consent/degradation/uninstall contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:45:41 -07:00
Garry TanandClaude Fable 5 619d502a04 feat(bootstrap): format spec, question bank, identity templates, bundled assets
agent.json manifest (format_version 1, initialized sentinel) + machine-local
install receipt [CX2-1, CX2-12]; 12-question/6-required interview bank with
consent keys and a persist:false sink for the optional provider key [CX2-13];
ten {{TOKEN}} identity templates (generic, adapted to gbrain ops — gates call
recall/query/put_page, write-through-ops rule, keyless agent-authored facts,
silence contract); assets embedded compiled-binary-safe via file-type imports
[ENG-6].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:39:42 -07:00
Garry TanandClaude Fable 5 75e708b745 docs(designs): agent-bootstrap plan + design docs (normative, review-absorbed)
The scrubbed, in-repo sources of truth for the gbrain bootstrap wave:
AGENT_BOOTSTRAP_DESIGN.md (product scope/sequencing) and
AGENT_BOOTSTRAP_PLAN.md (implementation; all review-finding IDs inlined).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:34:15 -07:00
266 changed files with 29991 additions and 2061 deletions
+57
View File
@@ -66,6 +66,20 @@ jobs:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun run test:heavy
# Agent-bootstrap offline Docker e2e [A7] — placeholder registration.
# The networkless cold-machine container test (interview → render →
# verify with a fake gh) lands with the bootstrap doors task at
# tests/docker/bootstrap-e2e.sh. Gated on file existence so this stays
# a visible no-op until then. It runs at heavy cadence (nightly +
# `heavy-tests` label) by design — never in the PR shard matrix.
- name: Bootstrap offline Docker e2e (placeholder)
run: |
if [ -f tests/docker/bootstrap-e2e.sh ]; then
bash tests/docker/bootstrap-e2e.sh
else
echo "SKIP: tests/docker/bootstrap-e2e.sh not present yet — placeholder until the bootstrap doors task lands."
fi
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
# run. Upload those + the rss workload JSON on failure for triage
# without re-running locally.
@@ -87,3 +101,46 @@ jobs:
path: heavy-artifacts/
retention-days: 14
if-no-files-found: ignore
# Real-agent door e2e: drives the ACTUAL `claude` + `codex` binaries (no PATH
# shims) against a real gbrain over MCP. These pay real API cost and need the
# binaries installed + authed, which a stock GitHub runner does NOT have — so
# both tests self-SKIP (describe.skipIf on binary/auth) and the job is a clean
# no-op here. It exists so a self-hosted / manually-provisioned runner WITH
# authed claude/codex (and ANTHROPIC/OPENAI creds) actually exercises the real
# binaries. Heavy cadence only (nightly + `real-agent-e2e` label + dispatch);
# NEVER the PR shard matrix.
real-agent-e2e:
name: Real-agent door e2e (skips without authed binaries)
if: |
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
# Reference both door tests; run only the ones present (the claude door
# may land in a sibling PR). Missing binary/auth → the file self-skips, so
# a stock runner reports a green no-op rather than failing.
- name: Run real-agent door tests
run: |
files=()
for f in \
test/e2e/bootstrap-real-claude.serial.test.ts \
test/e2e/bootstrap-real-codex.serial.test.ts; do
[ -f "$f" ] && files+=("$f")
done
if [ "${#files[@]}" -eq 0 ]; then
echo "SKIP: no real-agent door test files present yet."
exit 0
fi
echo "Running: ${files[*]}"
# --timeout: real-agent turns are slow (live claude/codex); the door
# tests self-skip without authed binaries so this is a no-op elsewhere.
bun test --timeout=600000 "${files[@]}"
+110
View File
@@ -38,6 +38,7 @@ jobs:
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
template_tree_hash: ${{ steps.v.outputs.template_tree_hash }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: v
@@ -47,6 +48,13 @@ jobs:
run: |
version="$(tr -d '[:space:]' < VERSION)"
echo "version=$version" >> "$GITHUB_OUTPUT"
# [C3] Record the vendored template tree's git hash alongside the
# asset completeness check. The publish-template job embeds it in
# the template repo's commit message, so template-repo HEAD can be
# audited back to the exact reviewed tree in THIS repo. Empty until
# templates/bootstrap/template-repo/ is vendored.
tree_hash="$(git rev-parse "HEAD:templates/bootstrap/template-repo" 2>/dev/null || true)"
echo "template_tree_hash=$tree_hash" >> "$GITHUB_OUTPUT"
# Complete = release exists AND carries every asset the self-updater
# can request. A partial release must NOT short-circuit, so a re-run
# can repair it.
@@ -137,3 +145,105 @@ jobs:
files: |
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
artifacts/gbrain-linux-x64/gbrain-linux-x64
# [C1 = D6-A] `latest-stable` is the single sanctioned distribution ref:
# the README paste block, the BOOTSTRAP_FOR_AGENTS.md fetch URL, and
# `bun install -g github:garrytan/gbrain#latest-stable` all reference it
# permanently, so paste blocks copied into the wild never rot and there
# is no 404 window between VERSION landing and assets publishing. It is
# a maintainer-controlled tag advanced ONLY here — the FINAL step of the
# release job, after binaries + provenance attestation have fully
# published — so a half-built release never moves it. The force-move
# (`+`) is intentional: latest-stable tracks the newest verified release;
# per-release history lives in the immutable v<VERSION> tags.
# scripts/check-bootstrap-tag.sh keeps the entry docs pinned to this ref.
#
# If this step ALONE fails, re-advance by hand (a full re-run would skip:
# the release already exists with all assets):
# git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
- name: Advance latest-stable to this release commit
run: git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"
# [G7/S3#4] Publishes the rendered agent-workspace template repo (the
# GitHub "Use this template" door) from CI ONLY — no human pushes it by
# hand, so what adopters clone is exactly what this repo reviewed. Guarded
# three ways:
# 1. The release above fully published (needs: release + the exists gate).
# 2. The vendored tree templates/bootstrap/template-repo/ exists — the
# generator/doors work may not have landed yet; skip, never fail.
# 3. The TEMPLATE_REPO_PAT secret is configured. Secrets are not readable
# in job-level `if:` expressions, so the secret is bound to env (the
# same env-indirection pattern as the release-notes step) and checked
# by the gate step's shell.
#
# TEMPLATE_REPO_PAT scope: a fine-grained PAT with `contents: write` on the
# template repository ONLY — no other repositories, no other permissions.
# Documented in docs/RELEASING.md.
publish-template:
needs: [version, release]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
permissions:
contents: read
env:
TEMPLATE_REPO_PAT: ${{ secrets.TEMPLATE_REPO_PAT }}
# owner/name of the template repo; override via repository variable.
TEMPLATE_REPO: ${{ vars.TEMPLATE_REPO || 'garrytan/gbrain-agent-template' }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: gate
name: Gate on PAT + vendored template tree
run: |
if [ -z "$TEMPLATE_REPO_PAT" ]; then
echo "publish=false" >> "$GITHUB_OUTPUT"
echo "SKIP: TEMPLATE_REPO_PAT secret not configured — template-repo publishing disabled."
elif [ ! -d templates/bootstrap/template-repo ]; then
echo "publish=false" >> "$GITHUB_OUTPUT"
echo "SKIP: templates/bootstrap/template-repo/ not vendored yet — nothing to publish."
else
echo "publish=true" >> "$GITHUB_OUTPUT"
fi
- if: steps.gate.outputs.publish == 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- if: steps.gate.outputs.publish == 'true'
run: bun install
- if: steps.gate.outputs.publish == 'true'
name: Generate template tree and byte-diff against the vendored copy
run: |
bun run scripts/generate-template-repo.ts --out /tmp/template-tree
# [A1] Publish gate: fresh generator output must equal the vendored
# tree byte-for-byte. A mismatch means the vendored tree is stale —
# regenerate + commit it (scripts/check-bootstrap-templates.sh runs
# this same diff offline in `bun run verify`).
diff -r /tmp/template-tree templates/bootstrap/template-repo
- if: steps.gate.outputs.publish == 'true'
name: Force-push the template repo
env:
RELEASE_VERSION: ${{ needs.version.outputs.version }}
TEMPLATE_TREE_HASH: ${{ needs.version.outputs.template_tree_hash }}
run: |
set -euo pipefail
cd /tmp/template-tree
git init -q -b main
git config user.name "gbrain-release-bot"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add -A
# History-less by design: each release force-publishes one commit
# whose message binds (gbrain version, vendored tree hash) [C3].
git commit -q -m "gbrain v${RELEASE_VERSION} template (tree ${TEMPLATE_TREE_HASH:-unvendored})"
# Out-of-band credential: a one-shot GIT_ASKPASS script reads the
# PAT from env at prompt time, so the token never rides argv (where
# `ps`, runner traces, and error messages echoing the remote URL
# could surface it).
ASKPASS="$(mktemp)"
# shellcheck disable=SC2016 # $1/$TEMPLATE_REPO_PAT are literal on
# purpose — they must expand when /bin/sh runs the askpass script at
# git's credential prompt, not when this outer shell writes the file.
printf '%s\n' '#!/bin/sh' \
'case "$1" in Username*) printf "x-access-token\n" ;; *) printf "%s\n" "$TEMPLATE_REPO_PAT" ;; esac' \
> "$ASKPASS"
chmod +x "$ASKPASS"
GIT_ASKPASS="$ASKPASS" GIT_TERMINAL_PROMPT=0 \
git push --force "https://github.com/${TEMPLATE_REPO}.git" HEAD:main
rm -f "$ASKPASS"
+19 -13
View File
@@ -4,8 +4,16 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
start here.
> **Becoming someone's persistent personal agent** (identity + memory + private repo)?
> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap`
> flow — instead of the plain install below, then come back here for the operating
> protocol. Connecting to an EXISTING remote brain from a laptop agent?
> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP
> table in [`README.md`](./README.md)).
## Install (5 min)
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
1. Install gbrain via Bun (the canonical path):
```bash
curl -fsSL https://bun.sh/install | bash
@@ -26,8 +34,8 @@ start here.
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
for existing users (search modes were added in v0.32.3).
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
(API keys, identity, cron, verification).
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step
flow (API keys, identity, cron, verification).
## Read this order
@@ -69,10 +77,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
and `gbrain eval replay --against base.ndjson`. For public benchmark
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
<dataset.jsonl>` runs against an isolated in-memory PGLite
per question — your `~/.gbrain` is never opened. Full guide:
[`docs/eval-bench.md`](./docs/eval-bench.md).
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
- **Drive the brain to a target health score:** the one-command
loop. `gbrain doctor --remediation-plan --json` previews what would be
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
walks a dependency-ordered plan (sync before extract, embed after
@@ -81,22 +89,20 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
keys hit a `max_reachable_score` ceiling and bail with what's missing.
Three phase handlers (synthesize / patterns / consolidate) are
PROTECTED — only trusted local callers can submit them; MCP cannot.
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
and the CHANGELOG entry for v0.36.4.0.
- **Track a founder/company over time (v0.35.7):** when an entity has
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md).
- **Track a founder/company over time:** when an entity has
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
`unit: USD`, `period: monthly` columns), run
`gbrain eval trajectory <entity-slug>` for the chronological history
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
for a four-signal JSON rollup (claim_accuracy / consistency /
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:**
`gbrain think` now uses this substrate automatically on temporal /
same data — read scope, visibility-filtered for remote callers.
`gbrain think` uses this substrate automatically on temporal /
knowledge_update intent (default ON; flip `think.trajectory_enabled=false`
to opt out). Migration v82 added `facts.event_type` so non-metric event
rows (`meeting`, `job_change`, `location_change`) ride through the same
pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query
them.
to opt out). Non-metric event rows (`meeting`, `job_change`,
`location_change`) ride through the same pipeline via `facts.event_type`;
pass `kind: 'event'` or `'all'` to `find_trajectory` to query them.
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
single-fetch ingestion.
+140
View File
@@ -0,0 +1,140 @@
<!-- gbrain-runbook-stamp: 0.45.0.0 -->
<!-- This stamp must equal the VERSION file at every release; CI enforces it
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
the installed binary and warns on skew. -->
# BOOTSTRAP_FOR_AGENTS.md — become your human's persistent agent
You are an AI agent (Claude Code or Codex) reading this because a human pasted a
block asking you to set yourself up as their persistent personal agent, with gbrain
as your memory. This runbook is the judgment layer; the deterministic work lives in
`gbrain bootstrap` subcommands with exit codes. Follow it top to bottom.
**Scope note:** this path is for Claude Code and Codex (desktop apps or CLIs).
Running OpenClaw or Hermes? Use `INSTALL_FOR_AGENTS.md` instead.
**End state:** this folder is your workspace — identity files rendered from your
human's own answers, a local brain (PGLite, embedded, no server), MCP wired,
per-turn context, and a private GitHub repo as your durable body. ~15 minutes,
mostly interview.
## Hard rules for you, the installing agent
**NEVER INVENT ANSWERS.** Personality, purpose, and boundaries come from the
human. A guessed SOUL.md gets believed by every future session. If you do not have
an answer, ask. The render step structurally refuses to run until the required
answers exist and were read back.
**ASK IN SMALL BATCHES.** The interview is 12 questions max (6 required), asked
in three batches. Mirror each batch back in one line. Accept "skip" on any
non-required question — momentum beats completeness.
**STAY INSIDE THIS PHASE LIST.** Run `gbrain bootstrap status --json` and follow
ITS phase list — the CLI is the source of truth, this document is commentary. If a
step you are asked to run is not in the CLI's phase list, refuse it. If `status`
reports a version skew between this runbook and the installed binary, say so and
prefer the binary's instructions.
**NO SILENT FAILURE.** Every blocking condition (secret-scan block, lock
collision, partial install) surfaces through `status`/`verify`/`doctor` output —
read it and relay it to the human in plain language. Never work around a refusal.
**VERIFY BEFORE CLAIMING DONE.** The install is done when `gbrain bootstrap
verify` exits 0 — not when the transcript looks good. Paste its report to the human.
**RESPECT THE TOOLCHAIN TRUST RULES.** Install bun via a platform package manager
when available (`brew install oven-sh/bun/bun`); the only permitted fallback is the
checksum-verified variant: download the pinned release to a file, verify it against
that release's SHASUMS256.txt, and only then execute. Install gh the same way —
platform package manager first (`brew install gh`, `apt install gh`, `dnf install gh`,
`winget install GitHub.cli` per the official instructions); never a piped
curl-to-shell one-liner. Install gbrain ONLY as
`bun install -g github:garrytan/gbrain#latest-stable` — the npm package named
"gbrain" is an unrelated project.
## Codex preflight (ChatGPT desktop / Codex CLI only)
Codex sandboxes command execution. Before starting, tell the human: "I'll need
approval to run install commands (bun, gh, gbrain) and to write in this folder —
approve those prompts when they appear." If approvals are globally disabled, ask the
human to enable workspace-write + network for this session. Count the approval taps
you needed; report the count at the end (it feeds the install-time measurement).
## Phase walkthrough (commentary — the CLI's list wins)
1. **Preflight.** `git`, `bun`, `gh` present. Install what's missing per the trust
rules above: bun via a platform package manager or the checksum-verified download
— the checksum-verified install is the ONLY permitted non-package-manager
variant; gh via the platform package manager. (On a clean Mac, `git` may trigger
the Xcode tools dialog — that download does not count against the 15 minutes,
tell the human to let it run.)
`gh auth status` — if logged out, the human's ONE manual step:
`gh auth login -h github.com -p https -w` (you run it; they click Authorize).
Then `gbrain bootstrap status` — it is idempotent and resume-aware; after any
partial failure, re-run it and continue where it points.
2. **Engine.** `gbrain init --pglite` (2 seconds, no server). Search mode defaults
to balanced silently — do NOT ask; the human can change it any time with
`gbrain search modes`. The one thing to raise here is the OPTIONAL provider
key — with no key you run keyless: keyword search plus memory you author
yourself through the write tools; everything works, one key upgrades search to
semantic and enables automatic fact extraction. Never pressure for a key. If the
human provides one, pass it to the CLI prompt — it goes to the 0600 config file,
never into the interview answers, never into chat logs you keep.
3. **Interview.** `gbrain bootstrap interview --init`, then ask the questions from
the bank (the CLI prints them) in three batches, recording each answer verbatim
with `--set KEY "value"`. Push once on vague answers to the required questions.
After the last batch: read ALL answers back in one compact block, ask "Is this
the thing you want in the room?", and only then run
`gbrain bootstrap interview --confirm <hash>` with the hash `--status` printed
for the read-back set. The gate fails if you confirm a set the human never saw.
4. **Render.** `gbrain bootstrap render` — identity files appear. Show the human
SOUL.md. Existing files are never overwritten (re-runs are safe; `--force`
backs up first).
5. **Skills + brain wiring.** The CLI scaffolds the skill set and registers
`brain/` as the workspace source. Nothing to judge here; relay the output.
6. **Wire the harness.** `gbrain bootstrap hooks --harness <detected>`:
- Claude Code: installs per-turn hooks ON by default — do NOT ask; loading the
brain every turn is the whole point of installing gbrain for your agent. Tell
the human it is on and how to turn it off (`GBRAIN_HOOKS=0`, or re-run with
`--no-hooks`, or `gbrain bootstrap uninstall`). The ONE consent to actually
ask in this phase is MCP scope: project (recommended — any other repo you open
cannot read your brain) vs user (your agent everywhere, but any repo you open
can query it, and two open sessions contend for the database).
- Codex: registers MCP (`codex mcp add`) and relies on the AGENTS.md protocol —
say plainly that Codex gets pull-based context, not per-turn push.
7. **Private repo.** `gbrain bootstrap repo` — creates a PRIVATE GitHub repo from
the workspace, verifies the privacy bit through the API, pushes. Asks the
background-persistence consent (15-minute scan-gated push job; declining still
persists at session end). If the human has no GitHub or declines: local-only
mode with an honest warning; `bootstrap repo` can run any time later.
8. **Verify.** `gbrain bootstrap verify` — the whole contract: brain round-trip
through the real write path, graph floor, token sweep, secret scan, repo
privacy, hooks smoke, capability report (keyless or keyed). Exit 0 or it is not
done. Paste the report. Then relay the first-run tour it prints (three prompts
the human should try, starting with restarting the session).
## Machine two
If this workspace was cloned from an existing agent repo (agent.json says
initialized), run `gbrain bootstrap attach` instead of the interview/render/repo
phases — it wires this machine (source, hooks, MCP) and verifies. If agent.json
says it is an uninitialized template, proceed with the normal flow from phase 1.
## Failure modes, and what they actually mean
| Symptom | Real cause | Fix |
|---|---|---|
| `interview --status` exits nonzero forever | A required answer is genuinely missing | Ask the human. Do not default it. |
| Render refuses with unresolved tokens | Interview incomplete or a template edit broke a token | Finish the interview; `status` names the tokens. |
| `verify` fails the magic-moment check | The fact never landed (keyless: the Facts fence was not written) | Re-run the write step it names; check `gbrain doctor`. |
| Secret-scan block on push | A credential-shaped string in a tracked file | Fix or allowlist deliberately (`.gbrain-scan-allow`); never force. |
| "bootstrap already running (pid N)" | A concurrent bootstrap holds the lock | Wait or investigate that pid; the lock self-clears when stale. |
| Brain tools fail with a lock error | Another live session's serve owns the database | Close the other session; sequential use is the v1 contract. |
| Hook reports "brain context unavailable" | serve not running or degraded | `gbrain doctor` names it; hooks fail open by design. |
## Hand off
Finish by telling the human: the private repo URL (or the local-only status), the
capability mode (keyless vs keyed), the three commands they will actually reuse
(`gbrain doctor`, `gbrain bootstrap verify`, `gbrain sources push`), and the
first-run tour. Then delete nothing — this runbook was fetched, not installed.
+57
View File
@@ -2,6 +2,63 @@
All notable changes to GBrain will be documented in this file.
## [0.45.0.0] - 2026-08-10
**Your coding agent can now become your personal agent.** Paste one block into Codex or Claude Code and it sets itself up as a persistent agent with a memory that survives across sessions: it interviews you, writes its own identity files from your answers, spins up a local brain, and keeps a private GitHub repo as its durable body. Close the laptop and reopen it tomorrow, and it still knows who you are, who you're talking to, and what you told it last time. This is the OpenClaw/Hermes experience — identity, memory, schedules, persistence — running on the subscription you already pay for, with nothing to deploy.
**Start with Codex.** It runs on your ChatGPT subscription, takes about fifteen minutes, and deploys nothing. Claude Code is the same install. OpenClaw and Hermes are still the way to run GBrain exactly as designed — always on, enriching around the clock — at real server and API cost; Codex is the recommended first step for anyone new to GBrain.
### How to use it
Pick the folder that will become your agent's home and paste (Codex shown; the Claude Code block is identical):
```
Read and follow every step of:
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
as your memory. Interview me before writing any identity file — never invent
answers. Ask before anything destructive. You are not done until
`gbrain bootstrap verify` exits 0.
```
The agent runs `gbrain bootstrap` — a new command family (`status`, `interview`, `render`, `repo`, `hooks`, `verify`, `attach`, `uninstall`) that drives the whole install. It works with **zero API keys**: your harness's model is the LLM, so the agent authors memory directly and search runs keyword-only; add one optional key (OpenAI, Anthropic, or Voyage) to unlock semantic search and automatic fact extraction. Everything is consent-gated — hooks, background push, MCP scope — and nothing runs while your harness is closed (the honest desktop contract; true 24/7 is what a hosted brain adds).
### What you get
- **A memory that compounds.** On Claude Code, session hooks inject relevant brain context at the start of each prompt and ingest the transcript when the session ends; on Codex, the rendered `AGENTS.md` carries the same pull protocol. Facts you state get written back through the brain's own tools and resurface next session.
- **A portable body.** Your workspace is a private GitHub repo (privacy verified via the API before anything is pushed, secret-scanned before every commit). Clone it on a second machine and run `gbrain bootstrap attach`. It mounts anywhere GBrain runs — the exit plan if a provider ever changes its terms.
- **Multi-model install.** Works in the ChatGPT desktop app, the Codex CLI, Claude Code desktop, and the Claude Code CLI. Same body format, two doors.
### What to know after upgrading
Existing installs are untouched — this is a new, opt-in surface. If you want it, run the paste block in a fresh folder. `gbrain doctor` gained bootstrap health checks (hook heartbeat, push staleness, serve/lock collisions, runbook skew); if anything looks off after an install, `gbrain doctor` names it and prints the fix.
### What we caught and fixed before merging
This shipped through a full review pass (architecture, security, adversarial, cross-model). Highlights, in plain terms: the credential scanner now recognizes today's API-key formats and reads the exact bytes being committed rather than a stale snapshot, so a key can't slip through a timing gap; private-key material is redacted whole, not just its header; and the scanner fails safe — an unreadable or oversized file blocks the push instead of passing unscanned. The GitHub repo is confirmed to belong to you and to be private before any content leaves the machine.
## To take advantage of v0.45.0.0
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
warns about a partial migration:
1. **Run the orchestrator manually:**
```bash
gbrain apply-migrations --yes
```
2. **To try the new agent-bootstrap surface,** paste the block above into Codex
or Claude Code in a fresh folder, or read
[`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) for the full contract.
Existing brains need no action — bootstrap is an opt-in new surface, not a
change to how your current install works.
3. **Verify the outcome:**
```bash
gbrain doctor
gbrain stats
```
4. **If any step fails or the numbers look wrong,** please file an issue:
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`
and `~/.gbrain/upgrade-errors.jsonl` if it exists.
## [0.44.1.0] - 2026-08-11
**Any current model works now. gbrain stops rejecting model ids it hasn't heard of.**
+9 -6
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~110 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`; v0.43.0.0 adds the five frozen MEMORY_VERBS `recall`, `remember`, `entity`, `synthesize`, `forget` servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md`). CLI and MCP
Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md` — and the five frozen MEMORY_VERBS `recall`/`remember`/`entity`/`synthesize`/`forget`, servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -126,6 +126,7 @@ detail on demand.)
| eval methodology / metrics | `docs/eval/` |
| brains vs sources / topology | `docs/architecture/brains-and-sources.md`, `topologies.md` |
| skill routing | `skills/RESOLVER.md` |
| agent bootstrap (paste-in install, hooks, `gbrain bootstrap`, sweep, keyless) | `docs/guides/bootstrap.md` + `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + the KEY_FILES bootstrap cluster |
| shipping a release / CHANGELOG / PR conventions | `docs/RELEASING.md` (ship IRON RULES stay inline below) |
The per-file index (`## Key files`), the thin-client routing seam, and the testing
@@ -191,9 +192,10 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
expensive one.
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
picker copy in `gbrain init` carries the same matrix verbatim** — update
both when refreshing.
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix
has three verbatim homes: this section, the `gbrain init` picker copy
(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step
3.5** — update all three when refreshing.
**Per-query math vs real-world spend.** The matrix above is what an
isolated benchmark would measure. Real agent loops with disciplined
@@ -273,8 +275,9 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 52 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills
(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md`
(`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
briefing, migrate, setup, publish.
+55 -75
View File
@@ -48,7 +48,9 @@ src/
core/
operations.ts Contract-first operation definitions (the foundation)
engine.ts BrainEngine interface
postgres-engine.ts Postgres implementation
engine-factory.ts Engine factory (dynamic import of the configured engine)
postgres-engine.ts Postgres + pgvector implementation
pglite-engine.ts PGLite (embedded Postgres via WASM) implementation
db.ts Connection management + schema loader
import-file.ts Import pipeline (chunk + embed + tags)
types.ts TypeScript types
@@ -59,12 +61,16 @@ src/
supabase-admin.ts Supabase admin API
file-resolver.ts MIME detection + content hashing
migrate.ts Migration helpers
bootstrap/ Agent-bootstrap flow (interview, hooks, repo, verify)
yaml-lite.ts Lightweight YAML parser
chunkers/ 3-tier chunking (recursive, semantic, llm)
search/ Hybrid search (vector, keyword, hybrid, expansion, dedup)
embedding.ts OpenAI embedding service
embedding.ts Embedding service (provider-routed; ZeroEntropy default)
mcp/
server.ts MCP stdio server (generated from operations)
http-transport.ts HTTP MCP transport (OAuth, body caps)
dispatch.ts Op dispatch + scope enforcement + param redaction
rate-limit.ts Rate limiting
schema.sql Postgres DDL
skills/ Fat markdown skills for AI agents
test/ Unit tests (bun test, no DB required)
@@ -77,15 +83,21 @@ test/e2e/ E2E tests (requires DATABASE_URL, real Postgres+pgvect
docs/ Architecture docs
```
Per-file invariants live in `docs/architecture/KEY_FILES.md` — read a file's entry
before editing it.
## Running tests
The canonical reference for test tiers, isolation rules, timing, and the E2E
lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version:
```bash
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
# Inner edit loop (~85s on a Mac dev box)
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
bun test test/markdown.test.ts # specific unit test
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck
# Pre-push gate (19+ parallel checks + typecheck)
bun run verify
# Pre-merge sanity (everything CI runs)
bun run test:full # verify + parallel unit + slow + smart e2e
@@ -103,18 +115,17 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
DATABASE_URL=postgresql://... bun run test:e2e
```
Use `bun run verify` before pushing. The guard chain catches: banned fork-name
leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
(`scripts/check-progress-to-stdout.sh`), test-isolation rule violations
(`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel
loop" below), silent fallback to recursive chunking in the compiled binary
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
(`scripts/check-admin-build.sh`), and resolver drift on bundled skills
(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any
warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift
before merge). `bun run check:all` runs the full historical sweep including the
trailing-newline and exports-count checks.
(`scripts/check-admin-build.sh`), resolver drift on bundled skills
(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full
historical sweep including the trailing-newline and exports-count checks.
### Writing tests that survive the parallel loop
@@ -123,75 +134,35 @@ capping total concurrency (shards × intra-shard files) to available memory and
re-running OOM-killed or externally-killed files serially before calling them
failures (see `docs/TESTING.md` for the rescue-pass details and knobs). Files
in the same shard share a process, so process-global state leaks between them.
Four lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
Four lint rules (`scripts/check-test-isolation.sh`, R1R4) enforce isolation:
no direct `process.env` mutation (use `withEnv()` from
`test/helpers/with-env.ts`), no `mock.module(...)` outside `*.serial.test.ts`,
and every `new PGLiteEngine(` goes inside the canonical `beforeAll` block with
a paired `afterAll(disconnect)`.
| Rule | What it bans | Fix |
|---|---|---|
| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` |
| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` |
| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) |
| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` |
**The full rules, the canonical PGLite block, the `withEnv` pattern, and the
`*.serial.test.ts` quarantine policy live in
[`docs/TESTING.md`](docs/TESTING.md#test-isolation-lint-and-helpers)
— read that before writing a new test file.** Files that predate the rules are
listed in `scripts/check-test-isolation.allowlist`; the allow-list MUST shrink
over time — never add new entries.
Canonical PGLite block (R3 + R4 compliant — paste this verbatim):
```ts
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => { await engine.disconnect(); });
beforeEach(async () => { await resetPgliteState(engine); });
```
Env-touching tests:
```ts
import { withEnv } from './helpers/with-env.ts';
test('reads OPENAI_API_KEY', async () => {
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
expect(loadConfig().openai_key).toBe('sk-test');
});
});
```
`withEnv` saves and restores keys via try/finally including when the callback
throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is
process-global). Files using `withEnv` stay outside the future
`test.concurrent()` codemod's eligibility filter.
When to quarantine instead of fix: rename to `*.serial.test.ts` if the file
uses `mock.module(...)`, is genuinely env-coupled (module-load env readers +
ESM caching defeat dynamic-import-after-env tricks), or intentionally shares
state across `it()` boundaries. Quarantine count cap: 10 (informational).
Files that violated these rules at the v0.26.7 baseline are listed in
`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over
time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep
+ codemod) remove entries as files get fixed.
### Local CI gate (recommended before pushing, v0.23.1+)
### Local CI gate (recommended before pushing)
```bash
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
bun run ci:local # full gate: gitleaks + guards/typecheck + 4-shard parallel unit + E2E
bun run ci:local:diff # gate with diff-aware E2E selector
bun run ci:select-e2e # print which E2E files the selector would run
```
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
`ci:local` spins up four pgvector services plus a transaction-mode PgBouncer via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite
sharded 4 ways in parallel, then tears down. Named volumes keep the install warm
across runs. Requires Docker (Docker Desktop, OrbStack, or Colima) and `gitleaks`
on host (`brew install gitleaks`). Override the postgres host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
Fail-closed selector: an unmapped `src/` change runs ALL E2E files. Hand-tune
narrower mappings via `scripts/e2e-test-map.ts`.
### PR-side security checks
@@ -237,7 +208,7 @@ Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
See `docs/ENGINES.md` for the full guide. In short:
1. Create `src/core/myengine-engine.ts` implementing `BrainEngine`
2. Add to engine factory in `src/core/engine.ts`
2. Add to the engine factory in `src/core/engine-factory.ts`
3. Run the test suite against your engine
4. Document in `docs/`
@@ -322,7 +293,7 @@ NDJSON wire format is documented in
[`docs/eval-capture.md`](./docs/eval-capture.md).
For public benchmark coverage on top of replay, `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
<dataset.jsonl>` runs LongMemEval against gbrain's hybrid
retrieval. One in-memory PGLite per question, runtime-enumerated
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
published `evaluate_qa.py`. Use it alongside replay when changes affect
@@ -331,9 +302,18 @@ regressions on YOUR queries, LongMemEval catches them on a public set the
benchmark community already cites. See the "Public benchmarks: LongMemEval"
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
## Shipping
Releases go through the `/ship` skill, never hand-rolled. The full release +
contributor process (CHANGELOG voice, version-locations sync, PR conventions,
community-PR-wave workflow) lives in [`docs/RELEASING.md`](docs/RELEASING.md).
Community PRs are batched into release waves rather than merged one-by-one;
contributor attribution stays attached via `Co-Authored-By:` trailers and every
accepted contribution is credited in `CHANGELOG.md`.
## Welcome PRs
- SQLite engine implementation
- Additional engine implementations (see [`docs/ENGINES.md`](docs/ENGINES.md))
- Docker Compose for self-hosted Postgres
- Additional migration sources
- New enrichment API integrations
+24 -6
View File
@@ -3,6 +3,11 @@
Read this entire file, then follow the steps. Ask the user for API keys when needed.
Target: ~30 minutes to a fully working brain.
> **Scope note:** this guide installs the BRAIN (for agent platforms like OpenClaw
> and Hermes, or a standalone brain). If you are Claude Code or Codex and the human
> asked you to become their persistent personal agent — identity + memory + private
> repo — follow `BOOTSTRAP_FOR_AGENTS.md` instead.
## Step 0: If you are not Claude Code
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
@@ -16,10 +21,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
> supported sources are `github:garrytan/gbrain` (optionally pinned as
> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a
> git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
@@ -88,6 +96,7 @@ default. Stop and ask the operator.
**Present this matrix verbatim:**
<!-- Cost matrix: three verbatim homes — CLAUDE.md "Search Mode", src/commands/init-mode-picker.ts, and this block. Sync all three when refreshing. -->
```
Per-query cost @ 10K queries/mo (typical single-user volume):
@@ -198,13 +207,13 @@ scaffold the bundled skills into it:
```bash
cd /path/to/agent/workspace
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md
```
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
diff against gbrain's bundle when you want upstream improvements. (The legacy
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
`gbrain skillpack install` managed-block model was removed in v0.33 — run
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
@@ -261,8 +270,17 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
## Step 9: Verify
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
actually works) is the most important.
Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4
(live sync actually works) is the most important.
Once verification passes and the brain has content, run the activation probe:
```bash
gbrain onboard --check --json
```
See "The onboard surface" below for what the recommendations mean and the
consent gates around unattended remediation.
## Upgrade
@@ -305,7 +323,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
`compiled_truth` from source markdown.
## v0.42.0+ onboard surface (NEW)
## The onboard surface
`gbrain onboard` is the activation surface gbrain did not have before.
Once your brain has any content, run `gbrain onboard --check --json` to
+88 -73
View File
@@ -2,7 +2,7 @@
**Search gives you raw pages. GBrain gives you the answer.** It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box. Run a full autonomous agent on top of it, or just wire it into Claude Code or Codex as a supercharged retrieval layer in one command; either way your coding agent stops being amnesiac about everything that isn't code.
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **155,795 pages, 24,589 people, 5,340 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
**And now it works as a company brain too.** Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the [company-brain](https://www.ycombinator.com/rfs#company-brain) shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. **[Tutorial: set up GBrain as your company brain →](docs/tutorials/company-brain.md)**
@@ -11,7 +11,7 @@ Lots of personal-knowledge systems give you keyword matching and grep in a box.
- **A synthesis layer that gives you the actual answer.** Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
- **A self-wiring knowledge graph.** Every page write extracts entity refs and creates typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, **+31.4 points P@5** over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
The point of building a 150K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.
@@ -75,11 +75,41 @@ This is the difference between a search engine and a brain. Search finds the pag
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
GBrain is designed to be installed and operated by an AI agent. **New to GBrain? Start with Codex** — it runs on the ChatGPT subscription you already have, takes ~15 minutes, and deploys nothing. Already living in Claude Code? Its path is identical. Want GBrain running the way it was designed to run — always on, enriching your brain around the clock? That's OpenClaw or Hermes, at real server + API cost. Each path below is complete on its own. (Wiring it up by hand instead? Jump to [CLI standalone](#cli-standalone-no-agent) or the [MCP table](#connect-gbrain-to-your-ai-client-mcp).)
### Have your agent install it (recommended)
### For Codex — the recommended first step
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
Turn Codex into your persistent personal agent. Works in the **ChatGPT desktop app** (open Codex on a folder) and in the **Codex CLI** (`codex` in a terminal) — same install, same result. Pick the folder that will become your agent's home, and paste:
```
Read and follow every step of:
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
as your memory. Interview me before writing any identity file — never invent
answers. Ask before anything destructive. You are not done until
`gbrain bootstrap verify` exits 0.
```
Codex will ask for command approvals during the install — approving them is the sandbox working as intended. What you get, in about 15 minutes: a short interview (6 required questions) → your agent's identity (SOUL.md, USER.md, MEMORY.md) rendered from your own answers, never invented → a local PGLite brain (2 seconds, no server, no Docker) → MCP wired so every session can search and write memory → a **private** GitHub repo, created and privacy-verified, as your agent's durable body. Works with **zero API keys** — keyword search plus memory your agent writes itself; one optional key (OpenAI, Anthropic, or Voyage) upgrades to semantic search and automatic fact extraction. Codex reads brain context through its tools each turn (pull-based).
### For Claude Code — turn it into your persistent personal agent
Works in the **desktop app** and in the **CLI** (`claude` in a terminal) — identical harness, identical result. Open Claude Code in the folder that will become your agent's home, and paste the same block:
```
Read and follow every step of:
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
as your memory. Interview me before writing any identity file — never invent
answers. Ask before anything destructive. You are not done until
`gbrain bootstrap verify` exits 0.
```
Everything from the Codex path applies — interview, identity from your own answers, local brain, private repo, keyless mode — plus Claude Code gets **per-turn context hooks** (consent-gated): your brain loads automatically into every prompt, and each session persists itself to your private repo at exit. Restart the session after install and ask "what did I tell you my top jobs were?" — that's the moment it clicks. Full contract, security posture, and uninstall: [docs/guides/bootstrap.md](docs/guides/bootstrap.md).
### For OpenClaw or Hermes — GBrain as intended, always on
This is GBrain used the way it was designed to be used: a server-hosted agent with 24/7 crons, continuous ingestion, and the overnight dream cycle that enriches your brain while you sleep — your agent works whether your laptop is open or not. It's also the highest-cost path: a deployed server (8GB+ RAM) plus raw API token usage that scales with how hard your agent runs, well beyond a chat subscription. Start here if you want the full experience from day one; start with Codex above if you want to feel it first. If you don't have a platform running yet, both deploy in one click:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
@@ -91,15 +121,13 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
The agent installs GBrain, creates the brain, asks for your API keys, loads 52 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
### Quick start: Claude Code or Codex
### Lighter ways in
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full 110-tool wall; drop the flag for every operation:
**Just want a memory for your coding agent — no identity, no repo.** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full tool wall; drop the flag for every operation:
```bash
gbrain init --pglite # 2-second local brain (no Docker)
@@ -115,18 +143,9 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --install
gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install # Codex
```
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — both paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
**Brain-only install into another coding agent** (Cursor, Claude Cowork, or anything that can fetch a URL and run shell commands) — paste the OpenClaw/Hermes block above (`INSTALL_FOR_AGENTS.md`); it installs the brain, skills, and dream cycle without the personal-agent identity layer. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
### Install the full autonomous setup into your existing agent
Want the whole thing — local brain, 52 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
```
Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — the memory-only paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
### CLI standalone (no agent)
@@ -142,7 +161,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
### Connect GBrain to your AI client (MCP)
GBrain exposes 110 tools over MCP (stdio and HTTP) — or exactly the five memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side) — or exactly the five memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
@@ -178,7 +197,7 @@ gbrain think "who's working on AI agents at portfolio companies?"
**`gbrain think`** runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 150K-page brain is worth the effort.
`gbrain agent run "..."` exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.
@@ -264,7 +283,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
## Capabilities
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
@@ -288,7 +307,7 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. `gbrain eval brainbench` runs the cross-harness memory conformance suite: know-to-ask, push precision/recall, write-back fidelity, and cross-session continuity, scored per harness seam (your OpenClaw's production pipeline plus Claude Code and Codex injection contracts) against a committed 141-fixture synthetic corpus — hermetic by default (in-memory PGLite, no keys, seconds), and CI gates every PR against master's committed baseline. Methodology in [`docs/eval/BRAINBENCH.md`](docs/eval/BRAINBENCH.md); search-mode methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
@@ -302,14 +321,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
## Architecture
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source.
**Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md).
@@ -323,10 +342,9 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
two flags + a recommended pattern. Switch your cron to a per-source loop
with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating
gracefully half-a-minute earlier:
**Hourly cron sync keeps timing out on a federated brain?** Switch your
cron to a per-source loop with shell `timeout(1)` doing the OS-level kill
and gbrain self-terminating gracefully half-a-minute earlier:
```bash
gbrain sync --break-lock --all --max-age 1800
@@ -339,19 +357,17 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status
`partial` and `last_commit` UNCHANGED — the next run re-walks the same
diff and `content_hash` short-circuits already-imported files. The
`--max-age 1800` first command self-heals any wedged-but-alive locks
left by a hung previous run, using the v98 `last_refreshed_at` semantic
(NOT `acquired_at`) so healthy long-running holders are safe by
construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md)
for the honest scope notes (extract + embed phases run to completion;
30-min rollout window for `--max-age` post-migration v98; full-sync
triggers deferred to v0.42+).
left by a hung previous run, keyed on the lock's last refresh time
(NOT when it was acquired) so healthy long-running holders are safe by
construction. Scope note: the extract + embed phases still run to
completion once started; `--timeout` interrupts the import walk only.
**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes
the bug class structurally. The engine now self-retries every bulk batch
write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on
Supavisor pooler blips, with a 12s worst-case wait that covers the full
5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents
via the new `batch_retry_health` check (reads the last 24h of
**Dream cycle silently losing wiki links on Supabase?** The engine
self-retries every bulk batch write (`addLinksBatch` /
`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips,
with a 12s worst-case wait that covers the full 5-10s circuit-breaker
recovery window. `gbrain doctor` surfaces incidents via the
`batch_retry_health` check (reads the last 24h of
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually
slow pooler:
@@ -369,34 +385,33 @@ retry wrap is engine-level, but PGLite has no pooler so retries never
fire in practice.
**Dream cycle losing ~150 link rows per run with `'No database
connection: connect() has not been called'` errors in the log?** v0.41.27.0
makes the retry layer self-heal on a nulled-out database singleton. A
new `reconnect` callback on `withRetry` rebuilds the connection between
attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()`
connection: connect() has not been called'` errors in the log?** The
retry layer self-heals on a nulled-out database singleton: a
`reconnect` callback on `withRetry` rebuilds the connection between
attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()`
so engine-level batch writes survive a mid-cycle disconnect by something
else in the same process. Same release: `gbrain capture` no longer trails
a `'No database connection'` stderr line from a background facts:absorb
worker firing after CLI exit — the op-dispatch finally block awaits
else in the same process. `gbrain capture` also no longer trails a
`'No database connection'` stderr line from a background facts:absorb
worker firing after CLI exit — op dispatch awaits
`getFactsQueue().drainPending({timeout: 1000})` before
`engine.disconnect()`. To find which code path is still calling
disconnect mid-process, run `gbrain doctor --json | jq '.checks[] |
select(.id=="batch_retry_health")'`; the extended check now surfaces
24h disconnect-call count and the most-recent caller frame from a new
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.)
select(.id=="batch_retry_health")'`; the check surfaces the
24h disconnect-call count and the most-recent caller frame from the
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit.
**`gbrain brainstorm` returning `judge_failed: true` with 0 scored
ideas?** v0.41.21.0 closes the two bugs that caused it. The judge
hard-coded a 4K-token output cap; for any run past ~40 ideas the call
truncated mid-JSON and the parser threw. Same release closes a slash-
form pricing miss: `gbrain brainstorm --judge-model
anthropic/claude-sonnet-4-6 --max-cost 5` failed with
`BudgetExhausted reason=no_pricing` because every pricing site only
matched the colon form. Both shapes work now. No config change, no
schema migration — `gbrain upgrade` is the whole fix.
ideas?** Two historical bugs caused it, both fixed: the judge
hard-coded a 4K-token output cap (any run past ~40 ideas truncated
mid-JSON and the parser threw), and slash-form model ids
(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because
pricing lookups only matched the colon form. Both shapes work now. No
config change, no schema migration — `gbrain upgrade` is the whole fix.
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
`reindex --markdown` now ADD current frontmatter tags and never delete,
tags?** Upgrade — tag reconciliation is add-only now. Re-import and
`reindex --markdown` ADD current frontmatter tags and never delete,
so enrichment tags written to the DB (auto-tag, dream synthesize,
signal-detector) survive a re-chunk. The reindex DB-only fallback also
reconstructs the full markdown (frontmatter + body + timeline) before
@@ -404,10 +419,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter,
title, and timeline instead of getting overwritten with empty
frontmatter. Trade-off: removing a tag from a page's frontmatter no
longer removes it from the DB on the next sync (frontmatter-tag removal
needs a provenance column, deferred). (Closes #1621.)
needs a provenance column, deferred).
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
v0.41.37.0 ships three things. First, name the stalling file:
Three tools. First, name the stalling file:
```bash
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
@@ -422,28 +437,28 @@ the sync with the pack disabled and re-run extraction later:
gbrain sync --no-schema-pack --no-pull --no-embed --yes
```
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
`gbrain schema lint` warns on the classic nested-quantifier ReDoS
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
PGLite is single-writer and a live MCP server contends for the write
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
for the full triage. (Closes #1569.)
for the full triage.
**`gbrain init --migrate-only` / a schema migration fails on Windows
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
with `getaddrinfo ENOTFOUND`?** Upgrade schema bring-up now runs its
phases in-process instead of spawning a child `gbrain init
--migrate-only` per phase. The spawned child died on
Windows + bun + Supabase pooler with a DNS-resolution failure even
though the parent connected fine; running in-process removes the spawn
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
completes in ~1-2 seconds. (Closes #1605, #1581.)
entirely. The grandfather migration that used to hang 70+ minutes on an
80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on
the page PK, soft-delete-filtered, source-safe) and completes in seconds.
## Docs
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall)
- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows`
- [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory
+148
View File
@@ -5015,3 +5015,151 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
+ the models-doctor tests.
**Depends on:** nothing.
## Agent-bootstrap wave follow-ups (filed at build time)
- [x] **P2 — compiled `gbrain` binary can now `serve` a PGLite brain.** FIXED:
`src/core/pglite-embedded-assets.ts` embeds PGLite's runtime payload
(`pglite.wasm`, `initdb.wasm`, `pglite.data`, `vector.tar.gz`,
`pg_trgm.tar.gz`) with Bun's `import ... with { type: 'file' }` and hands them
to PGLite via `PGliteOptions` (`pgliteWasmModule` / `initdbWasmModule` /
`fsBundle` + custom `vector`/`pg_trgm` extensions whose `setup()` returns a
`bundlePath` pointing at the embedded tarball, materialized to a temp file
because `createReadStream` — unlike `readFileSync` — can't read `/$bunfs`
paths). `src/core/pglite-engine.ts` spreads `getEmbeddedPgliteOptions()` at
both `PGlite.create()` sites, so the Bun-vfs #1340 ENOENT no longer fires for
a correctly-built binary. Guarded by `scripts/check-pglite-embedded.sh`
(compiles a smoketest and asserts a real PGLite query round-trips), wired into
`bun run verify` + `check:all` + `check:pglite-embedded`. The real-agent e2e
harness (`test/helpers/agent-harness.ts`) now resolves to the fast compiled
MCP server; the `bun run` fallback stays as a safety net. Upstream Bun issue
(still open since Nov 2024) is now moot for gbrain.
- [ ] **P1 — `--background --follow` spawns a nonexistent subcommand.**
`src/core/cli-options.ts` (~:391) spawns `gbrain jobs follow <id>` after a
background submit, but jobs.ts has no `follow` subcommand (`jobs watch
--follow` exists and takes no id; `jobs submit --follow` is inline-only).
The spawned child hits the unknown-subcommand path, so `--background
--follow` submits fine but the live stream never attaches. Fix: implement
`jobs follow <id>` (poll get_job + stream progress) or retarget the spawn.
Found during the v0.45.0.0 doc audit; KEY_FILES documents current behavior.
- [ ] **P1 — bootstrap status is not workspace-scoped.** status.ts reads the
newest GLOBAL verify snapshot + global push-status, so a green verify in
workspace A can report workspace B as verified/healthy. Key the verify
snapshots + push-status by workspace path (or store them under the
workspace's own state dir) so status reflects the workspace it runs in.
Found by the v0.45.0.0 adversarial pass (Codex P2, raised to P1 — multi-
workspace is the graduation/multi-device story).
- [ ] **P2 — mkdir-lock ABA steal race (folds into the shared-primitive TODO).**
Both acquirePushLock (workspace-push.ts) and acquireBootstrapLock (lock.ts)
steal a dead+stale lock as rmSync-then-mkdir with no re-verify between, so
two simultaneous stealers can both believe they hold it (same known class as
pglite-lock; git index.lock bounds the push damage). When extracting the
shared mkdir-lock primitive, harden the steal: re-read + token-compare the
stale owner immediately before rmSync, or use an atomic rename-based steal.
Cross-model finding (Claude + Codex) v0.45.0.0 adversarial.
- [ ] **P2 — uninstall TOCTOU on a live PGLite DB.** uninstall probes the serve
lock then recursively deletes later; a serve starting in between wins the
race. Gated today by the lock probe + refuse-on-live-serve, so this is the
residual known lock-class window — close it when the shared lock primitive
lands (hold the lock across the probe→delete span). Codex P1 v0.45.0.0.
- [ ] **P2 — sweep fairness: recency-only selection starves old pages.**
sweep.ts pass 2 repeatedly takes the newest batchLimit pages with no
extraction-watermark filter, so frequently-updated pages monopolize every
sweep and older pages never get link/timeline extraction. Add an
extracted-watermark (or round-robin) so the backlog drains. Codex P2.
- [ ] **P2 — untracked deny-glob exclusion is invisible under detached push.**
A first-time deny-glob file (e.g. a fresh .env) is dropped from staging and
reported only via the logger callback + excludedUntracked — but session-end/
session-start auto-push runs via spawnDetachedPush with stdio:'ignore', so
the warning is discarded. Fail-safe (never leaves the machine) but the user
gets a false "synced" impression. Surface excluded deny-matches via the
heartbeat/push-status file so doctor + status can report them. Claude
adversarial (minor) v0.45.0.0.
- [ ] **P2 — shared receipt upsert helper.** InstallReceipt construct/merge
logic is quadruplicated (bootstrap.ts writeRenderReceipt +
appendReceiptRegistration, attach.ts, repo.ts recordRepoInReceipt) with
slightly different defaults; guardReceiptOverwrite is wired at each site
individually. One `upsertReceipt(home, ws, patch)` in format.ts.
- [ ] **P2 — promote the atomic tmp+rename write idiom to one core helper.**
~7 hand-rolled copies across format.ts/interview.ts/lock.ts/render.ts/
hook.ts/workspace-push.ts/verify.ts; hooks.ts already has a private
atomicWriteJson to promote (plus a text variant).
- [ ] **P2 — extract a shared mkdir-lock primitive.** bootstrap/lock.ts and
workspace-push.ts both implement atomic-mkdir + PID/age/token steal rules
citing the same pglite-lock learnings (#2058/#2348); three parallel
implementations counting pglite-lock. One primitive with injectable stale
window + throw-vs-result adapters.
- [ ] **P2 — move the hook heartbeat read surface into core.** core/bootstrap/
status.ts and core/bootstrap/verify.ts dynamic-import readHeartbeatTail
from commands/hook.ts (core→commands inversion). A core/hooks-telemetry.ts
owning the read/write surface removes the reach-ins.
- [ ] **P2 — consolidate the two GitHub remote parsers + privacy probes.**
workspace-push.ts parseGithubOwnerRepo (accepts ssh:// + trailing /) vs
repo.ts parseGithubRemote (scp-form only), and verifyRemotePrivacy (gh repo
view) vs verifyRepoPrivate (gh api): grammars/failure classes can drift.
One exported parser + one privacy-probe core returning the verdict union.
- [ ] **P2 — connector-ingest capability probe (with v1.1 connector ingest).**
The plan's build-order-2 probe (email/calendar connector availability wired
into install output) deferred with the feature it gates; the capability
report today covers embeddings + extraction only.
- [ ] **P2 — G15 retention warnings in doctor.** MEMORY.md size cap + corpus
retention + orphaned stop-hook buffers are enforced/pruned but doctor never
warns when they accumulate; add the three checks to the bootstrap group.
- [ ] **P2 — NER-based graph floor in verify.** The graph-floor check asserts
wikilink-edge/backlink link-table rows; a true entity-extraction floor
(≥1 NER entity + one edge-only query op) needs the extraction path or a
keyed provider, so it rides with a keyed-mode verify extension.
- [ ] **P2 — `--isolated` flag productization.** GBRAIN_HOME threading through
MCP registration env, hooks, and uninstall guards ships in v1; the
documented `bootstrap --isolated` one-flag wrapper (set env + assert
database_path containment + gitignore check, per D2=C) is not yet a flag.
- [ ] **P2 — sweep N+1 page fetch.** The links/timeline pass getPage-per-slug
loop is bounded (batchLimit 20) but serializes round-trips on the live
serve connection; add a getPagesBySlugs batch read to both engines (engine
parity + bootstrap-probe obligations) and use it.
- [ ] **P2 — `gbrain quota` meter command.** Productize the release-gate quota
measurement (script+doc ship with the bootstrap wave) into a command that reports
a session/day's token burn against the harness subscription allowance. Blocked on
a proven per-harness token-count method — an inaccurate meter erodes the trust it
exists to build (CEO review D3.4; not promoted by a passing quota gate, by
decision). Start: the spike's measurement scripts under docs/designs/.
- [ ] **P2 — Networked Docker paste-flow e2e.** The offline container e2e (interview
→ render → verify with fake gh) ships with the wave; the full networked flow (bun
install, gh auth, repo create, MCP registration) stays manual. Unblocks after 10
consecutive green offline runs in heavy-tests (CEO review D3.3b). Start:
tests/docker/ harness + the fake-gh recording shim.
- [ ] **P2 — Real-agent door e2e needs a provisioned runner.** The real-binary door
tests (`test/e2e/bootstrap-real-{claude,codex}.serial.test.ts`) drive the ACTUAL
`claude`/`codex` binaries against a real gbrain and pay API cost. They self-SKIP
(`describe.skipIf` on binary/auth) everywhere else, so the `real-agent-e2e` job in
`.github/workflows/heavy-tests.yml` is a green no-op on stock GitHub runners. To
actually EXERCISE them we need a self-hosted / manually-provisioned runner with
authed `claude` + `codex` and provider creds exported
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`). Until then they
run locally on an operator machine only. Start: stand up a labeled runner with the
binaries pre-authed, or a scheduled self-hosted lane.
- [ ] **P1 — unit-shard exit hang: bun test process leaks a ref'd handle and never
exits after all tests pass.** Probabilistic (scales with file count/duration),
PRE-EXISTING: reproduces on origin/master with the same 274-file combination
(`bun test $(cat list) --max-concurrency=2` — shard-2 composition from the
agent-bootstrap wave's reshuffled split; solo halves/eighths pass, full list
hangs on both branches; a repeat of the same 68-file quarter passed, so it is a
race, not a single file). Forensics: the "[process-cleanup] db-lock:test:r1:
PGlite is closed" tail line is printed BY the SIGTERM cleanup at kill time, not
pre-hang; db-lock refresh + process-watchdog intervals are already unref'd —
the leak is elsewhere (candidate class: a ref'd handle in a PGLite/worker
teardown race). run-unit-parallel.sh now classifies the exact signature
(killed at cap + all assigned files started + zero fail markers) as a loud
warn-pass so it can't red-X unrelated work; fixing the leak removes the
classifier's reason to exist. Start: reproduce with `--inspect` /
Bun.unsafe process handle dumps on the full shard-2 list.
- [ ] **P2 — WAL-repair unit tests flake under parallel shard load.**
`test/pglite-repair.test.ts` (`repairPgliteWal rename-based backup`,
`inspectPgliteDataDir verdicts`) failed once in a full 4-shard run and pass
44/44 standalone — load-sensitive, from the v0.42.75.0 WAL auto-repair wave
(pre-existing on master; observed during the agent-bootstrap gate runs).
Start: run the file under `--max-concurrency=4` alongside PGLite-heavy
neighbors to reproduce; suspect tmp-dir or timing assumptions.
+1 -1
View File
@@ -1 +1 @@
0.44.1.0
0.45.0.0
+44 -82
View File
@@ -4,7 +4,7 @@
Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else.
v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
Two engines ship today: `PGLiteEngine` embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
## Why this matters
@@ -12,7 +12,7 @@ Different users have different constraints:
| User | Needs | Best engine |
|------|-------|-------------|
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) |
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) |
| Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase |
| Open source hacker | Single file, no server, git-friendly | PGLiteEngine |
| Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted |
@@ -23,72 +23,30 @@ The engine interface means we don't have to choose. PGLite is the zero-friction
## The interface
```typescript
// src/core/engine.ts
**The single source of truth is `export interface BrainEngine` in
`src/core/engine.ts`.** It is large (100+ methods) and grows with every
feature wave — do NOT work from any snapshot of it, including an old copy of
this doc. Read the interface itself, and let
`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you
whether both engines agree.
export interface BrainEngine {
// Lifecycle
connect(config: EngineConfig): Promise<void>;
disconnect(): Promise<void>;
initSchema(): Promise<void>;
transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T>;
The method families, to orient you before opening the file:
// Pages CRUD
getPage(slug: string): Promise<Page | null>;
putPage(slug: string, page: PageInput): Promise<Page>;
deletePage(slug: string): Promise<void>;
listPages(filters: PageFilters): Promise<Page[]>;
// Search
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
// Chunks
upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void>;
getChunks(slug: string): Promise<Chunk[]>;
// Links
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
removeLink(from: string, to: string): Promise<void>;
getLinks(slug: string): Promise<Link[]>;
getBacklinks(slug: string): Promise<Link[]>;
traverseGraph(slug: string, depth?: number): Promise<GraphNode[]>;
// Tags
addTag(slug: string, tag: string): Promise<void>;
removeTag(slug: string, tag: string): Promise<void>;
getTags(slug: string): Promise<string[]>;
// Timeline
addTimelineEntry(slug: string, entry: TimelineInput): Promise<void>;
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
// Raw data
putRawData(slug: string, source: string, data: object): Promise<void>;
getRawData(slug: string, source?: string): Promise<RawData[]>;
// Versions
createVersion(slug: string): Promise<PageVersion>;
getVersions(slug: string): Promise<PageVersion[]>;
revertToVersion(slug: string, versionId: number): Promise<void>;
// Stats + health
getStats(): Promise<BrainStats>;
getHealth(): Promise<BrainHealth>;
// Ingest log
logIngest(entry: IngestLogInput): Promise<void>;
getIngestLog(opts?: IngestLogOpts): Promise<IngestLogEntry[]>;
// Config
getConfig(key: string): Promise<string | null>;
setConfig(key: string, value: string): Promise<void>;
// Migration + advanced (added v0.7)
runMigration(sql: string): Promise<void>;
getChunksWithEmbeddings(slug: string): Promise<ChunkWithEmbedding[]>;
}
```
- **Lifecycle + identity** — `connect` / `disconnect` / `reconnect`,
`initSchema`, `transaction`, `withReservedConnection`, and the `kind`
discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch.
- **Pages CRUD** — `getPage`, `putPage`, `deletePage`, `listPages`, slug
resolution.
- **Search** — `searchKeyword`, `searchVector`, chunk-level variants, takes
search (keyword + vector), and `relationalFanout` (the typed-edge recall
arm).
- **Chunks + embeddings** — upsert/get, embedding-bearing variants.
- **Graph** — links (single + batch writers), backlinks, `traverseGraph`,
`traversePaths`.
- **Tags, timeline (single + batch), raw data, versions.**
- **Takes / facts / eval / salience** — the epistemological layer and the
instruments over it.
- **Stats, health, ingest log, config, migrations.**
### Key design choices
@@ -131,7 +89,7 @@ export interface BrainEngine {
RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific.
## PostgresEngine (v0, ships)
## PostgresEngine
**Dependencies:** `postgres` (porsager/postgres), `pgvector`
@@ -144,9 +102,7 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
- JSONB for frontmatter with GIN index
- Connection pooling via Supabase Supavisor (port 6543)
**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in.
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
@@ -193,17 +149,17 @@ run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine (v0.7, ships)
## PGLiteEngine
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
**Dependencies:** `@electric-sql/pglite`
**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented.
**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep.
**PGLite-specific details:**
- Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes)
- Parameterized queries throughout (shared utilities in `src/core/utils.ts`)
- `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set
- Data stored at `~/.gbrain/brain.db` (configurable)
- Data stored at `~/.gbrain/brain.pglite` (configurable)
- pgvector HNSW index for cosine similarity vector search (same as Postgres)
- tsvector + ts_rank for full-text search (same as Postgres)
- pg_trgm for fuzzy slug resolution (same as Postgres)
@@ -319,16 +275,22 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make.
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
2. Add to engine factory in `src/core/engine-factory.ts`:
```typescript
export function createEngine(type: string): BrainEngine {
switch (type) {
case 'pglite': return new PGLiteEngine();
case 'postgres': return new PostgresEngine();
case 'myengine': return new MyEngine();
default: throw new Error(`Unknown engine: ${type}`);
export async function createEngine(config: EngineConfig): Promise<BrainEngine> {
switch (config.engine || 'postgres') {
case 'pglite': {
const { PGLiteEngine } = await import('./pglite-engine.ts');
return new PGLiteEngine();
}
case 'myengine': {
const { MyEngine } = await import('./my-engine.ts');
return new MyEngine();
}
// ...
}
}
```
The factory uses dynamic imports so engines are only loaded when selected.
The factory uses dynamic imports so an engine's dependencies (e.g. the
PGLite WASM blob) are only loaded when that engine is selected.
3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }`
4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor.
5. Document in this file + add a design doc in `docs/`
@@ -359,7 +321,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu
| JSONB queries | GIN index | GIN index | Identical |
| Concurrent access | Connection pooling | Single process | PGLite limitation |
| Hosting | Supabase, self-hosted, Docker | Local file | |
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 |
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical |
## Future engine ideas
+8
View File
@@ -6,6 +6,14 @@ A system prompt for any AI agent that wants to build and maintain a personal kno
Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest.
> **Relationship to schema packs:** this document is the prose, paste-in
> version of the schema pattern. gbrain also ships a machine-enforced
> counterpart — schema packs (`gbrain schema`, typed pages, extraction,
> aliases, lint) — documented in `docs/architecture/schema-packs.md` and
> `docs/schema-author-tutorial.md`. The prose schema here and the active
> schema pack should describe the same brain; when you evolve one, evolve
> the other.
---
## What this is
+4 -2
View File
@@ -1,10 +1,10 @@
<!-- skillpack-version: 0.7.0 -->
<!-- source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md -->
# GBrain Skillpack: Reference Architecture for AI Agents
This is a reference architecture for how a production AI agent uses gbrain as its
knowledge backbone. Based on patterns from a real deployment with 14,700+ brain
files, 40+ skills, and 20+ cron jobs running continuously.
files, the 50+ bundled skills (`skills/manifest.json`), and 20+ cron jobs running
continuously.
**The memex vision, realized.** Vannevar Bush imagined a device where an individual
stores everything, mechanized so it may be consulted with exceeding speed. GBrain is
@@ -25,6 +25,7 @@ The foundational read-write loop and data model.
| [Entity Detection](guides/entity-detection.md) | Run it on every message. Capture original thinking + entity mentions |
| [The Originals Folder](guides/originals-folder.md) | Capturing WHAT YOU THINK, not just what you found |
| [Brain-First Lookup](guides/brain-first-lookup.md) | Check the brain before calling any external API |
| [Push-Based Context](guides/push-context.md) | volunteer_context: the brain volunteers relevant pages instead of waiting to be asked |
| [Compiled Truth + Timeline](guides/compiled-truth.md) | Above the line: current synthesis. Below: append-only evidence |
| [Source Attribution](guides/source-attribution.md) | Every fact needs a citation. Format and hierarchy |
@@ -99,6 +100,7 @@ Keeping it running and up to date.
| Guide | What It Covers |
|-------|---------------|
| [Agent Bootstrap](guides/bootstrap.md) | The paste-in install: `gbrain bootstrap`, hooks, `bootstrap verify`, keyless mode |
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
+28 -14
View File
@@ -1,5 +1,13 @@
# GBrain Installation Verification Runbook
> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install
> contract (round-trip, graph floor, and more) automatically and exits non-zero
> on failure — it is the modern first thing to run after any install. See
> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the
> **manual, deep-verification** companion: use it when `bootstrap verify` fails
> and you need to isolate which layer broke, or when you want to understand
> what "healthy" looks like check by check.
Run these checks after install to confirm every part of GBrain is working.
Each check includes the command, expected output, and what to do if it fails.
@@ -20,7 +28,8 @@ gbrain doctor --json
**Expected:** All checks return `"ok"`:
- `connection`: connected, N pages
- `pgvector`: extension installed
- `rls`: enabled on all tables
- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains
skip this check; the embedded engine has no remote surface)
- `schema_version`: current
- `embeddings`: coverage percentage
@@ -33,12 +42,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table.
**Check:** Ask the agent: "What is the brain-agent loop?"
**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes
the read-write cycle: detect entities, read brain, respond with context, write
brain, sync.
**Expected:** The agent describes the read-write cycle documented in
[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities,
read brain, respond with context, write brain, sync.
**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the
install paste (read `docs/GBRAIN_SKILLPACK.md`).
**If it fails:** The agent hasn't loaded the skillpack. Have it read
`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links.
---
@@ -53,8 +62,8 @@ gbrain check-update --json
**Expected:** Returns JSON with `current_version`, `latest_version`,
`update_available` (boolean). The cron `gbrain-update-check` is registered.
**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md
Section 17.
**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md)
for how to register the update-check cron.
---
@@ -88,8 +97,9 @@ find /data/brain -name '*.md' \
Some difference is normal (files added since last sync), but if page count is
less than half the file count, sync is silently skipping pages.
**If page count is way too low:** The #1 cause is an unreachable direct
connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543)
**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an
unreachable direct connection on an IPv4-only host. (PGLite brains have no
network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543)
for reads, but routes migrations, DDL, and sync transactions to a derived direct
connection (`db.<ref>.supabase.co:5432`), which is IPv6-only.
- On an IPv4-only host, reads work but sync transactions fail and silently skip
@@ -122,7 +132,7 @@ This is the real test. Edit a brain page, push, wait, search.
1. Edit a page in the brain repo (e.g., correct a fact on a person's page):
```bash
# Example: fix a line in Gustaf's page
# Example: fix a line in alice-example's page
cd /data/brain
# Make a small edit to any .md file
git add -A && git commit -m "test: verify live sync" && git push
@@ -253,19 +263,23 @@ gbrain repair-jsonb
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
**Bonus check** — frontmatter-keyed queries actually resolve:
**Bonus check** — the doctor's dedicated JSONB scan agrees:
```bash
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*'
```
If this returns rows on a brain with person pages, the JSONB path is healthy.
**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store
objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`.
---
## Quick Verification (all checks in one pass)
```bash
# 0. The one-command contract check (exits non-zero on failure)
gbrain bootstrap verify
# 1. Schema
gbrain doctor --json
+31 -46
View File
@@ -1,15 +1,24 @@
# Install
Three install paths. Pick one. Mix later if needed.
**Recommended door: the agent bootstrap.** Open your agent (Codex, Claude Code,
or any harness) in the folder that will become its home and paste the block
from the [README's install section](../README.md) — the agent fetches
`BOOTSTRAP_FOR_AGENTS.md` from the `latest-stable` tag, installs the CLI,
initializes a local PGLite brain, wires MCP, and isn't done until
`gbrain bootstrap verify` exits 0. Full contract, security posture, and
uninstall: [docs/guides/bootstrap.md](guides/bootstrap.md).
## 1. Run with an agent platform (recommended)
The paths below are the manual equivalents and deep-dive detail. Pick one.
Mix later if needed.
## 1. Run with an agent platform
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
```bash
bun install -g github:garrytan/gbrain
bun install -g github:garrytan/gbrain#latest-stable
gbrain init --pglite # 2 seconds; no server
gbrain skillpack scaffold --all # 52 skills scaffolded into your agent workspace
gbrain skillpack scaffold --all # scaffolds every bundled skill (skills/manifest.json) into your agent workspace
gbrain doctor # green checks all the way down
```
@@ -24,7 +33,7 @@ To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts
No agent platform, just shell + MCP-aware editor.
```bash
bun install -g github:garrytan/gbrain
bun install -g github:garrytan/gbrain#latest-stable
gbrain init --pglite
```
@@ -107,55 +116,31 @@ Useful for: team mounts, brain-as-a-service deployments, dev machines without di
## Verifying the install
```bash
gbrain bootstrap verify # the whole install contract; exits non-zero on failure
gbrain doctor --json # full health check
gbrain models # which AI models are configured for what
gbrain models doctor # 1-token probe per configured model
```
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). For the manual check-by-check runbook, see [docs/GBRAIN_VERIFY.md](GBRAIN_VERIFY.md).
## Troubleshooting
### PGLite crashes on macOS 26.x (Tahoe)
### PGLite crashes at startup (`RuntimeError: Aborted()`)
This crash (`RuntimeError: Aborted()` at engine startup, typically first seen
after a macOS upgrade) is **not** a macOS/WASM incompatibility. The upgrade
reboot kills gbrain mid-write and tears the data dir's write-ahead log; every
subsequent open then fails WAL replay. Recovery ladder:
This crash (typically first seen after a macOS upgrade) is **not** a
macOS/WASM incompatibility — an unclean shutdown tore the data dir's
write-ahead log, and every subsequent open fails WAL replay. The short
version of the recovery ladder:
1. **Auto-repair (default):** just run any gbrain command — gbrain detects the
abort, resets the WAL in place (data preserved; a backup of the pre-repair
state is kept next to the data dir), and continues. Then run `gbrain doctor`.
2. **Manual repair:** `gbrain pglite-repair --dry-run` to diagnose,
`gbrain pglite-repair --yes` to repair in place.
3. **Rebuild:** `gbrain reinit-pglite` (wipes and re-creates the brain from
your brain repo; embedding settings default from your config).
4. **Switch engines** — if you prefer a server database anyway, native
Homebrew PostgreSQL works great and supports multiple concurrent agents:
1. **Auto-repair (default):** run any gbrain command — gbrain detects the
abort, resets the WAL in place (data preserved, backup kept), and
continues. Then run `gbrain doctor`.
2. **Manual repair:** `gbrain pglite-repair --dry-run`, then
`gbrain pglite-repair --yes`.
3. **Rebuild:** `gbrain reinit-pglite`.
4. **Switch engines:** Supabase or native Homebrew Postgres + pgvector.
```bash
# Install PostgreSQL + pgvector
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
# Build pgvector from source (required for vector search)
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# Point gbrain at your local Postgres
cat > ~/.gbrain/config.json << 'EOF'
{
"engine": "postgres",
"database_url": "postgresql://localhost:5432/gbrain",
"schema_pack": "gbrain-base-v2"
}
EOF
# Run migrations and verify
gbrain apply-migrations --yes
gbrain doctor
```
Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend (plus multi-connection support: several agents can share one Postgres brain, which PGLite's single-process lock doesn't allow).
The full ladder — safety bounds, kill-switches, when WAL repair can't help,
and the Homebrew Postgres recipe — lives in
[docs/ENGINES.md](ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted).
+57 -11
View File
@@ -26,19 +26,21 @@ Two equivalent paths:
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
run `bun run test:e2e`, then tear it down.
- Follow the "E2E test DB lifecycle" steps in
[docs/TESTING.md](TESTING.md) to spin up the test DB, run
`bun run test:e2e`, then tear it down.
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
**Always run typecheck before pushing.** `bun test` (the bun runner)
skips TypeScript type checking — it only enforces runtime behavior.
**Always run typecheck before pushing.** Neither `bun test` (the bun runner)
nor `bun run test` gates on types — `bun run test` is just
`bash scripts/run-unit-parallel.sh` (the sharded unit runner; no typecheck,
no shell pre-checks — see the test-tier table in [docs/TESTING.md](TESTING.md)).
Three ways to actually gate on types:
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
1. `bun run verify` — runs the shell guard checks (privacy, jsonb, source-id,
progress-to-stdout, …) plus `bun run typecheck` in parallel
(`scripts/run-verify-parallel.sh`). Use this mid-branch.
2. `bun run typecheck``tsc --noEmit` standalone. Fast (~5s on this repo).
3. `bun run ci:local` — the full local CI gate from Path A.
@@ -290,7 +292,8 @@ matter" with BrainBench-style before/after table, "what this means" closer, then
Create a migration file at `skills/migrations/v[version].md` when a release
includes changes that existing users need to act on. The auto-update agent
reads these files post-upgrade (Section 17, Step 4) and executes them.
reads these files post-upgrade (see `docs/guides/upgrades-auto-update.md`)
and executes them.
**You need a migration file when:**
- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
@@ -343,8 +346,8 @@ canonical.
## Schema state tracking
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
reads this during upgrades to suggest new schema additions without re-suggesting
adopted, declined, or added custom. The auto-update agent
(`docs/guides/upgrades-auto-update.md`) reads this during upgrades to suggest new schema additions without re-suggesting
things the user already declined. The setup skill writes the initial state during
Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
@@ -373,6 +376,46 @@ the release the same, uses that version's `CHANGELOG.md` entry as the notes
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
missing), and attaches the compiled binaries.
### The `latest-stable` tag
The **final step of the release job** force-advances the `latest-stable` tag to
the release commit (`git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"`).
`latest-stable` is the single sanctioned distribution ref: the README paste
block, the `BOOTSTRAP_FOR_AGENTS.md` fetch URL, and
`bun install -g github:garrytan/gbrain#latest-stable` all reference it
permanently, so paste blocks copied into the wild never rot and there is no 404
window between VERSION landing and assets publishing.
`scripts/check-bootstrap-tag.sh` keeps the entry docs pinned to this ref.
Because it moves ONLY after binaries + provenance attestation have fully
published, a half-built release never advances it. If the tag-advance step
alone fails, re-advance by hand (a full workflow re-run would skip — the
release already exists with all assets):
```bash
git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
```
### The `publish-template` job
After the release job, a `publish-template` job force-pushes the rendered
agent-workspace template repo (the GitHub "Use this template" door,
`vars.TEMPLATE_REPO`, default `garrytan/gbrain-agent-template`) from CI only —
no human pushes it by hand, so what adopters clone is exactly what this repo
reviewed. It is guarded three ways: the release above fully published; the
vendored tree `templates/bootstrap/template-repo/` exists (skip, never fail,
if not); and the `TEMPLATE_REPO_PAT` secret is configured (skip if not).
Before pushing, it regenerates the template tree
(`bun run scripts/generate-template-repo.ts`) and byte-diffs it against the
vendored copy — a mismatch fails the job; regenerate + commit the vendored
tree (`scripts/check-bootstrap-templates.sh` runs the same diff offline in
`bun run verify`).
**`TEMPLATE_REPO_PAT` scope:** a fine-grained PAT with `contents: write` on
the template repository ONLY — no other repositories, no other permissions.
Configure it as a repo secret; when absent, template publishing is disabled
and the job skips cleanly.
Why every bump, not selective: `gbrain check-update` resolves the latest
version from `VERSION` on master, while binary self-update
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
@@ -395,6 +438,9 @@ Invariants:
history; every new 4-segment `VERSION` mints a fresh tag.
- **Permissions stay scoped.** `contents: write` lives on the release job
only; everything else runs read-only.
- **Never advance `latest-stable` on a partial release.** The tag moves only
as the final release-job step, after every asset has published. Manual
re-advances must point at a fully published `v<VERSION>` release.
## PR descriptions cover the whole branch
+11 -4
View File
@@ -11,13 +11,13 @@ Seven test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. Up-to-4-shard fan-out via `scripts/run-unit-parallel.sh` (min(CPUs, 4)), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | ~85s on a Mac dev box (3700+ tests) | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### Shell dispatch and Windows
@@ -61,7 +61,12 @@ When `bun run test` finds any failure, the wrapper:
3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 3000s; `GBRAIN_TEST_SHARD_KILL_AFTER` grace after TERM before KILL, default 30s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
If a shard hits the per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap (default 3000s — sized so the heaviest count-balanced shard finishes under 4-way contention; `GBRAIN_TEST_SHARD_KILL_AFTER` sets the grace after TERM before KILL, default 30s), the wrapper classifies the kill one of two ways:
- **EXIT-HANG → warn-pass.** If the shard's log had been silent for ≥300s at kill time AND shows zero `(fail)` markers, the shard finished all its work, leaked a handle, and never exited (a pre-existing, master-reproducible PGLite-adjacent leak — see TODOS.md "unit-shard exit hang"). The wrapper prints a `⚠️ shard N/M: EXIT-HANG ... Treating as pass-with-warning` banner, writes `EXIT-HANG (idle Ns, 0 fails) ... warn-pass` to the summary, and does NOT fail the run. Its pass counts are undercounted (bun never printed its final summary). Bun's per-test `--timeout` turns a genuinely hung TEST into a printed `(fail)` — new output — so this classification cannot mask a hung test; the residual maskable case is a file-level import hang in the very last file, which the banner keeps visible.
- **WEDGED → hard failure.** Anything else (failures present, or the log was still growing) writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log with the last 50 lines of the shard log, marks the run failed, and proceeds with other shards' results.
Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT a test failure — don't burn time bisecting it; a `WEDGED` line is.
### File taxonomy
@@ -83,6 +88,8 @@ Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-
### Test-isolation lint and helpers
**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules.
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
| Rule | What it bans | Fix |
+8
View File
@@ -1,5 +1,13 @@
# Upgrading Downstream Agents
> **Currency note:** this file is an append-only historical log and stopped
> receiving new sections after v0.36.5.0. **The canonical, maintained upgrade
> channel is `skills/migrations/v*.md`** (the agent-executed migration files
> that `gbrain upgrade` / `gbrain post-upgrade` route through), plus
> `CHANGELOG.md` for what each release changed. Use this file only to catch a
> long-diverged fork up through the versions it covers; for anything after
> v0.36.5.0, walk the migration files and CHANGELOG instead.
GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments,
agent forks of any kind) often **copy** these skill files into their own workspace and
diverge over time — adding agent-specific phases, removing irrelevant ones, tightening
File diff suppressed because one or more lines are too long
+46 -19
View File
@@ -4,14 +4,14 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
## The four strategies in concert
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at acme-example?" → pages mentioning "alice-example + retrieval" even when the user never typed "acme".
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
## Why each one alone fails
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in alice-example's portfolio" returns essays about portfolios, not company pages.
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
@@ -36,8 +36,8 @@ BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/ga
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
- Standard markdown links: `[Alice Example](wiki/people/alice-example)`
- Obsidian wikilinks: `[[wiki/people/alice-example|Alice Example]]`
- Typed-link blockquotes: `> **Convention:** see [path](path).`
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') JOIN pages ON CONFLICT DO NOTHING RETURNING 1` (free-text-safe; the prior `unnest(${arr}::text[])` form crashed on calendar/Zoom context per gbrain#1861). The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
@@ -46,7 +46,7 @@ Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`,
## ZeroEntropy as reranker: 60% top-1 reshuffle
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
ZeroEntropy's `zerank-2` is the default reranker (on for the `balanced` and `tokenmax` mode bundles, off for `conservative`). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
@@ -62,8 +62,9 @@ The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `Sea
## Named-thing retrieval (per-page pool + title + alias + evidence)
A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than
embedding proximity. Four layers, added after the incident in
A brain organized around *chosen names* (project codenames, place nicknames —
say a project named "Helios" whose page is also known as "the Sun Room") needs
more than embedding proximity. Four layers, added after the incident in
[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md):
- **Per-page max-pool**`searchVector` (both engines) collapses chunk-grain
@@ -79,7 +80,7 @@ embedding proximity. Four layers, added after the incident in
`page_aliases` table (separate from the `slug_aliases` wikilink redirect) and
consulted at query time: a full normalized-query match injects/boosts the
canonical page (`applyAliasHop`). The only layer that bridges true synonyms
with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill
with zero surface overlap ("the Sun Room" → the Helios page). Backfill
existing pages with `gbrain reindex --aliases`.
- **Evidence contract** — every result carries `evidence`
(`alias_hit | exact_title_match | high_vector_match | keyword_exact |
@@ -107,7 +108,7 @@ specific miss with `gbrain search diagnose "<q>" --target <slug>`.
## Intent-aware query rewriting
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
`src/core/search/query-intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
@@ -127,35 +128,61 @@ Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `con
The full pipeline for a `query` op:
```
intent classify
intent classify (query-intent.ts — deterministic, no LLM)
expansion (if enabled)
expansion (if enabled — tokenmax only by default)
hybrid search:
├── vector (HNSW on chunk embeddings)
hybrid recall + fusion:
├── vector (HNSW on chunk embeddings, per-page max-pool)
├── keyword (BM25 via tsvector)
├── relational (v0.42.34.0: typed-edge recall arm — relational queries only)
├── title-phrase arm
├── relational (typed-edge recall arm — relational queries only)
├── source-aware re-rank (CASE in SQL)
└── RRF fusion → top 30
└── RRF fusion → cosine re-score → post-fusion boosts
(backlink / salience / recency / graph signals / exact-match)
graph augment (typed-edge traversal from any seed)
graph augment (optional two-pass structural expansion — walkDepth > 0)
reranker (zerank-2 cross-encoder, top 30 → reordered)
deduplication (4-layer: per-page cap, Jaccard, type diversity)
token-budget enforcement (per mode bundle)
reranker (zerank-2 cross-encoder — balanced/tokenmax; fail-open)
deduplication (same slug, different chunks → keep best)
alias hop (exact alias match injects/boosts the canonical page)
evidence stamp → adaptive return (opt-in) → autocut (reranked modes)
limit slice → token-budget enforcement (per mode bundle)
results
```
The stage order is pinned by `hybridSearch` in `src/core/search/hybrid.ts`:
dedup runs BEFORE the reranker (so the reranker sees a diverse candidate pool,
capped by its own `topNIn`), the alias hop runs AFTER the reranker (so a query
that is a page's declared name reliably surfaces that page regardless of how
the reranker scored body chunks), and the token budget is enforced last, on
the final slice.
### Autocut: score-discontinuity result-sizing
Default-on for `balanced` and `tokenmax` (off for `conservative`, which has no
reranker and therefore no trustworthy cliff signal). `applyAutocut`
(`src/core/search/autocut.ts`) cuts the ranked set at the largest
cross-encoder rerank-score cliff, before the limit slice, first page only.
Never-empty failsafe (`minKeep`), no-op when fewer than 2 results carry a
finite rerank score (covers the fail-open reranker path), and alias-hop exact
matches are preserved through the cut. Knobs: per-call `SearchOpts.autocut`
`search.autocut` / `search.autocut_jump` config → mode bundle.
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
## How to verify on your own brain
+16 -9
View File
@@ -19,18 +19,18 @@ need to understand both of them, or queries misroute silently.
A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase.
Each brain has:
- Its own `pages` table, `chunks` table, `embeddings`, etc.
- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2).
- Its own OAuth surface if served over HTTP MCP.
- Its own separate lifecycle, backup, access control.
Brains are enumerated by:
- **host** — your default brain, configured in `~/.gbrain/config.json`.
- **mounts** — additional brains registered in `~/.gbrain/mounts.json` via
`gbrain mounts add <id>` (v0.19+).
`gbrain mounts add <id>`.
Routing: `--brain <id>`, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or
longest-path match against registered mount paths. Falls back to `host`.
### Sources (the repo axis, v0.18.0+)
### Sources (the repo axis)
A **source** is a named content repo *inside* one brain. Every `pages` row
carries a `source_id`. Slugs are unique per source, not globally.
@@ -142,7 +142,7 @@ Use this topology when:
You're senior enough to sit across multiple teams. You maintain your personal
brain (with N sources inside) AND mount several work team brains. Each team
brain is itself a multi-source brain in the v0.18.0 sense — organized
brain is itself a multi-source brain — organized
internally however the team owner chose.
```
@@ -181,7 +181,7 @@ Use this topology when:
- You need latent-space federation (agent decides when to query across
brains), not SQL federation.
Cross-brain queries are **not deterministic** in v0.19. The agent sees the
Cross-brain queries are **not deterministic**. The agent sees the
brain list and re-queries as needed. That's the feature — it keeps debugging
sane and access control clean.
@@ -202,6 +202,13 @@ WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)?
Both axes follow the same layered pattern on purpose. If you know one, you
know the other.
One addition on the source axis for remote (MCP/OAuth) callers: a client
registered with federated reads carries `ctx.auth.allowedSources` — an
ARRAY of readable sources that takes precedence over the scalar
`ctx.sourceId` on every read path (`sourceScopeOpts(ctx)` in the
operations layer). Local CLI callers never set it; the scalar chain above
is the whole story for them.
---
## For agents reading this
@@ -236,7 +243,7 @@ know the other.
## Further reading
- v0.18.0 CHANGELOG — introduced `sources` primitive.
- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`.
- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain
publisher, not just the subscriber.
- [`topologies.md`](./topologies.md) — where the DB lives (operator recipes
for each deployment shape).
- `skills/conventions/brain-routing.md` — the agent-facing decision table.
- `CHANGELOG.md` — release history for the `sources` and `mounts` primitives.
@@ -10,14 +10,16 @@
> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80
> during successive master merges — v0.37.0.0's autonomous-remediation wave
> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79.
> - **Follow-up minor** (forthcoming): falsifiability + category extraction at
> `propose_takes`, SQL-side grade gate, per-category calibration scorecards,
> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation
> against the v0.36.1.0 fixtures.
> - **Follow-up minor — NEVER IMPLEMENTED.** The falsifiability + category
> extraction at `propose_takes`, SQL-side grade gate, per-category
> calibration scorecards, and pg_trgm-based proposal dedup described in the
> sections below remain UNSHIPPED design. Do not read §§14 as current
> behavior; only the `unresolvable` hotfix above landed.
>
> Preserved here per the hotfix plan's PR #1191 close protocol so the
> production context (96K-page brain, 6.8% falsifiability rate, category
> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation.
> production context (falsifiability rate + category breakdown observed on a
> large real brain) doesn't get lost in the CHANGELOG → release-notes
> condensation.
## Problem
@@ -68,11 +68,13 @@ existing.
## Migration shape
```ts
// src/core/migrate.ts — append after the v80 entry
// src/core/migrate.ts — append after the CURRENT last entry in the
// MIGRATIONS array (take the next unused version number at implementation
// time; the numbers below are placeholders, not a reserved slot)
const migrations = [
// ...existing v1-v80...
// ...existing entries...
{
version: 81,
version: NEXT_VERSION, // next unused number in the MIGRATIONS array
name: 'frontmatter_scan_state',
sql: `
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
@@ -194,7 +196,7 @@ stale data as authoritative.
```
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
Design lives at docs/architecture/frontmatter-scan-incremental.md.
Schema migration v81 + sync-side UPSERT + incremental scan command
New schema migration + sync-side UPSERT + incremental scan command
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
table + writes first; flip the reader one release later.
```
+18 -90
View File
@@ -1,105 +1,33 @@
# GBrain Infrastructure Layer
# GBrain Infrastructure Layer (orientation pointer)
The shared foundation that all skills, recipes, and integrations build on.
This page is a router — the detailed, current-state references live in the
docs below (this file once carried its own copies of the pipeline and schema;
those rotted, so each concept now has exactly one home).
## Data Pipeline
## Where things live
```
INPUT (markdown files, git repo)
FILE RESOLUTION (local → .redirect → .supabase → error)
MARKDOWN PARSER (gray-matter frontmatter + body)
→ compiled_truth + timeline separation
CONTENT HASH (SHA-256 idempotency check — skip if unchanged)
CHUNKING (3 strategies, configurable)
├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy
├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing
└── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates
EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions)
→ batch 100, exponential backoff, non-fatal if fails
DATABASE TRANSACTION (atomic: page + chunks + tags + version)
SEARCH (hybrid, available immediately)
```
## Search Architecture
GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search:
```
User Query
EXPANSION (optional: Claude Haiku generates 2 alternative phrasings)
├── VECTOR SEARCH (pgvector HNSW, cosine distance)
│ → 2x limit results per query variant
└── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank)
→ 2x limit results
RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly)
4-LAYER DEDUP
├── Best 3 chunks per page (source dedup)
├── Jaccard similarity > 0.85 (text dedup)
├── No type exceeds 60% (diversity)
└── Max 2 chunks per page (page cap)
TOP N RESULTS (default 20)
```
## Key Components
| File | Purpose |
|------|---------|
| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) |
| `src/core/postgres-engine.ts` | Postgres + pgvector implementation |
| `src/core/import-file.ts` | importFromFile + importFromContent pipeline |
| `src/core/sync.ts` | Git-based incremental change detection |
| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing |
| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff |
| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) |
| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection |
| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking |
| `src/core/search/hybrid.ts` | RRF merge of vector + keyword |
| `src/core/search/dedup.ts` | 4-layer result deduplication |
| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku |
| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) |
| `src/core/operations.ts` | Contract-first operation definitions (31 ops) |
| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) |
## Schema Overview
10 tables in Postgres:
- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB)
- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline)
- **links** — typed edges (knows, works_at, invested_in, founded, etc.)
- **tags** — many-to-many page tagging
- **timeline_entries** — structured events (date, source, summary, detail)
- **page_versions** — snapshot history for diff/revert
- **raw_data** — sidecar JSON from external APIs (preserves provenance)
- **files** — binary attachments in storage backend
- **ingest_log** — audit trail of import operations
- **config** — brain-level settings (version, embedding model, chunk strategy)
Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C).
Vector search uses HNSW index with cosine distance on content_chunks.embedding.
| Topic | Home |
|---|---|
| Ingest pipeline (file resolution → frontmatter parse → content-hash idempotency → chunking → embedding → atomic write) | per-file entries in [`KEY_FILES.md`](./KEY_FILES.md): `src/core/import-file.ts`, `src/core/sync.ts`, `src/core/markdown.ts`, `src/core/embedding.ts`, `src/core/chunkers/*` |
| Chunking strategies (recursive / semantic / LLM-guided) | `src/core/chunkers/{recursive,semantic,llm}.ts` entries in [`KEY_FILES.md`](./KEY_FILES.md) |
| Search pipeline (hybrid RRF, graph, reranker, autocut, dedup, budgets) | [`RETRIEVAL.md`](./RETRIEVAL.md) |
| Search modes + cost knobs | `docs/guides/search-modes.md` + the CLAUDE.md Search Mode table |
| Per-file index of `src/` (what each file does + its invariants) | [`KEY_FILES.md`](./KEY_FILES.md) |
| Schema DDL | the `MIGRATIONS` array in `src/core/migrate.ts` (source of truth) + `src/schema.sql`; per-table classification in [`system-of-record.md`](./system-of-record.md) |
| Engines (PGLite vs Postgres, parity rules) | `docs/ENGINES.md` + the engine entries in [`KEY_FILES.md`](./KEY_FILES.md) |
| Operations contract (CLI + MCP generated from one source) | `src/core/operations.ts` (100+ operations; run `gbrain --tools-json` for the live list) |
| Brains vs sources (which database vs which repo inside it) | [`brains-and-sources.md`](./brains-and-sources.md) |
## The Thin Harness Principle
GBrain is the deterministic layer. Skills and recipes are the latent space layer.
GBrain is the deterministic layer. Skills and recipes are the latent-space layer.
See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full
architecture philosophy.
- **GBrain CLI** = thin harness (same input → same output)
- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills
- **Skills** (the bundled set routed by `skills/RESOLVER.md`) = fat skills
- **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure
The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work.
+25 -29
View File
@@ -1,4 +1,4 @@
# Lens packs (v0.41.2.0)
# Lens packs
Four bundled schema packs that turn the gbrain dream cycle into a multi-lens
brain. Activate one with `gbrain config set schema_pack <name>` and the cycle
@@ -7,7 +7,7 @@ picks up the pack's declared phases on the next `gbrain dream` run.
## The four packs
```
gbrain-base (shipped v0.38)
gbrain-base
│ extends
┌──────────────┼──────────────────────┐
@@ -60,37 +60,33 @@ conviction so high-stakes misses cost more).
### gbrain-engineer
Bridge-only pack. Declares `learning` page type + reuses base `code`.
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource
(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
each JSONL line as a `learning` page when this pack is active. Three
calibration domains: `architecture_calls` (scalar_brier),
`effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier).
Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred
to v0.42+ — they'll ship when a real user authors the first one (D8).
Speculative ADR/postmortem/refactor_thesis/tech_debt types are
deferred — they'll ship when a real user authors the first one.
### gbrain-everything
Meta-pack stacking creator + investor + engineer via the v0.38
Meta-pack stacking creator + investor + engineer via the
`extends` + `borrow_from` chain. Single-active-pack constraint
preserved — this IS the active pack; the registry walks extends +
borrow to materialize the merged view.
**Merge contract (T20 / #1749).** `resolvePack` merges parent → child
(child-wins) for the six ingest/query-shaping fields: `page_types`,
`link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`,
and `takes_kinds` (unioned — a child cannot narrow it). `phases` and
`calibration_domains` are **NOT** inherited: they gate cycle execution,
so each pack must declare its own participation explicitly. That is why
`gbrain-everything` re-declares all its phases and all 7
`calibration_domains` — inheritance does not carry them.
**Merge contract.** The full `extends` + `borrow_from` merge rules live in
[`schema-packs.md` § Merge contract](./schema-packs.md#merge-contract-extends--borrow_from).
The one rule that matters here: `phases` and `calibration_domains` are
**NOT** inherited (they gate cycle execution, so each pack must declare
its own participation explicitly) — which is why `gbrain-everything`
re-declares all its phases and all 7 `calibration_domains`.
Activate via `gbrain config set schema_pack gbrain-everything` and
calibration_profile produces all 7 domain scorecards in one JSONB.
## Calibration profile widening (T10)
## Calibration profile domains
Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a
`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared
Each declared
domain produces a `{n, brier, accuracy, aggregator, page_types,
extras}` entry. Four aggregator algorithms (closed enum):
@@ -109,20 +105,20 @@ Domain names are OPEN (third-party packs can declare new domain labels
without a gbrain release). Aggregator algorithms are CLOSED (safe SQL
stays in code, validated at pack-load).
## take_domain_assignments table (T1)
## take_domain_assignments table
New JOIN table (migration v94):
JOIN table (migration v94):
`take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT,
source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id,
domain))`. Multi-domain assignment honest — a take about "Sequoia's
investment in Anthropic" can land in BOTH `deal_success` AND
domain))`. Multi-domain assignment honest — a take about "fund-a's
investment in acme-example" can land in BOTH `deal_success` AND
`market_call` rather than being force-bucketed.
## What this enables for the user
- **Atoms + concepts ship in the binary.** Your OpenClaw's parallel
atom-pipeline-coordinator + atom-backfill-coordinator + concept-
synthesis crons can retire (T12 follow-up). One `gbrain dream` cron
synthesis crons can retire. One `gbrain dream` cron
covers everything.
- **gstack learnings reach gbrain.** Engineer-pack-active brains
surface every gstack-logged learning as a queryable page within
@@ -131,22 +127,22 @@ investment in Anthropic" can land in BOTH `deal_success` AND
often you're wrong on deals AND market calls AND architecture
AND effort estimates in one `gbrain calibration --json` call.
- **Lossless OpenClaw migration.** The `markdown-greenfield`
importer (T7, mode='migration') re-ingests existing OpenClaw
importer (mode='migration') re-ingests existing OpenClaw
pages with permanent slug-keyed idempotency + per-row JSONL audit
+ the `imported_from` marker so extract_atoms + synthesize_concepts
don't re-extract already-atomized material.
## v0.41.2.1 follow-ups (filed in plan)
## Known gaps / deferred follow-ups
- Per-page-type `frontmatter_validators` on PageTypeSchema so the
atom_type enum (currently hardcoded in extract_atoms.ts) reads from
the active pack manifest at runtime per D11.
the active pack manifest at runtime.
- 3-check quality gate (truism / punchline / entity-page reject) as
a multi-pass extract_atoms refinement.
- Embedding-similarity dedup in synthesize_concepts (currently
exact-string concept ref match only).
- Voice gate integration for T1 Canon narratives.
- Voice gate integration for concept narratives.
- op_checkpoint resumability for cross-cycle continuation in both
phases.
- Parity-baseline eval gates against your OpenClaw's existing 13K atoms
+ 11K concepts on a 500-page sample subset.
- Parity-baseline eval gates against a pre-existing downstream
atom/concept corpus on a sample subset.
+11 -11
View File
@@ -127,10 +127,10 @@ candidate ≠ the active pack name, loads the manifest via
migration_from.version)`. Returns matching packs sorted by version
descending.
v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed
packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the
filesystem-scan cost needs the cache invalidation strategy from
`registry.ts`).
Successor detection covers bundled packs only. Future work: enumerate
user-installed packs at `~/.gbrain/schema-packs/*/pack.yaml` (deferred
because the filesystem-scan cost needs the cache invalidation strategy
from `registry.ts`).
## The manual_only apply policy
@@ -173,8 +173,8 @@ migration_from:
version: "1.x"
page_types:
# Inherit gbrain-base-v2's 15 types here (or use extends to merge
# automatically once v0.43+ extends-chain composition lands)
# Inherit gbrain-base-v2's 15 types here (or declare `extends:
# gbrain-base-v2` and let the merge contract in schema-packs.md merge them)
- { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true }
- { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true }
# ... all 13 other v2 canonicals ...
@@ -222,17 +222,17 @@ Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl`
identities (before + after), per-phase counts (would_apply + applied),
warnings, completion timestamp. Privacy: page slugs are NOT logged in
bulk (only the per-rule sample_slugs[≤10]); for forensic debugging
add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired.
## What's NOT yet supported
- Subprocess sandbox for the publish-gate (v0.43+ TODO)
- Subprocess sandbox for the publish-gate
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
- LLM-assisted mapping_rules codegen from production data (a proposed
`gbrain schema detect-mappings`)
## Reference
@@ -242,6 +242,6 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
- Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable`
- Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS`
- Handler: `src/core/schema-pack/unify-types-handler.ts`
- Migration: `src/core/migrate.ts:105` (slug_aliases table)
- Migration: the `slug_aliases` entry in `src/core/migrate.ts`'s `MIGRATIONS` array
- Type taxonomy doc: `docs/architecture/type-taxonomy.md`
- Skill: `skills/schema-unify/SKILL.md`
+54 -49
View File
@@ -7,20 +7,25 @@ paths, and which link verbs connect what to what. The schema pack is the
querying, or routing experts. It is the single source of truth for
"what's in your brain."
The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the
user-facing reference; for implementation details see
`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in
`src/core/schema-pack/`.
This doc is the user-facing reference; for implementation details see
`docs/designs/V038_SCHEMA_PACKS.md` (the original design) and the engine
layer in `src/core/schema-pack/`.
## What ships in the box
Two bundled packs:
Seven bundled packs (`src/core/schema-pack/base/`):
- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior
byte-for-byte. Existing brains see zero behavior change after upgrade.
Covers: person, company, deal, meeting, project, place, concept, writing,
analysis, guide, hardware, architecture, etc. (the original
`ALL_PAGE_TYPES` list).
- **`gbrain-base-v2`** — the 15-type canonical taxonomy. Fresh installs
(`gbrain init`) activate this by default. See
[`type-taxonomy.md`](./type-taxonomy.md) for the full type list and the
upgrade path from `gbrain-base`.
- **`gbrain-base`** — the original hardcoded behavior, byte-for-byte
(person, company, deal, meeting, project, place, concept, writing,
analysis, guide, hardware, architecture, etc. — the original
`ALL_PAGE_TYPES` list). Still the resolution-chain fallback (tier 7)
for brains with no pack configured anywhere, so pre-existing brains see
zero behavior change until they opt in to something newer.
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional
directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal,
@@ -32,12 +37,17 @@ Two bundled packs:
gbrain schema use gbrain-recommended
```
- **`gbrain-creator`**, **`gbrain-investor`**, **`gbrain-engineer`**,
**`gbrain-everything`** — the lens packs, which add cycle phases and
calibration domains on top of the base taxonomy. See
[`lens-packs.md`](./lens-packs.md).
Plus user-installed packs at `~/.gbrain/schema-packs/<name>/pack.yaml`
that you author with `gbrain schema init` or `gbrain schema fork`.
## CLI surface
Five inspection verbs (shipped in v0.38):
Inspection verbs:
```bash
gbrain schema active # show resolved pack + which tier set it
@@ -47,7 +57,7 @@ gbrain schema validate # validate a manifest's shape
gbrain schema use <pack> # activate a pack (writes ~/.gbrain/config.json)
```
Eight authoring + discovery verbs (shipped in v0.39):
Authoring + discovery verbs:
```bash
gbrain schema detect # propose types matching brain shape
@@ -62,12 +72,12 @@ gbrain schema graph # ASCII type listing (experimental)
gbrain schema lint # flag duplicates + missing prefixes
gbrain schema explain <type> # plain-English type description (experimental)
gbrain schema downgrade --to <p> # restore previous pack (recovery)
gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry)
gbrain schema usage --since 30d # per-verb invocation counts (telemetry)
```
The verbs marked `experimental` are demand-gated per D14: their usage is
tracked via T15's schema-events audit, and v0.40+ retro decides whether
to deprecate any that stay <5% usage.
The verbs marked `experimental` are demand-gated: usage is tracked via the
schema-events audit (`gbrain schema usage`), which informs whether
rarely-used verbs get deprecated.
## Resolution chain (7 tiers)
@@ -78,10 +88,10 @@ this chain top-down. First match wins.
|------|--------|-------|
| 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. |
| 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. |
| 3 | Per-source DB config key `schema_pack:source:<id>` | New in v0.38. |
| 3 | Per-source DB config key `schema_pack:source:<id>` | |
| 4 | Brain-wide DB config key `schema_pack` | |
| 5 | `gbrain.yml schema:` section | Repo-checked. |
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. |
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` (and `gbrain init`, which sets `gbrain-base-v2`) writes. |
| 7 | Default: `gbrain-base` | Always present. |
## How the agent uses the active pack
@@ -97,18 +107,18 @@ Every read + write path consults the active pack at runtime:
- **`extract_facts`** runs only on `extractable: true` types.
- **`enrichment-service`** routes person/company enrichment based on the
pack's primitive declarations.
- **Search hybrid cache** (`knobsHash`) folds in pack name + version
(v0.39 T21). A cache row written under pack A is unreachable when pack
- **Search hybrid cache** (`knobsHash`) folds in pack name + version.
A cache row written under pack A is unreachable when pack
B is active. Cross-pack contamination is structurally impossible.
## The magical moment (T2-T4 + T10)
## The magical moment
Persona A (Notion refugee) installs gbrain, imports her exports, and the
brain looks unfamiliar — the default `gbrain-base` pack expects
`people/`, `companies/`, etc., but her files live under `Projects/`,
`Reading/`, `Daily Notes/`. The friction signal fires in two places:
1. **Import warn (T7):** the end of `gbrain import` prints
1. **Import warn:** the end of `gbrain import` prints
`[schema] X of Y pages (Z%) have no type matching the active schema
pack. Run gbrain schema detect to propose a pack matching your
content shape.`
@@ -124,7 +134,7 @@ gbrain schema review-candidates # human gate on promotion
gbrain schema review-candidates --apply Projects/ # accept
```
The agent (via the new EIIRP skill) automates phases 1-3 of this for any
The agent (via the EIIRP skill, `skills/eiirp/SKILL.md`) automates phases 1-3 of this for any
significant work session. The brain's schema becomes a living artifact
the agent maintains, not a hardcoded ceremony the user authors.
@@ -172,9 +182,10 @@ filing_rules: []
## Merge contract (`extends` + `borrow_from`)
This section is the single home for the merge rules (other docs link here).
`resolvePack` composes a pack against its `extends` chain (and any
`borrow_from` targets) into the `resolved.manifest` every consumer reads
(T20 / #1749). The rules:
`borrow_from` targets) into the `resolved.manifest` every consumer reads.
The rules:
- **Six fields inherit, child-wins:** `page_types`, `link_types`,
`frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`.
@@ -200,9 +211,8 @@ filing_rules: []
## Recovery + revert
The single-PR cathedral is hard to revert atomically. Per codex finding
#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore
the active-pack config field:
A pack activation is config, not code, so reverting code alone doesn't
undo it. `gbrain schema downgrade` restores the active-pack config field:
```bash
gbrain schema downgrade --to gbrain-base
@@ -214,19 +224,19 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain purge-deleted --older-than 0h`drops
v0.39-typed pages that no longer have a matching type in the active
3. (Optional) `gbrain pages purge-deleted --older-than 0h`hard-deletes
soft-deleted pages that no longer have a matching type in the active
pack.
The cache + eval rows that pack-aware code wrote are isolated by the
`knobsHash` pack-folding (T21) — they become unreachable under the
`knobsHash` pack-folding — they become unreachable under the
restored pack so no eviction is needed.
## Distribution
`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as
`.gbrain-skillpack` tarballs (T14 artifact abstraction). The
discriminator is `api_version` in the manifest:
`.gbrain-schema` tarballs ride the same distribution pipeline as
`.gbrain-skillpack` tarballs. The discriminator is `api_version` in the
manifest:
- `gbrain-schema-pack-v1` → schemapack
- `gbrain-skillpack-v1` → skillpack
@@ -237,22 +247,17 @@ respectively.
Publication to the public registries (`garrytan/gbrain-schema-registry`,
`garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR
workflow as v0.37 skillpack publishing.
workflow as skillpack publishing.
## What's deferred to v0.40+
## Known limits / deferred work
- **Per-source pack federation across mounts.** A query crossing multiple
sources currently rejects with `permission_denied` when those sources
have divergent active packs (T19 + codex finding #2). The v0.40+ work
computes a true per-source closure via the existing
`buildSourceClosureCte` engine surface.
- **`extends` chain semver compatibility checks** between pack versions.
- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack
can declare "I work best with these primitives present in your pack."
- **Live schema migration helpers** — when you add a type, auto-suggest
backfill of existing pages.
- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the
full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+
retro reads T23 usage telemetry to decide which to deprecate.
sources rejects with `permission_denied` when those sources have
divergent active packs (`src/core/schema-pack/op-trust-gate.ts`). A true
per-source closure via the existing `buildSourceClosureCte` engine
surface remains future work.
- **Pack version upgrades** (e.g. `gbrain-base``gbrain-base-v2`) are
handled by the successor-detection + unify-types mechanism — see
[`pack-upgrade-mechanism.md`](./pack-upgrade-mechanism.md).
See `TODOS.md` v0.40+ section for the full deferred list.
The live deferred list is in `TODOS.md`.
@@ -52,3 +52,13 @@ gbrain sync --no-schema-pack --no-pull --no-embed --yes
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
The manual diagnosis above has an automated cousin: the progress-aware stall
watchdog. If the import drain makes no forward progress for
`GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900; keyed on file-import
progress, not the lock heartbeat), the run aborts with
`reason: 'stall_timeout'` and releases the per-source lock so the next
`gbrain sync` resumes from the checkpoint. It fires BETWEEN files — a hang
inside one file's import runs until the wall-clock hard deadline. `0`
disables it. The full sync-resumability knob table lives in CLAUDE.md
("Sync resumability + lock tuning").
+1 -3
View File
@@ -88,6 +88,7 @@ the repo. The architectural rule still holds — these aren't
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
| `op_checkpoint_paths` | Sync-resume checkpoint. Append-only progress banking; a completed sync makes it irrelevant. |
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
A new derived table that holds user-knowledge MUST land FS-first.
@@ -189,9 +190,6 @@ reconciler / migration layer without the explicit allow-list comment.
## Related
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
and round 2 finds)
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
- `scripts/check-system-of-record.sh` — the CI gate that enforces
+38 -43
View File
@@ -3,68 +3,63 @@
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
only; release history lives in `CHANGELOG.md` + git.
`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local
brain content, just an OAuth client pointing at a remote `gbrain serve --http`.
v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25
silently fell through to `connectEngine()` and opened the empty local PGLite,
returning "No results." against a populated remote brain. v0.31.1 fixes the
silent-empty-results bug class for every operation surface.
`gbrain init --mcp-only` sets up a thin-client install: no local brain content,
just an OAuth client pointing at a remote `gbrain serve --http`. Every operation
surface routes through the remote brain — a thin-client install never opens the
empty local PGLite, so a populated remote brain can't silently return
"No results." Local-only commands refuse with a pinpoint hint instead of
falling through.
Key files:
Key files (per-file detail lives in each file's `KEY_FILES.md` entry; this doc
carries the routing-seam picture):
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (no
parallel `src/core/thin-client/` module; routing is a ~80-line conditional
in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine`
so thin-client installs never open the empty PGLite. localOnly ops on
thin-client refuse via `refuseThinClient` (with pinpoint hint table
`THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort`
`THIN_CLIENT_REFUSE_HINTS`, which covers the full DB-bound command surface —
sync, embed, extract, migrate, enrich, dream, jobs, sources, pages, files,
eval, code-*, and more). Banner via `printIdentityBannerBestEffort`
before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`,
non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason`
for canned, actionable error messages. ENG-2 renderer parity: local-engine
for canned, actionable error messages. Renderer parity: the local-engine
path runs `JSON.parse(JSON.stringify(result))` so renderers see the same
shape on both paths (kills Date/bigint/Buffer drift class).
- `src/core/mcp-client.ts``callRemoteTool(config, toolName, args, opts)`.
Hardened in v0.31.1 (CDX-4): all transport errors normalized to
`RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions
{timeoutMs, signal}`; `buildAbortController` composes external signal with
timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind`
('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code`
field carrying server-supplied error codes (e.g. `missing_scope`).
`extractToolErrorCode` parses JSON envelopes first, falls back to substring
detection for legacy server messages. `unpackToolResult<T>(res)` unchanged
(parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape.
- `src/core/cli-options.ts``parseGlobalFlags` adds `--timeout=Ns` (accepts
`30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s
for most ops, 180s for `think`). `parseTimeout(s)` exported helper.
- `src/core/doctor-remote.ts``gbrain remote doctor` adds the
`oauth_client_scopes_probe` check (CDX-5). Probes the read tier via
`get_brain_identity` and admin tier via `get_health`; reports per-tier
shape on both paths (kills the Date/bigint/Buffer drift class).
- `src/core/mcp-client.ts``callRemoteTool(config, toolName, args, opts)`,
the transport under the routing seam. All transport errors normalize to
`RemoteMcpError` via the `toRemoteMcpError` funnel, with a stable
`RemoteMcpErrorReason` union the dispatcher's `never` switch keys off.
Full symbol-level detail: the `src/core/mcp-client.ts` entry in
[`KEY_FILES.md`](./KEY_FILES.md).
- `src/core/cli-options.ts``parseGlobalFlags` supports `--timeout=Ns`
(accepts `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command
default (30s for most ops, 180s for `think`). `parseTimeout(s)` exported
helper.
- `src/core/doctor-remote.ts``gbrain remote doctor` includes the
`oauth_client_scopes_probe` check. Probes the read tier via
`get_brain_identity` and the admin tier via `get_health`; reports per-tier
status with pinpoint remediation when admin is missing. `buildScopeCheck`
+ `ScopeProbeResult` exported for test access. Skippable via
`GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC
initialize level only (MCP SDK Client hangs on shape mismatch).
- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries.
- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller).
- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality``'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error.
- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).
- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex).
- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed.
- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on).
- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command).
- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows.
- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)``date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query.
- `src/core/operations.ts``get_brain_identity` op (read scope, no params,
banner-only): cheap counter packet `{version, engine, page_count,
chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses
`engine.getStats()`; banner's 60s client-side TTL bounds frequency to
≤1/60s per CLI process (well below the Fly.io health-check cadence that
motivated the original `getStats` cost warning).
`engine.getStats()`; the banner's 60s client-side TTL bounds frequency to
≤1/60s per CLI process.
- `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command
thin-client routing branches. These commands bypass the operation-layer
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
to op params. `think` is a special case: the server's `think` op is
read-scoped for OAuth/MCP and intentionally disables `--save`/`--take` for
remote callers in its trust-boundary gate; thin-client `think` warns loudly
when those flags are set.
remote callers (the `safeSave`/`safeTake` trust-boundary gate in the `think`
handler in `operations.ts`); thin-client `think` warns loudly when those
flags are set.
Cross-modal search files (image query, SSRF-guarded image loading, spend
tracking, multimodal reindex) are indexed per-file in
[`KEY_FILES.md`](./KEY_FILES.md) and described behaviorally in
[`RETRIEVAL.md`](./RETRIEVAL.md) — they are not part of the thin-client
routing seam.
+10 -5
View File
@@ -108,11 +108,13 @@ instead of a local DB connection:
}
```
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
`integrity`, `serve`) on a thin-client install with a clear error pointing
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
(OAuth discovery, token round-trip, MCP smoke).
The CLI dispatch guard refuses every DB-bound command (`sync`, `embed`,
`extract`, `migrate`, `serve`, `enrich`, `jobs`, `sources`, `pages`,
`files`, `eval`, and the rest of the local-only surface — the full hint
table is `THIN_CLIENT_REFUSE_HINTS` in `src/cli.ts`) on a thin-client
install with a clear error pointing at the remote host. `gbrain doctor`
runs a dedicated thin-client check set (OAuth discovery, token round-trip,
MCP smoke). See [`thin-client.md`](./thin-client.md) for the routing seam.
### Setup
@@ -394,6 +396,9 @@ simultaneously — that's by design.
## See also
- `docs/guides/bootstrap.md``gbrain bootstrap`, the paved-road paste-in
install for Topology 1 with a desktop coding agent (interview, hooks,
MCP registration, verify).
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
vs sources axes).
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
+8 -7
View File
@@ -1,7 +1,7 @@
# Type Taxonomy (v0.41.22: gbrain-base-v2)
# Type Taxonomy (gbrain-base-v2)
> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor
> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs
> The 14-canonical-type DRY/MECE taxonomy. Predecessor
> `gbrain-base` (24 types) stays bundled for back-compat; fresh installs
> default to `gbrain-base-v2`.
## Why
@@ -79,7 +79,7 @@ gbrain jobs submit unify-types \ # PROTECTED + manual_only
--params '{"target_pack":"gbrain-base-v2","apply":true}'
# omit "apply":true → dry-run (default)
Handler runs 4 phases:
Handler runs 8 phases:
┌─────────────────────────────────────┐
│ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL)
├─────────────────────────────────────┤
@@ -165,8 +165,10 @@ explicitly disambiguated this as canonical, so it should outrank fuzzy
matches that hit aliases by accident."
`SearchResult.alias_resolved_boost` is stamped on touched results for
`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to
invalidate pre-v0.42 cache rows that don't reflect the new stage.
`--explain` formatter visibility. The stage participates in the search
cache key (`KNOBS_HASH_VERSION` in `src/core/search/mode.ts` is the
single source of truth for the current cache-key version), so cache rows
written before the stage existed are unreachable.
## Reference
@@ -176,4 +178,3 @@ invalidate pre-v0.42 cache rows that don't reflect the new stage.
- Migration handler: `src/core/schema-pack/unify-types-handler.ts`
- Onboard checks: `src/core/onboard/checks.ts`
- Skill: `skills/schema-unify/SKILL.md`
- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md`
+21 -19
View File
@@ -1,4 +1,4 @@
# gbrain eval suspected-contradictions (v0.32.6)
# gbrain eval suspected-contradictions
The contradiction probe samples retrieval results, asks an LLM judge whether
any pair contradicts on a factual claim relevant to the user's query, and
@@ -142,25 +142,27 @@ pay near-zero on re-runs (until you bump PROMPT_VERSION).
gate makes accidental private-data commits hard, but the operator MUST
inspect every redaction before commit.
## Temporal axis
The judge distinguishes real contradictions from legitimate change-over-time.
The verdict enum has six members (`no_contradiction | contradiction |
temporal_supersession | temporal_regression | temporal_evolution |
negation_artifact`), and `pages.effective_date` is threaded into the judge
prompt so the probe doesn't cry wolf on facts that simply changed.
The trajectory substrate builds on the same signal:
`gbrain eval trajectory <entity>` shows the chronological typed-claim
history with regressions flagged inline; `gbrain founder scorecard
<entity>` rolls up four signals (accuracy, consistency, growth
direction, red flags) into a stable JSON contract. MCP op
`find_trajectory` (read scope, visibility-filtered for remote callers)
exposes the same data to agents. The probe's `temporal_supersession`
verdict and the consolidate phase's `valid_until` writeback both
preserve the `auto-supersession.ts` "NEVER auto-applies" invariant
— the probe only emits paste-ready commands; only `consolidate`
writes `valid_until` (a grep guard pins this).
## See also
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
+ trend-tracking workflow.
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
six-member verdict enum (`no_contradiction | contradiction |
temporal_supersession | temporal_regression | temporal_evolution |
negation_artifact`) and threaded `pages.effective_date` into the judge
prompt so the probe stops crying wolf on legitimate change-over-time.
v0.35.7 lands the trajectory substrate the probe pointed at:
`gbrain eval trajectory <entity>` shows the chronological typed-claim
history with regressions flagged inline; `gbrain founder scorecard
<entity>` rolls up four signals (accuracy, consistency, growth
direction, red flags) into a stable JSON contract. MCP op
`find_trajectory` (read scope, visibility-filtered for remote callers)
exposes the same data to agents. The probe's `temporal_supersession`
verdict and the consolidate phase's `valid_until` writeback both
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
— the probe still emits paste-ready commands, only `consolidate`
writes `valid_until` (R1+R8 grep guard pins this).
+130
View File
@@ -0,0 +1,130 @@
# Agent Bootstrap — Product Design (normative for scope & sequencing)
**Status:** APPROVED (product review, 3 adversarial rounds; CEO review; eng review —
0 unresolved decisions). Implementation source of truth:
[AGENT_BOOTSTRAP_PLAN.md](AGENT_BOOTSTRAP_PLAN.md).
## Problem
Agent platforms (OpenClaw, Hermes) deliver the full personal-agent experience —
scheduled work, skill files, SOUL.md identity, persistent memory — but setting one up
means deploying a server, wiring API keys, and paying raw token prices. Meanwhile
nearly everyone already runs Claude Code or Codex, both subsidized by subscriptions,
both capable of executing an install protocol themselves.
**The feature:** a paste-in install that adds memory + skills + identity +
session-triggered schedules + knowledge persistence to a private GitHub repo,
mimicking the agent-platform experience on the desktop apps people already have.
"Just use your local harness as your agent."
## Target surfaces & order
| Surface | In v1? | Order | Per-turn context seam | Persistence (write) seam |
|---|---|---|---|---|
| Codex in ChatGPT desktop | YES | 1st | spike exit question #2 | spike exit question #1 (blocking); fallback: mandated MCP writes + end-of-session sweep |
| Codex CLI | YES | with 1st | AGENTS.md pull protocol + `volunteer_context` | mandated MCP write ops |
| Claude Code desktop | YES | 2nd | hooks: SessionStart / UserPromptSubmit via serve IPC | SessionEnd hook: transcript ingest + scan-gated push |
| Claude Code CLI | YES | with 2nd | same hooks | same hooks |
The wedge is NOT platform feature-parity on day one. It is: paste → interview → an
agent that knows who it is and who you are → recalls what you told it in the next
session → visibly compounds during week one.
## The repo format (the contract)
A private GitHub repo is the product artifact — the agent's portable body:
- **Manifest:** `agent.json` (`format_version: 1`, provisional; `initialized`
sentinel distinguishes a template clone from a bootstrapped workspace).
- **Identity:** SOUL.md, USER.md, MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md,
ACCESS_POLICY.md, GITHUB.md — rendered ONLY from interview answers, never invented.
- **Content:** `brain/` (the gbrain source), `memory/` (daily notes), `skills/`,
`state/` (committed: interview.json, portable mcp.json; local-only files gitignored).
- **Compatibility promise:** hosted gbrain mounts `format_version: 1` repos natively;
compatibility is a test against this spec.
## Premises (all settled)
1. **Free desktop tier of the hosted ladder** — the desktop ceiling (laptop asleep =
agent asleep; data outgrows the disk) is the graduation mechanic to hosted gbrain.
2. **One agent-body format, two doors.** Portability up the ladder is moat #1.
3. **Awake-when-you-are is the honest desktop contract.** Session-triggered schedules
(jobs run at turn/session boundaries while the harness is open); true 24/7 crons
are hosted-tier, stated in-product.
4. **Day-one-empty-brain is the #1 churn risk.** Magic moment with zero corpus
(interview → next-session recall) + fast ramp (file import; connector ingest v1.1).
5. **Wire-level truth before build:** a clean-machine spike gates door 1 (write-seam
pass bar: 0 durable-write failures in 20 sessions over ≥3 days, else extend to 50).
6. **The graph is moat #2:** v1 exercises entity extraction, backlinks, and
graph-aware recall; verify enforces a graph floor.
7. **Keyless mode is first-class:** the harness agent IS the subsidized LLM — with
zero API keys, memory is agent-authored through write ops, search is keyword-only,
and the magic moment still passes. One optional key unlocks embeddings +
auto-extraction.
## Build order (one cathedral PR; size trip-wire: split at build order 2 if PR is
open >10 days from first code commit)
0. **Spike + quota gate** (manual, gates door-1 ship; per-harness quota measured; a
p90 day must fit ≤10% of weekly subscription quota or schedule scope is cut).
1. **Shared body + engine machinery:** `gbrain bootstrap` family, templates, format
spec, secret-scan-gated persistence, verify, uninstall (v1 via CEO-review
expansion; receipt-keyed scope per the PLAN's CX2-12).
2. **Codex door ships first** (runbook variant + approvals preflight + capability
probe; CLI path not spike-gated).
3. **Claude Code door:** hooks, IPC turn_context, transcript ingestion, greeting
digest, schedule mechanism.
4. **Graduation seam, desktop half:** format spec + documented upgrade path (advisor
nudge ships with the hosted mount in v1.1).
## Out of scope for v1
| Deferred | Lands | Why |
|---|---|---|
| Connector-driven ingest (email/calendar) | v1.1 (keyed to probe) | unverified host capability |
| Hosted mount + "outgrowing this laptop" nudge | v1.1 together | never point at a destination that can't accept the repo |
| `serve --attach` (simultaneous multi-harness) | fast-follow | v1 documents one-live-serve politely |
| Codex `notify` transcript sweeper | fast-follow | mandated MCP writes cover v1 |
| Windows (named-pipe IPC) | deferred | v1 = macOS + Linux |
| Networked Docker paste-flow e2e | fast-follow | offline container e2e covers 80% at 20% of the flake |
| `gbrain quota` meter command | TODOS | measurement ships as script+doc; productize when per-harness token counting is proven |
| True 24/7 crons on desktop | never | hosted-tier by design |
## Success criteria
- **TTFM ≤15 min** paste→verified install, excluding first-run toolchain downloads
(published separately); every human action counted.
- **Magic moment, deterministically:** verify asserts an interview fact is retrievable
through the agent's own MCP path; fresh-session end-to-end is a scripted human
confirmation. Must pass keyless.
- **Pilot tests:** a non-developer pilot completes the door-1 install unaided and
still uses it in week two; a developer completes the door-2 README install ≤10 min
with a week-two recall check.
- **Graph floor:** ≥1 entity extracted, ≥1 backlink resolved, one edge-only query
answered — via the real MCP write path.
- **Ladder proof:** a desktop-grown repo validates against `format_version: 1`.
- **Honesty checks:** quota number published; desktop contract stated in-product;
door 1 demotes to documented-beta on its trigger rather than shipping flaky.
## Distribution
Paste block + tag-pinned runbook (`BOOTSTRAP_FOR_AGENTS.md`, fetched at the
`latest-stable` ref — advanced by the release job only after assets publish, so
published copies never rot); optional GitHub template repo (generated at release from
the same renderer); binary via `bun install -g github:garrytan/gbrain#latest-stable`
(never npm). The paste block lives in the README's `## Install` section, as
per-harness subsections ordered "For Codex — the recommended first step" → "For
Claude Code" → "For OpenClaw or Hermes" (the 2026-08-09 ordering decision, recorded
in the PLAN's artifact table). `INSTALL_FOR_AGENTS.md` remains the paste path for
agent platforms and lives inside the OpenClaw/Hermes subsection.
## Threat model (v1 summary)
Tag-pinned fetch + version-stamp skew check + runbook phase allowlist; secret-scan
gates every commit AND corpus write (loud block, per-finding override); repo privacy
verified via API after create; hooks in gitignored local settings with a kill switch;
interview answers rendered as fenced data (never instructions) with escaping and
caps; uninstall keyed to a machine-local receipt, never deletes a brain it didn't
create; provider-policy drift acknowledged as residual risk — posture: measured
sustainable load, no absent-user background burn, portable body as the exit plan.
Full posture: `docs/guides/bootstrap.md`.
+817
View File
@@ -0,0 +1,817 @@
# Agent Bootstrap — Implementation Plan (normative)
The engineering source of truth for `gbrain bootstrap` — the paste-in install that
turns Claude Code / Codex desktop apps into a persistent personal agent (identity +
memory + skills + session-triggered schedules + private-repo persistence + local
PGLite brain). Product scope/sequencing source of truth:
[AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md). Where they disagree, the
design doc wins on scope; this doc wins on implementation.
Reviews absorbed: 3-designer panel + adversarial critique; product design review (3
rounds); CEO review (SELECTIVE EXPANSION, ~55 findings); outside voice x2 (37
findings, 35 accepted); eng review (32 findings). All accepted fixes are inlined
below with their finding IDs. 0 unresolved decisions.
---
## As-shipped deltas (read this first — where the code moved after the plan froze)
This plan is layered: later absorption sections (the post-design-review deltas, the
CX2 series) override earlier prose, and THIS section overrides everything below it.
The shipped implementation matches the plan except for these deltas:
1. **Verify runs LAST, not before host registration.** [CX2-5]'s determinism goal
survived, but the shipped phase order (single TS source:
`src/core/bootstrap/status.ts` `PHASES`) is
preflight → engine → interview → render → skills → wire → repo → **verify**,
and verify runs in-process on the caller-held engine, calling
`runMaintenanceSweep` directly — no transient serve. It works pre-registration
AND as the weekly re-run (`src/core/bootstrap/verify.ts`).
2. **Uninstall scope: [CX2-12] wins over the CEO-expansion bullet.** `~/.gbrain` is
NEVER deleted wholesale — only receipt-enumerated bootstrap-created state
(`src/core/bootstrap/uninstall.ts`).
3. **Module naming/layout:** `private-repo.ts` shipped as `repo.ts`; additional
shipped modules the artifact table doesn't list: `attach.ts, assets.ts,
format.ts, host-specs.ts, hooks.ts, lock.ts, status.ts, template-repo.ts,
uninstall.ts`.
4. **Templates layout:** all bootstrap templates live under `templates/bootstrap/`
(not at `templates/` root).
5. **Test filenames:** `test/hook-command.serial.test.ts` and
`test/e2e/bootstrap-*.serial.test.ts` — the `.serial` variants the plan's own
[A7] mandated; the artifact table predates that.
6. **README ordering:** the D5 placement was superseded by the 2026-08-09 user
decision — per-harness `## Install` sections ordered Codex → Claude Code →
OpenClaw/Hermes, with `INSTALL_FOR_AGENTS.md` living inside the OpenClaw/Hermes
section (annotated in the artifact table; the D5 prose at the bottom is stale).
## Post-design-review deltas (2026-08-07, /office-hours APPROVED — these override below)
Product: **"GBrain for Codex" + "GBrain for Claude Code"** (names contingent on
trademark review) — the free desktop tier of the hosted-gbrain ladder (the hosted service is the
graduation path; the desktop ceiling is the mechanic, not a bug). Target surfaces are
the DESKTOP APPS; **Codex door ships first** (named pilot user is a non-developer
ChatGPT-app user). CLIs come along via shared machinery.
**Build order (replaces the PR1/FF phasing below where they conflict):**
0. **Spike (gates door-1 ship; starts immediately, before build):** clean-machine
reproduction of Garry's working Codex-in-ChatGPT + gbrain prototype. Exit questions
with pass bars: (#1 blocking) reliable per-turn/per-session WRITE trigger — 0
durable-write failures in 20 sessions across ≥3 days on a non-founder machine, else
extend to 50; causes logged. (#2) per-turn READ seam — context block present at turn
start or degraded pull-mode documented. Also: folder access, MCP registration path,
approval-tap count, connector availability (yes/no/degraded). PLUS in parallel: the
**quota release gate** measured against Garry's production usage — pass: p90 day
≤10% of weekly Max/Pro quota; failure = scope change to "session-agent, no schedules
by default" BEFORE schedule build effort. Codex CLI is NOT spike-gated.
1. **Shared body + engine machinery (PR1):** everything in this plan's PR1 (bootstrap
family, gbrain hook + IPC turn_context, templates, question bank, secret scan,
sources push, skillpack scaffold wiring, doctor checks) PLUS: `agent.json` repo
manifest with `format_version: 1` (the ladder contract — the hosted gbrain service mounts this
format; compatibility is a test, not a hope); wire EXISTING import + entity
extraction + backlinks + graph-aware query into install/verify (funds design
Premises 4/6 — new verify check: ≥1 entity extracted, ≥1 backlink resolved, one
edge-only query answered); byte floors scaled to answered-question count (they catch
skipped interviews, NOT invention — never pressure padding); commit this plan to
docs/designs/AGENT_BOOTSTRAP_PLAN.md.
2. **Codex door (ships first):** runbook variant for Codex desktop approvals/sandbox
(seam per spike; fallback = mandated MCP writes via AGENTS.md gates + end-of-session
sweep, shipped only with measured reliability) + Codex CLI path (AGENTS.md pull
protocol; session-start gate includes "run due jobs"). Connector-ingest capability
probe wired into install output; connector ingest itself is v1.1.
3. **Claude Code door:** hooks per this plan (SessionStart/UserPromptSubmit/SessionEnd
via serve IPC), MCP project-scope default, transcript ingestion. Includes the v1
schedule mechanism: hook layer checks HEARTBEAT.md due-job list at session start +
turn boundaries. If the spike finds no turn-boundary trigger on ChatGPT desktop,
schedules are Claude-Code-only in v1 and per-surface copy says so.
4. **Graduation seam, desktop half only:** format spec + documented upgrade path +
honest-contract copy. The "outgrowing this laptop" advisor nudge moves to v1.1
WITH the hosted mount (never point at a destination that can't accept the repo).
**New v1 requirements from the design review (fold into implementation):**
- Binary install is tag-pinned: `bun install -g github:garrytan/gbrain#<release-tag>`,
same stamp as the runbook, so `bootstrap status` skew check is meaningful.
- Non-terminal-buyer error channel: every blocking condition (scan block, lock
collision, partial install, verify fail) surfaces through the agent's reply channel
(agent-readable status/verify/doctor output the runbook + AGENTS.md instruct the
agent to relay; hook failures write a status file surfaced at session start).
- Both doors, one machine: shared brain, one live serve; installer detects + reuses
existing registration; simultaneous-session collision fails politely, doctor names
it, agent relays it. serve --attach lifts the limit later.
- state/ boundary: interview.json + mcp.json COMMITTED (machine-2 re-render must not
hard-fail); *.local/caches/heartbeat files gitignored; DB never committed.
- OS scope v1: macOS + Linux (unix-socket IPC). Windows deferred (named-pipe = spike
question if prioritized).
- Success criteria additions: TTFM ≤15 min EXCLUDING first-run toolchain downloads
(published separately); door-2 developer test (README paste → ≤10 min install,
week-two recall check); graph floor verify check; magic moment = deterministic
retrieval assertion + scripted human confirmation (never fake-automated).
- Provider-policy drift acknowledged as unmitigable external risk; posture = quota
gate + no absent-user background burn + portable body as exit plan.
- README: bootstrap paste block is the "full agent" option INSIDE the existing "Quick
start: Claude Code or Codex" section; the 2-line memory-only quick start STAYS,
labeled "just memory, no agent"; INSTALL_FOR_AGENTS.md remains the platform headline.
- Consolidated v1 non-goals (from design doc, amended by CEO review): connector ingest
(v1.1), hosted mount + advisor nudge (v1.1), serve --attach, notify sweeper, Windows,
networked-paste Docker e2e (fast-follows); 24/7 crons never on desktop.
**CEO-review accepted expansions (2026-08-07, SELECTIVE EXPANSION — now v1 scope):**
- **GitHub template repo** ("Use this template") as a second distribution artifact for
the ChatGPT door (resolves design OQ2 = yes). Same rendered file set, published;
kept in sync with templates/ by extending scripts/check-bootstrap-templates.sh to
diff the template repo content. Build order 2.
- **`gbrain bootstrap uninstall`** in v1 (was fast-follow): removes MCP registration +
hooks + bootstrap-created state (confirm-gated), leaves the repo ("the body remains
yours"). [Scope superseded by CX2-12 + as-shipped delta 2: `~/.gbrain` is never
deleted wholesale — only receipt-enumerated bootstrap-created state.]
- **Docker cold-machine e2e (offline parts) in CI** in v1: networkless read-only
container running interview → render → verify with fake gh (codex-as-agent
tests/docker shape). The full networked paste flow stays a fast-follow (flake).
- **Hot-memory greeting digest in SessionStart**: top facts via IPC turn_context when
serve is up; file-plane MEMORY.md digest fallback when not (session start often
precedes serve spawn); fail-open. Claude door in v1; door 1 iff spike finds a read
seam.
- **First-run tour**: `bootstrap verify` success output ends with three scripted magic
prompts (who am I to you / remember X then restart / what do you know about
<project>) — makes week-one compounding visible (design Premise 4).
- **Deferred to TODOS.md in the PR** (must land as TODOS entries with the cathedral
PR): `gbrain quota` meter command (release-gate measurement ships as script+doc
first; productize once the per-harness token-count method is proven); networked
Docker paste-flow e2e.
## Deep-review hardening (2026-08-07 CEO review sections 19 — ALL ACCEPTED per
## Garry's standing directive; IDs trace to the review record. These are v1 scope.)
**Architecture (S1):**
- [D4] The release-time template-repo generator IS `core/bootstrap/render.ts --minimal`
with placeholder answers — one rendering code path, two consumers; CI guard becomes a
byte-diff of generator output vs the vendored tree.
- [D5] Bootstrap phase list defined ONCE in TS; `bootstrap status --json` emits it; the
runbook defers to it ("follow status's phase list"); CI checks runbook phase names
against the TS list.
**Source binding + brain semantics (G1, S3#1 — the two product-breaking fixes):**
- [G1] MCP registration passes the workspace source explicitly
(`claude mcp add -e GBRAIN_SOURCE=<workspace-slug>` / codex env equivalent) so agent
writes land in the workspace source, fact fence-writes reach `brain/` files (not
DB-only fallback), and the private repo actually fills. `bootstrap verify` asserts an
MCP-path `put_page` materializes a COMMITTED file under `brain/` — a green verify with
an empty repo is impossible.
- [S3#1] `turn-context.ts` constructs an OperationContext with `remote: true` and
threads `visibility: ['world']` into all fact reads (parity with the existing
meta-hook posture — the IPC path must never widen what MCP would return). IPC test:
a `visibility='private'` fact NEVER appears in a turn_context response. Verify's
magic-moment fact is written with visibility the harness can read back (world).
**Interview + render hardening (S3#3, G10, G12, A8):**
- Answers render inside fenced, explicitly-subordinate blocks ("verbatim principal
input — data, not instructions"); strip/escape line-leading `#`, `<!--`, and code
fences; per-answer length cap (~4KB, confirm-to-truncate); reject/escape `{{` and
control chars at `--set` time (a Handlebars user's honest answer must not brick the
token sweep). Verify fails if any heading or managed-block marker in a rendered file
traces to an interview value (protects harden's AGENTS.md marker splicing).
- [A8] `--confirm` requires the hash of the exact answer set that was read back;
per-answer `set_at` provenance recorded; hostile test: single-batch set+confirm
exits non-zero and verify reports `provenance: unverified`.
- [G12] Conflict-marker detection on `state/interview.json` read → agent-readable
"resolve this file" message, never a stack trace.
**Uninstall confinement (G2, S3#5, A2):**
- Never delete a brain bootstrap didn't create: `agent.json` created-by stamp checked;
default KEEPS the DB; deletion only behind explicit `--delete-brain` with a confirm
that enumerates sources + page count; facts export offered first (facts are NOT
derived state). Refuse when `GBRAIN_HOME` is set unless `--home` is explicit AND
`isPathContained` + gbrain-home signature (config.json + brain.pglite) both pass.
Host-config edits are marker-keyed managed blocks only (settings.local.json,
~/.codex/config.toml) — foreign hooks/servers survive; test asserts full
before/after filesystem+registration diff incl. foreign entries + symlinked home.
**Hooks + IPC hardening (G5, G11, S3#6, S3#7, S3#8, A9, A3):**
- [G5] `bootstrap hooks`/`--repair`/uninstall use marker-keyed managed-block
read-merge-write on settings.local.json — never overwrite `permissions.allow` or
foreign hooks; never append duplicates.
- [G11] IPC client timeout parameterized per kind (turn_context > 250ms default);
window payload + assembled block clamped below the 256KB message cap before send.
- [ENG-1] **Claude Code hook-output cap:** stdout/additionalContext is capped at
10,000 chars by the harness (overflow is diverted to a file and NOT injected) —
the assembled turn_context block is budgeted to ≤8KB (pointers + facts trimmed by
confidence to fit), asserted in the hook snapshot test. Docs: code.claude.com hooks
reference (verified 2026-08-08; the hook writers are dated spec-targets, so this cap
lives with the settings-shape module).
## Eng-review hardening (2026-08-08 /plan-eng-review — ALL ACCEPTED per standing
## directive; seams verified against source with file:line quotes)
- [ENG-2] (9/10) **cli.ts registration is THREE touchpoints, not one:** `CLI_ONLY` set
(cli.ts:58), the engine-free if-chain inside `handleCliOnly` (add before the
`connectEngine` terminator at cli.ts:1840), and `CLI_ONLY_SELF_HELP` (cli.ts:61 —
omit it and the subcommands' `--help` is dead code, the documented init.ts:117 trap).
`bootstrap`/`hook` must NOT enter `THIN_CLIENT_REFUSED_COMMANDS`. Membership test per
the #2035 precedent (test/cli-bigint-normalize.test.ts:46 shape).
- [ENG-3] (9/10) **IPC widening = handler restructure, not a field add.** Correct path
is `src/core/context/resolve-ipc.ts` (constants at :26-28). `ResolveHandler` is a
single-function type (:42) and the server handler dispatches unconditionally
(:127-128) — turn_context needs a discriminated-union request + handler map
(restructuring the closure at src/mcp/server.ts:90-108), NAMED response types
(today's responses are inline literals), and per-kind MAX_MSG_BYTES/CLIENT_TIMEOUT.
Absent `kind` defaults to 'resolve'; old-serve grace confirmed (malformed → client
IPC_UNAVAILABLE fail-soft, resolve-ipc.ts:80-84).
- [ENG-4] (8/10) **Postgres-hook silent degrade on unmigrated brains:** volunteer.ts
(:247-249) and retrieval-reflex.ts (:180-183) swallow missing-table errors
(pre-v110/v117), so a direct-engine hook returns empty, not an error. Fix: hook
heartbeat records `degraded_reason: 'schema_pre_vNNN'`; doctor pairs
hook-in-use + unmigrated-brain into a named warning.
- [ENG-5] (8/10) **Sweep layer ownership decided:** startup sweep attaches in
src/mcp/server.ts immediately after `server.connect` (:77) in the same best-effort
try/catch shape as the resolve-IPC block (:85-112), with cleanup added to shutdown
(:122); the idle sweep lives in src/commands/serve.ts reusing the `armIdle` pattern
(:437-451) through the injectable `deps.setInterval` seam (:281,297), every timer
`unref()`d (the :424/:444 convention) so the sweep can never hold the process open.
- [ENG-6] (9/10) **Compiled-binary asset rule:** `bin/gbrain` ships via
`bun build --compile`; `dirname(dirname(__dirname))` template resolution
(init.ts:1514 pattern) breaks in the binary. Templates + questions.json + runbook
stamps are STATICALLY IMPORTED (bundled) — same mechanism as the existing
`skills/_brain-filing-rules.json` static import in brain-repo-durability.ts. A
compiled-binary e2e asserts `bootstrap render` works with NO repo checkout present.
- [ENG-7] (7/10) **Host-format module precedents named:** registration strings follow
connect.ts `AGENT_SPECS` + argv builders (:61-67, :262-267); file-writing hook/
settings writers follow the integrations.ts / frontmatter-install-hook.ts idiom
(backup + marker + restore). The dated-spec-target scaffold is imported from
codex-as-agent (greenfield here — docs/plans/ has n=1 file); it becomes
`src/core/bootstrap/host-specs.ts` with TARGETS entries carrying id/status/
verifiedAt/references.
- [ENG-8] (9/10) **Facts visibility knob = ONE resolver helper.** The 'private'
default is duplicated at backstop.ts:185, :352, operations.ts:4468, :5812 — and the
:4468 ternary coerces any non-'world' to 'private', so a config default needs an
explicit caller-unset check. Implement `resolveDefaultVisibility(engine)` (reads
`facts.default_visibility` via the getConfig precedent, extract.ts:44) feeding
ctx.visibility at ALL FOUR sites; no schema change (CHECK already permits 'world',
migrate.ts:2319). Documented as security-relevant: it widens what remote/MCP
callers read via meta-hook.ts:66 — the intended effect, stated as such.
- [ENG-9] (8/10) **Secret-scan module reuses residents:** seed exclusion list from
`.gitleaks.toml` allowlist paths (test/, skills/, .claude/skills/) so the scanner
doesn't fire on fixtures CI already ignores; findings render through
`redactSecretsInText` (shell-redact.ts:36) for consistent `<REDACTED:name>` output.
- [ENG-10] (7/10) **Renderer must not eat intentional literals:** skillpack scaffolds
carry a literal `{{output-from-skill}}` token (init-scaffold.ts:269) that must
survive to disk — the renderer is never pointed at skillpack scaffold paths, pinned
by a negative test.
- [ENG-11] (7/10) **turn_context reuses the existing hot-memory cache** (30s TTL keyed
on source+session, meta-hook.ts) instead of a fresh facts query per turn — the
per-turn cost profile is then identical to what every MCP tool call already pays.
- [ENG-tests] New tests from this review: CLI_ONLY membership (bootstrap, hook);
compiled-binary render e2e (no repo checkout); Postgres-hook degraded_reason;
resolveDefaultVisibility across all 4 call sites (unset/world/private × config);
IPC handler-map dispatch (kind absent/resolve/turn_context/unknown); renderer
negative test on scaffold literals; 8KB block-budget snapshot assert.
- [ENG-tests-2] Coverage-trace gaps (Section 3): **attach-mode e2e** (clone a
bootstrap-created fixture repo on "machine 2" → attach → hooks repair → verify);
**sweep tests** (bounded batch per idle tick, spend-gate off ⇒ no LLM calls,
unref/shutdown never held open, corpus file marked processed exactly once);
**keyless-mode e2e** (zero API keys: install → agent-authored fact via ops → BM25
recall → magic-moment passes → capability report says keyless);
**decline-everything e2e** (no gh, no keys, hooks declined: install completes
local-only with honest warnings, verify exits 0-with-warnings, nothing silently
broken).
## Eng outside-voice absorption (Codex round 2, 2026-08-08 — 17/17 ACCEPTED,
## all with file:line evidence; CX2 ids)
- [CX2-1 P0] **Template-vs-attach discriminator:** agent.json presence cannot
discriminate a template clone from a machine-2 clone. The template ships
`agent.json` with `initialized: false`; `bootstrap render` flips it true atomically
AND writes a machine-local install receipt (`~/.gbrain/bootstrap/receipt.json`);
`attach` requires `initialized: true`. [pairs with CX2-12]
- [CX2-2 P0] **One-live-serve wording clarified:** each door spawns its OWN stdio
serve via its MCP registration (process-bound transport, server.ts:76); "reuse
registration" means config, never process. v1 contract restated: one live serve at
a time per brain; sequential across doors works; simultaneous fails politely.
- [CX2-3 P0] **Durability must be parent-repo-aware:** write-through targets
`repo/brain` but hardening asserts `.git` in that exact dir (sources-harden.ts:83,
brain-repo-durability.ts:621) — would fail on the workspace layout. Fix: durability
resolves the repo root via `git rev-parse --show-toplevel` (the sync.ts:1002
precedent); commit/push operate on the parent repo; integration test on the
workspace fixture.
- [CX2-4 P0] **Keyless facts ingestion made deterministic:** put_page only queues
extraction (operations.ts:1337); facts-fence reconciliation lives in the cycle
extractor (extract-facts.ts:337). The serve sweep INCLUDES the zero-LLM facts-fence
reconciliation pass, so agent-authored `## Facts` fences populate the facts table
with no API key. Keyless e2e asserts it.
- [CX2-5 P0] **Graph-floor verify made deterministic:** verify cannot command the
host's serve (stdio owned by the desktop app). Fix: the sweep gets a trusted
local-only CLI entry (`gbrain sweep --once`, CLI_ONLY, never over MCP), and
`bootstrap verify` runs BEFORE host registration on its own transient serve/engine:
write via op → `sweep --once` → edge query. No timing nondeterminism.
[Sequencing superseded by as-shipped delta 1: verify shipped as the LAST phase,
in-process on the caller-held engine; the determinism goal is unchanged.]
- [CX2-6 P1] **Cross-platform lock replaces flock dependence:** flock(1) absent ⇒
locking silently disabled (brain-repo-durability.ts:137) — macOS is the v1 target.
One cross-platform lock (atomic mkdir/lockfile with PID+age+token semantics) spans
scan → stage → commit → pull → push, coordinated with the post-commit hook.
[As-shipped delta: a TOCTOU fix reordered the scanned phase to stage FIRST, then
secret-scan the STAGED index blobs (`git cat-file`), so scanned bytes == committed
bytes; unscannable staged blobs fail closed (`blocked_unscannable`). Lock span
otherwise unchanged.]
- [CX2-7 P1] **Push ordering pinned:** commit FIRST, then divergenceSafePull, then
push (the existing durability ordering, brain-repo-durability.ts:200 —
divergenceSafePull returns skipped_dirty on a dirty tree, git-remote.ts:489); test:
dirty local + advanced remote.
- [CX2-8 P1] **GBRAIN_HOME dual semantics normalized:** config appends `.gbrain`
(config.ts:1210) while durability uses the value directly
(brain-repo-durability.ts:95) — the S3#10 `ensureGbrainHome()` choke point is also
the single semantic resolver; the --isolated e2e asserts the credential store's
ACTUAL location is gitignored, not just the expected path.
- [CX2-9 P1] **Provider-key resolution re-specified honestly:** env legitimately
overrides file config (config.ts:568) — the CX-P1.4 claim "never from shell env" is
wrong. Real contract: interview-provided keys go to the 0600 config file so
GUI-spawned serves (which lack shell env) find them; normal env>file precedence
stands; test = GUI-launch simulation with empty env + file key.
- [CX2-10 P1] **IPC authorization, not just authentication:** turn_context binds
server-side to the registered GBRAIN_SOURCE and rejects caller-supplied cross-source
requests (existing handler accepts caller sourceId, server.ts:89); cross-source
rejection test.
- [CX2-11 P1] **Hot-memory cache session key made real:** meta-hook reads an ad-hoc
`source_session` that dispatch never sets (meta-hook.ts:49, dispatch.ts:208) — all
callers collapse to the null-session cache key today. Session identity becomes
typed OperationContext state set from MCP `_meta.session_id`; two-session isolation
test.
- [CX2-12 P1] **Uninstall ownership = machine-local receipt** (never the repo-carried
agent.json, which template/attach clones inherit); uninstall stops/refuses a live
serve before touching state; `~/.gbrain` global config/sources/clones are NEVER
deleted wholesale — only receipt-enumerated bootstrap-created state.
- [CX2-13 P1] **Committed-state hygiene:** the optional interview API key bypasses
answers/hashes/provenance/logs entirely → written only to the 0600 config sink
(config.ts:1138); committed `state/mcp.json` is the PORTABLE snippet (no absolute
paths, no machine GBRAIN_HOME) — machine-specific wiring lives in local state and
is regenerated by attach/`hooks --repair`.
- [CX2-14 P1] **Generator determinism:** template-repo renders use canonical
placeholder provenance (frozen timestamps), exclude runtime state; guard test =
two independent renders are byte-identical.
- [CX2-15 P1] **Two scan policies, not one:** the `.gitleaks.toml` allowlist is a CI
fixture policy for a PUBLIC repo — importing it into the personal-repo runtime
scanner creates blind spots (skills/ trees). Runtime scanner ships its own minimal
allowlist + per-finding override; ENG-9 amended.
- [CX2-16 P1] **Bootstrap lock done properly:** atomic acquisition + PID liveness +
age guard + ownership token (the pid-reuse learning applied); kill→immediate-rerun
recovery test; export `LiveServeLockError` (currently unexported, pglite-lock.ts:28)
so the planned class assertion can import it.
- [CX2-17 P2] **Format-aware host-config writers:** JSON has no comment-marker
boundary — settings.local.json gets a structural JSON merger (gbrain entries keyed
by a `_gbrain` marker property, semantic dup detection surviving reordering);
config.toml gets a TOML-aware writer; both atomic write+backup; the
frontmatter-install-hook replace/backup idiom applies only to whole-file targets.
G5 amended.
- [S3#6] Socket binds in a 0700 dir with mode set BEFORE exposure; turn_context
requires a shared secret from a 0600 file in the data dir; heartbeat counts
turn_context serves so doctor can flag unexplained callers.
- [S3#7] Heartbeat JSONL schema pinned to counters + durations + error codes — NO
prompt/fact/slug text; dir 0700; line cap; CI test greps fixture for keys outside
the allowlist.
- [S3#8] `transcript_path` confined: `isPathContained(path, ~/.claude/projects)`,
`.jsonl` + valid envelope on line 1, lstat-reject symlinks, byte cap.
- [A9] IPC request carries `protocol: 2`; hook treats a response lacking the protocol
echo as "stale serve" and degrades LOUDLY (heartbeat entry + doctor warn); live test
against a v1-shaped server.
- [A3] Session-start digest: explicit list of digest-eligible MEMORY.md sections
(respecting the template's own security-boundary note); three-case test (socket up /
socket absent / malformed MEMORY.md).
**Persistence + sync (G4, G6, G8, G9, G13, G14/A5, G15, S3#2, S3#10):**
- [G4] `hook session-start` checks for unpushed commits / dirty tree from crashed
sessions, pushes, and names it in the greeting digest (SessionEnd is not the only
persistence path anymore).
- [G6] Verify + every push gate run `git ls-files` against a deny-glob list
(`*.pglite`, `.env*`, keys) — a truncated or pre-existing .gitignore can't leak.
- [G8] `bootstrap repo` refuses any pre-existing `origin` (always creates a dedicated
repo); "couldn't verify visibility" is refuse-and-name-the-reason, never fail-open;
idempotency keys off the remote URL, not the name probe.
- [G9] Workspace lockfile (pid+timestamp) makes concurrent `bootstrap` runs impossible;
second run exits "bootstrap already running (pid N)".
- [G13] Fixed verify probe slug; sweep any prior probe before writing; excluded from
retrieval; delete failure reported as a verify warning.
- [G14/A5] The ENTIRE `sources push` (add+commit+push) runs under the existing
durability flock; single-flight test: N concurrent pushes → 1 winner, N-1 clean
"skipped, push in flight" exits, zero leftover locks.
- [G15] Retention policy everywhere: MEMORY.md size cap in the template contract with
rotation into `memory/reference/`; corpus pruned via
`dream.synthesize.corpus_retention_days` (default 30); orphaned stop-hook buffers
GC'd; doctor warns on all three.
- [S3#2] Secret scan runs at corpus-WRITE time (redact matched span + log redaction);
bootstrap adds one consent line naming the extraction/embedding provider that will
see session text.
- [S3#10] One `ensureGbrainHome()` choke point creates ~/.gbrain 0700 (all callers);
brain-push.log 0600 + rotated; `--isolated` e2e asserts `git check-ignore
.gbrain/git-credentials` passes and push never stages it.
**Transcripts (G3, A6):**
- Parser registered as a dated spec-target (same pattern as host-format writers);
heartbeat logs parsed-turns/bytes; `bytes>0 && turns==0` raises a LOUD status-file
failure ("the agent stopped learning" is never silent); scrubbed
`claude-code.jsonl` fixture (tool_use/tool_result/thinking/image/sidechain/summary/
compact-boundary shapes) added to `test/fixtures/conversation-formats/` and wired
into `check:conversation-parser` + fixture-privacy guard; session-id-keyed corpus
filenames prevent double-ingest on resume.
**Template repo (G7, S3#4, A1, C3):**
- Published ONLY from a CI release job (branch-protected, no direct pushes) behind the
placeholder assertion + secret scan + privacy scripts run against the RENDERED
artifact; fine-grained PAT secret scoped `contents:write` to the template repo only,
documented in docs/RELEASING.md; version-job completeness check verifies template
repo HEAD tree hash == vendored tree. Vendored rendered tree lives at
`templates/bootstrap/template-repo/` — the CI guard diffs generator↔vendored OFFLINE
(network comparison happens only in the release workflow). Template's first runbook
step is `bootstrap status`, which hard-fails on a public `origin` before any
identity file lands (closes the no-privacy-gate-on-this-door hole).
**Toolchain trust (S3#9):**
- Prefer platform package managers for bun/gh; curl fallback downloads to a file,
verifies against the pinned bun release's SHASUMS256.txt, then executes; the
runbook's phase allowlist names the verified form as the only permitted variant.
**Observability (B1B5):**
- [B1] Append-only `~/.gbrain/bootstrap/install.jsonl` ({ts, phase, outcome,
duration_ms, binary_version, harness, workspace}) written by every subcommand;
`status` renders the tail.
- [B2] Every verify run persisted to `~/.gbrain/bootstrap/verify-<ts>.json` (keep 5);
doctor reports last verify timestamp/status/deltas.
- [B3] Hook heartbeat records outcome + reason on EVERY invocation; trailing-20
failure rate over threshold → one visible line inside the injected context block
("brain context unavailable for the last N turns — run `gbrain doctor`").
- [B4] `~/.gbrain/bootstrap/push-status.json`; doctor fails when last successful push
>48h old with a dirty tree; surfaced in the SessionStart digest.
- [B5] `bootstrap status --json` emits a support blob (workspace, binary version,
harness, engine, last verify, last push, hook failure rate); AGENTS.md instructs
the agent to relay it verbatim on any "something's broken" report.
**Tests (A1A9 not already covered above):**
- [A4] First-run tour prompts pinned in the questions.json↔template bijection guard +
verify success-output snapshot test.
- [A7] Flake pinning: hook deadline asserted against an injected slow-IPC stub (real
latency kept as non-gating benchmark); kill-mid-phase via deterministic
`GBRAIN_BOOTSTRAP_ABORT_AFTER=<phase>` injection; lock-contention asserts on
LiveServeLockError class; all subprocess tests named `*.serial.test.ts` with
explicit --timeout; offline Docker e2e runs from heavy-tests.yml (nightly + label),
not the PR shard matrix.
**Release mechanics (C1 RESOLVED + C2, C4, C5, C6, C8):**
- [C1 = D6-A, decided by Garry] **Distribution ref: single `latest-stable` ref.** The
release job force-updates `latest-stable` to the just-verified release commit as its
FINAL step (after binary assets publish + provenance attestation). README paste
block, runbook URL, and `bun install -g github:garrytan/gbrain#latest-stable` all
reference it permanently — no per-release tag pins, no 404 window, no per-ship
README edits. The fetched runbook embeds the concrete VERSION it was cut from;
`bootstrap status` compares that stamp to the installed binary (skew check intact).
New guard `scripts/check-bootstrap-tag.sh`: README/runbook reference ONLY the
sanctioned ref AND the runbook's embedded stamp equals VERSION. Modeled on
test/release-workflow.test.ts.
- [C2] README.md + BOOTSTRAP_FOR_AGENTS.md re-admitted to `scripts/ci-cache-hash.sh`
ALLOW_PATTERNS (a README-only paste-block change must never skip CI green).
- [C4/C5] `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + a SCRUBBED
`docs/designs/AGENT_BOOTSTRAP_DESIGN.md` (banned names → capability-class phrasing,
founder quotes/pilot identifiers/pricing-funnel strategy removed) land in the SAME
commit; normativity paragraphs point in-repo; `bash scripts/check-privacy.sh` run
against the staged index before that commit.
- [C6] CHANGELOG states capabilities functionally ("installs hooks that run on each
prompt; installs an opt-in background push job") linking to docs/guides/bootstrap.md
which owns the full security/consent posture together with the rendered
ACCESS_POLICY.md; zero plan IDs / review-round references.
- [C7] Public product names ("GBrain for Codex"/"GBrain for Claude Code") are a
pre-merge checklist item owned by Garry (trademark review); all committed copy uses
the neutral `gbrain bootstrap` verb until sign-off.
- [C8] Zero-migration release confirmed (facts/context_volunteer_events already
exist); ALL new bootstrap telemetry stays on the filesystem — the moment it moves
into a table it becomes migration v126 + bootstrap-coverage + engine-parity
obligations.
- [G16] User-scope consent question names the project-scope tradeoff explicitly;
CLAUDE.md renders a one-line "this agent lives in <dir>" note.
## Outside-voice absorption (Codex, 2026-08-07 — dispositions under Garry's
## auto-accept directive; CX ids trace to the codex output)
**ACCEPTED — product-critical (the two the whole review chain missed):**
- [CX-P0.5 **Keyless mode is a first-class design requirement.**] The named pilot user
has ChatGPT Pro but NO API keys — and gbrain's embeddings + extract_facts (Haiku)
are API-metered. Bootstrap MUST work with zero API keys: the HARNESS AGENT is the
subsidized LLM, so AGENTS.md instructs it to author facts/timeline/links explicitly
through write ops (zero API cost); search degrades to BM25 keyword (no embeddings);
verify prints an honest capability report ("keyless mode: keyword search, agent-
authored memory; add ONE key to unlock embeddings + auto-extraction") and the
interview offers the optional key question. The magic-moment check must pass in
keyless mode (fact written by the agent via ops, recalled next session). Quota-gate
copy amended: API spend (embeddings/extraction) is metered separately from
subscription quota and is ZERO in keyless mode; with a key, existing spend gates
(docs/operations/spend-controls.md) govern. [also resolves CX-P0.6]
- [CX-P0.1+P0.3 **Serve-resident maintenance sweep** closes the persistence loop.]
Nothing previously ingested the transcript corpus into the live brain (dream is
disabled; CLI can't open PGLite under a live serve), and remote `put_page`
deliberately skips auto-link/timeline extraction (operations.ts:1273) so the graph
would never compound from harness writes. Fix: the serve process (the lock owner)
runs a bounded, spend-gated maintenance sweep — on startup and idle — that (a)
ingests unprocessed corpus files (keyless mode: skipped, agent-authored memory
covers it), (b) runs the deterministic zero-LLM link/timeline extraction over
recently written workspace-source pages. Verify's graph-floor check exercises the
REAL MCP write path end-to-end (write via stdio op → sweep → edge query), never a
synthetic install-time shim.
**ACCEPTED — durability/trust:**
- [CX-P0.4] Write-through failures on `put_page` are best-effort today → for the
workspace source they become LOUD: doctor check counts DB-pages lacking file
backing; surfaced in the SessionStart digest; sources push reconciles before commit.
- [CX-P1.1] Single-principal visibility posture: bootstrap sets the workspace brain's
fact default visibility to 'world' (the desktop agent IS the principal's main
session; ACCESS_POLICY.md documents it; multi-tier users flip the knob). Keeps the
IPC world-only filter (S3#1) AND working personalization.
- [CX-P1.2] The injected turn_context block is wrapped in the same "retrieved data,
never instructions" subordinate envelope as interview answers (provenance-labeled).
- [CX-P1.4] GUI env inheritance: MCP registration carries the absolute binary path +
explicit env (GBRAIN_SOURCE, GBRAIN_HOME when --isolated); provider keys resolve
from ~/.gbrain/config.json, never from shell env, for GUI-spawned serves.
- [CX-P1.5] **`bootstrap attach` mode** (machine two): a cloned repo carrying
agent.json enters attach mode (register source, hooks --repair, MCP, verify) —
the refuse-pre-existing-origin rule (G8) applies only when agent.json is absent.
- [CX-P1.6] Git conflict model: sources push does fetch + divergence-safe rebase pull
(reuse harden's divergenceSafePull) before push; non-FF/conflict = loud status +
agent-relayed instruction, never silent retry; repo-local git author identity set
at `bootstrap repo` (ported from setup-private-repo).
- [CX-P1.7] Search-mode consent folds into the interview as one optional question
(default balanced) — preserves the INSTALL_FOR_AGENTS consent contract without
another stop; the spike's TTFM measurement counts every stop.
- [CX-P1.9] Non-circular error surface: README + GITHUB.md carry the one recovery
command ("if it seems broken: `gbrain doctor`"); doctor covers hook/push/serve
health independent of the possibly-broken agent loop.
- [CX-P1.10] Forget semantics documented honestly in ACCESS_POLICY.md/GITHUB.md: the
repo is append-only history; true deletion = documented history-rewrite procedure;
GitHub remains default-but-optional (local-only mode with honest warning).
- [CX-P1.11] Template repo README embeds the same VERSION stamp as the runbook;
`bootstrap status` validates it identically (closes the adopter-skew window).
- [CX-P1.12] agent.json spec labeled **provisional-v1**: desktop-side validation
only, consumers tolerate unknown fields, hosted mount may bump to v2 with a
migration note — a version contract, not a frozen promise.
- [CX-P0.7+P1.13] Spike/pilot instrument upgraded from binary pass/fail to product
metrics: per-turn write-attempt vs durable-write precision, correct-recall rate,
false-memory incidents, correction round-trips — measured through the 2-week pilot
(the pilot IS the extended sample); the 0-failures-in-20 bar remains only the
minimum to START the pilot.
**REJECTED (with reasons, recorded):**
- [CX-P0.2] "Two desktop apps cannot share the brain" — TRUE for simultaneous
sessions and already the documented v1 limit (polite collision + doctor + attach
proxy fast-follow); sequential use works. No change beyond what's accepted.
- [CX-P1.8] "Networked paste flow untested" — known; manual clean-machine acceptance
in v1 + networked Docker e2e deliberately deferred (D3.3b). Stands.
**CROSS-MODEL TENSION (recorded, not re-litigated):** Codex's "fundamental
simplification" (Codex-only, 3-question interview, no hooks/template/GitHub, 2-week
manual pilot before building) is a REDUCTION-mode argument against the settled
cathedral decision (D1-A) and accepted Approach B scope. Disposition: rejected as
sequencing (Garry chose the cathedral 30 minutes prior, with the two-wave case
recorded for revisit + a size trip-wire), but its measurement substance was absorbed
via CX-P0.7/P1.13, and the spike + pilot ARE the "prove the loop" instrument, run
before the doors ship rather than instead of building them.
## Design (synthesized)
Synthesis of three independent designs (DX lens, runtime-parity lens, architecture lens) +
an adversarial critique that verified the load-bearing claims against both repos. Where the
designs disagreed, the critique's evidence-checked winner is taken.
### The experience (end state)
One paste block (README + tweet) → the agent fetches `BOOTSTRAP_FOR_AGENTS.md` (raw GitHub
URL **pinned to a release tag**, same mechanism as INSTALL_FOR_AGENTS.md) and drives:
preflight → interview (chat, 12Q/6-required, hard gate) → render identity files → skills →
MCP + hooks wiring → sources add/sync/embed → private GitHub repo (created, privacy-verified,
pushed) → `gbrain bootstrap verify` (exit-code contract) → completion manifest.
Human actions: paste, answer interview in chat, `gh auth login` if needed, ~2 consent
questions. Target ≤15 min. Definition of done includes the magic moment: after restart, the
agent recalls a fact the human said during the interview.
Second session: SessionStart hook injects identity digest + "since last time"; every prompt
gets Live Context + ≤3 volunteered brain pointers + hot memory injected via hook; the agent
writes facts back same-turn through MCP ops; SessionEnd ingests the transcript into the dream
corpus and fires a scan-gated commit+push. "Your local harness IS your agent."
### Decisions (settled by panel + critique)
- **D1 Topology: single private repo.** Workspace root = identity files (SOUL.md, USER.md,
MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md, GITHUB.md, ACCESS_POLICY.md) + `skills/` +
`memory/` (daily notes + README) + `state/` + **`brain/`** (people/ companies/ meetings/
concepts/ daily/). **Only `brain/` is registered as the gbrain source** (critique: indexing
the whole repo puts the contract files into the retrieval corpus — noise + self-referential
injection surface). `.gbrain-source` dotfile at root routes CLI calls. PGLite DB is NEVER
in the repo (`.gitignore` renders first: `*.pglite`, `.env*`, `state/*.local`, corpus paths,
keys/PEM). Two-repo model stays documented as the graduation path in GITHUB.md/docs.
- **D2 Workspace = the cwd the user pasted in.** Guard: if cwd is an existing code project
(tracked files/remote), ask ONE relocation question. DB default: global `~/.gbrain` host
brain (workspace is a source); `--isolated` escape hatch per resolved D2.
- **D3 Delivery: hybrid.** New `gbrain bootstrap` command family (deterministic, idempotent,
exit-coded) + fetched runbook (judgment: interview wording, read-backs, consent). Bootstrap
is **CLI-only like init/connect — NOT an operation**; zero new `ctx.remote` surface (a
remote-triggerable `gh repo create` + filesystem render is forbidden by the trust model).
Subcommands: `status` (resume entrypoint; also verifies runbook version stamp vs binary —
supply-chain skew check), `interview --init|--set K V|--skip K|--status|--confirm|--show`,
`render [--force] [--only F] [--minimal]`, `repo`, `hooks --harness claude-code|codex
[--repair]`, `verify [--json]`.
- **D4 Interview: one question bank, two entry points.** Port codex-as-agent's
`questions.json` (scrubbed) to `templates/bootstrap/questions.json`: 12 asked / 6 required
(AGENT_NAME, PRINCIPAL_NAME, AGENT_PURPOSE, AGENT_TOP_JOBS, PRINCIPAL_CONTEXT,
VOICE_REGISTER — the wince question and vibe samples from soul-audit Phase 2 fold in as
optional keys). Answers → `<ws>/state/interview.json` (committed; identity source of truth,
same sensitivity as the rendered USER.md; makes re-render-on-new-machine work).
`--status` exits non-zero until required present; `render` additionally refuses until
`--confirm` (set only after the read-back "Is this the thing you want in the room?").
Hard rules verbatim in runbook: NEVER INVENT ANSWERS, accept "skip", quote literal words.
`skills/soul-audit/SKILL.md` becomes the re-run/deepen surface over the same bank
(`interview --set` + `render --only SOUL.md`) — one bank, CI-guarded against drift.
- **D5 Per-turn context (critique-decided; the PGLite lock is the constraint).**
`gbrain serve` holds the PGLite single-writer lock for its lifetime (#2348: live holders
are never stolen) — so hooks must NEVER open the engine on PGLite. New engine-free
`gbrain hook <event>` command (no-engine dispatch branch in cli.ts) talks ONLY to serve's
existing resolve-IPC unix socket (`src/core/context/resolve-ipc.ts`), widened with
back-compat `kind: 'resolve' | 'turn_context'`. `turn_context` request carries
{window, priorContextText, sourceId}; serve assembles server-side (new
`src/core/context/turn-context.ts`): reflex pointers + volunteerContext pages (≤3) +
hot-memory facts (the same content `_meta.brain_hot_memory` carries — this ROUTES AROUND
the _meta invisibility rather than fixing the harness). Postgres fallback: hook opens the
engine directly (multi-connection safe) since the IPC socket only exists on PGLite.
- Claude Code hooks (written to **`.claude/settings.local.json`** — gitignored; committed
hooks with absolute paths are a portability trap + persistence-of-execution surface;
`bootstrap hooks --repair` re-renders on a new machine):
- SessionStart → `gbrain hook session-start`: zero-DB file reads (date/tz, MEMORY.md
open-commitments, ops/tasks.md if present) + best-effort socket warm probe. ≤1.5s.
- UserPromptSubmit → `gbrain hook user-prompt`: stdin hook JSON → tail last 4 turns from
transcript_path → socket turn_context → stdout additionalContext block.
**800ms hard self-deadline**, fail-open (exit 0, empty stdout), heartbeat JSONL at
`~/.gbrain/integrations/hooks/` for doctor.
- Stop → `gbrain hook stop`: append turn to live per-session buffer (pure file append).
- SessionEnd → `gbrain hook session-end`: parse full transcript .jsonl → corpus .txt in
`dream.synthesize.session_corpus_dir` (default `~/.gbrain/transcripts/corpus/`, 0700,
never in the repo) + fire scan-gated commit+push (D6). Closes the transcript gap in PR1.
- Absolute binary path everywhere; `GBRAIN_HOOKS=0` kill switch.
- Codex (no hooks): honest pull model. Rendered AGENTS.md carries adapted per-message
gates 07 (entity lookup = "call recall/volunteer_context with recent window";
receipts; WRITE IT DOWN same turn via extract_facts/put_page) +
`codex mcp add gbrain -- gbrain serve`. FF2: `notify` hook in ~/.codex/config.toml as
transcript sweeper (validate the event semantics first). Runbook/verify state the
degradation plainly: Claude Code = push-on-hook, Codex = pull-on-protocol.
- **Write-path rule rendered into AGENTS.md** (critique hole 2): on PGLite, durable
knowledge is written through MCP ops (put_page/extract_facts/add_timeline_entry), never
by editing brain/ files directly — file edits are invisible to retrieval until a sync
can run, and sync can't run while serve holds the lock. MEMORY.md/memory/ file edits are
fine (file-plane, loaded by path not retrieval).
- **D6 Private repo + persistence.** `gbrain bootstrap repo` = TS port of
setup-private-repo.mjs: gh-auth exit-2 gate (only human step: `gh auth login`), slugified
`<agent-name>-workspace` collision probe, `gh repo create --private --source . --push`,
**privacy verified via `gh api ... --jq .private` (hard fail)**. Sync: extract secret scan
to `src/core/secret-scan.ts` (sk-/gh[pousr]_/github_pat_/xox[baprs]-/PEM; blocks commit);
new `gbrain sources push [<id>|--path]` = scan-gated add+commit+push, refuses public
remotes, pushes even on clean tree. `hardenBrainRepo` gains the scan as a step (its
existing post-commit hook + cron machinery is reused, NOT a parallel sync system).
Cadence per resolved D3: 15-min cron installed after an explicit consent question;
SessionEnd-hook push always on as the no-daemon backstop/fallback. GITHUB.md persistence
contract rendered.
- **D7 Rendered files.** AGENTS.md (adapted gates + hard gates: WRITE IT DOWN, NO SILENT
FAILURE, VERIFY BEFORE CLAIMING DONE, RED LINES, PRIVATE REPO PERSISTENCE + brain-first
protocol from docs/tutorials/connect-coding-agent.md + brain filing contract rendered from
skills/_brain-filing-rules.md), CLAUDE.md (thin: @AGENTS.md @SOUL.md @USER.md @MEMORY.md +
hooks note), SOUL.md (codex-as-agent section skeleton: Identity/Mission/Worldview/The
Standard/Honesty/Voice+wince/Good vs bad output/High agency/Never — IDENTITY.md merged in),
USER.md ("their literal words are ground truth"), **MEMORY.md (new template**: hot state,
corrections format `- YYYY-MM-DD — rule (Bug: ...)`, open commitments, security-boundary
note), ACCESS_POLICY.md, HEARTBEAT.md (quiet hours, verify-time-first, silence contract,
jobs disabled), GITHUB.md, memory/README.md, brain/ dirs + READMEs, .gitignore,
state/interview.json, state/mcp.json (portable snippet). Skills via existing
`gbrain skillpack scaffold --all` (resolved D4) + inert-skill report (skillpack check
wired into verify AND into the completion manifest's next-steps block with the exact API
keys to add). ALL templates REWRITTEN generic (privacy IRON RULE —
never copy codex-as-agent prose verbatim; CI placeholder assertion). TOOLS.md deferred
(essential lines fold into AGENTS.md). `installDefaultTemplates` (init.ts:1513) finally
gets its caller via `render --minimal`.
- **D8 Verify (union).** (1) doctor green; (2) DB round-trip: put_page → get → query-with-
score → DELETE probe page; (3) MCP registered (`claude|codex mcp list`) + probeBrainIdentity
smoke; (4) {{TOKEN}} sweep hard-fail; (5) byte floors (SOUL.md ≥3000B, USER.md ≥1000B);
(6) secret scan clean; (7) repo private via API; (8) **hooks smoke UNDER LIVE SERVE**
(spawn real serve, pipe fixture UserPromptSubmit stdin, assert non-empty block + <800ms +
never acquires the lock — bootstrap-time-only smoke is a false green); (9) one manual
sources push succeeded; (10) inert-skill report; (11) transcript parser dry-run on fixture.
Prints ranked completion manifest. Re-runnable weekly as the workspace rot self-check.
- **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
daemon). OPT-IN: 15-min harden cron. Autopilot NOT default on PGLite (verified: its
sync/embed children would contend with every live serve for the single-writer lock, and
nothing handles LiveServeLockError politely today) — recommended on Postgres; any future
scheduled job must treat lock-held as skip-silently-and-log. LLM crons (briefing, dream via
`claude -p`/`codex exec`) ship rendered-but-disabled with the enable-one-at-a-time ritual.
- **D10 Phasing.**
- **PR1 (complete usable experience):** BOOTSTRAP_FOR_AGENTS.md + README paste block;
`gbrain bootstrap` family; `gbrain hook` + IPC v2 turn_context; new templates + question
bank; secret-scan + `sources push` + harden integration; SessionEnd transcript ingest
(Claude Code .jsonl parser); skillpack scaffold wiring; soul-audit re-run update; doctor
checks (hooks heartbeat, dual-serve report, sync check); docs; unit + e2e (incl.
hook-under-serve + lock-contention pins); CI guards.
- **FF2:** Codex notify sweeper + ~/.codex/sessions parser. **FF3:** `gbrain serve
--attach` stdio proxy (two harnesses fully concurrent on one PGLite brain). **FF4:**
cron fleet + cron-doctor port + heartbeat activation. **FF5:** Docker fresh-machine e2e
in CI, _meta surfacing experiments, per-turn-context BrainBench eval, upgrade re-render
nudge via runPostUpgrade (`bootstrap render --diff`).
### Security/trust invariants (critique holes, addressed)
1. Paste block pinned to release tag; runbook version-stamped; `bootstrap status` compares
stamp vs binary and warns; runbook instructs refusing steps outside its phase list.
2. Write-through-ops rule in AGENTS.md (above).
3. MCP scope: project scope default, user-scope opt-in (resolved D1); threat named in
ACCESS_POLICY.md either way.
4. Codex sandbox reality: runbook carries a Codex-specific preflight (approval mode /
workspace-write + network consent) — the paste block warns the human they'll be asked.
5. Hooks in settings.local.json (gitignored) + --repair.
6. Upgrade story: FF5 re-render nudge; render never clobbers (backup on --force).
7. Transcript corpus + answers privacy: corpus 0700 outside repo; interview.json committed
(same sensitivity as rendered USER.md, which is committed); responsible-disclosure
phrasing in CHANGELOG (functional, no attack-surface enumeration).
8. Process invariants: CLAUDE.md dispatcher row edit → `bun run build:llms` same commit;
KEY_FILES.md entries current-state prose; version-first PR title; ship via /ship;
/document-release after.
### New/changed artifacts (paths)
| Path | New/changed |
|---|---|
| `BOOTSTRAP_FOR_AGENTS.md` | NEW root runbook (fetched by paste block) |
| `README.md` | dedicated "For Codex" / "For Claude Code" paste-block sections, ordered Codex → Claude Code → OpenClaw/Hermes at equal weight (user decision 2026-08-09, supersedes D5's ordering; both platform paths preserved) |
| `src/commands/bootstrap.ts` (+ `src/commands/bootstrap/*.ts`) | NEW dispatcher + subcommands |
| `src/commands/hook.ts` | NEW `gbrain hook session-start|user-prompt|stop|session-end` |
| `src/cli.ts` | CHANGED: `bootstrap`+`hook` in no-engine dispatch branch |
| `src/core/bootstrap/{interview,render,private-repo,verify}.ts` | NEW (TS ports) |
| `src/core/secret-scan.ts` | NEW (shared: sources push, harden, verify) |
| `src/commands/sources.ts` + `src/core/brain-repo-durability.ts` | CHANGED: `sources push`, scan-gated hook/cron |
| `src/core/context/resolve-ipc.ts` + `src/mcp/server.ts` | CHANGED: IPC v2 `turn_context` (back-compat) |
| `src/core/context/turn-context.ts` | NEW server-side block assembly |
| `src/core/transcripts/claude-code-jsonl.ts` | NEW parser + corpus writer |
| `templates/{MEMORY,AGENTS,CLAUDE,GITHUB,memory-README}.md.template` + `gitignore.template` + enriched SOUL/USER/HEARTBEAT/ACCESS_POLICY | NEW/CHANGED (generic, scrubbed) |
| `templates/bootstrap/questions.json` | NEW shared question bank |
| `skills/soul-audit/SKILL.md` | CHANGED: re-run surface over the bank |
| `src/commands/doctor.ts` | CHANGED: hooks heartbeat, dual-serve, sync checks |
| `docs/guides/bootstrap.md` + docs/mcp/ + connect-coding-agent.md cross-links | NEW/CHANGED |
| `scripts/check-bootstrap-templates.sh` | NEW CI guard (token↔bank bijection + placeholder-only assertion) |
| `docs/architecture/KEY_FILES.md`, `CLAUDE.md` (+build:llms) | CHANGED |
| `test/bootstrap-*.test.ts`, `test/hook-command.test.ts`, `test/secret-scan.test.ts`, `test/e2e/bootstrap-lifecycle.test.ts`, IPC back-compat tests | NEW |
### Port map (codex-as-agent → gbrain)
interview.mjs → core/bootstrap/interview.ts · render-templates.mjs → core/bootstrap/render.ts
· setup-private-repo.mjs → core/bootstrap/private-repo.ts · verify-install.mjs (+
install-gbrain.mjs round-trip) → core/bootstrap/verify.ts · git-sync.mjs → secret-scan.ts +
sources push · questions.json → templates/bootstrap/questions.json (scrubbed) · AGENTS.md
gates 07 / SOUL/USER/MEMORY/HEARTBEAT/GITHUB templates → REFERENCE structure, REWRITE
content (privacy rule) · install-skills.mjs → REFERENCE (skillpack scaffold exists) ·
cron fleet/cron-doctor → FF4 · codex-plugin-spec one-file-owns-format pattern → governs the
settings.local.json + config.toml writers (single module owns each host format).
### Verification (how we know it works end-to-end)
- Unit: interview gate exit codes; render token hard-fail/no-clobber/backup; secret-scan
fixture corpus (positives + benign lookalikes); hook stdin→JSON contract; IPC v1↔v2
back-compat both directions; questions.json↔template token bijection.
- E2E: full lifecycle in temp dir (sandboxed GBRAIN_HOME, PATH-shimmed fake gh/claude/codex
recording invocations): render → repo → verify exit 0; idempotency (second run no-op);
kill-mid-phase → `status` resumes. **hook-under-serve** and **lock-contention** pins
(permanent). Engine parity for turn-context on both engines.
- Manual acceptance at ship: fresh macOS account, real paste, both harnesses, timed
(≤15 min, ≤3 human actions).
### Resolved user decisions (Garry, 2026-08-07)
- **D1 = A. MCP scope: project scope default, user-scope opt-in** (consent question during
bootstrap; threat named in ACCESS_POLICY.md).
- **D2 = C. DB: global `~/.gbrain` default + documented `--isolated` escape hatch.**
`gbrain bootstrap --isolated` threads `GBRAIN_HOME=<workspace>/.gbrain` through init, MCP
registration env (`claude mcp add -e` / codex config env), and the hook commands in
settings.local.json. Port install-gbrain.mjs's guards: GBRAIN_HOME does NOT isolate
`sync.repo_path` (set it explicitly), strip ambient GBRAIN_DATABASE_URL/DATABASE_URL/
GBRAIN_BRAIN_ID, assert database_path is inside the workspace. `.gbrain/` already
gitignored by the rendered template.
- **D3 = B with consent. 15-min scan-gated commit+push cron is the default posture, but the
runbook ASKS PERMISSION before installing it** ("Enable background persistence? Installs a
15-min launchd/cron job that commits + pushes this workspace, secret-scan-gated"). The
SessionEnd-hook push stays as the always-available, no-daemon backstop and is the fallback
when the cron is declined — persistence never silently disappears. Autopilot posture
unchanged: off on PGLite, recommended on Postgres.
- **D4 = B. ALL bundled skills scaffold in** (`skillpack scaffold --all`). The onboarding
ENDS with a next-steps block in the completion manifest: the inert-skill report (which
installed skills are dormant for lack of which API key), the exact keys to add and where,
and pointers to soul-audit (deepen identity this week) + cold-start (fill the brain with
your data). The routing-table-size concern is mitigated by frontmatter-trigger routing
(authoritative since v0.36) and noted for a future curated-profile fast-follow if dispatch
accuracy suffers in practice.
- **D5 = Codex/Claude-Code-scoped placement.** [Superseded by the 2026-08-09 user
decision — see as-shipped delta 6 and the artifact table's README row.] This is NOT the new headline install —
most users still use GBrain with OpenClaw/Hermes, so `INSTALL_FOR_AGENTS.md` remains the
primary paste path at the top of the README. The bootstrap paste block becomes the
flagship "For Codex" / "For Claude Code" README sections, ahead of the OpenClaw/Hermes path at equal weight (and
docs/tutorials/connect-coding-agent.md cross-links it). Command name stays
`gbrain bootstrap`; paste block is pinned to a release tag (supply-chain integrity).
BOOTSTRAP_FOR_AGENTS.md opens with a scope note: "For Claude Code / Codex. Running
OpenClaw or Hermes? Use INSTALL_FOR_AGENTS.md instead."
+63
View File
@@ -0,0 +1,63 @@
# Agent Bootstrap — Spike Instrument (build order 0)
Manual validation that gates door-1 (Codex/ChatGPT desktop) ship. Run on a machine
the maintainer does not own, with fresh accounts. Owner: the maintainer. Timebox:
~1 week wall-clock. Outcomes feed the design doc's gate
([AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md)).
## Exit questions and pass bars
**#1 (blocking) — the write seam.** What reliably persists memory per turn/session
on the ChatGPT-desktop surface?
- Protocol: run 20 sessions across ≥3 days of ordinary use. Each session must
produce at least one durable write (a page/fact retrievable in the NEXT session).
- Pass: 0 durable-write failures in 20 sessions; else extend to 50 and log every
failure's cause (crash, sleep, approval friction, format drift, model forgot).
- Below the bar → door 1 demotes to documented beta (Codex CLI unaffected).
**#2 — the read seam.** Is injected/pulled context demonstrably present at turn
start? Pass: context block present (or the degraded pull-mode documented as
door-1's v1 behavior). Pass expands the greeting digest to door 1.
**#3 — capability surface.** Record, with screenshots: local folder access
(yes/no/how), MCP registration path (config file? `codex mcp add`? UI?), approval
taps for each toolchain step (count them), connector availability for
email/calendar (yes/no/degraded).
**#4 — quota.** Per harness the doors run on (Claude Code: Max plan; Codex: the
ChatGPT plan): log each day's usage-meter readings during the pilot.
- Load model: ordinary sessions + hooks + one session-triggered schedule.
- Pass: a p90 day consumes ≤10% of the weekly allowance (per-door; one harness
failing cuts schedule scope for that door only).
- Measurement: the harness's own usage UI (screenshot at day start/end) + a tally
of sessions/turns from the transcript dir. No telemetry — this is a manual
instrument by design.
**#5 — TTFM baseline.** One full paste-to-verified install, timed. Count every
human action (paste / auth click / interview answers / consents / approval taps).
Toolchain download time recorded separately (excluded from the 15-minute target).
## Pilot metrics (continue for 2 weeks after the spike)
Per week, from session review (screen recordings + self-report — no telemetry):
correct-write rate (things worth remembering that got written), correct-recall
rate (recalls that were right), false-memory incidents (recalled things that were
wrong), correction round-trips (corrections that stuck as standing rules). The
0-failures-in-20 bar is the minimum to START the pilot, not the proof — the pilot
is the sample.
## Log template (one row per session)
| # | date | door | duration | writes attempted | writes durable | recalls right/wrong | approvals | notes |
|---|---|---|---|---|---|---|---|---|
## Deliverable
A filled copy of this doc committed as `AGENT_BOOTSTRAP_SPIKE_RESULTS.md`
(scrubbed: no real names beyond the maintainer, no account identifiers), plus the
gate decision recorded in the design doc: door-1 ships full / ships as documented
beta / schedule scope cut per quota.
**Gate status:** not yet run — no `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` is committed,
so no gate decision is recorded and door 1 has not been promoted past the
documented-beta bar by this instrument. Update this line when the results land.
+22 -29
View File
@@ -17,11 +17,10 @@ result sets; BrainBench gates the memory behaviors above them, with its own
committed baseline (`evals/brainbench/baselines/main.json`) compared against
MAIN's copy in CI so a PR can't self-approve a regression.
## v0.41 update — the LOOP is now real
## The eval gate loop
Before v0.41, you could capture eval rows and replay them but nothing
stitched them into a gate. `gbrain bench publish` + `gbrain eval gate`
close the loop. Two gates:
`gbrain bench publish` + `gbrain eval gate` stitch captured eval rows into
a pass/fail gate. Two gates:
- **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline
you captured against your current brain. Catches: "did my refactor break
@@ -48,7 +47,7 @@ gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/person
gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson
```
### Privacy posture (D9)
### Privacy posture
**Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real
user captures stay local in `~/.gbrain/baselines/`. The boundary is
@@ -140,14 +139,9 @@ gbrain query "anything" >/dev/null
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
```
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
```json
{"eval": {"capture": true}} // force on
{"eval": {"capture": false}} // force off
```
Explicit config beats the env var both directions.
The full on/off resolution order (config beats env var, both directions) is
documented once in [`eval-capture.md`](./eval-capture.md) — that file is the
capture contract.
## The 4-command loop
@@ -214,7 +208,7 @@ retrieval, and which queries did it move most?"
For a third evaluation axis — public benchmark, ground-truth labels, full
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
<dataset.jsonl>` runs the LongMemEval benchmark against gbrain's
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
imported, the question asked, the hypothesis emitted as JSONL — exactly the
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
@@ -346,7 +340,7 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
## Public benchmarks: LongMemEval (v0.28.8)
## Public benchmarks: LongMemEval
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
benchmark directly against gbrain's hybrid retrieval. Different evaluation
@@ -407,7 +401,7 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
LLM latency.
## Measuring brain consistency over time (v0.32.6)
## Measuring brain consistency over time
`gbrain eval suspected-contradictions` is a complementary measurement
instrument: it samples retrieval results for unmarked semantic
@@ -444,20 +438,19 @@ commands per high-severity finding.
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
decision criteria gated on Wilson CI lower-bound.
## v0.40.1.0 Track D — Eval infrastructure
## Eval infrastructure: by-type breakdowns, the hermetic gate, batch scoring
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
covers the dev loop that uses them and the gates they enforce.
Three further eval surfaces, and the dev loop that uses them.
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
surfaces it in machine-readable form. Two additive changes:
LongMemEval computes per-question-type recall internally, and surfaces it in
machine-readable form:
1. Every per-question JSONL row now includes a `question: string` field so the
1. Every per-question JSONL row includes a `question: string` field so the
`gbrain eval cross-modal --batch` consumer (below) can read it without
joining back against the source dataset.
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
2. The `--by-type` flag emits a final aggregate line keyed by `question_type`:
```json
{"schema_version": 1, "kind": "by_type_summary",
@@ -489,11 +482,11 @@ echo "exit=$?" # 1 if any type fell below 0.80
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
silently regress retrieval." Replaces the original "replay against captured
eval_candidates" design (which Codex caught as non-functional in CI — see
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
real-query version).
The structural fix for "PRs touching `src/core/search/` silently regress
retrieval." A "replay against captured eval_candidates" design can't work in
CI (CI has no captured production queries), so the gate is hermetic; see the
`contributor-mode CI capture` TODO in `TODOS.md` for the deferred
real-query version.
How it works:
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
@@ -508,7 +501,7 @@ How it works:
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
#### Refreshing the qrels fixture (the `Why:` discipline)
When CI fails because a legitimate ranking change moved expected slugs, the
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
+7 -7
View File
@@ -28,7 +28,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
|---|---|---|
| `--limit N` | 100 | Random sample of N takes from the brain. |
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud rather than silently blowing the budget. |
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
@@ -73,8 +73,8 @@ receipt file from disk and re-renders it. The other modes need the brain.
- `schema_version` — locks the contract. Adding optional fields is additive
and compatible. Renaming, removing, or changing semantics bumps the version.
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
(codex review #3). When the rubric definition changes, both fields update,
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch.
When the rubric definition changes, both fields update,
and trend mode groups runs accordingly so a stricter rubric doesn't
silently look like a quality drop.
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
@@ -83,7 +83,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
models in `--models` doesn't change the sha (sort is stable).
- `successes_per_cycle` — count of contributing models per cycle. A model
contributes when (a) its JSON parsed AND (b) every declared rubric dim
has a finite score (codex review #5 missing-dim drops the contribution).
has a finite score (a missing dim drops the whole contribution).
- `verdict``pass` if every dim mean >= 7 AND every dim min across
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
2/3 models contributed complete scores.
@@ -93,11 +93,11 @@ receipt file from disk and re-renders it. The other modes need the brain.
## Receipt persistence
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
Receipts persist to **`eval_takes_quality_runs`** (the DB is authoritative)
AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
as a best-effort artifact. The DB row carries the full receipt JSON in the
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
can still reconstruct via `loadReceiptFromDb`.
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
+10 -9
View File
@@ -1,6 +1,6 @@
# Search Mode Evaluation Methodology
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
_How gbrain measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible — pinned datasets, recorded seeds, and the exact run commands below._
## 1. What this measures and what it doesn't
@@ -21,14 +21,15 @@ If you want to know how a mode behaves on YOUR brain, run `gbrain search stats -
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
- **BrainBench v1**`n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
No private brain content is used in any reported result. The NDJSON run records under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
## 3. Sample selection
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode bundle is the only independent variable. A mode Δ therefore measures the joint effect of every knob the bundles differ on — today that's `tokenBudget`, `expansion`, `relationalRetrieval` (the typed-edge fourth recall arm, ON for balanced/tokenmax, OFF for conservative), and `searchLimit`; the canonical diff is `MODE_BUNDLES` in `src/core/search/mode.ts`.
- **Cache comparability across upgrades.** The query cache keys on a versioned knobs hash (`KNOBS_HASH_VERSION` in `mode.ts`) that folds in the active knob set + embedding column/provider, so one mode's cached results can't be served to another mode's queries — and a version bump makes prior rows unreachable (one-time miss spike). Cross-run comparisons that straddle a knobs-hash bump see a cold cache on the first re-run.
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from a run's `--output` dumps.
## 4. Run procedure
@@ -46,14 +47,14 @@ gbrain eval run-all \
--limit 500 \
--budget-usd-retrieval 5 \
--budget-usd-answer 20 \
--output docs/eval/results/v0.32.3/
--output docs/eval/results/<version>/
# Render the comparison.
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
gbrain eval compare --md > docs/eval/results/<version>/README.md
gbrain eval compare --json > docs/eval/results/<version>/comparison.json
```
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. When a release publishes eval numbers, the `--output` dumps under `docs/eval/results/<version>/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. **No dumps are committed in the repo right now** — reproduce by running the commands above; determinism (§3) means your re-run matches the reported orderings.
## 5. Threats to validity
@@ -68,7 +69,7 @@ Honest list. We name what would let a critic dismiss the numbers.
## 6. Per-question raw outputs
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
Every reported metric is reproducible from the NDJSON dumps a run writes to its `--output` directory (`docs/eval/results/<version>/` when a release publishes numbers; none are committed right now — see §4). The commit SHA in the methodology footer pins the code version.
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
+43 -36
View File
@@ -12,13 +12,13 @@ surfaces**, and which one you pick depends on the operation.
│ gbrain process │
│ │
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
│ │ (HTTP + OAuth) │ │ admin ops │ │
openclaw, fork) ────┼──▶ MCP ops surface │ │ local-only │ │
│ │ (HTTP + OAuth) │ │ commands │ │
│ │ │ │ │ │
│ │ search, query, │ │ sync, embed, │ │
│ │ put_page, │ │ dream, doctor,│ │
│ │ get_page, │ │ autopilot, │ │
│ │ find_experts, │ │ init, secrets │ │
│ │ put_page, │ │ extract, │ │
│ │ get_page, │ │ dream, │ │
│ │ find_experts, │ │ enrich, ... │ │
│ │ ... │ │ │ │
│ └──────────────────┘ └────────────────┘ │
│ ▲ ▲ │
@@ -26,7 +26,7 @@ surfaces**, and which one you pick depends on the operation.
│ │ │ │
│ thin-client OAuth shell-job `inherit:`
│ (preferred for (only path for │
│ MCP-equivalent ops) localOnly ops)
│ MCP-equivalent ops) local-only work)
└─────────────────────────────────────────────┘
```
@@ -45,7 +45,7 @@ the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
The host runs gbrain as a long-lived HTTP server:
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
gbrain serve --http --port 3131
```
The agent registers as an OAuth client (one-time):
@@ -75,22 +75,27 @@ commands through the configured remote MCP. The agent can call
agent to a specific source within a federated brain.
- One audit surface (`mcp_request_log`) covers every op call uniformly.
## Surface 2 — localOnly admin ops via shell-job `inherit:`
## Surface 2 — local-only work via shell-job `inherit:`
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
list (as of v0.36.5.0) includes:
Two mechanisms keep local-only work off the remote surface, and they operate
at different layers:
- `sync` (filesystem walks need local FS access)
- `embed` (orchestrates the embed pipeline)
- `extract` (walks markdown files)
- `dream` (synthesis cycle)
- `doctor` (filesystem hygiene checks)
- `autopilot` (background daemon orchestration)
- `init` (creates `~/.gbrain/`)
- `secrets` (config management)
- **Op layer:** operations flagged `localOnly: true` in
`src/core/operations.ts` are filtered out of the HTTP MCP surface entirely
— a remote caller never sees them.
- **CLI layer:** on a thin-client install (remote MCP configured, no local
engine), commands that require a local engine or the local filesystem are
refused at dispatch with a pinpoint hint naming the closest alternative.
The authoritative set is `THIN_CLIENT_REFUSED_COMMANDS` in `src/cli.ts`
read it there rather than trusting any list copied into a doc; it covers
`sync`, `embed`, `extract`, `dream`, `enrich`, `serve`, `config`, and a
couple dozen more.
For these, the agent cannot route through HTTP MCP. The only path is to run
Notable non-members: `doctor` is NOT refused on a thin client — it reroutes
to an outbound-HTTP probe set (`src/core/doctor-remote.ts`); `bootstrap` and
`hook` are engine-free and work on any install shape.
For refused commands, the agent cannot route through HTTP MCP. The path is to run
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
DLQ / audit trail all come for free.
@@ -114,12 +119,11 @@ full validation rules and error catalog.
### Why this is preferred over writing secrets into `env:` per-job
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
validation. The error message names `inherit: ["database_url"]` as the
replacement.
- Passing `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` per job would
land the URL plaintext in `minion_jobs.data` and the shell-audit JSONL —
visible to anyone with brain-DB read access (or a brain dump, or a shared
brain via mounts). Pre-enqueue validation rejects it; the error message
names `inherit: ["database_url"]` as the replacement.
### Worker setup (one-time, per host)
@@ -146,11 +150,11 @@ proxy for worker env.
| `get_page` / `list_pages` | HTTP MCP | Same. |
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
| `sync` / `embed` / `extract` | Shell job + `inherit:` | Thin-client refused; needs local engine + FS. |
| `dream` | Shell job + `inherit:` | Thin-client refused; synthesis runs on the host. |
| `doctor` | Run directly (any install) | Not refused: thin clients get the remote probe set. |
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
| `init` / `config` | One-time host setup | Operator action, not agent action. |
## Recommended patterns
@@ -166,16 +170,19 @@ proxy for worker env.
- **`env:` still works** for non-secret values, or for cases where you
WANT the value in the row (e.g. an opaque correlation token your audit
flow needs to read back later). The validator doesn't second-guess you.
- **Never try to route a `localOnly` op through thin-client MCP.** It will
fail with `localOnly op refused in thin-client mode`. Use shell-job +
`inherit:` (for secrets) or `env:` (for non-secrets).
- **Never try to route a refused command through a thin client.** The CLI
refuses it at dispatch with a hint. Use shell-job + `inherit:` (for
secrets) or `env:` (for non-secrets) on the host instead.
- **Push-based context.** Beyond request/response ops, MCP clients can
receive volunteered context via the `volunteer_context` op — see
[push-context.md](./push-context.md).
## Migration: from pre-v0.36.5.0
## Migration: from `env:`-passed secrets
If your agent submits shell jobs that pass secrets via `env:`:
```jsonc
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
// Rejected at submit: the URL would persist in minion_jobs.data plaintext.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
@@ -186,7 +193,7 @@ If your agent submits shell jobs that pass secrets via `env:`:
Switch to (recommended):
```jsonc
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
// Name in row, value resolved at child-spawn from worker config.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
+159
View File
@@ -0,0 +1,159 @@
# GBrain Bootstrap — your harness as your agent
`gbrain bootstrap` turns a Claude Code or Codex session into a persistent personal
agent: identity files rendered from your own answers, a local PGLite brain,
per-turn context, session-triggered schedules, and a private GitHub repo as the
agent's durable, portable body. This guide is the full contract — what gets
installed, what runs when, what it can and cannot do, and how to undo all of it.
Normative design docs: [AGENT_BOOTSTRAP_DESIGN.md](../designs/AGENT_BOOTSTRAP_DESIGN.md)
(scope) and [AGENT_BOOTSTRAP_PLAN.md](../designs/AGENT_BOOTSTRAP_PLAN.md)
(implementation). The paste block lives in the README; the runbook your agent
follows is `BOOTSTRAP_FOR_AGENTS.md` at the repo root, fetched at the
`latest-stable` ref.
## What gets installed, exactly
| Piece | Where | Runs when |
|---|---|---|
| Identity files (SOUL/USER/MEMORY/AGENTS/CLAUDE/HEARTBEAT/ACCESS_POLICY/GITHUB) | your workspace folder | loaded at session start |
| `agent.json` manifest + `brain/`, `memory/`, `skills/`, `state/` | workspace | — |
| Local brain (PGLite) | `~/.gbrain/` (never in the repo) | while a session's MCP serve is open |
| MCP registration (`gbrain serve`) | project scope by default | spawned by your harness per session |
| Hooks (Claude Code, ON by default) | `.claude/settings.local.json` (gitignored) | each prompt; fail-open; `--no-hooks` opts out at install, `GBRAIN_HOOKS=0` disables at runtime |
| Session persistence | SessionEnd hook → scan-gated commit+push | at session end |
| Optional 15-min push job | launchd/cron (consent-gated) | while logged in |
| Private GitHub repo | your account, created by `bootstrap repo` | privacy verified via API |
| Machine receipt | `~/.gbrain/bootstrap/receipt.json` | uninstall is keyed to it |
**What does NOT run:** anything while the harness is closed. Session-triggered
schedules fire at turn/session boundaries only. True 24/7 operation is what a
hosted brain provides — this is the honest desktop contract.
## The awake-when-you-are contract
Your agent is awake when your harness is. Laptop asleep = agent asleep. What this
buys you: no daemon fleet, no background token burn while you're away, and a load
profile that fits inside a subscription plan. The measured sustainable load and the
per-harness numbers are published with each release; if a provider changes quota or
policy, the portable body (your repo) is the exit plan — it mounts anywhere gbrain
runs.
## Keyless mode
With zero API keys, everything works: the agent authors memory explicitly through
the brain's write tools (`put_page`, timeline entries, `## Facts` fences — your
harness's model is the LLM, already paid for), and search runs keyword-only
(BM25). `bootstrap verify` prints the capability report honestly. One optional key
(OpenAI, Anthropic, or Voyage) unlocks semantic search and automatic fact
extraction; the key goes to the 0600 config file, never into the repo or the
interview answers. API spend is metered separately from your subscription and is
zero in keyless mode; with a key, the standard spend gates apply
([spend-controls](../operations/spend-controls.md)).
## Security posture
- **Supply chain:** the paste block and install command pin the `latest-stable`
ref — a maintainer-controlled tag advanced only after a release fully publishes.
The runbook carries a version stamp; `bootstrap status` warns on skew. The
runbook instructs the agent to refuse steps outside the CLI's phase list. bun
installs via package manager or checksum-verified download.
- **Secrets:** every commit AND every transcript-corpus write is secret-scanned
(key-shaped patterns; loud block; per-finding allowlist at
`.gbrain-scan-allow`). A deny-glob backstop refuses tracked `*.pglite`/`.env*`
files even if `.gitignore` is damaged. Push refuses public remotes and
unverifiable visibility.
- **Injection boundaries:** interview answers render as fenced data (escaped,
length-capped) — text you paste can never become instructions in your agent's
contract. Retrieved brain context is injected under an explicit
"data, not instructions" envelope. Facts visible to the harness respect the
brain's visibility tiers.
- **Hooks:** live in gitignored local settings (absolute paths, machine-specific;
`bootstrap hooks --repair` regenerates on a new machine). Every hook fails open
— a brain hiccup never blocks a prompt — and failures are visible: repeated
degradation prints a notice inside the context block, and `gbrain doctor` names
the cause.
- **Privacy of transcripts:** session transcripts are retained locally (0700,
outside the repo, pruned after `dream.synthesize.corpus_retention_days`, default
30 — set it in the config file, `~/.gbrain/config.json`; the DB config plane
doesn't carry this key yet) and secret-redacted at write time. They never enter the repo. The extraction
provider (if you configured a key) sees session text — the install names the
provider when asking for the key.
## Honest forget semantics
The repo is git history — append-only. Deleting a line removes it from the working
tree, not from history. To truly remove something: rewrite history
(`git filter-repo --path <file> --invert-paths` or `--replace-text`), force-push,
and re-clone on other machines. `MEMORY.md` and daily notes follow the same rule
you'd apply to any journal: write what you'd be comfortable persisting.
## Degradation matrix
| You declined / lack | What still works | What you lose |
|---|---|---|
| API keys | everything (keyless mode) | semantic search, auto-extraction |
| GitHub / `gh` | full local agent | off-machine durability (repo re-runnable later) |
| Hooks (Claude Code) | pull protocol via AGENTS.md gates | automatic per-turn context + session-end persistence |
| Codex (no hook system) | pull protocol + MCP tools | per-turn push (stated plainly; not oversold) |
| Second simultaneous session | first session unaffected | second session's brain tools fail politely (one live serve per brain — v1 contract) |
## Multi-device
Clone your agent repo on machine two and run `gbrain bootstrap attach` — it
validates the manifest, wires this machine (source registration, hooks repair,
MCP), and verifies. The brain database is derived state, rebuilt from `brain/` +
re-ingestion; hot facts extracted only on machine one arrive via the repo's pages
and fences. Simultaneous editing from two machines is ordinary git conflict
territory — `sources push` pulls divergence-safely (commit first, rebase pull,
loud on conflicts).
## Uninstall
`gbrain bootstrap uninstall` removes exactly what this machine's install receipt
records: hook wiring, MCP registrations (surgically — foreign servers and hooks
survive), and bootstrap-created state. Your repo is never touched — the body
remains yours. The brain database is KEPT by default; `--delete-brain` is offered
only when bootstrap created the brain, offers a facts export first, and enumerates
what it is about to remove. It refuses to run while a session's serve is live.
## If something seems broken
One command: `gbrain doctor`. It covers hook health, push staleness, serve/lock
collisions, schema state, and prints fixes. `gbrain bootstrap status --json` emits
a support blob (versions, harness, last verify/push, hook failure rate) your agent
can relay verbatim when you report a problem.
## Real-agent e2e
Most bootstrap tests drive the dispatcher with PATH-shimmed `claude`/`codex`
recorders — fast, hermetic, no API cost. Two additional "door" tests drive the
ACTUAL binaries end to end so we catch real-world drift (a `codex mcp add` flag
that changed shape, a harness that stopped calling our MCP server):
- `test/e2e/bootstrap-real-claude.serial.test.ts` — real `claude -p` over MCP.
- `test/e2e/bootstrap-real-codex.serial.test.ts` — real `codex exec`. It runs the
keyless-`init` → interview → render → `gbrain bootstrap hooks --harness codex`
path (executing the real `codex mcp add` into a hermetic `~/.codex/config.toml`),
asserts the rendered `AGENTS.md` carries the Gate-3 brain-first pull protocol
(Codex has no hook system, so the pull protocol is its per-turn seam), then
spends one live `codex exec` turn to prove real codex → gbrain MCP → brain →
a seeded, brain-only fact (falling back to a shell `gbrain query` if headless
stdio-MCP is unavailable).
These pay real API cost and take 30s2min per turn, so they are NOT in the PR
shard. Everything is hermetic (temp `HOME` / `CODEX_HOME` / `CLAUDE_CONFIG_DIR` /
`GBRAIN_HOME` per test — the operator's real `~/.claude`, `~/.gbrain`, `~/.codex`
are never touched; auth is copied read-only). Each file self-SKIPS via
`describe.skipIf` when its binary or auth is absent, so on a machine without the
tool it is a clean no-op that never fails. CI wires them into the `real-agent-e2e`
job in `.github/workflows/heavy-tests.yml` (nightly + the `real-agent-e2e` /
`heavy-tests` label); on a stock runner they self-skip. To actually exercise the
binaries you need a runner with authed `claude`/`codex` and the provider creds
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`) exported.
Run locally (where both are installed + authed):
```bash
bun test test/e2e/bootstrap-real-codex.serial.test.ts
```
+8 -5
View File
@@ -27,6 +27,7 @@ READ: check brain FIRST (before responding)
│ → gbrain search "{entity name}"
│ → gbrain get {slug} (if you know it)
│ → gbrain query "what do we know about {topic}"
│ → full protocol: brain-first-lookup.md
RESPOND with brain context (every answer is better with context)
@@ -100,12 +101,14 @@ on_message(text):
Write immediately after the conversation, while the context is fresh.
3. **Sync after every write batch.** Without sync, the brain search index is
stale. The next query won't find what you just wrote.
stale. The next query won't find what you just wrote. On installs set up
via `gbrain bootstrap`, per-turn context injection and session-end
persistence hooks automate parts of this loop — see
[bootstrap.md](bootstrap.md) and [push-context.md](push-context.md).
4. **External APIs are fallback, not primary.** `gbrain search` before
Brave Search. `gbrain get` before Crustdata. The brain has relationship
history, your own assessments, meeting transcripts, cross-references.
No external API can provide that.
4. **External APIs are fallback, not primary.** `gbrain search` before any
web or enrichment API. The full brain-before-external protocol (and why)
lives in [brain-first-lookup.md](brain-first-lookup.md).
## How to Verify It Works
+12 -9
View File
@@ -51,22 +51,25 @@ The brain has context no external API can provide:
- Timeline (what changed recently, what's trending)
A LinkedIn scrape gives you their job title. The brain gives you: "co-founded
Brex, you had coffee with him 3 times, last discussed the payments infrastructure
thesis, he's interested in your take on AI agents."
widget-co, you had coffee with her 3 times, last discussed the payments
infrastructure thesis, she's interested in your take on AI agents."
## Tricky Spots
1. **Try keyword first, then hybrid.** Keyword search works without embeddings
(day one). Hybrid search needs embeddings but finds semantic matches. Try
both in sequence.
(day one — and it's ALL you get in keyless mode, see
[bootstrap.md](bootstrap.md)). Hybrid search needs embeddings but finds
semantic matches. Try both in sequence.
2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact
slug doesn't exist, it suggests alternatives. Use this for name variants
("Pedro" → "pedro-franceschi").
2. **Fuzzy slug matching is opt-in.** Pass `--fuzzy` (the `fuzzy: true` param
on `get_page`) and a near-miss slug resolves to the unique candidate, or
returns an `ambiguous_slug` error listing the candidates. WITHOUT the flag
a miss just throws `page_not_found` (with a hint to retry with
`fuzzy: true`). Use it for name variants ("Alice" → "alice-example").
3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?"
should check the brain first. The brain might have it, and the lookup adds
no latency (< 100ms for keyword search).
should check the brain first. The brain might have it, and a keyword
lookup is fast enough to be effectively free.
4. **Load compiled truth + recent timeline.** The compiled truth gives you the
state of play in 30 seconds. The timeline gives you what changed recently.
+5 -4
View File
@@ -17,8 +17,8 @@ on new_information(info):
# This is world knowledge -- facts about entities external to the agent
gbrain put <slug> --content "..."
# Examples:
# "Pedro is CEO of Brex" -> gbrain (person page)
# "Brex raised Series D at $12B" -> gbrain (company page)
# "alice-example is CEO of widget-co" -> gbrain (person page)
# "widget-co raised Series D at $12B" -> gbrain (company page)
# "Tuesday's meeting covered Q2" -> gbrain (meeting page)
# "The meatsuit maintenance tax" -> gbrain (originals page)
@@ -57,15 +57,16 @@ on user_asks(question):
## Tricky Spots
1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world.
1. **Don't store people in agent memory.** "alice-example prefers email over Slack" feels like a preference, but it's a fact about Alice -- it goes in GBrain on her page. Agent memory is for the agent's own operational state, not facts about people in the world.
2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration.
3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory.
4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything.
On installs set up via `gbrain bootstrap`, "agent memory" has a concrete file form: MEMORY.md and the other identity files in the agent repo (see [bootstrap.md](bootstrap.md)). The routing rule is unchanged -- those files hold operational state and identity, not world knowledge.
5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session.
## How to Verify
1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
1. Ask the agent "Who is alice-example?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state.
3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers.
4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory).
+16 -6
View File
@@ -44,12 +44,12 @@ Sharp technical leader. Under-appreciated internally. Watch for signs of burnout
Ascending. Likely CTO track if the migration succeeds.
## Relationship
Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis.
Met through alice-example. Had coffee 3x. Last: discussed API architecture thesis.
## Contact
sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
---
<!-- timeline -->
## Timeline
@@ -58,7 +58,7 @@ sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
[Source: Meeting notes, 2026-04-07 2:00 PM PT]
- **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops.
[Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT]
- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background.
- **2026-03-15** | First meeting. Intro from alice-example. Strong technical background.
[Source: User, direct conversation, 2026-03-15 3:00 PM PT]
```
@@ -113,9 +113,19 @@ support that claim.
truth chunks with higher relevance than timeline chunks. This means the freshest
synthesis surfaces first in search results.
4. **The --- separator matters.** GBrain uses the first standalone `---` after
frontmatter to split compiled_truth from timeline. Everything above is compiled
truth, everything below is timeline.
4. **The timeline sentinel matters — and a bare `---` is NOT one.** GBrain
splits compiled_truth from timeline at the first recognized sentinel, in
order of precedence:
1. `<!-- timeline -->` — preferred; unambiguous, and what GBrain itself
emits when it writes a page.
2. `--- timeline ---` — decorated separator.
3. `---` ONLY when the next non-empty line is `## Timeline` or
`## History` (backward-compat for older gbrain-written files).
A plain `---` line anywhere else is a markdown horizontal rule, not a
separator. Author new pages with `<!-- timeline -->` (as in the example
above); everything above it is compiled truth, everything below is
timeline.
5. **Don't skip the Assessment section.** The assessment is the value. "Strong
technical leader" is something no API can provide. It's YOUR read on this
+7 -1
View File
@@ -8,6 +8,12 @@ Without this: media links are bookmarks that decay -- you remember watching a vi
## Implementation
gbrain's own media surfaces complement this pattern: the bundled
`media-ingest` skill (`skills/media-ingest/`) ships the ingestion workflow,
and `gbrain files` handles binary/file upload for attachments that should
live alongside pages. For meeting recordings specifically, see
[meeting-ingestion.md](meeting-ingestion.md).
```
on user_shares_media(url_or_file):
@@ -119,7 +125,7 @@ on user_shares_media(url_or_file):
## Tricky Spots
1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it.
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain.
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "The speaker made a specific claim about compute scaling that contradicts what another researcher said in their NeurIPS talk -- see media/youtube/a-researcher-neurips-2025" is useful. The analysis connects the new media to the existing brain.
3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page.
4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry.
5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power.
+37 -16
View File
@@ -26,6 +26,27 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
| Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) |
| Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below |
### Prefer gbrain's native schedulers where they fit
System cron is the lowest common denominator, but gbrain ships its own
scheduling surfaces — reach for these first:
- **`gbrain dream`** — the shipped nightly maintenance cycle (lint,
backlinks, extract, sync, embed, synthesize). Schedule THIS instead of
hand-rolling the dream cycle below.
- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry,
backoff, and an audit trail. See the `minion-orchestrator` skill.
- **`gbrain autopilot`** — the long-lived background daemon that runs cycles
on its own cadence.
- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to
manage its harness's scheduler.
- **Bootstrap session-triggered schedules**`gbrain bootstrap` installs
HEARTBEAT.md-driven schedules that fire on session activity; see
[bootstrap.md](bootstrap.md).
For scheduling `sync` + `embed --stale` specifically, the home doc is
[live-sync.md](live-sync.md).
## Implementation: Setting Up Cron Jobs
```bash
@@ -50,18 +71,12 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
### Quiet Hours Gate (MANDATORY)
Every cron job that sends notifications MUST check quiet hours first.
See [Quiet Hours](quiet-hours.md) for the full pattern.
```bash
# In every cron script:
if ! bash scripts/quiet-hours-gate.sh; then
mkdir -p /tmp/cron-held
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
exit 0
fi
# Not quiet hours — send normally
```
Every cron job that sends notifications MUST check quiet hours first. The
gate is a small script YOU create (it doesn't ship with gbrain) and call at
the top of every notification-sending cron script; held output goes to a
holding directory that the morning briefing drains. See
[Quiet Hours](quiet-hours.md) for the gate script and the full pattern —
don't copy a snippet from here, that page is the single home.
### Travel-Aware Timezone Handling
@@ -88,6 +103,12 @@ morning briefing. Zero config change needed.
The most important cron job. Runs while you sleep.
**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle
(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule
it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is
covered. The pseudocode that follows is the harness-side variant for agents
that also do LLM-driven entity sweeps and memory consolidation on top.
### What It Does
```
@@ -150,11 +171,11 @@ echo "Dream cycle starting at $(date)"
# Phase 1: Entity sweep (spawn sub-agent)
# Read today's conversation logs, extract entities, update brain
# Phase 2: Citation hygiene
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize)
gbrain dream
# Phase 3: Embed any stale content
gbrain embed --stale
# Phase 3: Surface anything the cycle flagged
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
echo "Dream cycle complete at $(date)"
```
+4 -1
View File
@@ -143,4 +143,7 @@ the same pass.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The
[email-to-brain recipe](../../recipes/email-to-brain.md) implements this
collector pattern end-to-end; [cron-schedule.md](cron-schedule.md) covers
scheduling it.*
+4 -1
View File
@@ -53,6 +53,8 @@ Write extracted content to `brain/diligence/{company}/{doc-name}.md`:
**Step 5: Save Raw Files.**
Copy original PDFs/files to `brain/diligence/{company}/.raw/`
Preserve originals for reference. The diarized version is for search.
This is safe by design: `.raw/` directories are excluded from sync, so the
originals never enter the search index — only your extracted markdown does.
**Step 6: Create or Update index.md.**
Every diligence directory needs an `index.md`:
@@ -103,7 +105,8 @@ cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} in
**Step 9: Publish (if asked).**
When the user wants a shareable brief, create a password-protected
published version. Strip internal notes and raw assessment language.
published version via the `publish` skill (`skills/publish/`). Strip
internal notes and raw assessment language.
### Quality Bar
+2 -2
View File
@@ -7,8 +7,8 @@ sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
but it is provider-agnostic: any configured `provider:model` works as a
target.
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
README reference).
Also reachable as `gbrain retrieval-upgrade` the alias that `gbrain doctor`
repair hints and the README point at.
## Quick start
+10 -4
View File
@@ -8,6 +8,11 @@ Without this: brain pages are thin shells with only what the user manually typed
## Implementation
gbrain ships both halves of this: `gbrain enrich` is the batch enrichment
primitive (finds thin pages and enriches at scale), and the `enrich` skill
(`skills/enrich/`) is the agent-driven page-at-a-time workflow. The pipeline
below is the pattern they implement — use it to customize or extend.
```
on enrich(entity, trigger):
# trigger: meeting mention, email thread, social interaction, user request
@@ -68,9 +73,10 @@ on enrich(entity, trigger):
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
# Executive Summary, State, What They Believe, What They're Building,
# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline
# People page sections: use the person-page structure from compiled-truth.md
# (Executive Summary, State, What They Believe, ... Timeline) -- that doc is
# the single home for the section taxonomy. Enrichment can add texture
# sections on top (What Motivates Them, Hobby Horses, Open Threads).
# Facts are table stakes. TEXTURE is the value.
# Extract texture, not just facts:
@@ -100,4 +106,4 @@ on enrich(entity, trigger):
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Compiled Truth](compiled-truth.md) for the person-page section taxonomy, [Spend Controls](../operations/spend-controls.md) for gbrain's own embedding/LLM spend gates.*
+21 -9
View File
@@ -7,10 +7,10 @@ Every inbound message gets scanned for original thinking AND entity mentions so
## What the User Gets
Without this: the agent answers questions but forgets everything. You mention
Pedro in a meeting, and next week the agent doesn't know who Pedro is.
Alice in a meeting, and next week the agent doesn't know who Alice is.
With this: every person, company, and idea mentioned in conversation gets a
brain page. Next time Pedro comes up, the agent already has context. The
brain page. Next time Alice comes up, the agent already has context. The
brain compounds.
## Implementation
@@ -18,6 +18,11 @@ brain compounds.
Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it
to finish before responding. It runs in parallel.
This pattern is harness-side by design, but gbrain ships help on both ends:
the `signal-detector` skill (`skills/signal-detector/`) is the bundled
version of this detection loop, and `gbrain extract` runs gbrain's own
extraction machinery (entities, facts) over already-synced content.
```
on_every_message(message_text, source_context):
@@ -109,6 +114,7 @@ is_notable(entity):
| Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions |
| Hot takes with reasoning | Routine operational messages |
| Metaphors that reveal new angles | Requests without embedded insight |
| Emotional/psychological insights about self or others | |
### Filing Rules
@@ -116,11 +122,17 @@ is_notable(entity):
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| Person mentioned | `brain/people/{slug}.md` |
| Company mentioned | `brain/companies/{slug}.md` |
| Media referenced | `brain/media/{type}/{slug}.md` |
| Article ABOUT the user | `brain/media/writings/{slug}.md` |
This table is the single home for the capture/filing taxonomy. Other guides
([idea-capture](idea-capture.md) especially) link here rather than carrying
their own copy.
### The Iron Law of Back-Linking
@@ -128,21 +140,21 @@ Every entity mention MUST create a back-link FROM the entity page TO the
source. This is not optional.
```
// When message mentions "Pedro" and creates a meeting page:
// When message mentions "Alice" and creates a meeting page:
// 1. Update the meeting page (normal)
brain/meetings/2026-04-10-board-sync.md:
- Pedro presented Q1 numbers
- Alice presented Q1 numbers
// 2. ALSO update Pedro's page (back-link)
brain/people/pedro-franceschi.md:
// 2. ALSO update Alice's page (back-link)
brain/people/alice-example.md:
## Timeline
- **2026-04-10** | Presented Q1 numbers at board sync
[Source: User, board meeting, 2026-04-10]
```
Without back-links, you can't traverse the graph. "Show me everything related
to Pedro" only works if Pedro's page links back to every mention.
to Alice" only works if Alice's page links back to every mention.
## Tricky Spots
@@ -163,7 +175,7 @@ to Pedro" only works if Pedro's page links back to every mention.
5. **Dedup before creating.** Always `gbrain search` before creating a page.
Variant spellings, nicknames, and company abbreviations cause duplicates.
"Pedro Franceschi" and "Pedro" might be the same person.
"Alice Example" and "Alice" might be the same person.
## How to Verify
@@ -182,7 +194,7 @@ to Pedro" only works if Pedro's page links back to every mention.
4. **Send a boring message.** Say "ok sounds good." Verify: nothing was
created. The detector should report "No signals detected."
5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi."
5. **Check for duplicates.** Mention "Alice" then later "Alice Example."
Verify: one page, not two.
---
+9 -4
View File
@@ -8,6 +8,11 @@ Without this: the agent triages email mechanically ("you have 12 unread"), preps
## Implementation
Before hand-rolling these: gbrain bundles the morning-briefing half of this
pattern as the `briefing` skill (`skills/briefing/`) and the task-prep half
as `daily-task-prep` (`skills/daily-task-prep/`). Use the workflows below to
extend or customize what those skills already ship.
```
# WORKFLOW 1: Email Triage
on email_batch(emails):
@@ -59,8 +64,8 @@ on upcoming_meeting(meeting):
briefing[attendee] = "No brain page -- consider enriching"
# Surface: shared history, what to follow up on, what to watch for
# "Last time you discussed the Series B timeline. Pedro was concerned
# about burn rate. Here's the latest from his company page."
# "Last time you discussed the Series B timeline. alice-example was
# concerned about burn rate. Here's the latest from her company page."
# WORKFLOW 3: Post-Inbox Brain Updates
on inbox_cleared():
@@ -93,9 +98,9 @@ on schedule_request(meeting):
1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical.
2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise.
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate."
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with alice-example -- last time you discussed the Series B, she was concerned about burn rate."
4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip.
5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
5. **Scheduling nudges require timeline data.** "You haven't met with charlie-example in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
## How to Verify
+13 -39
View File
@@ -61,14 +61,13 @@ capture_idea(message_text, source_context):
### The Authorship Test
| Signal | Destination |
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
| Article ABOUT user | `brain/media/writings/` |
Who authored the idea determines where it files: user-generated ideas,
syntheses, and ghostwritten work go to `brain/originals/`; borrowed world
concepts to `brain/concepts/`; product ideas to `brain/ideas/`; articles
ABOUT the user to `brain/media/writings/`. The full filing-rules table (and
the what-counts-as-original-thinking criteria) lives in
[entity-detection.md](entity-detection.md) — the single home for the capture
taxonomy.
### Capture Standards
@@ -78,21 +77,6 @@ capture_idea(message_text, source_context):
"tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase.
The vivid version is the real version.
**What counts as worth capturing:**
- Original observations about how the world works
- Novel connections between disparate things
- Frameworks and mental models
- Pattern recognition moments ("I keep seeing X in every Y")
- Hot takes with reasoning behind them
- Metaphors that reveal new angles
- Emotional/psychological insights about self or others
**What does NOT count:**
- Routine operational messages ("ok", "do it")
- Pure questions without embedded observations
- Echoing back something the agent said
- Acknowledgments and reactions
### The Depth Test
**Could someone unfamiliar with the user read this page and understand not
@@ -137,21 +121,9 @@ Every original MUST link to:
### Notability Filtering
Before creating any entity page, check notability:
**Create a page for:**
- People you know or discuss with specificity
- Companies you're evaluating, working with, or investing in
- Media you mention with personal reaction
- Anyone you've explicitly engaged with
**Don't create pages for:**
- Generic references or passing examples
- Low-engagement accounts who mentioned you once
- Pure metaphors ("like the Roman Empire...")
- One-off encounters with no follow-up
**Decision:** If notable AND no page exists, create a full page with web
Before creating any entity page, check notability — the full create/skip
criteria live in [entity-detection.md](entity-detection.md#notability-filtering).
The decision rule: if notable AND no page exists, create a FULL page with web
search enrichment. No stubs. If you make a page, make it good.
## Tricky Spots
@@ -187,4 +159,6 @@ search enrichment. No stubs. If you make a page, make it good.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The bundled
`idea-ingest` skill (`skills/idea-ingest/`) ships this workflow. See also:
[Entity Detection](entity-detection.md).*
+22 -5
View File
@@ -43,7 +43,12 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
- `gbrain sync --repo <path>` -- one-shot incremental sync. Detects changes via
`git diff`, imports only what changed. For small changesets (<= 100 files),
embeddings are generated inline during import.
embeddings are generated inline during import — unless the inline cost gate
intervenes: when the estimated embedding spend crosses the configured floor
in a non-interactive session (cron, `--json`), sync auto-defers embeds to a
capped `embed-backfill` job instead of spending silently. Either way the
chunks get embedded; a deferred run just finishes asynchronously. See
[spend controls](../operations/spend-controls.md).
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
@@ -97,15 +102,27 @@ Triggers sync on push events for instant sync (<5s).
### What Gets Synced
Sync only indexes "syncable" markdown files. These are excluded by design:
- Hidden paths (`.git/`, `.raw/`, etc.)
- The `ops/` directory
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`
- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees
(`node_modules/`, `dist/`, `build/`, `venv/`)
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md`
### Sync is Idempotent
Everything else is ordinary synced content — including `ops/` (the bundled
daily-task-manager skill files its canonical page under `ops/tasks`).
### Sync is Idempotent — and Resumable
Concurrent runs are safe. Two syncs on the same commit no-op because content
hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
Long syncs also survive being killed: progress checkpoints into the database
as files drain, so a killed or aborted run resumes from where it stopped, and
the sync bookmark only advances on true completion. A progress-aware stall
watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts
a run that stops making forward progress and releases the per-source lock so
the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence
and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env
vars — incident-time escape hatches, not everyday knobs.
## Tricky Spots
1. **Always chain sync + embed.** Running `gbrain sync` without
+13 -2
View File
@@ -58,15 +58,20 @@ on new_meeting_transcript(meeting):
# Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings
# Source: Circleback (https://circleback.ai) or any service with
# speaker diarization + API/webhook access
# Automation: the built-in `extract-timeline-from-meetings` Minion job
# automates step 3 (entity timeline propagation) for already-ingested
# meeting pages: gbrain jobs submit extract-timeline-from-meetings --follow
```
## Tricky Spots
1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth.
2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation.
3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room.
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
3. **Mentioned people are not just attendees.** If the meeting discussed "Alice's team at widget-co," then Alice's page AND widget-co's page need updates -- even though Alice wasn't in the room.
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Alice pushed back on the burn rate, Charlie didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always.
6. **`--source` on `timeline-add` is the citation text, not source routing.** Because the op declares its own `source` parameter, the CLI binds `--source` to it. To write into a different registered source, use the `.gbrain-source` dotfile or `GBRAIN_SOURCE` env for routing instead.
## How to Verify
@@ -76,5 +81,11 @@ on new_meeting_transcript(meeting):
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
## Related
- `skills/meeting-ingestion/SKILL.md` — the bundled, agent-executable skill
for this workflow (the canonical step-by-step home; this guide is the
pattern overview).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+29 -22
View File
@@ -40,6 +40,10 @@ gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
# Optional: cap worker memory in MB (--max-rss). Without the flag the RSS
# watchdog is still on, at a RAM-relative auto-sized cap.
gbrain jobs supervisor --concurrency 4 --max-rss 4096
```
**Exit codes:**
@@ -50,9 +54,11 @@ gbrain jobs supervisor stop
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
| 4 | Queue-scoped DB lock lost mid-run (`LOCK_LOST` — exited rather than risk a split-brain) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=1 should page a human.
exit=4 as "restart me — the DB lock refresh failed"; exit=1 should page
a human.
### Lowering scheduling priority (`--nice`)
@@ -233,7 +239,7 @@ use a dedicated queue name like `nightly-enrich` above.
## Upgrading from an older deployment
### From `minion-watchdog.sh` (pre-v0.20)
### From `minion-watchdog.sh`
Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
@@ -270,10 +276,10 @@ Regardless of which deployment path you're upgrading from:
in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** from v0.14 onward, pass
`--allow-shell-jobs` to the supervisor (or keep
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
need the flag; only the worker does.
3. **If you run shell jobs:** pass `--allow-shell-jobs` to the
supervisor (or keep `GBRAIN_ALLOW_SHELL_JOBS=1` in
`/etc/gbrain.env`). Submitters don't need the flag; only the worker
does.
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
should show no unexplained growth in `dead` between pre- and
@@ -283,29 +289,30 @@ Regardless of which deployment path you're upgrading from:
### Supabase connection drops
The worker uses a single Postgres connection. If Supabase drops it
(maintenance, connection limits, network blip), lock renewal fails
silently. The stall detector then dead-letters the job after
`max_stalled` misses.
If Supabase drops the worker's Postgres connection (maintenance,
connection limits, network blip), this now self-heals under the
supervisor: the worker's DB-liveness probe self-exits (`db_dead`) on a
dead pool and the supervisor respawns it with a fresh pool, and the
supervisor also restarts a worker that stops making progress while
claimable work waits. The escalation commands and thresholds live in the
[queue operations runbook](queue-operations-runbook.md) — that's the
canonical home for wedge recovery.
**Current defaults that make this worse:**
What can still bite: a *brief* blip during a long-running job can make
lock renewal miss, and the stall detector dead-letters the job after
`max_stalled` misses (schema column default 5; lock duration and stall
check interval are both 30 s).
- `lockDuration: 30000` (30 s) — too short for long jobs during
connection blips.
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`,
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today.
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags.
These write onto the job row at submit time — which is what
`handleStalled()` reads — so per-job tuning is the real knob.
### DO NOT pass `maxStalledCount` to `MinionWorker`
It's a no-op. The stall detector reads the row's `max_stalled` column
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
(set at submit time), not the worker opt in `src/core/minions/worker.ts`.
Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children
+9 -6
View File
@@ -1,5 +1,9 @@
# Minions fix — repairing a half-migrated install
> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're
> on any recent release, the canonical fix below (`gbrain apply-migrations
> --yes`) is all you need; the stopgap sections exist for archaeology.
**tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially
set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs
still on `agentTurn`), run:
@@ -34,17 +38,16 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`.
gbrain doctor
```
If the install is half-migrated, you'll see:
If the install is half-migrated, you'll see the `minions_migration` check
fail:
```
[FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes
```
or
```
[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes
```
(Missing `~/.gbrain/preferences.json` on a fresh install is a valid
pre-`apply-migrations` state — doctor deliberately does NOT fail on that
alone; the partial-migration record is the canonical half-migration signal.)
For a machine-readable report (cron-friendly):
+6 -7
View File
@@ -129,11 +129,10 @@ JSONL records the same. Pre-enqueue validation rejects the submission if the
worker can't resolve the requested key, with a paste-ready
`gbrain config set database_url <value>` hint.
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
wrote things like:
**Why not just write the URL into `env:` directly?** You *can*:
```jsonc
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
// ❌ Works, but plants the secret in the job row. Prefer inherit:.
{
"cmd": "gbrain stats",
"cwd": "/data/gbrain",
@@ -141,14 +140,14 @@ wrote things like:
}
```
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
This plants plaintext secrets in `minion_jobs.data` (DB row) and in the
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
forbid that pattern — the validator trusts the agent — but **prefer
or a shared brain via the mounts feature) sees the URL. The validator
doesn't forbid the pattern — it trusts the agent — but **prefer
`inherit:`** for any secret you want kept out of the row. Names land in the
row; values resolve at child-spawn from the worker's config.
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
**Scope:** `inherit:` is **free-form**. Pass any snake_case
config-key name and the worker resolves the value from `loadConfig()` at
child-spawn time:
+34 -19
View File
@@ -5,6 +5,11 @@ is a `source`: a logical brain-within-the-brain with its own slug
namespace, its own sync state, and its own federation policy. The rest
of this guide walks the three canonical scenarios.
(Sources are the *within-one-database* axis. If you want to connect a
whole separate database — a team-published brain with its own access
policy — that's the *brain* axis: `gbrain mounts add`. See
`docs/architecture/brains-and-sources.md` for the two-axis topology.)
## The three scenarios
### 1. Unified knowledge recall (wiki + gstack)
@@ -66,14 +71,14 @@ gbrain search "tech layoffs" --source yc-media,garrys-list
### 3. Mixed (wiki federated + sessions isolated)
Your main wiki is federated with a few trusted sources. Your session
transcripts (coming in v0.18) land in a separate isolated source so
they don't dominate every search result.
transcripts (`gbrain transcripts` ingests them) land in a separate
isolated source so they don't dominate every search result.
```bash
# Federated sources
gbrain sources add gstack --path ~/.gstack --federated
# Isolated source (future v0.18 — sessions use this shape today for ingest)
# Isolated source for session transcripts
gbrain sources add sessions --path ~/.claude/sessions --no-federated
```
@@ -104,14 +109,16 @@ Every source row stores `config.federated: boolean` in its JSONB config.
| `true` | Source participates in unqualified `gbrain search "X"` results. |
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
The seeded `default` source is `federated=true` so pre-v0.17 brains
behave exactly as before — every page appears in search.
The seeded `default` source is `federated=true` so single-source brains
behave as you'd expect — every page appears in search.
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
## Commands
Full subcommand reference:
The most-used subcommands (run `gbrain sources --help` for the full,
always-current reference — it also covers `status`, `current`,
`set-cr-mode`, and the `push`/`pull` durability surface):
```
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
@@ -119,9 +126,17 @@ gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--
--path must be a git repo (or a subdirectory of one) — see
"The git requirement for --path sources" below. --force
skips that check to register before git-init exists.
gbrain sources add <id> --url <git-url> [--pat-file <p>] [--clone-dir <path>] [--no-harden]
Clone + register a remote repo in one step; auto-hardens
for durability when a PAT is provided (see "Durability" below).
gbrain sources list [--json] List all sources with page counts + federation state.
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
Cascade-delete a source (pages, chunks, timeline).
gbrain sources archive <id> Soft-delete: hide from search, keep data for a TTL
grace window. Prefer this over `remove`.
gbrain sources restore <id> Un-archive. `gbrain sources archived` lists expiries;
`gbrain sources purge` permanently deletes expired archives.
gbrain sources remove <id> [--confirm-destructive] [--dry-run]
Permanently cascade-delete a source (pages, chunks,
timeline). Shows an impact preview first.
gbrain sources rename <id> <new-name>
Change display name only; id is immutable.
gbrain sources default <id> Set the brain-level default.
@@ -253,8 +268,8 @@ reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
## Upgrading an existing brain
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
existing pages all move under `source_id='default'`. Behavior is
`gbrain upgrade` runs the needed schema migrations automatically. Your
existing pages all live under `source_id='default'`. Behavior is
unchanged until you add a second source.
To add one:
@@ -266,13 +281,13 @@ cd ~/.gstack && gbrain sources attach gstack && gbrain sync
Two commands. The existing default source is untouched.
## Not in v0.18.0
## Related features that build on sources
- Session transcript ingest (`.jsonl`, raised size cap, session
PageType) — v0.18.
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
- ACL enforcement via caller-identity — v0.17.1.
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
release after the core plumbing stabilizes.
All of these build on the `sources` primitive shipped here.
- **Session transcript ingest**`gbrain transcripts` (server-private:
raw chat exports stay on the host machine).
- **Per-source retention**`gbrain sources archive` / `archived` /
`purge` (soft-delete with a TTL grace window).
- **One-shot remote bootstrap**`gbrain sources add <id> --url <git-url>`
(clone + register + auto-harden).
- **Access control across brains** — the *brain* axis (`gbrain mounts`);
see `docs/architecture/brains-and-sources.md`.
+13 -10
View File
@@ -43,13 +43,14 @@ on information_needed(topic):
# An agent that reaches for the web before checking its own brain
# is wasting money and giving worse answers.
# DISCIPLINE 3: Sync After Every Write (MANDATORY)
on brain_write_complete():
# DISCIPLINE 3: Sync After Every Repo Write (MANDATORY)
on brain_repo_files_changed():
gbrain sync
# Without this, search results are stale.
# The page you just wrote won't appear in gbrain search or gbrain query
# until sync runs. Skipping this means the next lookup misses the
# most recent data.
# `gbrain put` indexes immediately -- pages written through the CLI/MCP
# are searchable the moment the command returns. No sync needed there.
# But files written DIRECTLY to the brain repo (an editor, a script,
# another agent committing markdown) are invisible to search until
# `gbrain sync` imports them. If anything touched repo files, sync.
# DISCIPLINE 4: Daily Heartbeat Check
on daily_schedule("09:00"):
@@ -62,6 +63,8 @@ on daily_schedule("09:00"):
on nightly_schedule("02:00"):
# The dream cycle is the most important discipline.
# The brain COMPOUNDS overnight.
# gbrain ships this as a first-class command: `gbrain dream`.
# The pseudocode below is the shape of the work it does.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list
@@ -103,16 +106,16 @@ on nightly_schedule("02:00"):
## Tricky Spots
1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots.
2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes.
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context.
2. **Skipping Discipline 3 (sync after repo writes) means stale search results.** A file lands in the brain repo, then you search for it -- and get nothing back. The file exists but isn't imported. Always sync after repo-file writes. (`gbrain put` is exempt: it indexes on write.)
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Alice yesterday about the board seat" in passing -- that's a timeline entry on Alice's page, a potential update to her State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Alice Example" returns a LinkedIn profile. The brain returns the full picture including private context.
5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss.
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
3. Write a markdown file directly into the brain repo (not via `gbrain put`), run `gbrain sync`, then `gbrain search` for it. Confirm it appears in results (verifies the sync discipline). A `gbrain put` page should appear in search immediately, with no sync.
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
+14 -10
View File
@@ -1,4 +1,4 @@
# Plugin authors guide (v0.15)
# Plugin authors guide — subagent definitions
`gbrain` discovers subagent definitions from outside this repo via
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
@@ -7,6 +7,11 @@ subagents alongside it, drop a plugin directory on that env path.
This guide is for plugin authors. The CLI user doesn't need to read it.
> **Two plugin systems.** This doc covers *subagent definitions*
> (markdown prompts the `subagent` job handler runs). Custom *job
> handlers* (code the Minion worker executes) are a separate system —
> see [plugin-handlers.md](plugin-handlers.md).
## Minimum viable plugin
```
@@ -65,7 +70,7 @@ You control where your plugin lives on disk; `gbrain` doesn't guess.
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
with a warning naming both sources.
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
**Trust policy.** Plugins ship subagent definitions ONLY:
- You **cannot** declare new tools.
- You **cannot** extend the brain tool allow-list.
@@ -76,8 +81,8 @@ with a warning naming both sources.
your plugin gives you a loud startup error, not a silent "tool never
fires" at 3am.
v0.16+ may open up plugin-declared tools with a separate contract. Don't
expect it.
Plugin-declared tools would require a new `plugin_version` contract;
nothing under `gbrain-plugin-v1` opens that up.
## `gbrain.plugin.json`
@@ -85,9 +90,9 @@ expect it.
|------------------|--------|----------|--------------------------------------------------------------------|
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"`. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Shown in a future plugin-listing command. |
| `description` | string | no | Informational; appears in load/collision warnings. |
## Subagent definition files
@@ -103,8 +108,7 @@ Recognized frontmatter fields:
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
may consume more of them.
Unknown frontmatter fields are preserved but ignored by the handler.
## Caveats that will bite you
@@ -115,8 +119,8 @@ may consume more of them.
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
worker runs on a different host than the `gbrain agent logs` caller,
the CLI won't see heartbeats from that worker. v0.16 will unify this;
for now assume worker + CLI share a filesystem.
the CLI won't see heartbeats from that worker. Assume worker + CLI
share a filesystem.
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
invocation. Tools that gate on `remote=true` (file_upload's strict
+26 -7
View File
@@ -1,7 +1,12 @@
# Plugin handlers — registering host-specific Minion handlers
GBrain's Minion worker ships with seven built-in handlers: `sync`,
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
GBrain's Minion worker ships with a full set of built-in handlers,
registered by `registerBuiltinHandlers` in `src/commands/jobs.ts`
that registry is the source of truth. Examples: `sync`, `embed`,
`lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`, `shell`,
`subagent`, `orphans`, `integrity`, plus dream-cycle phases and other
maintenance jobs. Submitting an unknown job name with
`gbrain jobs submit <name> --follow` prints the full registered list.
These cover every background operation the gbrain CLI itself performs.
Host platforms (OpenClaw deployments, future hosts) register their own
@@ -10,6 +15,11 @@ handlers via a plugin bootstrap that imports
code, loaded by the worker, with the same trust model as any other
code in the host's repo.
> **Two plugin systems.** This doc covers *job handlers* (code the Minion
> worker runs). Custom *subagent definitions* (markdown prompts loaded via
> `GBRAIN_PLUGIN_PATH`) are a separate system — see
> [plugin-authors.md](plugin-authors.md).
## Why code, not data
An earlier design draft shipped `~/.claude/gbrain-handlers.json` where
@@ -64,14 +74,23 @@ auto-loads on startup (configurable via a host-provided entry point).
## Handler contract
Every handler receives a `MinionJobContext`:
Every handler receives a `MinionJobContext` (canonical definition:
`src/core/minions/types.ts`). The load-bearing fields:
```ts
interface MinionJobContext {
data: Record<string, unknown>; // job params (whatever the cron submit passed)
job: MinionJob; // full job row (id, queue, attempts, etc.)
signal: AbortSignal; // set to aborted when the worker is shutting down
inbox: MinionInbox; // read messages sent to this job while it runs
id: number; // job id
name: string; // job type
data: Record<string, unknown>; // job params (whatever the cron submit passed)
attempts_made: number;
signal: AbortSignal; // fires on timeout, cancel, pause, or lock loss
shutdownSignal: AbortSignal; // fires only on worker SIGTERM/SIGINT
deadlineAtMs: number | null; // wall-clock deadline from timeout_at, if set
updateProgress(progress: unknown): Promise<void>;
updateTokens(tokens: TokenUpdate): Promise<void>;
log(message: string | TranscriptEntry): Promise<void>;
isActive(): Promise<boolean>; // is the job lock still held?
readInbox(): Promise<InboxMessage[]>; // unread messages sent to this job
}
```
+19 -13
View File
@@ -1,8 +1,8 @@
# Queue operations runbook
"My queue looks wedged — what do I run?" The commands below are in the order
you probably want them. Shipped with v0.19.1 after a production incident
where the queue held for 90+ minutes before the operator noticed.
you probably want them. Born from a production incident where the queue held
for 90+ minutes before the operator noticed.
## First signal: jobs aren't running
@@ -23,16 +23,18 @@ container health), but its DB connection died (common behind a transaction
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
pile up with **0 active**. Liveness checks all pass; nothing crashes.
As of v0.42.22.0 this self-heals — you usually won't have to do anything:
This self-heals — you usually won't have to do anything:
- **The worker exits on its own dead pool.** Under a supervisor, the worker's
DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the
supervisor respawns it with a fresh pool.
- **The supervisor restarts a worker that stops making progress.** If a queue
has claimable work, **0 live-lock active jobs**, and no completions for 15
minutes while the child is alive, the supervisor restarts it (covers stuck
handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
`--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables).
minutes across 3 consecutive health checks while the child is alive, the
supervisor restarts it (covers stuck handlers too, not just dead pools).
These thresholds are built in — there are no CLI flags to tune them. A
restart-loop breaker caps wedge restarts at 3 per 30-minute window, then
switches to a one-shot `wedge_restart_loop` alert in the audit log.
The signal is loud now — check either:
@@ -45,8 +47,12 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
stale completions). Manual fix if you ever need it:
```bash
gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool
gbrain jobs retry <id> # dead-lettered jobs
# Restart the supervisor with a fresh pool. `start` alone runs in the
# FOREGROUND (blocks); use --detach to get your shell back.
gbrain jobs supervisor stop && gbrain jobs supervisor start --detach --json
# Re-queue any jobs that were dead-lettered during the wedge.
gbrain jobs retry <id>
```
## Triage commands
@@ -101,9 +107,9 @@ claiming. Start one:
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
```
## Follow-ups tracked for v0.20+
## Related
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
subcheck both need this).
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
exit codes, and per-platform deployment (systemd / Fly / Render).
- [Minions shell jobs](minions-shell-jobs.md) — the `shell` job type's
security model and error table.
+21 -1
View File
@@ -113,6 +113,22 @@ fi
send_notification "$OUTPUT"
```
### GBrain-native hooks
Two places gbrain already understands quiet hours natively — use these
before rolling your own gate for the same job:
- **Self-upgrade**`auto` mode only applies upgrades during quiet hours,
configured via `gbrain config set self_upgrade.quiet_hours
'{"start":23,"end":8,"tz":"US/Pacific"}'`. See
[upgrades-auto-update.md](upgrades-auto-update.md).
- **Cron prompts** — schedule-driven notification jobs should carry the
gate described in this doc; [cron-schedule.md](cron-schedule.md) covers
the scheduling side.
The shell pattern below is for everything else: your own cron jobs,
collectors, and notification paths that gbrain doesn't gate for you.
### Configurable Hours
Some users want different quiet hours. Store the config:
@@ -140,7 +156,11 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring)
skill reads and clears the held directory. Orphaned held files mean the
pickup integration is broken.
3. **Timezone auto-detection is fragile.** Calendar-based timezone detection
3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a
held message must not be lost across a restart, use a durable held
directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`.
4. **Timezone auto-detection is fragile.** Calendar-based timezone detection
relies on the user having airline/hotel events with location data. If the
user books travel without calendar entries, the system won't detect the
move. Fall back to activity-hour analysis (responding at 3 AM PT = probably
+18 -6
View File
@@ -9,8 +9,9 @@ Separate agent behavior (replaceable) from world knowledge (permanent) into two
Without this: agent config and world knowledge are mixed together. Switch agents
and you lose your knowledge. Switch knowledge tools and you lose your agent setup.
With this: your brain (14,700+ files of people, companies, meetings, ideas)
survives any agent swap. Your agent config survives any knowledge tool swap.
With this: your brain (thousands of files of people, companies, meetings,
ideas) survives any agent swap. Your agent config survives any knowledge
tool swap.
## Implementation
@@ -119,7 +120,15 @@ without losing your agent setup.
notes). The agent repo contains operational config. Different access controls.
**GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep
the search index current. The agent repo is never indexed by GBrain.
the search index current. The agent repo is not indexed by default.
**Multi-source nuance.** With multi-source brains
([multi-source-brains.md](multi-source-brains.md)), "the brain repo" means
"each registered source." You CAN deliberately register a non-brain repo
(e.g. `~/.gstack`) as its own isolated or federated source — that's a
conscious registration with its own slug namespace, not a violation of the
boundary. The rule below is about *unregistered, accidental* indexing of
agent config.
## Tricky Spots
@@ -133,9 +142,12 @@ the search index current. The agent repo is never indexed by GBrain.
belongs in the brain. Agent configs, skills, cron jobs, and operational
state are replaceable. People, companies, ideas, and meetings are not.
3. **Don't index the agent repo.** GBrain indexes the brain repo only.
Running `gbrain sync` against the agent repo pollutes search results
with operational config instead of world knowledge.
3. **Don't casually index the agent repo.** Running `gbrain sync` against
the agent repo pollutes search results with operational config instead
of world knowledge. (Registering it deliberately as an isolated source
is different — see the multi-source nuance above. Pin the working
directory to the right source with a `.gbrain-source` dotfile via
`gbrain sources attach <id>`.)
## How to Verify
+43 -20
View File
@@ -34,35 +34,35 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
## v0.26.7 — auto-RLS event trigger and one-time backfill
## Auto-RLS: the event trigger and the one-time backfill
Starting in v0.26.7 (migration v35), gbrain ships two changes that close the
gap where a table could exist in your `public` schema without RLS for any
amount of time at all.
gbrain ships two mechanisms (schema migration v35) that close the gap where a
table could exist in your `public` schema without RLS for any amount of time
at all.
**1. The event trigger.** A Postgres DDL event trigger named
`auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY`
on every newly created `public.*` table. It covers `CREATE TABLE`,
`CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres
reports as a table-creation command. Tables created by gbrain itself, by
your other apps sharing the same Supabase project (Baku, Hermes, anything),
or by a human running raw SQL all get RLS enabled the moment they exist.
any other app sharing the same Supabase project, or by a human running raw
SQL all get RLS enabled the moment they exist.
Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly
ignored — Supabase manages those, and we should not touch them.
**2. The one-time backfill.** When you upgrade to v0.26.7, the migration
**2. The one-time backfill.** The first upgrade that applies migration v35
walks every existing `public.*` base table whose RLS is off and whose comment
doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS
on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op
on every brain.
### Breaking change: read this before upgrading
### Read this before upgrading a pre-auto-RLS brain
If you have public tables that are intentionally RLS-off and you want them
to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before**
running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public
table that doesn't carry the exact comment contract documented below. There
is no `--dry-run` flag on the migration.
the upgrade that applies migration v35. The backfill flips RLS on for any
public table that doesn't carry the exact comment contract documented below.
There is no `--dry-run` flag on the migration.
The minimum cost of getting this wrong is one round-trip: the operator runs
the SQL to enable RLS on a table that should have been exempt, then
@@ -71,7 +71,7 @@ prevent a re-flip on a later doctor run. No data is lost.
### Cross-app implications
If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates
If a non-gbrain app (a side project, a script you wrote, anything) creates
tables in the same Supabase project, the trigger will enable RLS on those
tables too. Two ways to handle that:
@@ -90,17 +90,40 @@ ship a policy.
### What if the trigger gets dropped?
`gbrain doctor` includes a new `rls_event_trigger` check that verifies the
`gbrain doctor` includes an `rls_event_trigger` check that verifies the
trigger is installed and enabled. If you drop it manually for any reason
(debugging, migration testing, anything), doctor warns and gives you the
recovery command:
(debugging, migration testing, anything), doctor warns and points you here.
```
gbrain apply-migrations --force-retry 35
Recreate it by re-running the trigger DDL from migration v35 — idempotent
(`CREATE OR REPLACE` + `DROP EVENT TRIGGER IF EXISTS`), safe to paste into
psql as a BYPASSRLS role (e.g. `postgres`):
```sql
CREATE OR REPLACE FUNCTION auto_enable_rls()
RETURNS event_trigger AS $$
DECLARE
obj record;
BEGIN
FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands()
WHERE object_type = 'table'
AND schema_name = 'public'
LOOP
EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', obj.object_identity);
END LOOP;
END;
$$ LANGUAGE plpgsql;
DROP EVENT TRIGGER IF EXISTS auto_rls_on_create_table;
CREATE EVENT TRIGGER auto_rls_on_create_table
ON ddl_command_end
WHEN TAG IN ('CREATE TABLE', 'CREATE TABLE AS', 'SELECT INTO')
EXECUTE FUNCTION auto_enable_rls();
```
Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS`
and recreates cleanly.
(This is the same DDL migration v35 runs — the canonical copy lives in the
`MIGRATIONS` array in `src/core/migrate.ts`. There's no CLI shortcut:
`gbrain apply-migrations --force-retry` targets the vX.Y.Z orchestrator
registry, not numeric schema migrations like v35.)
### Why no FORCE ROW LEVEL SECURITY?
@@ -147,7 +170,7 @@ Rules:
```sql
ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY;
COMMENT ON TABLE public.expenses_ramp IS
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22';
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=you, 2026-04-22';
```
After that, `gbrain doctor` reports:
+10 -11
View File
@@ -127,9 +127,9 @@ you can use as a reference shape. The skillpack story for distributing
your own resolvers across machines is covered in
[skillpacks as scaffolding](skillpacks-as-scaffolding.md).
## The compact list format (v0.41.7.0)
## The compact list format
GBrain's resolver parser used to require markdown tables:
GBrain's resolver parser originally required markdown tables:
```markdown
| Trigger | Skill |
@@ -146,14 +146,14 @@ format that scales better:
- **flight-tracker**: track my flight | flight status | when does my flight land
```
Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a
306-skill compact-format resolver, the doctor reported every skill as
unreachable: **238 FAIL errors on every doctor run**. The parser was
silently treating the compact dialect as zero skills.
When `gbrain doctor` only spoke the table dialect, a 306-skill
compact-format resolver reported every skill as unreachable: **238 FAIL
errors on every doctor run**. The parser was silently treating the compact
dialect as zero skills.
v0.41.7.0 ships dual-format support. The same `parseResolverEntries`
function reads both table rows and list rows in the same file, with the
v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
Today the parser supports both. The same `parseResolverEntries`
function reads table rows and list rows in the same file, with the
multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
`../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor`
and the 238 FAILs collapse to 0.
@@ -267,8 +267,7 @@ I initially converted my resolver from a clean list format to a table
format because the validator only spoke tables. That was wrong. When a
tool fails against valid data, the right move is to fix the tool, not
reshape the data. The list format was correct, compact, readable, easy
to maintain. The parser needed to support both shapes. v0.41.7.0 is
that fix.
to maintain. The parser needed to support both shapes — and now it does.
The same principle applies everywhere in agent systems. Your SKILL.md is
the source of truth. Your AGENTS.md is the source of truth. Your resolver
+100 -9
View File
@@ -1,19 +1,109 @@
# Search Modes
## Goal
Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result.
Two decisions shape every gbrain lookup, and this guide covers both:
## What the User Gets
Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls.
1. **Which mode bundle** your brain runs — `conservative` / `balanced` /
`tokenmax`, the named cost-knob presets that control cache, token budget,
query expansion, and result count. This is the config-level decision you
make once (at `gbrain init` or via `gbrain config set search.mode`).
2. **Which lookup verb** to use per call — `gbrain search` (keyword),
`gbrain query` (hybrid), or `gbrain get` (direct). This is the
per-lookup decision an agent makes on every question.
## Implementation
## The three mode bundles
A search mode is a named preset that sets every search-cost knob at once.
The bundles are frozen in `src/core/search/mode.ts` (`MODE_BUNDLES`):
| Knob | `conservative` | `balanced` | `tokenmax` |
|-------------------------------|----------------|------------|----------------|
| `cache.enabled` | true | true | true |
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
| `intentWeighting` | true | true | true |
| `tokenBudget` | **4000** | **12000** | **off** |
| `expansion` (LLM multi-query) | false | false | **true** |
| `relationalRetrieval` | false | **true** | **true** |
| `searchLimit` default | 10 | 25 | 50 |
- **`conservative`** — smallest payloads. Pairs naturally with a cheap
downstream model (Haiku-class) or a high query volume.
- **`balanced`** — the default and the fallback when no mode is set.
- **`tokenmax`** — no token budget, LLM query expansion on, 50 results.
Pairs with an expensive downstream model you want fully fed.
Two of the knobs deserve a sentence:
- **`expansion`** rewrites your query into multiple variants via a cheap
LLM call per search (adds roughly $1.50 per 1K queries) — better recall,
small extra cost.
- **`relationalRetrieval`** adds a graph-walk recall arm for relational
questions ("who invested in X", "what connects A and B"); it's a pure
no-op for non-relational queries. The `query` op's `relational` flag
forces it on/off per call.
### Setting and resolving the mode
```bash
gbrain config set search.mode tokenmax
```
Per-knob resolution (highest first):
per-call SearchOpts → per-key config override (search.cache.enabled, …) →
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
Mode resolution lives in bare `hybridSearch`, not just the cached wrapper,
so eval replays test the same mode-affected behavior as the production
`query` op. The query cache folds the active knobs into its key
(`knobs_hash`), so switching modes never serves you a stale result set
from a different configuration.
### Cost intuition
gbrain's own cost is rounding error; what the mode really controls is how
many tokens your *downstream agent* pays to read per query. The
corner-to-corner spread is ~25x once you pair mode with downstream model.
Rough anchors at 10K queries/month, full payload, no cache savings:
| Mode \ Downstream | Haiku-class (\$1/M in) | Sonnet-class (\$3/M in) | Opus-class (\$5/M in) |
|---|---|---|---|
| conservative (~4K tok) | **\$40/mo** | \$120/mo | \$200/mo |
| balanced (~10K tok) | \$100/mo | \$300/mo | \$500/mo |
| tokenmax (~20K tok) | \$200/mo | \$600/mo | **\$1,000/mo** |
Scales linearly with volume. Cache hits cut all numbers ~50%; disciplined
prompt caching in the agent loop cuts further. Mismatched pairings waste
capacity in both directions — a tokenmax payload overwhelms a cheap model,
a conservative payload starves an expensive one. The full methodology and
realistic-scale walkthrough live in
[`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
### CLI surfaces
```bash
gbrain search modes # what is running, with per-knob attribution
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
gbrain search tune [--apply] # data-driven recommendations
gbrain search diagnose "<query>" --target <slug>
# trace where a page surfaces (or fails to)
# across the keyword/vector/alias/hybrid layers
```
The mode picker runs inside `gbrain init` (non-TTY auto-selects `balanced`).
## Choosing a lookup verb (search vs query vs get)
Independent of which bundle is active, every individual lookup should use
the cheapest verb that answers the question.
```
on user_asks_about(topic):
# Decision tree: pick the right search mode
# Decision tree: pick the right lookup verb
if know_exact_slug(topic):
# MODE 3: Direct get -- instant, no search overhead
# Direct get -- instant, no search overhead
result = gbrain get <slug>
# e.g., "Tell me about Alice" -> gbrain get alice-example
# Returns the FULL page -- compiled truth + timeline
@@ -66,7 +156,7 @@ on user_asks_about(topic):
# 4. External sources (web search, APIs)
```
## Tricky Spots
### Tricky Spots
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
2. **Search works without embeddings.** On day one before any embedding run, `gbrain search` still works (the keyword arm carries it; the vector arm joins once embeddings exist). Don't tell the user "search isn't available yet" -- search is always available.
@@ -75,13 +165,14 @@ on user_asks_about(topic):
5. **Full hybrid needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
6. **A populated `gbrain search` result set is not proof you found everything.** Search runs without query expansion, so synonym- and outcome-phrased matches can be missed even when it returns plenty of hits. For "find every / all / the landscape of" questions, use `gbrain query`; for literal exhaustive enumeration ("list every page of type X"), use `list_pages` pagination. A nonzero count is not a completeness signal.
## How to Verify
### How to Verify
1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references.
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline.
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
6. Run `gbrain search modes` -- confirm the active mode bundle and any per-key overrides are what you expect.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+5 -2
View File
@@ -40,7 +40,9 @@ Show the user the results. Get feedback.
- Revise the process based on what you learned.
**Step 4: Codify into a Skill.**
Write the SKILL.md. Either:
Write the SKILL.md (`gbrain skillify scaffold <name>` scaffolds the tree for
you; `gbrain skillopt` optimizes an existing skill against a benchmark).
Either:
- **New skill** -- genuinely new capability
- **Add to existing skill** -- variation of something that exists (parameterize it)
@@ -63,7 +65,8 @@ Skills should be **Mutually Exclusive, Collectively Exhaustive**:
- Each signal source has exactly ONE owner skill
- Two skills creating the same brain page = MECE violation
**Example ownership (no overlap):**
**Example ownership (no overlap — illustrative; your skill roster will
differ):**
| Signal Source | Owner Skill | Creates |
|--------------|-------------|---------|
+4 -4
View File
@@ -65,14 +65,14 @@ For each step:
more than 0.05.
6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe.
After each epoch with no improvement: D6 slow-update fires one meta-edit
proposal (this lives in v0.42 follow-up; v1 emits the audit event).
After each epoch with no improvement: D6 slow-update fires. Today it emits
the audit event only; the full meta-edit proposal is a tracked follow-up.
## Flags
| Flag | Default | Purpose |
|---|---|---|
| `--benchmark <path>` | `skills/<n>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
| `--benchmark <path>` | `skills/<name>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
| `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) |
| `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) |
| `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl |
@@ -136,7 +136,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
- **No benchmark.** Optimizing against guesses is worse than not optimizing.
- **Write-flavored skills.** Skills whose job is to `put_page` heavily can't
use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up.
use the read-only sandbox; mocked-write capture is a tracked follow-up.
- **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful
validation needs ≥20 tasks total per the paper.
+23 -14
View File
@@ -1,12 +1,11 @@
# Skillpacks as scaffolding, not amber
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
scaffold + reference library. This guide explains the model and the
workflow.
`gbrain skillpack` is a scaffold + reference library, not a package
manager. This guide explains the model and the workflow.
## Why we changed it
## Why it works this way
Pre-v0.33 (the "amber" model):
An earlier design (the "amber" model):
- `gbrain skillpack install <name>` copied bundled skills into your
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
@@ -26,7 +25,16 @@ repo. You scaffold once, you own them, you fork and edit freely. When
gbrain ships a new version, you ask "what changed?" — the agent reads
the diff and decides what (if anything) to integrate.
## The five commands
## The core workflow commands
The five commands below are the scaffold-and-own workflow. The full
`gbrain skillpack` surface is larger (`list`, `diff`, `check`, `search`,
`info`, `registry`, `doctor`, `init`, `pack`, `endorse`, …) — run
`gbrain skillpack --help` for the always-current list. One worth calling
out here: **`gbrain skillpack init-brain-pack <name>`** scaffolds a
*brain-resident* pack inside a brain/source repo (`brain_resident: true`
plus a machine-parseable README) that connecting harnesses discover on
`gbrain sources add`.
### `gbrain skillpack scaffold <name> [--workspace PATH]`
@@ -77,15 +85,15 @@ gbrain skillpack reference book-mirror
`reference <name> --apply-clean-hunks` is the auto-apply path. It
parses the diff between gbrain's bundle and your local copy, applies
every hunk whose pre-change context matches uniquely. **Two-way merge
limitation**: without scaffold-time base tracking (intentionally
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
limitation**: without scaffold-time base tracking (intentionally out of
scope), this cannot distinguish "gbrain changed X"
from "you changed X." Applied hunks align everything to gbrain. Use
`--dry-run` first to preview, or run plain `reference` to inspect the
diff before letting auto-apply touch anything.
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
One-shot conversion for workspaces on the pre-v0.33 managed-block
One-shot conversion for workspaces still on the legacy managed-block
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
markers and the manifest receipt comment from your resolver file.
@@ -157,7 +165,7 @@ Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
frontmatter, and match the user's intent against every skill's
`triggers:` array. When a match scores high enough, invoke that skill.
This replaces the v0.32 model where `gbrain skillpack install` wrote
This replaces the legacy model where `gbrain skillpack install` wrote
table rows into your `RESOLVER.md`. Rows are gone (or, for users
migrating from the old model, preserved transitionally by
`migrate-fence` until they run `scrub-legacy-fence-rows`).
@@ -173,7 +181,7 @@ If you're a downstream agent author updating to this model:
## Removing a scaffolded skill
There's no `gbrain skillpack uninstall` command in v0.33. The files
There's no `gbrain skillpack uninstall` command. The files
in your `skills/<slug>/` are first-class members of your repo —
delete them like any other code:
@@ -195,16 +203,17 @@ You own the files. There's no manifest to update, no fence to rebuild.
## When to use which command (quick decision tree)
- **New host repo, want a gbrain skill**`scaffold`
- **Shipping a pack from inside a brain/source repo**`init-brain-pack`
- **gbrain shipped a new version, want to see what's changed**
`reference` (read-only) or `reference --apply-clean-hunks` (auto)
- **Upgrading from v0.32 or earlier**`migrate-fence` (one-shot)
- **Upgrading from the legacy managed-block model**`migrate-fence` (one-shot)
- **Cleanup after `migrate-fence`**`scrub-legacy-fence-rows`
- **Lift your fork's skill back into gbrain**`harvest` + the
`skillpack-harvest` editorial skill
## What about `install` and `uninstall`?
Both are removed in v0.33. Running either prints an error pointing at
the replacement command. No deprecated alias — this is a clean break.
Both are removed. Running either prints an error pointing at the
replacement command. No deprecated alias — this is a clean break.
If you have existing scripts referencing the old names, update them
once and move on.
+13 -5
View File
@@ -4,7 +4,7 @@
Every fact in the brain traces to where it came from -- who said it, in what context, and when.
## What the User Gets
Without this: six months from now, someone reads a brain page and has no idea if "Pedro co-founded Brex" came from Pedro himself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
Without this: six months from now, someone reads a brain page and has no idea if "Alice co-founded widget-co" came from Alice herself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
## Implementation
@@ -23,11 +23,11 @@ on brain_write(page, fact):
# [Source: Crustdata LinkedIn enrichment, 2026-04-07 12:35 PM PT]
elif source.type == "social_media":
# MUST include full URL -- not just @handle
# [Source: X/@pedroh96 tweet, product launch, 2026-04-07](https://x.com/pedroh96/status/...)
# [Source: X/@alice_example tweet, product launch, 2026-04-07](https://x.com/alice_example/status/...)
elif source.type == "email":
# [Source: email from Sarah Chen re Q2 board deck, 2026-04-05 2:30 PM PT]
# [Source: email from Alice Example re Q2 board deck, 2026-04-05 2:30 PM PT]
elif source.type == "workspace":
# [Source: Slack #engineering, Keith re deploy schedule, 2026-04-06 11:45 AM PT]
# [Source: Slack #engineering, Charlie re deploy schedule, 2026-04-06 11:45 AM PT]
elif source.type == "web":
# [Source: Happenstance research, 2026-04-07 12:35 PM PT]
elif source.type == "published":
@@ -57,7 +57,7 @@ SOURCE_PRIORITY = [
## Tricky Spots
1. **Compiled truth is NOT exempt from citations.** "Pedro co-founded Brex" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
1. **Compiled truth is NOT exempt from citations.** "Alice co-founded widget-co" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
2. **Tweet URLs are mandatory.** `[Source: X/@handle tweet, topic, date]` without a URL is a broken citation. Hundreds of brain pages end up with unreachable tweet references when the URL is omitted. Always: `[Source: X/@handle tweet, topic, date](https://x.com/handle/status/ID)`.
3. **"User said it" isn't enough.** WHERE, ABOUT WHAT, WHEN. `[Source: User, direct message, 2026-04-07 12:33 PM PT]` -- not just `[Source: User]`.
4. **Don't silently resolve conflicts.** When the user says one thing and an API says another, note the contradiction in compiled truth with both citations. Let the reader decide.
@@ -71,5 +71,13 @@ SOURCE_PRIORITY = [
4. Check timeline entries on 3 random pages. Each entry should have a source citation with date and context.
5. Look for a page where the user stated something that contradicts an API result. Confirm the contradiction is noted, not silently resolved.
## Related
- `skills/_brain-filing-rules.md` — the canonical citation-format rules every
brain write follows (this guide is the narrative walkthrough of the same
contract).
- `skills/citation-fixer/SKILL.md` — audits and repairs existing pages
against that format.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+50 -23
View File
@@ -2,19 +2,42 @@
## Goal
Route sub-agents to the cheapest model that can do the job, saving 10-40x on costs without sacrificing quality.
Route sub-agents to the cheapest model that can do the job, saving large
multiples on cost without sacrificing quality.
## What the User Gets
Without this: every sub-agent runs on Opus ($15/MTok). Entity detection on
every message costs $3-5/day. Research tasks cost $10+ each.
Without this: every sub-agent runs on your most expensive model. Entity
detection fires on every message at top-tier rates; research tasks cost
several dollars each.
With this: entity detection runs on Sonnet ($3/MTok, 5x cheaper). Research
runs on DeepSeek ($0.50/MTok, 30x cheaper). Main session stays on Opus for
quality. Total cost drops 70-80%.
With this: entity detection runs on a cheap fast model, research execution
runs on a budget model, and only planning/synthesis touch the expensive
model. Total cost drops 70-80%.
(Illustrative input-token anchors from gbrain's canonical pricing table,
`src/core/model-pricing.ts`: Opus-class $5/MTok, Sonnet-class $3/MTok,
Haiku-class $1/MTok. Budget providers run well under $1/MTok. Prices
drift — the pricing table is the source of truth, not this doc.)
## Implementation
### GBrain's native mechanism: model tiers
Before hardcoding vendors, use gbrain's tier routing. Every gbrain
subagent/LLM call resolves through a named tier
(`utility` / `reasoning` / `deep` / `subagent`), and you point each tier
at whatever model you want once:
```bash
gbrain config set models.tier.subagent anthropic:claude-haiku-4-5
gbrain config set models.tier.deep anthropic:claude-opus-4-7
```
Per-call override: `gbrain agent run --model <provider:model>`. The
conventions file `skills/conventions/model-routing.md` is the canonical
routing policy; this guide is the cost rationale behind it.
### Routing Table
| Task Type | Recommended Model | Why |
@@ -44,14 +67,16 @@ on_every_message(text):
3. FACTS: New info about existing entities -> update timeline
4. CITATIONS: Every fact needs [Source: ...] attribution
5. Sync changes to brain repo`,
model: "sonnet-class", // fast + cheap
model: "sonnet-class", // fast + cheap; haiku-class is cheaper still
timeout: 120s
})
```
**Why Sonnet-class for detection:** Entity detection is pattern matching, not
deep reasoning. Sonnet is 5-10x cheaper than Opus and fast enough for async
detection. The main session continues on Opus while detection runs in parallel.
**Why a cheaper class for detection:** Entity detection is pattern matching,
not deep reasoning. Sonnet-class runs at a fraction of Opus-class cost, and
Haiku-class at a fraction of that — both fast enough for async detection.
The main session continues on your best model while detection runs in
parallel.
### Research Pipeline Pattern
@@ -64,9 +89,9 @@ For research-heavy tasks, use a multi-model pipeline:
```
**Why this works:** The planning and synthesis steps need taste and judgment
(Opus). The execution step is mechanical data gathering (DeepSeek at 25-40x
lower cost). You get Opus-quality output at DeepSeek-level cost for 80% of
the work.
(Opus-class). The execution step is mechanical data gathering (a budget
model at a small fraction of the cost). You get top-tier output at
budget-model cost for 80% of the work.
### When to Spawn Sub-Agents
@@ -82,15 +107,16 @@ the work.
The main session runs on your best model. Everything else runs on the
cheapest model that can do the job. In practice, 60-70% of sub-agent
work is entity detection (Sonnet) and research execution (DeepSeek),
which are 10-40x cheaper than the main session model.
work is entity detection and research execution, which run at a small
fraction of the main session model's cost.
## Tricky Spots
1. **Sonnet, not Opus, for detection.** The most common mistake is running
entity detection on Opus. Detection is pattern matching, not deep reasoning.
Sonnet is 5-10x cheaper and fast enough. Reserve Opus for the main session
where reasoning quality matters.
1. **A cheap class, not Opus, for detection.** The most common mistake is
running entity detection on Opus-class. Detection is pattern matching, not
deep reasoning. Sonnet- or Haiku-class is several times cheaper and fast
enough. Reserve Opus-class for the main session where reasoning quality
matters.
2. **Don't block the main thread.** Sub-agents must run asynchronously. If the
signal detector runs synchronously, the user waits 30-120 seconds for every
@@ -98,10 +124,11 @@ which are 10-40x cheaper than the main session model.
a response immediately.
3. **Cost optimization is multiplicative.** Entity detection runs on every
single message. If you use Opus at $15/MTok for detection across 50
messages/day, that's $3-5/day just for detection. Sonnet at $3/MTok brings
that to $0.60-1.00/day. Over a month, the wrong model choice costs $100+
more than necessary.
single message, so the per-call price difference compounds across 50+
messages/day. Routing detection from Opus-class ($5/MTok in) to
Haiku-class ($1/MTok in) is a flat 5x cut on your highest-frequency LLM
call — over a month, the wrong model choice for detection alone costs
real money. (Current per-model rates: `src/core/model-pricing.ts`.)
## How to Verify
+18 -7
View File
@@ -16,9 +16,9 @@ benefit-focused bullets, waits for explicit permission, then runs the full
upgrade flow including re-reading skills, running migrations, and syncing
schema. The user gets new capabilities automatically.
## Self-upgrade modes (v0.42)
## Self-upgrade modes
gbrain now stays current the way gstack does: it rides invocation frequency. A
gbrain stays current the way gstack does: it rides invocation frequency. A
throttled, cache-read-only check runs at the start of every `gbrain` invocation
(CLI and MCP) and emits an `UPGRADE_AVAILABLE <old> <new>` marker on stderr. No
host cron required — every agent kind (Claude Code, Codex, OpenClaw, Hermes, the
@@ -44,6 +44,14 @@ execution. The trust model is TLS + GitHub (same as `gbrain upgrade`);
signature verification is a tracked follow-up. Apply manually any time with
`gbrain self-upgrade`.
The `auto` quiet-hours window is configured via the
`self_upgrade.quiet_hours` config key
(`gbrain config set self_upgrade.quiet_hours '{"start":23,"end":8,"tz":"US/Pacific"}'`).
The quiet-hours *pattern* itself — gating any notification or background
action on the user's local sleep window — is owned by
[quiet-hours.md](quiet-hours.md); this doc only covers the self-upgrade
hook into it.
## Implementation
### The Check (cron-initiated)
@@ -163,15 +171,18 @@ Also persist in `~/.gbrain/update-state.json` so it survives agent context reset
If you loaded this SKILLPACK directly (copied or read from GitHub) without
installing gbrain, you can still stay current. Both GBRAIN_SKILLPACK.md and
GBRAIN_RECOMMENDED_SCHEMA.md have version markers:
GBRAIN_RECOMMENDED_SCHEMA.md carry a `<!-- source: ... -->` header pointing
at their canonical copies, and GBRAIN_RECOMMENDED_SCHEMA.md also carries a
version marker:
```bash
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md | head -1
# Returns: <!-- skillpack-version: X.Y.Z -->
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md | head -1
# Returns: <!-- schema-version: X.Y.Z -->
```
If the remote version is newer, fetch the full file and replace your local
copy. Set up a weekly cron to check automatically.
If the remote version is newer (or the remote SKILLPACK content differs from
your local copy), fetch the full file and replace your local copy. Set up a
weekly cron to check automatically.
## Tricky Spots
+9 -7
View File
@@ -2,8 +2,9 @@
GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it
accessible from other devices and AI clients, run `gbrain serve --http`
(built-in HTTP transport with bearer auth, Postgres-only ... see
[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options.
(built-in HTTP transport with OAuth 2.1 + bearer auth, works on both PGLite
and Postgres brains — see [DEPLOY.md](DEPLOY.md)) behind a public tunnel.
Here are your tunnel options.
## ngrok (recommended)
@@ -58,10 +59,11 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits.
| Works when laptop is off | No | No | Yes |
| Cold start | None | None | None |
| Timeout limits | None | None | None |
| All 30 operations | Yes | Yes | Yes |
| Full remote operation surface (100+ ops, minus `localOnly`) | Yes | Yes | Yes |
| Setup time | 5 min | 10 min | 15 min |
**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth
against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap,
per-request audit log. Postgres-only by design (PGLite is local-only). See
[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables.
**Note:** `gbrain serve --http` is the built-in HTTP transport. OAuth 2.1 plus
bearer auth against the `access_tokens` table, default-deny CORS, two-bucket rate
limit, body cap, per-request audit log. Works on both PGLite and Postgres brains.
See [DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and
tunables.
+28 -34
View File
@@ -1,35 +1,39 @@
# Connect GBrain to ChatGPT
**Status (v0.26.0):** Unblocked. GBrain's `gbrain serve --http` ships OAuth 2.1
with PKCE, which is the ChatGPT MCP connector's hard requirement. Before v1.0,
this was a P0 TODO — the only major AI client that could not connect.
ChatGPT's MCP connector requires OAuth 2.1 with PKCE — it does not support
bearer-token MCP servers. GBrain's `gbrain serve --http` speaks exactly that,
so ChatGPT connects natively.
ChatGPT does not support bearer-token MCP servers. You must use the OAuth 2.1
HTTP server.
This page covers only the ChatGPT-specific parts. The full server setup —
starting `gbrain serve --http`, the admin bootstrap token, the `/admin`
dashboard, tunnels, and `--bind` / `--public-url` — lives in
[DEPLOY.md](DEPLOY.md). Do steps 1 (start the server) and 3 (expose it)
from there, then come back for the ChatGPT client.
## Setup
### 1. Start the HTTP server
### 1. Start and expose the server (DEPLOY.md steps 1 + 3)
```bash
gbrain serve --http --port 3131
```
Save the admin bootstrap token printed on stderr. Open
`http://localhost:3131/admin` and paste it to access the dashboard.
Follow [DEPLOY.md — OAuth 2.1 Setup](DEPLOY.md#oauth-21-setup) to start
`gbrain serve --http`, save the admin bootstrap token, and expose the server
at a public HTTPS URL (e.g. `https://your-brain.ngrok.app`). ChatGPT's
connector auto-discovers the spec-compliant endpoint at
`/.well-known/oauth-authorization-server`.
### 2. Register a ChatGPT client
ChatGPT uses the authorization code flow with PKCE (browser-based OAuth).
Register from the `/admin` dashboard:
The ChatGPT-specific delta: ChatGPT uses the **authorization code flow with
PKCE** (browser-based OAuth), so the client needs the `authorization_code`
grant type and a redirect URI. Register from the `/admin` dashboard:
1. Click **Register client**.
2. Name: `chatgpt`.
3. Grant type: `authorization_code`.
4. Scopes: `read`, `write` (leave `admin` unchecked for ChatGPT).
5. Redirect URI: ChatGPT's OAuth redirect (copy it from the ChatGPT
connector setup screen — something like
`https://chat.openai.com/connector_platform_oauth_redirect`).
5. Redirect URI: ChatGPT's OAuth redirect — **always copy the exact value
from the ChatGPT connector setup screen** (it looks like
`https://chatgpt.com/connector_platform_oauth_redirect`, but the domain
has changed before; trust the setup screen, not this doc).
6. Hit **Register**. The credential-reveal modal shows the `client_id` once
with Copy and Download JSON buttons. There is no client secret for
PKCE-based public clients.
@@ -41,22 +45,11 @@ await oauthProvider.registerClientManual(
'chatgpt',
['authorization_code'],
'read write',
['https://chat.openai.com/connector_platform_oauth_redirect'],
['<ChatGPT redirect URI from the connector setup screen>'],
);
```
### 3. Expose the server publicly
```bash
brew install ngrok
ngrok http 3131 --url your-brain.ngrok.app
```
Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. ChatGPT's
connector auto-discovers the spec-compliant endpoint at
`/.well-known/oauth-authorization-server`.
### 4. Add the connector in ChatGPT
### 3. Add the connector in ChatGPT
1. Open ChatGPT > Settings > Connectors.
2. Click **Add connector**.
@@ -71,10 +64,11 @@ calls show up in the admin dashboard's live SSE feed in real time.
## Scopes
ChatGPT clients can request any combination of `read`, `write`, `admin`. The
scopes granted at consent time are enforced on every tool call. Four
operations are `localOnly` and rejected over HTTP regardless of scope:
`sync_brain`, `file_upload`, `file_list`, `file_url`. The HTTP server fails
closed for any attempt to reach local filesystem surface area.
scopes granted at consent time are enforced on every tool call. Operations
flagged `localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain`
and the `file_*` ops among them) are rejected over HTTP regardless of scope.
The HTTP server fails closed for any attempt to reach local filesystem
surface area.
Recommended ChatGPT scope: `read write`. Leave `admin` for your local CLI
and the admin dashboard.
+5
View File
@@ -4,6 +4,10 @@
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
> end to end, plus the brain-first protocol that makes it worth it. This page is
> the connection reference.
>
> Want the **full agent** — identity, per-turn context, schedules, and a private
> repo as its durable body — not just a memory? That's `gbrain bootstrap`:
> see the paste block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
## Option 1: Local (recommended, zero server needed)
@@ -90,6 +94,7 @@ You should see results from your GBrain knowledge base.
> full-surface — on `--surface verbs` the agent sees only the five memory verbs,
> and `list_skills` isn't on the surface at all. Note: `capture` is a
> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`.
> Why brains differ on the default: [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
## Remove
+13
View File
@@ -27,6 +27,19 @@ Desktop bridges local MCP servers into Cowork via its SDK layer.
This means: if `gbrain serve` is running and configured in Claude Desktop,
you don't need a separate server for Cowork.
## Verify
In a Cowork session, try:
```
Call get_brain_identity, then search my brain for [any topic]
```
You should get pages from your brain back. If `list_skills` returns nothing,
skill publishing is off on the host — enable it with
`gbrain config set mcp.publish_skills true` (see
[CLAUDE_CODE.md](CLAUDE_CODE.md) for the full gotcha).
## Which to use?
- **Remote server:** works even when your laptop is closed, available to all org members
+5
View File
@@ -1,5 +1,10 @@
# Connect GBrain to Claude Desktop
This page covers connecting Claude Desktop to a **remote** brain. For a brain
on the same machine as Claude Desktop, a local stdio entry in
`claude_desktop_config.json` with `"command": "gbrain", "args": ["serve"]`
works too — but only against a full local install, never a thin-client one.
**Important:** Claude Desktop does NOT connect to remote MCP servers via
`claude_desktop_config.json`. That file only works for local stdio servers.
Remote HTTP servers must be added through the GUI.
+12 -8
View File
@@ -4,10 +4,14 @@
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
> end to end, plus the brain-first protocol that makes it worth it. This page is
> the connection reference.
>
> Want the **full agent** — identity, memory, schedules, and a private repo as its
> durable body — not just a connection? That's `gbrain bootstrap`: see the paste
> block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
Codex CLI (`@openai/codex`, v0.130+) supports remote streamable-HTTP MCP servers
with a bearer token read from an environment variable. The token lives in your
shell env, not in Codex's config file.
Recent versions of the Codex CLI (`@openai/codex`) support remote
streamable-HTTP MCP servers with a bearer token read from an environment
variable. The token lives in your shell env, not in Codex's config file.
## Fastest path: `gbrain connect`
@@ -51,11 +55,11 @@ Call get_brain_identity, then search my brain for [topic].
`get_brain_identity` confirms whose brain you're connected to; `list_skills` shows
everything it can do.
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host (default
> ON for `gbrain init` brains, OFF for brains upgraded from older releases). Enable
> it on the host: `gbrain config set mcp.publish_skills true`. The core tools
> (search, query, get_page, put_page, think, find_experts) work regardless.
> `capture` is CLI-only, not an MCP tool — write over MCP with `put_page`.
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host — enable
> it with `gbrain config set mcp.publish_skills true`. The core tools (search,
> query, get_page, put_page, think, find_experts) work regardless; `capture` is
> CLI-only, so write over MCP with `put_page`. Why brains differ on the default:
> [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
## Remove
+57 -47
View File
@@ -1,17 +1,17 @@
# Deploy GBrain Remote MCP Server
> **v0.26.0+:** `gbrain serve --http` ships full OAuth 2.1 (client credentials,
> auth code + PKCE, refresh rotation, optional DCR), an embedded React admin
> dashboard at `/admin`, scoped operations, and a live SSE activity feed.
> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back
> to the `access_tokens` table and grandfathers tokens to `read+write+admin`.
> Both the legacy fallback and the OAuth tables work on PGLite and Postgres
> (both engine schemas carry `access_tokens`). See [SECURITY.md](../../SECURITY.md)
> for env vars and tunable defaults.
> `gbrain serve --http` ships full OAuth 2.1 (client credentials, auth code +
> PKCE, refresh rotation, optional DCR), an embedded React admin dashboard at
> `/admin`, scoped operations, and a live SSE activity feed. Legacy bearer
> tokens still work — `verifyAccessToken` falls back to the `access_tokens`
> table and grandfathers tokens to `read+write+admin`. Both the legacy fallback
> and the OAuth tables work on PGLite and Postgres (both engine schemas carry
> `access_tokens`). See [SECURITY.md](../../SECURITY.md) for env vars and
> tunable defaults.
Access your brain from any device, any AI client. GBrain ships two transports:
`gbrain serve` (stdio) for local agents, and `gbrain serve --http` (v0.26.0+)
for remote clients over OAuth 2.1.
`gbrain serve` (stdio) for local agents, and `gbrain serve --http` for remote
clients over OAuth 2.1.
## Three Paths
@@ -29,7 +29,7 @@ No server, no tunnel, no token needed. Works on both PGLite and Postgres engines
[MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)) instead of the full catalog;
omit the flag (default `full`) for every operation.
### Remote over OAuth 2.1 (recommended, v0.26.0+)
### Remote over OAuth 2.1 (recommended)
```bash
gbrain serve --http --port 3131
@@ -50,28 +50,27 @@ Supported clients:
- **Perplexity** — OAuth 2.1 client credentials grant.
- **Claude Code, Cursor, Windsurf** — can use OAuth or legacy bearer.
See the [OAuth 2.1 setup](#oauth-21-setup-v100) section below.
See the [OAuth 2.1 setup](#oauth-21-setup) section below.
### Remote with legacy bearer tokens (pre-v0.26 deployments) — Postgres only
### Remote with legacy bearer tokens (simplest)
```
Your AI client (Claude Desktop, Perplexity, etc.)
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
→ gbrain serve --http (built-in transport with bearer auth)
→ Postgres (pooler connection or self-hosted)
→ Postgres or PGLite
```
This requires:
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
running `gbrain serve --http` against a PGLite install fails fast at startup)
2. A machine running `gbrain serve --http`
3. A public tunnel (ngrok, Tailscale, or cloud host)
4. A bearer token created via `gbrain auth create <name>`
1. A machine running `gbrain serve --http` (works on both PGLite and Postgres
brains)
2. A public tunnel (ngrok, Tailscale, or cloud host)
3. A bearer token created via `gbrain auth create <name>`
Pre-v1.0 tokens are grandfathered as `read+write+admin` scopes when you upgrade
to the HTTP server, so no migration is required.
Existing bearer tokens are grandfathered as `read+write+admin` scopes on the
OAuth-capable HTTP server, so no migration is required.
## OAuth 2.1 Setup (v0.26.0+)
## OAuth 2.1 Setup
### 1. Start the HTTP server
@@ -97,8 +96,8 @@ Save this token. Open `http://localhost:3131/admin` and paste it to access the
dashboard. The dashboard shows live activity, registered clients, request logs,
and per-client config export.
> **v0.26.9+:** `mcp_request_log.params` and the live SSE activity feed default
> to a redacted summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`.
> `mcp_request_log.params` and the live SSE activity feed default to a redacted
> summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`.
> Declared param keys are kept (intersected against the operation's spec); unknown
> keys are counted but never named, and byte sizes round up to 1KB so size-probe
> attacks can't binary-search secret content. Operators on a personal laptop who
@@ -129,9 +128,9 @@ gbrain auth register-client perplexity \
--scopes "read write"
```
**v0.34 — source-scoped clients.** Multi-source brains can scope a client's
write authority to one source and its read scope to a curated set with the
new `--source` and `--federated-read` flags:
**Source-scoped clients.** Multi-source brains can scope a client's write
authority to one source and its read scope to a curated set with the
`--source` and `--federated-read` flags:
```bash
gbrain auth register-client dept-x-agent \
@@ -143,9 +142,12 @@ gbrain auth register-client dept-x-agent \
`--source` controls the write authority — `put_page` / `add_link` / etc only
land in `dept-x`. `--federated-read` controls the read axis independently;
queries return rows from any of the listed sources. Omit both flags for the
v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to
`source_id='default'` on `gbrain upgrade`.
queries return rows from any of the listed sources. Omit both flags for an
unscoped super-client. Clients registered before source scoping existed are
backfilled to `source_id='default'` on `gbrain upgrade`. Within a source,
slug-level write fencing is also available: `--bound-slug-prefixes p1/,p2/`
rejects slug-mutating writes outside the listed prefixes (update later with
`gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`).
Host-repo wrappers can register programmatically:
@@ -163,7 +165,7 @@ start the server with `--enable-dcr`. DCR is off by default.
### 3. Expose the server
**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
**Bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
To accept connections from the ngrok tunnel (or any non-loopback source),
restart with `--bind`:
@@ -187,10 +189,10 @@ router exposes the spec-compliant discovery endpoint at
### 4. Scopes and localOnly
Every operation is tagged `read | write | admin`. Four operations are
`localOnly` and rejected over HTTP regardless of scope: `sync_brain`,
`file_upload`, `file_list`, `file_url`. Remote agents cannot reach local
filesystem surface area.
Every operation is tagged `read | write | admin`. Operations flagged
`localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain` and
the `file_*` ops among them) are rejected over HTTP regardless of scope.
Remote agents cannot reach local filesystem surface area.
| Scope | What it allows |
|-------|---------------|
@@ -198,10 +200,13 @@ filesystem surface area.
| `write` | `put_page`, `delete_page`, `add_link`, `add_timeline_entry` |
| `admin` | Client management, token revocation, sweep, local-only ops |
Write ops can additionally be fenced per client with `--bound-slug-prefixes`
(see [Register OAuth clients](#2-register-oauth-clients) above).
## Legacy Bearer Token Setup
Keep using pre-v0.26 bearer tokens if you aren't ready to migrate. They
grandfather to `read+write+admin` scopes on the HTTP server.
Bearer tokens are the simple path when you don't need per-client scoping.
They grandfather to `read+write+admin` scopes on the HTTP server.
### 1. Set up the tunnel
@@ -248,15 +253,20 @@ gbrain auth test \
## Operations
All 30 GBrain operations are available remotely, including `sync_brain` and
`file_upload` (no timeout limits with self-hosted server).
GBrain's full operation catalog (100+ operations in `src/core/operations.ts`)
is available remotely, with no timeout limits on a self-hosted server. The
only exceptions are the operations flagged `localOnly: true``sync_brain`
and the `file_*` ops among them — which are rejected over HTTP regardless of
scope (see [Scopes and localOnly](#4-scopes-and-localonly) above).
**Security note on `file_upload`:** remote MCP callers are confined to the working
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
**Security note on file access:** the `file_*` operations being localOnly is
the first line of defense; as defense-in-depth, `file_upload` also confines
any caller that isn't verifiably the trusted local CLI to the working
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and
absolute paths outside cwd are rejected, and page slugs and filenames are
allowlist-validated (alphanumeric + hyphens; no control chars, RTL overrides,
or backslashes). Local CLI callers (`gbrain files upload ...`) keep
unrestricted filesystem access since the user owns the machine.
## Deployment Options
@@ -326,8 +336,8 @@ Remote servers must be added via Settings > Integrations, NOT
| put_page | 100-500ms | Write + trigger search_vector update |
| get_stats | < 100ms | Aggregate query |
**Note:** `gbrain serve --http` shipped in v0.26.0 with OAuth 2.1 + admin
dashboard baked into the binary. The custom HTTP wrapper pattern (see
**Note:** `gbrain serve --http` has OAuth 2.1 + the admin dashboard baked
into the binary. The custom HTTP wrapper pattern (see
[voice recipe](../../recipes/twilio-voice-brain.md)) is still supported for
teams that need bespoke middleware, but for most remote deployments the
built-in server is the recommended path.
+11 -5
View File
@@ -18,13 +18,16 @@ gbrain serve --http --port 3131 --bind 0.0.0.0 \
--public-url https://YOUR-DOMAIN.ngrok.app
```
- **`--bind 0.0.0.0` is required.** Since v0.34, `--http` defaults to
`127.0.0.1`, so without it the tunnel reaches the server but the connection is
refused (`ECONNREFUSED`).
- **`--bind 0.0.0.0` is required.** `--http` defaults to `127.0.0.1`, so
without it the tunnel reaches the server but the connection is refused
(`ECONNREFUSED`).
- **`--public-url` must match the tunnel.** The OAuth issuer in the discovery
metadata has to line up with the URL Perplexity actually hits (RFC 8414 §3.3),
or OAuth client-credentials auth fails.
Full detail on both flags (and the rest of the server setup) lives in
[DEPLOY.md — Expose the server](DEPLOY.md#3-expose-the-server).
## 2. Expose it with a tunnel
```bash
@@ -36,9 +39,12 @@ tunnel.
## 3. Create credentials
Two supported auth paths.
Two supported auth paths. (Full client-registration mechanics — the `/admin`
dashboard flow, grant types, scope format — live in
[DEPLOY.md — Register OAuth clients](DEPLOY.md#2-register-oauth-clients);
below is the Perplexity-specific shape.)
**OAuth 2.1 client credentials (recommended, v0.26.0+).** Perplexity is a cloud
**OAuth 2.1 client credentials (recommended).** Perplexity is a cloud
service, so it holds whatever credential you give it. OAuth is the correct choice:
least-privilege scopes + short-lived rotating access tokens instead of a
long-lived full-access secret. Mint a client and print the connector fields in
+20 -6
View File
@@ -1,8 +1,8 @@
# Headless install: Docker, CI, postinstall
As of v0.37, `gbrain init --pglite` in a non-TTY context (Docker `RUN`, CI step, postinstall hook) exits 1 when no embedding-provider API key is present in the environment. This is a deliberate fail-loud — the alternative was the v0.36 silent-broken-state class where init succeeded with a default that didn't match any real key.
`gbrain init --pglite` in a non-TTY context (Docker `RUN`, CI step, postinstall hook) exits 1 when no embedding-provider API key is present in the environment. This is a deliberate fail-loud — the alternative is a silent-broken state where init succeeds with a default that doesn't match any real key.
Two patterns work for headless installs. Pick whichever fits your image lifecycle.
Three patterns work for headless installs. Pick whichever fits your image lifecycle.
## Pattern 1: Provider key available at image build time
@@ -16,7 +16,7 @@ FROM oven/bun:1 AS builder
ARG OPENAI_API_KEY
ENV OPENAI_API_KEY=$OPENAI_API_KEY
RUN bun install -g github:garrytan/gbrain
RUN bun install -g github:garrytan/gbrain#latest-stable
RUN gbrain init --pglite # auto-picks OpenAI, persists config
```
@@ -26,7 +26,7 @@ RUN gbrain init --pglite # auto-picks OpenAI, persists config
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
bun install -g github:garrytan/gbrain
bun install -g github:garrytan/gbrain#latest-stable
gbrain init --pglite
```
@@ -38,7 +38,7 @@ If the API key is a runtime secret (Kubernetes secret, runtime env injection, en
```dockerfile
FROM oven/bun:1
RUN bun install -g github:garrytan/gbrain
RUN bun install -g github:garrytan/gbrain#latest-stable
# Build the brain shape without a provider — schema lands at the default
# width, but no embed callsite will actually run until runtime config.
@@ -59,6 +59,20 @@ The runtime `gbrain init --force` re-runs the init flow against the now-populate
- Resolves the provider via env detection.
- Re-templates the PGLite schema if dim differs from the build-time default.
## Pattern 3: No key, ever (keyless mode)
`--no-embedding` isn't only a deferral — it's also the install shape for **keyless mode**, a first-class supported end state (not a broken one). With zero provider keys, gbrain runs keyword-only (BM25) search and takes memory from agent-authored `## Facts` fences and write ops; embedding and extraction paths refuse cleanly instead of failing silently.
```dockerfile
FROM oven/bun:1
RUN bun install -g github:garrytan/gbrain#latest-stable
RUN gbrain init --pglite --no-embedding # keyless install — done; no runtime re-init needed
```
`gbrain bootstrap verify` (and the agent-bootstrap flow generally) prints an honest capability report for this posture — a "keyless mode" banner, per-touchpoint lines, and the one-key upsell (`src/core/capability.ts`). Keyless installs for the agent-bootstrap path are covered in `docs/guides/bootstrap.md`; this doc covers the Docker/CI shape. Adding a single provider key later upgrades in place via Pattern 2's runtime `gbrain init --force`.
Since every embedding cost gate is structurally moot with no key, none of `docs/operations/spend-controls.md` applies until you add one.
## What WON'T work
```dockerfile
@@ -67,7 +81,7 @@ The runtime `gbrain init --force` re-runs the init flow against the now-populate
RUN gbrain init --pglite
```
If you upgrade from a pre-v0.37 image that used this pattern, `gbrain doctor` will surface the mismatch on first run after upgrade and print a paste-ready repair command (`gbrain init --force --embedding-model …` for empty brains, `gbrain retrieval-upgrade --reindex` for non-empty).
If an older image used this pattern, `gbrain doctor` will surface the mismatch on first run after upgrade and print a paste-ready repair command `gbrain init --force --pglite --embedding-model <model> --embedding-dimensions <dims>` for brains with no embeddings yet, `gbrain migrate embeddings --to <model> --dim <dims>` for non-empty brains.
## Verifying a headless install
+6 -1
View File
@@ -8,6 +8,11 @@ The orienting idea: **GBrain itself is rounding error; the spend that matters is
downstream embedding.** These gates exist so a routine sync or enrich can't run up
an unexpected embedding bill, while never wedging an unattended cron.
**Keyless mode:** if you run with zero provider keys (`gbrain init --no-embedding`,
the keyless bootstrap posture — see `docs/guides/bootstrap.md` and
`docs/operations/headless-install.md`), nothing here can spend and none of these
gates ever fire. This doc applies once you add a key.
## `spend.posture` — one switch for "cost is not my constraint"
```bash
@@ -18,7 +23,7 @@ gbrain config set spend.posture gated # default — gates enforce
| Value | Effect |
|-------|--------|
| `gated` (default) | Every cost gate enforces its limit as documented below. |
| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. |
| `tokenmax` | Every embedding-spend gate in the table below prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. (Commands with their own LLM cost caps outside this doc's embedding scope — e.g. `extract-conversation-facts --max-cost-usd` — don't resolve posture; their per-call flags govern.) |
`spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs
retrieval payload size, not embedding spend). When a gate fires and
+2 -3
View File
@@ -212,13 +212,12 @@ gbrain schema lint --with-db
**Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal).
**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher.
**For agents (MCP):** the same operations are reachable over HTTPS MCP as schema ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher.
**Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`.
## Related docs
- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture.
- **Reference:** `gbrain schema --help` for the full CLI surface (30+ subcommands); the "Schema Cathedral v3" section of `docs/architecture/KEY_FILES.md` for the module-by-module architecture.
- **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit).
- **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix.
- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private).
+15
View File
@@ -105,6 +105,21 @@ gbrain skillpack scaffold <source> # owner/repo, https, ./dir, ./*.tgz
gbrain skillpack registry --url X # point at a custom registry
```
## Brain-resident packs
A brain/source repo can carry its own publishable skillpack (`brain_resident: true`
in `skillpack.json`, plus a `schema_pack` declaration). Scaffold one with:
```bash
gbrain skillpack init-brain-pack <name> # inside the brain repo; --dry-run to preview
```
Connecting harnesses discover the pack on `gbrain sources add`, and remote
agents reach it over MCP via the source-scoped `list_brain_skillpack` op +
`get_skill --source_id` (gated by the `mcp.publish_skills` config key). The
anatomy above applies unchanged — a brain-resident pack is a normal pack that
happens to live inside a brain repo.
## See also
- `examples/skillpack-reference/` — the live 10/10 reference pack
+1 -1
View File
@@ -4,7 +4,7 @@
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands.
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning; rename them in your config by hand to silence it.
## Configuration
+9 -9
View File
@@ -14,10 +14,10 @@ The epistemological layer. WHO believes WHAT, with confidence weight and time.
- **Scale:** 100K+ rows across thousands of holders in a mature brain
**Example takes:**
- `holder=people/garry-tan kind=bet` "AI will replace 50% of coding by 2030" (w=0.75)
- `holder=people/jared-friedman kind=take` "Momo has strong retention" (w=0.80)
- `holder=world kind=fact` "Clipboard raised $100M Series C" (w=1.0)
- `holder=brain kind=hunch` "Garry has a hero/rescuer pattern" (w=0.70)
- `holder=people/alice-example kind=bet` "AI will replace 50% of coding by 2030" (w=0.75)
- `holder=people/bob-example kind=take` "widget-co has strong retention" (w=0.80)
- `holder=world kind=fact` "acme-example raised a Series C" (w=1.0)
- `holder=brain kind=hunch` "alice-example has a hero/rescuer pattern" (w=0.70)
**Query surface:** `gbrain takes list`, `gbrain takes search`, `gbrain think`
@@ -32,9 +32,9 @@ Personal knowledge from the brain owner's conversations. Real-time capture.
- **Bridge:** Dream cycle `consolidate` phase promotes hot facts → cold takes nightly
**Example facts:**
- `kind=event` "I have a meeting with Brian tomorrow"
- `kind=event` "I have a meeting with alice-example tomorrow"
- `kind=preference` "I don't drink coffee"
- `kind=commitment` "We decided on nesting custody"
- `kind=commitment` "We decided to move the offsite to March"
- `kind=belief` "I think the market is overheated"
**Query surface:** `gbrain recall`, MCP `_meta.brain_hot_memory`
@@ -42,8 +42,8 @@ Personal knowledge from the brain owner's conversations. Real-time capture.
## The Category Error
**Never dump takes into the facts table.** Takes include other people's attributed
beliefs (Jared's assessment of a company, PG's view on schools, a founder's
revenue claims). These are NOT the brain owner's personal facts.
beliefs (a partner's assessment of a company, an investor's view on markets, a
founder's revenue claims). These are NOT the brain owner's personal facts.
**Never dump facts into the takes table without transformation.** Facts are
scoped to what the owner said in conversation. They become takes only through
@@ -85,7 +85,7 @@ First full takes extraction run on a ~100K-page brain:
### Key Learnings for Extraction Prompts
1. **Holder ≠ subject.** "Garry has a hero/rescuer pattern" → holder=brain, NOT people/garry-tan
1. **Holder ≠ subject.** "alice-example has a hero/rescuer pattern" → holder=brain, NOT people/alice-example
2. **Atomic claims.** Split compound claims into separate rows
3. **Amplification ≠ endorsement.** Retweet-only → max weight 0.55
4. **Self-reported ≠ verified.** "Reports 7 figures" → holder=person, weight=0.75, NOT world/1.0
+2 -1
View File
@@ -32,5 +32,6 @@ Tutorials follow the [Diataxis](https://diataxis.fr/) tutorial pattern: learning
- **Reference:** [`docs/architecture/`](../architecture/) — system design, topologies, retrieval theory
- **How-to:** [`docs/guides/`](../guides/) — task-oriented runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion). Highlight: [scaling skills past 300](../guides/scaling-skills.md) — the three-tier architecture for agents that have outgrown the always-loaded skill manifest.
- **Integrations:** [`docs/integrations/`](../integrations/) — connecting external data sources (voice, email, calendar, embedding providers)
- **MCP setup:** [`docs/mcp/`](../mcp/) — per-client setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
- **MCP setup:** [`docs/mcp/`](../mcp/) — per-client setup (Claude Desktop, Claude Code, Codex, ChatGPT, Perplexity, Cowork)
- **Install paths:** [`docs/INSTALL.md`](../INSTALL.md) — every install path, end to end
- **Agent bootstrap:** [`docs/guides/bootstrap.md`](../guides/bootstrap.md) — the paste-in path that turns a coding agent into a full agent (identity, per-turn context, schedules, a private repo as its durable body)
+40 -39
View File
@@ -85,7 +85,7 @@ git clone git@github.com:your-org/customers.git customers
git clone git@github.com:your-org/internal-docs.git internal
```
You can also keep the existing personal-brain repo as one of the sources. Just pick the role it plays (probably `shared` if it's already org-wide content).
You can also keep the existing personal-brain repo as one of the sources. Just pick the role it plays (probably `shared` if it's already org-wide content). When agents on the host write pages into a source, `gbrain sources push <id>` (run on the host) commits and pushes those changes back to the source's git repo, so the repo stays the durable system of record.
### Two scoping models (pick the one that matches your shape)
@@ -156,7 +156,7 @@ The personal brain talks to you through the AlphaClaw harness over Telegram. For
gbrain serve --http --port 3131 --bind 0.0.0.0
```
The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface.
The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface. (Full detail on `--bind` / `--public-url`, including the ECONNREFUSED failure mode they prevent, lives in [DEPLOY.md — Expose the server](../mcp/DEPLOY.md#3-expose-the-server).)
The server prints an admin bootstrap token to stderr on first start when run in an interactive terminal. Save it. You'll use it once for the admin dashboard. On a non-TTY start (systemd, Docker, piped logs) the token is hidden from logs — set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` yourself or pass `--print-admin-token` on a trusted terminal instead.
@@ -186,56 +186,67 @@ Each teammate (or each AI agent for a teammate) gets their own OAuth client. The
# Alice (sales): writes customers/alice-example, reads customers + shared
gbrain auth register-client alice-example \
--grant-types client_credentials \
--scopes read,write \
--scopes "read write" \
--source customers \
--federated-read customers,shared
# Bob (ops): writes internal/bob-example, reads internal + shared
gbrain auth register-client bob-example \
--grant-types client_credentials \
--scopes read,write \
--scopes "read write" \
--source internal \
--federated-read internal,shared
# Carol (legal): writes shared/legal, reads all three
gbrain auth register-client carol-example \
--grant-types client_credentials \
--scopes read,write \
--scopes "read write" \
--source shared \
--federated-read shared,customers,internal
```
Each `register-client` command prints a `client_id` and a `client_secret`. Save both for each teammate. They go into the teammate's local agent config.
Each `register-client` command prints a `client_id` and a `client_secret`. Save both for each teammate. They go into the teammate's local agent config. (The full registration reference — grant types, the `/admin` dashboard flow, DCR — lives in [DEPLOY.md — Register OAuth clients](../mcp/DEPLOY.md#2-register-oauth-clients). What follows is the multi-user delta.)
A note on the flags:
- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--scopes "read write"` (space-separated, quoted — the OAuth wire format; a comma-separated list is rejected at registration) lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder — and you can make that server-enforced with `--bound-slug-prefixes alice-example/` (v0.42.72.0+): every slug-mutating write op (put_page, delete_page, tags, links, timeline, revert, raw data) outside the bound prefixes is rejected with `permission_denied`. Update the binding later with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`. **Adding a binding to an existing client narrows it in ways you should expect:** ops that write by something other than a slug (`extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`) and `POST /ingest` become unavailable to that client, and `put_page`'s automatic fact extraction is skipped — all because none of them can be confined to a prefix. Reads are unaffected. See [the qm-harness guide](../integrations/qm-harness.md) for the full model.
- `--federated-read` controls read scope. A client can read from one or more sources.
### Verify the scoping actually scopes
Before you hand the brain to teammates, verify isolation. Two terminal windows on your local machine using each client's credentials:
Before you hand the brain to teammates, verify isolation. The clean way is a **thin-client install** on a second machine (or a scratch shell): `gbrain init --mcp-only` writes a config that routes every CLI command through your remote server as one specific OAuth client, so a plain `gbrain search` exercises exactly the path teammates will use.
```bash
# Terminal 1, as Alice
export GBRAIN_REMOTE_CLIENT_ID=<Alice's client_id>
export GBRAIN_REMOTE_CLIENT_SECRET=<Alice's client_secret>
export GBRAIN_REMOTE_MCP_URL=https://brain.acme-co.com/mcp
# As Alice (on a machine that is NOT the brain host)
gbrain init --mcp-only \
--issuer-url https://brain.acme-co.com \
--mcp-url https://brain.acme-co.com/mcp \
--oauth-client-id <Alice's client_id> \
--oauth-client-secret <Alice's client_secret>
gbrain search "performance review" --remote
gbrain whoami # confirms which client you're acting as
gbrain search "performance review"
```
Alice should see results only from `customers` and `shared`. The performance-review notes live in `internal`, which she's not scoped to read. She shouldn't see them.
Now re-run the same check as Bob. On the same test machine, swap the credentials with `--force` (it overwrites the thin-client config):
```bash
# Terminal 2, as Bob (export his credentials similarly)
gbrain search "performance review" --remote
gbrain init --mcp-only --force \
--issuer-url https://brain.acme-co.com \
--mcp-url https://brain.acme-co.com/mcp \
--oauth-client-id <Bob's client_id> \
--oauth-client-secret <Bob's client_secret>
gbrain whoami
gbrain search "performance review"
```
Bob should see the performance-review notes from `internal`, plus anything related from `shared`. He shouldn't see anything that lives only in `customers`.
If both queries return correctly scoped results, isolation is working.
If both queries return correctly scoped results, isolation is working. (There is no per-query "act as client X" flag — the thin-client config decides which credential the CLI uses; only the client secret can be overridden at call time via `GBRAIN_REMOTE_CLIENT_SECRET`.)
---
@@ -274,13 +285,13 @@ copy to customers/alice-example/digests/YYYY-MM-DD-pipeline.md.
The `client:` field tells the cron runner which OAuth client to use, which enforces the scoping. Alice's cron can only read Alice's sources and write to Alice's folder. It cannot accidentally touch Bob's customer notes.
To install the cron schedule, commit the file to the workspace repo and let AlphaClaw pick it up on next deploy. The cron-scheduler skill (one of the 60 that GBrain installed) handles the dispatch.
To install the cron schedule, commit the file to the workspace repo and let AlphaClaw pick it up on next deploy. The cron-scheduler skill (one of the bundled skills GBrain installed) handles the dispatch. (The `client:` frontmatter field is a workspace/harness convention — your cron runner reads it and picks the matching OAuth credential; GBrain enforces the scoping once the credential is used.)
---
## Part 7: Add per-person skills
The 60+ skills GBrain installs are generic. Your team probably wants a few that are specific to them. Examples:
The bundled skills GBrain installs are generic. Your team probably wants a few that are specific to them. Examples:
- `onboarding-new-hire`. Only Carol (HR) runs this. Walks through generating a welcome packet, scheduling intro meetings, provisioning accounts.
- `customer-success-followup`. Only Alice (sales) runs this. Pulls latest customer page, drafts a follow-up email, posts to her review queue.
@@ -307,7 +318,7 @@ gbrain skillify scaffold onboarding-new-hire
That creates the directory + SKILL.md + routing entry. Edit the SKILL.md to describe the procedure, commit, deploy. The agent picks up the new skill on next request.
Per-person scoping for skills is handled at the routing layer: a skill can declare `allowed_clients: [carol-example]` in its frontmatter. If Alice asks her agent to run that skill, the agent refuses with "this skill is scoped to carol-example."
Per-person scoping for skills is a routing-layer **convention, enforced by your agent harness, not by GBrain**: declare something like `allowed_clients: [carol-example]` in the skill's frontmatter and instruct your agent (in its routing rules) to refuse the skill for anyone else. The hard guarantee stays at the data layer — even if the agent runs the skill anyway, Alice's OAuth credential still can't read or write outside her scoped sources.
### Shared rule files at the skills root
@@ -382,9 +393,9 @@ Repeat this flow for every new teammate. About 45 minutes per person, total. Com
## Part 10: Connect each teammate's AI client
Each teammate runs their AI client (Claude Code, Cursor, Claude Desktop, OpenClaw, Hermes, whatever) configured to point at your brain server through their OAuth credentials.
Each teammate runs their AI client (Claude Code, Codex, Claude Desktop, OpenClaw, Hermes, whatever) configured to point at your brain server. Two pieces, both direct-to-server — there is no local relay in between:
Recommended path for each teammate: the thin-client install. On their machine:
**1. The GBrain CLI, as a thin client (recommended for everyone).** On their machine:
```bash
curl -fsSL https://bun.sh/install | bash
@@ -397,24 +408,14 @@ gbrain init --mcp-only \
--oauth-client-secret <their client_secret>
```
The thin-client install creates a local config that knows how to talk to your brain but never opens its own database. Most CLI commands route through the remote server transparently.
The thin-client install creates a local config that knows how to talk to your brain but never opens its own database. From then on, plain CLI commands (`gbrain search`, `gbrain query`, `gbrain think`, `gbrain whoami`, ...) route through your remote server transparently, as that teammate's OAuth client. Local-only commands (`gbrain sync`, `gbrain serve`, `gbrain embed`, ...) are refused with a hint — those run on the brain host, not on teammate laptops.
Now they configure their AI client. For Claude Desktop, the teammate adds an MCP server entry in `~/Library/Application Support/Claude/claude_desktop_config.json`:
**2. Their AI client, connected directly to `https://brain.acme-co.com/mcp`.** Each client has its own connection shape; the per-client pages in [`docs/mcp/`](../mcp/) are the reference:
```jsonc
{
"mcpServers": {
"company-brain": {
"command": "gbrain",
"args": ["serve"]
}
}
}
```
When Claude Desktop launches, it talks to the local `gbrain serve` stdio bridge, which forwards every request to your remote brain over HTTPS with their OAuth token attached. From Claude Desktop's perspective it's just one MCP server.
For Claude Code, Cursor, OpenClaw, Hermes, and other clients, per-client setup steps live in [`docs/mcp/`](../mcp/). They all follow the same shape: point the agent at the local `gbrain serve` bridge, which knows about the remote.
- **Claude Code / Codex** — one command from anywhere `gbrain` is installed: `gbrain connect https://brain.acme-co.com/mcp --token <token> --install` (see [CLAUDE_CODE.md](../mcp/CLAUDE_CODE.md) / [CODEX.md](../mcp/CODEX.md)). Note the credential type: `gbrain connect` for these two agents uses **bearer tokens** (`gbrain auth create <name>`), which are full-access. That's fine for you as the admin; for source-scoped teammates, the scoped credential is their OAuth client — use it via the thin-client CLI above and the OAuth-capable clients below.
- **Claude Desktop** — remote servers are added through the GUI: **Settings > Integrations**, URL `https://brain.acme-co.com/mcp`. Do **not** put a remote server in `claude_desktop_config.json`; that file only works for local stdio servers and fails silently for remote ones. See [CLAUDE_DESKTOP.md](../mcp/CLAUDE_DESKTOP.md).
- **ChatGPT** ([CHATGPT.md](../mcp/CHATGPT.md)) and **Perplexity** ([PERPLEXITY.md](../mcp/PERPLEXITY.md)) — both speak OAuth to the server directly, so per-teammate scoping carries into those tools. Perplexity uses the same `client_credentials` clients you registered in Part 5. ChatGPT needs an `authorization_code` (PKCE) client — register one per teammate with the same `--source` / `--federated-read` flags.
- **OpenClaw / Hermes forks** — if the teammate's own agent runs on a machine with a full local gbrain install, it can use local stdio (`gbrain serve`) against its own brain and reach yours over HTTP MCP like any other remote client.
---
@@ -518,7 +519,7 @@ The first sync embeds every page, which takes time. Check `gbrain sources status
### "I see a page I shouldn't see"
This shouldn't happen, but if you suspect it, run `gbrain search <query> --remote --json` as the constrained client and inspect the `source_id` field on every returned result. Every row should be in the client's `--federated-read` set. If one isn't, file an issue with the exact slug and source IDs.
This shouldn't happen, but if you suspect it, run `gbrain search "<query>" --json` from a thin-client install configured with the constrained client's credentials (the Part 5 verification setup) and inspect the `source_id` field on every returned result. Every row should be in the client's `--federated-read` set. If one isn't, file an issue with the exact slug and source IDs.
### "The synthesized answer is wrong"
@@ -537,7 +538,7 @@ Each parallel sync worker opens its own pool. With three sources and the default
```bash
gbrain auth register-client diana-example \
--grant-types client_credentials \
--scopes read,write \
--scopes "read write" \
--source shared \
--federated-read shared,customers,internal
```
+10 -5
View File
@@ -6,6 +6,11 @@ you decided three meetings ago, and re-derive context you already have written
down somewhere. GBrain is the retrieval layer that fixes that: search, synthesis,
and a self-wiring knowledge graph, wired into your agent over MCP.
> This tutorial is the **memory-only** tier. Want the whole agent — identity,
> per-turn context, schedules, and a private GitHub repo as its durable body?
> That's `gbrain bootstrap`: the "For Codex" / "For Claude Code" paste blocks in
> the README, full contract in [docs/guides/bootstrap.md](../guides/bootstrap.md).
There are two ways to do this. Pick the one that matches where you are:
- **Path A — I already run a brain** (OpenClaw, Hermes, or any `gbrain serve`
@@ -61,9 +66,9 @@ special). Turn it on:
gbrain config set mcp.publish_skills true
```
(New brains from `gbrain init` default this ON. Brains upgraded from before
v0.41.36 stay OFF until you opt in, so this is the common gotcha for existing
OpenClaw users.)
(New brains from `gbrain init` default this ON. Brains upgraded from a release
before skill publishing existed stay OFF until you opt in, so this is the
common gotcha for existing OpenClaw users.)
### A2. On the host: mint a token
@@ -91,7 +96,7 @@ token fails right now, not silently on the agent's first request. You'll see:
```
Added MCP server 'gbrain' -> https://your-host.example.com/mcp.
Verified: {"version":"0.42.x","engine":"postgres","page_count":146646,...}
Verified: {"version":"0.42.x","engine":"postgres","page_count":1204,...}
```
Drop `--install` to print a paste-ready block instead (useful when the host and
@@ -243,7 +248,7 @@ habits to build. Your agent stops being amnesiac.
## Next steps
- Go full autonomous: the overnight enrichment daemon ([dream cycle](../../CHANGELOG.md)) fixes citations, dedupes people, builds scorecards while you sleep. See `gbrain autopilot --install`.
- Go full autonomous: the overnight enrichment daemon ([dream cycle](../guides/operational-disciplines.md)) fixes citations, dedupes people, builds scorecards while you sleep. See `gbrain autopilot --install`.
- Run a real agent platform on top: [personal-brain tutorial](personal-brain.md).
- Scale to a team: [company-brain tutorial](company-brain.md).
- Every MCP client's exact setup: [`docs/mcp/`](../mcp/).
+4 -4
View File
@@ -6,7 +6,7 @@ This is the install I'd run if I were setting up the whole stack from scratch to
> "This is the Apple I, we're just soldering breadboards over here."
If you only want the **brain layer** (no agent, no Telegram, just gbrain as memory for an MCP client you already use), skip to the [CLI standalone install](../INSTALL.md#2-cli-standalone) in INSTALL.md. If you want the whole agent **shared with a team**, read the [company brain tutorial](company-brain.md) instead. This tutorial is the solo, full-stack, talk-to-it-on-Telegram path.
If you only want the **brain layer** (no agent, no Telegram, just gbrain as memory for an MCP client you already use), skip to the [CLI standalone install](../INSTALL.md#2-cli-standalone) in INSTALL.md. If your daily driver is a **coding agent** (Claude Code / Codex) and you want it to bootstrap its own full agent — identity, per-turn context, schedules, a private repo as its durable body — that's `gbrain bootstrap`: see the paste blocks in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md). If you want the whole agent **shared with a team**, read the [company brain tutorial](company-brain.md) instead. This tutorial is the solo, full-stack, talk-to-it-on-Telegram path.
---
@@ -17,7 +17,7 @@ A personal AI agent with four pieces:
- **A brain** (git repo). Your knowledge base, constantly ingesting and growing.
- **A harness** (OpenClaw via AlphaClaw). The runtime that gives the LLM tools, memory, and integrations.
- **A chat interface** (Telegram). How you talk to it.
- **Skills** (60+ installed via GBrain). Reusable capabilities the agent can invoke.
- **Skills** (50+ installed via GBrain). Reusable capabilities the agent can invoke.
Architecture:
@@ -127,7 +127,7 @@ gbrain skillpack scaffold --all
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
`gbrain skillpack scaffold --all` copies the 50+ bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired; see `docs/INSTALL.md` if you're upgrading from an older release.)
From this point, the agent has working memory and access to every skill.
@@ -252,7 +252,7 @@ My production setup is about $10,000 a month, but that's 10 instances, 200 crons
2. **GitHub PAT can't see the repos.** Reload the page after creating repos. Make sure the fine-grained token has the correct repo selection.
3. **Telegram bot doesn't respond.** Check the bot token in AlphaClaw. Make sure the Render instance is actually running.
4. **Supabase bottleneck on heavy ingestion.** Upgrade the DB instance size before the small one chokes.
5. **GBrain.io provisioning fails.** The hosted instance may need Pro tier. Check the machine allocation in the AlphaClaw UI.
5. **Hosted-instance provisioning fails.** If you're using a hosted brain instance instead of self-hosting, it may need the Pro tier. Check the machine allocation in the AlphaClaw UI.
---
+4 -4
View File
@@ -107,7 +107,7 @@ The schema is the team's tribal knowledge made explicit. Two engineers on differ
This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for.
Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes:
Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes:
> You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page.
@@ -147,7 +147,7 @@ Three things gbrain does that generic note systems can't:
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees.
**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through atomic CLI verbs + MCP ops (`gbrain schema --help` for the full surface) with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees.
## What changed in v0.40.7.0 specifically
@@ -156,7 +156,7 @@ v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pac
v0.40.7.0 closed those gaps:
- **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race.
- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking."
- **Per-pack `O_CREAT|O_EXCL` atomic lock** (deliberately NOT the TOCTOU-prone `existsSync+writeFileSync` pattern). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking."
- **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`.
- **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:<clientId8>`).
- **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`.
@@ -169,7 +169,7 @@ The cumulative effect: an agent can safely co-curate your ontology with a comple
- **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end.
- **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity.
- **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types.
- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it.
- **Want the architecture?** The "Schema Cathedral v3" section in `docs/architecture/KEY_FILES.md` has the module-by-module breakdown.
- **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes.
The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app.
+385 -289
View File
File diff suppressed because it is too large Load Diff
+5 -2
View File
@@ -50,13 +50,14 @@
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
"check:cli-exec": "bash scripts/check-cli-executable.sh",
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-pglite-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
"check:wasm": "bash scripts/check-wasm-embedded.sh",
"check:pglite-embedded": "bash scripts/check-pglite-embedded.sh",
"check:newlines": "bash scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
@@ -72,6 +73,8 @@
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
"check:bootstrap-tag": "bash scripts/check-bootstrap-tag.sh",
"check:bootstrap-templates": "bash scripts/check-bootstrap-templates.sh",
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
"check:privacy": "bash scripts/check-privacy.sh",
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
@@ -149,7 +152,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.44.1.0",
"version": "0.45.0.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.5",
+15 -19
View File
@@ -10,15 +10,9 @@ secrets:
- name: OPENAI_API_KEY
description: OpenAI API key with Realtime API access enabled
where: https://platform.openai.com/api-keys — click "+ Create new secret key", copy immediately
- name: TWILIO_ACCOUNT_SID
description: (optional) Twilio Account SID — only if wiring inbound Twilio calls
where: https://www.twilio.com/console
- name: TWILIO_AUTH_TOKEN
description: (optional) Twilio auth token — only if wiring inbound Twilio calls
where: https://www.twilio.com/console
health_checks:
- type: env_exists
var: OPENAI_API_KEY
name: OPENAI_API_KEY
label: OPENAI_API_KEY present
setup_time: 10 min
cost_estimate: "$0.06-0.24/min OpenAI Realtime, optional $1-2/mo Twilio number"
@@ -34,20 +28,22 @@ A reference voice agent (WebRTC-first; OpenAI Realtime) shipped as **copy-into-y
- **WebRTC browser client** at `/call?test=1` for the production-grade voice loop. Production load installs zero test instrumentation; `?test=1` enables Web Audio API tee → MediaRecorder capture for the E2E.
- **Tool router** with a read-only allow-list by default (search, query, get_page, list_pages, find_experts, get_recent_salience, get_recent_transcripts, read_article). Write ops are denylisted; operators opt in to a bounded set via local override.
- **Persona-aware prompt builder** with identity-first composition + Unicode sanitization for Realtime API safety.
- **Optional Twilio adapter** (`/voice` TwiML, WSS bridge) for phone inbound. Skip if you only want browser voice.
- **Optional Twilio adapter** (`/voice` TwiML, WSS bridge) for phone inbound. Skip if you only want browser voice. If you wire it, set `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` in `$TARGET_REPO/.env` (from https://www.twilio.com/console). They're deliberately NOT in this recipe's `secrets:` frontmatter — every listed secret must be set before the integration reports `configured`, and Twilio is genuinely optional.
- **Three skills** for resolver routing: `voice-persona-mars`, `voice-persona-venus`, `voice-post-call`.
- **Unit + E2E tests** that ride with the copy. PII-shape regex guards every prompt, classifier triages upstream vs plumbing failures.
- **Unit tests** that ride with the copy (PII-shape regex guards every prompt; a classifier triages upstream vs plumbing failures). The E2E and eval suites stay gbrain-side under `recipes/agent-voice/tests/` — see Tests below.
## The skillpack-as-reference paradigm
Earlier gbrain skillpacks installed to `~/.gbrain/skills/<name>/` as managed-block-canonical first-class skills. The user's local edits drifted from the canonical and updates were either "overwrite local" or "skip update" — neither is what an operator wants on code they've extended.
This recipe ships a different shape: gbrain holds the up-to-date REFERENCE, and `gbrain integrations install agent-voice --target <host-repo>` COPIES it into the operator's repo. The code now lives in the host repo, on the operator's release cadence, with the operator's edits. Subsequent `--refresh` invocations diff host-side files against gbrain's reference and propose changes; the operator picks per-file (keep mine / take theirs / merge).
This recipe ships a different shape: gbrain holds the up-to-date REFERENCE, and `gbrain integrations install agent-voice --target <host-repo>` COPIES it into the operator's repo. The code now lives in the host repo, on the operator's release cadence, with the operator's edits. Subsequent `--refresh` invocations diff host-side files against gbrain's reference and apply updates while preserving local edits by default (`--auto keep-mine|take-theirs` for CI lanes).
The shipped reference does NOT contain personal names, hardcoded private paths, or upstream-agent codenames. A CI guard (`scripts/check-no-pii-in-agent-voice.sh`) blocks any drift back; a deterministic import script (`scripts/import-from-upstream.sh`) refreshes the gbrain reference from an upstream voice-agent source.
## Install
> Note: this recipe's category is `voice`, which the `gbrain integrations list` dashboard does not render (it shows infra / sense / reflex sections only). Install it directly by id, as below.
```bash
# 1. Detect target repo
export TARGET_REPO=$OPENCLAW_WORKSPACE # or your agent repo path
@@ -85,15 +81,9 @@ git -C $(which gbrain | xargs -I{} dirname {})/.. pull # or your gbrain update
gbrain integrations install agent-voice --target $TARGET_REPO --refresh
```
`--refresh` reads the `.gbrain-source.json` manifest written by the original install, re-computes per-file SHA-256 against gbrain's current reference, and classifies each file:
`--refresh` reads the `.gbrain-source.json` manifest written by the original install, re-computes per-file SHA-256 against gbrain's current reference, and classifies each file into one of six states (identical / stale / locally-modified / host-deleted / source-deleted / new-in-manifest). Stale and new files are updated automatically; **locally-modified files are preserved by default** (`--auto take-theirs` to overwrite; `--dry-run` to preview). An append-only audit journal is written to `<target>/services/voice-agent/.gbrain-source.refresh.log`.
- **unchanged-identical** — host file matches gbrain reference; skip.
- **unchanged-stale** — host file matches the recorded SHA but reference moved; offer to update.
- **locally-modified** — host file diverges from the recorded SHA; show diff, offer three options (keep mine / take theirs / merge).
- **source-deleted** — gbrain reference removed a file; offer cleanup.
- **source-renamed** — detected via path-mapping; offer to follow.
A transaction journal at `<target>/services/voice-agent/.gbrain-source.refresh.log` allows partial-apply recovery if the refresh is interrupted.
The full state machine and per-state decisions live in [`recipes/agent-voice/install/refresh-algorithm.md`](agent-voice/install/refresh-algorithm.md) — the single home for refresh semantics.
## Architecture
@@ -142,9 +132,15 @@ Reference code ships intentionally minimal. Before public deployment:
## Tests
The install copies the **unit suites only**; the E2E and eval suites stay gbrain-side under `recipes/agent-voice/tests/` (they carry puppeteer fixtures and live-API costs the host repo shouldn't inherit).
```bash
# Host-side (rides with the copy)
cd $TARGET_REPO/services/voice-agent
bun run test # host-side unit tests (5 suites, ~100 cases)
bun run test # unit tests
# gbrain-side (from your gbrain checkout)
cd <gbrain-checkout>/recipes/agent-voice && bun install
AGENT_VOICE_E2E=1 bun run test:e2e # WebRTC roundtrip (~$0.10/run)
AGENT_VOICE_FULL_E2E=1 bun run test:full-flow # openclaw-driven install + roundtrip (~$1-2/run)
```
+5 -5
View File
@@ -25,10 +25,10 @@ recipes/<name>/
├── README.md # paradigm doc; gbrain-side only (not copied)
├── package.json # top-of-bundle; copied to <host>/services/<name>/package.json
├── code/ # copied to <host>/services/<name>/code/
├── tests/ # copied to <host>/services/<name>/tests/
│ ├── unit/
│ ├── e2e/
│ └── evals/
├── tests/
│ ├── unit/ # copied to <host>/services/<name>/tests/unit/
│ ├── e2e/ # gbrain-side only (puppeteer + live-API costs)
│ └── evals/ # gbrain-side only (LLM-judge suites)
├── skills/ # copied to <host>/skills/<skill-name>/
├── install/ # gbrain-side only; install metadata
│ ├── manifest.json # src → target map + per-file SHA-256
@@ -54,7 +54,7 @@ Three rules for new recipes following this shape:
- `code/lib/personas/private-name-blocklist.json` — privacy guard source of truth (read by the shipped guard script and by host-side prompt-shape tests).
- `code/lib/personas/context-builder.contract.md` — API the operator implements for live brain context.
## Files (in `bundle = code/ + tests/ + skills/ + package.json`) — copied to host repo
## Files (in `bundle = code/ + tests/unit/ + skills/ + package.json`) — copied to host repo
The install subcommand reads `install/manifest.json` and copies each listed file to its target path under the host repo. SHA-256s computed at copy time get persisted into `<host>/services/<name>/.gbrain-source.json` so `--refresh` can do three-way classification (unchanged-identical / unchanged-stale / locally-modified) without re-walking the entire bundle.
@@ -67,9 +67,14 @@ bun run start # or `npm start`
Open `http://localhost:8765/call` in a browser, click Connect, grant mic permission. You should be talking to Venus (or Mars if you set `DEFAULT_PERSONA=mars`).
### 6. (Optional) Run the WebRTC roundtrip E2E
### 6. (Optional) Run the WebRTC roundtrip E2E — from the gbrain checkout
The install copies **unit tests only**. The E2E and eval suites stay gbrain-side
(under `recipes/agent-voice/tests/`), so run them from your gbrain checkout,
not from `<target-repo>`:
```bash
cd <gbrain-checkout>/recipes/agent-voice && bun install
export AGENT_VOICE_E2E=1 OPENAI_API_KEY=sk-...
bun run test:e2e
# → ~$0.10/run; spawns server, drives puppeteer with a fake-audio WAV
@@ -83,15 +88,15 @@ gbrain claw-test --scenario voice-agent-install --live --agent openclaw
# → ~$1-2/run; friction-discovery test, NOT a ship gate
```
### 7. (Optional) Run the LLM-judge persona evals
### 7. (Optional) Run the LLM-judge persona evals — from the gbrain checkout
```bash
cd <target-repo>/services/voice-agent
cd <gbrain-checkout>/recipes/agent-voice
node tests/evals/mars-eval.mjs # ~$1-3 for the full 3-model judge sweep
node tests/evals/venus-eval.mjs
```
Synthetic canonical baselines are committed under `tests/evals/baseline-runs/canonical/`. Live receipts you generate go to `tests/evals/baseline-runs/` (gitignored — they may contain residual brain content from your live personas).
Live receipts you generate go to `tests/evals/baseline-runs/` (gitignored — they may contain residual brain content from your live personas). See `tests/evals/README.md` for the pass criteria and failure triage.
### 8. Update later
@@ -101,4 +106,4 @@ When gbrain ships a new agent-voice reference, refresh your local copy:
gbrain integrations install agent-voice --target <target-repo> --refresh
```
The refresh classifies each file (identical / stale / locally-modified / source-deleted / host-deleted) and lets you decide per-file. See `<target>/services/voice-agent/code/install/refresh-algorithm.md` (copied from gbrain) for the contract.
Refresh classifies each file (six states — identical / stale / locally-modified / host-deleted / source-deleted / new-in-manifest) and applies a deterministic decision per state: local edits are preserved by default; pass `--auto take-theirs` to take upstream everywhere, or `--dry-run` to preview. The full contract lives gbrain-side at `recipes/agent-voice/install/refresh-algorithm.md` (not copied to the host repo).

Some files were not shown because too many files have changed in this diff Show More