Compare commits

..
Author SHA1 Message Date
Garry Tan e22b8fb555 Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	llms-full.txt
#	package.json
#	src/commands/sync.ts
2026-04-29 22:49:21 -07:00
Garry Tan 2a9feb859f Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	llms-full.txt
#	package.json
2026-04-29 22:18:26 -07:00
Garry Tan 15b9316dbf Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
2026-04-29 11:37:09 -07:00
Garry TanandClaude Opus 4.7 f739de5521 chore: regenerate llms-full.txt for v0.22.13 doc updates
CI's build-llms generator test failed because llms-full.txt was stale
relative to the README + CLAUDE.md updates this PR added (--workers
flag in the IMPORT section, sync-concurrency.ts/db-lock.ts/sync.ts
entries in the Key files section).

Per CLAUDE.md: "Run \`bun run build:llms\` after adding a new doc."
The test test/build-llms.test.ts:67 verifies committed bundles match
generator output — now they do again.

llms.txt was already in sync (no curated config additions); only
llms-full.txt needed the regen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 08:23:44 -07:00
Garry TanandClaude Opus 4.7 02d585c0a4 chore: bump version slot to v0.22.13
VERSION 0.22.10 → 0.22.13. Master moved to 0.22.8 plus claimed slots
0.22.9-0.22.12 in sibling workspaces; 0.22.13 is the next free slot for
this PR's parallel-sync hardening work.

Updated all v0.22.10 references in CHANGELOG.md (release header +
self-repair block), TODOS.md (D-PR490-1 follow-up tag), CLAUDE.md
(Key files entries + tests + commands subsection), and the inline
v0.22.10 markers in src/core/sync-concurrency.ts, src/core/db-lock.ts,
src/commands/sync.ts, src/commands/import.ts, src/commands/jobs.ts,
test/sync-parallel.test.ts, test/e2e/sync-parallel.test.ts.

No behavioral change. CHANGELOG header rewrite, content unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 20:05:17 -07:00
Garry Tan e573fa6988 Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	package.json
2026-04-28 19:56:25 -07:00
Garry Tan ff6320e552 Merge remote-tracking branch 'origin/master' into pr-490
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	TODOS.md
#	VERSION
#	package.json
2026-04-28 08:53:54 -07:00
Garry TanandClaude Opus 4.7 36c750bbec docs: update CLAUDE.md + README for v0.22.10 sync hardening
CLAUDE.md:
- New "Key files" entries for src/core/sync-concurrency.ts and
  src/core/db-lock.ts (both v0.22.10).
- New "Key files" entry for src/commands/sync.ts (covers the lock,
  head-drift gate, engine.kind discriminator, vanished-file failure
  capture, parallel branch wiring).
- Updated src/commands/jobs.ts entry with v0.22.10 sourceId
  resolution + autoConcurrency policy + noEmbed contract.
- Added test/sync-concurrency.test.ts and test/sync-parallel.test.ts
  to the unit-test list with case counts.
- Added test/e2e/sync-parallel.test.ts to the E2E section with the
  SYNC_PARALLEL_BENCH grep marker for CHANGELOG quoting.
- Added "Key commands added in v0.22.10" section: gbrain sync --workers,
  gbrain import --workers (parseWorkers validation).

README.md: added --workers flag to the IMPORT section's gbrain sync
and gbrain import lines, with the >100-file auto-parallelize note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:47:19 -07:00
Garry TanandClaude Opus 4.7 7f2c81f929 chore: v0.22.10 release notes + sync follow-up TODO
VERSION + package.json + bun.lock: 0.22.5/0.22.6 → 0.22.10. Repo had
existing drift between VERSION and package.json on master; this commit
brings them back in sync at the bumped value.

CHANGELOG.md: v0.22.10 entry replaces the unfinished v0.23.0 stub from
PR #490's original commit. Voice-rule clean (no em dashes, no AI
vocabulary), real benchmark numbers from the new E2E test
(serial=289ms parallel(4)=221ms speedup=1.31x), additive worker-pool
note (A3), 'To take advantage of v0.22.10' self-repair block per
CLAUDE.md convention.

TODOS.md: A4 follow-up filed — plumb resolved database_url through
SyncOpts so performSync / performFullSync / import.ts don't each call
loadConfig() separately. Deferred to a future patch; not on the
v0.22.10 critical path.

Patch (not minor) framing held even though new CLI surface lands here;
release-notes prose names the behavior change explicitly so users know
to read them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:43:07 -07:00
Garry TanandClaude Opus 4.7 1353366b5f test: e2e parallel sync against real Postgres + benchmark
DATABASE_URL-gated E2E coverage that PGLite-only tests can't reach:

T2 — happy path: 60 files imported at concurrency=4, all 60 pages land
in the DB, with a pg_stat_activity probe before/after to confirm worker
engines (4 × 2 connections) actually disconnected.

P4 — benchmark: 120-file fixture, serial vs concurrency=4 timing.
Emits a single-line `SYNC_PARALLEL_BENCH 120 files | serial=Xms |
parallel(4)=Yms | speedup=Zx` so the CHANGELOG can quote a real
number instead of an unbacked '~4×' claim. Asserts parallel <=
serial * 1.5 to allow for noisy CI but fail genuine regressions.

Skips gracefully when DATABASE_URL is unset (consistent with the rest
of test/e2e/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:54 -07:00
Garry TanandClaude Opus 4.7 93ae40dd3a fix: jobs.ts sync handler — resolve sourceId, autoConcurrency
CODEX-1: resolve sourceId at handler entry by looking up sources.local_path.
Mirrors cycle.ts:480's autopilot-cycle fix (PR #475). Without this, every
Minion sync job on a multi-source brain reads global config.sync.last_commit
instead of the per-source anchor, which on a regularly-GC'd repo can drop
out of git history and trigger 30-min full reimports every cycle.

The handler accepts an optional sourceId job param for callers that want
to override; falls back to the resolveSourceForDir lookup when absent.

CODEX-4: replace the hardcoded concurrency=4 default with the shared
autoConcurrency policy. Behavior is now consistent between CLI sync,
the Minion handler, and the autopilot cycle's sync phase. Jobs that
request a specific concurrency via job.data.concurrency still win.

noEmbed default stays at true — embed is a separate job (submit
gbrain embed --stale, OR rely on the autopilot cycle's embed phase).
The doc comment makes that contract explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:43 -07:00
Garry TanandClaude Opus 4.7 8fcd2737bf fix: import.ts — engine.kind discriminator, worker try/finally, parseWorkers
A1: replace the config?.engine === 'pglite' string sniff with
engine.kind === 'pglite' to match sync.ts and the v0.13.1 contract.

A2: wrap worker engine creation + the parallel loop in try/finally so
disconnects always fire — same pattern as sync.ts. Worker engines now
push onto an array as they connect (rather than Promise.all) so the
finally block can clean up partial-connect state.

Q2: route --workers parsing through the shared parseWorkers() helper.
parseInt-with-no-validation is gone — '0', '-3', 'foo', '1.5' now exit
with a clear error message instead of silently falling through.

Q3: drop the config!.database_url! non-null assertion; fall back to
serial when database_url is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:30 -07:00
Garry TanandClaude Opus 4.7 b23f24f91b fix: harden performSync — writer lock, head-drift gate, engine.kind
CODEX-2: wrap performSync body in a gbrain-sync DB lock so two concurrent
syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot
both read last_commit, both write it unconditionally, and let the last
writer win. cycle.ts continues to hold gbrain-cycle for its broader scope;
the two ids nest cleanly.

CODEX-3: capture git HEAD at sync entry, re-rev-parse after the import
phase, refuse to advance last_commit if HEAD drifted (someone ran
git checkout / git pull mid-sync). Vanished files now go into failedFiles
instead of silent-skip — same gating mechanism, no more bookmark advance
past unimported work.

A1: replace both PGLite detection sites with engine.kind === 'pglite'.
The constructor.name sniff is gone (breaks under bundling) and so is the
inconsistent config?.engine string check.

A2: connect worker engines serially into an array, run inside try/finally
so disconnect always fires — even on partial connect failure, OOM, or
mid-import abort. Prior Promise.all(...disconnect) leaked the 8 worker
connections on any panic path.

Q1: explicit --workers / opts.concurrency now bypasses the >50-file floor.
User opt-in beats the auto-path safety net.

Q3: drop the config!.database_url! non-null assertions; fall back to serial
when database_url is unset instead of crashing on TypeError.

Q4: worker-count banner moves from console.log to console.error so stdout
stays clean for --json output.

test/sync-parallel.test.ts — 7 cases over PGLite covering the bookmark
gate under concurrency request, the head-drift gate, vanished-file
failure capture, PGLite-stays-serial, and the writer-lock contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:21 -07:00
Garry TanandClaude Opus 4.7 10d96545a4 feat: shared concurrency policy + db-lock primitive
src/core/sync-concurrency.ts — single source of truth for autoConcurrency()
+ parseWorkers() + shouldRunParallel() + constants. Replaces three drifted
call-site policies (performSync, performFullSync, jobs handler).

src/core/db-lock.ts — generic tryAcquireDbLock(engine, lockId, ttlMinutes)
over the existing gbrain_cycle_locks table. Parameterized lock id so
performSync (gbrain-sync) can nest cleanly under cycle.ts (gbrain-cycle)
without deadlock.

test/sync-concurrency.test.ts — 17 cases covering PGLite-forces-serial,
explicit override clamping, auto-path threshold, parseWorkers validation
(rejects 0, negatives, NaN, decimals, trailing chars).

No consumers yet; subsequent commits wire sync.ts, import.ts, and jobs.ts
to use these helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:02 -07:00
root 6b2f3bc321 feat: parallel sync — bounded concurrent imports (#489)
gbrain sync --concurrency N (alias --workers N) parallelizes the import
phase using per-worker Postgres engine instances with an atomic queue
index (same proven pattern as gbrain import --workers N).

Auto-concurrency: when a sync touches >100 files and the user didn't
explicitly set --concurrency, defaults to 4 workers. Small incremental
syncs (<50 files) stay serial. Full syncs auto-detect Postgres and
default to 4 workers.

Minion sync handler defaults to concurrency=4, configurable via job
params: {"concurrency": 8}.

Delete and rename phases remain serial (order-dependent, fast).
PGLite falls back to serial automatically (single-connection engine).

Changes:
- src/commands/sync.ts: SyncOpts.concurrency, parallel import loop in
  performSync incremental path, --workers passthrough in performFullSync
- src/commands/jobs.ts: sync handler accepts concurrency param (default 4)
- CHANGELOG.md: v0.23.0 parallel sync entry

All 37 existing sync tests pass. Typecheck clean.
2026-04-28 06:43:12 +00:00
2002 changed files with 8555 additions and 415947 deletions
+4 -12
View File
@@ -34,7 +34,7 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
bun-version: latest
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
@@ -44,10 +44,7 @@ jobs:
tier2:
name: Tier 2 (LLM Skills)
runs-on: ubuntu-latest
# Runs on every push/PR now (promoted from schedule-only in v0.19.0).
# Tier 1 must pass first; Tier 2 uses OPENAI_API_KEY + ANTHROPIC_API_KEY
# from repo/org secrets. Nightly + manual triggers still supported via
# the workflow-level `on:` list.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: tier1
services:
postgres:
@@ -67,7 +64,7 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
bun-version: latest
- run: bun install
- name: Install OpenClaw
run: npm install -g openclaw@2026.4.9
@@ -88,13 +85,8 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test test/e2e/skills.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
# v0.33.3.0: ZE live API tests skip gracefully when this is unset,
# so forks without the secret stay green. The test exercises the
# zeroEntropyCompatFetch response-rewriter + URL rewrite + flexible
# dim handling + gateway.rerank against the real provider.
ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}
-89
View File
@@ -1,89 +0,0 @@
name: Heavy Tests
# Heavy ops-shape tests under tests/heavy/. Cost minutes per run; NOT part
# of default PR CI. Two triggers:
# - Nightly schedule (catches regressions within 24h of merge to master).
# - On-demand opt-in via PR label `heavy-tests` (slow loop kept off by default).
# - Manual workflow_dispatch for triage.
#
# See CLAUDE.md "tests/heavy/*.sh" entry and tests/heavy/README.md.
on:
schedule:
- cron: '17 8 * * *' # 08:17 UTC daily — staggered to avoid noisy slots
pull_request:
# `synchronize` + `reopened` fire on subsequent pushes / reopens — without
# them, a PR labeled `heavy-tests` would NEVER re-run heavy on later
# commits. The job-level `if:` below filters to PRs that still carry the
# label so we don't fan out on unrelated label changes.
types: [labeled, synchronize, reopened]
workflow_dispatch:
permissions:
contents: read
# When a PR gets the heavy-tests label, cancel any in-flight heavy-tests run on
# the same ref so we only ever measure the latest commit.
concurrency:
group: heavy-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
heavy:
name: Heavy tests
# On pull_request: only run when the PR currently carries the `heavy-tests`
# label. Works for all three trigger types (labeled, synchronize, reopened)
# because `contains(labels.*.name, ...)` reads the live label set, not the
# event payload's `label.name` (which is only populated for `labeled`).
if: |
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
runs-on: ubuntu-latest
timeout-minutes: 30
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- name: Run heavy tests
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun run test:heavy
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
# run. Upload those + the rss workload JSON on failure for triage
# without re-running locally.
#
# actions/upload-artifact runs as a node action — `~` is NOT expanded by
# the shell here. Stage logs into the workspace first, then upload from
# the stable workspace-relative path.
- name: Stage heavy-test logs into workspace
if: always()
run: |
mkdir -p heavy-artifacts
cp -r "$HOME/.gbrain/audit"/heavy-* heavy-artifacts/ 2>/dev/null || true
cp tests/heavy/rss-baseline.json heavy-artifacts/ 2>/dev/null || true
- name: Upload heavy-test artifacts
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: heavy-tests-${{ github.run_id }}-${{ github.run_attempt }}
path: heavy-artifacts/
retention-days: 14
if-no-files-found: ignore
+1 -1
View File
@@ -23,7 +23,7 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
bun-version: latest
- run: bun install
- run: bun test
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
+10 -236
View File
@@ -5,72 +5,12 @@ on:
branches: [master]
pull_request:
branches: [master]
# Manual dispatch lets a local dev/agent offload the suite to GitHub's
# on-demand runners from ANY branch (see scripts/ship-remote-tests.sh).
# Frees a load-saturated local machine (e.g. many Conductor agents running
# their own bun-test suites at once — load avg 120 on 16 cores).
workflow_dispatch:
permissions:
contents: read
jobs:
# ──────────────────────────────────────────────────────────────────────
# cache-check: runs first, computes the content hash of every tracked
# file EXCEPT the deny-list (CHANGELOG.md, README.md, docs/**/*.md, etc.
# — see scripts/ci-cache-hash.sh for the full list). Looks up
# `ci-pass-<hash>` in actions/cache; if hit, the test matrix + verify
# + serial jobs all skip and test-status reports green immediately.
# If miss, the full suite runs and cache-write seals it on success.
#
# Hit rate covers re-pushes (same SHA twice), branch rebases that
# don't touch tracked code, and any branch update that touches only
# the deny-listed doc files.
# ──────────────────────────────────────────────────────────────────────
cache-check:
runs-on: ubuntu-latest
outputs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Compute content hash
id: compute
run: |
# --verbose writes the "X/Y files in hash" diagnostic to stderr;
# stdout carries the 16-char hash. Capture both.
HASH=$(bash scripts/ci-cache-hash.sh --verbose 2>/tmp/cache-diag)
cat /tmp/cache-diag
echo "Computed cache hash: $HASH"
echo "hash=$HASH" >> "$GITHUB_OUTPUT"
- name: Lookup actions/cache for ci-pass-<hash>
id: lookup
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: ci-pass-${{ steps.compute.outputs.hash }}
path: .ci-cache-marker
# `lookup-only: true` means we only probe whether the cache
# entry exists — we don't download it (the marker contents
# don't matter, only the key match does). `cache-hit` returns
# true only on EXACT key match (per actions/cache docs); a
# restore-keys prefix fallback would set cache-hit=false, so
# it's deliberately omitted here. Cross-branch scoping works
# naturally: PR branches can read default-branch (master)
# cache entries via exact key match when the content hash
# matches, which happens whenever the tree is doc-only
# different from a green master run.
lookup-only: true
- name: Cache status
run: |
if [ "${{ steps.lookup.outputs.cache-hit }}" = "true" ]; then
echo "✓ cache HIT for hash ${{ steps.compute.outputs.hash }} — test jobs will skip"
else
echo "✗ cache MISS for hash ${{ steps.compute.outputs.hash }} — full suite will run"
fi
gitleaks:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@@ -80,189 +20,23 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
verify:
# Pre-test gates: privacy/jsonb/source-id/etc + typecheck + admin-build.
# Lives in its own runner so the matrix shards aren't carrying ~2-3min
# of verify work in addition to their test files (the old shape stuffed
# this into `test (1)` via `if: matrix.shard == 1`, which made shard 1
# the slowest matrix worker). scripts/run-verify-parallel.sh fans out
# the 20 checks via & + wait (~5s vs ~15-25s sequential).
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
# the matrix shards aren't carrying the serial-pass tail (the old shape
# stuffed this into `test (1)` after the matrix work, which compounded
# shard 1's overload).
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run test:serial
slow-eval-longmemeval:
# Dedicated runner for the LongMemEval end-to-end test file. The file
# was originally 359s. TODO #1 (engine-sharing in runEvalLongMemEval
# via RunOpts.engine) cut it to ~200s by amortizing PGLite cold-create
# across all 13 runEvalLongMemEval calls in one beforeAll-shared brain.
# Pulled out of the matrix (see scripts/test-shard.sh) so a single 200s
# atom doesn't dominate a shard's wallclock. Companion file
# test/eval-longmemeval.slow.test.ts (the pure-bucket half) stays in
# the matrix because it's light (~42s).
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
slow-entity-resolve-perf:
# Dedicated runner for the entity-resolve perf test (~159s, single perf
# describe with one test that builds 5000+ pages and asserts the NEW
# tryPrefixExpansion shape is 5x faster than the OLD shape — not
# subdivisible without weakening the perf guarantee). Pulled out of the
# matrix (see scripts/test-shard.sh) so a single 159s atom doesn't
# dominate a shard's wallclock. Runs in parallel with the matrix.
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
test:
# Pure matrix shard — no verify, no serial. Each shard runs its slice
# of the unit test set under one `bun test` invocation.
#
# 10 shards (was 6) drops per-shard total from 532s → 287s. With the two
# dedicated jobs (slow-eval-longmemeval, slow-entity-resolve-perf) also
# pulled out, the matrix is bounded by ~287s ≈ 4.8 min. Total CI ≈ max
# of matrix + slow-eval (~3.3 min after engine-sharing in TODO #1) +
# slow-entity-resolve-perf (~2.6 min) ≈ 4.8 min.
#
# Concurrency budget: 10 shards + verify + serial + slow-eval +
# slow-entity-resolve-perf + gitleaks + cache-check + cache-write +
# test-status = ~18 jobs × 2 concurrent PRs = 36. GitHub free-tier
# caps at ~20 concurrent jobs, so multi-PR days will see some queue
# pressure. Single-PR runs are unaffected.
#
# Partition policy is weight-aware LPT bin-packing via scripts/sharding.ts
# (replaces FNV-1a path hash). Weights live in scripts/test-weights.json,
# mined from real CI logs via scripts/mine-shard-weights.ts. Missing
# weights fall back to corpus median — new test files work immediately.
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
# ubuntu-latest is free 2-core/7GB. Larger runners (16-cores, etc.) require
# a provisioned runner pool in repo settings. Falling back to default keeps
# the matrix shard speedup (~5-6x via parallelism) at zero cost.
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
bun-version: latest
- run: bun install
- name: Run test shard ${{ matrix.shard }}/10
run: scripts/test-shard.sh ${{ matrix.shard }} 10
# ──────────────────────────────────────────────────────────────────────
# cache-write: ONLY runs when every gated job succeeded. Writes the
# cache entry under `ci-pass-<hash>` so future runs at the same hash
# hit cache. Codex's load-bearing correctness point: writing the
# cache before the matrix completes would permanently bless bad states
# (a future run at the same hash would skip tests because of a cache
# entry written when tests hadn't actually passed).
# ──────────────────────────────────────────────────────────────────────
cache-write:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: success() && needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
steps:
- name: Create cache marker
run: |
mkdir -p .ci-cache-marker
echo "${{ needs.cache-check.outputs.hash }}" > .ci-cache-marker/hash
echo "$GITHUB_SHA" > .ci-cache-marker/sha
echo "$GITHUB_REF" > .ci-cache-marker/ref
- uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: ci-pass-${{ needs.cache-check.outputs.hash }}
path: .ci-cache-marker
# ──────────────────────────────────────────────────────────────────────
# test-status: the single user-visible "did CI pass?" check.
# Runs always (if: always()), succeeds when EITHER cache-check.hit==true
# OR all gated jobs (gitleaks, verify, serial-tests, test) succeeded.
# Branch protection (when configured) gates on this single job name.
# ──────────────────────────────────────────────────────────────────────
test-status:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: always()
runs-on: ubuntu-latest
steps:
- name: Aggregate result
run: |
HIT="${{ needs.cache-check.outputs.hit }}"
GITLEAKS="${{ needs.gitleaks.result }}"
VERIFY="${{ needs.verify.result }}"
SERIAL="${{ needs.serial-tests.result }}"
SLOW_EVAL="${{ needs.slow-eval-longmemeval.result }}"
SLOW_PERF="${{ needs.slow-entity-resolve-perf.result }}"
TEST="${{ needs.test.result }}"
echo "cache-check.hit=$HIT"
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF test=$TEST"
if [ "$HIT" = "true" ]; then
echo "✓ cache HIT for hash ${{ needs.cache-check.outputs.hash }} — CI green"
exit 0
fi
# Cache miss: every gated job must have succeeded.
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$TEST"; do
if [ "$r" != "success" ]; then
echo "✗ gated job did not succeed (got $r) — CI fail"
exit 1
fi
done
echo "✓ all gated jobs succeeded — CI green"
- name: Pre-test gates (shard 1 only — they're not test files)
if: matrix.shard == 1
run: scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck
- name: Run test shard ${{ matrix.shard }}/4
run: scripts/test-shard.sh ${{ matrix.shard }} 4
-23
View File
@@ -11,33 +11,10 @@ bin/
.gstack/
supabase/.temp/
.claude/skills/
# admin/dist/ is the React SPA bundle. CLAUDE.md says it's committed for
# self-contained binaries (the bun --compile path embeds it via
# `import path from 'admin/dist/index.html' with { type: 'file' }`).
# Build via: cd admin && bun install && bun run build.
admin/node_modules/
.idea
eval/reports/
eval/data/world-v1/world.html
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
eval/data/amara-life-v1/_cache/
# claw-test E2E build cache (shim + scratch outputs)
test/.cache/
.claude/
export/
# Conductor workspace-local agent artifacts: plans, todos, run-unit-parallel
# failure logs and per-shard test output. v0.26.4 (run-unit-parallel.sh)
# writes .context/test-failures.log + .context/test-summary.txt +
# .context/test-shards/. Workspace-local by design — never committed.
.context/
# Tier 3 PGLite snapshot fixture (built on demand by build:pglite-snapshot)
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
# Private brain reports — never check these in (per CLAUDE.md privacy rule)
reports/network-intelligence/
+9 -72
View File
@@ -6,26 +6,10 @@ start here.
## Install (5 min)
1. Install gbrain via Bun (the canonical path):
```bash
curl -fsSL https://bun.sh/install | bash
export PATH="$HOME/.bun/bin:$PATH"
bun install -g github:garrytan/gbrain
```
If `bun install -g` aborts or `gbrain doctor` reports `schema_version: 0`,
the CLI prints a recovery hint pointing at [#218](https://github.com/garrytan/gbrain/issues/218).
Run `gbrain apply-migrations --yes` to recover, or fall back to the
deterministic install: `git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain && bun install && bun link`.
2. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
1. Clone: `git clone https://github.com/garrytan/gbrain ~/gbrain && cd ~/gbrain`
2. Install: `bun install`
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
multi-machine sync, init suggests Postgres + pgvector via Supabase.
3. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
default but printed a 9-cell cost matrix (mode × downstream model)
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
and confirm their choice before continuing. Cost spread between corners
is 25x — silent acceptance is the wrong default. See
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
for existing users (search modes were added in v0.32.3).
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
(API keys, identity, cron, verification).
@@ -34,13 +18,7 @@ start here.
1. `./AGENTS.md` (this file) — install + operating protocol.
2. [`./CLAUDE.md`](./CLAUDE.md) — architecture reference, key files, trust boundaries,
test layout.
3. [`./docs/architecture/brains-and-sources.md`](./docs/architecture/brains-and-sources.md)
— the two-axis mental model (brain = which DB, source = which repo in the DB). Every
query routes on both axes. Read before writing anything that touches brain ops.
4. [`./skills/conventions/brain-routing.md`](./skills/conventions/brain-routing.md) —
agent-facing decision table: when to switch brain, when to switch source, how
cross-brain federation works (latent-space only; the agent decides).
5. [`./skills/RESOLVER.md`](./skills/RESOLVER.md) — skill dispatcher. Read before any task.
3. [`./skills/RESOLVER.md`](./skills/RESOLVER.md) — skill dispatcher. Read before any task.
## Trust boundary (critical)
@@ -57,58 +35,17 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
[`docs/mcp/DEPLOY.md`](./docs/mcp/DEPLOY.md).
- **Debug:** [`docs/GBRAIN_VERIFY.md`](./docs/GBRAIN_VERIFY.md),
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
- **Migrate / upgrade:** `gbrain upgrade` (binary self-update + schema migrations + post-upgrade prompts),
[`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations --yes` (manual schema-only).
- **Eval retrieval changes:** capture is off by default. To benchmark a
retrieval change against real captured queries, set
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
and `gbrain eval replay --against base.ndjson`. For public benchmark
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
per question — your `~/.gbrain` is never opened. Full guide:
[`docs/eval-bench.md`](./docs/eval-bench.md).
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
loop. `gbrain doctor --remediation-plan --json` previews what would be
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
walks a dependency-ordered plan (sync before extract, embed after
consolidate), re-checking score between every step, refusing to spend
past the cost cap. Empty brains (no entity pages) or unconfigured embedding
keys hit a `max_reachable_score` ceiling and bail with what's missing.
Three phase handlers (synthesize / patterns / consolidate) are
PROTECTED — only trusted local callers can submit them; MCP cannot.
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
and the CHANGELOG entry for v0.36.4.0.
- **Track a founder/company over time (v0.35.7):** when an entity has
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
`unit: USD`, `period: monthly` columns), run
`gbrain eval trajectory <entity-slug>` for the chronological history
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
for a four-signal JSON rollup (claim_accuracy / consistency /
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:**
`gbrain think` now uses this substrate automatically on temporal /
knowledge_update intent (default ON; flip `think.trajectory_enabled=false`
to opt out). Migration v82 added `facts.event_type` so non-metric event
rows (`meeting`, `job_change`, `location_change`) ride through the same
pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query
them.
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
single-fetch ingestion.
## Before shipping
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
Manual path: `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin
up the test Postgres container, run `bun run test:e2e`, tear it down).
Ship via the `/ship` skill, not by hand.
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
not by hand.
## Privacy
+20 -16490
View File
File diff suppressed because it is too large Load Diff
+81 -1228
View File
File diff suppressed because one or more lines are too long
+3 -191
View File
@@ -52,22 +52,10 @@ docs/ Architecture docs
## Running tests
```bash
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
bun run test # parallel 8-shard fan-out + serial post-pass
bun test # all tests (unit + E2E skipped without DB)
bun test test/markdown.test.ts # specific unit test
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck
# Pre-merge sanity (everything CI runs)
bun run test:full # verify + parallel unit + slow + smart e2e
# Slow / serial / e2e in isolation
bun run test:slow # *.slow.test.ts only (cold-path correctness)
bun run test:serial # *.serial.test.ts only (--max-concurrency=1)
bun run test:e2e # real-Postgres E2E (requires DATABASE_URL)
# E2E setup (Postgres with pgvector)
# E2E tests (requires Postgres with pgvector)
docker compose -f docker-compose.test.yml up -d
DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run test:e2e
@@ -75,94 +63,6 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
DATABASE_URL=postgresql://... bun run test:e2e
```
Use `bun run verify` before pushing. The guard chain catches: banned fork-name
leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
(`scripts/check-progress-to-stdout.sh`), test-isolation rule violations
(`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel
loop" below), silent fallback to recursive chunking in the compiled binary
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
(`scripts/check-admin-build.sh`), and resolver drift on bundled skills
(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any
warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift
before merge). `bun run check:all` runs the full historical sweep including the
trailing-newline and exports-count checks.
### Writing tests that survive the parallel loop
`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the
same shard share a process, so process-global state leaks between them. Four
lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
| Rule | What it bans | Fix |
|---|---|---|
| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` |
| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` |
| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) |
| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` |
Canonical PGLite block (R3 + R4 compliant — paste this verbatim):
```ts
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => { await engine.disconnect(); });
beforeEach(async () => { await resetPgliteState(engine); });
```
Env-touching tests:
```ts
import { withEnv } from './helpers/with-env.ts';
test('reads OPENAI_API_KEY', async () => {
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
expect(loadConfig().openai_key).toBe('sk-test');
});
});
```
`withEnv` saves and restores keys via try/finally including when the callback
throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is
process-global). Files using `withEnv` stay outside the future
`test.concurrent()` codemod's eligibility filter.
When to quarantine instead of fix: rename to `*.serial.test.ts` if the file
uses `mock.module(...)`, is genuinely env-coupled (module-load env readers +
ESM caching defeat dynamic-import-after-env tricks), or intentionally shares
state across `it()` boundaries. Quarantine count cap: 10 (informational).
Files that violated these rules at the v0.26.7 baseline are listed in
`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over
time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep
+ codemod) remove entries as files get fixed.
### Local CI gate (recommended before pushing, v0.23.1+)
```bash
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
bun run ci:local:diff # gate with diff-aware E2E selector
bun run ci:select-e2e # print which E2E files the selector would run
```
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
narrower mappings via `scripts/e2e-test-map.ts`.
## Building
```bash
@@ -193,95 +93,7 @@ See `docs/ENGINES.md` for the full guide. In short:
3. Run the test suite against your engine
4. Document in `docs/`
The original SQLite engine plan was superseded by PGLite (embedded Postgres 17 via WASM), which uses the same SQL dialect as Postgres and eliminates the need for a separate FTS5/sqlite-vss translation layer. See [`docs/ENGINES.md`](docs/ENGINES.md) for the engine architecture and the rationale.
## CONTRIBUTOR_MODE — turn on the dev loop
gbrain captures retrieval traffic so you can replay real queries against
your code changes before merging. **This is off by default** (production
users get a quiet brain, no surprise data accumulation). Contributors turn
it on with one shell rc line:
```bash
# In ~/.zshrc or ~/.bashrc:
export GBRAIN_CONTRIBUTOR_MODE=1
```
That's it. Every `query` / `search` you (or agents pointed at your dev
brain) run from that shell now writes a row to `eval_candidates`, and the
[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code)
has data to work against.
What CONTRIBUTOR_MODE actually does:
- Turns on `query`/`search` capture into the local `eval_candidates` table.
Without it the gate is closed and capture is a no-op.
- That's all. PII scrubbing, retention, and replay are independent.
Resolution order (most explicit wins):
1. `eval.capture: true` in `~/.gbrain/config.json` → on
2. `eval.capture: false` in `~/.gbrain/config.json` → off
3. `GBRAIN_CONTRIBUTOR_MODE=1` → on
4. otherwise → off
Quick check that capture is actually running:
```bash
gbrain query "anything" >/dev/null
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
# (or `gbrain doctor` — surfaces silent capture failures cross-process)
```
To disable capture even with the env var set, write
`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config
beats the env var both directions.
## Running real-world eval benchmarks (touching retrieval code)
If your PR touches retrieval — search ranking, RRF fusion, embeddings,
intent classification, query expansion, source boost, or the `query` /
`search` op handlers — run `gbrain eval replay` against a snapshot of
real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you
have captured rows to replay against.
Quick loop:
```bash
gbrain eval export --since 7d > baseline.ndjson # snapshot before your change
# ... make your change ...
gbrain eval replay --against baseline.ndjson # diff retrieval, get Jaccard@k
```
Three numbers come back: mean Jaccard@k between captured and current slug
sets, top-1 stability, and mean latency Δ. The replay tool flags the worst
regressions so you can eyeball whether the change is hurting real queries.
Trigger paths (rerun if your diff touches any of these):
- `src/core/search/hybrid.ts`
- `src/core/search/source-boost.ts`, `sql-ranking.ts`
- `src/core/search/intent.ts`, `expansion.ts`, `dedup.ts`
- `src/core/embedding.ts`
- `src/core/operations.ts` (query / search handlers)
- `src/core/postgres-engine.ts` / `pglite-engine.ts` (searchKeyword /
searchVector SQL)
See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide
including CI integration, hand-crafted NDJSON corpora (so a fresh checkout
without captured data can still replay), and cost considerations. The
NDJSON wire format is documented in
[`docs/eval-capture.md`](./docs/eval-capture.md).
For public benchmark coverage on top of replay, `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
retrieval. One in-memory PGLite per question, runtime-enumerated
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
published `evaluate_qa.py`. Use it alongside replay when changes affect
retrieval quality on long-context conversational data — replay catches
regressions on YOUR queries, LongMemEval catches them on a public set the
benchmark community already cites. See the "Public benchmarks: LongMemEval"
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
## Welcome PRs
-148
View File
@@ -1,148 +0,0 @@
# DESIGN.md
The design system source of truth for gbrain. Born from the de facto tokens
that landed in `admin/src/index.css` during the v0.26.0 admin SPA work and
formalized during the v0.36.1.0 Hindsight calibration wave's design review.
This doc is the calibration target for `/plan-design-review` and `/design-review`.
When a question is "does this UI fit the system?", the answer is here.
## Voice
GBrain talks like a smart friend who knows your past, not a clinical scoring
system. Every user-facing string passes through this filter:
- Second person, contractions allowed.
- Grounded in concrete data the user can verify ("2 of 3 missed" beats
"Brier 0.31").
- Never preachy. Never "we recommend." Never "according to your data."
- Short. Under 25 words for narrative; under one line for status.
- Numbers grounded in real outcomes, never abstract metrics without
translation.
Five surfaces use this voice (v0.36.1.0+):
`pattern_statement`, `nudge`, `forecast_blurb`, `dashboard_caption`,
`morning_pulse`. All five pass through `gateVoice()` in
`src/core/calibration/voice-gate.ts` with mode-specific rubrics. A Haiku
judge rejects academic-sounding candidates; up to 2 regens; then fall
back to a hand-written template from `src/core/calibration/templates.ts`.
## Color tokens
CSS variables in `admin/src/index.css`. SVG renderer inlines literals
matching these tokens (`src/core/calibration/svg-renderer.ts`).
| Token | Value | Use |
|--------------------|-----------|-------------------------------------------|
| `--bg-primary` | `#0a0a0f` | Page background |
| `--bg-secondary` | `#14141f` | Sidebar, cards |
| `--bg-tertiary` | `#1e1e2e` | Subtle surfaces, borders |
| `--text-primary` | `#e0e0e0` | Body text |
| `--text-secondary` | `#888` | Headings, labels |
| `--text-muted` | `#777` | Tertiary text — TD2 bumped from #555 for WCAG AA contrast (~5.5:1) |
| `--accent` | `#3b82f6` | Active states, links, primary CTAs |
| `--success` | `#22c55e` | Healthy / ok status |
| `--warning` | `#f59e0b` | Doctor warnings |
| `--error` | `#ef4444` | Failures, destructive confirmations |
Dark theme is the only theme. No light mode toggle planned — admin is an
operator tool, not a marketing surface. Users live in the terminal with a
dark theme already.
WCAG contrast:
- Body text (#e0e0e0 on #0a0a0f) → ~14:1, AAA
- Muted text (#777 on #0a0a0f) → ~5.5:1, AA (was 4.0 / fail before TD2)
- Accent links (#3b82f6 on #0a0a0f) → ~5.7:1, AA
## Typography
| Variable | Value | Use |
|--------------------|-----------------------------|---------------------------------|
| `--font-sans` | `Inter, system-ui, sans-serif` | UI text, headings, body |
| `--font-mono` | `JetBrains Mono, monospace` | Numbers, slugs, code, terminal-ish data |
Type scale (de facto, not formalized yet):
- 18px: sidebar logo / page title
- 14px: body
- 13px: nav items
- 12px: chart captions, secondary labels
- 11px: tertiary labels in dense charts
Numbers in tables and metrics use JetBrains Mono so column alignment is
mechanical. Avoid mixing Inter and JetBrains Mono in the same line.
## Spacing scale
4 / 8 / 16 / 24 / 32px. Linear-app-style density: 24-32px between major
sections, 16px between row groups, 8px within a row. The Calibration tab
(approved variant-B mockup) is the canonical example.
## Layout
- Sidebar 200px on the left. Active item gets a 3px left-border in `--accent`.
- Main content area uses the remaining width.
- Max content width: 720px for text-heavy pages (Calibration), 960px for
data tables (Request Log).
- No 3-column feature grids. No icons in colored circles. No decorative blobs.
- Cards earn their existence — heading + content works without a card frame
in most cases.
## Charts
Server-rendered SVG via `src/core/calibration/svg-renderer.ts`. Pure
functions: data → SVG string. No DOM, no React component, no chart library.
XSS posture: server-side `escapeXml()` on every caller-controlled string.
Numeric inputs `.toFixed()`-coerced. Admin SPA renders via
`<TrustedSVG>` wrapper with `dangerouslySetInnerHTML`. Endpoint gated by
`requireAdmin` middleware.
Why server-rendered SVG (per D23):
- Chart logic stays close to the data math.
- Zero new client-side chart-library dep.
- SVG is accessible (text labels), scalable, copy-paste-friendly to PR
descriptions and docs.
- Sets the precedent for future admin charts (contradictions trend, takes
scorecard, etc.).
Four chart renderers in v0.36.1.0:
- `renderBrierTrend({ series })` — sparkline + baseline reference at 0.25
- `renderDomainBars({ bars })` — horizontal accuracy bars
- `renderAbandonedThreadsCard(threads)` — text rows + "revisit now" links
- `renderPatternStatementsCard(statements)` — clickable drill-down anchors
## Interaction patterns
- Keyboard navigation is REQUIRED for all CLI interaction surfaces. The
propose-queue review uses J/K/space/u/q shortcuts (gmail-style).
- Loading states: "Loading...". Don't show spinners on sub-200ms operations.
- Empty states ARE features: warmth + primary action + context. Cold-brain
Calibration page tells the user EXACTLY how to build a profile, not
"no data available."
- Error states: name what failed + name the next step. Never "an error
occurred — please try again."
## What's NOT here yet (v0.37+ roadmap)
- Type scale formalization (current values are de facto, not enforced)
- Animation tokens (admin SPA has zero animations on purpose; v0.37 may
add subtle progress / loading transitions)
- Print stylesheet
- Light mode (NOT planned — see "Dark theme is the only theme" above)
- Component library extraction (the React components live inline in admin/src/pages/;
no `<Button>` / `<Card>` abstraction layer yet)
## How to use this document
When adding a new UI surface to gbrain:
1. Pick existing tokens before introducing new ones. New tokens go through
`/plan-design-review`.
2. Match the voice rules. Run candidates through `gateVoice()` before
shipping any user-facing string in the calibration surfaces.
3. Match the spacing scale and density. Linear-calm-clarity over
dashboard-card-mosaic.
4. Match the typography: Inter for UI, JetBrains Mono for numbers.
When updating this document: it's a living target, not a frozen spec.
Major changes go through `/plan-design-review` to keep the system coherent.
+19 -180
View File
@@ -16,43 +16,32 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain
curl -fsSL https://bun.sh/install | bash
export PATH="$HOME/.bun/bin:$PATH"
bun install -g github:garrytan/gbrain
bun install && bun link
```
Verify: `gbrain --version` should print a version number. If `gbrain` is not found,
restart the shell or add the PATH export to the shell profile.
> **If `bun install -g` aborts or `gbrain doctor` reports `schema_version: 0`** (Bun
> occasionally blocks the top-level postinstall hook on global installs, so schema
> migrations don't run automatically), the CLI prints a recovery hint pointing at
> [#218](https://github.com/garrytan/gbrain/issues/218). Run `gbrain apply-migrations --yes`
> to recover. If that doesn't work, fall back to the deterministic install path:
>
> ```bash
> git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain
> bun install && bun link
> ```
> **Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
> postinstall hook on global installs, so schema migrations never run and the CLI
> aborts with `Aborted()` when it opens PGLite. Use the `git clone + bun link` path
> above. Tracking issue: [#218](https://github.com/garrytan/gbrain/issues/218).
## Step 2: API Keys
Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack
(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config
set embedding_model <provider:model>`.
Ask the user for these:
```bash
export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+)
export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion
export OPENAI_API_KEY=sk-... # required for vector search
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality
```
Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically
or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider,
keyword search still works. Without Anthropic, search works but skips query expansion.
Save to shell profile or `.env`. Without OpenAI, keyword search still works.
Without Anthropic, search works but skips query expansion.
## Step 3: Create the Brain
@@ -72,63 +61,6 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
NOT inside ~/gbrain.
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
tier is Haiku-class or no OpenAI key is configured). The init output included the
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
default. Stop and ask the operator.
**Present this matrix verbatim:**
```
Per-query cost @ 10K queries/mo (typical single-user volume):
Haiku 4.5 Sonnet 4.6 Opus 4.7
($1/M) ($3/M) ($5/M)
conservative $40/mo $120/mo $200/mo
balanced $100/mo $300/mo $500/mo
tokenmax $200/mo $600/mo $1,000/mo
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
```
**Ask the operator (paraphrase if needed):**
> Your gbrain just installed with search mode `<auto-applied default>`. This is
> a one-time setup decision that controls retrieval payload size. Which mode
> do you want?
>
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
>
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
>
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
>
> Cost depends on BOTH the mode AND the downstream model you run. See the
> matrix above for the 9-cell breakdown.
If the operator picks a non-default mode, run:
```bash
gbrain config set search.mode <mode>
```
If they pick tokenmax AND want to preserve the literal v0.31.x default
(limit=20 instead of tokenmax's 50), also run:
```bash
gbrain config set search.searchLimit 20
```
Verify the choice with `gbrain search modes` before continuing.
**Why this matters:** the cost spread between corners of the matrix is 25x.
An agent that silently accepts the default and starts running queries against
a user who didn't expect tokenmax-class context loads can rack up surprise
spend. Confirm before continuing.
## Step 4: Import and Index
```bash
@@ -163,24 +95,8 @@ and supports `--since YYYY-MM-DD` for incremental runs.
## Step 5: Load Skills
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
scaffold the bundled skills into it:
```bash
cd /path/to/agent/workspace
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
```
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
diff against gbrain's bundle when you want upstream improvements. (The legacy
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
bundled copy at `~/gbrain/skills/RESOLVER.md` when running from the cloned repo). It's
the skill dispatcher — tells you which skill to read for any task. Save this to your
memory permanently.
Read `~/gbrain/skills/RESOLVER.md`. This is the skill dispatcher. It tells you which
skill to read for any task. Save this to your memory permanently.
The three most important skills to adopt immediately:
@@ -208,17 +124,13 @@ If skipped, minimal defaults are installed automatically.
## Step 7: Recurring Jobs
Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), or skip the
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
— or `gbrain sync --watch` for a continuous loop.
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
synthesis and cross-session pattern detection. One cron-friendly command. This is what
makes the brain compound. Do not skip it. See `docs/guides/cron-schedule.md` for the
full protocol.
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
Entity sweep, citation fixes, memory consolidation. This is what makes the brain
compound. Do not skip it.
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
## Step 8: Integrations
@@ -236,18 +148,9 @@ actually works) is the most important.
## Upgrade
If you installed via `bun install -g`:
```bash
gbrain upgrade # self-updates the binary, runs schema migrations,
# and prints post-upgrade notes for the version range
```
If you installed via `git clone + bun link`:
```bash
cd ~/gbrain && git pull origin master && bun install
gbrain apply-migrations --yes # apply schema migrations (idempotent)
gbrain init # apply schema migrations (idempotent)
gbrain post-upgrade # show migration notes for the version range
```
@@ -255,15 +158,6 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
versions you skipped) and run any backfill or verification steps it lists. Skipping
this is how features ship in the binary but stay dormant in the user's brain.
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
**Do NOT silently move past the banner.** Present the matrix to the operator
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
same matrix and same default.
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
`gbrain extract links --source db && gbrain extract timeline --source db` to
backfill the new graph layer (see Step 4.5 above).
@@ -274,58 +168,3 @@ automatically during `gbrain post-upgrade` to fix the double-encoded JSONB
columns. PGLite brains no-op. If wiki-style imports were truncated by the old
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
`compiled_truth` from source markdown.
## v0.42.0+ onboard surface (NEW)
`gbrain onboard` is the activation surface gbrain did not have before.
Once your brain has any content, run `gbrain onboard --check --json` to
see structured recommendations across 5 brain-health axes (orphans,
stale embeddings, entity link coverage, timeline coverage, takes count).
**On first connect (after `gbrain init`):**
```bash
gbrain onboard --check --json
```
The JSON envelope (`schema_version: 1`) carries `recommendations[]` with
`apply_policy` per item: `auto_apply` (safe to run unattended),
`prompt_required` (needs explicit user consent), or `manual_only`
(LLM-bearing, user must run themselves).
**After every `gbrain upgrade`:**
```bash
gbrain onboard --check --json
```
New versions may surface new opportunities. The post-upgrade banner
nudges the user when it runs, but agents should re-probe as a hygiene
step regardless.
**Unattended remediation (cron / autopilot):**
```bash
gbrain onboard --auto --max-usd 5
```
Refuses without `--max-usd N`. Runs auto-eligible items only. The
autopilot daemon also consults onboard recommendations on its tick — no
explicit agent action needed for the autonomous path.
**Remote / federated brain installs (MCP):**
The `run_onboard` MCP op (admin scope) lets thin-client agents probe
brain health + drive remediation over OAuth-authenticated MCP. Protected
LLM-bearing handlers (synthesize, patterns, consolidate, takes-bootstrap,
contextual_reindex_per_chunk) require the additional `run_protected_onboard`
scope — admin alone is insufficient. The MCP op returns
`skipped_missing_scope[]` listing what would have run with the right
grants.
**Privacy + consent gates:**
- `gbrain takes extract --from-pages` sends concept/atom/lore/briefing/
writing/originals page content to your configured chat model (default
Anthropic Haiku). Refuses to run unless `takes.bootstrap_enabled=true`
is set in config AND `--yes` is passed. Two-gate opt-in by design.
- Autopilot's auto-apply tier for takes-bootstrap stays `manual_only`
until v0.42.1's eval gate (do not bypass).
**Suppress nudges in CI / scripted environments:**
```bash
export GBRAIN_NO_ONBOARD_NUDGE=1
```
Init + upgrade banners auto-skip in non-TTY too.
+629 -330
View File
File diff suppressed because it is too large Load Diff
+12 -104
View File
@@ -50,43 +50,6 @@ exclusively via `gbrain auth create/list/revoke`.
4. **Log all token issuance** — alert on unexpected registrations
5. **Rate-limit registration and token endpoints**
### Pre-registering claude.ai / ChatGPT clients without DCR (v0.41.3+)
The recommended hardening posture above is: ship `gbrain serve --http`
**without** `--enable-dcr` and pre-register every client manually. As of
v0.41.3, `gbrain auth register-client` accepts the OAuth fields
browser-based clients need:
```bash
# Pre-register claude.ai (confidential client; two redirect URIs)
gbrain auth register-client claude-ai \
--scopes "read write" \
--redirect-uri https://claude.ai/api/mcp/auth_callback \
--redirect-uri https://claude.com/api/mcp/auth_callback
# --grant-types is auto-set to authorization_code,refresh_token when
# --redirect-uri is passed; pass --grant-types explicitly to override.
# Pre-register ChatGPT (public PKCE client; no client_secret minted)
gbrain auth register-client chatgpt \
--scopes "read write" \
--redirect-uri https://chatgpt.com/connector/oauth/<HASH> \
--token-endpoint-auth-method none
```
Auth methods (`--token-endpoint-auth-method`):
- `client_secret_post` (default) — confidential client, secret in body
- `client_secret_basic` — confidential client, secret in `Authorization` header
- `none` — public PKCE-only client (no secret minted; ChatGPT custom
connector, Claude Code, Cursor)
The validator rejects unknown methods at the registration boundary, and
the same gate applies to the admin endpoint `POST /admin/api/register-client`
and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
`redirect_uris = []` and `token_endpoint_auth_method = NULL`, forcing
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
without `--enable-dcr`. That footgun is gone.
### Token Management
```bash
@@ -105,16 +68,6 @@ The built-in HTTP transport ships with several layers of hardening on by
default. All env vars below are optional; the defaults are intentionally
conservative.
### Bind address (v0.34: loopback by default)
`gbrain serve --http` listens on `127.0.0.1` by default. Personal-laptop
installs cannot accidentally publish the brain to the LAN. Self-hosted
deployments that need remote access pass `--bind 0.0.0.0` (all
interfaces) or `--bind <interface-ip>` (specific NIC). A stderr WARN
fires when `--public-url` is set without `--bind` so the operator sees
the binding before the first request — common cause of "ngrok forwards
to me but the agent can't reach the upstream" misconfigurations.
### Postgres-only
`gbrain serve --http` requires a Postgres engine. PGLite is local-only by
@@ -138,19 +91,6 @@ When the request `Origin` matches the allowlist, the server echoes it
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
CORS header is sent and the browser blocks the request.
**v0.41.3:** the same allowlist now gates every OAuth endpoint (`/mcp`,
`/token`, `/authorize`, `/register`, `/revoke`). Pre-v0.41.3 these used
default-wide-open `cors()` middleware, leaking
`Access-Control-Allow-Origin: *` on every response — any web origin could
complete a token exchange from a logged-in operator's browser. The CORS
preflight handler in the legacy bearer transport was also asymmetric
(actual-request path correctly default-deny, but OPTIONS preflight leaked
`Access-Control-Allow-Methods` + `Access-Control-Allow-Headers` to every
Origin); both are now consolidated through a single allowlist-gated path.
A startup stderr WARN fires when `--bind 0.0.0.0` is set without
`GBRAIN_HTTP_CORS_ORIGIN`, surfacing the default-deny posture before the
first request.
### Rate limiting
Two buckets, both stored in a bounded LRU map (default 10K keys, evicts
@@ -174,51 +114,30 @@ deployments.
### Reverse-proxy trust
**Loopback-only by default** (v0.41.3+ Express server agrees with the
legacy transport; pre-v0.41.3 the Express server hardcoded `'loopback'`
while docs claimed "disabled by default" — that disagreement is gone).
The default trusts only same-host proxies (127.0.0.1, ::1, fc00::/7);
external forwarded-for headers are ignored regardless. To widen or
narrow trust:
Disabled by default. To honor `X-Forwarded-For` (or `X-Real-IP`) when
gbrain runs behind a trusted reverse proxy:
```bash
# Trust exactly one hop — Fly.io, Render, Vercel, single-layer nginx
GBRAIN_HTTP_TRUST_PROXY=1 gbrain serve --http --port 8787
# Trust N hops — Cloudflare → nginx → gbrain
GBRAIN_HTTP_TRUST_PROXY=2 gbrain serve --http --port 8787
# Disable entirely — direct-exposure deployment with no proxy
GBRAIN_HTTP_TRUST_PROXY=0 gbrain serve --http --port 8787
# Named Express modes (uniquelocal, linklocal) or CIDR lists pass through
GBRAIN_HTTP_TRUST_PROXY=uniquelocal gbrain serve --http --port 8787
GBRAIN_HTTP_TRUST_PROXY="10.0.0.0/8,192.168.1.0/24" gbrain serve --http --port 8787
```
Both transports (Express OAuth server in `src/commands/serve-http.ts` and
the legacy bearer transport in `src/mcp/http-transport.ts`) read the same
env var, so single source of truth.
**Critical safety contract:** only widen past `'loopback'` when **both**
of these are true:
**Critical safety contract:** only set `GBRAIN_HTTP_TRUST_PROXY=1` when
**both** of these are true:
1. gbrain is reachable only via a trusted reverse proxy (not directly
exposed to the internet on the configured port). As of v0.34
`gbrain serve --http` binds `127.0.0.1` by default, so the
reverse-proxy-only posture is the out-of-the-box shape; only
override with `--bind 0.0.0.0` (or a specific interface IP) when
gbrain itself needs to accept remote connections directly.
exposed to the internet on the configured port). The simplest
guarantee is to bind gbrain to `127.0.0.1` or a private interface
and have the proxy forward to it.
2. The proxy strips any client-supplied `X-Forwarded-For` and `X-Real-IP`
headers, then sets them itself. (nginx with `proxy_set_header
X-Forwarded-For $remote_addr` does this; Cloudflare and most cloud
load balancers handle it automatically.)
If gbrain is reachable directly AND `GBRAIN_HTTP_TRUST_PROXY=1` (or any
non-loopback value) is set, clients can spoof their IP by sending
arbitrary `X-Forwarded-For` headers, defeating the pre-auth IP rate
limit. The `'loopback'` default protects against this by ignoring all
forwarded-for headers and using the socket peer address.
If gbrain is reachable directly AND `GBRAIN_HTTP_TRUST_PROXY=1` is set,
clients can spoof their IP by sending arbitrary `X-Forwarded-For`
headers, defeating the pre-auth IP rate limit. Without the flag, gbrain
ignores all forwarded-for headers and uses the socket peer address,
which is the safe default for direct-exposure deployments.
### Body size cap
@@ -247,14 +166,3 @@ psql "$DATABASE_URL" -c \
`body_too_large`, `parse_error`, `unknown_method`. Failed-auth rows have
`token_name = NULL`. Inserts are fire-and-forget so audit failures
never block requests.
**v0.26.9 redaction default.** The `params` column now stores
`{redacted, kind, declared_keys, unknown_key_count, approx_bytes}` instead
of raw JSON-RPC payloads. Declared keys (intersected against the operation's
spec) preserve for debug visibility; unknown keys are counted but never
named so attackers can't probe key existence; byte sizes bucket to 1KB so
content sizes can't be binary-searched. The same shape is broadcast on the
admin SSE feed at `/admin/events`. Operators on a personal laptop who want
raw payloads back can pass `gbrain serve --http --log-full-params` (loud
stderr warning at startup). Multi-tenant deployments should leave it
on the redacted default.
+14 -3042
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -1 +1 @@
0.42.3.0
0.22.13
-158
View File
@@ -1,158 +0,0 @@
# Design System — GBrain Admin Dashboard
## Product Context
- **What this is:** Admin dashboard for GBrain MCP server — manage OAuth agents, API keys, monitor requests
- **Who it's for:** GBrain operators managing multi-agent access to their brain
- **Space/industry:** Developer infrastructure (peers: Supabase dashboard, Vercel, Railway)
- **Project type:** Dense utilitarian admin panel — Steve Krug "Don't Make Me Think"
## Aesthetic Direction
- **Direction:** Industrial/Utilitarian — function-first, data-dense, zero decoration
- **Decoration level:** None — every pixel earns its place with information
- **Mood:** Ops dashboard for someone who builds. Not a marketing site. Not a consumer app. A cockpit.
- **Reference:** Supabase dashboard (dark + dense), Linear (restrained), Grafana (data-forward)
## Alignment
- **Text alignment:** Left-align everything. No centered text in tables, cards, forms, or labels.
- **Headings:** Left-aligned
- **Table data:** Left-aligned (including numbers — contextual readability over columnar alignment)
- **Form labels:** Left-aligned above inputs
- **Buttons in forms:** Right-aligned (action flows left-to-right: Cancel → Submit)
- **Modal titles:** Left-aligned
- **Page titles:** Left-aligned
- **Only exception:** Empty states and the login page lock icon can center for visual weight
## Typography
- **Display/Headings:** Inter (Semibold 600) — clean, neutral, disappears into the content
- **Body/UI:** Inter (Regular 400 / Medium 500)
- **Data/Tables/Code:** JetBrains Mono (Regular 400 / Medium 500) — monospace for anything the user might copy, any ID, any token, any technical value
- **Loading:** Google Fonts. `display=swap`.
- **Scale:**
- Page title: 24px / Inter Semibold
- Section title: 14px / Inter Semibold, uppercase, letter-spacing 0.5px
- Table header: 12px / Inter Medium, uppercase, letter-spacing 1px, muted color
- Body: 14px / Inter Regular
- Small/Caption: 13px
- Micro: 12px (badges, timestamps)
- Code/Data: 13px / JetBrains Mono
## Color
- **Approach:** Monochrome base + semantic color only. No primary brand color. Color means something.
- **Background:**
- Base: #0a0a0f (near-black with blue undertone)
- Surface/cards: #12121a
- Hover: #1a1a2a
- Input/code blocks: #0f0f1a
- **Borders:** #1e1e2e (default), #3a3a5a (hover/active)
- **Text:**
- Primary: #e0e0e0
- Secondary: #888888
- Muted: #555555
- Link: #88aaff
- **Semantic (badges only):**
- Success/active: #34a853
- Error/danger: #ff6b6b
- Warning: #f5a623
- Read scope: #3b82f6
- Write scope: #f59e0b
- Admin scope: #ef4444
- **No accent color.** The data IS the interface. Badges carry all the color.
## Spacing
- **Base unit:** 4px
- **Density:** Dense — this is an ops tool, not a landing page
- **Scale:** 4px, 8px, 12px, 16px, 20px, 24px, 32px, 48px
- **Table row padding:** 10px 16px
- **Card padding:** 24px
- **Modal padding:** 24px
- **Section gaps:** 24px between sections, 12px between related elements
## Layout
- **Sidebar:** Fixed left, 200px wide, dark (#0a0a0f)
- **Main content:** Fluid, max-width none (fills available space)
- **Grid:** Single column for tables (full width), 2-column for stats cards
- **Border radius:**
- Cards/panels: 16px
- Buttons/inputs: 8px
- Badges: 9999px (pill)
- Tables: 0 (sharp edges — data is rectangular)
## Components
### Tables
- Full-width, no outer border
- Header row: uppercase, letter-spaced, muted color, no background
- Data rows: subtle hover (#1a1a2a), pointer cursor when clickable
- All text left-aligned
- Monospace for IDs, tokens, latency values
### Badges
- Pill shape (border-radius: 9999px)
- Padding: 2px 8px
- Font: 12px
- Scoped to semantic meaning: `success`, `danger`, `read`, `write`, `admin`
### Buttons
- Primary: white text on #3a3a5a, hover brightens
- Secondary: muted text on transparent, border #1e1e2e
- Danger: white text on #ff6b6b background
- Size: 13px font, 6px 14px padding
### Modals
- Overlay: rgba(0,0,0,0.7)
- Card: #12121a, border #1e1e2e, border-radius 16px, max-width 480px
- Title: 18px Semibold, left-aligned
- Close: top-right ✕ button
### Drawers
- Right-side panel, 400px wide
- Slide in from right
- Dark overlay behind
- Close button top-right
- Sections separated by section titles (uppercase, muted)
### Tabs
- Inline horizontal, wrapping allowed
- Active: white text, bottom border
- Inactive: muted text, no border
- No background color on tabs
### Code blocks
- Background: rgba(0,0,0,0.3)
- Border-radius: 8px
- Padding: 10px 14px
- Font: JetBrains Mono 12px
- Copy button: right-aligned, subtle
### Empty states
- Centered text (only exception to left-align rule)
- Muted color
- Suggest next action
## Motion
- **Approach:** Minimal — transitions for hover states only
- **Duration:** 150ms for hovers, 200ms for drawer slide
- **No loading spinners** — show stale data until fresh arrives
- **SSE live feed:** Real-time, no animation on new entries (just prepend)
## Anti-Patterns (do NOT do these)
- ❌ Center-aligned table data
- ❌ Center-aligned headings or labels (except empty states)
- ❌ Gradient backgrounds
- ❌ Shadows (the dark theme IS the depth model)
- ❌ Rounded table corners
- ❌ Icons as navigation (use text labels)
- ❌ Loading skeletons (show real data or nothing)
- ❌ Confirmation toasts (action → result is immediate and visible)
- ❌ Color for decoration (every color means something)
## Decisions Log
| Date | Decision | Rationale |
|------|----------|-----------|
| 2026-05-01 | Dark theme only | Ops dashboard. No light mode needed. |
| 2026-05-01 | Steve Krug lens | Zero happy talk, mindless choices, scannable tables, billboard-speed comprehension. |
| 2026-05-01 | JetBrains Mono for data | Anything copyable or technical should be monospace. |
| 2026-05-03 | Left-align everything | Garry preference. Centered text is a design crutch. Left-align forces hierarchy through typography weight and spacing, not position. |
| 2026-05-03 | Incorporate GStack design DNA | Same family: Inter + JetBrains Mono, dark base, semantic-only color. Diverges on accent (GStack: amber; GBrain: none — data is the color). |
| 2026-05-03 | Per-client config export tabs | Claude Code, ChatGPT, Claude.ai, Cursor, Perplexity, JSON. Every agent has a copy-paste setup path. |
| 2026-05-03 | Magic link auth | Login page tells you to ask your agent. No pasting hex strings into forms. |
-257
View File
@@ -1,257 +0,0 @@
{
"lockfileVersion": 1,
"configVersion": 1,
"workspaces": {
"": {
"name": "gbrain-admin",
"dependencies": {
"react": "^19.1.0",
"react-dom": "^19.1.0",
},
"devDependencies": {
"@types/react": "^19.1.2",
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"typescript": "^5.8.3",
"vite": "^6.3.3",
},
},
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.0", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.28.5", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw=="],
"@babel/compat-data": ["@babel/compat-data@7.29.0", "", {}, "sha512-T1NCJqT/j9+cn8fvkt7jtwbLBfLC/1y1c7NtCeXFRgzGTsafi68MRv8yzkYSapBnFA6L3U2VSc02ciDzoAJhJg=="],
"@babel/core": ["@babel/core@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-compilation-targets": "^7.28.6", "@babel/helper-module-transforms": "^7.28.6", "@babel/helpers": "^7.28.6", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/traverse": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA=="],
"@babel/generator": ["@babel/generator@7.29.1", "", { "dependencies": { "@babel/parser": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw=="],
"@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.28.6", "", { "dependencies": { "@babel/compat-data": "^7.28.6", "@babel/helper-validator-option": "^7.27.1", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-JYtls3hqi15fcx5GaSNL7SCTJ2MNmjrkHXg4FSpOA/grxK8KwyZ5bubHsCq8FXCkua6xhuaaBit+3b7+VZRfcA=="],
"@babel/helper-globals": ["@babel/helper-globals@7.28.0", "", {}, "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw=="],
"@babel/helper-module-imports": ["@babel/helper-module-imports@7.28.6", "", { "dependencies": { "@babel/traverse": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw=="],
"@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.28.6", "", { "dependencies": { "@babel/helper-module-imports": "^7.28.6", "@babel/helper-validator-identifier": "^7.28.5", "@babel/traverse": "^7.28.6" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA=="],
"@babel/helper-plugin-utils": ["@babel/helper-plugin-utils@7.28.6", "", {}, "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug=="],
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@babel/helper-validator-option": ["@babel/helper-validator-option@7.27.1", "", {}, "sha512-YvjJow9FxbhFFKDSuFnVCe2WxXk1zWc22fFePVNEaWJEu8IrZVlda6N0uHwzZrUM1il7NC9Mlp4MaJYbYd9JSg=="],
"@babel/helpers": ["@babel/helpers@7.29.2", "", { "dependencies": { "@babel/template": "^7.28.6", "@babel/types": "^7.29.0" } }, "sha512-HoGuUs4sCZNezVEKdVcwqmZN8GoHirLUcLaYVNBK2J0DadGtdcqgr3BCbvH8+XUo4NGjNl3VOtSjEKNzqfFgKw=="],
"@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@babel/plugin-transform-react-jsx-self": ["@babel/plugin-transform-react-jsx-self@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-6UzkCs+ejGdZ5mFFC/OCUrv028ab2fp1znZmCZjAOBKiBK2jXD1O+BPSfX8X2qjJ75fZBMSnQn3Rq2mrBJK2mw=="],
"@babel/plugin-transform-react-jsx-source": ["@babel/plugin-transform-react-jsx-source@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-zbwoTsBruTeKB9hSq73ha66iFeJHuaFkUbwvqElnygoNbj/jHRsSeokowZFN3CZ64IvEqcmmkVe89OPXc7ldAw=="],
"@babel/template": ["@babel/template@7.28.6", "", { "dependencies": { "@babel/code-frame": "^7.28.6", "@babel/parser": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ=="],
"@babel/traverse": ["@babel/traverse@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-globals": "^7.28.0", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/types": "^7.29.0", "debug": "^4.3.1" } }, "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA=="],
"@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.25.12", "", { "os": "aix", "cpu": "ppc64" }, "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA=="],
"@esbuild/android-arm": ["@esbuild/android-arm@0.25.12", "", { "os": "android", "cpu": "arm" }, "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg=="],
"@esbuild/android-arm64": ["@esbuild/android-arm64@0.25.12", "", { "os": "android", "cpu": "arm64" }, "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg=="],
"@esbuild/android-x64": ["@esbuild/android-x64@0.25.12", "", { "os": "android", "cpu": "x64" }, "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg=="],
"@esbuild/darwin-arm64": ["@esbuild/darwin-arm64@0.25.12", "", { "os": "darwin", "cpu": "arm64" }, "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg=="],
"@esbuild/darwin-x64": ["@esbuild/darwin-x64@0.25.12", "", { "os": "darwin", "cpu": "x64" }, "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA=="],
"@esbuild/freebsd-arm64": ["@esbuild/freebsd-arm64@0.25.12", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg=="],
"@esbuild/freebsd-x64": ["@esbuild/freebsd-x64@0.25.12", "", { "os": "freebsd", "cpu": "x64" }, "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ=="],
"@esbuild/linux-arm": ["@esbuild/linux-arm@0.25.12", "", { "os": "linux", "cpu": "arm" }, "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw=="],
"@esbuild/linux-arm64": ["@esbuild/linux-arm64@0.25.12", "", { "os": "linux", "cpu": "arm64" }, "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ=="],
"@esbuild/linux-ia32": ["@esbuild/linux-ia32@0.25.12", "", { "os": "linux", "cpu": "ia32" }, "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA=="],
"@esbuild/linux-loong64": ["@esbuild/linux-loong64@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng=="],
"@esbuild/linux-mips64el": ["@esbuild/linux-mips64el@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw=="],
"@esbuild/linux-ppc64": ["@esbuild/linux-ppc64@0.25.12", "", { "os": "linux", "cpu": "ppc64" }, "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA=="],
"@esbuild/linux-riscv64": ["@esbuild/linux-riscv64@0.25.12", "", { "os": "linux", "cpu": "none" }, "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w=="],
"@esbuild/linux-s390x": ["@esbuild/linux-s390x@0.25.12", "", { "os": "linux", "cpu": "s390x" }, "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg=="],
"@esbuild/linux-x64": ["@esbuild/linux-x64@0.25.12", "", { "os": "linux", "cpu": "x64" }, "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw=="],
"@esbuild/netbsd-arm64": ["@esbuild/netbsd-arm64@0.25.12", "", { "os": "none", "cpu": "arm64" }, "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg=="],
"@esbuild/netbsd-x64": ["@esbuild/netbsd-x64@0.25.12", "", { "os": "none", "cpu": "x64" }, "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ=="],
"@esbuild/openbsd-arm64": ["@esbuild/openbsd-arm64@0.25.12", "", { "os": "openbsd", "cpu": "arm64" }, "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A=="],
"@esbuild/openbsd-x64": ["@esbuild/openbsd-x64@0.25.12", "", { "os": "openbsd", "cpu": "x64" }, "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw=="],
"@esbuild/openharmony-arm64": ["@esbuild/openharmony-arm64@0.25.12", "", { "os": "none", "cpu": "arm64" }, "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg=="],
"@esbuild/sunos-x64": ["@esbuild/sunos-x64@0.25.12", "", { "os": "sunos", "cpu": "x64" }, "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w=="],
"@esbuild/win32-arm64": ["@esbuild/win32-arm64@0.25.12", "", { "os": "win32", "cpu": "arm64" }, "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg=="],
"@esbuild/win32-ia32": ["@esbuild/win32-ia32@0.25.12", "", { "os": "win32", "cpu": "ia32" }, "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ=="],
"@esbuild/win32-x64": ["@esbuild/win32-x64@0.25.12", "", { "os": "win32", "cpu": "x64" }, "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA=="],
"@jridgewell/gen-mapping": ["@jridgewell/gen-mapping@0.3.13", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.0", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA=="],
"@jridgewell/remapping": ["@jridgewell/remapping@2.3.5", "", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.5", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ=="],
"@jridgewell/resolve-uri": ["@jridgewell/resolve-uri@3.1.2", "", {}, "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw=="],
"@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.5.5", "", {}, "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og=="],
"@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="],
"@rolldown/pluginutils": ["@rolldown/pluginutils@1.0.0-beta.27", "", {}, "sha512-+d0F4MKMCbeVUJwG96uQ4SgAznZNSq93I3V+9NHA4OpvqG8mRCpGdKmK8l/dl02h2CCDHwW2FqilnTyDcAnqjA=="],
"@rollup/rollup-android-arm-eabi": ["@rollup/rollup-android-arm-eabi@4.60.1", "", { "os": "android", "cpu": "arm" }, "sha512-d6FinEBLdIiK+1uACUttJKfgZREXrF0Qc2SmLII7W2AD8FfiZ9Wjd+rD/iRuf5s5dWrr1GgwXCvPqOuDquOowA=="],
"@rollup/rollup-android-arm64": ["@rollup/rollup-android-arm64@4.60.1", "", { "os": "android", "cpu": "arm64" }, "sha512-YjG/EwIDvvYI1YvYbHvDz/BYHtkY4ygUIXHnTdLhG+hKIQFBiosfWiACWortsKPKU/+dUwQQCKQM3qrDe8c9BA=="],
"@rollup/rollup-darwin-arm64": ["@rollup/rollup-darwin-arm64@4.60.1", "", { "os": "darwin", "cpu": "arm64" }, "sha512-mjCpF7GmkRtSJwon+Rq1N8+pI+8l7w5g9Z3vWj4T7abguC4Czwi3Yu/pFaLvA3TTeMVjnu3ctigusqWUfjZzvw=="],
"@rollup/rollup-darwin-x64": ["@rollup/rollup-darwin-x64@4.60.1", "", { "os": "darwin", "cpu": "x64" }, "sha512-haZ7hJ1JT4e9hqkoT9R/19XW2QKqjfJVv+i5AGg57S+nLk9lQnJ1F/eZloRO3o9Scy9CM3wQ9l+dkXtcBgN5Ew=="],
"@rollup/rollup-freebsd-arm64": ["@rollup/rollup-freebsd-arm64@4.60.1", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-czw90wpQq3ZsAVBlinZjAYTKduOjTywlG7fEeWKUA7oCmpA8xdTkxZZlwNJKWqILlq0wehoZcJYfBvOyhPTQ6w=="],
"@rollup/rollup-freebsd-x64": ["@rollup/rollup-freebsd-x64@4.60.1", "", { "os": "freebsd", "cpu": "x64" }, "sha512-KVB2rqsxTHuBtfOeySEyzEOB7ltlB/ux38iu2rBQzkjbwRVlkhAGIEDiiYnO2kFOkJp+Z7pUXKyrRRFuFUKt+g=="],
"@rollup/rollup-linux-arm-gnueabihf": ["@rollup/rollup-linux-arm-gnueabihf@4.60.1", "", { "os": "linux", "cpu": "arm" }, "sha512-L+34Qqil+v5uC0zEubW7uByo78WOCIrBvci69E7sFASRl0X7b/MB6Cqd1lky/CtcSVTydWa2WZwFuWexjS5o6g=="],
"@rollup/rollup-linux-arm-musleabihf": ["@rollup/rollup-linux-arm-musleabihf@4.60.1", "", { "os": "linux", "cpu": "arm" }, "sha512-n83O8rt4v34hgFzlkb1ycniJh7IR5RCIqt6mz1VRJD6pmhRi0CXdmfnLu9dIUS6buzh60IvACM842Ffb3xd6Gg=="],
"@rollup/rollup-linux-arm64-gnu": ["@rollup/rollup-linux-arm64-gnu@4.60.1", "", { "os": "linux", "cpu": "arm64" }, "sha512-Nql7sTeAzhTAja3QXeAI48+/+GjBJ+QmAH13snn0AJSNL50JsDqotyudHyMbO2RbJkskbMbFJfIJKWA6R1LCJQ=="],
"@rollup/rollup-linux-arm64-musl": ["@rollup/rollup-linux-arm64-musl@4.60.1", "", { "os": "linux", "cpu": "arm64" }, "sha512-+pUymDhd0ys9GcKZPPWlFiZ67sTWV5UU6zOJat02M1+PiuSGDziyRuI/pPue3hoUwm2uGfxdL+trT6Z9rxnlMA=="],
"@rollup/rollup-linux-loong64-gnu": ["@rollup/rollup-linux-loong64-gnu@4.60.1", "", { "os": "linux", "cpu": "none" }, "sha512-VSvgvQeIcsEvY4bKDHEDWcpW4Yw7BtlKG1GUT4FzBUlEKQK0rWHYBqQt6Fm2taXS+1bXvJT6kICu5ZwqKCnvlQ=="],
"@rollup/rollup-linux-loong64-musl": ["@rollup/rollup-linux-loong64-musl@4.60.1", "", { "os": "linux", "cpu": "none" }, "sha512-4LqhUomJqwe641gsPp6xLfhqWMbQV04KtPp7/dIp0nzPxAkNY1AbwL5W0MQpcalLYk07vaW9Kp1PBhdpZYYcEw=="],
"@rollup/rollup-linux-ppc64-gnu": ["@rollup/rollup-linux-ppc64-gnu@4.60.1", "", { "os": "linux", "cpu": "ppc64" }, "sha512-tLQQ9aPvkBxOc/EUT6j3pyeMD6Hb8QF2BTBnCQWP/uu1lhc9AIrIjKnLYMEroIz/JvtGYgI9dF3AxHZNaEH0rw=="],
"@rollup/rollup-linux-ppc64-musl": ["@rollup/rollup-linux-ppc64-musl@4.60.1", "", { "os": "linux", "cpu": "ppc64" }, "sha512-RMxFhJwc9fSXP6PqmAz4cbv3kAyvD1etJFjTx4ONqFP9DkTkXsAMU4v3Vyc5BgzC+anz7nS/9tp4obsKfqkDHg=="],
"@rollup/rollup-linux-riscv64-gnu": ["@rollup/rollup-linux-riscv64-gnu@4.60.1", "", { "os": "linux", "cpu": "none" }, "sha512-QKgFl+Yc1eEk6MmOBfRHYF6lTxiiiV3/z/BRrbSiW2I7AFTXoBFvdMEyglohPj//2mZS4hDOqeB0H1ACh3sBbg=="],
"@rollup/rollup-linux-riscv64-musl": ["@rollup/rollup-linux-riscv64-musl@4.60.1", "", { "os": "linux", "cpu": "none" }, "sha512-RAjXjP/8c6ZtzatZcA1RaQr6O1TRhzC+adn8YZDnChliZHviqIjmvFwHcxi4JKPSDAt6Uhf/7vqcBzQJy0PDJg=="],
"@rollup/rollup-linux-s390x-gnu": ["@rollup/rollup-linux-s390x-gnu@4.60.1", "", { "os": "linux", "cpu": "s390x" }, "sha512-wcuocpaOlaL1COBYiA89O6yfjlp3RwKDeTIA0hM7OpmhR1Bjo9j31G1uQVpDlTvwxGn2nQs65fBFL5UFd76FcQ=="],
"@rollup/rollup-linux-x64-gnu": ["@rollup/rollup-linux-x64-gnu@4.60.1", "", { "os": "linux", "cpu": "x64" }, "sha512-77PpsFQUCOiZR9+LQEFg9GClyfkNXj1MP6wRnzYs0EeWbPcHs02AXu4xuUbM1zhwn3wqaizle3AEYg5aeoohhg=="],
"@rollup/rollup-linux-x64-musl": ["@rollup/rollup-linux-x64-musl@4.60.1", "", { "os": "linux", "cpu": "x64" }, "sha512-5cIATbk5vynAjqqmyBjlciMJl1+R/CwX9oLk/EyiFXDWd95KpHdrOJT//rnUl4cUcskrd0jCCw3wpZnhIHdD9w=="],
"@rollup/rollup-openbsd-x64": ["@rollup/rollup-openbsd-x64@4.60.1", "", { "os": "openbsd", "cpu": "x64" }, "sha512-cl0w09WsCi17mcmWqqglez9Gk8isgeWvoUZ3WiJFYSR3zjBQc2J5/ihSjpl+VLjPqjQ/1hJRcqBfLjssREQILw=="],
"@rollup/rollup-openharmony-arm64": ["@rollup/rollup-openharmony-arm64@4.60.1", "", { "os": "none", "cpu": "arm64" }, "sha512-4Cv23ZrONRbNtbZa37mLSueXUCtN7MXccChtKpUnQNgF010rjrjfHx3QxkS2PI7LqGT5xXyYs1a7LbzAwT0iCA=="],
"@rollup/rollup-win32-arm64-msvc": ["@rollup/rollup-win32-arm64-msvc@4.60.1", "", { "os": "win32", "cpu": "arm64" }, "sha512-i1okWYkA4FJICtr7KpYzFpRTHgy5jdDbZiWfvny21iIKky5YExiDXP+zbXzm3dUcFpkEeYNHgQ5fuG236JPq0g=="],
"@rollup/rollup-win32-ia32-msvc": ["@rollup/rollup-win32-ia32-msvc@4.60.1", "", { "os": "win32", "cpu": "ia32" }, "sha512-u09m3CuwLzShA0EYKMNiFgcjjzwqtUMLmuCJLeZWjjOYA3IT2Di09KaxGBTP9xVztWyIWjVdsB2E9goMjZvTQg=="],
"@rollup/rollup-win32-x64-gnu": ["@rollup/rollup-win32-x64-gnu@4.60.1", "", { "os": "win32", "cpu": "x64" }, "sha512-k+600V9Zl1CM7eZxJgMyTUzmrmhB/0XZnF4pRypKAlAgxmedUA+1v9R+XOFv56W4SlHEzfeMtzujLJD22Uz5zg=="],
"@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.60.1", "", { "os": "win32", "cpu": "x64" }, "sha512-lWMnixq/QzxyhTV6NjQJ4SFo1J6PvOX8vUx5Wb4bBPsEb+8xZ89Bz6kOXpfXj9ak9AHTQVQzlgzBEc1SyM27xQ=="],
"@types/babel__core": ["@types/babel__core@7.20.5", "", { "dependencies": { "@babel/parser": "^7.20.7", "@babel/types": "^7.20.7", "@types/babel__generator": "*", "@types/babel__template": "*", "@types/babel__traverse": "*" } }, "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA=="],
"@types/babel__generator": ["@types/babel__generator@7.27.0", "", { "dependencies": { "@babel/types": "^7.0.0" } }, "sha512-ufFd2Xi92OAVPYsy+P4n7/U7e68fex0+Ee8gSG9KX7eo084CWiQ4sdxktvdl0bOPupXtVJPY19zk6EwWqUQ8lg=="],
"@types/babel__template": ["@types/babel__template@7.4.4", "", { "dependencies": { "@babel/parser": "^7.1.0", "@babel/types": "^7.0.0" } }, "sha512-h/NUaSyG5EyxBIp8YRxo4RMe2/qQgvyowRwVMzhYhBCONbW8PUsg4lkFMrhgZhUe5z3L3MiLDuvyJ/CaPa2A8A=="],
"@types/babel__traverse": ["@types/babel__traverse@7.28.0", "", { "dependencies": { "@babel/types": "^7.28.2" } }, "sha512-8PvcXf70gTDZBgt9ptxJ8elBeBjcLOAcOtoO/mPJjtji1+CdGbHgm77om1GrsPxsiE+uXIpNSK64UYaIwQXd4Q=="],
"@types/estree": ["@types/estree@1.0.8", "", {}, "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w=="],
"@types/react": ["@types/react@19.2.14", "", { "dependencies": { "csstype": "^3.2.2" } }, "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w=="],
"@types/react-dom": ["@types/react-dom@19.2.3", "", { "peerDependencies": { "@types/react": "^19.2.0" } }, "sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ=="],
"@vitejs/plugin-react": ["@vitejs/plugin-react@4.7.0", "", { "dependencies": { "@babel/core": "^7.28.0", "@babel/plugin-transform-react-jsx-self": "^7.27.1", "@babel/plugin-transform-react-jsx-source": "^7.27.1", "@rolldown/pluginutils": "1.0.0-beta.27", "@types/babel__core": "^7.20.5", "react-refresh": "^0.17.0" }, "peerDependencies": { "vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0" } }, "sha512-gUu9hwfWvvEDBBmgtAowQCojwZmJ5mcLn3aufeCsitijs3+f2NsrPtlAWIR6OPiqljl96GVCUbLe0HyqIpVaoA=="],
"baseline-browser-mapping": ["baseline-browser-mapping@2.10.18", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-VSnGQAOLtP5mib/DPyg2/t+Tlv65NTBz83BJBJvmLVHHuKJVaDOBvJJykiT5TR++em5nfAySPccDZDa4oSrn8A=="],
"browserslist": ["browserslist@4.28.2", "", { "dependencies": { "baseline-browser-mapping": "^2.10.12", "caniuse-lite": "^1.0.30001782", "electron-to-chromium": "^1.5.328", "node-releases": "^2.0.36", "update-browserslist-db": "^1.2.3" }, "bin": { "browserslist": "cli.js" } }, "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg=="],
"caniuse-lite": ["caniuse-lite@1.0.30001788", "", {}, "sha512-6q8HFp+lOQtcf7wBK+uEenxymVWkGKkjFpCvw5W25cmMwEDU45p1xQFBQv8JDlMMry7eNxyBaR+qxgmTUZkIRQ=="],
"convert-source-map": ["convert-source-map@2.0.0", "", {}, "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg=="],
"csstype": ["csstype@3.2.3", "", {}, "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ=="],
"debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],
"electron-to-chromium": ["electron-to-chromium@1.5.336", "", {}, "sha512-AbH9q9J455r/nLmdNZes0G0ZKcRX73FicwowalLs6ijwOmCJSRRrLX63lcAlzy9ux3dWK1w1+1nsBJEWN11hcQ=="],
"esbuild": ["esbuild@0.25.12", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.25.12", "@esbuild/android-arm": "0.25.12", "@esbuild/android-arm64": "0.25.12", "@esbuild/android-x64": "0.25.12", "@esbuild/darwin-arm64": "0.25.12", "@esbuild/darwin-x64": "0.25.12", "@esbuild/freebsd-arm64": "0.25.12", "@esbuild/freebsd-x64": "0.25.12", "@esbuild/linux-arm": "0.25.12", "@esbuild/linux-arm64": "0.25.12", "@esbuild/linux-ia32": "0.25.12", "@esbuild/linux-loong64": "0.25.12", "@esbuild/linux-mips64el": "0.25.12", "@esbuild/linux-ppc64": "0.25.12", "@esbuild/linux-riscv64": "0.25.12", "@esbuild/linux-s390x": "0.25.12", "@esbuild/linux-x64": "0.25.12", "@esbuild/netbsd-arm64": "0.25.12", "@esbuild/netbsd-x64": "0.25.12", "@esbuild/openbsd-arm64": "0.25.12", "@esbuild/openbsd-x64": "0.25.12", "@esbuild/openharmony-arm64": "0.25.12", "@esbuild/sunos-x64": "0.25.12", "@esbuild/win32-arm64": "0.25.12", "@esbuild/win32-ia32": "0.25.12", "@esbuild/win32-x64": "0.25.12" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg=="],
"escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],
"fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
"fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="],
"gensync": ["gensync@1.0.0-beta.2", "", {}, "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
"jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="],
"json5": ["json5@2.2.3", "", { "bin": { "json5": "lib/cli.js" } }, "sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg=="],
"lru-cache": ["lru-cache@5.1.1", "", { "dependencies": { "yallist": "^3.0.2" } }, "sha512-KpNARQA3Iwv+jTA0utUVVbrh+Jlrr1Fv0e56GGzAFOXN7dk/FviaDW8LHmK52DlcH4WP2n6gI8vN1aesBFgo9w=="],
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nanoid": ["nanoid@3.3.11", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="],
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
"picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="],
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.9", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-7a70Nsot+EMX9fFU3064K/kdHWZqGVY+BADLyXc8Dfv+mTLLVl6JzJpPaCZ2kQL9gIJvKXSLMHhqdRRjwQeFtw=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
"react-dom": ["react-dom@19.2.5", "", { "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { "react": "^19.2.5" } }, "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag=="],
"react-refresh": ["react-refresh@0.17.0", "", {}, "sha512-z6F7K9bV85EfseRCp2bzrpyQ0Gkw1uLoCel9XBVWPg/TjRj94SkJzUTGfOa4bs7iJvBWtQG0Wq7wnI0syw3EBQ=="],
"rollup": ["rollup@4.60.1", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.60.1", "@rollup/rollup-android-arm64": "4.60.1", "@rollup/rollup-darwin-arm64": "4.60.1", "@rollup/rollup-darwin-x64": "4.60.1", "@rollup/rollup-freebsd-arm64": "4.60.1", "@rollup/rollup-freebsd-x64": "4.60.1", "@rollup/rollup-linux-arm-gnueabihf": "4.60.1", "@rollup/rollup-linux-arm-musleabihf": "4.60.1", "@rollup/rollup-linux-arm64-gnu": "4.60.1", "@rollup/rollup-linux-arm64-musl": "4.60.1", "@rollup/rollup-linux-loong64-gnu": "4.60.1", "@rollup/rollup-linux-loong64-musl": "4.60.1", "@rollup/rollup-linux-ppc64-gnu": "4.60.1", "@rollup/rollup-linux-ppc64-musl": "4.60.1", "@rollup/rollup-linux-riscv64-gnu": "4.60.1", "@rollup/rollup-linux-riscv64-musl": "4.60.1", "@rollup/rollup-linux-s390x-gnu": "4.60.1", "@rollup/rollup-linux-x64-gnu": "4.60.1", "@rollup/rollup-linux-x64-musl": "4.60.1", "@rollup/rollup-openbsd-x64": "4.60.1", "@rollup/rollup-openharmony-arm64": "4.60.1", "@rollup/rollup-win32-arm64-msvc": "4.60.1", "@rollup/rollup-win32-ia32-msvc": "4.60.1", "@rollup/rollup-win32-x64-gnu": "4.60.1", "@rollup/rollup-win32-x64-msvc": "4.60.1", "fsevents": "~2.3.2" }, "bin": { "rollup": "dist/bin/rollup" } }, "sha512-VmtB2rFU/GroZ4oL8+ZqXgSA38O6GR8KSIvWmEFv63pQ0G6KaBH9s07PO8XTXP4vI+3UJUEypOfjkGfmSBBR0w=="],
"scheduler": ["scheduler@0.27.0", "", {}, "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q=="],
"semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="],
"source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"tinyglobby": ["tinyglobby@0.2.16", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.4" } }, "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg=="],
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "^3.2.0", "picocolors": "^1.1.1" }, "peerDependencies": { "browserslist": ">= 4.21.0" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="],
"vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="],
"yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
}
}
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
-16
View File
@@ -1,16 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>GBrain Admin</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<script type="module" crossorigin src="/admin/assets/index-DqP-zmqH.js"></script>
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
</head>
<body>
<div id="root"></div>
</body>
</html>
-15
View File
@@ -1,15 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>GBrain Admin</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>
-21
View File
@@ -1,21 +0,0 @@
{
"name": "gbrain-admin",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"react": "^19.1.0",
"react-dom": "^19.1.0"
},
"devDependencies": {
"@types/react": "^19.1.2",
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"vite": "^6.3.3",
"typescript": "^5.8.3"
}
}
-92
View File
@@ -1,92 +0,0 @@
import React, { useState, useEffect } from 'react';
import { LoginPage } from './pages/Login';
import { DashboardPage } from './pages/Dashboard';
import { AgentsPage } from './pages/Agents';
import { RequestLogPage } from './pages/RequestLog';
import { CalibrationPage } from './pages/Calibration';
import { JobsWatchPage } from './pages/JobsWatch';
import { api } from './api';
type Page = 'login' | 'dashboard' | 'agents' | 'log' | 'calibration' | 'jobs';
function getPage(): Page {
const hash = window.location.hash.replace('#', '') || 'dashboard';
if (['login', 'dashboard', 'agents', 'log', 'calibration', 'jobs'].includes(hash)) return hash as Page;
return 'dashboard';
}
export function App() {
const [page, setPage] = useState<Page>(getPage);
useEffect(() => {
const onHash = () => setPage(getPage());
window.addEventListener('hashchange', onHash);
return () => window.removeEventListener('hashchange', onHash);
}, []);
const navigate = (p: Page) => {
window.location.hash = p;
setPage(p);
};
if (page === 'login') {
return <LoginPage onLogin={() => navigate('dashboard')} />;
}
const handleSignOutEverywhere = async () => {
if (!confirm('Sign out every active admin session, including other browsers and tabs? Each one will need to re-authenticate via a fresh magic link.')) {
return;
}
try {
await api.signOutEverywhere();
} catch {
// Even if the call fails, push to login — cookie is likely already invalid.
}
navigate('login');
};
return (
<div className="app">
<nav className="sidebar">
<div className="sidebar-logo">GBrain</div>
<div className="sidebar-nav">
<a className={`nav-item ${page === 'dashboard' ? 'active' : ''}`}
onClick={() => navigate('dashboard')}>Dashboard</a>
<a className={`nav-item ${page === 'agents' ? 'active' : ''}`}
onClick={() => navigate('agents')}>Agents</a>
<a className={`nav-item ${page === 'log' ? 'active' : ''}`}
onClick={() => navigate('log')}>Request Log</a>
<a className={`nav-item ${page === 'calibration' ? 'active' : ''}`}
onClick={() => navigate('calibration')}>Calibration</a>
<a className={`nav-item ${page === 'jobs' ? 'active' : ''}`}
onClick={() => navigate('jobs')}>Jobs Watch</a>
</div>
<div style={{ marginTop: 'auto', padding: '16px 12px', borderTop: '1px solid var(--border)' }}>
<button
onClick={handleSignOutEverywhere}
style={{
background: 'transparent',
border: '1px solid var(--border)',
color: 'var(--text-secondary)',
padding: '6px 10px',
borderRadius: 6,
fontSize: 12,
cursor: 'pointer',
width: '100%',
}}
title="Revoke every active admin session — every browser, every tab"
>
Sign out everywhere
</button>
</div>
</nav>
<main className="main">
{page === 'dashboard' && <DashboardPage />}
{page === 'agents' && <AgentsPage />}
{page === 'log' && <RequestLogPage />}
{page === 'calibration' && <CalibrationPage />}
{page === 'jobs' && <JobsWatchPage />}
</main>
</div>
);
}
-55
View File
@@ -1,55 +0,0 @@
const BASE = '';
// v0.26.3 trust model (D11 + D12): the admin UI does NOT cache the
// bootstrap token in browser JS state. On 401, redirect to login —
// no auto-reauth via saved token, no localStorage/sessionStorage read.
// The HttpOnly cookie set by /admin/login is the only session credential.
async function apiFetch(path: string, options?: RequestInit) {
const res = await fetch(`${BASE}${path}`, {
...options,
credentials: 'same-origin',
headers: { 'Content-Type': 'application/json', ...options?.headers },
});
if (res.status === 401) {
// No token cache to retry from. Redirect to login.
window.location.hash = '#login';
throw new Error('Unauthorized');
}
if (!res.ok) {
const body = await res.json().catch(() => ({}));
throw new Error(body.error || `HTTP ${res.status}`);
}
return res.json();
}
// v0.36.1.0 (T15 / E6) — SVG fetch (text/plain payload, NOT JSON).
async function apiFetchText(path: string) {
const res = await fetch(`${BASE}${path}`, { credentials: 'same-origin' });
if (res.status === 401) {
window.location.hash = '#login';
throw new Error('Unauthorized');
}
if (!res.ok) throw new Error(`HTTP ${res.status}`);
return res.text();
}
export const api = {
login: (token: string) => apiFetch('/admin/login', { method: 'POST', body: JSON.stringify({ token }) }),
signOutEverywhere: () => apiFetch('/admin/api/sign-out-everywhere', { method: 'POST' }),
stats: () => apiFetch('/admin/api/stats'),
health: () => apiFetch('/admin/api/health-indicators'),
agents: () => apiFetch('/admin/api/agents'),
requests: (page = 1, qs = '') => apiFetch(`/admin/api/requests?page=${page}${qs}`),
apiKeys: () => apiFetch('/admin/api/api-keys'),
createApiKey: (name: string) => apiFetch('/admin/api/api-keys', { method: 'POST', body: JSON.stringify({ name }) }),
revokeApiKey: (name: string) => apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name }) }),
updateClientTtl: (clientId: string, tokenTtl: number | null) => apiFetch('/admin/api/update-client-ttl', { method: 'POST', body: JSON.stringify({ clientId, tokenTtl }) }),
revokeClient: (clientId: string) => apiFetch('/admin/api/revoke-client', { method: 'POST', body: JSON.stringify({ clientId }) }),
// v0.36.1.0 (T15 / E6) — calibration endpoints.
calibrationProfile: (holder?: string) =>
apiFetch(`/admin/api/calibration/profile${holder ? `?holder=${encodeURIComponent(holder)}` : ''}`),
calibrationChart: (type: string, holder?: string) =>
apiFetchText(`/admin/api/calibration/charts/${encodeURIComponent(type)}${holder ? `?holder=${encodeURIComponent(holder)}` : ''}`),
// v0.41 D2 — live minion-jobs dashboard snapshot.
jobsWatch: () => apiFetch('/admin/api/jobs/watch'),
};
-359
View File
@@ -1,359 +0,0 @@
:root {
--bg-primary: #0a0a0f;
--bg-secondary: #14141f;
--bg-tertiary: #1e1e2e;
--text-primary: #e0e0e0;
--text-secondary: #888;
/* v0.36.1.0 TD2 bumped from #555 (contrast 4.0 on #0a0a0f bg, below WCAG AA
4.5 for body text) to #777 (contrast ~5.5, passes AA). Applies globally
to Dashboard, Agents, RequestLog, and the new Calibration tab. */
--text-muted: #777;
--accent: #3b82f6;
--success: #22c55e;
--warning: #f59e0b;
--error: #ef4444;
--font-mono: 'JetBrains Mono', monospace;
--font-sans: 'Inter', system-ui, sans-serif;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: var(--font-sans);
background: var(--bg-primary);
color: var(--text-primary);
font-size: 14px;
line-height: 1.5;
}
/* Layout */
.app { display: flex; min-height: 100vh; }
.sidebar {
width: 200px;
background: var(--bg-secondary);
border-right: 1px solid #1e1e2e;
padding: 16px 0;
flex-shrink: 0;
display: flex;
flex-direction: column;
}
.sidebar-logo {
font-size: 18px;
font-weight: 600;
padding: 0 16px 24px;
color: var(--text-primary);
}
.sidebar-nav { display: flex; flex-direction: column; gap: 2px; }
.nav-item {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 16px;
color: var(--text-secondary);
text-decoration: none;
font-size: 13px;
cursor: pointer;
border-left: 3px solid transparent;
transition: all 0.15s;
}
.nav-item:hover { background: var(--bg-tertiary); color: var(--text-primary); }
.nav-item.active {
border-left-color: var(--accent);
background: var(--bg-tertiary);
color: var(--text-primary);
}
.main { flex: 1; padding: 24px 32px; overflow-y: auto; }
.page-title {
font-size: 24px;
font-weight: 600;
margin-bottom: 24px;
}
/* Metrics bar */
.metrics { display: flex; gap: 16px; margin-bottom: 24px; }
.metric {
background: var(--bg-secondary);
padding: 16px 20px;
border-radius: 6px;
min-width: 140px;
}
.metric-value {
font-family: var(--font-mono);
font-size: 28px;
font-weight: 500;
}
.metric-label { font-size: 12px; color: var(--text-secondary); margin-top: 4px; }
/* Tables */
table { width: 100%; border-collapse: collapse; }
th {
text-align: left;
font-size: 11px;
text-transform: uppercase;
color: var(--text-muted);
padding: 8px 12px;
font-weight: 500;
letter-spacing: 0.5px;
}
td {
padding: 10px 12px;
font-size: 13px;
border-top: 1px solid #1a1a2a;
}
tr:hover td { background: var(--bg-tertiary); }
/* Badges */
.badge {
display: inline-block;
padding: 2px 8px;
border-radius: 10px;
font-size: 11px;
font-weight: 500;
}
.badge-read { background: rgba(59,130,246,0.15); color: var(--accent); }
.badge-write { background: rgba(245,158,11,0.15); color: var(--warning); }
.badge-admin { background: rgba(239,68,68,0.15); color: var(--error); }
.badge-success { background: rgba(34,197,94,0.15); color: var(--success); }
.badge-error { background: rgba(239,68,68,0.15); color: var(--error); }
/* Status dots */
.status-dot {
display: inline-block;
width: 8px;
height: 8px;
border-radius: 50%;
margin-right: 6px;
}
.status-active { background: var(--success); }
.status-warning { background: var(--warning); }
.status-inactive { background: var(--text-muted); }
/* Buttons */
.btn {
padding: 8px 16px;
border-radius: 6px;
font-size: 13px;
font-weight: 500;
cursor: pointer;
border: none;
transition: all 0.15s;
}
.btn-primary { background: var(--accent); color: white; }
.btn-primary:hover { background: #2563eb; }
.btn-secondary { background: transparent; color: var(--text-secondary); border: 1px solid #333; }
.btn-secondary:hover { border-color: var(--text-secondary); color: var(--text-primary); }
.btn-danger { background: transparent; color: var(--error); border: 1px solid var(--error); }
.btn-danger:hover { background: rgba(239,68,68,0.1); }
/* Forms */
input, select {
background: var(--bg-primary);
border: 1px solid #333;
color: var(--text-primary);
padding: 8px 12px;
border-radius: 6px;
font-size: 13px;
font-family: var(--font-sans);
width: 100%;
}
input:focus, select:focus {
outline: none;
border-color: var(--accent);
box-shadow: 0 0 0 2px rgba(59,130,246,0.2);
}
input::placeholder { color: var(--text-muted); }
label { display: block; font-size: 13px; font-weight: 500; margin-bottom: 6px; }
/* Modal */
.modal-overlay {
position: fixed;
inset: 0;
background: rgba(0,0,0,0.7);
display: flex;
align-items: center;
justify-content: center;
z-index: 100;
}
.modal {
background: var(--bg-secondary);
border-radius: 8px;
padding: 24px;
min-width: 420px;
max-width: 520px;
}
.modal-title { font-size: 18px; font-weight: 600; margin-bottom: 20px; }
/* Drawer */
.drawer-overlay {
position: fixed;
inset: 0;
background: rgba(0,0,0,0.5);
z-index: 90;
}
.drawer {
position: fixed;
right: 0;
top: 0;
bottom: 0;
width: 420px;
background: var(--bg-secondary);
border-left: 1px solid var(--accent);
padding: 24px;
z-index: 91;
overflow-y: auto;
}
.drawer-close {
position: absolute;
top: 16px;
right: 16px;
background: none;
border: none;
color: var(--text-muted);
font-size: 18px;
cursor: pointer;
}
/* Section headers */
.section-title {
font-size: 11px;
text-transform: uppercase;
color: var(--text-muted);
letter-spacing: 0.5px;
margin: 20px 0 12px;
font-weight: 500;
}
/* Health panel */
.health-panel {
background: var(--bg-secondary);
border-radius: 6px;
padding: 16px;
}
.health-row {
display: flex;
justify-content: space-between;
padding: 6px 0;
font-size: 13px;
}
/* Code block */
.code-block {
background: var(--bg-primary);
border-radius: 6px;
padding: 12px;
font-family: var(--font-mono);
font-size: 12px;
overflow-x: auto;
position: relative;
}
.code-block .copy-btn {
position: absolute;
top: 8px;
right: 8px;
background: var(--accent);
color: white;
border: none;
padding: 4px 10px;
border-radius: 4px;
font-size: 11px;
cursor: pointer;
}
/* Activity feed */
.feed { max-height: 400px; overflow-y: auto; }
.feed-empty {
color: var(--text-muted);
text-align: center;
padding: 32px;
font-size: 13px;
}
/* Sparkline */
.sparkline { display: inline-block; vertical-align: middle; }
/* Filter bar */
.filter-bar { display: flex; gap: 12px; margin-bottom: 16px; align-items: center; }
.filter-bar select { width: auto; min-width: 140px; }
/* Pagination */
.pagination {
display: flex;
justify-content: space-between;
align-items: center;
padding: 12px 0;
font-size: 13px;
color: var(--text-secondary);
}
.pagination button {
background: var(--bg-secondary);
border: 1px solid #333;
color: var(--text-primary);
padding: 6px 12px;
border-radius: 4px;
cursor: pointer;
font-size: 12px;
}
.pagination button:disabled { opacity: 0.3; cursor: default; }
/* Warning bar */
.warning-bar {
background: rgba(245,158,11,0.15);
border: 1px solid var(--warning);
color: var(--warning);
padding: 10px 16px;
border-radius: 6px;
font-size: 13px;
margin: 12px 0;
}
/* Checkbox */
.checkbox-group { display: flex; gap: 16px; flex-wrap: wrap; }
.checkbox-label {
display: flex;
align-items: center;
gap: 6px;
font-size: 13px;
cursor: pointer;
}
/* Tabs */
.tabs { display: flex; gap: 0; margin-bottom: 12px; }
.tab {
padding: 6px 12px;
font-size: 13px;
color: var(--text-secondary);
cursor: pointer;
border-bottom: 2px solid transparent;
}
.tab.active { color: var(--accent); border-bottom-color: var(--accent); }
/* Login page */
.login-page {
display: flex;
align-items: center;
justify-content: center;
min-height: 100vh;
background: var(--bg-primary);
}
.login-box { text-align: left; width: 340px; }
.login-logo { font-size: 32px; font-weight: 600; margin-bottom: 32px; }
.login-hint { color: var(--text-muted); font-size: 12px; margin-top: 12px; }
.login-error { color: var(--error); font-size: 13px; margin-top: 8px; }
/* Monospace data */
.mono { font-family: var(--font-mono); font-size: 12px; }
/* Responsive */
@media (max-width: 768px) {
.sidebar { display: none; }
.main { padding: 16px; }
.metrics { flex-wrap: wrap; }
.drawer { width: 100%; }
}
-25
View File
@@ -1,25 +0,0 @@
/**
* Admin SPA scope constants HAND-MAINTAINED MIRROR of src/core/scope.ts.
*
* The admin tsconfig.json scopes `include: ['src']` to admin/src/, so we
* cannot directly import from ../../src/core/scope.ts without breaking the
* SPA's compile boundary. Instead, this file is a hand-maintained duplicate;
* scripts/check-admin-scope-drift.sh fails the build if the two lists drift.
*
* If you change ALLOWED_SCOPES in src/core/scope.ts, update this file too,
* or `bun run verify` will reject the change.
*/
export type Scope = 'read' | 'write' | 'admin' | 'sources_admin' | 'users_admin' | 'agent';
// MIRROR OF src/core/scope.ts ALLOWED_SCOPES_LIST — keep alphabetically sorted.
// v0.38: 'agent' added for the submit_agent remote-MCP op (sibling to admin,
// NOT implied — existing admin clients must re-register to opt in).
export const ALLOWED_SCOPES_LIST: ReadonlyArray<Scope> = [
'admin',
'agent',
'read',
'sources_admin',
'users_admin',
'write',
];
-10
View File
@@ -1,10 +0,0 @@
import React from 'react';
import ReactDOM from 'react-dom/client';
import { App } from './App';
import './index.css';
ReactDOM.createRoot(document.getElementById('root')!).render(
<React.StrictMode>
<App />
</React.StrictMode>,
);
-633
View File
@@ -1,633 +0,0 @@
import React, { useState, useEffect } from 'react';
import { api } from '../api';
import { ALLOWED_SCOPES_LIST, type Scope } from '../lib/scope-constants';
function timeAgo(date: Date): string {
const s = Math.floor((Date.now() - date.getTime()) / 1000);
if (s < 60) return 'just now';
if (s < 3600) return `${Math.floor(s / 60)}m ago`;
if (s < 86400) return `${Math.floor(s / 3600)}h ago`;
return `${Math.floor(s / 86400)}d ago`;
}
interface Agent {
id: string;
name: string;
auth_type: 'oauth' | 'api_key';
client_id?: string; // compat
client_name?: string; // compat
grant_types: string[];
scope: string;
created_at: string;
last_used_at: string | null;
total_requests: number;
requests_today: number;
token_ttl: number | null;
status: 'active' | 'revoked';
}
interface ApiKey {
id: string;
name: string;
created_at: string;
last_used_at: string | null;
status: 'active' | 'revoked';
}
export function AgentsPage() {
const [agents, setAgents] = useState<Agent[]>([]);
const [hideRevoked, setHideRevoked] = useState(true);
const [showRegister, setShowRegister] = useState(false);
const [showCredentials, setShowCredentials] = useState<{ clientId: string; clientSecret: string; name: string } | null>(null);
const [showApiKeyCreate, setShowApiKeyCreate] = useState(false);
const [showApiKeyToken, setShowApiKeyToken] = useState<{ name: string; token: string } | null>(null);
const [selectedAgent, setSelectedAgent] = useState<Agent | null>(null);
useEffect(() => { loadAgents(); }, []);
const loadAgents = () => { api.agents().then(setAgents).catch(() => {}); };
return (
<>
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: 24 }}>
<h1 className="page-title" style={{ marginBottom: 0 }}>Agents</h1>
<div style={{ display: 'flex', gap: 8, alignItems: 'center' }}>
<label style={{ fontSize: 13, color: 'var(--text-secondary)', display: 'flex', alignItems: 'center', gap: 6, cursor: 'pointer' }}>
<input type="checkbox" checked={hideRevoked} onChange={e => setHideRevoked(e.target.checked)} /> Hide revoked
</label>
<button className="btn btn-secondary" onClick={() => setShowApiKeyCreate(true)}>+ API Key</button>
<button className="btn btn-primary" onClick={() => setShowRegister(true)}>+ OAuth Client</button>
</div>
</div>
{(() => {
// Filter once and reuse, so the empty-state guard sees the same
// rows the table renders. Pre-fix: agents.length === 0 used the
// unfiltered array, so an all-revoked dataset with hideRevoked=on
// showed a header-only table with no placeholder.
const visibleAgents = agents.filter(a => !hideRevoked || a.status !== 'revoked');
if (agents.length === 0) {
return (
<div style={{ textAlign: 'center', padding: 48, color: 'var(--text-muted)' }}>
No agents registered. Register your first agent to get started.
</div>
);
}
if (visibleAgents.length === 0) {
return (
<div style={{ textAlign: 'center', padding: 48, color: 'var(--text-muted)' }}>
All agents are revoked. Uncheck "Hide revoked" to view them.
</div>
);
}
return (
<>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Scopes</th>
<th>Status</th>
<th>Requests</th>
<th>Last Used</th>
</tr>
</thead>
<tbody>
{visibleAgents.map(a => (
<tr key={a.id} onClick={() => setSelectedAgent(a)}
style={{ cursor: 'pointer' }}>
<td style={{ fontWeight: 500 }}>{a.name || a.client_name}</td>
<td>
<span className={`badge ${a.auth_type === 'oauth' ? 'badge-read' : 'badge-write'}`} style={{ fontSize: 11 }}>
{a.auth_type === 'oauth' ? 'OAuth' : 'API Key'}
</span>
</td>
<td>
{(a.scope || '').split(' ').filter(Boolean).map(s => (
<span key={s} className={`badge badge-${s}`} style={{ marginRight: 4 }}>{s}</span>
))}
</td>
<td>
<span className={`badge ${a.status === 'active' ? 'badge-success' : 'badge-danger'}`}>{a.status}</span>
</td>
<td>
<span style={{ fontWeight: 500 }}>{a.requests_today || 0}</span>
<span style={{ color: 'var(--text-muted)', fontSize: 12 }}> / {a.total_requests || 0}</span>
</td>
<td style={{ color: 'var(--text-secondary)' }}>
{a.last_used_at ? timeAgo(new Date(a.last_used_at)) : 'Never'}
</td>
</tr>
))}
</tbody>
</table>
<div style={{ color: 'var(--text-muted)', fontSize: 13, marginTop: 12 }}>
{agents.filter(a => a.status === 'active').length} active / {agents.length} total
</div>
</>
);
})()}
{showRegister && (
<RegisterModal
onClose={() => setShowRegister(false)}
onRegistered={(creds) => { setShowRegister(false); setShowCredentials(creds); loadAgents(); }}
/>
)}
{showCredentials && (
<CredentialsModal
credentials={showCredentials}
onClose={() => setShowCredentials(null)}
/>
)}
{selectedAgent && (
<AgentDrawer agent={selectedAgent} onClose={() => setSelectedAgent(null)} onRevoked={loadAgents} />
)}
{showApiKeyCreate && (
<ApiKeyCreateModal
onClose={() => setShowApiKeyCreate(false)}
onCreated={(result) => { setShowApiKeyCreate(false); setShowApiKeyToken(result); loadAgents(); }}
/>
)}
{showApiKeyToken && (
<ApiKeyTokenModal token={showApiKeyToken} onClose={() => setShowApiKeyToken(null)} />
)}
</>
);
}
function ApiKeyCreateModal({ onClose, onCreated }: {
onClose: () => void;
onCreated: (result: { name: string; token: string }) => void;
}) {
const [name, setName] = useState('');
const [loading, setLoading] = useState(false);
const [error, setError] = useState('');
const handleSubmit = async (e: React.FormEvent) => {
e.preventDefault();
if (!name.trim()) { setError('Name required'); return; }
setLoading(true);
try {
const data = await api.createApiKey(name.trim());
onCreated({ name: data.name, token: data.token });
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed');
} finally { setLoading(false); }
};
return (
<div className="modal-overlay" onClick={onClose}>
<form className="modal" onClick={e => e.stopPropagation()} onSubmit={handleSubmit}>
<div className="modal-title">Create API Key</div>
<p style={{ color: 'var(--text-secondary)', fontSize: 13, marginBottom: 16 }}>
API keys use simple bearer token auth. They grant full read+write+admin access.
For scoped access, use OAuth clients instead.
</p>
<div style={{ marginBottom: 16 }}>
<label>Key Name</label>
<input placeholder="e.g. claude-code-local" value={name} onChange={e => setName(e.target.value)} autoFocus />
</div>
{error && <div style={{ color: 'var(--error)', fontSize: 13, marginBottom: 12 }}>{error}</div>}
<div style={{ display: 'flex', gap: 12, justifyContent: 'flex-end' }}>
<button type="button" className="btn btn-secondary" onClick={onClose}>Cancel</button>
<button type="submit" className="btn btn-primary" disabled={loading}>
{loading ? 'Creating...' : 'Create Key'}
</button>
</div>
</form>
</div>
);
}
function ApiKeyTokenModal({ token, onClose }: {
token: { name: string; token: string };
onClose: () => void;
}) {
const copy = (text: string) => navigator.clipboard.writeText(text);
return (
<div className="modal-overlay">
<div className="modal" style={{ maxWidth: 560 }}>
<div style={{ textAlign: 'center', marginBottom: 16 }}>
<div style={{ fontSize: 36, color: 'var(--success)', marginBottom: 8 }}>&#10003;</div>
<div style={{ fontSize: 20, fontWeight: 600 }}>API Key Created</div>
</div>
<div style={{ marginBottom: 12 }}>
<label style={{ fontSize: 12 }}>Name</label>
<div className="code-block"><span>{token.name}</span></div>
</div>
<div style={{ marginBottom: 12 }}>
<label style={{ fontSize: 12 }}>Bearer Token</label>
<div className="code-block">
<span>{token.token}</span>
<button className="copy-btn" onClick={() => copy(token.token)}>Copy</button>
</div>
</div>
<div style={{ marginBottom: 12 }}>
<label style={{ fontSize: 12 }}>Usage</label>
<div className="code-block">
<pre style={{ whiteSpace: 'pre-wrap', margin: 0, fontSize: 12 }}>{`Authorization: Bearer ${token.token}`}</pre>
<button className="copy-btn" onClick={() => copy(`Authorization: Bearer ${token.token}`)}>Copy</button>
</div>
</div>
<div className="warning-bar">Save this token now. It will not be shown again.</div>
<div style={{ display: 'flex', gap: 12, justifyContent: 'flex-end', marginTop: 20 }}>
<button className="btn btn-primary" onClick={onClose}>Done</button>
</div>
</div>
</div>
);
}
function RegisterModal({ onClose, onRegistered }: {
onClose: () => void;
onRegistered: (creds: { clientId: string; clientSecret: string; name: string }) => void;
}) {
const [name, setName] = useState('');
// v0.28: scope set sourced from admin/src/lib/scope-constants.ts (mirror
// of src/core/scope.ts). CI drift check at scripts/check-admin-scope-drift.sh
// fails the build if these diverge.
const [scopes, setScopes] = useState<Record<Scope, boolean>>(() =>
Object.fromEntries(ALLOWED_SCOPES_LIST.map(s => [s, s === 'read'])) as Record<Scope, boolean>,
);
const [ttl, setTtl] = useState('86400'); // 24h default
const [loading, setLoading] = useState(false);
const [error, setError] = useState('');
const ttlOptions = [
{ label: '1 hour', value: '3600' },
{ label: '24 hours', value: '86400' },
{ label: '7 days', value: '604800' },
{ label: '30 days', value: '2592000' },
{ label: '1 year', value: '31536000' },
{ label: 'No expiry', value: '0' },
];
const handleSubmit = async (e: React.FormEvent) => {
e.preventDefault();
if (!name.trim()) { setError('Name required'); return; }
setLoading(true);
setError('');
try {
// Use the CLI registration endpoint (POST to admin API)
const selectedScopes = Object.entries(scopes).filter(([, v]) => v).map(([k]) => k).join(' ');
const res = await fetch('/admin/api/register-client', {
method: 'POST',
credentials: 'same-origin',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: name.trim(), scopes: selectedScopes, tokenTtl: ttl === '0' ? 315360000 : Number(ttl) }),
});
if (!res.ok) throw new Error('Registration failed');
const data = await res.json();
onRegistered({ clientId: data.clientId, clientSecret: data.clientSecret, name: name.trim() });
} catch (err) {
setError(err instanceof Error ? err.message : 'Registration failed');
} finally {
setLoading(false);
}
};
return (
<div className="modal-overlay" onClick={onClose}>
<form className="modal" onClick={e => e.stopPropagation()} onSubmit={handleSubmit}>
<div className="modal-title">Register Agent</div>
<div style={{ marginBottom: 16 }}>
<label>Agent Name</label>
<input placeholder="e.g. perplexity-production" value={name} onChange={e => setName(e.target.value)} autoFocus />
</div>
<div style={{ marginBottom: 16 }}>
<label>Scopes</label>
<div className="checkbox-group">
{ALLOWED_SCOPES_LIST.map(s => (
<label key={s} className="checkbox-label">
<input type="checkbox" checked={scopes[s]} onChange={e => setScopes(p => ({ ...p, [s]: e.target.checked }))} />
{s}
</label>
))}
</div>
</div>
<div style={{ marginBottom: 20 }}>
<label>Token Lifetime</label>
<select value={ttl} onChange={e => setTtl(e.target.value)}
style={{ width: '100%', background: 'var(--bg-secondary)', color: 'var(--text-primary)', border: '1px solid var(--border)', borderRadius: 6, padding: '6px 10px', fontSize: 14 }}>
{ttlOptions.map(o => <option key={o.value} value={o.value}>{o.label}</option>)}
</select>
</div>
{error && <div style={{ color: 'var(--error)', fontSize: 13, marginBottom: 12 }}>{error}</div>}
<div style={{ display: 'flex', gap: 12, justifyContent: 'flex-end' }}>
<button type="button" className="btn btn-secondary" onClick={onClose}>Cancel</button>
<button type="submit" className="btn btn-primary" disabled={loading}>
{loading ? 'Registering...' : 'Register'}
</button>
</div>
</form>
</div>
);
}
function CredentialsModal({ credentials, onClose }: {
credentials: { clientId: string; clientSecret: string; name: string };
onClose: () => void;
}) {
const copy = (text: string) => navigator.clipboard.writeText(text);
const downloadJson = () => {
const blob = new Blob([JSON.stringify(credentials, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url; a.download = `${credentials.name}-credentials.json`; a.click();
URL.revokeObjectURL(url);
};
return (
<div className="modal-overlay">
<div className="modal" style={{ maxWidth: 560 }}>
<div style={{ textAlign: 'center', marginBottom: 16 }}>
<div style={{ fontSize: 36, color: 'var(--success)', marginBottom: 8 }}>&#10003;</div>
<div style={{ fontSize: 20, fontWeight: 600 }}>Agent Registered</div>
</div>
<div style={{ marginBottom: 12 }}>
<label style={{ fontSize: 12 }}>Client ID</label>
<div className="code-block">
<span>{credentials.clientId}</span>
<button className="copy-btn" onClick={() => copy(credentials.clientId)}>Copy</button>
</div>
</div>
<div style={{ marginBottom: 12 }}>
<label style={{ fontSize: 12 }}>Client Secret</label>
<div className="code-block">
<span>{credentials.clientSecret}</span>
<button className="copy-btn" onClick={() => copy(credentials.clientSecret)}>Copy</button>
</div>
</div>
<div className="warning-bar">
Save this secret now. It will not be shown again.
</div>
<div style={{ display: 'flex', gap: 12, justifyContent: 'flex-end', marginTop: 20 }}>
<button className="btn btn-secondary" onClick={downloadJson}>Download as JSON</button>
<button className="btn btn-primary" onClick={onClose}>Done</button>
</div>
</div>
</div>
);
}
function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: () => void; onRevoked: () => void }) {
const [tab, setTab] = useState<'claude-code' | 'chatgpt' | 'claude-cowork' | 'perplexity' | 'cursor' | 'json'>('claude-code');
const copy = (text: string) => navigator.clipboard.writeText(text);
const serverUrl = window.location.origin;
const cid = agent.id || agent.client_id || '';
const isOAuth = agent.auth_type === 'oauth';
const agentName = agent.name || agent.client_name || 'unknown';
// For API keys, we can't show the actual token (it was shown once at creation).
// For OAuth, we show the client_id and tell them to use their secret.
const configSnippets: Record<string, string> = {
'claude-code': isOAuth
? [
`The user wants to connect GBrain MCP to your context. Here's how.`,
``,
`# Step 1 — read the client secret from the operator without putting it in shell history.`,
`# read -s suppresses the echo; export keeps it scoped to this shell session.`,
`read -rs -p "Client Secret: " GBRAIN_CS && echo`,
``,
`# Step 2 — mint a token and register GBrain in one go.`,
`claude mcp add --transport http gbrain ${serverUrl}/mcp \\`,
` --header "Authorization: Bearer $(curl -s -X POST ${serverUrl}/token \\`,
` -d 'grant_type=client_credentials' \\`,
` -d 'client_id=${cid}' \\`,
` --data-urlencode "client_secret=$GBRAIN_CS" \\`,
` -d 'scope=${agent.scope || 'read write'}' | jq -r .access_token)"`,
``,
`# Step 3 — clear the secret from this shell.`,
`unset GBRAIN_CS`,
``,
`The token will last ${agent.token_ttl ? (agent.token_ttl >= 86400 ? Math.floor(agent.token_ttl / 86400) + ' days' : Math.floor(agent.token_ttl / 3600) + ' hours') : '1 hour (default)'}.`,
``,
`─── Fallback: 2-step curl + paste ───`,
``,
`If your shell doesn't support read -s, mint the token first, then paste:`,
``,
`curl -s -X POST ${serverUrl}/token \\`,
` -d 'grant_type=client_credentials' \\`,
` -d 'client_id=${cid}' \\`,
` -d 'client_secret=PASTE_YOUR_CLIENT_SECRET_HERE' \\`,
` -d 'scope=${agent.scope || 'read write'}' | jq -r .access_token`,
``,
`claude mcp add --transport http gbrain ${serverUrl}/mcp \\`,
` --header "Authorization: Bearer PASTE_TOKEN_FROM_ABOVE"`,
``,
`# Then run: history -d $((HISTCMD-1)) # zsh: print -s -- '' && fc -p`,
].join('\n')
: [
`The user wants to connect GBrain MCP to your context. Here's how.`,
``,
`Run this command in terminal to register GBrain:`,
``,
`claude mcp add --transport http gbrain ${serverUrl}/mcp \\`,
` --header "Authorization: Bearer PASTE_YOUR_API_KEY_HERE"`,
``,
`Replace PASTE_YOUR_API_KEY_HERE with the API key shown when "${agentName}" was created.`,
`API keys never expire.`,
].join('\n'),
'chatgpt': [
`The user wants to connect GBrain MCP to ChatGPT.`,
``,
`1. Open ChatGPT > Settings > Tools & Integrations > Add MCP Server`,
`2. Enter this URL — ChatGPT will auto-discover the OAuth configuration:`,
``,
` ${serverUrl}/.well-known/oauth-authorization-server`,
``,
`3. When prompted for credentials:`,
` Client ID: ${cid}`,
` Client Secret: (the secret from agent registration)`,
` Grant Type: client_credentials`,
` Scope: ${agent.scope || 'read write'}`,
].join('\n'),
'claude-cowork': [
`The user wants to connect GBrain MCP to Claude.ai.`,
``,
`1. Open claude.ai > Settings > Connected Apps > Add MCP Server`,
`2. Server URL: ${serverUrl}/mcp`,
`3. When prompted for auth:`,
` Token endpoint: ${serverUrl}/token`,
` Client ID: ${cid}`,
` Client Secret: (the secret from agent registration)`,
` Scope: ${agent.scope || 'read write'}`,
``,
`Discovery URL: ${serverUrl}/.well-known/oauth-authorization-server`,
].join('\n'),
cursor: isOAuth
? [
`The user wants to connect GBrain MCP to Cursor.`,
``,
`Cursor supports OAuth for remote MCP. Add to .cursor/mcp.json:`,
``,
`{`,
` "mcpServers": {`,
` "gbrain": {`,
` "url": "${serverUrl}/mcp",`,
` "transport": "sse"`,
` }`,
` }`,
`}`,
``,
`Cursor will auto-discover OAuth via:`,
`${serverUrl}/.well-known/oauth-authorization-server`,
``,
`When prompted: Client ID ${cid}, use the secret from registration.`,
].join('\n')
: [
`The user wants to connect GBrain MCP to Cursor.`,
``,
`Add to .cursor/mcp.json:`,
``,
`{`,
` "mcpServers": {`,
` "gbrain": {`,
` "url": "${serverUrl}/mcp",`,
` "transport": "sse",`,
` "headers": {`,
` "Authorization": "Bearer PASTE_YOUR_API_KEY_HERE"`,
` }`,
` }`,
` }`,
`}`,
``,
`Replace PASTE_YOUR_API_KEY_HERE with the API key shown when "${agentName}" was created.`,
].join('\n'),
perplexity: [
`The user wants to connect GBrain MCP to Perplexity.`,
``,
`1. Go to Settings > Connectors > Add MCP`,
`2. Server URL: ${serverUrl}/mcp`,
`3. Client ID: ${cid}`,
`4. Client Secret: (the secret from agent registration)`,
].join('\n'),
json: JSON.stringify({
server_url: serverUrl + '/mcp',
token_url: serverUrl + '/token',
discovery_url: serverUrl + '/.well-known/oauth-authorization-server',
client_id: cid,
client_name: agentName,
auth_type: agent.auth_type,
scope: agent.scope,
}, null, 2),
};
return (
<>
<div className="drawer-overlay" onClick={onClose} />
<div className="drawer">
<button className="drawer-close" onClick={onClose}>&#10005;</button>
<div style={{ fontSize: 18, fontWeight: 600, marginBottom: 4 }}>{agent.name || agent.client_name}</div>
<span className={`badge ${agent.status === 'active' ? 'badge-success' : 'badge-danger'}`}>{agent.status}</span>
<div className="section-title">Details</div>
<div style={{ display: 'grid', gridTemplateColumns: '100px 1fr', gap: '6px 12px', fontSize: 13 }}>
<span style={{ color: 'var(--text-secondary)' }}>Client ID</span>
<span className="mono">{(agent.id || agent.id || agent.client_id || '').substring(0, 24)}...</span>
<span style={{ color: 'var(--text-secondary)' }}>Scopes</span>
<span>{(agent.scope || '').split(' ').filter(Boolean).map(s => (
<span key={s} className={`badge badge-${s}`} style={{ marginRight: 4 }}>{s}</span>
))}</span>
<span style={{ color: 'var(--text-secondary)' }}>Registered</span>
<span>{new Date(agent.created_at).toLocaleDateString()}</span>
<span style={{ color: 'var(--text-secondary)' }}>Token TTL</span>
<span>{agent.token_ttl ? (agent.token_ttl >= 31536000 ? 'No expiry' : agent.token_ttl >= 86400 ? `${Math.floor(agent.token_ttl / 86400)}d` : agent.token_ttl >= 3600 ? `${Math.floor(agent.token_ttl / 3600)}h` : `${agent.token_ttl}s`) : '1h (default)'}</span>
</div>
{/*
Config Export visible for both auth_type=oauth AND auth_type=api_key.
Claude Code + Cursor + JSON tabs render real snippets regardless
(commit 15's snippets are auth-type-aware for those two clients;
JSON is just structured metadata). ChatGPT, Claude.ai, and
Perplexity tabs render an "OAuth client required" message on
api_key agents those MCP clients only speak OAuth 2.0
client_credentials, not raw bearer tokens.
Pre-fix (Wintermute commit 16): the entire Config Export
section was hidden for api_key agents, dropping the working
Claude Code + Cursor snippets along with the broken ones.
(D5=C in the eng review.)
*/}
<div className="section-title">Config Export</div>
<div className="tabs" style={{ flexWrap: 'wrap' }}>
<div className={`tab ${tab === 'claude-code' ? 'active' : ''}`} onClick={() => setTab('claude-code')}>Claude Code</div>
<div className={`tab ${tab === 'chatgpt' ? 'active' : ''}`} onClick={() => setTab('chatgpt')}>ChatGPT</div>
<div className={`tab ${tab === 'claude-cowork' ? 'active' : ''}`} onClick={() => setTab('claude-cowork')}>Claude.ai</div>
<div className={`tab ${tab === 'cursor' ? 'active' : ''}`} onClick={() => setTab('cursor')}>Cursor</div>
<div className={`tab ${tab === 'perplexity' ? 'active' : ''}`} onClick={() => setTab('perplexity')}>Perplexity</div>
<div className={`tab ${tab === 'json' ? 'active' : ''}`} onClick={() => setTab('json')}>JSON</div>
</div>
{(() => {
const oauthOnlyTabs = new Set(['chatgpt', 'claude-cowork', 'perplexity']);
if (!isOAuth && oauthOnlyTabs.has(tab)) {
const clientName = { chatgpt: 'ChatGPT', 'claude-cowork': 'Claude.ai', perplexity: 'Perplexity' }[tab] || tab;
return (
<div style={{
background: 'rgba(255, 200, 100, 0.08)',
border: '1px solid rgba(255, 200, 100, 0.2)',
borderRadius: 8,
padding: '14px 16px',
marginTop: 12,
fontSize: 13,
lineHeight: 1.6,
color: 'var(--text-secondary)',
}}>
<div style={{ fontWeight: 600, color: 'var(--text-primary)', marginBottom: 6 }}>
{clientName} requires an OAuth client
</div>
{clientName} only supports OAuth 2.0 (client_credentials). API keys use raw bearer tokens, which {clientName} does not accept. Register a separate OAuth client and use that to connect this AI.
</div>
);
}
return (
<div className="code-block">
<pre style={{ whiteSpace: 'pre-wrap', margin: 0 }}>{configSnippets[tab]}</pre>
<button className="copy-btn" onClick={() => copy(configSnippets[tab])}>Copy</button>
</div>
);
})()}
<div style={{ marginTop: 32 }}>
{agent.status === 'active' && (
<button className="btn btn-danger" onClick={async () => {
if (!confirm(`Revoke ${agent.name || agent.client_name}? All active tokens will be invalidated.`)) return;
try {
if (agent.auth_type === 'oauth') {
await api.revokeClient(agent.id || agent.client_id || '');
} else {
await api.revokeApiKey(agent.name || '');
}
onRevoked();
onClose();
} catch (e) {
alert('Revoke failed: ' + (e instanceof Error ? e.message : 'unknown error'));
}
}}>Revoke Agent</button>
)}
{agent.status === 'revoked' && (
<span style={{ color: 'var(--text-muted)', fontSize: 13 }}>This agent has been revoked.</span>
)}
</div>
</div>
</>
);
}
-174
View File
@@ -1,174 +0,0 @@
/**
* v0.36.1.0 (T15 / E6) Calibration tab.
*
* Fetches the active calibration profile + 4 server-rendered SVG charts.
* Layout: Linear calm clarity (per D23 mockup variant-B) single column,
* generous whitespace, ONE big sparkline as hero, then patterns, then
* domain bars, then abandoned threads.
*
* Per D23 SVG markup comes from the server (image/svg+xml endpoint).
* Admin SPA renders inside a TrustedSVG wrapper that uses
* dangerouslySetInnerHTML. XSS posture: server-side escapeXml() on all
* caller-controlled strings + requireAdmin middleware on the endpoint.
*/
import React, { useEffect, useState } from 'react';
import { api } from '../api';
interface CalibrationProfileSummary {
holder: string;
source_id: string;
generated_at: string;
published: boolean;
total_resolved: number;
brier: number | null;
accuracy: number | null;
partial_rate: number | null;
grade_completion: number;
pattern_statements: string[];
active_bias_tags: string[];
voice_gate_passed: boolean;
voice_gate_attempts: number;
}
interface ChartSvgProps {
type: string;
ariaLabel: string;
}
function TrustedSVG({ markup }: { markup: string }) {
return (
<div
style={{ width: '100%', overflow: 'auto' }}
// Server-rendered SVG (image/svg+xml) gated by requireAdmin middleware.
// All caller-controlled strings pass through escapeXml() server-side.
dangerouslySetInnerHTML={{ __html: markup }}
/>
);
}
function ChartSvg({ type, ariaLabel }: ChartSvgProps) {
const [markup, setMarkup] = useState<string>('');
const [error, setError] = useState<string>('');
useEffect(() => {
let cancelled = false;
api
.calibrationChart(type)
.then(svg => {
if (!cancelled) setMarkup(svg);
})
.catch(err => {
if (!cancelled) setError(err.message ?? 'fetch failed');
});
return () => {
cancelled = true;
};
}, [type]);
if (error) {
return (
<div style={{ padding: 16, color: 'var(--error)' }} role="alert">
{ariaLabel}: {error}
</div>
);
}
if (!markup) {
return <div style={{ padding: 16, color: 'var(--text-muted)' }}>{ariaLabel} loading...</div>;
}
return <TrustedSVG markup={markup} />;
}
export function CalibrationPage() {
const [profile, setProfile] = useState<CalibrationProfileSummary | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string>('');
useEffect(() => {
api
.calibrationProfile()
.then(p => {
setProfile(p);
setLoading(false);
})
.catch(err => {
setError(err.message ?? 'fetch failed');
setLoading(false);
});
}, []);
if (loading) {
return <div style={{ padding: 24, color: 'var(--text-secondary)' }}>Loading calibration profile</div>;
}
if (error) {
return (
<div style={{ padding: 24, color: 'var(--error)' }} role="alert">
Could not load calibration profile: {error}
</div>
);
}
if (!profile) {
return (
<div style={{ padding: 24, maxWidth: 700 }}>
<h1 style={{ marginBottom: 16 }}>Calibration</h1>
<p style={{ color: 'var(--text-secondary)' }}>
No calibration profile yet. Builds after 5+ resolved takes.
</p>
<pre
style={{
background: 'var(--bg-secondary)',
padding: 12,
borderRadius: 4,
color: 'var(--text-primary)',
marginTop: 12,
fontFamily: 'var(--font-mono)',
}}
>
gbrain dream --phase calibration_profile
</pre>
</div>
);
}
const generated = new Date(profile.generated_at);
const generatedAgo = Math.floor((Date.now() - generated.getTime()) / (1000 * 60 * 60 * 24));
return (
<div style={{ padding: 32, maxWidth: 720 }}>
<h1 style={{ marginBottom: 8 }}>Calibration</h1>
<div style={{ color: 'var(--text-muted)', fontSize: 13, marginBottom: 24 }}>
Holder: {profile.holder}
{' · '}
Updated {generatedAgo === 0 ? 'today' : `${generatedAgo}d ago`}
{profile.published && ' · published'}
{profile.grade_completion < 0.9 && ` · ~${Math.round(profile.grade_completion * 100)}% graded`}
{!profile.voice_gate_passed && ' · voice gate fell back to template'}
</div>
<section style={{ marginBottom: 32 }}>
<ChartSvg type="brier-trend" ariaLabel="Brier trend" />
</section>
<section style={{ marginBottom: 32 }}>
<h2 style={{ fontSize: 14, color: 'var(--text-secondary)', marginBottom: 12, fontWeight: 400 }}>
Pattern statements
</h2>
<ChartSvg type="pattern-statements" ariaLabel="Pattern statements" />
</section>
<section style={{ marginBottom: 32 }}>
<ChartSvg type="domain-bars" ariaLabel="Per-domain accuracy" />
</section>
<section style={{ marginBottom: 32 }}>
<ChartSvg type="abandoned-threads" ariaLabel="Abandoned threads" />
</section>
{profile.active_bias_tags.length > 0 && (
<section style={{ marginBottom: 32, color: 'var(--text-muted)', fontSize: 13 }}>
Active bias tags: {profile.active_bias_tags.join(', ')}
</section>
)}
</div>
);
}
-137
View File
@@ -1,137 +0,0 @@
import React, { useState, useEffect, useRef } from 'react';
import { api } from '../api';
interface FeedEvent {
agent: string;
operation: string;
scopes: string;
latency_ms: number;
status: string;
timestamp: string;
}
export function DashboardPage() {
const [stats, setStats] = useState({ connected_agents: 0, requests_today: 0, active_tokens: 0 });
const [health, setHealth] = useState({ expiring_soon: 0, error_rate: '0%' });
const [events, setEvents] = useState<FeedEvent[]>([]);
const [sseStatus, setSseStatus] = useState<'connecting' | 'connected' | 'disconnected'>('connecting');
const eventSourceRef = useRef<EventSource | null>(null);
useEffect(() => {
api.stats().then(setStats).catch(() => {});
api.health().then(setHealth).catch(() => {});
const es = new EventSource('/admin/events');
eventSourceRef.current = es;
es.onopen = () => setSseStatus('connected');
es.onmessage = (e) => {
try {
const event = JSON.parse(e.data) as FeedEvent;
setEvents(prev => [event, ...prev].slice(0, 50));
} catch {}
};
es.onerror = () => {
setSseStatus('disconnected');
setTimeout(() => {
setSseStatus('connecting');
es.close();
// Reconnect handled by browser EventSource auto-retry
}, 3000);
};
const interval = setInterval(() => {
api.stats().then(setStats).catch(() => {});
api.health().then(setHealth).catch(() => {});
}, 30000);
return () => { es.close(); clearInterval(interval); };
}, []);
const timeAgo = (ts: string) => {
const diff = Date.now() - new Date(ts).getTime();
if (diff < 60000) return `${Math.floor(diff / 1000)}s ago`;
if (diff < 3600000) return `${Math.floor(diff / 60000)} min ago`;
return `${Math.floor(diff / 3600000)}h ago`;
};
return (
<>
<h1 className="page-title">Dashboard</h1>
<div style={{ display: 'flex', gap: 24 }}>
<div style={{ flex: 1 }}>
<div className="metrics">
<div className="metric">
<div className="metric-value">{stats.connected_agents}</div>
<div className="metric-label">Connected Agents</div>
</div>
<div className="metric">
<div className="metric-value">{stats.requests_today}</div>
<div className="metric-label">Requests Today</div>
</div>
<div className="metric">
<div className="metric-value">{stats.active_tokens}</div>
<div className="metric-label">Active Tokens</div>
</div>
</div>
<h2 className="section-title">
Live Activity
<span style={{ marginLeft: 8, fontSize: 10, color: sseStatus === 'connected' ? 'var(--success)' : sseStatus === 'connecting' ? 'var(--warning)' : 'var(--error)' }}>
{sseStatus === 'connected' ? '● connected' : sseStatus === 'connecting' ? '● connecting...' : '● disconnected'}
</span>
</h2>
<div className="feed">
{events.length === 0 ? (
<div className="feed-empty">
{sseStatus === 'connected' ? 'No requests yet. Agents will appear when they connect.' : 'Connecting...'}
</div>
) : (
<table>
<thead>
<tr>
<th>Agent</th>
<th>Operation</th>
<th>Scopes</th>
<th>Latency</th>
<th>Status</th>
<th>Time</th>
</tr>
</thead>
<tbody>
{events.map((e, i) => (
<tr key={i}>
<td className="mono">{e.agent}</td>
<td className="mono">{e.operation}</td>
<td>{e.scopes.split(',').map(s => (
<span key={s} className={`badge badge-${s.trim()}`} style={{ marginRight: 4 }}>{s.trim()}</span>
))}</td>
<td className="mono">{e.latency_ms} ms</td>
<td><span className={`badge badge-${e.status}`}>{e.status}</span></td>
<td style={{ color: 'var(--text-secondary)' }}>{timeAgo(e.timestamp)}</td>
</tr>
))}
</tbody>
</table>
)}
</div>
</div>
<div style={{ width: 220 }}>
<h2 className="section-title">Token Health</h2>
<div className="health-panel">
<div className="health-row">
<span style={{ color: 'var(--warning)' }}>Expiring Soon</span>
<span className="mono">{health.expiring_soon}</span>
</div>
<div className="health-row">
<span style={{ color: 'var(--error)' }}>Error Rate</span>
<span className="mono">{health.error_rate}</span>
</div>
</div>
</div>
</div>
</>
);
}
-174
View File
@@ -1,174 +0,0 @@
import React, { useEffect, useState } from 'react';
import { api } from '../api';
/**
* v0.41 D2 live jobs dashboard. Browser counterpart to the TTY
* `gbrain jobs watch` command. Polls `/admin/api/jobs/watch` every
* 1s (matches TTY refresh cadence; SSE upgrade is a v0.42 follow-up
* once the same wiring lands in serve-http for the TTY command).
*
* Layout intentionally matches the TTY 1:1 so an operator looking at
* both surfaces sees the same panels in the same order.
*/
interface WatchSnapshot {
ts_ms: number;
by_type: Array<{ name: string; total: number; completed: number; failed: number; dead: number }>;
queue_health: { waiting: number; active: number; stalled: number };
lease_pressure_1h: number;
top_errors: Array<{ cluster: string; count: number }>;
budget_owners: Array<{ owner_id: number; remaining_cents: number; total_spent_cents: number }>;
}
function leasePressureColor(n: number): string {
if (n === 0) return 'var(--accent-success, #2ea043)';
if (n >= 100) return 'var(--accent-danger, #f85149)';
return 'var(--accent-warn, #d29922)';
}
function dollars(cents: number): string {
return `$${(cents / 100).toFixed(2)}`;
}
export function JobsWatchPage() {
const [snap, setSnap] = useState<WatchSnapshot | null>(null);
const [err, setErr] = useState<string | null>(null);
useEffect(() => {
let alive = true;
let timer: ReturnType<typeof setTimeout> | null = null;
const tick = async () => {
try {
const data = await api.jobsWatch();
if (alive) {
setSnap(data);
setErr(null);
}
} catch (e) {
if (alive) setErr(e instanceof Error ? e.message : String(e));
}
if (alive) timer = setTimeout(tick, 1000);
};
tick();
return () => {
alive = false;
if (timer) clearTimeout(timer);
};
}, []);
if (err) {
return (
<div style={{ padding: 24, color: 'var(--accent-danger, #f85149)' }}>
<h2>Jobs Watch error</h2>
<pre style={{ whiteSpace: 'pre-wrap' }}>{err}</pre>
</div>
);
}
if (!snap) {
return <div style={{ padding: 24, color: 'var(--text-muted, #777)' }}>Loading jobs watch</div>;
}
const ts = new Date(snap.ts_ms).toLocaleTimeString();
return (
<div style={{ padding: 24, fontFamily: 'var(--font-mono, "JetBrains Mono", monospace)' }}>
<h1 style={{ fontSize: 18, marginBottom: 4 }}>
Jobs Watch
<span style={{ marginLeft: 12, color: 'var(--text-muted, #777)', fontSize: 12, fontWeight: 'normal' }}>
updated {ts}
</span>
</h1>
<section style={{ marginTop: 24 }}>
<h2 style={{ fontSize: 14, marginBottom: 8 }}>Queue</h2>
<div>
waiting=<b>{snap.queue_health.waiting}</b>{' '}
active=<b>{snap.queue_health.active}</b>{' '}
stalled=<b style={{ color: snap.queue_health.stalled > 0 ? 'var(--accent-warn, #d29922)' : undefined }}>
{snap.queue_health.stalled}
</b>
</div>
</section>
{snap.by_type.length > 0 && (
<section style={{ marginTop: 24 }}>
<h2 style={{ fontSize: 14, marginBottom: 8 }}>By type (24h)</h2>
<table style={{ borderCollapse: 'collapse' }}>
<thead>
<tr style={{ color: 'var(--text-muted, #777)', fontSize: 12 }}>
<th style={{ textAlign: 'left', padding: '4px 12px 4px 0' }}>name</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>total</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>done</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>fail</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>dead</th>
</tr>
</thead>
<tbody>
{snap.by_type.slice(0, 6).map(t => (
<tr key={t.name}>
<td style={{ padding: '4px 12px 4px 0' }}>{t.name}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{t.total}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{t.completed}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{t.failed}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{t.dead}</td>
</tr>
))}
</tbody>
</table>
</section>
)}
<section style={{ marginTop: 24 }}>
<h2 style={{ fontSize: 14, marginBottom: 8 }}>Lease pressure (1h)</h2>
<div style={{ color: leasePressureColor(snap.lease_pressure_1h) }}>
{snap.lease_pressure_1h} bounce{snap.lease_pressure_1h === 1 ? '' : 's'}
</div>
</section>
{snap.top_errors.length > 0 && (
<section style={{ marginTop: 24 }}>
<h2 style={{ fontSize: 14, marginBottom: 8 }}>Top errors (24h)</h2>
<table style={{ borderCollapse: 'collapse' }}>
<tbody>
{snap.top_errors.slice(0, 5).map(e => (
<tr key={e.cluster}>
<td style={{ textAlign: 'right', padding: '4px 12px 4px 0', color: 'var(--text-muted, #777)' }}>
{e.count}×
</td>
<td style={{ padding: '4px 12px 4px 0' }}>{e.cluster}</td>
</tr>
))}
</tbody>
</table>
</section>
)}
{snap.budget_owners.length > 0 && (
<section style={{ marginTop: 24 }}>
<h2 style={{ fontSize: 14, marginBottom: 8 }}>Budget owners</h2>
<table style={{ borderCollapse: 'collapse' }}>
<thead>
<tr style={{ color: 'var(--text-muted, #777)', fontSize: 12 }}>
<th style={{ textAlign: 'left', padding: '4px 12px 4px 0' }}>owner</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>spent</th>
<th style={{ textAlign: 'right', padding: '4px 12px' }}>remaining</th>
</tr>
</thead>
<tbody>
{snap.budget_owners.slice(0, 5).map(b => (
<tr key={b.owner_id}>
<td style={{ padding: '4px 12px 4px 0' }}>{b.owner_id}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{dollars(b.total_spent_cents)}</td>
<td style={{ textAlign: 'right', padding: '4px 12px' }}>{dollars(b.remaining_cents)}</td>
</tr>
))}
</tbody>
</table>
</section>
)}
</div>
);
}
-96
View File
@@ -1,96 +0,0 @@
import React, { useState } from 'react';
import { api } from '../api';
// v0.26.3 trust model (D11 + D12):
// - The bootstrap token is NEVER stored in browser JS state. No
// localStorage, no sessionStorage, no React state beyond the form
// submit cycle. After successful POST /admin/login the operator's
// token only lives in the HttpOnly cookie that the server set.
// - Magic-link URLs use single-use server-issued nonces, not the
// bootstrap token itself (see /admin/api/issue-magic-link). The
// bootstrap token never appears in a URL.
// - Closing the tab ends the session client-side. Reopening the
// dashboard 401s and shows this page again. Operator asks the agent
// for a fresh magic link or pastes the bootstrap token from the
// server's terminal scrollback.
export function LoginPage({ onLogin }: { onLogin: () => void }) {
const [token, setToken] = useState('');
const [error, setError] = useState('');
const [loading, setLoading] = useState(false);
const handleSubmit = async (e: React.FormEvent) => {
e.preventDefault();
setError('');
setLoading(true);
try {
await api.login(token);
// Don't persist the token. The HttpOnly cookie is the only
// session credential after this point.
setToken('');
onLogin();
} catch (err) {
setError('Invalid token.');
} finally {
setLoading(false);
}
};
return (
<div className="login-page">
<div className="login-box">
<div className="login-logo">GBrain</div>
<div style={{
background: 'rgba(136, 170, 255, 0.08)',
border: '1px solid rgba(136, 170, 255, 0.2)',
borderRadius: 8,
padding: '14px 16px',
marginBottom: 20,
fontSize: 13,
lineHeight: 1.5,
color: 'var(--text-secondary)',
}}>
<div style={{ fontWeight: 600, color: 'var(--text-primary)', marginBottom: 6 }}>
🔒 This is a protected dashboard
</div>
Ask your AI agent for the admin login link:
<div style={{
background: 'rgba(0,0,0,0.3)',
borderRadius: 6,
padding: '8px 12px',
marginTop: 8,
fontFamily: 'var(--font-mono)',
fontSize: 12,
color: '#88aaff',
wordBreak: 'break-all',
}}>
"Give me the GBrain admin login link"
</div>
<div style={{ marginTop: 8, fontSize: 12, color: 'var(--text-muted)' }}>
Each link is single-use. Your agent generates a fresh one each time.
</div>
</div>
<details style={{ marginBottom: 16 }}>
<summary style={{ cursor: 'pointer', fontSize: 13, color: 'var(--text-muted)' }}>
Or paste bootstrap token manually
</summary>
<form onSubmit={handleSubmit} style={{ marginTop: 12 }}>
<div style={{ marginBottom: 12 }}>
<input
type="password"
placeholder="Admin Token"
value={token}
onChange={e => setToken(e.target.value)}
/>
</div>
<button className="btn btn-primary" style={{ width: '100%' }} disabled={loading}>
{loading ? 'Authenticating...' : 'Submit'}
</button>
{error && <div className="login-error">{error}</div>}
</form>
</details>
</div>
</div>
);
}
-150
View File
@@ -1,150 +0,0 @@
import React, { useState, useEffect } from 'react';
import { api } from '../api';
interface LogEntry {
id: number;
token_name: string;
agent_name: string;
operation: string;
latency_ms: number;
status: string;
params: Record<string, unknown> | null;
error_message: string | null;
created_at: string;
}
export function RequestLogPage() {
const [data, setData] = useState<{ rows: LogEntry[]; total: number; page: number; pages: number }>({
rows: [], total: 0, page: 1, pages: 1,
});
const [page, setPage] = useState(1);
const [agentFilter, setAgentFilter] = useState('all');
const [expandedRow, setExpandedRow] = useState<number | null>(null);
useEffect(() => { loadPage(page); }, [page, agentFilter]);
const loadPage = (p: number) => {
const qs = agentFilter !== 'all' ? `&agent=${encodeURIComponent(agentFilter)}` : '';
api.requests(p, qs).then(setData).catch(() => {});
};
const timeAgo = (ts: string) => {
const diff = Date.now() - new Date(ts).getTime();
if (diff < 60000) return `${Math.floor(diff / 1000)}s ago`;
if (diff < 3600000) return `${Math.floor(diff / 60000)} min ago`;
if (diff < 86400000) return `${Math.floor(diff / 3600000)}h ago`;
return new Date(ts).toLocaleDateString();
};
const formatParams = (params: Record<string, unknown> | null) => {
if (!params) return null;
const { query, slug, partial, limit, ...rest } = params as any;
const parts: string[] = [];
if (query) parts.push(`"${query}"`);
if (slug) parts.push(slug);
if (partial) parts.push(`~${partial}`);
if (limit) parts.push(`limit=${limit}`);
if (Object.keys(rest).length > 0) parts.push(`+${Object.keys(rest).length} params`);
return parts.join(' ');
};
// Collect unique agents for filter (use name for display, token_name for value)
const agentMap = new Map<string, string>();
data.rows.forEach(r => { if (r.token_name) agentMap.set(r.token_name, r.agent_name || r.token_name); });
return (
<>
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: 24 }}>
<h1 className="page-title" style={{ marginBottom: 0 }}>Request Log</h1>
<select value={agentFilter} onChange={e => { setAgentFilter(e.target.value); setPage(1); }}
style={{ background: 'var(--bg-secondary)', color: 'var(--text-primary)', border: '1px solid var(--border)', borderRadius: 6, padding: '4px 8px', fontSize: 13 }}>
<option value="all">All agents</option>
{[...agentMap.entries()].map(([id, name]) => <option key={id} value={id}>{name}</option>)}
</select>
</div>
{data.rows.length === 0 ? (
<div style={{ textAlign: 'center', padding: 48, color: 'var(--text-muted)' }}>
No requests yet.
</div>
) : (
<>
<table>
<thead>
<tr>
<th>Time</th>
<th>Agent</th>
<th>Operation</th>
<th>Params</th>
<th>Latency</th>
<th>Status</th>
</tr>
</thead>
<tbody>
{data.rows.map(r => (
<React.Fragment key={r.id}>
<tr onClick={() => setExpandedRow(expandedRow === r.id ? null : r.id)}
style={{ cursor: 'pointer' }}>
<td style={{ color: 'var(--text-secondary)', whiteSpace: 'nowrap' }}>{timeAgo(r.created_at)}</td>
<td>
<a style={{ color: 'var(--text-link, #88aaff)', cursor: 'pointer', textDecoration: 'none', fontWeight: 500 }}
onClick={(e) => { e.stopPropagation(); setAgentFilter(r.token_name); setPage(1); }}>
{r.agent_name || r.token_name}
</a>
</td>
<td className="mono">{r.operation}</td>
<td style={{ color: 'var(--text-secondary)', fontSize: 12, maxWidth: 200, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>
{formatParams(r.params)}
</td>
<td className="mono">{r.latency_ms}ms</td>
<td><span className={`badge badge-${r.status}`}>{r.status}</span></td>
</tr>
{expandedRow === r.id && (
<tr>
<td colSpan={6} style={{ background: 'var(--bg-secondary, #0f0f1a)', padding: 16 }}>
<div style={{ display: 'grid', gridTemplateColumns: '100px 1fr', gap: '6px 12px', fontSize: 13 }}>
<span style={{ color: 'var(--text-muted)' }}>Time</span>
<span>{new Date(r.created_at).toLocaleString()}</span>
<span style={{ color: 'var(--text-muted)' }}>Agent</span>
<span className="mono">{r.token_name}</span>
<span style={{ color: 'var(--text-muted)' }}>Operation</span>
<span className="mono">{r.operation}</span>
<span style={{ color: 'var(--text-muted)' }}>Latency</span>
<span>{r.latency_ms}ms</span>
{r.params && (
<>
<span style={{ color: 'var(--text-muted)' }}>Params</span>
<pre className="mono" style={{ margin: 0, whiteSpace: 'pre-wrap', fontSize: 12 }}>
{JSON.stringify(r.params, null, 2)}
</pre>
</>
)}
{r.error_message && (
<>
<span style={{ color: 'var(--error, #ff6b6b)' }}>Error</span>
<span style={{ color: 'var(--error, #ff6b6b)' }}>{r.error_message}</span>
</>
)}
</div>
</td>
</tr>
)}
</React.Fragment>
))}
</tbody>
</table>
<div className="pagination">
<span>Page {data.page} of {data.pages} ({data.total} total)</span>
<div style={{ display: 'flex', gap: 8 }}>
<button disabled={data.page <= 1} onClick={() => setPage(p => p - 1)}>Previous</button>
<button disabled={data.page >= data.pages} onClick={() => setPage(p => p + 1)}>Next</button>
</div>
</div>
</>
)}
</>
);
}
-1
View File
@@ -1 +0,0 @@
/// <reference types="vite/client" />
-17
View File
@@ -1,17 +0,0 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"isolatedModules": true,
"moduleDetection": "force",
"noEmit": true,
"jsx": "react-jsx",
"strict": true
},
"include": ["src"]
}
-11
View File
@@ -1,11 +0,0 @@
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';
export default defineConfig({
plugins: [react()],
base: '/admin/',
build: {
outDir: 'dist',
emptyOutDir: true,
},
});
+4 -104
View File
@@ -5,44 +5,22 @@
"": {
"name": "gbrain",
"dependencies": {
"@ai-sdk/anthropic": "^3.0.71",
"@ai-sdk/google": "^3.0.64",
"@ai-sdk/openai": "^3.0.53",
"@ai-sdk/openai-compatible": "^2.0.41",
"@anthropic-ai/sdk": "^0.30.0",
"@aws-sdk/client-s3": "^3.1028.0",
"@dqbd/tiktoken": "^1.0.22",
"@electric-sql/pglite": "0.4.3",
"@jsquash/avif": "^2.1.1",
"@jsquash/png": "^3.1.1",
"@modelcontextprotocol/sdk": "1.29.0",
"ai": "^6.0.168",
"chokidar": "^4.0.3",
"cookie-parser": "^1.4.7",
"cors": "^2.8.5",
"eventsource-parser": "^3.0.8",
"exifr": "^7.1.3",
"express": "^5.1.0",
"express-rate-limit": "^7.5.0",
"@modelcontextprotocol/sdk": "^1.0.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
"tree-sitter-wasms": "0.1.13",
"web-tree-sitter": "0.22.6",
"zod": "^4.3.6",
},
"devDependencies": {
"@types/bun": "latest",
"@types/cookie-parser": "^1.4.7",
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"@types/js-yaml": "^3.12.10",
"bun-types": "^1.3.13",
"fast-check": "^4.8.0",
"typescript": "^5.6.0",
},
},
@@ -51,20 +29,6 @@
"@electric-sql/pglite",
],
"packages": {
"@ai-sdk/anthropic": ["@ai-sdk/anthropic@3.0.74", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-Xew9rfz9WWhDSyF8rNhjT/XWOWelNfJrMlmG0Ahw210hStisRpQZ1s+7VeI9JTJOZ5y5tXqBi5kfPwYnCfyRTA=="],
"@ai-sdk/gateway": ["@ai-sdk/gateway@3.0.109", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26", "@vercel/oidc": "3.2.0" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-r6dOqThjODp1vOhGRJg2OCmyB/ZOQtGx1esZ2SDvwDX5XoX8dBqYaYjLg8MPXTzMGJSgOkJyCxWgUcZtAl16pw=="],
"@ai-sdk/google": ["@ai-sdk/google@3.0.67", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-Qeq+SidYtzMrcf0fdw3L0QLmtXK+ErwdBzbxS4+0Q/2UP85Ges8RJJcbAj7SO8e2JbeJoM35BLqkeNy1o3wJvQ=="],
"@ai-sdk/openai": ["@ai-sdk/openai@3.0.58", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-2+5xGMROmrBboJuoOwqLL3b/o3i56+NRdxXDNVAiTyYjLiBj6KzembeuyuBT217be1X+zkEfAqD1H0irJlGIyw=="],
"@ai-sdk/openai-compatible": ["@ai-sdk/openai-compatible@2.0.45", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-5YBvurNL7Oj7mT3srws4Rh4cQidoorfEGObAOb5jV40eld8IC7EkXWARZjnWYqgYzabUs6Sn6muiXfQVkgOyOQ=="],
"@ai-sdk/provider": ["@ai-sdk/provider@3.0.10", "", { "dependencies": { "json-schema": "^0.4.0" } }, "sha512-Q3BZ27qfpYqnCYGvE3vt+Qi6LGOF9R5Nmzn+9JoM1lCRsD9mYaIhfJLkSunN48nfGXJ6n+XNV0J/XVpqGQl7Dw=="],
"@ai-sdk/provider-utils": ["@ai-sdk/provider-utils@4.0.26", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@standard-schema/spec": "^1.1.0", "eventsource-parser": "^3.0.8" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-CsKNLKsOpvPujRlIYvoz+Ybw+kGn7J4/fIZa/58+R7iWLLfwn6ifE2G6Yq8K9XvH/I/3bzaDAJ3NhRwEMsLBKQ=="],
"@anthropic-ai/sdk": ["@anthropic-ai/sdk@0.30.1", "", { "dependencies": { "@types/node": "^18.11.18", "@types/node-fetch": "^2.6.4", "abort-controller": "^3.0.0", "agentkeepalive": "^4.2.1", "form-data-encoder": "1.7.2", "formdata-node": "^4.3.2", "node-fetch": "^2.6.7" } }, "sha512-nuKvp7wOIz6BFei8WrTdhmSsx5mwnArYyJgh4+vYu3V4J0Ltb8Xm3odPm51n1aSI0XxNCrDl7O88cxCtUdAkaw=="],
"@aws-crypto/crc32": ["@aws-crypto/crc32@5.2.0", "", { "dependencies": { "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "tslib": "^2.6.2" } }, "sha512-nLbCWqQNgUiwwtFsen1AdzAtvuLRsQS8rYgMuxCrdKf9kOssamGLuPwyTY9wyYblNr9+1XM8v6zoDTPPSIeANg=="],
@@ -153,14 +117,8 @@
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
"@jsquash/png": ["@jsquash/png@3.1.1", "", {}, "sha512-C10pc+0H6j0h8fENOfnGOvkXCmvpSQTDGlfGd0sHphZhPSGTyLjIrHba0FaZZdsKqA/wlmhYicUHb92vfZphaw=="],
"@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="],
"@opentelemetry/api": ["@opentelemetry/api@1.9.0", "", {}, "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg=="],
"@smithy/chunked-blob-reader": ["@smithy/chunked-blob-reader@5.2.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-St+kVicSyayWQca+I1rGitaOEH6uKgE8IUWoYnnEX26SWdWQcL6LvMSD19Lg+vYHKdT9B2Zuu7rd3i6Wnyb/iw=="],
"@smithy/chunked-blob-reader-native": ["@smithy/chunked-blob-reader-native@4.2.3", "", { "dependencies": { "@smithy/util-base64": "^4.3.2", "tslib": "^2.6.2" } }, "sha512-jA5k5Udn7Y5717L86h4EIv06wIr3xn8GM1qHRi/Nf31annXcXHJjBKvgztnbn2TxH3xWrPBfgwHsOwZf0UmQWw=="],
@@ -261,48 +219,18 @@
"@smithy/uuid": ["@smithy/uuid@1.1.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-O/IEdcCUKkubz60tFbGA7ceITTAJsty+lBjNoorP4Z6XRqaFb/OjQjZODophEcuq68nKm6/0r+6/lLQ+XVpk8g=="],
"@standard-schema/spec": ["@standard-schema/spec@1.1.0", "", {}, "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w=="],
"@types/body-parser": ["@types/body-parser@1.19.6", "", { "dependencies": { "@types/connect": "*", "@types/node": "*" } }, "sha512-HLFeCYgz89uk22N5Qg3dvGvsv46B8GLvKKo1zKG4NybA8U2DiEO3w9lqGg29t/tfLRJpJ6iQxnVw4OnB7MoM9g=="],
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
"@types/connect": ["@types/connect@3.4.38", "", { "dependencies": { "@types/node": "*" } }, "sha512-K6uROf1LD88uDQqJCktA4yzL1YYAK6NgfsI0v/mTgyPKWsX1CnJ0XPSDhViejru1GcRkLWb8RlzFYJRqGUbaug=="],
"@types/cookie-parser": ["@types/cookie-parser@1.4.10", "", { "peerDependencies": { "@types/express": "*" } }, "sha512-B4xqkqfZ8Wek+rCOeRxsjMS9OgvzebEzzLYw7NHYuvzb7IdxOkI0ZHGgeEBX4PUM7QGVvNSK60T3OvWj3YfBRg=="],
"@types/cors": ["@types/cors@2.8.19", "", { "dependencies": { "@types/node": "*" } }, "sha512-mFNylyeyqN93lfe/9CSxOGREz8cpzAhH+E93xJ4xWQf62V8sQ/24reV2nyzUWM6H6Xji+GGHpkbLe7pVoUEskg=="],
"@types/express": ["@types/express@5.0.6", "", { "dependencies": { "@types/body-parser": "*", "@types/express-serve-static-core": "^5.0.0", "@types/serve-static": "^2" } }, "sha512-sKYVuV7Sv9fbPIt/442koC7+IIwK5olP1KWeD88e/idgoJqDm3JV/YUiPwkoKK92ylff2MGxSz1CSjsXelx0YA=="],
"@types/express-serve-static-core": ["@types/express-serve-static-core@5.1.1", "", { "dependencies": { "@types/node": "*", "@types/qs": "*", "@types/range-parser": "*", "@types/send": "*" } }, "sha512-v4zIMr/cX7/d2BpAEX3KNKL/JrT1s43s96lLvvdTmza1oEvDudCqK9aF/djc/SWgy8Yh0h30TZx5VpzqFCxk5A=="],
"@types/http-errors": ["@types/http-errors@2.0.5", "", {}, "sha512-r8Tayk8HJnX0FztbZN7oVqGccWgw98T/0neJphO91KkmOzug1KkofZURD4UaD5uH8AqcFLfdPErnBod0u71/qg=="],
"@types/js-yaml": ["@types/js-yaml@3.12.10", "", {}, "sha512-/Mtaq/wf+HxXpvhzFYzrzCqNRcA958sW++7JOFC8nPrZcvfi/TrzOaaGbvt27ltJB2NQbHVAg5a1wUCsyMH7NA=="],
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
"@types/qs": ["@types/qs@6.15.0", "", {}, "sha512-JawvT8iBVWpzTrz3EGw9BTQFg3BQNmwERdKE22vlTxawwtbyUSlMppvZYKLZzB5zgACXdXxbD3m1bXaMqP/9ow=="],
"@types/range-parser": ["@types/range-parser@1.2.7", "", {}, "sha512-hKormJbkJqzQGhziax5PItDUTMAM9uE2XXQmM37dyd4hVM+5aVl7oVxMVUiVQn2oCQFN/LKCZdvSM0pFRqbSmQ=="],
"@types/send": ["@types/send@1.2.1", "", { "dependencies": { "@types/node": "*" } }, "sha512-arsCikDvlU99zl1g69TcAB3mzZPpxgw0UQnaHeC1Nwb015xp8bknZv5rIfri9xTOcMuaVgvabfIRA7PSZVuZIQ=="],
"@types/serve-static": ["@types/serve-static@2.2.0", "", { "dependencies": { "@types/http-errors": "*", "@types/node": "*" } }, "sha512-8mam4H1NHLtu7nmtalF7eyBH14QyOASmcxHhSfEoRyr0nP/YdoesEtU+uSRvMe96TW/HPTtkoKqQLl53N7UXMQ=="],
"@vercel/oidc": ["@vercel/oidc@3.2.0", "", {}, "sha512-UycprH3T6n3jH0k44NHMa7pnFHGu/N05MjojYr+Mc6I7obkoLIJujSWwin1pCvdy/eOxrI/l3uDLQsmcrOb4ug=="],
"abort-controller": ["abort-controller@3.0.0", "", { "dependencies": { "event-target-shim": "^5.0.0" } }, "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg=="],
"accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "^3.0.0", "negotiator": "^1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="],
"agentkeepalive": ["agentkeepalive@4.6.0", "", { "dependencies": { "humanize-ms": "^1.2.1" } }, "sha512-kja8j7PjmncONqaTsB8fQ+wE2mSU2DJ9D4XKoJ5PFWIdRMa6SLSN1ff4mOr4jCbfRSsxR4keIiySJU0N9T5hIQ=="],
"ai": ["ai@6.0.174", "", { "dependencies": { "@ai-sdk/gateway": "3.0.109", "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26", "@opentelemetry/api": "1.9.0" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-bTrfLUWHWtkjzWyCY4bmyuk4Qvmj4S4NSNsXyNSVVqkmftQNtxRj7dzUoMeQDBBwlJO6fC7m2Q/lNOPqQQfAGA=="],
"ajv": ["ajv@8.18.0", "", { "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", "json-schema-traverse": "^1.0.0", "require-from-string": "^2.0.2" } }, "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A=="],
"ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],
@@ -323,8 +251,6 @@
"call-bound": ["call-bound@1.0.4", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="],
"chokidar": ["chokidar@4.0.3", "", { "dependencies": { "readdirp": "^4.0.1" } }, "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA=="],
"combined-stream": ["combined-stream@1.0.8", "", { "dependencies": { "delayed-stream": "~1.0.0" } }, "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg=="],
"content-disposition": ["content-disposition@1.0.1", "", {}, "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q=="],
@@ -333,9 +259,7 @@
"cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="],
"cookie-parser": ["cookie-parser@1.4.7", "", { "dependencies": { "cookie": "0.7.2", "cookie-signature": "1.0.6" } }, "sha512-nGUvgXnotP3BsjiLX2ypbQnWoGUPIIfHQNZkkC668ntrzGWEZVW70HDEB1qnNGMicPje6EttlIgzo51YSwNQGw=="],
"cookie-signature": ["cookie-signature@1.0.6", "", {}, "sha512-QADzlaHc8icV8I7vbaJXJwod9HWYp8uCqf1xa4OfNu1T7JVxQIrUgOWtHdNDtPiywmFbiS12VjotIXLrKM3orQ=="],
"cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],
"cors": ["cors@2.8.6", "", { "dependencies": { "object-assign": "^4", "vary": "^1" } }, "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw=="],
@@ -371,18 +295,14 @@
"eventsource": ["eventsource@3.0.7", "", { "dependencies": { "eventsource-parser": "^3.0.1" } }, "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA=="],
"eventsource-parser": ["eventsource-parser@3.0.8", "", {}, "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ=="],
"exifr": ["exifr@7.1.3", "", {}, "sha512-g/aje2noHivrRSLbAUtBPWFbxKdKhgj/xr1vATDdUXPOFYJlQ62Ft0oy+72V6XLIpDJfHs6gXLbBLAolqOXYRw=="],
"eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
"express": ["express@5.2.1", "", { "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", "content-disposition": "^1.0.0", "content-type": "^1.0.5", "cookie": "^0.7.1", "cookie-signature": "^1.2.1", "debug": "^4.4.0", "depd": "^2.0.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "finalhandler": "^2.1.0", "fresh": "^2.0.0", "http-errors": "^2.0.0", "merge-descriptors": "^2.0.0", "mime-types": "^3.0.0", "on-finished": "^2.4.1", "once": "^1.4.0", "parseurl": "^1.3.3", "proxy-addr": "^2.0.7", "qs": "^6.14.0", "range-parser": "^1.2.1", "router": "^2.2.0", "send": "^1.1.0", "serve-static": "^2.2.0", "statuses": "^2.0.1", "type-is": "^2.0.1", "vary": "^1.1.2" } }, "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw=="],
"express-rate-limit": ["express-rate-limit@7.5.1", "", { "peerDependencies": { "express": ">= 4.11" } }, "sha512-7iN8iPMDzOMHPUYllBEsQdWVB6fPDMPqwjBaFrgr4Jgr/+okjvzAy+UHlYYL/Vs0OsOrMkwS6PJDkFlJwoxUnw=="],
"express-rate-limit": ["express-rate-limit@8.3.2", "", { "dependencies": { "ip-address": "10.1.0" }, "peerDependencies": { "express": ">= 4.11" } }, "sha512-77VmFeJkO0/rvimEDuUC5H30oqUC4EyOhyGccfqoLebB0oiEYfM7nwPrsDsBL1gsTpwfzX8SFy2MT3TDyRq+bg=="],
"extend-shallow": ["extend-shallow@2.0.1", "", { "dependencies": { "is-extendable": "^0.1.0" } }, "sha512-zCnTtlxNoAiDc3gqY2aYAWFx7XWWiasuF2K8Me5WbN8otHKTUKBwjPtNpRs/rbUZm7KxWAaNj7P1a/p52GbVug=="],
"fast-check": ["fast-check@4.8.0", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-GOJ158CUMnN6cSahsv4+ExARvIDuzzinFjkp0E9WtiBa5zcVeLozVkWaE4IzFcc+Y48Wp1EDlUZsXRyAztQcSg=="],
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
@@ -419,8 +339,6 @@
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.12.10", "", {}, "sha512-mx/p18PLy5og9ufies2GOSUqep98Td9q4i/EF6X7yJgAiIopxqdfIO3jbqsi3jRgTgw88jMDEzVKi+V2EF+27w=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -445,16 +363,12 @@
"js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
"json-schema-traverse": ["json-schema-traverse@1.0.0", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="],
"json-schema-typed": ["json-schema-typed@8.0.2", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="],
"kind-of": ["kind-of@6.0.3", "", {}, "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw=="],
"libheif-js": ["libheif-js@1.19.8", "", {}, "sha512-vQJWusIxO7wavpON1dusciL8Go9jsIQ+EUrckauFYAiSTjcmLAsuJh3SszLpvkwPci3JcL41ek2n+LUZGFpPIQ=="],
"marked": ["marked@18.0.0", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-2e7Qiv/HJSXj8rDEpgTvGKsP8yYtI9xXHKDnrftrmnrJPaFNM7VRb2YCzWaX4BP1iCJ/XPduzDJZMFoqTCcIMA=="],
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
@@ -501,16 +415,12 @@
"proxy-addr": ["proxy-addr@2.0.7", "", { "dependencies": { "forwarded": "0.2.0", "ipaddr.js": "1.9.1" } }, "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg=="],
"pure-rand": ["pure-rand@8.4.0", "", {}, "sha512-IoM8YF/jY0hiugFo/wOWqfmarlE6J0wc6fDK1PhftMk7MGhVZl88sZimmqBBFomLOCSmcCCpsfj7wXASCpvK9A=="],
"qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="],
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],
"raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "~3.1.2", "http-errors": "~2.0.1", "iconv-lite": "~0.7.0", "unpipe": "~1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="],
"readdirp": ["readdirp@4.1.2", "", {}, "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg=="],
"require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="],
"router": ["router@2.2.0", "", { "dependencies": { "debug": "^4.4.0", "depd": "^2.0.0", "is-promise": "^4.0.0", "parseurl": "^1.3.3", "path-to-regexp": "^8.0.0" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="],
@@ -563,8 +473,6 @@
"vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="],
"wasm-feature-detect": ["wasm-feature-detect@1.8.0", "", {}, "sha512-zksaLKM2fVlnB5jQQDqKXXwYHLQUVH9es+5TOOHwGOVJOCeRBCiPjwSg+3tN2AdTCzjgli4jijCH290kXb/zWQ=="],
"web-streams-polyfill": ["web-streams-polyfill@4.0.0-beta.3", "", {}, "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug=="],
"web-tree-sitter": ["web-tree-sitter@0.22.6", "", {}, "sha512-hS87TH71Zd6mGAmYCvlgxeGDjqd9GTeqXNqTT+u0Gs51uIozNIaaq/kUAbV/Zf56jb2ZOyG8BxZs2GG9wbLi6Q=="],
@@ -589,16 +497,8 @@
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@modelcontextprotocol/sdk/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
"@modelcontextprotocol/sdk/express-rate-limit": ["express-rate-limit@8.3.2", "", { "dependencies": { "ip-address": "10.1.0" }, "peerDependencies": { "express": ">= 4.11" } }, "sha512-77VmFeJkO0/rvimEDuUC5H30oqUC4EyOhyGccfqoLebB0oiEYfM7nwPrsDsBL1gsTpwfzX8SFy2MT3TDyRq+bg=="],
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"eventsource/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
"express/cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
+9 -13
View File
@@ -1,16 +1,12 @@
[test]
# PGLite WASM cold start + initSchema() runs ~520s on loaded machines.
# Default 5s is too short for those tests' beforeAll hooks. 60s is the
# empirical ceiling we observed for the slowest cold-init paths.
# PGLite initialization can be slow under parallel test execution.
# Default 5s is too short when many test files boot PGLite instances at once.
# 60s is the empirical ceiling we observed before the first file's beforeAll
# completed on a loaded machine.
#
# v0.26.4: scripts/run-unit-parallel.sh and scripts/run-unit-shard.sh
# also pass `--timeout=60000` explicitly so the ceiling is consistent
# whether tests are invoked through the wrapper or directly via bun test.
# NOTE: this bunfig.toml `timeout` key is read by `bun test` but empirically
# does NOT apply to beforeEach/afterEach hook timeouts under `bun run test`
# chained behind `bun run typecheck`. The test script in package.json passes
# `--timeout=60000` explicitly to cover both per-test and per-hook timeouts.
# Leaving both in place as belt-and-suspenders.
timeout = 60_000
# v0.37 fix wave: pin gateway defaults to legacy OpenAI/1536 BEFORE any
# test runs, so the 20+ test files with hardcoded 1536-d Float32Array
# fixtures still match the schema. v0.37's production default is ZE/1280;
# tests that want the new default call configureGateway() explicitly in
# their own beforeAll.
preload = ["./test/helpers/legacy-embedding-preload.ts"]
-117
View File
@@ -1,117 +0,0 @@
# docker-compose.ci.yml
#
# Local CI gate with 4-way E2E sharding. Spins up 4 pgvector services + a bun
# runner that bind-mounts the repo. Used by `bun run ci:local` and
# `bun run ci:local:diff` (see scripts/ci-local.sh).
#
# All services are pulled as `image:` (no build) so `docker compose pull`
# refreshes everything. The bun version floats with `oven/bun:1` to track CI's
# `bun-version: latest`. Named volumes isolate the Linux container's deps from
# the host's darwin-arm64 deps and keep bun + postgres data warm across runs.
#
# Why 4 postgres services: bun's E2E suite shares one DB across 36 files and
# uses TRUNCATE CASCADE in setupDB(). Running files in parallel against ONE DB
# races (file A's TRUNCATE clobbers file B's fixture import). 4 separate DBs
# remove the race; we shard the file list 1/4..4/4 and run shards in parallel.
# Within a shard, files still run sequentially. Total wall-time on a 16-core
# host: ~6 min sequential -> ~1.5-2 min sharded.
#
# Postgres host ports default to 5434-5437 (avoid 5432 manual `gbrain-test-pg`
# and 5433 sibling-project conflicts). Override BASE port with GBRAIN_CI_PG_PORT;
# shards take BASE..BASE+3.
services:
postgres-1:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT:-5434}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-1:/var/lib/postgresql/data
postgres-2:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_2:-5435}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-2:/var/lib/postgresql/data
postgres-3:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_3:-5436}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-3:/var/lib/postgresql/data
postgres-4:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_4:-5437}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
runner:
image: oven/bun:1
working_dir: /app
depends_on:
postgres-1:
condition: service_healthy
postgres-2:
condition: service_healthy
postgres-3:
condition: service_healthy
postgres-4:
condition: service_healthy
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
volumes:
- .:/app
# Linux container's node_modules MUST be isolated from host darwin-arm64.
# Without this, container `bun install` stomps host node_modules and
# subsequent `bun test` on host fails with binary-incompat errors.
- gbrain-ci-node-modules:/app/node_modules
# Warm install cache across runs.
- gbrain-ci-bun-cache:/root/.bun/install/cache
volumes:
gbrain-ci-pg-data-1:
gbrain-ci-pg-data-2:
gbrain-ci-pg-data-3:
gbrain-ci-pg-data-4:
gbrain-ci-node-modules:
gbrain-ci-bun-cache:
-10
View File
@@ -102,16 +102,6 @@ Keeping it running and up to date.
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
## Getting Started
After setup, the brain is empty. The cold-start skill sequences the highest-leverage
data sources to populate it:
| Guide | What It Covers |
|-------|---------------|
| [Cold Start](../skills/cold-start/SKILL.md) | Day-one bootstrapping: contacts, calendar, email, conversations, social, archives. Uses ClawVisor for safe credential handling — agents never hold raw API keys. |
| [Ask User](../skills/ask-user/SKILL.md) | Choice-gate pattern for human input at decision points. Used by cold-start and other skills. |
---
## Appendix: GBrain CLI Quick Reference
+2 -9
View File
@@ -1,12 +1,5 @@
# GBrain v0: Postgres-Native Personal Knowledge Brain
> **Historical design doc.** This is the original v0 spec from before PGLite landed. Several
> forward-looking sections — most notably the SQLite engine plan — were superseded by
> PGLite (embedded Postgres via WASM), which uses the same SQL dialect as Postgres and
> eliminates the need for a separate FTS5/sqlite-vss translation layer. Kept here for
> historical context; see [`ENGINES.md`](ENGINES.md) for the current engine architecture and
> the [`CHANGELOG.md`](../CHANGELOG.md) for the actual implementation history.
## What this is
GBrain is a compiled intelligence system. Not a note-taking app. Not "chat with your notes."
@@ -524,7 +517,7 @@ See `docs/ENGINES.md` for the pluggable engine architecture and future backend p
- **Intelligence compiler.** Treat every fact as a first-class claim with source span, entity links, validity window, confidence, and contradiction status. "What changed, why, and what evidence would flip it again?" From Codex review. Builds on compiled truth model.
- **Active skills via Trigger.dev.** Application-specific briefings, meeting prep. Belongs in OpenClaw, not generic brain infra.
- **Multi-user access.** Supabase RLS + per-user API keys. v0 is single-user.
- **SQLite engine.** Superseded by PGLite (embedded Postgres 17 via WASM) before v1. See [`ENGINES.md`](ENGINES.md) for the current engine architecture.
- **SQLite engine.** Community PRs welcome. See `docs/SQLITE_ENGINE.md`.
- **Docker Compose for self-hosted Postgres.** Community PRs welcome.
- **Web UI.** Optional Vercel-hosted dashboard for browsing brain pages.
@@ -538,7 +531,7 @@ This means:
- A future DuckDB engine could implement analytics-heavy workloads
- The CLI, MCP server, and library consumers never know which engine runs underneath
See [`ENGINES.md`](ENGINES.md) for the full interface spec. (The original SQLite engine plan was superseded by PGLite; the contract-first `BrainEngine` interface made that swap clean.)
See `docs/ENGINES.md` for the full interface spec and `docs/SQLITE_ENGINE.md` for the SQLite implementation plan.
## Review history
-113
View File
@@ -1,113 +0,0 @@
# Install
Three install paths. Pick one. Mix later if needed.
## 1. Run with an agent platform (recommended)
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
```bash
bun install -g github:garrytan/gbrain
gbrain init --pglite # 2 seconds; no server
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
gbrain doctor # green checks all the way down
```
Your agent now reads `skills/RESOLVER.md` once per request, routes intent to the right skill, executes. New entity mentions create new pages. Daily cron runs enrichment overnight.
Scaffolded skills are first-class files in your agent repo — edit freely. To pull upstream gbrain improvements later, `gbrain skillpack reference <name>` diffs your local copy vs the bundle. The legacy `skillpack install` managed-block model was retired in v0.36.0.0; if you're upgrading from an older release, run `gbrain skillpack migrate-fence` once to strip the legacy fence and keep your existing skill rows.
To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts (chunker bumps, the v0.36.2.0 ZeroEntropy switch). Always TTY-only; non-TTY upgrades skip prompts with informational stderr lines.
## 2. CLI standalone
No agent platform, just shell + MCP-aware editor.
```bash
bun install -g github:garrytan/gbrain
gbrain init --pglite
```
> **If `bun install -g` hits a postinstall error** (Bun blocks postinstall hooks in some environments), the CLI prints a recovery hint pointing at [#218](https://github.com/garrytan/gbrain/issues/218). Run `gbrain doctor` to diagnose, then `gbrain apply-migrations --yes` manually. The deterministic fallback is `git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain && bun install && bun link`.
The init flow detects your repo size and suggests Supabase for brains > 1000 markdown files. To switch later:
```bash
gbrain migrate --to supabase # PGLite → Postgres
gbrain migrate --to pglite # Postgres → PGLite (rare)
```
For shared / large / multi-machine deployments (a team or company brain with multiple users hitting one server over HTTP MCP with OAuth scoping per user), follow the dedicated walkthrough: **[Tutorial: set up GBrain as your company brain](tutorials/company-brain.md)**.
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
```bash
gbrain config set zeroentropy_api_key sk-...
gbrain config set anthropic_api_key sk-ant-...
```
Common follow-ups:
```bash
gbrain import ~/my-knowledge # bulk-import a markdown folder
gbrain sync --watch # live-sync a git repo (autopilot mode)
gbrain autopilot --install # background daemon for nightly enrichment
```
**Wire this same local brain into your coding agent** — zero server, zero token:
```bash
claude mcp add gbrain -- gbrain serve # Claude Code
codex mcp add gbrain -- gbrain serve # Codex
```
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
## 3. MCP server (any MCP client)
```bash
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
```
**Wire a coding agent to a remote brain in one command** (when you have an HTTP
server + a bearer token): `gbrain connect` prints a paste-ready setup block, or
`--install` runs it and smoke-tests the token.
```bash
gbrain auth create "claude-code"
gbrain connect https://your-host/mcp --token gbrain_xxx # Claude Code (default)
gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex # Codex (env-var bearer)
gbrain connect https://your-host/mcp --agent perplexity --oauth --register # Perplexity (OAuth)
```
Per-client setup guides live in [`docs/mcp/`](mcp/):
- [`docs/mcp/CLAUDE_CODE.md`](mcp/CLAUDE_CODE.md)
- [`docs/mcp/CODEX.md`](mcp/CODEX.md)
- [`docs/mcp/CLAUDE_DESKTOP.md`](mcp/CLAUDE_DESKTOP.md)
- [`docs/mcp/CHATGPT.md`](mcp/CHATGPT.md)
- [`docs/mcp/PERPLEXITY.md`](mcp/PERPLEXITY.md)
- [`docs/mcp/DEPLOY.md`](mcp/DEPLOY.md) — production deploy patterns
The HTTP server ships with an admin SPA at `/admin`, an SSE activity feed at `/admin/events`, DCR-style client registration, scope-gated `read`/`write`/`admin` access, and rate limiting.
## Thin-client mode
Connect to someone else's brain without running a local engine:
```bash
gbrain init --mcp-only # configures remote MCP, skips local DB
```
Useful for: team mounts, brain-as-a-service deployments, dev machines without disk space. Most local commands refuse with a paste-ready hint. See [`docs/architecture/topologies.md`](architecture/topologies.md).
## Verifying the install
```bash
gbrain doctor --json # full health check
gbrain models # which AI models are configured for what
gbrain models doctor # 1-token probe per configured model
```
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
-72
View File
@@ -538,75 +538,3 @@ To check what your fork is missing:
diff <(grep -A3 "Based on gbrain" ~/<your-fork>/skills/brain-ops/SKILL.md) \
<(grep "v[0-9]" ~/gbrain/skills/migrations/ | tail -3)
```
## v0.36.5.0 — Free-form secret inheritance for shell jobs calling `gbrain` CLI
**The change.** Shell-job params get a new `inherit:` field. Pass any
snake_case config-key name on it; the worker resolves the value from its
`loadConfig()` at child-spawn time and injects it into the child env. Names
land in the row; values never persist from `inherit:`. Validation runs
**pre-enqueue** in both submit paths (CLI + `submit_job` op), so a malformed
payload never lands in `minion_jobs.data`.
**Why.** Pre-v0.36.5.0, agents that wanted to call `gbrain` from shell jobs
had to either write `database_url` to `~/.gbrain/config.json` plaintext or
pass `env: { GBRAIN_DATABASE_URL: "..." }` per-job. Both left plaintext
secrets somewhere — disk or DB row. `inherit:` keeps names in the row and
resolves values at spawn time.
**What your agent can do.** `inherit:` is free-form. Pass any config-key:
```jsonc
{
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
"cwd": "/data/gbrain",
"inherit": ["database_url", "anthropic_api_key", "voyage_api_key"]
}
```
The env-key name in the child is derived by uppercasing the config-key:
`database_url``GBRAIN_DATABASE_URL`, `anthropic_api_key`
`ANTHROPIC_API_KEY`, `voyage_api_key``VOYAGE_API_KEY`, etc. The validator
does NOT police which config keys you inherit — the agent is in the same
uid as the worker, so it's the agent's call.
**You can still use `env:`.** v0.36.5.0 does not forbid `env:{ ANYTHING }`.
If you have a reason to put a value in the row plaintext (a non-secret
correlation token, or a secret you know is OK to persist), pass it via
`env:`. Prefer `inherit:` when you want the value out of the row.
**Worker setup** (one-time, per host):
- `gbrain config set database_url postgresql://...` (or any other key you
want available for inherit)
- OR put the key in `~/.gbrain/config.json` directly
- OR set `GBRAIN_DATABASE_URL` / `DATABASE_URL` / per-provider env on the
worker process
If the worker can't resolve a requested name, the validator fail-fasts at
submit time with `gbrain config set <X>` hint. No more silent "No database
URL" failures in child stderr minutes after submission.
**Also new.** A `gbrain doctor` check `home_dir_in_worktree` warns if
`~/.gbrain/` lives inside a git worktree. A retroactive `~/.gbrain/.gitignore`
(single line `*`) is now laid down by every `saveConfig()` call AND by
`gbrain post-upgrade`, so existing users get coverage without re-running
`gbrain init`. Honest scope: the `.gitignore` covers casual `git add` but does
NOT cover already-tracked files, screenshots, backups, or `git add -f`.
**Strategy framing.** For agent-to-gbrain calls, the new canonical guide is
`docs/guides/agent-to-gbrain.md`. Two distinct surfaces: HTTP MCP via OAuth
for ops with MCP equivalents (`search`, `query`, `put_page`, etc.), and shell
job + `inherit:` for `localOnly` admin ops (`sync`, `embed`, `dream`,
`doctor`, etc.). Not a fallback hierarchy — pick by op.
**Errors to handle** (your agent submits shell jobs; surface these clearly):
| Error | What it means | Agent action |
|---|---|---|
| `shell: inherit must be an array of config-key names` | `inherit` wasn't an array. | Pass `"inherit": ["database_url", ...]`. |
| `shell: inherit entries must be non-empty strings` | Element was empty, non-string, or null. | Use snake_case config-key names. |
| `shell: inherit name "<X>" must match [a-z][a-z0-9_]*` | Name failed snake_case regex (uppercase, leading underscore, etc.). | Use the config-key verbatim — `database_url`, not `DATABASE_URL`. |
| `shell: inherit requested "<X>" but worker has no <X> configured` | Worker can't resolve the name from its `loadConfig()`. | Run `gbrain config set <X> <value>` on the worker host. |
-161
View File
@@ -1,161 +0,0 @@
# llama-server reranker (local) — Qwen3-Reranker, self-hosted ZE, any ZE-wire-shape provider
[`llama-server`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
is the HTTP wrapper that ships with llama.cpp. With `--reranking`, it
exposes an OpenAI-style `POST /v1/rerank` endpoint that returns
`{results: [{index, relevance_score}]}` — exactly the wire shape gbrain
already drives for ZeroEntropy's hosted reranker. The
`llama-server-reranker` recipe (added in v0.40.6.1) routes
`gateway.rerank()` at your local llama.cpp instance instead of ZE.
Two flavors of "local" this recipe covers:
- **Qwen3-Reranker** (0.6B / 4B / 8B) — open-weight cross-encoder; pull
the GGUF from HuggingFace and serve.
- **Self-hosted ZeroEntropy** (`zerank-2`, `zerank-1-small`) — the
weights are on HuggingFace too. GGUF-convert them and serve them the
same way. **Quality is not guaranteed to match ZE-hosted:** GGUF
conversion + quantization + pooling/rank metadata + tokenizer special
tokens all affect scores. If you self-host ZE for production
retrieval, pin your own brain-relevant eval (
[docs/eval-bench.md](../eval-bench.md)) as a regression guard.
This recipe is the path override + recipe shape. Any provider whose
request/response wire matches ZE/llama.cpp can use it by just pointing
at a different base URL. Providers whose wire shape differs (Voyage uses
`top_k` not `top_n`, returns `data[]` not `results[]`) need a separate
recipe with adapter hooks — that lands in a follow-up plan.
## Setup
### 1. Build llama.cpp (or download a release)
```bash
# Clone and build (CPU only; add `-DGGML_CUDA=ON` for GPU)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release -j
```
Pin a specific commit when you ship — `llama-server`'s path aliases
(`/rerank`, `/v1/rerank`, `/reranking`, `/v1/reranking`) have shifted
across releases. The recipe sends to `/v1/rerank`.
### 2. Pull a reranker GGUF
For Qwen3-Reranker-4B (quantized Q4_K_M is the sweet spot for CPU):
```bash
# Pick a quant level — Q4_K_M is the usual CPU sweet spot.
huggingface-cli download \
Qwen/Qwen3-Reranker-4B-GGUF qwen3-reranker-4b-q4_k_m.gguf \
--local-dir ./models
```
For self-hosted ZeroEntropy weights, find a community GGUF conversion
or convert from the HuggingFace weights yourself (out of scope of this
doc — see llama.cpp's `convert_hf_to_gguf.py`).
### 3. Launch llama-server with --reranking AND --alias
```bash
./build/bin/llama-server \
--model ./models/qwen3-reranker-4b-q4_k_m.gguf \
--alias qwen3-reranker-4b \
--reranking \
--port 8081
```
The `--alias` matters: without it, llama-server's `/v1/models` (and the
`model` field rerank requests echo) defaults to the full gguf file
path, which makes the gbrain config string ugly and brittle. With
`--alias qwen3-reranker-4b`, your config string is short and stable.
`--reranking` and `--embeddings` are mutually exclusive at server
launch. If you also run a local embedder via the
[`llama-server`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
recipe, run two separate llama-server processes on two different ports
(typically 8080 for embeddings, 8081 for reranking — gbrain's defaults
match that convention).
### 4. Wire gbrain at your server
```bash
# Point gbrain at the llama.cpp host (skip if running locally on default port)
gbrain config set provider_base_urls.llama-server-reranker http://your-host:8081/v1
# Tell search to use this reranker
gbrain config set search.reranker.model llama-server-reranker:qwen3-reranker-4b
gbrain config set search.reranker.enabled true
```
The `qwen3-reranker-4b` after the colon is your `--alias` value from
step 3. Any string works as long as it matches your server's alias.
Env vars work too as an alternative to the config set above:
```bash
export LLAMA_SERVER_RERANKER_BASE_URL=http://your-host:8081/v1
# Optional: if you front llama-server with nginx + bearer auth
export LLAMA_SERVER_RERANKER_API_KEY=your-bearer-token
```
### 5. Verify
```bash
gbrain models doctor
# Expect: ✔ reranker_config llama-server-reranker:qwen3-reranker-4b ok
# ✔ reranker_config llama-server-reranker:qwen3-reranker-4b ok (reachability)
gbrain search "some query" --json | jq '.[].rerank_score'
# Expect: rerank_score on every row
```
If `gbrain models doctor` reports the reachability probe as `network`
status, two common causes:
1. The server is reachable but in embedding mode, not reranking mode.
`--reranking` and `--embeddings` are mutually exclusive at launch
— relaunch the right one.
2. The recipe path doesn't match what your llama.cpp version serves.
This recipe sends `/v1/rerank`; older llama.cpp installs may only
serve `/rerank`. Pin to a recent llama.cpp commit.
## Cold-start headroom
CPU-only first-call warmup on a 4B reranker can take 8-15 seconds. The
recipe declares `default_timeout_ms: 30000` so the first call after a
server restart doesn't fail-open silently. That value flows through
search-mode resolution unless you override it:
```bash
# Tighten or loosen per-search timeout (overrides recipe default):
gbrain config set search.reranker.timeout_ms 60000
```
Per-call overrides in `SearchOpts.reranker_timeout_ms` still win for
any single call.
## Budget caps + local rerank
The recipe declares `cost_per_1m_tokens_usd: 0` and registers under
`FREE_LOCAL_RERANK_PROVIDERS` in the budget tracker, so
`--max-cost`-bounded callers (autopilot loops, batch jobs) do NOT
hard-fail when configured for local rerank. Local rerank costs
electricity, not API tokens.
```bash
GBRAIN_MAX_USD=0.01 gbrain search "..." --reranker llama-server-reranker:qwen3-reranker-4b
# Works: rerank fires, recorded at $0, cumulative cap untouched.
```
## Fail-open contract preserved
`applyReranker` in `src/core/search/rerank.ts` still has the
fail-open posture: any error class (network, timeout, malformed
response) logs to `~/.gbrain/audit/rerank-failures-*.jsonl` and
returns the original RRF order unchanged. Search reliability beats
reranker quality. If your llama.cpp host goes down, your searches keep
working — they just stop ranking against the cross-encoder until you
restart the server.
-173
View File
@@ -1,173 +0,0 @@
# ZeroEntropy — zembed-1 + zerank-2
[ZeroEntropy](https://zeroentropy.dev) ships two specialized small models
for retrieval pipelines:
- **`zembed-1`** — multilingual embedding distilled from zerank-2.
Flexible Matryoshka dims (2560/1280/640/320/160/80/40), 32K context,
asymmetric `input_type: query|document` encoding. $0.025/1M tokens
(sale) / $0.05 regular.
- **`zerank-2`** — SOTA multilingual cross-encoder reranker.
$0.025/1M tokens (~50% cheaper than Cohere/Voyage rerankers).
Plus `zerank-1` and `zerank-1-small` for legacy / open-source needs.
Both land in gbrain v0.35.0.0 behind the openai-compatible recipe path,
alongside OpenAI and Voyage.
## Setup
1. Get an API key at
[dashboard.zeroentropy.dev](https://dashboard.zeroentropy.dev).
2. Export it:
```bash
export ZEROENTROPY_API_KEY=<your-key>
```
## Embedding switch — zembed-1
**Important:** `gbrain config set embedding_model …` is NOT a live
gateway switch. `embedding_model` and `embedding_dimensions` size the
schema and must be stable across engine connects, so they only resolve
from the **file plane** (`~/.gbrain/config.json`) and the **env plane**
(`GBRAIN_EMBEDDING_MODEL` / `GBRAIN_EMBEDDING_DIMENSIONS`). The DB plane
is intentionally ignored for these two keys (same posture as today's
Voyage setup).
### Option A — file plane (recommended for stable installs)
Edit `~/.gbrain/config.json`:
```json
{
"embedding_model": "zeroentropyai:zembed-1",
"embedding_dimensions": 2560
}
```
Valid dims: `2560` (default), `1280`, `640`, `320`, `160`, `80`, `40`.
Matryoshka-style — smaller trades quality for storage monotonically.
Pick the largest that fits your column width.
### Option B — env plane (CI / Docker)
```bash
export GBRAIN_EMBEDDING_MODEL=zeroentropyai:zembed-1
export GBRAIN_EMBEDDING_DIMENSIONS=2560
```
### Re-embed
Switching embedding models invalidates the vector index. Re-embed:
```bash
gbrain embed --stale --limit 50 # smoke a small batch
gbrain embed --stale # full re-embed
```
### Verify
```bash
gbrain models doctor --json | jq '.probes[] | select(.touchpoint=="embedding_config")'
```
Expected: `status: "ok"`. Invalid dims (e.g. `1024`, `1536`, `3072`)
surface as `status: "config"` with a paste-ready
`gbrain config set embedding_dimensions <one of 2560|1280|640|320|160|80|40>` fix hint.
## Reranker switch — zerank-2
The reranker is the bigger story: gbrain had no cross-encoder reranker
stage before v0.35.0.0. It slots between RRF dedup and token-budget
enforcement in hybrid search.
### Default-on with `tokenmax` mode
`tokenmax` mode now defaults `search.reranker.enabled = true` with
`zerank-2`. If you already use `tokenmax` AND have `ZEROENTROPY_API_KEY`
set, reranker fires automatically. Without the key, every rerank call
fails-open (audit-logged) and search returns RRF order — same UX as
before, just with an observable failure surfaced via `gbrain doctor`.
### Opt-in on `conservative` or `balanced` mode
```bash
gbrain config set search.reranker.enabled true
```
The override sits above the mode-bundle default; opt-out is one flip.
### Cost anchor
At 30 candidates × ~400 tokens/chunk × $0.025/1M = **~$0.0003/query**.
Rounding error against the `tokenmax + Opus` pairing's ~$700/mo at
single-user volume per the CLAUDE.md cost matrix.
### Verify
```bash
gbrain models doctor --json | jq '.probes[] | select(.touchpoint=="reranker_config")'
```
Two probes run for reranker:
- `reranker_config` (zero-network) — validates the model resolves
through the recipe registry and is in the touchpoint's allowlist.
- A reachability probe sends a minimal `{query: "probe", documents:
["probe"]}` rerank to verify auth + URL.
## Knobs reference
| Config key | Default | Notes |
|---|---|---|
| `search.reranker.enabled` | `true` for tokenmax, `false` for others | One-flip opt-in/out |
| `search.reranker.model` | `zeroentropyai:zerank-2` | Try `zerank-1` (older SOTA) or `zerank-1-small` (Apache-2.0 open) |
| `search.reranker.top_n_in` | `30` | Candidates sent to reranker (caps API spend) |
| `search.reranker.top_n_out` | `null` (no truncate) | Truncate reranked output to this many; `null` preserves full length |
| `search.reranker.timeout_ms` | `5000` | HTTP timeout; long stalls degrade UX worse than RRF fallback |
## Failure observability
Reranker is fail-open by construction: every error class (auth, rate-limit,
network, timeout, payload-too-large, unknown) returns the original RRF
order unchanged. Failures log to
`~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation).
`gbrain doctor` reads the audit and surfaces:
- **auth failures** — any single one warns (config-time problem doctor's
own probe should have caught)
- **payload-too-large** — any single one warns (workload-mismatch signal)
- **transient (network/timeout/rate_limit)** — warns at >=5 in 7 days
Query text is SHA-256 hashed in the audit; never logged raw.
## Asymmetric input_type
ZE zembed-1 (and Voyage v3+) use asymmetric query/document encoding for
better retrieval. The gateway's `embedQuery(text)` companion threads
`input_type: 'query'`; standard `embed(texts)` defaults to
`'document'`. Hybrid search's two query-side embed sites use
`embedQuery()` automatically; all ingest paths use `embed()`.
Symmetric providers (OpenAI text-embedding-3, fixed-dim Voyage models)
ignore the field — no behavior change.
## Cache key versioning
v0.35.0.0 bumped `KNOBS_HASH_VERSION` 1 → 2 to fold reranker config into
the `query_cache.knobs_hash` column. During a rolling deploy:
- Expect a temporary cache hit-rate dip (~1 hour at default
`cache.ttl_seconds = 3600s`)
- Hot queries may briefly double their cache row count (one row per
version)
Both clear naturally; no operator action required.
## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| `embedding_config` probe says invalid dim | Defaulting to 1536 (OpenAI default) | Set `embedding_dimensions` to one of 2560/1280/640/320/160/80/40 |
| `reranker_config` probe says model not in allowlist | Typo in `search.reranker.model` | Use one of `zerank-2` / `zerank-1` / `zerank-1-small` |
| `reranker_health` doctor warns about auth | `ZEROENTROPY_API_KEY` not set or invalid | Re-export the env var; `gbrain models doctor` to verify |
| `reranker_health` doctor warns about transient failures | Upstream flake or rate limit | Reranker fails open to RRF; check ZE status page if persistent |
| Cache hit rate dipped after upgrade | Expected during rolling deploy | Clears within `cache.ttl_seconds` (default 3600s) |
-162
View File
@@ -1,162 +0,0 @@
# Why the hybrid + graph stack works
Vector search alone underdelivers on real personal-knowledge queries. This doc explains why gbrain layers four strategies together and how they compound.
## The four strategies in concert
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
## Why each one alone fails
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
**Graph only.** Excellent at "neighbors of Alice" but blind to anything not yet linked. Sparse on fresh pages until backlinks accumulate.
**Hybrid (vector + keyword + RRF), no graph.** Decent at "what is X?" type queries. Fails on "what is Y's relationship to X?" — those are graph queries and no amount of embedding tuning recovers them.
## The benchmark
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus.
| Strategy | P@5 | R@5 | Notes |
|---|---|---|---|
| ripgrep BM25 only | ~18 | ~75 | Lexical-only baseline |
| vector-only RAG | ~18 | ~80 | Standard RAG implementation |
| gbrain graph-disabled (hybrid + RRF, no graph traversal) | ~18 | ~85 | Hybrid alone |
| **gbrain default (full stack)** | **49.1** | **97.9** | Graph + extract-quality lift |
**+31 P@5 points** from the graph + extract quality work. The graph isn't a marginal feature; it's the load-bearing wall.
## Auto-link: why zero-LLM-call edge extraction works
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
- Typed-link blockquotes: `> **Convention:** see [path](path).`
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM unnest(...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1`. The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`, `advises`) fires from surrounding sentence context — also LLM-free. Power users who want richer types add them via the typed-link blockquote convention.
## ZeroEntropy as reranker: 60% top-1 reshuffle
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
The cost: +150ms p50 latency, ~$0.025/M tokens. Disabled with `gbrain config set search.reranker.enabled false`. For agent loops that do downstream LLM work after retrieval, the latency is invisible.
## Source-aware ranking
Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank.
The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `SearchOpts.exclude_slug_prefixes`. Temporal queries (`detail: 'high'`) bypass the boost so chat pages re-surface for time-sensitive lookups.
## Named-thing retrieval (per-page pool + title + alias + evidence)
A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than
embedding proximity. Four layers, added after the incident in
[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md):
- **Per-page max-pool**`searchVector` (both engines) collapses chunk-grain
candidates to the best chunk per page (`DISTINCT ON (slug)`) over the full
candidate set before the user `LIMIT`, via the shared `buildBestPerPagePoolCte`
in `sql-ranking.ts`. The vector side returns N distinct pages by best chunk,
not N chunks that collapse to fewer pages downstream.
- **Title-phrase boost** — when the normalized query is a contiguous token-run
inside `page.title` (or an exact full-title match), a floor-ratio-gated,
bounded multiplier fires (`applyTitleBoost`, `search.title_boost` knob). A
query that is a phrase from the title can't lose to a body chunk by luck.
- **Alias hop** — free-text `aliases:` frontmatter is projected into a
`page_aliases` table (separate from the `slug_aliases` wikilink redirect) and
consulted at query time: a full normalized-query match injects/boosts the
canonical page (`applyAliasHop`). The only layer that bridges true synonyms
with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill
existing pages with `gbrain reindex --aliases`.
- **Evidence contract** — every result carries `evidence`
(`alias_hit | exact_title_match | high_vector_match | keyword_exact |
weak_semantic`) and `create_safety` (`exists | probable | unknown`). An agent
deciding "is this page already here, safe to NOT write a duplicate?" keys off
`create_safety`, not a raw blended score.
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
specific miss with `gbrain search diagnose "<q>" --target <slug>`.
## Intent-aware query rewriting
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
- **Event** queries ("Acme AI Series A") engage the timeline index.
- **General** queries hit the standard hybrid stack.
The classifier is deterministic (no LLM call). Wrong classification degrades gracefully — the hybrid stack still works without it.
## Multi-query expansion
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
## Putting it together
The full pipeline for a `query` op:
```
intent classify
expansion (if enabled)
hybrid search:
├── vector (HNSW on chunk embeddings)
├── keyword (BM25 via tsvector)
├── source-aware re-rank (CASE in SQL)
└── RRF fusion → top 30
graph augment (typed-edge traversal from any seed)
reranker (zerank-2 cross-encoder, top 30 → reordered)
token-budget enforcement (per mode bundle)
deduplication (same slug, different chunks → keep best)
results
```
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
## How to verify on your own brain
```bash
# Run the public LongMemEval benchmark
gbrain eval longmemeval datasets/longmemeval_s.jsonl
# Capture your own queries and replay against retrieval changes
export GBRAIN_CONTRIBUTOR_MODE=1
# ... use gbrain normally ...
gbrain eval export > before.ndjson
# ... change something ...
gbrain eval replay --against before.ndjson
# A/B retrieval strategies on a labeled fixture
gbrain eval --qrels labels.tsv --config balanced.json
```
Methodology + metric glossary in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
@@ -1,97 +0,0 @@
# Retrieval Incident: a chosen-name page was missed, and the fix
**Status:** Resolved (retrieval-cathedral wave). Supersedes the docs-only RFC in
closed PR #1616 — the diagnosis there was directionally right about the disease
but wrong on several mechanics; this is the corrected record + what shipped.
**Original author:** Garry Tan's OpenClaw. **Severity at the time:** High.
**Related:** [`RETRIEVAL.md`](./RETRIEVAL.md), [`../eval/METRIC_GLOSSARY.md`](../eval/METRIC_GLOSSARY.md).
---
## 1. What happened
The agent was asked to log that Garry "wants to build a Greek amphitheater." It
ran a retrieval for the concept, the canonical concept page (titled "...Indoor
Greek Amphitheater...") did **not** surface with enough confidence to be
recognized as the existing page, and the agent wrote a **duplicate stub** on top
of a fully-developed concept doc. Garry caught it: "It's in the brain. It's the
Hall of Light. Why did you forget?"
The page is *about* a Greek amphitheater — the phrase is in its title and first
sentence. A healthy index returns it at the top. It didn't.
## 2. The disease (the RFC got this right)
The brain is stored by **meaning and chosen name** (Mingtang, Hall of Light) but
was retrieved by **literal embedding proximity to a body chunk**, and the agent's
"is this already here?" decision keyed off a single fuzzy blended score. Three
retrieval gaps plus one contract gap produced the miss.
## 3. Verified ground truth (corrections to the RFC)
These were checked in code during the fix; several change the remedy:
1. **`gbrain search` was keyword-only**, not hybrid — so the RFC's cosine scores
(0.64/0.98) came from the hybrid `query`/MCP path the agent actually hit, not
`gbrain search`. The repro command in the RFC was mislabeled.
2. **`--mode` was never a CLI param** — mode resolves server-side from the
`search.mode` config key, which is why all three "modes" returned identical
results (the flag was silently dropped; `thorough` isn't a real mode).
3. **`hybridSearch` already max-pooled per page at the dedup layer.** So the
per-page max-pool fix's real win is *candidate-set page recall* (the vector
side returned N chunks that could collapse to fewer pages), and it is
necessary-but-not-sufficient: if a page's title chunk scores below a body
chunk on a 2-word query, or falls outside the candidate pool, pooling alone
doesn't rescue it.
4. **Frontmatter `aliases:` was dead to search** — stored in `pages.frontmatter`
JSONB, never consulted. `slug_aliases` is a *slug→slug* wikilink redirect, a
different concept.
## 4. The fix that shipped (four layers + a contract)
| Layer | Fixes | Where |
|---|---|---|
| **Per-page max-pool** (T1) | a page scored by its weakest chunk; vector page-recall | `searchVector` both engines, shared `buildBestPerPagePoolCte` |
| **Title-phrase boost** (T2) | query is a phrase in the title but matched a body chunk | `applyTitleBoost` (reads `page.title`), `title_boost` mode knob |
| **Alias hop** (T3) | true synonyms with zero surface overlap ("Hall of Light" → Mingtang) | `page_aliases` table, `applyAliasHop`, ingest projection + `reindex --aliases` backfill |
| **Evidence contract** (T4) | the agent keyed "don't duplicate" off a fuzzy score | `evidence` + `create_safety` on every result; the agent keys off `create_safety='exists'`, not a threshold |
Plus: `gbrain search "<text>"` is now cheap-hybrid (the obvious verb gives the
good path); `modes/stats/tune` stay subcommands; `--mode` works per-call for
local callers; rank-1 score drift telemetry; and **NamedThingBench**, a CI gate
that hard-gates the families that ARE this incident.
## 5. How to confirm / triage a recurrence
```
# Which layer surfaces (or misses) the target page?
gbrain search diagnose "Greek amphitheater" --target projects/new-greek-theater/concept_v0
# Backfill aliases for existing pages whose frontmatter predates the alias layer:
gbrain reindex --aliases
# Watch retrieval quality over time (a downward avg rank-1 score = regressing):
gbrain search stats --days 30
# The gate that prevents silent reintroduction:
gbrain eval retrieval-quality test/fixtures/retrieval-quality/namedthing.jsonl
```
For a page to be reliably found by its chosen name, give it `aliases:` frontmatter:
```yaml
---
title: The Mingtang — Indoor Greek Amphitheater
aliases:
- Hall of Light
- 明堂
---
```
## 6. The discipline this teaches
A benchmark that scores 97.9 R@5 while production returns a flagship page at 0.64
means the benchmark and the shipped path diverged. NamedThingBench runs the same
families through the real pipeline on every PR, and the evidence contract means
the agent's duplicate-or-not decision is grounded in *why* a page matched, not a
number that was never a calibrated probability.
-242
View File
@@ -1,242 +0,0 @@
# Brains and Sources — the mental model
GBrain has two orthogonal axes for organizing knowledge. Users and agents both
need to understand both of them, or queries misroute silently.
**TL;DR:**
- A **brain** is a database. You can have many.
- A **source** is a named repo of content *inside* a brain. One brain can hold many.
- `--brain <id>` picks WHICH DATABASE.
- `--source <id>` picks WHICH REPO WITHIN that database.
- They're independent. You can target any combination.
---
## The two axes
### Brains (the DB axis)
A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase.
Each brain has:
- Its own `pages` table, `chunks` table, `embeddings`, etc.
- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2).
- Its own separate lifecycle, backup, access control.
Brains are enumerated by:
- **host** — your default brain, configured in `~/.gbrain/config.json`.
- **mounts** — additional brains registered in `~/.gbrain/mounts.json` via
`gbrain mounts add <id>` (v0.19+).
Routing: `--brain <id>`, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or
longest-path match against registered mount paths. Falls back to `host`.
### Sources (the repo axis, v0.18.0+)
A **source** is a named content repo *inside* one brain. Every `pages` row
carries a `source_id`. Slugs are unique per source, not globally.
Example: in one brain, the slug `topics/ai` can exist under `source=wiki`
AND under `source=gstack` — they're different pages.
Routing: `--source <id>`, `GBRAIN_SOURCE`, `.gbrain-source` dotfile, or
registered `local_path` match in the `sources` table.
### When does each axis move?
| You want to | Adjust |
|---|---|
| Work in a different repo within the same brain (wiki → gstack notes) | `--source` |
| Query a team-published brain that isn't yours | `--brain` |
| Isolate a topic so it never leaks into personal search | `--source` with `federated=false` |
| Share a brain with teammates | `--brain` (mount the team brain) |
| Add a new repo to your personal brain | `--source` via `gbrain sources add` |
| Add a team brain | `--brain` via `gbrain mounts add` |
**Rule of thumb:** if the data owner changes, it's a brain boundary. If the
data owner stays the same but the topic/repo changes, it's a source boundary.
---
## Topology: a single-person developer
Simplest case. One brain, one source.
```
┌─────────────────────────────────────────┐
│ host brain (~/.gbrain) │
│ ├── source: default (federated=true) │
│ │ └── all pages │
└─────────────────────────────────────────┘
```
`gbrain query "retry budgets"` finds everything. No `--brain`, no `--source`
needed.
---
## Topology: a personal brain with multiple repos
You maintain several codebases or writing streams. Each is its own source
inside one brain. Cross-source search is on by default so a query about
"caching" returns hits from every repo.
```
┌──────────────────────────────────────────────┐
│ host brain (~/.gbrain) │
│ ├── source: wiki (federated=true) │
│ │ └── personal notes, people, companies │
│ ├── source: gstack (federated=true) │
│ │ └── gstack plans, learnings │
│ ├── source: openclaw (federated=true) │
│ │ └── openclaw docs, memos │
│ └── source: essays (federated=false) │
│ └── draft essays, isolated on purpose │
└──────────────────────────────────────────────┘
```
Inside `~/openclaw/` the `.gbrain-source` dotfile pins every command to
`source=openclaw`. Inside `~/gstack/` the dotfile pins to `source=gstack`.
Everything still targets one DB.
Use this topology when:
- You own all the content.
- You want cross-repo search to just work.
- You don't need to share any of it with someone who isn't you.
---
## Topology: personal brain + one team brain
You're on a team that publishes a shared brain. Your personal brain stays
as-is; you mount the team brain alongside it.
```
┌──────────────────────────────────────────────┐
│ host brain (~/.gbrain) — YOUR personal DB │
│ ├── source: wiki │
│ ├── source: gstack │
│ └── ... │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ mount: media-team │
│ path: ~/team-brains/media │
│ engine: postgres (team's Supabase) │
│ └── sources: wiki, raw, enriched │
└──────────────────────────────────────────────┘
```
`gbrain query "X"` (no flags) → runs against host (your personal brain).
`gbrain query "X" --brain media-team` → runs against the team's DB.
Inside `~/team-brains/media/` a `.gbrain-mount` dotfile pins brain to
`media-team` automatically.
Use this topology when:
- You're on a team and someone publishes a brain the team subscribes to.
- You need data isolation between work and personal.
- Different teams/orgs own different brains.
---
## Topology: a CEO-class user with multiple team memberships
You're senior enough to sit across multiple teams. You maintain your personal
brain (with N sources inside) AND mount several work team brains. Each team
brain is itself a multi-source brain in the v0.18.0 sense — organized
internally however the team owner chose.
```
┌──────────────────────────────────────────────┐
│ host brain — YOUR personal DB │
│ ├── source: wiki │
│ ├── source: essays │
│ ├── source: gstack │
│ └── source: openclaw │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ mount: media-team (your media team's brain) │
│ └── sources: wiki, pipeline, enriched │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ mount: policy-team (your policy team's) │
│ └── sources: wiki, research, letters │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ mount: portfolio (another team's) │
│ └── sources: companies, deals, diligence │
└──────────────────────────────────────────────┘
```
Inside each team's checkout, a `.gbrain-mount` dotfile pins the brain. Inside
a specific subdirectory, a `.gbrain-source` dotfile pins the source. So `cd
~/team-brains/policy/research && gbrain query "X"` targets
`brain=policy-team, source=research` with zero flags.
Use this topology when:
- You cross-cut multiple teams.
- Each team owns its own brain with its own access policy.
- You need latent-space federation (agent decides when to query across
brains), not SQL federation.
Cross-brain queries are **not deterministic** in v0.19. The agent sees the
brain list and re-queries as needed. That's the feature — it keeps debugging
sane and access control clean.
---
## Resolution precedence (one page to remember)
```
WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)?
1. --brain <id> 1. --source <id>
2. GBRAIN_BRAIN_ID env 2. GBRAIN_SOURCE env
3. .gbrain-mount dotfile 3. .gbrain-source dotfile
4. longest-prefix mount path match 4. longest-prefix source path match
5. (reserved: brains.default v2) 5. sources.default config
6. fallback: 'host' 6. fallback: 'default'
```
Both axes follow the same layered pattern on purpose. If you know one, you
know the other.
---
## For agents reading this
- Default assumption when the user asks a question: start in the current
brain (resolved via the precedence above). Don't jump brains without a
reason.
- If the user asks a question that crosses topic areas a team might own
(e.g. "what did Team X decide last week?"), the right move is to *query
the team's brain explicitly* rather than searching host with "team x".
- Cross-brain federation is YOUR JOB, not the DB's. You have the brain list
(`gbrain mounts list`). You decide when to fan out. You synthesize
findings. You cite `brain:source:slug`.
- When writing a page, respect the brain boundary. A fact about a team's
work belongs in the team's brain, not in the user's personal brain. Ask
before writing cross-brain.
- See `skills/conventions/brain-routing.md` for the full decision table.
## For users reading this
- **Default path:** set up your personal brain (`gbrain init`), add a source
per repo you care about (`gbrain sources add gstack --path ~/gstack`).
You'll almost never need `--brain`.
- **When a team publishes a brain:** `gbrain mounts add <team-id> --path
<clone> --db-url <url>` and the `.gbrain-mount` dotfile in that checkout
routes queries there automatically.
- **When you are the CEO-class user with multiple team memberships:** mount
each team brain. Trust the resolver — inside a team's directory the
dotfile picks the brain, inside a subdirectory the dotfile picks the
source. The flags are for when you want to query across the boundary
deliberately.
## Further reading
- v0.18.0 CHANGELOG — introduced `sources` primitive.
- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`.
- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain
publisher, not just the subscriber.
@@ -1,215 +0,0 @@
# Calibration Quality Gate — Falsifiability Filter + Category Classification
> **Historical context.** This is the source spec absorbed from PR #1191 into
> two waves of implementation:
>
> - **v0.37.2.0 hotfix** (this release): widens the `takes_resolution_consistency`
> CHECK constraint to accept `quality='unresolvable'` as a 4th valid state.
> Unblocks the production grading script. Adds `unresolvable_count` +
> `unresolvable_rate` to `TakesScorecard` as sibling fields (preserves
> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80
> during successive master merges — v0.37.0.0's autonomous-remediation wave
> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79.
> - **Follow-up minor** (forthcoming): falsifiability + category extraction at
> `propose_takes`, SQL-side grade gate, per-category calibration scorecards,
> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation
> against the v0.36.1.0 fixtures.
>
> Preserved here per the hotfix plan's PR #1191 close protocol so the
> production context (96K-page brain, 6.8% falsifiability rate, category
> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation.
## Problem
v0.36.1.0 ships `propose_takes`, `grade_takes`, and `calibration_profile` as a
connected pipeline: extract claims → grade them against outcomes → build a
calibration profile showing systematic biases.
In production on a 96K-page brain with 36K takes across 6,239 holders, the
grade_takes phase produces noisy results:
- **6.8% falsifiability rate**: Of 500 candidate takes (weight ≥ 0.7), only 34
passed an LLM falsifiability filter. The other 93% were philosophical beliefs,
present-state observations, advice, logistics, or vague vibes.
- **50% unresolvable**: Even after filtering, 17/34 predictions couldn't be
graded because evidence was insufficient or the claim was too ambiguous.
- **Duplicates**: Same claim from the same page extracted multiple times with
slightly different wording.
The root cause: `propose_takes` extracts everything that looks like a belief or
assertion. That's correct for the *takes* table (epistemological layer), but
`grade_takes` needs a much narrower subset: **falsifiable predictions about
future outcomes** where we can check what actually happened.
### Example classifications from production testing
**Genuine predictions (grade-worthy):**
- "X will reach $1M ARR very soon" → company_outcome
- "X is going to leave Y" → people_move
- "AI will make authentic authorship more important" → technology
- "X was convinced Y would win the Z market" → market_call
**Not predictions (should skip grading):**
- "Desire is mimetic" → philosophical belief
- "X should charge 10x more" → advice
- "Return from Toronto on Monday" → logistics
- "Something is going to happen there" → vague/unfalsifiable
- "X is growing very quickly" → present-state observation
## Solution
### 1. Falsifiability score at extraction time
Add a `falsifiability` column to the `takes` table (real, 0.01.0, nullable,
default null). `propose_takes` sets this during extraction using the same LLM
call that already produces the take — one additional field in the JSON schema.
```sql
ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability real;
ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability_category text;
```
The LLM prompt addition (appended to the existing propose_takes extraction prompt):
```
For each claim, also assess:
- falsifiability (0.0-1.0): Can this claim be checked against future reality?
1.0 = specific, measurable, time-bounded prediction about an outcome
0.5 = directional claim that's partially checkable
0.0 = philosophical belief, advice, observation, or unfalsifiable assertion
- falsifiability_category: one of
company_outcome | fundraising | technology | people_move | market_call | other_prediction | not_prediction
```
Cost: ~0 incremental tokens (the claim is already being extracted; this adds
two fields to the JSON output schema).
### 2. Grade gate in `grade_takes`
Before attempting grading, filter:
```typescript
const gradeable = candidates.filter(t =>
t.falsifiability !== null && t.falsifiability >= 0.7
&& t.falsifiability_category !== 'not_prediction'
);
```
This reduces grading volume by ~93% in production, which means:
- LLM cost for grading drops proportionally
- Evidence retrieval load drops (each grade attempt triggers hybrid search)
- Calibration profiles are built on real predictions, not noise
### 3. Deduplication at extraction
`propose_takes` should check for near-duplicate claims before inserting:
```typescript
// Before inserting a new take, check if a similar claim exists
// for the same holder from the same page
const existing = await engine.sql`
SELECT id, claim FROM takes
WHERE holder = ${holder}
AND page_id = ${pageId}
AND similarity(claim, ${newClaim}) > 0.8
LIMIT 1
`;
if (existing.length > 0) {
// Skip — near-duplicate
continue;
}
```
Requires `pg_trgm` extension (already available on most Postgres installations).
Falls back gracefully: if `similarity()` isn't available, skip the dedup check.
### 4. Category-aware calibration profiles
The `calibration_profile` phase can now group resolved takes by
`falsifiability_category` to produce per-domain scorecards:
```
"Your company_outcome calls are 73% accurate.
Your people_move calls are 90% accurate.
Your technology calls are 60% accurate — you tend to be ~18 months early."
```
This is the tweetable output: a calibration profile that says "here's how you're
systematically right and wrong by category."
## Schema Changes
```sql
-- Migration: add falsifiability columns to takes
ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability real;
ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability_category text;
-- Index for grade_takes filter
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_takes_falsifiability
ON takes (falsifiability)
WHERE falsifiability IS NOT NULL AND falsifiability >= 0.7;
-- Optional: pg_trgm for dedup (CREATE EXTENSION IF NOT EXISTS pg_trgm;)
```
## Evidence Retrieval (v0.36.1.0 → v0.37 enhancement)
The current `grade_takes` evidence retriever returns a stub placeholder. In
production testing, we wired real evidence retrieval via `gbrain query` (hybrid
search). The pattern that works:
1. Extract the core claim from the take (first 150 chars)
2. Run `engine.query(claim)` to get relevant pages
3. Filter to pages updated AFTER the take's `since_date` (evidence must be newer)
4. Pass top-5 chunks as the evidence block to the judge
This should replace the stub in the `evidenceRetriever` injection point.
## Production Results
After implementing the falsifiability filter (as a pre-processing step outside
the cycle):
| Metric | Before (v2, no filter) | After (v3, with filter) |
|--------|----------------------|----------------------|
| Candidates evaluated | 50 | 34 (from 500 screened) |
| Falsifiable predictions | ~19 (38%) | 34 (100%) |
| Correct | 10 (52.6% of resolvable) | 10 (58.8% of resolvable) |
| Incorrect | 5 (26.3%) | 2 (11.8%) |
| Partial | 4 (21.1%) | 5 (29.4%) |
| Unresolvable | 31 (62%) | 17 (50%) |
| Category breakdown | N/A | people_move:13, company_outcome:11, technology:4, market_call:2 |
Key improvement: **the false positive rate dropped from 62% noise to 0% noise**
in the gradeable set. The remaining 50% unresolvable rate is genuine — those
predictions are about outcomes that haven't happened yet or where the brain
lacks evidence. That's correct behavior, not noise.
## Files to Change
1. **`src/core/cycle/propose-takes.ts`** — Add falsifiability + category to
extraction prompt and output schema
2. **`src/core/cycle/grade-takes.ts`** — Add falsifiability gate before grading;
wire real evidence retrieval
3. **`src/core/cycle/calibration-profile.ts`** — Group scorecards by category
4. **`src/core/engine.ts`** — Add `similarity()` helper for dedup (graceful
fallback)
5. **New migration** — Add columns + index
## Testing
- Unit test: falsifiability classifier on 20 known-good and 20 known-noise takes
- Unit test: dedup correctly merges near-identical claims
- Unit test: grade gate filters below threshold
- Integration test: full cycle with falsifiability → grade → profile pipeline
- Regression test: existing takes without falsifiability score are not broken
(null falsifiability = ungated, backward compatible)
## Backward Compatibility
- `falsifiability` defaults to null. Existing takes are unaffected.
- `grade_takes` with null falsifiability: configurable behavior. Default:
grade all (backward compat). Operator can set
`cycle.grade_takes.require_falsifiability: true` to gate.
- Category column is purely additive.
- Dedup is opt-in: `cycle.propose_takes.dedup.enabled: true`.
@@ -1,200 +0,0 @@
# Frontmatter scan: DB-backed incremental state (Phase 2 design sketch)
**Status:** Designed, not built. Captured here as the starting point for the
follow-up PR after v0.38.2.0.
## Why this exists
v0.38.2.0 fixed the load-bearing bug class that caused `gbrain doctor` to
hang on large brains: the disk walker descended into `node_modules/`, `.git/`,
and other vendor trees on every tick. After that fix doctor completes in
seconds on most brains, and bounded wall-clock (default 30s, with honest
partial-state surfacing) on any brain.
But the steady-state cost of `frontmatter_integrity` is still O(N) in real
syncable pages: every doctor tick re-walks the filesystem and re-parses
every `.md` file. For users with 200K+ pages the steady-state cost is in
the seconds even after Fix 1. For sub-second steady-state doctor (the
right shape for cron-monitored health checks), the scan needs to become
incremental.
This document captures the Phase 2 design before the follow-up PR starts,
so the implementer doesn't have to re-derive it.
## Goal
Doctor's `frontmatter_integrity` check completes in O(1) SQL queries
regardless of brain size, with the same per-source breakdown and partial-
state semantics as v0.38.2.0's bounded-walk approach. Incremental refresh
runs as a sync-side write + an autopilot cycle phase, so the steady-state
work is amortized across the workflow that already touches each file.
## Schema
New table:
```sql
CREATE TABLE frontmatter_scan_state (
source_id TEXT NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
path TEXT NOT NULL, -- relative to source.local_path
mtime_ms BIGINT NOT NULL,
content_hash TEXT NOT NULL, -- sha256 of file content at scan time
codes JSONB NOT NULL DEFAULT '[]'::jsonb, -- ParseValidationCode[]
last_scanned_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (source_id, path)
);
CREATE INDEX frontmatter_scan_state_has_issues_idx
ON frontmatter_scan_state (source_id)
WHERE codes != '[]'::jsonb;
```
Why these columns:
- `mtime_ms` + `content_hash`: incremental check picks one. mtime is faster
(no read); content_hash is the truth (defeats touch-without-change cases).
The incremental walker uses mtime as a fast gate and content_hash as the
fallback when mtime suggests change.
- `codes` JSONB: per-row error code list, NULL/`[]` means clean. Doctor
aggregates with `jsonb_array_length(codes) > 0`.
- Partial index on `WHERE codes != '[]'::jsonb`: doctor's aggregate query
only walks rows with issues, which is a small fraction of pages.
This follows the canonical `applyForwardReferenceBootstrap` pattern in
`src/core/pglite-engine.ts` (and `postgres-engine.ts`) — the new column /
table additions go into the bootstrap probe set per CLAUDE.md so old brains
walking forward through the schema chain don't wedge on the table not
existing.
## Migration shape
```ts
// src/core/migrate.ts — append after the v80 entry
const migrations = [
// ...existing v1-v80...
{
version: 81,
name: 'frontmatter_scan_state',
sql: `
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
CREATE INDEX IF NOT EXISTS frontmatter_scan_state_has_issues_idx ...;
`,
},
];
```
Plus the forward-reference probe entries in both engine bootstraps. Plus
the `REQUIRED_BOOTSTRAP_COVERAGE` extension in
`test/schema-bootstrap-coverage.test.ts`.
## Writers
Two paths write rows:
1. **Sync-side write** (canonical). `src/core/sync.ts:performSync` already
parses every file it touches. After the existing `parseMarkdown` call,
`UPSERT` into `frontmatter_scan_state` with the file's path / mtime /
content_hash / codes. Cost: one row per file synced. Zero extra parse
work — the parse already happened.
2. **Incremental scan** (`gbrain frontmatter scan --incremental`). Walks
the disk via `walkBrainTree`, for each file checks `mtime > last_scanned_at`
OR `content_hash != stored`, only re-parses changed files. Most ticks:
zero work after the first full backfill. Also exposed as an autopilot
cycle phase (`frontmatter_scan`) so it runs alongside the other periodic
maintenance phases.
The incremental walker handles two cases sync misses:
- Files edited outside sync (user opens an editor, saves, never `git
commit`s).
- Sources whose `local_path` isn't a git repo (sync only sees git-touched
files).
## Doctor reader
```ts
// src/commands/doctor.ts:frontmatter_integrity (Phase 2 shape)
const rows = await engine.executeRaw<{ source_id: string; issues: number }>(
`SELECT source_id, count(*) FILTER (WHERE jsonb_array_length(codes) > 0)::int AS issues
FROM frontmatter_scan_state
GROUP BY source_id`,
);
```
One SQL query, constant time regardless of brain size. The partial-state
surfacing from v0.38.2.0 stays — when `frontmatter_scan_state` is stale
(no rows for a registered source, or `last_scanned_at` >24h old for any
source), doctor warns about freshness rather than reporting potentially-
stale data as authoritative.
## Sequencing concerns
1. **First-ever scan.** A fresh upgrade has no rows in
`frontmatter_scan_state`. Two options:
- Lazy: doctor reports "no scan state yet; run `gbrain frontmatter scan
--incremental` once" (operator-driven).
- Eager: the migration that creates the table also enqueues an autopilot
cycle job to do the first full scan.
Recommendation: lazy, with a clear hint. The autopilot path is heavier
surface (must add the new `frontmatter_scan` phase to the existing
cycle.ts machinery + the doctor-routed background job system).
2. **Source archival / deletion.** `frontmatter_scan_state` has `ON DELETE
CASCADE` on `sources(id)`, so soft-delete + 72h TTL + purge already
clean it up. No additional logic needed.
3. **Path renames inside a source.** Sync would `DELETE` the old row by
path (via a periodic reconcile step) and `INSERT` the new row. Without
that step, the table accumulates stale path rows. Either:
- A reconcile step in the incremental scanner: any path-row not seen
during the walk gets deleted.
- Or: doctor reports "N stale rows in frontmatter_scan_state" as a
freshness signal, with `gbrain frontmatter scan --reconcile` as the
remediation.
## Cost estimate
- One UPSERT per file synced. Negligible vs the parse + DB write that sync
already does.
- Incremental refresh runtime: dominated by mtime stats. ~ms per 1000 files
on SSD.
- Doctor read: one indexed SQL query. Sub-100ms on any brain size.
## What this design deliberately does NOT do
- **Replace v0.38.2.0's bounded-walk safety net.** Phase 2 makes the
steady-state cheap, but the disk walker (with its deadline check) stays
as the source-of-truth fallback for sources whose scan state is missing
or stale. Belt-and-suspenders.
- **Introduce a separate frontmatter validation rule set.** Reuses
`parseMarkdown(..., {validate: true})` and the existing
`ParseValidationCode` enum. Single source of truth.
- **Add a new background daemon.** Wires into the existing
`autopilot-cycle` Minion handler as a new phase, alongside sync /
extract / embed / etc.
## Open questions for the implementer
1. **Path normalization.** `pages.source_path` and the disk walker's
relative path computation are similar but not identical (slashes,
leading `./`, etc.). The incremental scanner needs to match what sync
stores so UPSERTs key correctly. Audit before writing.
2. **Soft-delete interaction.** A page that gets soft-deleted in the DB
(v0.26.5) still has a file on disk. Should the incremental scan
continue to track its frontmatter state? Probably yes (so a future
`restore_page` doesn't surprise with stale frontmatter), but worth
confirming with the soft-delete owner.
3. **Two-phase rollout.** Land the table + writes first, let it backfill
for a release cycle, then switch the doctor reader. Avoids the
"Phase 2 ships but the table is empty" case where doctor regresses to
reporting "no scan state."
## TODO file entry
```
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
Design lives at docs/architecture/frontmatter-scan-incremental.md.
Schema migration v81 + sync-side UPSERT + incremental scan command
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
table + writes first; flip the reader one release later.
```
-143
View File
@@ -1,143 +0,0 @@
# Lens packs (v0.41.2.0)
Four bundled schema packs that turn the gbrain dream cycle into a multi-lens
brain. Activate one with `gbrain config set schema_pack <name>` and the cycle
picks up the pack's declared phases on the next `gbrain dream` run.
## The four packs
```
gbrain-base (shipped v0.38)
│ extends
┌──────────────┼──────────────────────┐
│ │ │
gbrain-creator gbrain-investor gbrain-engineer
(atom + concept (deal/thesis/ (learning bridge
lifecycle) bet_resolution) for gstack)
│ │ │
└──────────────┼───────────────────────┘
│ extends + borrow chain
gbrain-everything (meta-pack)
one brain, three lenses active
```
### gbrain-creator
Atom + concept content-creator lifecycle. Drives two cycle phases:
- `extract_atoms` — per source, Haiku extracts 1-3 atoms from each
transcript with the closed 11-value `atom_type` enum (insight,
anecdote, quote, framework, statistic, story_angle, strategy_angle,
strategy, endorsement, critique, collection). Writes
`atoms/{YYYY-MM-DD}/{slug}` pages. Budget cap $0.30/source/run.
- `synthesize_concepts` — globally aggregates atoms by frontmatter
`concepts:` ref. Tier by count: T1 ≥10, T2 ≥5, T3 ≥2. T1/T2 get
Sonnet narratives; T3 falls back to a deterministic stub. Writes
`concepts/{slug}` pages. Budget cap $1.50/run.
One calibration domain: `concept_themes` / cluster_summary / [concept]
— tier histogram + page count, not Brier (concepts don't have binary
outcomes to score against).
### gbrain-investor
YC / investor lens. Declares 2 net-new page types on top of
gbrain-base's deal/person/company/yc seed:
- `thesis` (NEW) — investment thesis with thesis_text + key_bets[] +
market_view + vintage. Files at `investing/theses/{slug}`. Extractable
(the LLM mines claims into facts).
- `bet_resolution_log` (NEW) — outcome record for a thesis's bet. FK
to a take row via take_id; carries resolved_outcome + resolved_at +
learned_pattern. Files at `investing/bets/{YYYY-MM}/{slug}`.
No new cycle phases — consumes the existing
extract_facts/propose_takes/grade_takes/calibration_profile loop. Three
calibration domains: `deal_success` (scalar_brier over deal-attached
takes), `founder_evaluation` (scalar_brier over person-attached takes),
`market_call` (weighted_brier over thesis-attached takes; weighted by
conviction so high-stakes misses cost more).
### gbrain-engineer
Bridge-only pack. Declares `learning` page type + reuses base `code`.
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource
(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
each JSONL line as a `learning` page when this pack is active. Three
calibration domains: `architecture_calls` (scalar_brier),
`effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier).
Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred
to v0.42+ — they'll ship when a real user authors the first one (D8).
### gbrain-everything
Meta-pack stacking creator + investor + engineer via the v0.38
`extends` + `borrow_from` chain. Single-active-pack constraint
preserved — this IS the active pack; the registry walks extends +
borrow to materialize the merged view.
Activate via `gbrain config set schema_pack gbrain-everything` and
calibration_profile produces all 7 domain scorecards in one JSONB.
## Calibration profile widening (T10)
Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a
`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared
domain produces a `{n, brier, accuracy, aggregator, page_types,
extras}` entry. Four aggregator algorithms (closed enum):
- **scalar_brier**`AVG(POWER(weight - outcome::int, 2))`. Default for
probabilistic predictions.
- **weighted_brier** — Brier weighted by `ABS(weight - 0.5) * 2`
(conviction proxy). High-conviction misses cost more.
- **count_based** — simple `SUM(hit) / COUNT(*)` accuracy without
Brier. Use when probability isn't natural.
- **cluster_summary** — descriptive rollup (page count + tier
histogram). For domains like `concept_themes` where there's no
binary outcome.
Pack manifests declare domains with `{name, aggregator, page_types}`.
Domain names are OPEN (third-party packs can declare new domain labels
without a gbrain release). Aggregator algorithms are CLOSED (safe SQL
stays in code, validated at pack-load).
## take_domain_assignments table (T1)
New JOIN table (migration v94):
`take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT,
source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id,
domain))`. Multi-domain assignment honest — a take about "Sequoia's
investment in Anthropic" can land in BOTH `deal_success` AND
`market_call` rather than being force-bucketed.
## What this enables for the user
- **Atoms + concepts ship in the binary.** Your OpenClaw's parallel
atom-pipeline-coordinator + atom-backfill-coordinator + concept-
synthesis crons can retire (T12 follow-up). One `gbrain dream` cron
covers everything.
- **gstack learnings reach gbrain.** Engineer-pack-active brains
surface every gstack-logged learning as a queryable page within
seconds of being written.
- **Multi-lens calibration.** Activate gbrain-everything and see how
often you're wrong on deals AND market calls AND architecture
AND effort estimates in one `gbrain calibration --json` call.
- **Lossless OpenClaw migration.** The `markdown-greenfield`
importer (T7, mode='migration') re-ingests existing OpenClaw
pages with permanent slug-keyed idempotency + per-row JSONL audit
+ the `imported_from` marker so extract_atoms + synthesize_concepts
don't re-extract already-atomized material.
## v0.41.2.1 follow-ups (filed in plan)
- Per-page-type `frontmatter_validators` on PageTypeSchema so the
atom_type enum (currently hardcoded in extract_atoms.ts) reads from
the active pack manifest at runtime per D11.
- 3-check quality gate (truism / punchline / entity-page reject) as
a multi-pass extract_atoms refinement.
- Embedding-similarity dedup in synthesize_concepts (currently
exact-string concept ref match only).
- Voice gate integration for T1 Canon narratives.
- op_checkpoint resumability for cross-cycle continuation in both
phases.
- Parity-baseline eval gates against your OpenClaw's existing 13K atoms
+ 11K concepts on a 500-page sample subset.
-246
View File
@@ -1,246 +0,0 @@
# Pack-Upgrade Mechanism (v0.41.22)
> How `gbrain-base@1.x → gbrain-base-v2@1.0.0` (and any future pack
> succession) wires through the onboard cathedral.
## The contract
A schema pack manifest can declare a `migration_from` field:
```yaml
api_version: gbrain-schema-pack-v1
name: gbrain-base-v2
version: 1.0.0
migration_from:
pack: gbrain-base
version: "1.x"
```
When this declaration is present + a `mapping_rules:` block is
populated, the pack registers itself as the successor to
`(parent_pack, version_range)`. Any brain whose active pack matches
that tuple lights up the `pack_upgrade_available` onboard check.
## End-to-end flow
```
┌────────────────────────────────────────────────────────────────┐
│ PACK AUTHORING │
│ │
│ Author declares: migration_from: {pack: P, version: R} │
│ + mapping_rules: [retype/page_to_link/page_to_alias] │
│ Pack ships bundled OR via ~/.gbrain/schema-packs/<name>/ │
└──────────────────────────┬─────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ ONBOARD CHECK DISCOVERY │
│ │
│ checkPackUpgradeAvailable(engine) at src/core/onboard/ │
│ checks.ts: │
│ 1. Read engine.getConfig('schema_pack') for dbConfig tier │
│ 2. loadActivePack({cfg: null, remote: false, dbConfig}) │
│ 3. findPackSuccessors(active.name, active.version) │
│ → walks BUNDLED_PACK_NAMES + ~/.gbrain/schema-packs/ │
│ → matches via _versionRangeMatches(version, range) │
│ → returns ResolvedPack[] sorted by successor version │
│ 4. If successors.length > 0, emit OnboardCheckResult │
│ with RemediationStep targeting `unify-types` handler │
│ + protected: true (D17 → manual_only via render │
│ allowlist) │
└──────────────────────────┬─────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ USER DECIDES │
│ │
│ gbrain onboard --check shows finding │
│ gbrain onboard --check --explain shows per-cluster narrative │
│ User reviews; if OK, runs: │
│ gbrain jobs submit unify-types --allow-protected \ │
│ --params '{"target_pack":"gbrain-base-v2"}' │
│ (Autopilot never auto-fires this; manual_only) │
└──────────────────────────┬─────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ HANDLER EXECUTION (src/core/schema-pack/unify-types-handler.ts) │
│ │
│ 1. Preflight: load target pack; assert mapping_rules present │
│ 2. Stats snapshot (pre-state for celebration) │
│ 3. Acquire gbrain-unify db-lock (60min TTL) │
│ 4. Apply phases (4): │
│ a. Explicit retype rules (chunked UPDATE 1000/batch) │
│ - frontmatter.legacy_type ALWAYS preserved (D8) │
│ - frontmatter.subtype stamped when subtype set │
│ b. Catch-all retype: synthesize per-unknown-type rule │
│ excluding declared types + explicit targets + page_to_ │
│ link/alias sources (D12 + critical bug fix) │
│ c. Page-to-link: parse body+frontmatter, insert link row, │
│ soft-delete source page (per-page atomicity per F7) │
│ d. Page-to-alias: insert slug_aliases row, soft-delete │
│ source page (NO rewriteLinks per D15) │
│ 5. Final sync: path-prefix typing for residual UNTYPED rows │
│ 6. ACTIVE-PACK FLIP (D13): │
│ - engine.setConfig('schema_pack', target_pack) │
│ - saveConfig({...existing, schema_pack: target_pack}) │
│ 7. Verify: re-run stats; warn if ≤ declared + 5 violated │
│ 8. Celebration summary to stderr + audit JSONL │
│ 9. Release db-lock │
└──────────────────────────┬─────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ POST-UPGRADE STATE │
│ │
│ • pages.type updated with canonical types │
│ • frontmatter.legacy_type preserved for rollback │
│ • slug_aliases populated for old-slug → canonical lookup │
│ • links table has new partner_of / relates_to rows │
│ • Source pages soft-deleted (72h TTL for restore) │
│ • Active pack flipped to target_pack │
│ • Next gbrain onboard --check shows ok │
└────────────────────────────────────────────────────────────────┘
```
## Version-range semantics
`migration_from.version` accepts three shapes:
| Form | Matches |
|------|---------|
| `1.0.0` (exact literal) | `1.0.0` only |
| `1.x` (major wildcard) | `1.0.0`, `1.5.2`, `1.99.99` |
| `1.0.x` (minor wildcard) | `1.0.0`, `1.0.5`, `1.0.99` |
`*` is accepted as an alias for `x`.
Implementation: `_versionRangeMatches(version, range)` in
`src/core/schema-pack/load-active.ts`. Pinned by
`test/schema-pack-find-pack-successors.test.ts`.
## findPackSuccessors discovery
Walks `BUNDLED_PACK_NAMES` (currently `gbrain-base`,
`gbrain-recommended`, `gbrain-creator`, `gbrain-investor`,
`gbrain-engineer`, `gbrain-everything`, `gbrain-base-v2`). For each
candidate ≠ the active pack name, loads the manifest via
`loadActivePack({ perCall: candidate })`, checks
`migration_from.pack === activeName && _versionRangeMatches(activeVer,
migration_from.version)`. Returns matching packs sorted by version
descending.
v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed
packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the
filesystem-scan cost needs the cache invalidation strategy from
`registry.ts`).
## The manual_only apply policy
The shipped onboard contract has 3 apply_policy values:
| Policy | Meaning |
|--------|---------|
| `auto_apply` | Autopilot runs unattended |
| `prompt_required` | Autopilot in `--auto-with-prompt` mode prompts user |
| `manual_only` | Autopilot NEVER auto-fires; user must explicitly submit |
`pack_upgrade_available` emits a `RemediationStep` with `protected:
true` + `job: 'unify-types'`. `toOnboardRecommendation` in
`src/core/onboard/render.ts` maps this to `manual_only` via the
`MANUAL_ONLY_PROTECTED_JOBS` allowlist (which also contains
`extract-takes-from-pages` per v0.41.18 A12+A24).
Rationale: pack upgrades change the brain's taxonomy. Taxonomy is a
user judgment call — not autopilot's call. Even with `--auto-with-
prompt`, prompting the user to confirm a pack upgrade mid-tick is the
wrong UX (the user came to fix orphans, not to be interrupted with
"hey want to migrate your taxonomy?"). Explicit submission is the
right boundary.
## Authoring a successor pack
Minimal example for an academic-research brain that adds a
`researcher` canonical:
```yaml
api_version: gbrain-schema-pack-v1
name: gbrain-academic-v1
version: 1.0.0
description: Academic research brain — adds researcher canonical
gbrain_min_version: 0.42.0
extends: null
migration_from:
pack: gbrain-base-v2
version: "1.x"
page_types:
# Inherit gbrain-base-v2's 15 types here (or use extends to merge
# automatically once v0.43+ extends-chain composition lands)
- { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true }
- { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true }
# ... all 13 other v2 canonicals ...
- { name: note, primitive: concept, path_prefixes: [notes/], extractable: true }
# Academic addition:
- name: researcher
primitive: entity
path_prefixes: [researchers/]
aliases: [academic, professor, scholar]
extractable: false
expert_routing: true
mapping_rules:
# All v2 mapping rules (copy from v2 yaml)
# ... ~40 rules ...
# Custom: relocate v2-tagged academics to researcher
- { kind: retype, from_type: person, to_type: researcher, path_filter: 'researchers/%' }
# Catch-all
- kind: retype
from_type: "*unknown*"
to_type: note
subtype_field: legacy_type
subtype: "*original_type*"
```
Drop at `~/.gbrain/schema-packs/gbrain-academic-v1/pack.yaml`.
Discoverable via `gbrain schema list`. Activatable via
`gbrain schema use gbrain-academic-v1`. Once active, the
`pack_upgrade_available` check fires for any brain on
`gbrain-base-v2@1.x` and surfaces a `unify-types` RemediationStep
targeting your pack.
## Lock + concurrency
`gbrain-unify` is a dedicated `gbrain_cycle_locks` row name (60min
TTL). The handler acquires it before any apply phase + releases in
`finally`. Two simultaneous `gbrain jobs submit unify-types`
invocations: second one fails fast at lock acquisition with a clear
error. Same pattern as `gbrain-sync` (v0.22.13 PR #490).
## Audit trail
Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl`
(ISO-week rotation, mirrors existing audit channels). Records: pack
identities (before + after), per-phase counts (would_apply + applied),
warnings, completion timestamp. Privacy: page slugs are NOT logged in
bulk (only the per-rule sample_slugs[≤10]); for forensic debugging
add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
## What's NOT yet supported
- Subprocess sandbox for the publish-gate (v0.43+ TODO)
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
## Reference
- Pack file: `src/core/schema-pack/base/gbrain-base-v2.yaml`
- Manifest extension: `src/core/schema-pack/manifest-v1.ts`
- Successor walker: `src/core/schema-pack/load-active.ts:findPackSuccessors`
- Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable`
- Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS`
- Handler: `src/core/schema-pack/unify-types-handler.ts`
- Migration: `src/core/migrate.ts:105` (slug_aliases table)
- Type taxonomy doc: `docs/architecture/type-taxonomy.md`
- Skill: `skills/schema-unify/SKILL.md`
-230
View File
@@ -1,230 +0,0 @@
# Schema Packs
A schema pack tells gbrain what shape your brain takes — which directories
exist, what types live in them, how the agent should infer types from
paths, and which link verbs connect what to what. The schema pack is the
**dynamic, always-consulted artifact** every skill reads when filing,
querying, or routing experts. It is the single source of truth for
"what's in your brain."
The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the
user-facing reference; for implementation details see
`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in
`src/core/schema-pack/`.
## What ships in the box
Two bundled packs:
- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior
byte-for-byte. Existing brains see zero behavior change after upgrade.
Covers: person, company, deal, meeting, project, place, concept, writing,
analysis, guide, hardware, architecture, etc. (the original
`ALL_PAGE_TYPES` list).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional
directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal,
meeting, concept, project, source, daily, personal, civic, original,
place, trip, conversation, writing. If you like the documented
operational-brain pattern, activate this with:
```bash
gbrain schema use gbrain-recommended
```
Plus user-installed packs at `~/.gbrain/schema-packs/<name>/pack.yaml`
that you author with `gbrain schema init` or `gbrain schema fork`.
## CLI surface
Five inspection verbs (shipped in v0.38):
```bash
gbrain schema active # show resolved pack + which tier set it
gbrain schema list # list bundled + installed packs
gbrain schema show # pretty-print the active pack
gbrain schema validate # validate a manifest's shape
gbrain schema use <pack> # activate a pack (writes ~/.gbrain/config.json)
```
Eight authoring + discovery verbs (shipped in v0.39):
```bash
gbrain schema detect # propose types matching brain shape
gbrain schema suggest # LLM-refined proposals on top of detect
gbrain schema review-candidates # promote / rename / ignore candidates
gbrain schema review-orphans # surface pages with no matching type
gbrain schema init <name> # scaffold a stub pack (experimental)
gbrain schema fork <a> <b> # copy + rename a pack (experimental)
gbrain schema edit <name> # surface the pack path (experimental)
gbrain schema diff <a> <b> # set-diff two packs (experimental)
gbrain schema graph # ASCII type listing (experimental)
gbrain schema lint # flag duplicates + missing prefixes
gbrain schema explain <type> # plain-English type description (experimental)
gbrain schema downgrade --to <p> # restore previous pack (recovery)
gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry)
```
The verbs marked `experimental` are demand-gated per D14: their usage is
tracked via T15's schema-events audit, and v0.40+ retro decides whether
to deprecate any that stay <5% usage.
## Resolution chain (7 tiers)
When the engine decides "which pack is active for this query?", it walks
this chain top-down. First match wins.
| Tier | Source | Notes |
|------|--------|-------|
| 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. |
| 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. |
| 3 | Per-source DB config key `schema_pack:source:<id>` | New in v0.38. |
| 4 | Brain-wide DB config key `schema_pack` | |
| 5 | `gbrain.yml schema:` section | Repo-checked. |
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. |
| 7 | Default: `gbrain-base` | Always present. |
## How the agent uses the active pack
Every read + write path consults the active pack at runtime:
- **`parseMarkdown`** infers page `type` from path prefixes declared in
the active pack (`page_types[].path_prefixes`). Without an active pack
threaded, falls back to the legacy hardcoded `inferType()` so the
byte-for-byte parity gate stays green.
- **`whoknows` / `find_experts`** scopes candidates to `expert_routing:
true` types in the active pack.
- **`extract_facts`** runs only on `extractable: true` types.
- **`enrichment-service`** routes person/company enrichment based on the
pack's primitive declarations.
- **Search hybrid cache** (`knobsHash`) folds in pack name + version
(v0.39 T21). A cache row written under pack A is unreachable when pack
B is active. Cross-pack contamination is structurally impossible.
## The magical moment (T2-T4 + T10)
Persona A (Notion refugee) installs gbrain, imports her exports, and the
brain looks unfamiliar — the default `gbrain-base` pack expects
`people/`, `companies/`, etc., but her files live under `Projects/`,
`Reading/`, `Daily Notes/`. The friction signal fires in two places:
1. **Import warn (T7):** the end of `gbrain import` prints
`[schema] X of Y pages (Z%) have no type matching the active schema
pack. Run gbrain schema detect to propose a pack matching your
content shape.`
2. **`gbrain doctor` schema_pack_consistency check** keeps surfacing
the warning persistently after the import session ends.
She runs the magical moment:
```bash
gbrain schema detect # heuristic clustering on her actual shape
gbrain schema suggest # LLM-refined proposals
gbrain schema review-candidates # human gate on promotion
gbrain schema review-candidates --apply Projects/ # accept
```
The agent (via the new EIIRP skill) automates phases 1-3 of this for any
significant work session. The brain's schema becomes a living artifact
the agent maintains, not a hardcoded ceremony the user authors.
## Authoring your own pack
```bash
gbrain schema init my-pack # scaffolds ~/.gbrain/schema-packs/my-pack/pack.yaml
$EDITOR ~/.gbrain/schema-packs/my-pack/pack.yaml
gbrain schema validate my-pack # check shape
gbrain schema use my-pack # activate
gbrain schema active # confirm
```
A minimal pack:
```yaml
api_version: gbrain-schema-pack-v1
name: my-pack
version: 0.0.1
gbrain_min_version: 0.39.0
extends: gbrain-base # inherits everything from base; add overrides below
description: |
My personal pack.
page_types:
- name: project-x
primitive: entity
path_prefixes:
- Projects/
aliases: []
extractable: false
expert_routing: false
# Add more types here. Each maps a path prefix to a primitive +
# opt-in flags. See src/core/schema-pack/base/gbrain-recommended.yaml
# for a worked example.
link_types: []
takes_kinds: [fact, take, bet, hunch]
borrow_from: []
frontmatter_links: []
enrichable_types: []
filing_rules: []
```
## Recovery + revert
The single-PR cathedral is hard to revert atomically. Per codex finding
#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore
the active-pack config field:
```bash
gbrain schema downgrade --to gbrain-base
# OR auto-detect previous from ~/.gbrain/schema-pack-history.jsonl:
gbrain schema downgrade
```
**Code revert alone is NOT sufficient.** The full revert procedure:
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
v0.39-typed pages that no longer have a matching type in the active
pack.
The cache + eval rows that pack-aware code wrote are isolated by the
`knobsHash` pack-folding (T21) — they become unreachable under the
restored pack so no eviction is needed.
## Distribution
`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as
`.gbrain-skillpack` tarballs (T14 artifact abstraction). The
discriminator is `api_version` in the manifest:
- `gbrain-schema-pack-v1` → schemapack
- `gbrain-skillpack-v1` → skillpack
Both install via the same scaffold + copy path; install targets are
`~/.gbrain/schema-packs/<name>/` and `~/.gbrain/skillpacks/<name>/`
respectively.
Publication to the public registries (`garrytan/gbrain-schema-registry`,
`garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR
workflow as v0.37 skillpack publishing.
## What's deferred to v0.40+
- **Per-source pack federation across mounts.** A query crossing multiple
sources currently rejects with `permission_denied` when those sources
have divergent active packs (T19 + codex finding #2). The v0.40+ work
computes a true per-source closure via the existing
`buildSourceClosureCte` engine surface.
- **`extends` chain semver compatibility checks** between pack versions.
- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack
can declare "I work best with these primitives present in your pack."
- **Live schema migration helpers** — when you add a type, auto-suggest
backfill of existing pages.
- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the
full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+
retro reads T23 usage telemetry to decide which to deprecate.
See `TODOS.md` v0.40+ section for the full deferred list.
@@ -1,54 +0,0 @@
# `gbrain serve``gbrain sync` concurrency (PGLite)
**Short version: on a PGLite brain, stop `gbrain serve` before a large sync.**
## Why
PGLite is a single-writer embedded Postgres (WASM). A running `gbrain serve`
(stdio or HTTP MCP) holds an open PGLite connection on the brain's data
directory. `gbrain sync` needs to write to that same data directory. The two
contend for PGLite's single-writer connection / write-lock — **this is NOT the
`gbrain-sync` advisory lock** (that's a separate, DB-row coordination lock for
two concurrent *syncs*). Confusing the two sends you debugging the wrong surface.
Symptoms of serve↔sync contention on PGLite:
- `gbrain sync` blocks acquiring the PGLite write lock, or makes very slow
progress, while a `gbrain serve` process is alive on the same brain.
- Killing stale `gbrain serve` MCP processes frees the lock and sync proceeds.
## What to do
1. Stop any `gbrain serve` process for this brain before a large sync:
```bash
pkill -f 'gbrain serve' # or stop your MCP client / Claude Desktop / Cursor
gbrain sync --no-pull --no-embed --yes
```
2. Restart `gbrain serve` after the sync completes.
This contention does **not** apply to the Postgres engine — Postgres tolerates
concurrent connections, so `serve` and `sync` can run simultaneously there.
## Diagnosing a sync hang
If a sync wedges (no progress, high CPU), re-run with the per-file begin trace
so the stalling file is named:
```bash
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
```
The last `[sync] begin import: <path>` line with no following completion is the
file being processed when the hang occurred. Under `--workers >1` / `--all`,
the stuck file is in the set of begin-lines without a matching completion.
If you suspect a schema-pack regex is the cause (a pack with a
catastrophic-backtracking `inference.regex`), complete the sync with the pack
disabled and re-run extraction afterward:
```bash
gbrain sync --no-schema-pack --no-pull --no-embed --yes
```
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
-198
View File
@@ -1,198 +0,0 @@
# System of record
**The GitHub repo (markdown + frontmatter) is the system of record.
The Postgres/PGLite database is a derived cache. We do not back up
the database — we rebuild it from the repo.**
This document is the canonical reference for that contract. Every code
path that writes user-knowledge state should match the pattern
described here. The CI gate at `scripts/check-system-of-record.sh`
enforces it programmatically.
## Why this matters
The DB is a derived index over the markdown content. It exists to make
search fast, to dedup embedding-similar claims, to materialize the
cross-page graph. None of that data is irreplaceable — as long as the
markdown is intact, `gbrain sync && gbrain extract all` rebuilds the
entire DB from scratch.
This means:
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
- **Privacy is in your hands.** Sensitive entity pages can be
gitignored (via `gbrain.yml` `db_only` paths or per-page) and they
stay on disk but not in git. The fence respects whatever git
tracking choice you make at the page level.
- **Cross-agent collaboration is possible.** Multiple agents can write
to the same brain because the fence is the merge point, not the DB.
Git handles concurrent edits the way git handles concurrent edits.
## The three categories
Every table in the gbrain schema belongs to exactly one of three
categories. The category determines how it gets rebuilt during
disaster recovery.
### FS-canonical (markdown is the source of truth)
These are user-authored knowledge. The DB row is a derived index over
the markdown — wipe the table and `gbrain extract` rebuilds it
identically. The CI gate keeps direct DB writes from drifting away
from the markdown contract.
| Category | How it's stored in markdown | Derived DB table | Reconciler |
|---|---|---|---|
| **Takes** (incl. hunches, bets) | `## Takes` fenced table between `<!--- gbrain:takes:begin -->` / `:end -->` markers | `takes` | `extract takes` |
| **Facts** | `## Facts` fenced table between `<!--- gbrain:facts:begin -->` / `:end -->` markers | `facts` | `extract_facts` cycle phase |
| **Links** | Inline `[text](slug)` / `[[slug]]` in markdown body + frontmatter `direction: incoming` | `links` | `extract links` |
| **Timeline** | `## Timeline` section after `<!-- timeline -->` sentinel | `timeline_entries` | `extract timeline` |
| **Tags** | Frontmatter `tags:` YAML array | `tags` | `importFromFile` (reconciles per-page on import) |
| **emotional_weight** | Recomputed from takes + tags | `pages.emotional_weight` (signal column) | `recompute_emotional_weight` cycle phase |
| **synthesis_evidence** | FK into `takes` rows (`slug#N`) inside synthesis pages | `synthesis_evidence` | `extract takes` (transitively) |
### Derived from FS but not user-authored
These hold derived state that's automatically reconstructible from the
markdown but not directly authored as markdown by the user. The
chunker + embedder rebuild these on import.
| Table | Source | Notes |
|---|---|---|
| `pages` | The markdown file as a whole | One row per file; `compiled_truth` + `frontmatter` come from parse |
| `content_chunks` | `pages.compiled_truth` after chunker strip | Re-chunked on content_hash change; embedded via configured model |
| `page_versions` | Each `pages` UPDATE | Audit history; rebuildable in principle but not in practice |
### DB-only by design (named exceptions)
These hold runtime / infrastructure state that's intentionally not in
the repo. The architectural rule still holds — these aren't
"user knowledge" — but they're DB-only by design.
| Category | Why it's OK to be DB-only |
|---|---|
| `raw_data` | Webhook/transcript sidecars; not user-authored knowledge. |
| `subagent_messages` / `subagent_tool_executions` / `subagent_rate_leases` | Runtime job state. Replay-only, not persistent knowledge. |
| `oauth_clients` / `oauth_tokens` / `access_tokens` | Credentials. Not in source control by definition. |
| `mcp_request_log` | Audit trail. Volatile by design. |
| `minion_jobs` / `minion_inbox` / `minion_attachments` | Job queue. Restarts re-enqueue or drop. |
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
A new derived table that holds user-knowledge MUST land FS-first.
If you're tempted to add one as "DB-only for now," the structural
question is: does it belong in this DB-only-by-design list? If not,
it's FS-canonical and needs a fence (or frontmatter field) plus a
reconciler.
## The privacy boundary
Private knowledge in a fence still lives in the markdown file. If the
user commits the page to git, the private data lands in git too. This
is the existing operational model — we don't infer git policy.
For untrusted readers (remote MCP, subagent), the v0.32.2 release ships
a 3-layer strip:
1. **Layer A (chunker):** `src/core/chunkers/recursive.ts` calls
`stripFactsFence({keepVisibility: ['world']})` + `stripTakesFence`
before chunking. Private fact text never reaches
`content_chunks.chunk_text`, embeddings, or search results.
2. **Layer B (get_page):** when `ctx.remote === true`, the response
body has both fences stripped (private rows from facts; entire
takes fence). Local CLI (`ctx.remote === false`) sees the full
fence.
3. **Layer C (git tracking):** the user decides whether to commit the
entity page. `gbrain.yml` `db_only` paths are gitignored
automatically; per-page choices via the user's normal git workflow.
For universally-private entities (a friend's name, an investor's
internal notes), mark the entity page's directory as `db_only` in
`gbrain.yml`. The file stays on disk but never lands in git.
## The forget contract
`gbrain forget <id>` and the MCP `forget_fact` op rewrite the fence
row with strikethrough + `valid_until = today` + `context: "forgotten:
<reason>"`. The DB's `expired_at = valid_until + now()` derivation
reconstructs the forget state on every rebuild because the fence is
canonical.
Strikethrough has two semantics distinguished by context:
- `~~claim~~` + `context: "superseded by #N"` → row was replaced by
a newer row in the same fence
- `~~claim~~` + `context: "forgotten: <reason>"` → row was retracted
via the forget op
Both encodings keep the row in the markdown for audit history. To
permanently delete a fact, edit the fence directly in markdown and
remove the row. The next `extract_facts` cycle wipes the DB row.
## Disaster recovery
The promise the rule makes:
```bash
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
# Counts match
gbrain stats > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt
```
The invariant E2E test at `test/e2e/system-of-record-invariant.test.ts`
exercises this exact flow on every CI run.
## Rule for new code
When you add a new user-knowledge category:
1. **Define the markdown shape.** Fence (`<!--- gbrain:NAME:begin
--> ... :end -->` table) or frontmatter field.
2. **Build a parser** that produces structured data from markdown.
See `src/core/fence-shared.ts` for the shared primitives.
3. **Build a writer** that round-trips: parse + edit + render produces
byte-identical markdown for identical input.
4. **Add the engine method** that takes parsed data and stamps a
derived table. The method gets an entry in the CI gate's
banned-direct-call list.
5. **Add a reconciler:** a cycle phase that walks pages, parses the
fence, and rebuilds the derived table from scratch. The reconciler
is the only legitimate call site for the engine method;
`// gbrain-allow-direct-insert: <reason>` annotates it explicitly.
6. **Add a round-trip test** in `test/e2e/system-of-record-invariant.test.ts`
that proves DELETE + reconcile rebuilds the table byte-identically.
The CI gate at `scripts/check-system-of-record.sh` fails any PR that
adds a new direct call to a derived-table writer outside the
reconciler / migration layer without the explicit allow-list comment.
## Related
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
and round 2 finds)
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
- `scripts/check-system-of-record.sh` — the CI gate that enforces
the rule
-403
View File
@@ -1,403 +0,0 @@
# GBrain Deployment Topologies
GBrain supports three deployment shapes. They compose: a single user can mix
all three on the same machine without conflict, because every shape resolves
to "which `~/.gbrain/config.json` is active right now?" and `GBRAIN_HOME`
controls that selection.
This page covers the three topologies, when each fits, and concrete setup
recipes. Pair this doc with `docs/architecture/brains-and-sources.md` (which
covers the in-brain organization axes) — that doc is about WHICH database;
this doc is about WHERE that database lives.
## Quick decision tree
```
"I'm setting up gbrain..."
Just for me, on one machine? ─── yes ───▶ Topology 1 (single brain)
no
Will a remote machine host the brain
while my agent runs locally? ──── yes ───▶ Topology 2 (cross-machine thin client)
no
Multiple Conductor worktrees that
shouldn't share a code index? ─── yes ───▶ Topology 3 (split-engine)
```
Topologies 2 and 3 stack: a thin-client install can also host per-worktree
code engines, and a per-worktree code engine can also point its artifact
brain at a remote server.
## Topology 1 — Single brain (today's default)
```
┌────────────────┐
│ one machine │
│ ┌──────────┐ │
│ │ gbrain │──┼──→ ~/.gbrain/ → PGLite or Supabase
│ │ CLI │ │
│ └──────────┘ │
└────────────────┘
```
What you get: one local DB (PGLite for small brains, Supabase for ~1000+
files). All commands work directly against it. `gbrain serve` exposes it
to a single agent over MCP.
When it fits: solo use, single machine, one agent, no Conductor parallelism.
This is the default; `gbrain init` (no flags) gives you this.
Setup:
```
gbrain init # interactive — defaults to PGLite
gbrain init --pglite # explicit local
gbrain init --supabase # remote Supabase (recommended for 1000+ files)
```
Nothing else here is special. The other two topologies are variations on
"who owns the DB" and "how does the agent talk to it."
## Topology 2 — Cross-machine thin client
```
┌────────────┐ ┌──────────────────┐
│ neuromancer│ │ brain-host │
│ ┌────────┐ │ HTTP MCP / OAuth │ ┌────────────┐ │
│ │ Hermes │─┼───────────────────→│ │ gbrain │──┼──→ Supabase
│ │ agent │ │ │ │ serve --http│ │
│ └────────┘ │ │ └────────────┘ │
│ │ │ (with autopilot)│
│ no local │ │ │
│ gbrain DB │ │ │
└────────────┘ └──────────────────┘
```
What you get: the agent on one machine ("neuromancer") consumes a brain
hosted on another machine ("brain-host") over HTTP MCP with OAuth. The
agent's machine has NO local engine. All queries, searches, embeddings,
and indexing happen on the host.
When it fits:
- Heavy brain (Supabase + autopilot) lives on a beefy machine; agents
elsewhere just consume it.
- You want one source of truth across many machines.
- Spinning up a parallel local install would create source-ID contention or
duplicate work.
The thin client's `~/.gbrain/config.json` carries a `remote_mcp` field
instead of a local DB connection:
```jsonc
{
"engine": "postgres", // ignored — never used
"remote_mcp": {
"issuer_url": "https://brain-host.local:3001",
"mcp_url": "https://brain-host.local:3001/mcp",
"oauth_client_id": "neuromancer-...",
"oauth_client_secret": "..." // or set GBRAIN_REMOTE_CLIENT_SECRET
}
}
```
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
`integrity`, `serve`) on a thin-client install with a clear error pointing
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
(OAuth discovery, token round-trip, MCP smoke).
### Setup
**Step 1 — On the host (brain-host):**
```bash
gbrain init --supabase # or --pglite, doesn't matter
gbrain serve --http --port 3001 --bind 0.0.0.0 # v0.34: bind explicitly for remote access
# (defaults to 127.0.0.1 since v0.34)
gbrain auth register-client neuromancer \
--grant-types client_credentials \
--scopes read,write,admin # admin needed for ping/doctor
# v0.34: source-scoped client (write to one source, federate reads across
# multiple sources). Omit both flags for a v0.33-compatible super-client.
gbrain auth register-client neuromancer-dept \
--grant-types client_credentials \
--scopes read,write \
--source dept-x \
--federated-read dept-x,shared,parent-canon
```
The `register-client` command prints a `client_id` and `client_secret`.
Note both. **Scope must include `admin`**`submit_job` (used by
`gbrain remote ping`) and `run_doctor` (used by `gbrain remote doctor`)
both require it.
**Step 2 — On the thin client (neuromancer):**
```bash
gbrain init --mcp-only \
--issuer-url https://brain-host.local:3001 \
--mcp-url https://brain-host.local:3001/mcp \
--oauth-client-id <id> \
--oauth-client-secret <secret>
```
Pre-flight smoke runs three probes (OAuth discovery, token round-trip,
MCP initialize). If any fails, init exits with an actionable error. On
success, `~/.gbrain/config.json` gets `remote_mcp` set and NO local DB
is created.
**Step 3 — Configure your agent's MCP client.**
For Claude Desktop / Hermes / openclaw, add a single MCP server entry
pointing at the host's `mcp_url` with the bearer token from `register-client`.
Example for Claude Desktop's `~/.config/claude/claude_desktop_config.json`:
```jsonc
{
"mcpServers": {
"gbrain": {
"type": "url",
"url": "https://brain-host.local:3001/mcp",
"headers": { "Authorization": "Bearer <client_secret>" }
}
}
}
```
**Step 4 — Verify.**
```bash
gbrain doctor # runs thin-client checks (no local DB needed)
gbrain remote ping # triggers an autopilot cycle on the host (Tier B)
gbrain remote doctor # asks the host to run its own doctor (Tier B)
```
`gbrain sync` and friends will refuse with a clear thin-client error
naming the `mcp_url`. That's the correct behavior — those commands need
a local engine that doesn't exist here.
### Re-run guard
Running `gbrain init` (no flags) on a machine that already has thin-client
config set refuses without `--force`. This catches the scripted-setup-loop
friction where an orchestrator keeps trying to create a local DB. Use
`gbrain init --mcp-only --force` to refresh thin-client config.
### Storing the OAuth secret
Three storage paths in priority order:
1. **`GBRAIN_REMOTE_CLIENT_SECRET` env var** (preferred for headless agents).
When set, overrides whatever's in the config file. The init flow doesn't
persist a config-file copy when the env var was the source.
2. **`~/.gbrain/config.json` with 0600 perms** (default for interactive
setup; mirrors how Supabase keys are stored today).
3. macOS Keychain integration is on the roadmap; not in v1.
## Topology 3 — Split-engine, per-worktree code + remote artifacts
```
┌──────────────────────────────────────────────────────┐
│ one machine │
│ │
│ ┌─ worktree A ──────────────┐ │
│ │ GBRAIN_HOME=A/.conductor │ │
│ │ gbrain serve --port 3001 │── PGLite (code A) │
│ └───────────────────────────┘ │
│ │
│ ┌─ worktree B ──────────────┐ │
│ │ GBRAIN_HOME=B/.conductor │ │
│ │ gbrain serve --port 3002 │── PGLite (code B) │
│ └───────────────────────────┘ │
│ │
│ ┌─ default ~/.gbrain ───────┐ HTTP MCP / OAuth │
│ │ gbrain serve --port 3000 │──────────────────────→ remote artifacts
│ └───────────────────────────┘ (Supabase / brain-host)
│ │
│ Agent's MCP config (Hermes / Claude Desktop): │
│ mcp__gbrain_code__* → http://localhost:3001 │
│ mcp__gbrain_artifacts__* → http://brain-host/mcp │
└──────────────────────────────────────────────────────┘
```
What you get: each Conductor worktree has its own per-worktree code index
(local PGLite, disposable when the worktree dies). Artifacts (plans,
learnings, transcripts) still live in a shared brain that all worktrees
can see and write to.
When it fits:
- Multiple Conductor worktrees on one machine, all touching the same code
repo.
- You don't want each worktree's code-import to clobber the others'
`last_commit`, source IDs, or symbol tables.
- You DO want artifacts (plans, learnings, retros, transcripts) to be
visible across worktrees.
### How it works
`GBRAIN_HOME` selects which `~/.gbrain` directory is active. Set per worktree:
```bash
export GBRAIN_HOME=/path/to/worktree-A/.conductor/gbrain
gbrain init --pglite
gbrain serve --http --port 3001
```
Each worktree's `gbrain serve` instance binds its own port and indexes its
own DB. Multiple `gbrain serve` processes coexist fine — they're separate
OS processes with separate config and separate connection pools.
The artifact brain runs as a separate `gbrain serve` instance with the
default `~/.gbrain` (no GBRAIN_HOME override) — or remote, in which case
it's a Topology 2 setup.
The agent's MCP client config lists multiple servers, each with a unique
alias. Tool names are namespaced as `mcp__<alias>__<tool>`, so the agent
calls `mcp__gbrain_code__search` for code lookups and `mcp__gbrain_artifacts__search`
for artifact lookups.
### Recommended embedding model
Per-worktree code brains index source files only — no meeting notes,
no people pages, no transcripts. Configure each code brain to use
Voyage's code-tuned model at init time so the config can't be lost to a
later `init` overwrite:
```bash
export GBRAIN_HOME=/path/to/worktree-A/.conductor/gbrain
gbrain init --pglite \
--embedding-model voyage:voyage-code-3 \
--embedding-dimensions 1024
```
`voyage-code-3` is Voyage's code-specialized embedding model with
head-to-head numbers above their general flagships on code retrieval
([voyageai.com/blog](https://voyageai.com/blog)). For already-initialized
brains, switch with the one-command wipe-and-reinit (preserves every
other config field):
```bash
gbrain reinit-pglite --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024
gbrain reindex --code --yes
```
(`gbrain config set embedding_model` is refused as of v0.37.11.0 because
the schema column has to resize alongside the config.)
`gbrain reindex --code` prints a recommendation when the configured
embedding model isn't code-tuned. Suppress with
`GBRAIN_NO_CODE_MODEL_NUDGE=1` if you've intentionally chosen another
provider (single-vendor procurement, compliance, no Voyage key).
### CRITICAL: alias-level routing is manual
Topology 3 has no smart per-tool routing inside gbrain. The agent picks
which brain to query when it picks the alias. **A wrong alias writes (or
queries) the wrong brain silently.** This is intentional (explicit beats
magic) but real:
- If the agent calls `mcp__gbrain_artifacts__put_page` with code-shaped
content, that page lands in the artifact brain forever.
- If the agent calls `mcp__gbrain_code__search` for a question that
actually wants artifact context, the search comes back empty.
Mitigations:
- Name aliases clearly. `gbrain_code` vs `gbrain_artifacts` is unambiguous;
`gbrain` vs `gbrain_local` is not.
- Document in your agent's system prompt or rules which alias goes where.
Be explicit about "code questions → `gbrain_code`; everything else →
`gbrain_artifacts`."
- Pair Topology 3 with `gstack`'s per-worktree wiring (which sets the
alias names + agent rules consistently across worktrees).
### Setup (manual; gstack automates this side)
The gbrain side requires zero new code — `GBRAIN_HOME` and `--port` already
exist. Setup looks like:
```bash
# Start the artifact brain (default ~/.gbrain) on port 3000
gbrain serve --http --port 3000 &
# Start a per-worktree code brain on port 3001
export GBRAIN_HOME=/path/to/worktree-A/.conductor/gbrain
gbrain init --pglite
gbrain serve --http --port 3001 &
unset GBRAIN_HOME
```
Then configure the agent's MCP config with two entries (different aliases,
different ports). For Claude Desktop:
```jsonc
{
"mcpServers": {
"gbrain_artifacts": {
"type": "url",
"url": "http://localhost:3000/mcp",
"headers": { "Authorization": "Bearer <token-A>" }
},
"gbrain_code": {
"type": "url",
"url": "http://localhost:3001/mcp",
"headers": { "Authorization": "Bearer <token-B>" }
}
}
}
```
The gstack-side wiring (per-worktree home setup, port allocation, automatic
MCP config generation, gitignore for the per-worktree DB) is in the gstack
repo's setup-gbrain skill — it composes these primitives, gbrain doesn't
have to know about Conductor.
## Combining topologies
The three shapes compose. A single machine can run:
- A thin-client default config pointing at a remote artifact brain
(Topology 2).
- Plus per-worktree code brains under their own `GBRAIN_HOME` (Topology 3).
- Each worktree's `gbrain serve` instance is local; the agent's MCP config
lists them alongside the remote artifact brain.
`GBRAIN_HOME` controls which config file is active for any one CLI
invocation. `gbrain serve --port` controls which port a server listens on.
The agent's MCP client picks the alias and thus the destination per tool
call. There's no global gbrain orchestrator that knows about all of them
simultaneously — that's by design.
## When NOT to use these topologies
- **Don't use Topology 2 if your agent only ever runs on the same machine
as the brain.** A local `gbrain` install + `gbrain serve` (stdio) is
simpler and faster.
- **Don't use Topology 3 if you only have one Conductor worktree at a
time.** Per-worktree engines exist to prevent contention; one-at-a-time
use has no contention.
- **Don't use a `remote_mcp` thin client AND a local engine on the same
machine in the same `GBRAIN_HOME`.** The dispatch guard refuses DB-bound
commands when `remote_mcp` is set. If you genuinely want both modes on
one machine, use `GBRAIN_HOME` to separate them (one home for the thin
client, another for the local engine).
## See also
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
vs sources axes).
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
- `gbrain init --help` and `gbrain auth --help` for command-level details.
- [`docs/tutorials/`](../tutorials/) — end-to-end walkthroughs that combine
these topologies into working setups (company brain, personal brain,
agent integration, etc.).
-177
View File
@@ -1,177 +0,0 @@
# Type Taxonomy (v0.41.22: gbrain-base-v2)
> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor
> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs
> default to `gbrain-base-v2`.
## Why
A production gbrain brain (186K pages) had accreted **94 distinct
`pages.type` values** in 9 clusters of redundancy. The type system is
the foundation for schema packs, search filtering, extract behavior,
enrichment routing, and expert routing. When types are noisy, every
downstream feature degrades:
- **Search filtering is ambiguous**`--type article` misses 2.2K
articles typed as `media/article`, `sources/article`, etc.
- **Enrichment routing is incomplete**`enrichable_types` could only
list a few canonical types; 80+ legacy types meant most pages never
got enriched.
- **Agent confusion** — when ingesting a new article, should it be
`article`, `media/article`, `sources/article`, or `source/article`?
Four reasonable choices, none of them right.
- **Orphan inflation** — 5,521 concept-redirect pages inflated orphan
counts without adding knowledge value.
Issue #1479 catalogues the 9 clusters with exact counts. This doc is
the response: a coherent 14-type taxonomy with subtypes/format/origin
pushed to frontmatter, alias-table rows for redirects, real link-table
rows for edge-shaped pages.
## The 14 canonical types (+ `note` catch-all)
| Type | Primitive | What it holds | Examples |
|------|-----------|---------------|----------|
| `person` | entity | People | Founders, partners, individuals |
| `company` | entity | Companies, products, orgs (subtype-distinguished) | Companies, YC-companies, products |
| `media` | media | Articles, videos, essays, books, podcasts (subtype-distinguished) | Substack posts, YouTube videos, books |
| `tweet` | media | Twitter posts (single/bundle/stub subtype) | Single tweets, threads, bundles |
| `social-digest` | temporal | Period-grouped social summaries (daily/monthly) | X account daily digests |
| `analysis` | media | Research + competitive intel | Market analysis, pricing analysis |
| `atom` | annotation | Knowledge units (extraction/manual/lore subtype) | Extracted facts, manual notes, lore |
| `concept` | concept | Ideas + reference pages | Wiki concepts |
| `source` | media | Transcripts, references | Interview transcripts |
| `deal` | temporal | Investment deals | Term sheets, investments |
| `email` | temporal | Email threads | Email correspondence |
| `slack` | temporal | Slack messages + threads | Slack conversations |
| `writing` | media | Original writing | Drafts, essays in progress |
| `project` | concept | Initiatives, workstreams | Internal projects |
| `note` | concept | **Catch-all** for one-offs (legacy_type preserved) | Memos, anecdotes, insights, etc. |
15 types total (14 canonical + `note`). The catch-all retype rule
binds any uncovered legacy type to `note` with
`frontmatter.legacy_type = <original>` preserved for rollback.
## Subtypes (declared in frontmatter post-unify)
| Canonical | Subtype field | Values |
|-----------|---------------|--------|
| `company` | `subtype` | `company` / `product` / `org` |
| `media` | `subtype` | `video` / `article` / `essay` / `book` / `podcast` / `blog` |
| `tweet` | `subtype` | `single` / `bundle` / `stub` |
| `social-digest` | `subtype` | `daily` / `monthly` |
| `atom` | `subtype` | `extraction` / `manual` / `lore` |
`subtype_field` for retype rules is restricted to an allowlist:
`{subtype, legacy_type, origin, format, kind, period, domain}`. This
prevents third-party packs from injecting `title`, `slug`, or `type`
via mapping_rules (codex D9 security hardening).
## Migration flow
```
gbrain onboard --check # surfaces pack_upgrade_available
gbrain onboard --check --explain # per-cluster narrative dry-run
gbrain jobs submit unify-types \ # PROTECTED + manual_only
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
Handler runs 4 phases:
┌─────────────────────────────────────┐
│ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL)
├─────────────────────────────────────┤
│ Phase 2: Retype explicit rules │ → chunked UPDATE 1000/batch
├─────────────────────────────────────┤
│ Phase 3: Retype catch-all sentinel │ → 'note' with legacy_type
├─────────────────────────────────────┤
│ Phase 4: Page-to-link conversions │ → insert links + soft-delete
├─────────────────────────────────────┤
│ Phase 5: Page-to-alias conversions │ → insert slug_aliases + soft-delete
├─────────────────────────────────────┤
│ Phase 6: Final sync (residual) │ → path-prefix typing
├─────────────────────────────────────┤
│ Phase 7: Flip active pack (D13) │ → engine.setConfig + saveConfig
├─────────────────────────────────────┤
│ Phase 8: Verify + celebrate │ → assert ≤16 types; stderr summary
└─────────────────────────────────────┘
gbrain onboard --check # pack_upgrade_available cleared
# type_proliferation cleared
```
## Rollback paths
Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
The catch-all retype rule (`from_type: '*unknown*'`) handles long-tail
types automatically — any page whose type isn't covered by an explicit
rule AND isn't a page_to_link / page_to_alias source gets retyped to
`note` with `legacy_type` preserved. Guarantees ≤16 distinct types
post-unify on ANY brain.
For brains with substantial custom types that deserve their own canonical
(e.g. `researcher` for an academic brain), the right move is:
1. Fork gbrain-base-v2: `gbrain schema fork gbrain-base-v2 my-pack`
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
--params '{"target_pack":"my-pack"}'`
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
version: "1.x"}` to register itself as a successor — future agents
discovering your pack via `pack_upgrade_available` will offer the
migration.
## Wikilink resolution post-unify
The slug_aliases table IS the resolver (D15: codex outside voice —
don't rewrite body-text wikilinks; the alias table is the right
primitive). Wikilinks like `[[old-redirect-slug]]` keep working post-
unify because:
1. The wikilink resolver short-circuits through
`engine.resolveSlugWithAlias(slug, sourceId)` BEFORE the existing
fuzzy/prefix cascade.
2. The lookup queries `slug_aliases` for any matching alias_slug in
the provided source(s).
3. If found, returns the canonical_slug. The renderer then resolves
the wikilink to the canonical page.
Multi-source ambiguity (same alias_slug in two registered sources)
emits a once-per-process `multi_match` stderr warning and returns the
first match by source array order. Federated reads pass the full
allowed-source array.
## Search ranking signal: alias_resolved_boost
Post-unify, search results whose slug is a canonical_slug in
slug_aliases get a 1.05x score multiplier via the
`applyAliasResolvedBoost` post-fusion stage. Semantic intent: "user
explicitly disambiguated this as canonical, so it should outrank fuzzy
matches that hit aliases by accident."
`SearchResult.alias_resolved_boost` is stamped on touched results for
`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to
invalidate pre-v0.42 cache rows that don't reflect the new stage.
## Reference
- Issue: https://github.com/garrytan/gbrain/issues/1479
- Pack file: `src/core/schema-pack/base/gbrain-base-v2.yaml`
- Pack-upgrade mechanism: `docs/architecture/pack-upgrade-mechanism.md`
- Migration handler: `src/core/schema-pack/unify-types-handler.ts`
- Onboard checks: `src/core/onboard/checks.ts`
- Skill: `skills/schema-unify/SKILL.md`
- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md`
-166
View File
@@ -1,166 +0,0 @@
# gbrain eval suspected-contradictions (v0.32.6)
The contradiction probe samples retrieval results, asks an LLM judge whether
any pair contradicts on a factual claim relevant to the user's query, and
aggregates into a calibrated report. The output is data — the operator
decides what to act on. This doc covers the architecture, severity rubric,
how to interpret the headline number, and when to act.
## Why this exists
gbrain handles contradictions for *curated* pages via compiled-truth-plus-
timeline and source-boost: when `companies/acme.md` says MRR is $2M and a
chat transcript from 2024 says MRR was $50K, the curated page outranks the
chat. `takes.active` filtering hides explicitly-superseded takes. Recency
decay biases ranking toward fresher content per source-tier.
What none of those mechanisms measure: how often do unmarked semantic
contradictions actually surface in retrieval? Without a probe, every
"should we build the bigger swing (chunk-level `revises` field + ranking
change)" decision is vibes. The probe produces evidence.
## Architecture
```
┌──────────────────────────────────────┐
│ gbrain eval suspected-contradictions │
└──────────────────┬───────────────────┘
┌──────────────────▼───────────────────┐
│ For each query: hybridSearch top-K │
│ → cross_slug_chunks + intra_page │
│ chunk-vs-take pairs │
└──────────────────┬───────────────────┘
┌──────────────────▼───────────────────┐
│ Date pre-filter: skip pairs whose │
│ dates are >30d apart (Codex fix: │
│ same-paragraph-dual-date overrides) │
└──────────────────┬───────────────────┘
┌──────────────────▼───────────────────┐
│ Persistent cache lookup │
│ (chunk_a_hash, chunk_b_hash, model, │
│ prompt_version, truncation_policy) │
└────────┬─────────┬────────────────────┘
hit│ │miss
│ ▼
│ ┌─────────────────────────┐
│ │ LLM judge call │
│ │ → JudgeVerdict │
│ │ confidence floor ≥ 0.7 │
│ └─────────┬───────────────┘
│ │
▼ ▼
┌──────────────────────────────────────┐
│ Aggregate per-query + global stats │
│ Wilson 95% CI on headline % │
│ source-tier breakdown │
│ hot pages + resolution proposals │
└──────────────────┬───────────────────┘
ProbeReport JSON
┌──────────────────┼──────────────────────┬───────────────┐
▼ ▼ ▼ ▼
doctor (M1) MCP (M3) synthesize (M2) trend (M5)
surfaces find_contradictions informational persistent
findings op for agents block in prompt tracking
```
## Severity rubric
The judge assigns severity per finding:
| Level | Rubric | Example |
|---|---|---|
| `low` | naming/format differences | "Alice Smith" vs "A. Smith" |
| `medium` | factual values that may be stale | revenue figure, headcount, valuation |
| `high` | identity / structural claims | founder/CEO/CFO role, company status |
Doctor sorts findings by severity DESC. The MCP op accepts a severity filter
so agents can fetch just the high-priority items.
## How to interpret the headline number
The probe outputs `queries_with_contradiction / queries_evaluated` with a
Wilson 95% confidence interval:
```
Queries with >=1 contradiction: 12 / 50 (24%) Wilson CI 95%: 1437%
```
What this says: with 95% confidence, the true rate is between 14% and 37%.
The 24% point estimate is the most-likely-value but bounded by sampling
noise. **`small_sample_note` fires when n < 30** — at that scale the CI is
too wide to act on.
Decision criteria for the bigger swing (chunk-level `revises` field):
| Wilson CI lower bound | What it says | Action |
|---|---|---|
| < 5% | Source-boost + recency-decay + curated pages handle the load | Stop here; this is the right scope |
| 515% | Real but bounded | Operator decides whether the cost justifies the swing |
| > 15% | Real and substantial | Plan the bigger swing in v0.34+ |
## When to act on findings
Each finding ships with a `resolution_command` field — paste-ready:
- `gbrain takes supersede <slug> --row N` — newer take should replace
the older chunk text on the same page (intra_page kind).
- `gbrain dream --phase synthesize --slug <slug>` — compiled_truth for
the curated entity needs an update (cross_slug curated-vs-bulk).
- `gbrain takes mark-debate <slug> --row N` — intentional disagreement
(e.g., two opinions you want to keep both of).
- `# manual review: <a> vs <b>` — judge wasn't sure; operator decides.
Run `gbrain eval suspected-contradictions review --severity high` to
inspect findings without re-running the probe.
## Cost model
Default judge is `claude-haiku-4-5` at ~$1/Mtok in, $5/Mtok out. With
the v0.32.6 truncation at 1500 chars per pair, ~500 input + 80 output
tokens per judge call. Budget cap defaults to $5 in TTY / $1 non-TTY.
- ~$0.0006 per judge call
- ~$0.005 per query (after date pre-filter + cache hits)
- ~$0.50 per 100 queries
The persistent cache means nightly runs against the same query set
pay near-zero on re-runs (until you bump PROMPT_VERSION).
## Trust posture
- Probe never mutates the brain. Runs only read pages/takes/chunks.
Writes go only to `eval_contradictions_runs` and `eval_contradictions_cache`.
- MCP `find_contradictions` is read-scope. NOT in the subagent allowlist —
user-initiated only, not autonomous-action surface.
- Build-fixture script is local-only. The redactor + `isCleanForCommit`
gate makes accidental private-data commits hard, but the operator MUST
inspect every redaction before commit.
## See also
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
+ trend-tracking workflow.
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
six-member verdict enum (`no_contradiction | contradiction |
temporal_supersession | temporal_regression | temporal_evolution |
negation_artifact`) and threaded `pages.effective_date` into the judge
prompt so the probe stops crying wolf on legitimate change-over-time.
v0.35.7 lands the trajectory substrate the probe pointed at:
`gbrain eval trajectory <entity>` shows the chronological typed-claim
history with regressions flagged inline; `gbrain founder scorecard
<entity>` rolls up four signals (accuracy, consistency, growth
direction, red flags) into a stable JSON contract. MCP op
`find_trajectory` (read scope, visibility-filtered for remote callers)
exposes the same data to agents. The probe's `temporal_supersession`
verdict and the consolidate phase's `valid_until` writeback both
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
— the probe still emits paste-ready commands, only `consolidate`
writes `valid_until` (R1+R8 grep guard pins this).
-580
View File
@@ -1,580 +0,0 @@
# Embedder Shootout — May 2026 Eval Plan
**Status:** approved, ready to execute
**Owner:** Garry
**Plan source:** `~/.claude/plans/system-instruction-you-are-working-linear-origami.md` (review log)
**Target wallclock:** ~2 weeks
**Target API spend:** ~$525 (hard cap $700)
## What this is
A head-to-head A/B/C comparison of three embedding providers under v0.35.0.0's new
multi-vendor gateway routing:
- **OpenAI** `text-embedding-3-large` @ 1536 dims
- **Voyage** `voyage-4-large` @ 2048 dims
- **ZeroEntropy** `zembed-1` @ 2560 dims (also 1280 in a Matryoshka ablation)
Each tested with and without the `zerank-2` reranker. Two corpora: public LongMemEval
(500q) and BrainBench in-house (145 relational queries + 50 newly-curated Cat 13
embedder-sensitive queries).
The goal: produce a publishable comparison report that answers "which embedder wins,
and does zerank-2 carry the win for ZeroEntropy" with bootstrap p-values, suitable
for a v0.35.2.0 release-note headline.
## Why this design
Locked decisions from the planning review (see plan file + `GSTACK REVIEW REPORT` at
the bottom of the linked plan):
- **Synthetic-only** — LongMemEval (public) + BrainBench (in-house). No `~/.gbrain` data.
- **Answer-gen mode**`gbrain eval longmemeval` runs the default answer-gen path
(Anthropic Sonnet), then feeds the resulting hypothesis JSONL to LongMemEval's
published `evaluate_qa.py` (OpenAI gpt-4o judge) for real correctness numbers.
`--retrieval-only` is NOT used (would produce an attackable headline; the judge
expects answer text, not retrieval text).
- **`tokenmax` search mode** pinned across all cells (expansion + reranker slot active).
- **Serial execution** in one workspace. Clean rate-limit profile; first-contact run on
ZE wants debuggable signal.
- **7-cell matrix** (no matched-dim cross-vendor row — no shared dim exists across
all three vendors; honest framing is "each vendor at marketed sweet spot").
## Architectural facts that constrain the plan
- `content_chunks.embedding vector(N)` dim is fixed per brain. Per-question PGLite in
LongMemEval makes this free; BrainBench needs separate brain per cell.
- pgvector HNSW caps at **2000 dims** (`PGVECTOR_HNSW_VECTOR_MAX_DIMS` in
`src/core/vector-index.ts:19`). Voyage 2048 and ZE 2560 fall back to exact vector
scan. Helps quality (no HNSW approximation) but adds latency. Footnoted in writeup.
- Reranker disable key is **`search.reranker.enabled false`**, NOT `reranker_model none`.
`tokenmax` mode defaults reranker=true.
- `gbrain/ai/gateway` is NOT exported in v0.35.0.0. PR α exposes it.
## Matrix
| Cell | Embedder | Dim | HNSW | Reranker | Notes |
|---|---|---|---|---|---|
| A0 | `openai:text-embedding-3-large` | 1536 | yes | none | OpenAI baseline |
| A1 | `openai:text-embedding-3-large` | 1536 | yes | `zerank-2` | mixed-vendor |
| B0 | `voyage:voyage-4-large` | 2048 | no (exact) | none | Voyage solo |
| B1 | `voyage:voyage-4-large` | 2048 | no (exact) | `zerank-2` | mixed-vendor |
| C0 | `zeroentropyai:zembed-1` | 2560 | no (exact) | none | ZE embedder solo |
| C1 | `zeroentropyai:zembed-1` | 2560 | no (exact) | `zerank-2` | **ZE full stack** |
| C2 | `zeroentropyai:zembed-1` | 1280 | yes | `zerank-2` | ZE-Matryoshka ablation |
## PR structure — as few as possible
**PR α — gbrain repo: v0.35.1.0 infra.** All gbrain changes bundled. Lands first.
Bisect-friendly commits inside, ship at the very end.
**PR β — gbrain-evals repo: adapter + smoke + curation + eval receipts + writeup.** The
big one. Includes the full eval-run output committed alongside the code that produced
it, plus the comparison writeup. Lands when everything is done.
**PR γ (optional) — gbrain repo: v0.35.2.0 release** that cross-links the gbrain-evals
benchmark in CHANGELOG. Small commit; no code changes.
Total: 2 substantive PRs + 1 optional release commit. **No mid-stream ships.**
## Conductor sessions
Each section below is a self-contained brief. Copy-paste into a fresh Conductor session
to hand off. Each session ends with a clean deliverable.
---
## Session 1 — PR α: gbrain infra (v0.35.1.0)
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from `master`)
**Branch:** `garrytan/v0.35.1.0-infra`
**Wallclock:** ~2h
**API spend:** $0
### What this session ships
Three changes in one PR, bundled so the embedder shootout in gbrain-evals (PR β) has a
clean prereq baseline:
1. Add `voyage:voyage-4-large` ($0.18/M) and `zeroentropyai:zembed-1` ($0.05/M) to the
embedding pricing table. Patch the `gbrain models doctor` cost estimator + test.
2. Expose `gbrain/ai/gateway` in `package.json` exports map so the gbrain-evals
adapters can call `configureGateway({embedding_model, embedding_dimensions, reranker_model})`
from outside the gbrain process.
3. Add `--resume-from <jsonl>` to `gbrain eval longmemeval` so a mid-run abort
(rate-limit, cost-cap, OS interrupt) doesn't lose the cells we already paid for.
Ships at the end as v0.35.1.0.
### Prereqs (verify before starting)
- On gbrain master at v0.35.0.0 baseline. `cat VERSION` shows `0.35.0.0`.
- `bun test` and `bun run verify` both pass on master.
### Commits (bisect-friendly, one feature per commit)
```
1. feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING
- src/core/embedding-pricing.ts: add both entries
- test/embedding-pricing.test.ts: pin both with $0.18 and $0.05
- Verify: bun test test/embedding-pricing.test.ts
2. feat(exports): expose gbrain/ai/gateway with canary test
- package.json: add "./ai/gateway" to exports map
- test/public-exports.test.ts: add canary for configureGateway + embed
- scripts/check-exports-count.sh: 17 -> 18
- Verify: bun run verify
3. feat(eval): add --resume-from <jsonl> to longmemeval
- src/commands/eval-longmemeval.ts: parse flag, skip questions already in input JSONL
- test/eval-longmemeval.test.ts: simulated mid-run abort + resume regression
- Verify: bun test test/eval-longmemeval.test.ts
4. chore: v0.35.1.0
- VERSION: 0.35.1.0
- package.json: 0.35.1.0
- CHANGELOG.md: new entry
- bun install (refresh lockfile)
```
### Verify before /ship
```bash
bun run typecheck
bun run verify
bun test test/embedding-pricing.test.ts test/public-exports.test.ts test/eval-longmemeval.test.ts
```
### Ship
```bash
/ship
```
### Deliverable
- `master` of gbrain at v0.35.1.0
- `gbrain/ai/gateway` reachable from external consumers (verified by canary test)
- `git tag eval-run-v0.35.1.0-baseline` (annotated, names this exact commit)
- `gbrain --version` prints `0.35.1.0`
### Hand-off to Session 2
- gbrain-evals can now `bun update gbrain` to v0.35.1.0
- The tag preserves the exact commit for any future reproducibility need
---
## Session 2 — PR β setup: gbrain-evals adapter + smoke + subset flag
**Repo:** `/Users/garrytan/git/gbrain-evals` (or a fresh Conductor workspace cloned from it)
**Branch:** `garrytan/embedder-shootout`
**Wallclock:** ~3-4h
**API spend:** ~$0.10 (smoke verification calls only)
### What this session ships into PR β (does NOT merge yet)
Wire the harness to drive 3 embedding providers via the newly-exposed gbrain gateway:
1. New typed `EvalAdapterConfig {embedder, dim, reranker?}` passed into each adapter.
2. Rewrite `vector.ts` + `hybrid-rrf.ts` to call `configureGateway()` from
`gbrain/ai/gateway` instead of the hardcoded `gbrain/embedding` import.
3. Critical: hybrid adapter must also route `search.reranker.enabled` (true/false) and
`search.mode` (tokenmax) — codex flagged that the existing hybrid never sets these.
4. New 3-phase smoke harness: wiring (5 queries × embed roundtrip + dim check) +
long-haystack (1 query × 50K-token synthetic haystack) + rerank-payload (1 query
× `topNIn=30`). Exit code is the gate.
5. New `--include-subset <name>` flag on the BrainBench runner (Cat 13 wiring; subset
itself comes in Session 3).
### Prereqs
- Session 1 done. gbrain master at v0.35.1.0.
- API keys present: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
`ZEROENTROPY_API_KEY`. Smoke fails-loud on missing key.
### Commits
```
1. chore(deps): bump gbrain pin to v0.35.1.0
- package.json + bun.lock
- Verify: bun install && bun run typecheck
2. feat(adapter): typed EvalAdapterConfig + gateway swap
- NEW: eval/runner/eval-adapter-config.ts (the type)
- eval/runner/adapters/vector.ts: constructor takes EvalAdapterConfig,
calls configureGateway({embedding_model, embedding_dimensions})
- Drop hardcoded gbrain/embedding import
- Verify: existing vector adapter unit tests still pass
3. feat(adapter): hybrid-rrf wires reranker_enabled + search.mode
- eval/runner/adapters/hybrid-rrf.ts: constructor takes EvalAdapterConfig,
plumbs search.reranker.enabled + search.mode = tokenmax through
- Verify: bun test eval/
4. feat(smoke): 3-phase smoke harness
- NEW: eval/runner/smoke.ts (CLI entry: bun run eval:smoke -- --embedder X --dim Y [--reranker Z])
- Phase 1: 5 queries × embed roundtrip, assert vector dim matches config
- Phase 2: 1 query × synthetic 50K-token haystack, assert no token-limit error
- Phase 3: 1 query × topNIn=30 documents, assert no 5MB payload cap hit
- Non-zero exit on any failure
- Verify: bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
5. feat(runner): --include-subset flag for BrainBench
- eval/runner/multi-adapter.ts: parse flag, filter queries by subset tag
- Subset itself comes in next commit (Session 3)
- Verify: bun run eval:run -- --include-subset cat13-embedder (errors politely because subset file doesn't exist yet)
```
### Smoke verification (run manually before opening PR)
```bash
bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
bun run eval:smoke -- --embedder voyage:voyage-4-large --dim 2048
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560 --reranker zeroentropyai:zerank-2
```
All four MUST exit 0. Reports should print the observed vector dim, matching the
configured dim.
### Open PR β
```bash
gh pr create --base main --title "feat: embedder shootout (adapter + smoke + Cat 13 + eval receipts)" --body "$(cat <<'EOF'
## Summary
v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support. This PR runs a head-to-head A/B/C comparison across OpenAI, Voyage, and ZeroEntropy under the new gateway routing.
This first commit batch lands the harness. Cat 13 curation, Phase 1+2 evals, and the
writeup follow in subsequent commits to this same PR.
## Test plan
- [x] Adapter unit tests pass
- [x] Smoke harness exits 0 against all 3 providers
- [ ] Cat 13 subset committed (Session 3)
- [ ] LongMemEval x 7 cells run (Session 4)
- [ ] BrainBench x 7 cells run (Session 5)
- [ ] Writeup committed (Session 5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
### Deliverable
- PR β open against gbrain-evals `main`, green CI
- Smoke verified against all 3 providers (paste the smoke output in the PR body)
- Branch ready for Session 3 (Cat 13 curation)
### Hand-off to Session 3
- Branch `garrytan/embedder-shootout` exists on origin
- The `--include-subset cat13-embedder` flag is wired but the subset file doesn't exist
yet — that's Session 3
---
## Session 3 — PR β: Cat 13 conceptual-recall curation
**Repo:** `/Users/garrytan/git/gbrain-evals`, branch `garrytan/embedder-shootout` (same as Session 2)
**Wallclock:** ~3-4h (heavily user-interactive; AI proposes, you review each)
**API spend:** $0
### What this session ships into PR β
Hand-curated 50 embedder-sensitive queries from BrainBench's Cat 13 (conceptual recall)
corpus. These are the queries where a graph/keyword adapter would likely miss but a
semantic adapter would find.
Codex flagged the existing 145-query relational corpus as graph/keyword-dominated and
weak for embedder claims. Cat 13 is closer to the embedder-sensitive workload but
needs hand-selection.
### Prereqs
- Session 2 done. PR β open with adapter + smoke + subset flag.
### Workflow
Interactive: Claude proposes queries in batches of 10, you accept/reject/edit each.
1. Claude reads the existing Cat 13 raw query pool:
```bash
ls eval/data/raw/ | grep -i cat13
cat eval/data/raw/cat13-*.json | jq '.'
```
2. Claude proposes 10 candidate queries per batch, each tagged with the inclusion
reasoning ("would a graph adapter miss this?")
3. User accepts/rejects/edits inline. Target: 50 queries × ~5 batches.
4. Claude commits to `eval/data/gold/brainbench-cat13-embedder-subset.json`:
```json
{
"schema_version": 1,
"subset": "cat13-embedder",
"queries": [
{
"id": "cat13-emb-001",
"query": "...",
"relevant_chunk_ids": ["..."],
"inclusion_reason": "paraphrase relationship; graph adapter wouldn't catch the synonym"
}
// ... 49 more
]
}
```
### Commit
```
feat(eval): curate Cat 13 conceptual-recall subset (50 embedder-sensitive queries)
- NEW: eval/data/gold/brainbench-cat13-embedder-subset.json
- Each query tagged with inclusion_reason for future audit
```
### Spot-check before commit
- Pick 5 random queries, run them against a hypothetical graph adapter (e.g. grep on
the relevant terms) and verify they would NOT surface the right chunk.
- Run the same 5 against the existing hybrid adapter and verify they DO.
### Deliverable
- `eval/data/gold/brainbench-cat13-embedder-subset.json` committed to PR β
- Exactly 50 queries
- Spot-check evidence in the commit message
### Hand-off to Session 4
- PR β now has: adapter + smoke + Cat 13 subset
- Ready for the actual eval runs
---
## Session 4 — PR β Phase 1: LongMemEval × 7 cells (overnight)
**Repo:** Same gbrain-evals branch
**Wallclock:** ~10.5h (mostly hands-off, kick off and walk away)
**API spend:** ~$476 (LongMemEval-heavy; 7 × $68/cell)
### What this session ships into PR β
7 LongMemEval scored receipts (one per matrix cell). Each is a JSONL of 500
hypotheses + a JSON file of correctness scores from `evaluate_qa.py`.
### Prereqs
- Sessions 1+2+3 done. PR β has adapter + smoke + Cat 13.
- LongMemEval dataset downloaded (gated HuggingFace; one-time setup).
- `evaluate_qa.py` checked out somewhere (from
https://github.com/xiaowu0162/LongMemEval) with its own venv set up.
- API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
`ZEROENTROPY_API_KEY`.
### Wrapper script
Claude writes `scripts/run-shootout-phase1.sh` in the gbrain-evals branch. Single
entry point that loops the 7 cells serially with smoke gating + cost-cap aborts.
```
NEW: scripts/run-shootout-phase1.sh
- Per cell: gbrain config set (embedder, dim, reranker, search.reranker.enabled, search.mode=tokenmax)
- Per cell: bun run eval:smoke (abort cell on non-zero)
- Per cell: gbrain eval longmemeval ... --output results/longmemeval-{cell}.jsonl
- Per cell: cost-cap check ($90/cell hard stop)
- Per cell: --resume-from existing results/longmemeval-{cell}.jsonl if present
- Logs to results/phase1-run-log.txt
```
### Run
```bash
# Kick off in background; check back in 10-12h
bash scripts/run-shootout-phase1.sh 2>&1 | tee results/phase1-run-log.txt &
```
Use `run_in_background: true` if running through Claude. Check back periodically.
### Scoring (after all 7 cells done)
```bash
for cell in A0 A1 B0 B1 C0 C1 C2; do
python evaluate_qa.py \
--input results/longmemeval-${cell}.jsonl \
--output results/longmemeval-${cell}-scored.json
done
```
Each scored file has correctness %.
### Commits
```
1. feat(scripts): Phase 1 LongMemEval wrapper with smoke gating + cost cap
- NEW: scripts/run-shootout-phase1.sh
2. data(phase1): 7 LongMemEval cells (raw hypothesis JSONL)
- results/longmemeval-{A0,A1,B0,B1,C0,C1,C2}.jsonl
- results/phase1-run-log.txt (run timing + cost ledger)
3. data(phase1): evaluate_qa.py scoring results
- results/longmemeval-{cell}-scored.json × 7
```
### Verify
- Each `longmemeval-{cell}.jsonl` has exactly 500 lines
- Each `hypothesis` field is non-empty AND is actual answer text (NOT retrieval text)
- Each `scored.json` has a `correctness_score` field
### Deliverable
- 7 scored LongMemEval receipts committed to PR β
- Real cost ledger committed alongside (compare against estimate)
### Hand-off to Session 5
- Phase 1 done. Phase 2 (BrainBench, ~3.5h) and writeup remaining.
---
## Session 5 — PR β Phase 2 + writeup + ship
**Repo:** Same gbrain-evals branch
**Wallclock:** ~7h (3.5h BrainBench + 3h writeup + /ship)
**API spend:** ~$56 (BrainBench is cheap)
### What this session ships into PR β
- 7 BrainBench cells (relational corpus + Cat 13 subset)
- Final comparison writeup
- PR β merged
### Prereqs
- Session 4 done. PR β has Phase 1 receipts.
### Phase 2 wrapper script
```
NEW: scripts/run-shootout-phase2.sh
- Per cell: configure provider (same as Phase 1)
- Per cell: bun run eval:run -- --N 10 --include-subset cat13-embedder
--output docs/benchmarks/2026-05-22-{cell}.md
- Cost-cap check
```
### Run
```bash
bash scripts/run-shootout-phase2.sh 2>&1 | tee results/phase2-run-log.txt
```
### Writeup
`docs/benchmarks/2026-05-22-embedder-shootout.md`. Structure:
1. **Headline table** — 7 cells × {LongMemEval correctness %, BrainBench relational MRR + P@5, Cat 13 correctness %, total cost}
2. **Two questions answered:**
- Which embedder wins solo? (A0 vs B0 vs C0)
- Does zerank-2 carry ZE's win? (C0 vs C1 vs A1 vs B1)
- Bonus: does dim matter for ZE? (C1 vs C2)
3. **Paired-bootstrap p-values** per headline pair (methodology in
`gbrain/docs/eval/SEARCH_MODE_METHODOLOGY.md`)
4. **HNSW footnote** — Voyage 2048 and ZE 2560 used exact vector scan; OpenAI 1536
and ZE 1280 used HNSW. Quality is primary, latency is secondary
5. **What this does NOT prove** — synthetic-only, tokenmax-only, no real-brain replay
6. **Recommendation:** explicit NON-recommendation to change `gbrain init` default;
defer to a v0.36.x evidence pass with real-brain replay data
### Commits
```
1. feat(scripts): Phase 2 BrainBench wrapper
- NEW: scripts/run-shootout-phase2.sh
2. data(phase2): 7 BrainBench cells
- docs/benchmarks/2026-05-22-{cell}.md × 7
3. docs(benchmark): embedder shootout comparison writeup
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md
- Bootstrap p-values, HNSW footnote, NOT-in-scope section
```
### Ship
```bash
# Merge PR β to gbrain-evals main
gh pr merge --squash --auto
# Or non-auto if reviewing one more time:
gh pr merge --squash
```
### Deliverable
- PR β merged to gbrain-evals `main`
- Comparison report public at
`gbrain-evals/docs/benchmarks/2026-05-22-embedder-shootout.md`
### Hand-off to Session 6 (optional)
- gbrain-evals master has the full data + writeup
- Ready for a v0.35.2.0 gbrain release that cross-links it
---
## Session 6 (optional) — PR γ: gbrain v0.35.2.0 release
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from master)
**Branch:** `garrytan/v0.35.2.0-benchmark-release`
**Wallclock:** ~30min
**API spend:** $0
### What this session ships
A release-notes-only PR that bumps gbrain to v0.35.2.0 with a CHANGELOG entry
cross-linking the embedder shootout benchmark. Optional — could be folded into the
next routine release if no rush.
### Prereqs
- Session 5 done. gbrain-evals merged with the comparison writeup.
### Commits
```
1. docs(benchmark): mirror embedder shootout summary
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md (slim mirror)
- Cross-link to gbrain-evals canonical version
2. chore: v0.35.2.0
- VERSION: 0.35.2.0
- package.json: 0.35.2.0
- CHANGELOG.md: new entry with the GStack-voice release summary
+ "numbers that matter" table from the benchmark
```
### Ship
```bash
/ship
```
### Deliverable
- gbrain v0.35.2.0 on master
- CHANGELOG entry that drives the release-note headline
---
## Cost ledger (revised, post-review)
| Component | Per cell | × 7 cells |
|---|---|---|
| LongMemEval embed | <$0.05 | <$0.35 |
| LongMemEval Sonnet answer-gen (500q × 2K tokens × $3/M) | $18 | $126 |
| LongMemEval gpt-4o judge (500q × $0.10/q) | $50 | $350 |
| BrainBench relational embed | $0.05-0.18 | <$1 |
| BrainBench Cat 13 answer-gen + judge (50q × $0.14) | $7 | $49 |
| Smoke harness (30 calls/cell) | <$0.10 | <$1 |
| **Total** | **~$75/cell** | **~$525** |
**Hard cap: $700.** Per-cell hard cap: $90 (wrapper aborts cell if exceeded; partial
JSONL preserved for resume).
## Failure modes and recovery
| Failure | Recovery |
|---|---|
| Voyage/ZE 429 rate-limit mid-cell | `gateway._shrinkState` halves safety_factor and retries. Cell continues. |
| ZE 5MB rerank payload cap hit | `applyReranker` fail-opens, returns un-reranked results. Stderr warn. |
| Mid-cell OS interrupt / cost-cap abort | Re-run with `gbrain eval longmemeval --resume-from results/longmemeval-{cell}.jsonl`. Picks up where it left off. |
| `evaluate_qa.py` auth fail | OPENAI_API_KEY check in wrapper aborts before any spend. |
| Adapter typo (bad dim) | `EvalAdapterConfig` runtime assertion at constructor throws AIConfigError. Cell aborts before API call. |
## NOT in scope (deliberate)
- **Real `~/.gbrain` replay** — adds 6-12h wallclock + $40-80 embed. Filed as v0.36.x.
- **All 3 search modes** — pinned to tokenmax. `conservative` + `balanced` are v0.35.3.0
follow-ups if reviewers push back.
- **Matched-dim cross-vendor row** — no shared dim exists across all 3 vendors.
Permanently out.
- **`gbrain eval whoknows` / `cross-modal` / `takes-quality`** — embedding-invariant;
rerunning across embedders produces noise.
- **`gbrain eval code-retrieval`** — code corpus, separate concern.
- **`gbrain eval suspected-contradictions`** — wants a real brain.
- **`gbrain init --recommended` default change** — codex correctly flagged the evidence
base as insufficient. Defer to v0.36.x with real-brain replay data.
## What already exists (reused, not rebuilt)
- `gbrain eval longmemeval` CLI (in-tree, answer-gen mode default)
- gbrain-evals BrainBench runner (`eval:run`) — needs adapter parameterization but
per-cell test plumbing is reused
- Gateway routing for Voyage + ZE (shipped v0.35.0.0)
- Reranker pipeline (`src/core/search/rerank.ts`, fail-open)
- Pricing table (extended, not rebuilt)
- Paired-bootstrap methodology (`docs/eval/SEARCH_MODE_METHODOLOGY.md`)
- LongMemEval published `evaluate_qa.py` (invoked externally, not bundled)
File diff suppressed because it is too large Load Diff
-221
View File
@@ -1,221 +0,0 @@
---
status: ACTIVE
---
# CEO Plan: v0.38 Schema Packs — Bring Your Own Shape
Generated by /plan-ceo-review on 2026-05-19
Branch: garrytan/houston-v1 | Mode: EXPANSION
Repo: garrytan/gbrain
## Definitions (terms used throughout)
- **Primitive** — a named bundle of (default link verbs, default
frontmatter fields, expert-routing flag, enrichment rubric slot).
Five built-in: `entity`, `media`, `temporal`, `annotation`,
`concept`. A pack type extends one primitive by name, inheriting
its defaults, then optionally overriding specific fields. Not a
table shape, not a schema in the SQL sense — a behavioral
template the engine consults at inference and search time.
- **Alias closure** — for read paths, when a pack declares type
`researcher` aliases base type `person`, queries for `researcher`
expand the WHERE clause to `type IN ('researcher','person', + any
other type aliasing person)`. The closure is computed once at
pack load, cached on the pack object, and inlined into search
SQL. Aliasing is one-directional (researcher → person; querying
`person` does NOT surface `researcher` rows unless the inverse is
declared).
- **Pack resolution chain (7 tiers)** — extends model-config's
6-tier pattern. Order: (1) per-call `schema_pack` opt, (2)
`GBRAIN_SCHEMA_PACK` env, (3) per-source `--source <id>` override
via DB config key `schema_pack:source:<id>`, (4) brain-wide DB
config key `schema_pack`, (5) `gbrain.yml schema:` section,
(6) `~/.gbrain/config.json schema_pack`, (7) default `gbrain-base`.
Tier 3 is the new tier introduced in v0.38; tiers 1, 2, 4-7
mirror existing patterns.
## Vision
### 10x Check
The plan as accepted ships a self-EXPANDING engine, not just a
self-describing one. The differences from the baseline plan:
- The brain watches what you create and proposes schema refinements
you didn't think to ask for (`schema suggest`)
- Schema is per-source (ISOLATED reads), so ~/git/brain and
~/git/zion-brain hold different mental models in the same engine
without renames. Cross-source federated reads still see per-source
packs in isolation — a query joining results across mounts does
NOT compute a closure across both packs. Federation (closure
across mounts) is explicitly deferred to v0.39.
- The pack is inspectable: ASCII graph, plain-English explanation,
consistency lint against actual content
- First unknown-type write asks "Add to pack?" with a primitive
inference, instead of silently logging
- Schema packs distribute as `.gbrain-schema` tarballs through the
v0.37 skillpack pipeline; skillpacks rename to `.gbrain-skillpack`
for symmetry. Community schema packs propagate the same way
community skillpacks do.
### Platonic Ideal
A new user clones gbrain and types `gbrain init`. Within 30 seconds
gbrain has read their existing markdown anywhere on disk, proposed a
schema matching their organic shape, asked 3-5 yes/no questions to
refine, and the brain is live. They never author YAML unless they
want to. They can publish their pack as a `.gbrain-schema` tarball
for anyone to install and fork.
The 12-month state: `gbrain init` runs `schema detect` automatically,
proposes a primitive structure, and 90% of users never see the
manifest format. The 10% who want to customize see a clean YAML they
can edit. Community packs cover the long tail of domains.
## Scope Decisions
| # | Proposal | Effort | Decision | Reasoning |
|---|----------|--------|----------|-----------|
| 0C-bis | Approach C (Full Cathedral) | ~4 weeks | ACCEPTED | User explicitly chose the most ambitious of three approaches; ecosystem + engine in one ship |
| D2 | Per-source schema packs | ~1 week | ACCEPTED | User owns two brains today; v0.34.1.0 source-isolation makes the seam architecturally clean |
| D3 | `gbrain schema suggest` (LLM-powered) | ~3-5 days | ACCEPTED | Closes the gap from "what exists" to "what your brain implies"; bounded cost via sampling |
| D4 | `schema graph` + `lint` + `explain` | ~2 days | ACCEPTED | Schema becomes legible and self-documenting; tiny effort, large UX delta |
| D5 | Auto-prompt on first unknown type | ~1-2 days | ACCEPTED | TTY-gated + per-type silenceable; turns lenient-mode from fallback to feature |
| D6-orig | `fork-from <brain-path>` (live-brain) | ~3-5 days | REJECTED | Privacy hazard (read access to whole repo); unclear value vs published tarballs |
| D6-reframed | Skillpack tarball reuse + extension expansion | ~3-5 days | ACCEPTED | Schema packs ship as `.gbrain-schema`; skillpacks gain `.gbrain-skillpack` extension alongside existing `.tgz`; both ride v0.37 pipeline parameterized on manifest discriminator. Extension is the install-time type detector — lets validation route to the right manifest validator before extraction. |
Total budget: **revised 9-11 weeks** (vs ~6.5-7 initial estimate;
spec review surfaced LLM prompt-tuning loops for `schema suggest`,
primitive-inference heuristics for auto-prompt, 7-tier × federated-
read interaction edges, full rename-migration surface, and 400-600
test cases at v0.36/v0.37 scope precedent). If budget pressure
emerges, the safest cuts in order are: D5 auto-prompt (~2 days),
D4 inspect triad (~2 days), reduce examples 7→3 (~3 days), defer
suggest LLM polish to v0.38.1 (~1 week).
## Accepted Scope (added to this plan)
- **Engine layer:** gbrain-base universal starter pack; 5 composable
primitives (entity, media, temporal, annotation, concept); alias
closure for read paths; lenient-by-default with audit for write
paths; strict mode opt-in.
- **Detect layer:** `gbrain schema detect` SQL-driven heuristic
clustering proposing a pack manifest matching brain shape.
- **Suggest layer:** `gbrain schema suggest` LLM-powered refinement
via gateway.chat() over a bounded sample.
- **Inspect layer:** `gbrain schema graph` (ASCII viz),
`gbrain schema lint` (consistency check), `gbrain schema explain
<type>` (plain English).
- **Author layer:** `gbrain schema init/use/fork/edit/validate/
diff/review-candidates` CLI.
- **Source layer:** per-source schema-pack resolution; pack
resolution gets a 7th tier (per-source override before per-brain);
`--source <id>` flag on every relevant command.
- **Auto-prompt layer:** TTY-gated interrupt on first unknown-type
`put_page` with primitive inference; per-type "always silent"
escape hatch.
- **Distribution layer:** `.gbrain-schema` tarball format; rename
skillpacks to `.gbrain-skillpack`; v0.37 skillpack pipeline
parameterized on artifact type (manifest discriminator drives
type-specific validation); both extensions accepted on install
for back-compat.
- **Examples:** 7 example packs in-tree (minimal, person-first,
media-archive, temporal-archive, research-notebook, founder-ops,
personal-archive) explicitly framed as sketches not products.
- **gbrain-base:** byte-for-byte reproduces today's hardcoded
behavior so existing brains see zero change after upgrade.
- **Migrations:** v76 drops `takes.kind` CHECK constraint;
validation moves to runtime against active pack's declared kinds.
- **Doctor checks:** schema_pack_active, schema_pack_consistency,
per-source pack drift.
- **Engine refactor coverage:** the v0.38 plan parameterizes EVERY
hardcoded type-coupling site listed in the original exploration,
not just `takes.kind`. Concretely: `inferType` path-prefix table,
`inferLinkType` regex bank, `FRONTMATTER_FIELD_OVERRIDES` table,
`find_experts` SQL (`type IN (…)`), `whoknows` `DEFAULT_TYPES`,
`enrichment-service` person/company restriction,
`completeness.ts` rubric map, dream-cycle entity-type prompts.
gbrain-base reproduces today's values for each.
- **Cache + rollback story:**
- `query_cache.knobs_hash` (v0.32.3 column) folds `schema_pack`
name + version into the hash so a cache row written under
`vc` is unreachable when `research-state` is active. Cross-
pack contamination structurally impossible.
- `eval_candidates` rows (v0.25.0) gain a `schema_pack` column
so `gbrain eval replay` reproduces the same retrieval space.
Migration v77 adds the column NULL-tolerant; pre-v0.38 rows
fall back to active pack during replay.
- HNSW indexes are pack-agnostic (vector columns don't change
shape across packs); no reindex needed on pack switch.
- Rollback: every `gbrain schema use` operation writes the
previous pack name to `~/.gbrain/schema-pack-history.jsonl`
so `gbrain schema use --previous` is one keystroke. Strict-
mode failures on switch surface the offending pages with
paste-ready "rename type to X" hints before any data
mutation runs. Soft-deletes from autopilot purge are NOT
triggered by pack changes.
- **Test budget:** ~400-600 cases across unit + e2e per the
v0.36/v0.37 precedent. Specifically: ~150 cases for engine layer
+ alias closure, ~50 for detect heuristic accuracy, ~50 for
suggest LLM prompts (hermetic via stubbed gateway), ~30 for per-
source resolution × 7-tier matrix, ~40 for auto-prompt UX
states, ~30 for inspect triad output stability, ~30 for tarball
type-detection + parameterized install, ~50 for migration v76 +
v77 + bootstrap parity, ~50 for examples × byte-for-byte
gbrain-base equivalence regression. **gbrain-base byte-for-byte
parity is a CI gate**, not a hope — pinned by
`test/regressions/gbrain-base-equivalence.test.ts` asserting the
pre-v0.38 hardcoded behavior reproduces from the pack-driven
paths on a fixture brain.
## Deferred to TODOS.md (v0.39+)
- Live-brain `fork-from <brain-path>` (rejected for privacy; revisit
if a sandboxed schema-only extraction path is designed)
- Per-source pack FEDERATION across mounts (a query crossing
multiple sources can use closure over each source's schema; right
now per-source is isolated reads only)
- Schema versioning + semver compatibility checks between pack
versions
- Skillpack ↔ schema-pack cross-reference (a skillpack can declare
"I work best with these primitives present in your pack")
- Live schema migration helpers (when you add a type, auto-suggest
backfill of existing pages)
- Schema diff in PR review (rendering pack changes as human-readable
diffs for community pack PRs)
## Reviewer Concerns (from spec review loop, partially addressed)
- Quality score on first pass: 6.5/10. Issues addressed in this
revision: definitions block (primitive, alias closure, 7-tier
resolution chain), per-source isolation vs federation contradiction
clarified, skillpack extension framing changed from rename to
expansion, full hardcoded-site coverage enumerated, cache +
rollback story added, test budget enumerated, budget revised to
9-11 weeks honestly.
- Issues NOT fully addressed, surfaced for the 11-section review:
- `schema suggest` LLM prompt-tuning iteration budget remains a
range estimate, not a measured number. The 11-section review
should pin a specific eval fixture set (size + diversity) and
a target accuracy threshold before code lands.
- 7-tier resolution × v0.34.1 federated_read OAuth scoping has
edge cases at the intersection that the 11-section review must
enumerate (specifically: an OAuth client with read scope across
federated sources but no source-specific pack override — which
pack drives the alias closure for cross-source queries?).
- The 7→3 example pack reduction is a real cut consideration. The
11-section review should decide whether 7 examples is the right
number or whether 3 + community-derived is more honest.
## Cathedral risks worth surfacing in 11-section review
1. The 7-week budget vs 4-week original ask. If pressure emerges,
D5 (auto-prompt) and D4 (inspect triad) are the safest cuts.
2. v0.37 skillpack registry currently has zero published packs.
The `.gbrain-schema` rename and tarball reuse doubles down on a
distribution layer with no usage signal.
3. Per-source pack resolution adds a 7th tier to the resolution
chain. The model-config 6-tier pattern is already cognitively
dense; tier 7 is an inflection point.
4. `schema suggest` introduces ongoing LLM cost per invocation.
Bounded by sampling, but sets a precedent for "gbrain commands
that cost money."
5. Auto-prompt UX is novel. TTY gate + per-type silencing helps,
but bulk-import flows could hit unexpected interruption patterns.
-179
View File
@@ -1,179 +0,0 @@
# Switching embedding models or dimensions on an existing brain
GBrain stores embeddings in a fixed-dimension `vector(N)` column on
`content_chunks`. If you switch to a model with a different dimension
(e.g. `openai:text-embedding-3-large` 1536 → `zeroentropyai:zembed-1`
1280, or `voyage:voyage-4-large` 2048), the on-disk column type doesn't
change automatically.
`gbrain init`, `gbrain doctor`, and `gbrain embed --stale` all detect
this mismatch and refuse to silently proceed. This doc is the recipe
they point at.
## Same-dimension model swaps (v0.41.31.0 — automatic)
If you switch to a different model at the **same** dimension count
(e.g. one 1536-dim provider to another, or a re-tuned model that keeps
its width), the column type doesn't change, so no `ALTER`/wipe recipe
is needed. As of v0.41.31.0, gbrain stamps an embedding-provenance
signature (`<provider:model>:<dims>`) onto each page when its chunks are
embedded. After you point the config at the new model, the stored
signatures differ from the current one, and `gbrain embed --stale`
re-embeds exactly those pages:
```bash
# After switching to the new same-dim model in your config:
gbrain embed --stale # re-embeds signature-drifted pages
gbrain embed --stale --dry-run # preview the count without re-embedding
```
Under federated_v2, the same drift is picked up by the per-source
`embed-backfill` jobs that `gbrain sync --all` enqueues (capped
`$X/source/24h`). **Grandfather:** pages embedded before v0.41.31.0
carry a NULL signature and are NEVER flagged stale, so upgrading to
v0.41.31.0 does NOT trigger a whole-corpus re-embed. Signatures only
get stamped going forward.
A **dimension** change still requires the wipe-and-reinit (PGLite) or
column-alter (Postgres) recipe below — the on-disk `vector(N)` width
genuinely has to change.
## Why we don't do this automatically
Switching dimensions requires:
1. Dropping the HNSW vector index (pgvector won't survive an `ALTER COLUMN TYPE`).
2. Altering the column type (Postgres only — PGLite cannot do this).
3. Wiping every existing embedding (the old vectors are unusable in the new space).
4. Re-embedding the entire corpus (can take hours on a 50K-page brain and costs $1-100 in API calls depending on model).
5. Conditionally recreating the index (HNSW supports up to 2000 dimensions per pgvector; above that you must use exact scans).
That's not an upgrade-time auto-run. It's a deliberate, expensive
operation. Run it when you've decided you actually want the new model.
## PGLite (default install)
**PGLite cannot `ALTER COLUMN TYPE vector(N)`.** pgvector ships as
embedded WASM, not a native extension, and the WASM build rejects the
column-type alter with `could not access file "$libdir/vector"`. The
SQL recipe below works against Postgres only.
The path that works on PGLite is **wipe-and-reinit**. v0.37 ships a
single-command wrapper:
```bash
gbrain reinit-pglite \
--embedding-model zeroentropyai:zembed-1 \
--embedding-dimensions 1280
```
This backs up the existing brain to `<path>.bak`, runs `gbrain init`
with the new flags (preserving every other field in
`~/.gbrain/config.json`), and re-syncs the brain repo. Add `--no-sync`
to skip the resync, `--yes` to skip the TTY confirmation, `--json` for
structured output.
Equivalent by hand:
```bash
# 1. Back up the existing brain (in case you want to roll back).
mv ~/.gbrain/brain.pglite ~/.gbrain/brain.pglite.bak
# 2. Re-init with the new model + dimensions. `gbrain init` writes
# the schema sized to the new dim, and (as of v0.37) preserves
# every other field in ~/.gbrain/config.json (chat model,
# expansion model, API keys).
gbrain init --pglite \
--embedding-model zeroentropyai:zembed-1 \
--embedding-dimensions 1280
# 3. Re-import your brain repo. `gbrain sync` reads the brain repo
# from disk and re-creates the page rows.
gbrain sync
# 4. Re-embed. The embed pipeline now uses the new model and the
# column accepts the new dim.
gbrain embed --stale
```
If your brain repo is large enough that re-syncing from disk is
expensive (>50K pages), see the Postgres section below — migrating to
Postgres temporarily lets you run the SQL recipe, then migrate back to
PGLite.
`GBRAIN_HOME` users: substitute the active database path (or use
`gbrain config get database_path` to find it).
## Postgres (Supabase / self-hosted)
Postgres supports the in-place column alter. Replace `<NEW_DIMS>` with
your target dimension count.
```sql
BEGIN;
-- 1. Drop the HNSW index. It can't survive the column type change.
DROP INDEX IF EXISTS idx_chunks_embedding;
-- 2. Alter the column type.
ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(<NEW_DIMS>);
-- 3. Clear stale embeddings so they don't survive into the new space.
UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;
-- 4. Recreate the HNSW index ONLY IF dims <= 2000. Above that, leave it
-- indexless and rely on exact scans (gbrain searchVector handles this
-- automatically — search just gets slower, not broken).
-- For dims <= 2000 (e.g. 1024, 1280, 1536, 768):
CREATE INDEX IF NOT EXISTS idx_chunks_embedding
ON content_chunks USING hnsw (embedding vector_cosine_ops);
-- For dims > 2000 (e.g. 2048 Voyage 4 Large): skip step 4.
COMMIT;
```
Then re-init config with the new model:
```bash
gbrain init --supabase \
--embedding-model <provider:model> \
--embedding-dimensions <NEW_DIMS>
```
And re-embed:
```bash
gbrain embed --stale
```
## A note on `gbrain config set`
Pre-v0.37 docs recommended `gbrain config set embedding_model X` to
switch models. **This is a no-op for the embed pipeline.** `config set`
writes the DB plane; the embed gateway reads the file plane
(`~/.gbrain/config.json`). The pre-v0.37 recipe shipped the lie because
the contract wasn't surfaced.
As of v0.37, `gbrain config set embedding_model` and `gbrain config set
embedding_dimensions` REFUSE and print the wipe-and-reinit recipe.
To change schema-sizing fields, use `gbrain init` (PGLite) or the SQL
recipe (Postgres). Both update the file plane AND the schema together.
## Verify
After the recipe lands, `gbrain doctor --fast` should report green and
`gbrain doctor` should pass the `embedding_width_consistency` check:
```
✓ embedding_width_consistency dim parity: config 1280 / column vector(1280)
```
If it doesn't, file an issue with the doctor output and the steps you
ran.
## v0.37+ followups
- Auto-fallback to alternative embedding providers when the primary
fails quota/auth. Tracked; requires explicit `--try-fallback`
consent because mixing provider vectors silently corrupts retrieval.
-29
View File
@@ -1,29 +0,0 @@
# Origin story
GBrain came out of building OpenClaw — Garry's personal AI agent fork. The first version had skills and a brain, but the brain was a flat directory of markdown files. Search was ripgrep. Memory was vibes.
Two problems surfaced almost immediately.
First, the agent forgot things between conversations. Every new session re-asked basic questions. Names of people Garry had introduced last week were gone. Decisions made on Tuesday didn't survive to Thursday. The brain existed but the agent couldn't actually use it.
Second, the agent kept duplicating work. Two different signals about the same company became two different people pages. Three meetings with the same person became three uncorrelated timeline entries. The signal-to-noise ratio decayed in real time.
GBrain is what you build when you decide both of those are unacceptable.
The fix wasn't one big idea. It was many small ones layered together:
- Brain-first lookup before any external API call.
- Auto-linking on every page write so the graph grows for free.
- Typed edges so "who works at Acme AI?" actually returns something.
- Hybrid search because vector alone underdelivers.
- Reranker on top because hybrid alone is locally optimal but globally suboptimal.
- Nightly cron to dedup, enrich, fix citations, surface contradictions.
- An agent that reads `skills/RESOLVER.md` once and knows what to do.
None of those are novel ideas. The contribution is shipping all of them together, on Postgres + pgvector that runs in WASM (no server), with skills that are markdown (not code), routed by a small text file (not a router LLM).
The production brain has been running for months now. 17,888 pages. 4,383 people. 723 companies. 21 cron jobs running autonomously. It wakes Garry up smarter than the day before.
GBrain is what happens when you write the brain you actually wanted to have.
The reason the brain is worth building is `gbrain think`. Without it, the brain is just a place that holds your notes. With it, the brain is a thing you can query about itself: what does it know, what does it not know yet, where does it contradict itself, where are the holes. The 24/7 cron cycle keeps the brain sharp. `think` is what makes a sharp brain useful.
-598
View File
@@ -1,598 +0,0 @@
# Running real-world eval benchmarks against your gbrain changes
Audience: gbrain maintainers and contributors. If you're touching retrieval
(search, ranking, embeddings, intent classification, query expansion, source
boost, hybrid fusion), this is the doc.
For the **NDJSON wire format** consumed by gbrain-evals, see
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
that lives on top of that format.
## v0.41 update — the LOOP is now real
Before v0.41, you could capture eval rows and replay them but nothing
stitched them into a gate. `gbrain bench publish` + `gbrain eval gate`
close the loop. Two gates:
- **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline
you captured against your current brain. Catches: "did my refactor break
search?" Compares jaccard / top-1 stability / latency multiplier.
- **Correctness gate** (`--qrels Y.qrels.json`): runs known-right queries
against your current brain via bare `hybridSearch`. Catches: "is my
retrieval actually any good?" Computes recall@K, first-relevant-hit-rate,
expected_top1-hit-rate.
Both can be passed together; both must pass for verdict `pass`. At least
one is required.
### The full LOOP for your own brain
```bash
# 1. Capture (one-time; uses queries already in eval_candidates)
gbrain eval export --limit 200 --tool query > /tmp/captured.ndjson
# 2. Publish a baseline
mkdir -p ~/.gbrain/baselines
gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/personal.baseline.ndjson --label "personal-$(date +%Y%m%d)"
# 3. Gate against it
gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson
```
### Privacy posture (D9)
**Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real
user captures stay local in `~/.gbrain/baselines/`. The boundary is
enforced at the file source, not by post-hoc scrubbing. If you publish a
baseline to `gbrain-evals`, generate it from a fixture-seeded test brain
(placeholder names like `alice-example`, `widget-co-example`) — never
from a real user's `eval_candidates` table.
### Deterministic-pipeline disclosure
`gbrain eval gate --qrels` uses bare `hybridSearch` (not the production
`query` op handler). This is deliberate: gates need to be deterministic in
CI. Production retrieval differs via the query cache, salience freshness,
expansion, etc. The gate measures retrieval quality with a fixed pipeline;
your users may see different results when the cache is warm.
### `.qrels.json` shape
Two equivalent representations per entry:
```json
{
"schema_version": 1,
"queries": [
{
"query_id": "q1",
"query": "fintech founder",
"relevant_slugs": ["people/alice-example"],
"first_relevant_slug": "people/alice-example"
}
]
}
```
For federated / multi-source brains, use the explicit shape (no defaults
to `source_id='default'`):
```json
{
"query_id": "q2",
"query": "anything",
"relevant": [
{"source_id": "host", "slug": "people/alice"},
{"source_id": "team-a", "slug": "people/alice"}
],
"expected_top1": {"source_id": "host", "slug": "people/alice"}
}
```
Without `source_id`, a hit from the wrong source could false-pass the
gate. The compare everywhere is `${source_id}::${slug}` strings.
### Example GitHub Actions workflow
```yaml
name: gbrain-eval-gate
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- run: bun install
- run: |
# Run both gates; CI fails on any breach.
gbrain eval gate \
--baseline gbrain-evals/baselines/v0.41-launch.baseline.ndjson \
--qrels gbrain-evals/qrels/v0.41-launch.qrels.json \
--json | tee /tmp/gate.json
```
---
## Prerequisite: turn on contributor mode
Capture is **off by default** for production users (privacy-positive — no
surprise data accumulation). Contributors flip it on with one line:
```bash
# In ~/.zshrc or ~/.bashrc:
export GBRAIN_CONTRIBUTOR_MODE=1
```
Verify:
```bash
gbrain query "anything" >/dev/null
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
```
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
```json
{"eval": {"capture": true}} // force on
{"eval": {"capture": false}} // force off
```
Explicit config beats the env var both directions.
## The 4-command loop
```bash
# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set.
# Inspect what's been collected:
gbrain doctor # surfaces capture failures
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
# ② Snapshot: freeze a baseline before your code change.
gbrain eval export --since 7d > baseline.ndjson
# ③ Code change: do whatever you want — tune RRF_K, swap embed model, edit
# hybrid.ts, add a new boost source, change the intent classifier.
# ④ Replay: re-run every captured query against the current build.
gbrain eval replay --against baseline.ndjson
```
Output:
```
Replaying 247 captured queries…
...25/247
...50/247
...
Replayed 247 of 247 captured queries (0 skipped, 0 errored)
Mean Jaccard@k: 0.927
Top-1 stability: 91.5%
Mean latency Δ: +14ms (current vs captured)
Top 5 regression(s):
jaccard=0.20 captured=12 current=3 "find every reference to widget-co"
jaccard=0.43 captured=14 current=8 "show me everything tagged for review"
jaccard=0.50 captured=8 current=4 "what did alice say about the spec"
...
```
Three numbers tell you whether the change is safe to land:
| Metric | What it means | Healthy range |
|---|---|---|
| **Mean Jaccard@k** | Average overlap between captured retrieved slugs and current run's slugs. 1.0 = identical sets. | ≥0.85 for "neutral" changes. <0.7 means major retrieval shift. |
| **Top-1 stability** | Fraction of queries whose #1 result didn't change. | ≥85% for tuning passes. <70% means top-of-funnel broke. |
| **Mean latency Δ** | Current minus captured. Positive = slower now. | Within ±50ms of captured. >2× anywhere = regression alarm. |
## What it actually does
`gbrain eval replay` reads your NDJSON snapshot and, for each row:
1. Re-executes the same op (`searchKeyword` for `tool_name='search'`,
`hybridSearch` for `tool_name='query'`) with the captured `detail` and
`expand_enabled` values threaded back in.
2. Captures the current `retrieved_slugs` (deduped, in result order).
3. Computes set-Jaccard between captured and current slug sets.
4. Records top-1 match (was the #1 result the same slug?).
5. Records latency delta vs captured `latency_ms`.
It does NOT compute MRR or nDCG — those need ground-truth relevance labels,
not a baseline comparison. For metric-against-truth eval, use
`gbrain eval --qrels <path>` (the legacy IR-eval path, still supported). The
replay tool answers a different question: "did my code change move
retrieval, and which queries did it move most?"
For a third evaluation axis — public benchmark, ground-truth labels, full
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
imported, the question asked, the hypothesis emitted as JSONL — exactly the
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
never opened. See `## Public benchmarks: LongMemEval` below.
## Best-effort by design
Replay is not pure. Three things can drift between capture and replay:
1. **Brain state** — your brain probably has more pages now than when the
snapshot was taken. Unless you explicitly seed a fixed corpus, mean
Jaccard will drop simply because new pages are eligible.
2. **Embedding source** — if you changed `OPENAI_API_KEY` between capture
and replay (or the embedding model rotated), vector-path results drift
even with identical code.
3. **Capture cap** — captured `retrieved_slugs` is a deduped set; it doesn't
preserve internal ranking metadata. Two tools can return the same slug
set with different scores — Jaccard will say 1.0, but a downstream
consumer that orders by score may behave differently.
The metrics are **regression alarms on real queries**, not a hash check.
Pair them with manual inspection of the top regressions.
## Cost
Every `query` row in the snapshot embeds the query string via OpenAI to run
the vector half of `hybridSearch`. Cost is identical to a normal `gbrain
query` invocation — text-embedding-3-large at OpenAI list price, batched
inside a single replay row.
If you're iterating locally and don't want to pay per change, use
`--limit 50` to cap rows replayed. The 50 most recent rows are usually
enough to catch direction; expand for the final pre-merge run.
```bash
# Iteration mode — 50 most recent queries
gbrain eval replay --against baseline.ndjson --limit 50
# Pre-merge — full snapshot
gbrain eval replay --against baseline.ndjson --top-regressions 20
```
## CI integration
```bash
gbrain eval replay --against baseline.ndjson --json > replay.json
jq -e '.summary.mean_jaccard >= 0.85' replay.json || exit 1
jq -e '.summary.top1_stability_rate >= 0.85' replay.json || exit 1
```
Stable JSON shape (schema_version: 1):
```json
{
"schema_version": 1,
"summary": {
"rows_total": 247,
"rows_replayed": 247,
"rows_skipped": 0,
"rows_errored": 0,
"mean_jaccard": 0.927,
"top1_stability_rate": 0.915,
"mean_latency_delta_ms": 14,
"rows_over_2x_latency": 0
}
}
```
`--verbose` adds a `results: [...]` array with one entry per replayed row
(useful for piping into jq or a notebook for deeper analysis).
## When to run this
Before merging anything that touches:
- `src/core/search/hybrid.ts` (RRF, fusion, dedup, two-pass retrieval)
- `src/core/search/source-boost.ts` / `sql-ranking.ts` (per-source ranking)
- `src/core/search/intent.ts` (auto-detail classification)
- `src/core/search/expansion.ts` (Haiku query expansion)
- `src/core/search/dedup.ts` (cross-page result collapse)
- `src/core/embedding.ts` or any embedding model swap
- `src/core/operations.ts` `query` or `search` op handlers (capture surface)
- `src/core/postgres-engine.ts` / `pglite-engine.ts` `searchKeyword` /
`searchVector` SQL
Skip for: schema-only migrations, doc changes, tests-only PRs, CLI ergonomics
that don't touch retrieval.
## Building your own corpus
If you don't have captured traffic yet (fresh install, can't dogfood for a
week before merging), you can hand-author an NDJSON file:
```jsonl
{"schema_version":1,"id":1,"tool_name":"query","query":"who is alice","retrieved_slugs":["people/alice","people/alice-bio"],"expand_enabled":false,"detail":null,"latency_ms":0,"remote":false}
{"schema_version":1,"id":2,"tool_name":"search","query":"acme deal","retrieved_slugs":["deals/acme-seed","companies/acme"],"latency_ms":0,"remote":false}
```
Then run `gbrain eval replay --against handcrafted.ndjson` to confirm the
authoritative slugs come back. This is the seam between the BrainBench-Real
pipeline (replay against live captures) and the BrainBench fixed-fixture
pipeline (`gbrain eval --qrels` with the sibling
[gbrain-evals](https://github.com/garrytan/gbrain-evals) corpus).
## Off-switch
Two ways to disable capture:
```bash
unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var
```
Or force off regardless of the env var via `~/.gbrain/config.json`:
```json
{"eval": {"capture": false}}
```
Existing `eval_candidates` rows stay until you `gbrain eval prune
--older-than 0d` (or just drop the table).
## Failure modes
| What you see | What it means |
|---|---|
| `Mean Jaccard@k: 0.4`, top regressions all in one source dir | Source boost or hard-exclude regression on that prefix |
| `Top-1 stability: 30%`, mean Jaccard still high | RRF tuning shifted the rank order without changing the set — re-tune `rrfK` |
| `Mean latency Δ: +500ms`, jaccard high | Vector path got slower; check embedding API or HNSW probes |
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
## Public benchmarks: LongMemEval (v0.28.8)
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
benchmark directly against gbrain's hybrid retrieval. Different evaluation
axis from `eval replay`: public dataset with ground-truth labels, end-to-end
question-answer pipeline, hermetic per-question brains.
```bash
# Download the dataset (visit the HF page in a browser; gated/manual download).
# Place longmemeval_oracle.json (or _s.json) somewhere local.
# Retrieval-only (no LLM answer-gen, fastest path, no Anthropic key needed):
gbrain eval longmemeval ./longmemeval_oracle.json --limit 50 --retrieval-only \
> /tmp/hypothesis.jsonl
# Full pipeline (Anthropic key required for answer-gen):
gbrain eval longmemeval ./longmemeval_oracle.json --limit 50 \
> /tmp/hypothesis.jsonl
# Score with LongMemEval's published evaluate_qa.py (not bundled — needs
# OpenAI gpt-4o per their spec):
python evaluate_qa.py /tmp/hypothesis.jsonl
```
### Architecture (read this if you're touching the harness)
- One in-memory PGLite per benchmark run via `createBenchmarkBrain` +
`withBenchmarkBrain`. Your `~/.gbrain` is never opened.
- Between questions: `TRUNCATE` over runtime-enumerated `pg_tables`, NOT a
hardcoded list — schema migrations don't silently leak data across
questions. Infrastructure tables (`sources`, `config`,
`gbrain_cycle_locks`, `subagent_rate_leases`) are preserved across resets.
- Sanitization parity: re-uses `INJECTION_PATTERNS` from
`src/core/think/sanitize.ts` so adding a new injection pattern
automatically covers takes AND benchmarks. One source of truth.
- Retrieved chat content is wrapped in `<chat_session id="..." date="...">`
framing; the answer-gen system prompt declares the content UNTRUSTED.
Same posture as `<take>` framing.
- LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})`.
Tests stub the client so the full pipeline runs hermetically without any
API key.
### Flags
| Flag | Default | Purpose |
|---|---|---|
| `--limit N` | run all | Cap question count (iterate fast) |
| `--retrieval-only` | off | Emit retrieved chunks; no LLM answer-gen |
| `--keyword-only` | off | Disable vector path (debug retrieval issues) |
| `--expansion` | **off** | Multi-query expansion. Off by default for determinism (no per-query Haiku call). Pass to opt in. |
| `--top-k K` | 10 | Retrieval depth |
| `--model M` | resolved | Default resolves through `resolveModel()` 6-tier chain (`models.eval.longmemeval` config key) |
| `--output FILE` | stdout | Write hypothesis JSONL to file instead of stdout |
### Numbers
p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
`test/eval-longmemeval.test.ts` perf gate). Per-question cost well under the
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
LLM latency.
## Measuring brain consistency over time (v0.32.6)
`gbrain eval suspected-contradictions` is a complementary measurement
instrument: it samples retrieval results for unmarked semantic
contradictions (e.g., compiled_truth vs chat content, intra-page chunk
vs active take). Where LongMemEval measures retrieval correctness on a
fixed labeled set, the contradiction probe measures how often a real
brain surfaces conflicting answers.
### Recommended nightly cadence
```bash
# Once a day, against your top 50 most-frequent queries:
gbrain eval suspected-contradictions \
--queries-file ~/.gbrain/queries.jsonl \
--top-k 5 \
--budget-usd 5 \
--output ~/.gbrain/probe-runs/$(date +%Y-%m-%d).json
```
Persistent cache (`eval_contradictions_cache`) makes re-runs near-zero
cost until you bump `PROMPT_VERSION`. Trend-track via:
```bash
gbrain eval suspected-contradictions trend --days 30
```
The ASCII bar chart shows total flagged per day. Headline % surfaces in
`gbrain doctor`'s `contradictions` check with paste-ready resolution
commands per high-severity finding.
### See also
- `docs/contradictions.md` — architecture, severity rubric, action criteria.
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
decision criteria gated on Wilson CI lower-bound.
## v0.40.1.0 Track D — Eval infrastructure
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
covers the dev loop that uses them and the gates they enforce.
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
surfaces it in machine-readable form. Two additive changes:
1. Every per-question JSONL row now includes a `question: string` field so the
`gbrain eval cross-modal --batch` consumer (below) can read it without
joining back against the source dataset.
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
```json
{"schema_version": 1, "kind": "by_type_summary",
"recall_by_type": {"single-session-user": {"hit": 18, "total": 19, "rate": 0.947}},
"aggregate": {"hit": 110, "total": 120, "rate": 0.917}}
```
**Resume-safe.** When `--resume-from` is the same path as `--output`, the
summary is rebuilt from the file (each per-row includes `question_type` and
`recall_hit`) so the final aggregate covers all resumed questions, not just
this run's slice. The prior summary at the file tail is replaced, not
appended — a brain that resumes 5 times across a 500-question run ends with
exactly ONE summary at the tail.
**Optional gate.** `--by-type-floor 0.85` exits non-zero when any
`question_type`'s rate falls below 0.85. Default: informational only.
```bash
# Diagnose per-type ranking quality after a search-touching change.
gbrain eval longmemeval ~/datasets/longmemeval_s.jsonl \
--by-type --output /tmp/run.jsonl
tail -1 /tmp/run.jsonl | jq . # summary line
# Strict gate in a CI script.
gbrain eval longmemeval test/fixtures/longmemeval-mini.jsonl \
--by-type --by-type-floor 0.80 --output /tmp/run.jsonl
echo "exit=$?" # 1 if any type fell below 0.80
```
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
silently regress retrieval." Replaces the original "replay against captured
eval_candidates" design (which Codex caught as non-functional in CI — see
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
real-query version).
How it works:
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
with PLACEHOLDER names only (no real people / companies per CLAUDE.md privacy
rule).
- The test seeds a PGLite engine with synthetic pages whose embeddings are
basis vectors (the same `basisEmbedding(idx)` pattern as
`test/e2e/search-quality.test.ts`). No API keys, no DATABASE_URL.
- For each qrels query, calls `engine.searchVector(basisEmbedding(dim))` and
computes `top1_match_rate` and `recall@10`. Asserts both meet floors
(`>= 0.80` and `>= 0.85` by default).
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
When CI fails because a legitimate ranking change moved expected slugs, the
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
in the commit body** so future maintainers can read the audit trail. Without
the `Why:`, the gate degrades to a rubber stamp within months. The convention
is informational (not a commit-hook block), but enforce it in PR review.
Example commit body:
```
chore(eval): refresh qrels for new source-boost ordering
Why: v0.40.x source-boost now weights originals/ over concepts/, so
q12 (founder-mode) now correctly surfaces originals/founder-mode-example
top-1. Manual verification: ran the production query; new ranking is
clearly better-aligned with the query intent.
```
#### Env-overrides for floors
```bash
GBRAIN_REPLAY_GATE_TOP1_FLOOR=0.85 \
GBRAIN_REPLAY_GATE_RECALL_FLOOR=0.90 \
bun test test/eval-replay-gate.test.ts
```
Use to tighten or loosen the gate as the qrels fixture matures.
### `gbrain eval cross-modal --batch` — batch quality scoring
Single-task cross-modal eval scores one (task, output) pair. Batch mode runs
the same scoring over an entire LongMemEval JSONL output, with cost guardrails.
```bash
# Step 1: produce LongMemEval hypotheses (real cost: depends on model + N).
gbrain eval longmemeval ~/datasets/longmemeval_s.jsonl \
--limit 10 --output /tmp/run.jsonl
# Step 2: batch-score those hypotheses (real cost: ~$0.70 for 10 questions,
# 1 cycle, 3 model slots at default --max-usd 5 budget cap).
gbrain eval cross-modal --batch /tmp/run.jsonl \
--limit 10 --cycles 1 --concurrent 3 --max-usd 5 --json
echo "exit=$?" # 0=all-pass, 1=any-fail, 2=any-error-or-inconclusive
```
**Key behaviors:**
- Default `--cycles 1` in batch mode (single-task default is 3 in TTY) to bound
cost. Pass `--cycles 3` to match single-task strictness.
- `--concurrent 3` runs up to 3 questions in parallel x 3 model slots each =
9 simultaneous API calls. Below tier-1 rate limits for all three providers.
- `--max-usd FLOAT` refuses to start if the pre-flight cost estimate exceeds
the cap, unless `--yes` bypasses (required for non-interactive cron / CI).
- Filters `kind: "by_type_summary"` rows automatically (the LongMemEval
`--by-type` summary line is metadata, not a question).
- `--batch` is mutually exclusive with `--task`; fail-fast usage error if both
are set.
- Exit precedence (fail-loud): ERROR > FAIL > INCONCLUSIVE > PASS.
- Per-question receipts land in a tempdir and are deleted at end of batch; the
summary inlines per-question verdicts so the audit trail is self-contained.
### Nightly cross-modal quality probe (opt-in, autopilot)
`src/core/cycle/nightly-quality-probe.ts` ships a phase that runs the longmemeval
+ cross-modal pipeline once per 24h. **Disabled by default** to avoid surprise
API spend. Enable per-host:
```bash
gbrain config set autopilot.nightly_quality_probe.enabled true
gbrain config set autopilot.nightly_quality_probe.max_usd 5.00 # optional override
```
Note: `--phase nightly_quality_probe` wiring into the autopilot scheduler is
deferred to a v0.41+ follow-up (see TODOS.md). For now the phase is callable
in isolation; the test harness exercises it via DI stubs.
```bash
# Manual smoke (exercises the path via DI stubs, no real API spend).
bun test test/nightly-quality-probe.test.ts
```
Observability:
- `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` — one event per run with
outcome (pass / fail / inconclusive / error / budget_exceeded /
rate_limited / no_embedding_key), pass/fail/inconclusive/error counts,
est_cost_usd, fixture_sha8. ISO-week rotation (mirrors slug-fallback
audit).
- `gbrain doctor` surfaces `nightly_quality_probe_health`:
- SKIPPED (disabled) — with paste-ready enable command.
- OK (enabled, no events yet) — autopilot hasn't fired its first run.
- OK (last 7d all PASS) — with timestamp of latest run.
- WARN — any FAIL / ERROR / BUDGET_EXCEEDED in the window, with outcome
counts and the latest run's reason.
Real expected cost: ~$0.35 per nightly run (5 questions x 3 slots x 1 cycle
x ~$0.02/call) ≈ $10.50/month. Worst-case under the default budget cap:
$150/month. Opt-in default prevents discovering this in your card statement.
-160
View File
@@ -1,160 +0,0 @@
# Eval capture — NDJSON schema reference
**Status:** stable from v0.21.0. Schema versioning via `schema_version`
on every row; additive changes increment the minor version; removals
are breaking-schema-v2.
**Audience:** downstream consumers (primarily the sibling
[gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) that
replay captured real-world queries as a BrainBench-Real fixture.
## The pipeline
```
MCP / CLI / subagent tool-bridge caller
src/core/operations.ts — query + search op handlers
│ (hybridSearch or searchKeyword)
{results, meta: HybridSearchMeta} ┌── captureEvalCandidate
│ │ (fire-and-forget)
▼ │
return to caller ▼
scrubPii(query) ←── src/core/eval-capture-scrub.ts
buildEvalCandidateInput
engine.logEvalCandidate
┌──────────────┴──────────────┐
│ success │ fail
▼ ▼
INSERT into eval_candidates engine.logEvalCaptureFailure
(reason: db_down | rls_reject |
check_violation |
scrubber_exception | other)
```
## `gbrain eval export` — the consumer contract
```sh
gbrain eval export [--since DUR] [--limit N] [--tool query|search]
```
Emits NDJSON to **stdout**. One JSON object per `\n`-terminated line.
stderr receives progress heartbeats. Every line starts with
`"schema_version": 1` so a forward-compat parser can fail loudly on
schema v2 instead of silently misparsing.
Typical usage from gbrain-evals:
```sh
# Snapshot the last week of real traffic for replay
gbrain eval export --since 7d > brainbench-real.ndjson
```
```sh
# Stream through jq for ad-hoc analysis
gbrain eval export --tool query | jq -c 'select(.latency_ms > 500)'
```
## Row schema (v1)
Every exported row has this shape. Field order in JSON output is not
guaranteed; consumers MUST key by name, not position.
| Field | Type | Notes |
|---|---|---|
| `schema_version` | number | Always `1` on v1 rows. Forward-compat gate. |
| `id` | number | Autoincrement primary key. Stable across exports. |
| `tool_name` | `"query"` \| `"search"` | Which MCP operation captured this row. |
| `query` | string | **Already PII-scrubbed** by `scrubPii` unless `eval.scrub_pii: false`. Emails / phones / SSN / Luhn-verified credit cards / JWTs / bearer tokens replaced with `[REDACTED]`. Max length 50KB (CHECK-enforced). |
| `retrieved_slugs` | string[] | Deduplicated slugs that came back in `SearchResult[]`. |
| `retrieved_chunk_ids` | number[] | Every chunk id in result order (duplicates preserved — one per hit). |
| `source_ids` | string[] | Distinct `sources.id` values across the result set (v0.18 multi-source). Empty for pre-v0.18 rows that lacked the column. |
| `expand_enabled` | boolean \| null | Whether the caller **requested** Haiku expansion. `null` for `search` (no expansion concept). |
| `detail` | `"low"` \| `"medium"` \| `"high"` \| null | Detail level the caller **requested**. `null` when omitted. |
| `detail_resolved` | `"low"` \| `"medium"` \| `"high"` \| null | What `hybridSearch` **actually used** after auto-detect. `null` when neither caller nor heuristic classified. |
| `vector_enabled` | boolean | True iff vector search actually ran. `false` when `OPENAI_API_KEY` was missing or the embed call failed. **Replay MUST respect this** — rows with `false` only exercised the keyword path. |
| `expansion_applied` | boolean | True iff Haiku expansion actually produced variants (not just "was requested"). |
| `latency_ms` | number | Wall-clock duration of the op handler (includes capture itself — negligible since it's fire-and-forget). |
| `remote` | boolean | `true` for MCP callers (untrusted), `false` for local CLI. Partitions "real agent traffic" from "operator probing." |
| `job_id` | number \| null | `OperationContext.jobId` when the caller was a subagent tool-bridge. Null for MCP + CLI. |
| `subagent_id` | number \| null | `OperationContext.subagentId` for subagent-owned runs. |
| `created_at` | string (ISO 8601) | UTC timestamp of insert. |
## Ordering + determinism
`listEvalCandidates` orders by `created_at DESC, id DESC`. Same-
millisecond inserts tie on `created_at`; `id DESC` is the stable
tiebreaker. Replay tools can consume rows in order and assume:
- no duplicate rows across calls with non-overlapping `--since` windows
- no missed rows across calls that chain `--since` windows (window end
of run 1 is the strict upper bound, not a soft cursor)
## Schema versioning promise
- **v1 (shipped v0.21.0)** — this document. All fields listed above.
- **Additive changes** increment gbrain minor version (v0.25.0, v0.23.0
…) and ship with new optional fields. Consumers keyed on known fields
ignore unknown keys and keep working.
- **Breaking changes** (rename, type change, removal) increment
`schema_version` to 2. Consumers MUST branch on `schema_version` to
stay compatible.
## `eval_capture_failures` — companion audit table
Not exported by `gbrain eval export`. Surfaced via `gbrain doctor`:
```sh
gbrain doctor # warns when failures in last 24h > 0
```
Reason enum (stable): `db_down` | `rls_reject` | `check_violation` |
`scrubber_exception` | `other`. Cross-process visibility is the whole
point — `gbrain doctor` runs in its own process and reads the table
directly, so in-process counters wouldn't work.
## Config + CONTRIBUTOR_MODE
Capture is **off by default** as of v0.25.0 (was on for everyone in
earlier drafts). Two paths to turn it on:
**Path A — env var (contributor opt-in, the common case):**
```bash
export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc
```
**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):**
```json
{
"engine": "postgres",
"database_url": "...",
"eval": {
"capture": true,
"scrub_pii": true
}
}
```
Resolution order (most explicit wins):
1. `eval.capture: true` in config → on
2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1)
3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on
4. otherwise → off
`scrub_pii` defaults to `true` independent of capture. Set
`eval.scrub_pii: false` to preserve raw query text (only if you control
the brain's distribution).
`gbrain config set eval.capture false` does **not** work — that
command writes the DB-plane config, and the MCP server reads the
file-plane. Edit the JSON directly or use the env var.
-159
View File
@@ -1,159 +0,0 @@
# `gbrain eval takes-quality` — reproducible cross-modal quality eval
v0.32+ ships a CI-able quality gate for the takes layer. Three frontier models
score a sample of takes against a 5-dimension rubric, the runner aggregates to
PASS / FAIL / INCONCLUSIVE, and the receipt persists to `eval_takes_quality_runs`
so a follow-up `trend` or `regress` can compare against history.
This doc is the consumer contract. The sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals)
repo and any future CI gate read receipts shaped exactly like the JSON below.
Fields are additive-stable at `schema_version: 1`. A breaking shape change
bumps the version.
## Subcommands
| Command | Brain required? | Exit codes |
|---|---|---|
| `gbrain eval takes-quality run [flags]` | yes (samples takes) | 0 PASS, 1 FAIL, 2 INCONCLUSIVE |
| `gbrain eval takes-quality replay <receipt>` | **no** (disk-only) | 0 PASS, 1 FAIL, 2 INCONCLUSIVE |
| `gbrain eval takes-quality trend [flags]` | yes (reads runs table) | 0 |
| `gbrain eval takes-quality regress --against <receipt>` | yes | 0 OK, 1 regression |
`replay` is the only mode that runs without `DATABASE_URL` — it reads the
receipt file from disk and re-renders it. The other modes need the brain.
## `run` flags
| Flag | Default | Notes |
|---|---|---|
| `--limit N` | 100 | Random sample of N takes from the brain. |
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
| `--json` | off | Emit the full receipt to stdout. |
## Receipt JSON shape (`schema_version: 1`)
```json
{
"schema_version": 1,
"ts": "2026-05-09T22:00:00.000Z",
"rubric_version": "v1.0",
"rubric_sha8": "abcd1234",
"corpus": {
"source": "db",
"n_takes": 100,
"slug_prefix": null,
"corpus_sha8": "abcd1234"
},
"prompt_sha8": "abcd1234",
"models_sha8": "abcd1234",
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
"cycles_run": 3,
"successes_per_cycle": [3, 3, 2],
"verdict": "pass",
"scores": {
"accuracy": { "mean": 7.8, "min": 7, "max": 9, "scores": [9,7,7], "per_model": {...} },
"attribution": { "mean": 7.0, "min": 7, "max": 7, "scores": [7,7,7], "per_model": {...} },
"weight_calibration": { "mean": 7.5, "min": 7, "max": 8, "scores": [8,7,7], "per_model": {...} },
"kind_classification": { "mean": 7.2, "min": 7, "max": 8, "scores": [7,8,7], "per_model": {...} },
"signal_density": { "mean": 7.0, "min": 6, "max": 8, "scores": [8,7,6], "per_model": {...} }
},
"overall_score": 7.3,
"cost_usd": 1.85,
"improvements": ["..."],
"errors": [],
"verdictMessage": "PASS: every dim mean >=7 and min >=5 ..."
}
```
### Field reference
- `schema_version` — locks the contract. Adding optional fields is additive
and compatible. Renaming, removing, or changing semantics bumps the version.
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
(codex review #3). When the rubric definition changes, both fields update,
and trend mode groups runs accordingly so a stricter rubric doesn't
silently look like a quality drop.
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
saw. Determines whether two runs are over the "same" sample.
- `models_sha8` — fingerprint over the sorted model id list. Re-ordering
models in `--models` doesn't change the sha (sort is stable).
- `successes_per_cycle` — count of contributing models per cycle. A model
contributes when (a) its JSON parsed AND (b) every declared rubric dim
has a finite score (codex review #5 — missing-dim drops the contribution).
- `verdict``pass` if every dim mean >= 7 AND every dim min across
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
2/3 models contributed complete scores.
- `cost_usd` — sum of per-call cost via `pricing.ts`. Unknown models when
`--budget-usd` is set produce a `PricingNotFoundError` before any call
fires.
## Receipt persistence
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
as a best-effort artifact. The DB row carries the full receipt JSON in the
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
## Trend output
Plain text (default):
```
ts rubric verdict overall cost corpus
─────────────────────────────────────────────────────────────────────────────
2026-05-09T22:00:00 v1.0 pass 7.3 $1.85 abcd1234
2026-05-08T18:30:00 v1.0 fail 6.8 $1.92 ef567890
```
JSON shape (`--json`):
```json
{
"schema_version": 1,
"rows": [
{ "id": 42, "ts": "...", "rubric_version": "v1.0", "verdict": "pass",
"overall_score": 7.3, "cost_usd": 1.85, "corpus_sha8": "abcd1234" }
]
}
```
## Regress: gating CI on quality
```bash
# Capture a baseline.
gbrain eval takes-quality run --limit 100 --json \
> .ci/takes-quality-baseline.json
# Later, after changing the extraction prompt:
gbrain eval takes-quality regress --against .ci/takes-quality-baseline.json \
--threshold 0.5
# exit 0 → no regression past threshold
# exit 1 → some dim dropped > 0.5; CI fails
```
The threshold is the per-dim-mean drop counting as regression. Default 0.5.
Regress reuses the **same** model panel + slug prefix + source as the prior
receipt for an apples-to-apples compare. Diffs in `corpus_sha8` /
`prompt_sha8` / `rubric_sha8` are surfaced as informational warnings (the
runner doesn't refuse — that's the caller's call).
## Contract stability
The shape above is the read contract for downstream consumers. Anything
not listed (e.g. internal aggregator state, gateway providerMetadata) is
**not** in the receipt and may change without notice.
When you need to evolve the schema:
1. Additive optional field → no version bump; old consumers ignore the
new key, new consumers read it.
2. Renamed or removed field, or changed semantics → bump
`schema_version` to `2`; runner emits both shapes for one release as
a deprecation runway.
-176
View File
@@ -1,176 +0,0 @@
# Evaluation Metric Glossary
**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.
Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.
## Retrieval Metrics
### Precision at k (P@k)
**Key:** `precision@k`
**Plain English:** Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.
**Range:** 0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.
### Recall at k (R@k)
**Key:** `recall@k`
**Plain English:** Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.
**Range:** 0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.
### Mean Reciprocal Rank (MRR)
**Key:** `mrr`
**Plain English:** On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it's typically at rank 2.
**Range:** 0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.
### Normalized Discounted Cumulative Gain at k (nDCG@k)
**Key:** `ndcg@k`
**Plain English:** Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.
**Range:** 0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.
## Retrieval-Quality / Evidence Metrics (NamedThingBench)
### Hit rate at 1 (Hit@1)
**Key:** `hit@1`
**Plain English:** Fraction of queries where the right page is the very first result. NamedThingBench hard-gates title-substring Hit@1 >= 0.95 and alias Hit@1 >= 0.98 — a query that is a page's name or title phrase should land it at rank 1, not "somewhere in the top 10".
**Range:** 0..1, higher is better.
### Hit rate at 3 (Hit@3)
**Key:** `hit@3`
**Plain English:** Fraction of queries where the right page is in the top 3 results. NamedThingBench requires the multi-chunk-dilution family to hit 1.0 — a page with one strong chunk among many weak ones must never be buried.
**Range:** 0..1, higher is better.
### Average rank-1 match score
**Key:** `avg_rank1_score`
**Plain English:** The mean base (pre-boost) retrieval score of the TOP result across recent searches, from `gbrain search stats`. It is NOT a labeled accuracy number — it is a drift signal: if this trends DOWN over time, retrieval quality is regressing (the early warning that would have caught the duplicate-page incident before a human did).
**Range:** 0..1. Watch the trend, not the absolute value; pair with the <0.6 / 0.6-0.85 / >=0.85 bucket counts for shape.
### Create-safety hint (evidence contract)
**Key:** `create_safety`
**Plain English:** A result's answer to "is this page already in the brain — safe to NOT write a new one?" Derived from the strongest evidence, NOT a raw score: exists (alias_hit / exact_title_match / high_vector_match — do not duplicate), probable (solid keyword match — prefer updating), unknown (weak match — look closer). An agent keys its don't-duplicate decision off this, which is what prevents the incident's duplicate-stub class.
**Range:** enum: exists | probable | unknown
## Set-Similarity / Stability Metrics
### Jaccard similarity at k (set Jaccard @k)
**Key:** `jaccard@k`
**Plain English:** How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.
**Range:** 0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.
### Top-1 stability rate
**Key:** `top1_stability`
**Plain English:** Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don't change the top answer don't hurt it.
**Range:** 0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.
## Statistical-Significance Metrics
### p-value (paired bootstrap)
**Key:** `p_value`
**Plain English:** How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).
**Range:** 0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.
### 95% Confidence Interval (CI)
**Key:** `confidence_interval`
**Plain English:** The range we're 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.
**Range:** Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn't statistically significant.
## Operational / Cost Metrics
### Cache hit rate
**Key:** `cache_hit_rate`
**Plain English:** Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.
**Range:** 0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.
### Average results returned
**Key:** `avg_results`
**Plain English:** Mean number of search-result rows the engine returned per call. Should be near the active mode's searchLimit unless the brain is small or the budget is dropping results.
**Range:** 0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.
### Average tokens delivered
**Key:** `avg_tokens`
**Plain English:** Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.
**Range:** 0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.
### Cost per query (USD)
**Key:** `cost_per_query_usd`
**Plain English:** Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.
**Range:** 0..unbounded. Conservative mode is typically <\$0.001 per call; tokenmax with answer-gen can exceed \$0.01.
### p99 latency (ms)
**Key:** `p99_latency_ms`
**Plain English:** 99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.
**Range:** 0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.
## Result-Sizing Metrics
### Autocut signal
**Key:** `autocut.signal`
**Plain English:** Which signal autocut used to size the result set. 'rerank' means it found a real score cliff in the cross-encoder rerank scores and cut there; 'none' means no trustworthy cliff (no reranker, <2 scored results, or the gap was too small) so it returned the full list.
**Range:** 'rerank' | 'none'. 'none' is not a failure — it means autocut declined to cut because the signal didn't justify it.
### Autocut gap ratio
**Key:** `autocut.gap_ratio`
**Plain English:** The size of the largest score drop autocut found, as a fraction of the top result's score. A gap of 0.40 means the score fell by 40% of the top score at the steepest point. Autocut cuts there only when this clears the sensitivity threshold (autocut_jump, default 0.20).
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
---
## Coverage
Every metric printed by any `gbrain eval *` or `gbrain search stats` command resolves through `getMetricGloss()` in `src/core/eval/metric-glossary.ts`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.
-285
View File
@@ -1,285 +0,0 @@
# Search Mode Evaluation Methodology
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
## 1. What this measures and what it doesn't
**Measures:** retrieval quality and operational cost on fixed public datasets, under each named search mode, against the same brain content.
**Does NOT measure:**
- Your specific brain content (this is a benchmark, not your bill).
- Your specific query distribution.
- End-user satisfaction or downstream task success.
- Latency under concurrent load.
- Production cost (the cost numbers are model-pricing estimates × dataset size, not your actual API spend).
If you want to know how a mode behaves on YOUR brain, run `gbrain search stats --days 30` after a real usage window, then run `gbrain search tune` for actionable recommendations.
## 2. Datasets and sizes
- **LongMemEval** — public split, `n=500` questions. Downloaded from [Hugging Face](https://huggingface.co/datasets/xiaowu0162/longmemeval). The corpus + answer keys are pinned to a specific commit; recorded in every per-run record.
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
- **BrainBench v1**`n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
## 3. Sample selection
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
## 4. Run procedure
The command is the doc. Anyone can reproduce.
```bash
# Setup: in your gbrain working tree, with OPENAI_API_KEY + ANTHROPIC_API_KEY exported.
git rev-parse HEAD # record the commit for the methodology footer
# Sweep all 3 modes × 2 retrieval-focused suites with seed 42.
gbrain eval run-all \
--modes conservative,balanced,tokenmax \
--suites longmemeval,replay \
--seed 42 \
--limit 500 \
--budget-usd-retrieval 5 \
--budget-usd-answer 20 \
--output docs/eval/results/v0.32.3/
# Render the comparison.
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
```
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
## 5. Threats to validity
Honest list. We name what would let a critic dismiss the numbers.
- **LongMemEval skews English + technical.** The questions are software-engineering and consumer-product flavored. Performance on a brain rich in non-English / non-technical content (writing, art history, etc.) may differ.
- **BrainBench is small** (1240 docs) relative to a production brain (10K-100K pages). Absolute scores aren't predictive of your hit rate; the _delta_ between modes is.
- **char/4 token heuristic.** Token-budget enforcement and cost estimates use a character-count / 4 heuristic. Accurate within ~5-10% for English with the OpenAI tiktoken family; off worse for Voyage (we don't use Voyage in chat retrieval, so it doesn't bias the reported numbers, but if you do, your budget caps will be approximate).
- **Expansion's quality lift varies by query distribution.** The eval data shows ~97.6% relative quality with LLM expansion vs without (i.e., barely measurable lift) on the LongMemEval corpus. On rarer-entity / longer-tail queries, the lift can be larger. We report the corpus we measured; YMMV.
- **Paired bootstrap assumes question-level independence.** Multi-hop questions within the same conversation thread aren't independent; the bootstrap CI is slightly tighter than reality.
- **Single brain instance per benchmark.** The benchmark spins up an in-memory PGLite per question. Cache hit rate measured here doesn't reflect a long-running production brain's cache state.
## 6. Per-question raw outputs
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
- **Wins:** the 3 questions where this mode's score exceeded the next-best mode by the largest margin.
- **Losses:** the 3 questions where this mode's score fell short of the next-best mode by the largest margin.
Picked by the score delta, NOT cherry-picked by hand. The README documents the rule so a critic can verify.
## 7. Pre-registered expectations
Before running, we expect:
1. **tokenmax wins Recall@10** by 5-15 percentage points over conservative. LLM expansion + 50-result ceiling helps rare-entity surface forms.
2. **conservative wins cost-per-query** by 5-15× over tokenmax. No Haiku expansion + tight 4K budget cap = single-digit-cent queries.
3. **balanced lands within 3pp of tokenmax** on Recall@10. Intent weighting (zero-LLM cost) closes most of the expansion gap on common queries.
4. **No mode breaks nDCG@10 ≥ 0.65** — the published "ship it" threshold for hybrid retrieval on technical corpora.
Then we publish whether the data agrees. **If a hypothesis fails, that's documented honestly** in the release README, not buried. Pre-registration is what makes the comparison defensible — without it, a "we expected X and got X" outcome is observation, not prediction.
## 8. Re-run cadence
This document + the eval results are regenerated on every release that touches retrieval-affecting code. The `gbrain doctor eval_drift` check surfaces changes to the curated watch-list in `src/core/eval/drift-watch.ts`:
- `src/core/search/**`
- `src/core/embedding.ts`
- `src/core/chunkers/**`
- `src/core/ai/recipes/anthropic.ts`
- `src/core/ai/recipes/openai.ts`
- `src/core/operations.ts`
Additions to the watch-list require a CHANGELOG line.
## Statistical-significance discipline
When `gbrain eval compare --md` reports a Δ between two modes, it computes:
- **Paired bootstrap** with 10,000 resamples per metric. Each resample draws _question-level_ pairs (same question, mode A vs mode B), so question-level variance is differenced out.
- **Bonferroni correction** across the 12 comparisons (3 modes × 4 metrics). The reported p-value is the comparison's raw p-value × 12 (clamped at 1.0).
- **95% confidence intervals** computed from the bootstrap distribution.
If the CI for a Δ includes 0 OR the Bonferroni-adjusted p-value exceeds 0.05, the difference is **not** statistically significant. The MD report says "not significant" verbatim.
## Glossary
Every metric the report prints has a plain-English entry in `docs/eval/METRIC_GLOSSARY.md`, auto-generated from `src/core/eval/metric-glossary.ts`. The CI guard at `scripts/check-eval-glossary-fresh.sh` regenerates and diffs against the committed file on every test run; a stale doc fails the build.
## Cost anchors
The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table both surface these rough cost anchors. Working through the math so they're auditable:
**Variables:**
- `T` = avg tokens per search-result chunk. The recursive chunker targets 300 words / chunk → ~400 tokens (English, OpenAI tiktoken approx).
- `N` = chunks delivered per query (capped by the mode's `searchLimit`).
- `R` = downstream model input rate. Sonnet 4.6 = \$3/M. Opus 4.7 = \$5/M. Haiku 4.5 = \$1/M.
- `Q` = queries per month.
**Per-query input cost** (downstream agent reads the chunks):
cost_per_query = T × N × R
| Mode | T (tokens) | N (chunks) | Sonnet (\$3/M) | Opus (\$5/M) | Haiku (\$1/M) |
|---|---|---|---|---|---|
| conservative (4K cap, 10 max) | ~400 | 10 (or fewer if budget hits) | \$0.012 | \$0.020 | \$0.004 |
| balanced (12K cap, 25 max) | ~400 | ~25 | \$0.030 | \$0.050 | \$0.010 |
| tokenmax (no cap, 50 max) | ~400 | ~50 | \$0.060 | \$0.100 | \$0.020 |
**Monthly cost** (Q × per-query):
| Mode @ Sonnet | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|---|---|---|---|
| conservative | \$12 | \$120 | \$1,200 |
| balanced | \$30 | \$300 | \$3,000 |
| tokenmax | \$60 | \$600 | \$6,000 |
| Mode @ Opus | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|---|---|---|---|
| conservative | \$20 | \$200 | \$2,000 |
| balanced | \$50 | \$500 | \$5,000 |
| tokenmax | \$100 | \$1,000 | \$10,000 |
**gbrain's own cost** on top:
- Query embedding (text-embedding-3-large @ \$0.13/M tokens): ~\$0.00001 per query. Negligible at every scale.
- Tokenmax Haiku expansion call (\$1/M input, \$5/M output, ~500 input + 200 output per call): ~\$0.0015 per query, or \$150/mo at 100K queries. Cache hits cut this in half.
- Per-page indexing (one-time): bounded by your import volume, not query volume. Not modeled here.
**Cache hit adjustment.** A warmed brain typically sees 30-50% cache hits on repeat-query traffic. Cache hits skip the downstream input cost entirely (the cached result was already in the agent's context once). So real-world costs run ~50-70% of the table above on a busy brain.
**Why these numbers DRIFT from your actual bill:**
- Your agent's system prompt + reasoning tokens add input that gbrain doesn't see.
- Compaction reduces input over a long session.
- Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot.
The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes.
## Mode × Model matrix (the 25x spread)
The per-query math above assumes Sonnet 4.6 downstream. In reality, the
downstream model tier is the BIGGER cost lever. Per-query cost at 10K
queries/month (typical single-user volume), search payload only (no cache
savings):
| Mode (search tokens) | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|---|---|---|---|
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
fleet); divide by 10 for 1K/mo (light usage).
**Natural pairings span ~4x** (cheap model + tight mode → frontier model + loose
mode). **Mismatches waste capacity:**
- `tokenmax + Haiku`: Haiku gets 20K of search results stuffed into its
context per query. Haiku's reasoning is weaker; more chunks = more noise,
not more signal. You pay Haiku rates but get sub-Haiku quality. Wrong
direction.
- `conservative + Opus`: Opus has 200K context window and can synthesize
across many chunks. Capping at 10 chunks / 4K tokens leaves Opus
reasoning underfed. You pay Opus rates but get conservative-shape
retrieval. Wasted spend.
**Right-sizing rule:** match the mode's `searchLimit` to the downstream
model's "useful context depth":
- Haiku struggles past ~5-10 chunks of cross-referenced content → conservative
- Sonnet handles ~25-40 chunks well → balanced
- Opus benefits from 50+ chunks for multi-hop reasoning → tokenmax
## Realistic-scale anchor (single power-user agent loop)
The per-query math above is honest but theoretical: it treats each search as an isolated billable event. Real agent loops amortize a lot of context across turns via Anthropic prompt caching. Here's what one heavy power-user loop actually looks like in production, anonymized + scaled so the numbers represent a representative power user rather than any specific deployment.
**Reference shape — tokenmax in production at a single-user scale:**
| Quantity | Approximate value |
|---|---|
| 30-day total agent spend | ~\$700/mo |
| 30-day total tokens billed | ~800M |
| Turns per month | ~860 (~29/day; one active agent loop) |
| Average tokens per turn | ~900K |
| Average cost per turn | ~\$0.85 |
| Anthropic prompt-cache hit rate | ~88% |
A "turn" here is one agent loop iteration: read user message, plan, execute tool calls (including gbrain searches), generate response. Each turn typically includes 2-4 gbrain searches.
**Per-mode scaling from the tokenmax anchor:**
The cost difference between modes is concentrated in the search-attributable fraction of per-turn cost. System prompt, tool definitions, conversation history, and reasoning tokens don't change with mode — only the chunks gbrain delivers do. Assume 3 searches per turn at the mode's `searchLimit`:
| Mode | Search tokens/turn | Search cost/turn (at \$3/M effective) | Search-attributable @ 860 turns | Δ vs tokenmax |
|---|---|---|---|---|
| tokenmax | ~60K (3 × 20K) | ~\$0.18 | ~\$155/mo | — |
| balanced | ~30K (3 × 10K) | ~\$0.09 | ~\$77/mo | -\$78 |
| conservative | ~12K (3 × 4K) | ~\$0.036 | ~\$31/mo | -\$124 |
**Implied total agent spend by NATURAL PAIRING** (mode + matched
downstream model). Per-turn cost scales with the downstream model's
per-token rate, since the cached prefix + uncached portion + reasoning
tokens all bill at that rate:
| Pairing | Per-turn cost | Total @ 860 turns/mo |
|---|---|---|
| tokenmax + Opus (frontier, max quality) | ~\$0.85 | ~\$700/mo |
| balanced + Sonnet (the sweet spot) | ~\$0.50 | ~\$430/mo |
| conservative + Haiku (cost-sensitive) | ~\$0.20 | ~\$170/mo |
**4x spread across natural pairings.** The model tier dominates because
the per-token rate applies to the WHOLE per-turn payload (system + tools
+ history + reasoning + search), not just gbrain's chunks. Mode choice
contributes ~10-20% on top of that base.
**Mismatched pairings push you off the curve:**
| Pairing | Per-turn estimate | Total @ 860 turns/mo | Compared to natural |
|---|---|---|---|
| tokenmax + Haiku | ~\$0.20 | ~\$170/mo | Same cost as conservative+Haiku, worse quality |
| conservative + Opus | ~\$0.75 | ~\$640/mo | 92% of tokenmax+Opus spend, conservative-shape retrieval |
The mismatch math says: a tokenmax+Haiku user pays the same as
conservative+Haiku but gets a noisier context (Haiku can't filter signal
from 50 chunks). A conservative+Opus user pays nearly the same as
tokenmax+Opus but starves Opus on retrieval depth. Both burn budget for
no improvement.
**What this anchor tells us that the per-query math doesn't:**
1. **At realistic agent-loop scale with disciplined prompt caching, mode choice saves 10-20% of total agent spend** — meaningful, but smaller than the per-query 5x ratio implies. Disciplined prompt-cache layouts blunt the mode delta because most of the per-turn cost is the cached prefix, not the search payload.
2. **Without that prompt-cache discipline, the per-query framing reasserts itself.** Setups that churn the prompt prefix on every turn (frequent system-prompt edits, untemplated tool defs, no prompt-cache structuring) see search payload contribute a much larger fraction of total cost. Those setups should care about mode choice more, not less.
3. **The cache hit rate quoted here (~88%) is achievable but not automatic.** It requires structuring the prompt so the cached prefix stays stable across turns: system prompt + tool defs first, history compacted but cache-aware, retrieved chunks appended LAST (where their volatility doesn't invalidate the prefix). Agents that interleave search results inside the cached region pay the prefix-rebuild tax on every turn.
**Caveats stacked here:**
- The anchor represents ONE power-user loop. Multi-user fleets aggregate proportionally; the per-user shape doesn't change.
- The "3 searches per turn" assumption varies wildly. A code-review agent might issue 10+ searches per turn; a chat-only loop might do 0.
- The 88% cache hit rate is the high end of what's achievable. Half that is closer to a default agent without cache-aware prompt layout.
- The "Δ vs tokenmax" math assumes the OTHER cost components (system, tools, history, reasoning) stay constant. In practice, conservative's smaller per-turn payload also leaves more room in the context window for history → which can change agent behavior in either direction.
This anchor + the per-query math both live in this doc on purpose. The per-query framing is what an isolated benchmark would measure (and what `gbrain eval run-all` will produce). The realistic-scale anchor is what an operator actually pays. Both are honest; neither is the whole truth.
## Reproducibility footer
Every release that publishes eval numbers includes a footer with:
- Code commit SHA
- Dataset SHA (LongMemEval, BrainBench, Replay)
- `--seed N`
- Run commands verbatim
- API model identifiers used (Anthropic + OpenAI + judge model)
Without these, the numbers are unfalsifiable. With them, anyone with API keys can re-score.
-102
View File
@@ -1,102 +0,0 @@
# Content Guardrail Seams
GBrain exposes **vendor-neutral guardrail seams** at the boundaries where
external content enters the retrieval layer and where queries/tool-inputs enter
the LLM gateway. A guardrail is any external classifier — a content firewall, a
prompt-injection detector, a PII scrubber — that wants to *observe* content at
those boundaries.
The OSS distribution ships **inert**: zero guardrails are registered by default,
and every seam is a no-op until an operator registers a provider.
## Design contract (hard invariants)
These hold for every seam and are enforced by `test/guardrails.test.ts`:
- **Observe-only.** `runGuardrails()` returns `void`. Callers never branch on a
provider verdict. A guardrail registered through this interface *cannot*
block, rewrite, drop, retry, or reorder GBrain behavior. Enforcement, if ever
added, will get its own explicitly-named seam and its own RFC — it will not
silently reuse this one.
- **Fail open.** Missing config, provider throw/reject, timeout, and network
error are all swallowed. A broken guardrail never breaks an ingest, a query,
or a tool call.
- **Inline await.** Hooks await the provider before proceeding, so the
classifier sees content at the exact pre-persist / pre-inference moment.
- **No verdict persistence.** GBrain writes no guardrail rows. Providers own
their own audit trail.
- **Content boundaries.** Hooks pass only the ingest/user-facing payload — the
markdown/code body, the last user message, the expansion query, the tool
input. They never pass system prompts, full chat history, tool *output*, LLM
output, embeddings, or multimodal/OCR/rerank payloads.
## The five seams
All seams call `runGuardrails({ hook, content, metadata })` from
`src/core/guardrails.ts`.
| `hook` | Location | Fires |
| --- | --- | --- |
| `file_storage.markdown` | `import-file.ts``importFromContent` | After `parseMarkdown` + size guard, **before** content-sanity, hashing, chunking, embedding, DB write |
| `file_storage.code` | `import-file.ts``importCodeFile` | After code size guard, **before** hashing, code-chunking, embedding, DB write |
| `ai_gateway.chat` | `ai/gateway.ts``chat` | On the **latest user message only**, before provider inference |
| `ai_gateway.expand` | `ai/gateway.ts``expand` | On the query, before the expansion model call |
| `ai_gateway.tool_input` | `ai/gateway.ts``toolLoop` | On `{toolName, input}`, before pending-persist and before tool execution |
The two `file_storage.*` hooks cover every natural ingest caller that routes
through `importFromContent` / `importCodeFile`: `gbrain import`, sync, capture,
`put_page`, subagent `brain_put_page`, trusted-workspace writes,
`ingest_capture`, inbox daemon dispatch, reindex, code reindex, and the public
import APIs.
## Writing a guardrail provider
```ts
import { registerGuardrailProvider, type GuardrailInput } from 'gbrain/core/guardrails';
registerGuardrailProvider({
id: 'my-firewall',
async classify(input: GuardrailInput) {
// input.hook — which boundary ('file_storage.markdown', etc.)
// input.content — the raw text to classify
// input.metadata — provider-opaque context (slug, source_kind, tool_name, model, ...)
//
// Do your own timeout/retry/logging here. The return value is IGNORED by
// GBrain — return a typed verdict only if your own audit code consumes it.
await fetch(MY_API, { method: 'POST', body: JSON.stringify({ text: input.content }) });
},
});
```
Register once at process init (e.g. from a plugin entry or an operator boot
hook). Registration is idempotent by `id`, so a re-init won't double-fire.
### Provider responsibilities
GBrain deliberately keeps the seam minimal. The provider owns:
- **Timeout discipline.** GBrain does not impose a timeout in `runGuardrails`
so you can tune per-deployment latency. Use an `AbortController`.
- **Secret handling.** Read API keys from env at call time. Never log the key.
- **Redacted logging.** Don't log raw classified content (it may itself be the
payload you're trying to protect). Log a hash + verdict, not the body.
- **Async fan-out.** If you don't want to block ingest on your classifier,
enqueue inside `classify` and return immediately. The seam awaits *your*
function; what it does is up to you.
## Example: shadow-mode firewall provider
A typical "shadow mode" provider (classify, log a redacted verdict, change
nothing) is ~80 lines and lives entirely in the provider's own package. See
the reference provider doc shipped to integration partners for a complete
`classify` implementation that:
1. resolves `<base>/classify` from an env URL,
2. posts `{ text, hook, metadata }` with an `x-api-key` header,
3. parses a `{ prediction, blocked, score, threshold }` response,
4. emits one redacted stderr line (`status=… prediction=… content_sha256=…`),
5. fails open on every error path.
Because the verdict is ignored by GBrain, "shadow mode" requires *no* special
GBrain flag — it is the only mode this interface supports. Enforcement would be
a separate, future, RFC-gated seam.
-199
View File
@@ -1,199 +0,0 @@
# How a downstream agent should talk to gbrain
This guide is for authors of downstream agents (hermes, openclaw, future
forks) that need to call gbrain operations from their own runtime. Reading
this first will save you a debugging cycle: gbrain has **two distinct
surfaces**, and which one you pick depends on the operation.
## The two surfaces
```
┌─────────────────────────────────────────────┐
│ gbrain process │
│ │
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
│ │ (HTTP + OAuth) │ │ admin ops │ │
│ │ │ │ │ │
│ │ search, query, │ │ sync, embed, │ │
│ │ put_page, │ │ dream, doctor,│ │
│ │ get_page, │ │ autopilot, │ │
│ │ find_experts, │ │ init, secrets │ │
│ │ ... │ │ │ │
│ └──────────────────┘ └────────────────┘ │
│ ▲ ▲ │
│ │ │ │
│ │ │ │
│ thin-client OAuth shell-job `inherit:`
│ (preferred for (only path for │
│ MCP-equivalent ops) localOnly ops) │
└─────────────────────────────────────────────┘
```
The two surfaces are **not interchangeable**. Pick by op, not by preference.
## Surface 1 — MCP ops over HTTP (thin-client + OAuth)
Use for any operation that has an MCP equivalent: `search`, `query`,
`put_page`, `get_page`, `find_experts`, `find_orphans`, `find_anomalies`,
`get_recent_salience`, `find_trajectory`, and so on. The canonical list is
the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
(or `false`).
### Setup
The host runs gbrain as a long-lived HTTP server:
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
```
The agent registers as an OAuth client (one-time):
```bash
gbrain auth register-client hermes \
--grant-types client_credentials \
--scopes read,write
# Prints client_id + client_secret one-time. Store securely.
```
The agent's runtime calls `/mcp` with a bearer token from `client_credentials`
grant. Secrets stay in the gbrain serve process; the agent never sees
DATABASE_URL or API keys.
Thin-client mode (`gbrain init --mcp-only`) gives the agent the same
client-credentials wiring, plus the `gbrain` CLI itself routes MCP-eligible
commands through the configured remote MCP. The agent can call
`gbrain search` / `gbrain query` directly and the CLI does the OAuth dance.
### Why this is preferred for MCP ops
- Secrets never leave the server process.
- OAuth scopes give you `read`, `write`, `admin` separation — agent only gets
what it needs.
- Source-scoped tokens (`--source dept-x` on `register-client`) confine the
agent to a specific source within a federated brain.
- One audit surface (`mcp_request_log`) covers every op call uniformly.
## Surface 2 — localOnly admin ops via shell-job `inherit:`
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
list (as of v0.36.5.0) includes:
- `sync` (filesystem walks need local FS access)
- `embed` (orchestrates the embed pipeline)
- `extract` (walks markdown files)
- `dream` (synthesis cycle)
- `doctor` (filesystem hygiene checks)
- `autopilot` (background daemon orchestration)
- `init` (creates `~/.gbrain/`)
- `secrets` (config management)
For these, the agent cannot route through HTTP MCP. The only path is to run
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
DLQ / audit trail all come for free.
### Setup
```bash
gbrain jobs submit shell --params '{
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
"cwd": "/data/gbrain",
"inherit": ["database_url"]
}'
```
The `inherit: ["database_url"]` field tells the worker to look up
`database_url` from its `loadConfig()` and inject the value into the child
env as `GBRAIN_DATABASE_URL`. The DB row in `minion_jobs.data` carries the
names only — `inherit: ["database_url"]` — never the value. See
[minions-shell-jobs.md#secrets](./minions-shell-jobs.md#secrets) for the
full validation rules and error catalog.
### Why this is preferred over writing secrets into `env:` per-job
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
validation. The error message names `inherit: ["database_url"]` as the
replacement.
### Worker setup (one-time, per host)
The agent's host needs a worker that processes shell jobs:
```bash
# One-shot inline execution (PGLite or Postgres):
gbrain jobs submit shell --params '{...}' --follow
# Persistent worker (Postgres only — PGLite uses --follow inline):
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work
```
`GBRAIN_ALLOW_SHELL_JOBS=1` is the worker-side opt-in. Without it, shell jobs
sit in `waiting` indefinitely. Set it on the worker process env (or in your
deploy unit / launchd plist), not per-submission — submitter env is a weak
proxy for worker env.
## Decision table
| Operation | Surface | Why |
|---|---|---|
| `search` / `query` | HTTP MCP via thin-client | Has MCP op; OAuth-scoped. |
| `get_page` / `list_pages` | HTTP MCP | Same. |
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
## Recommended patterns
- **Prefer `inherit:` for secrets you don't want in the row.** Names land in
`minion_jobs.data`; values resolve at child-spawn from the worker's config.
If a brain DB ever traverses a trust boundary, secrets stay out.
- **Free-form names.** `inherit:` accepts any snake_case config-key on your
worker — `database_url`, `anthropic_api_key`, `openai_api_key`,
`voyage_api_key`, `groq_api_key`, `zeroentropy_api_key`, or any custom
field you stuff into `~/.gbrain/config.json`. The agent picks what it
needs.
- **`env:` still works** for non-secret values, or for cases where you
WANT the value in the row (e.g. an opaque correlation token your audit
flow needs to read back later). The validator doesn't second-guess you.
- **Never try to route a `localOnly` op through thin-client MCP.** It will
fail with `localOnly op refused in thin-client mode`. Use shell-job +
`inherit:` (for secrets) or `env:` (for non-secrets).
## Migration: from pre-v0.36.5.0
If your agent submits shell jobs that pass secrets via `env:`:
```jsonc
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
"env": { "GBRAIN_DATABASE_URL": "postgresql://..." }
}
```
Switch to (recommended):
```jsonc
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
"inherit": ["database_url"]
}
```
Make sure the worker host has `database_url` configured (either via
`gbrain config set database_url <value>` or via `GBRAIN_DATABASE_URL` /
`DATABASE_URL` env on the worker process). If the worker can't resolve the
key, the validator rejects the job at submit time with a paste-ready hint.
+4 -121
View File
@@ -46,13 +46,10 @@ pass:
**What the env allowlist does AND does not do.** Shell jobs run with a minimal
env: `PATH, HOME, USER, LANG, TZ, NODE_ENV`. Your secrets like `OPENAI_API_KEY`
and `DATABASE_URL` are NOT passed to the child. You opt-in additional keys per
job via `env: { ... }` (non-secret values only — see "Secrets" below) or via
`inherit: ["database_url"]` (recommended for secrets — names only in the row,
values resolved at child-spawn from `gbrain config set`). This stops accidental
`$OPENAI_API_KEY` interpolation in a user-authored script. It does **not**
sandbox filesystem reads: a shell script can `cat ~/.env` or any file the
worker process can read. The operator picks a safe `cwd`. That is the trust
boundary.
job via `env: { ... }`. This stops accidental `$OPENAI_API_KEY` interpolation in
a user-authored script. It does **not** sandbox filesystem reads: a shell
script can `cat ~/.env` or any file the worker process can read. The operator
picks a safe `cwd`. That is the trust boundary.
**Audit trail, not forensic insurance.** Every submission writes a JSONL line
to `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override
@@ -109,115 +106,6 @@ Note: `--follow` blocks the crontab slot until the job finishes. If 14 shell
crons land at the same minute and each takes 30s, they serialize through
crontab's spawning limits. Postgres + persistent worker scales better.
### Calling `gbrain` itself from a shell job — use `inherit:` for DATABASE_URL {#secrets}
A common pattern is submitting shell jobs that run `gbrain` CLI commands:
```bash
gbrain jobs submit shell --params '{
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
"cwd": "/data/gbrain",
"inherit": ["database_url"]
}'
```
`inherit: ["database_url"]` tells the worker to look up `database_url` from its
own `loadConfig()` (file + env merged) and inject the value into the child's
env as `GBRAIN_DATABASE_URL`. The job row in `minion_jobs.data` stores
`inherit: ["database_url"]`**names only, never values**. The shell-audit
JSONL records the same. Pre-enqueue validation rejects the submission if the
worker can't resolve the requested key, with a paste-ready
`gbrain config set database_url <value>` hint.
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
wrote things like:
```jsonc
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
{
"cmd": "gbrain stats",
"cwd": "/data/gbrain",
"env": { "GBRAIN_DATABASE_URL": "postgresql://..." }
}
```
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
forbid that pattern — the validator trusts the agent — but **prefer
`inherit:`** for any secret you want kept out of the row. Names land in the
row; values resolve at child-spawn from the worker's config.
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
config-key name and the worker resolves the value from `loadConfig()` at
child-spawn time:
- `inherit: ["database_url"]` → child env `GBRAIN_DATABASE_URL`
- `inherit: ["anthropic_api_key"]` → child env `ANTHROPIC_API_KEY`
- `inherit: ["openai_api_key"]` → child env `OPENAI_API_KEY`
- `inherit: ["voyage_api_key"]` → child env `VOYAGE_API_KEY`
- `inherit: ["groq_api_key", "zeroentropy_api_key"]` → both injected
- Or any arbitrary config-key your worker has (`my_custom_field`
`MY_CUSTOM_FIELD`)
The env-key name is derived by uppercasing the config-key name. The one
override is `database_url``GBRAIN_DATABASE_URL` (plain `DATABASE_URL` is
ambiguous in most Postgres-app contexts).
Pre-enqueue validation fail-fasts if the worker can't resolve a requested
name. The validator does NOT police which secrets you choose to inherit —
the agent submitting the minion is in the same uid as the worker, so it's
your call.
**Output-side leakage (read this).** The `inherit:` allowlist prevents
secrets from landing in the JOB ROW INPUT fields (`data.cmd`, `data.argv`,
`data.env`). By default it does NOT scrub the OUTPUT fields — if your
script prints the secret to stdout or stderr (`echo "$GBRAIN_DATABASE_URL"`,
`psql "$GBRAIN_DATABASE_URL"` echoing the URL on error), the value lands
plaintext in `result.stdout_tail` / `result.stderr_tail` / `error_text`,
and from there into the brain DB row.
**`redact_secrets: true` opts into output-side scrubbing.** Set it per-job
(or pass `--redact-secrets` on the CLI):
```bash
gbrain jobs submit shell --params '{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
"inherit": ["database_url"],
"redact_secrets": true
}'
# Or, equivalently:
gbrain jobs submit shell \
--params '{"cmd":"gbrain sync --skip-failed","cwd":"/data/gbrain","inherit":["database_url"]}' \
--redact-secrets
```
When `redact_secrets: true`, the worker resolves each name in `inherit:` to
a value, runs the child, then string-replaces every occurrence of those
values in `stdout_tail` / `stderr_tail` (and in the `error_text` derived from
`stderr_tail` on non-zero exit) with `<REDACTED:name>` before persistence.
Only `inherit:`-resolved values are scrubbed; caller-supplied `env:` values
are not (those are the "I'm fine with this in the row" channel by design).
**Heuristic, not perfect.** The redactor uses literal string-replace. A
script that base64-encodes the secret before printing, or that emits it
one character at a time, will bypass the scrub. Those are adversarial
shapes — the agent + the script are in the same trust domain, so this
layer defends against accidental echo (the common case), not deliberate
exfiltration.
**Three rules for shell-job authors who deal with secrets:**
- **Prefer not to echo secrets at all.** Even with `redact_secrets`, less
output means less risk if the redactor ever has an edge-case miss.
- **Wrap noisy CLI tools to suppress URLs on error.** `psql --quiet`,
`pg_dump --quiet`, or pipe through
`2>&1 | sed 's|postgresql://[^@]*@|postgresql://REDACTED@|g'`.
- **Inspect with `gbrain jobs get <id>` after a failure** to verify what
actually persisted.
### Submitting with `argv` (no shell interpolation)
For programmatic callers assembling commands from JSON, use `argv` instead of
@@ -273,11 +161,6 @@ gbrain jobs list --status waiting --name shell
| `shell: cwd is required and must be an absolute path` | `cwd` must be a string starting with `/`. | Set `cwd` in `--params` to an absolute path. |
| `shell: argv must be an array of strings` | `argv` has a non-string entry or isn't an array. | Pass `argv: ["bin","arg1","arg2"]`. |
| `shell: env values must all be strings` | `env` has a number/bool/object value. | Stringify: `"env":{"COUNT":"3"}` not `"env":{"COUNT":3}`. |
| `shell: inherit must be an array of config-key names` | `inherit` wasn't an array. | Pass `"inherit": ["database_url", ...]`. |
| `shell: inherit entries must be non-empty strings` | An element of `inherit` was empty, non-string, or null. | Use snake_case config-key names like `database_url`, `anthropic_api_key`. |
| `shell: inherit name "<X>" must match [a-z][a-z0-9_]*` | Name failed snake_case regex (uppercase, leading digit/underscore, special char). | Use the config-key name verbatim — `database_url`, not `DATABASE_URL`. |
| `shell: inherit requested "<X>" but worker has no <X> configured` | Worker can't resolve the requested name from `loadConfig()`. | Run `gbrain config set <X> <value>` on the worker host, OR check the config file at `~/.gbrain/config.json`. |
| `shell: redact_secrets must be a boolean if set` | Caller passed a non-boolean for `redact_secrets`. | Pass `true` or `false` (or omit). The CLI `--redact-secrets` flag sets it automatically. |
| `permission_denied: shell jobs cannot be submitted over MCP` | An MCP client tried to submit a shell job. By design CLI-only. | Submit from CLI or via a trusted operation handler (`ctx.remote === false`). |
| `protected job name 'shell' requires CLI or operation-local submitter` | A caller invoked `MinionQueue.add('shell', ...)` without the `trusted` opt-in. | Pass `{ allowProtectedSubmit: true }` as the 4th arg. CLI and `submit_job` do this automatically. |
| `aborted: timeout` / `aborted: cancel` / `aborted: shutdown` / `aborted: lock-lost` | The worker's abort signal fired mid-execution. Child got SIGTERM, 5s grace, then SIGKILL. | Expected: timeout / user cancel / deploy restart / stall. Inspect `gbrain jobs get` to see which. |
-79
View File
@@ -34,85 +34,6 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
## v0.26.7 — auto-RLS event trigger and one-time backfill
Starting in v0.26.7 (migration v35), gbrain ships two changes that close the
gap where a table could exist in your `public` schema without RLS for any
amount of time at all.
**1. The event trigger.** A Postgres DDL event trigger named
`auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY`
on every newly created `public.*` table. It covers `CREATE TABLE`,
`CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres
reports as a table-creation command. Tables created by gbrain itself, by
your other apps sharing the same Supabase project (Baku, Hermes, anything),
or by a human running raw SQL all get RLS enabled the moment they exist.
Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly
ignored — Supabase manages those, and we should not touch them.
**2. The one-time backfill.** When you upgrade to v0.26.7, the migration
walks every existing `public.*` base table whose RLS is off and whose comment
doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS
on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op
on every brain.
### Breaking change: read this before upgrading
If you have public tables that are intentionally RLS-off and you want them
to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before**
running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public
table that doesn't carry the exact comment contract documented below. There
is no `--dry-run` flag on the migration.
The minimum cost of getting this wrong is one round-trip: the operator runs
the SQL to enable RLS on a table that should have been exempt, then
`ALTER TABLE … DISABLE ROW LEVEL SECURITY` and adds the exempt comment to
prevent a re-flip on a later doctor run. No data is lost.
### Cross-app implications
If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates
tables in the same Supabase project, the trigger will enable RLS on those
tables too. Two ways to handle that:
1. **The app's connection role has BYPASSRLS** (e.g. it's also using the
`postgres` role). Newly created tables get RLS on but the app reads/writes
freely because BYPASSRLS bypasses policies entirely.
2. **The app's role does NOT have BYPASSRLS.** Then the app needs to add a
`CREATE POLICY` immediately after creating the table, granting itself
the read/write access it needs. The trigger does NOT add policies — it
only enables RLS, leaving the deny-by-default posture in place until the
app's policy lands.
If neither condition holds, the app will fail to read its own freshly-created
tables. The fix is at the app side, not gbrain's: either grant BYPASSRLS or
ship a policy.
### What if the trigger gets dropped?
`gbrain doctor` includes a new `rls_event_trigger` check that verifies the
trigger is installed and enabled. If you drop it manually for any reason
(debugging, migration testing, anything), doctor warns and gives you the
recovery command:
```
gbrain apply-migrations --force-retry 35
```
Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS`
and recreates cleanly.
### Why no FORCE ROW LEVEL SECURITY?
Postgres has two RLS dials. `ENABLE` blocks anon/authenticated; `FORCE` also
blocks the table OWNER unless they hold BYPASSRLS. We use `ENABLE` only,
matching the posture in `src/schema.sql`, migrations v24, and v29. `FORCE`
would lock non-BYPASSRLS apps out of their own freshly-created tables (the
trigger function inherits the caller's role, not the gbrain role) — which
defeats the cross-app coexistence story above. If you want defense-in-depth
`FORCE` on a specific gbrain-owned table, add it explicitly in your own
migration; gbrain's auto-RLS does not opt you in by default.
## The 1% case: deliberate exemption
Sometimes a public table is supposed to be readable by the anon key. An
-310
View File
@@ -1,310 +0,0 @@
# Scaling skills past 300 without drowning the context window
When an agent grows past 100 skills, a wall starts forming. Sessions take
longer to start. The model gets a little dumber about which skill to pick.
Tokens that should be powering reasoning are powering a skill catalog the
model reads on every turn whether it needs to or not.
This guide is the recipe for breaking through that wall without deleting
capabilities. Three tiers, one resolver, one safety net. Production-tested
on a 306-skill agent (Garry's OpenClaw, the agent behind Y Combinator's
president). The pattern works whether you run OpenClaw, Hermes, Claude Code,
Cursor, or your own MCP-aware agent.
## The problem
OpenClaw scans every skill file on disk at session start and injects them
into the system prompt as `<available_skills>` entries. The model sees a
name, description, and file path for each one. When a request matches, the
model reads the full SKILL.md and follows it.
This is great architecture at 50 skills. At 100, it's fine. At 200, it
starts to drag. At 300, the system prompt eats more than 25,000 tokens on
skill descriptions alone. Tokens that aren't going to reasoning, context,
or actual work.
The symptoms compound:
- Sessions take noticeably longer to start.
- The model has less room for conversation history.
- Skill routing gets fuzzier. With 300 descriptions competing for attention,
the model occasionally picks the wrong one.
- Cost goes up because every turn carries the full skill manifest.
The naive fix is to delete skills you don't use often. Don't do this. The
whole point of skills is that capabilities compound. A gift pipeline that
fires twice a month saves 30 minutes each time it does. A flight tracker
fires once per trip and prevents a missed Uber. Deleting low-frequency
skills optimizes for prompt size at the cost of capability. You wouldn't
delete apps from your phone because the home screen is too crowded. You'd
organize them.
## The three tiers
Not all skills need to be visible to the model at all times. Some are core.
Some are specialized. Some are dormant.
### Tier A: always loaded (~35 skills)
The skills the model needs on every single turn. Brain search, email triage,
calendar, meeting ingestion, content creation, the executive assistant.
They stay in the system prompt's `<available_skills>` manifest. The model
sees them natively and routes to them without any lookup.
### Tier B: resolver-routed (~85 skills)
Real, active skills that fire regularly but don't need to pollute every
turn. Gift pipeline, flight tracker, investor update ingestion, adversary
tracking, book mirror, civic intelligence. They live on disk. They have
full SKILL.md files. But OpenClaw doesn't inject them into the prompt.
Instead, a compact RESOLVER.md handles routing. One line per skill with
trigger phrases:
```markdown
- **gift-advisor**: gift idea | what should I bring | birthday gift | housewarming
- **flight-tracker**: track my flight | flight status | when does my flight land
- **investor-update-ingest**: investor update | portfolio update | company metrics
```
When the model sees "what should I bring to Jessica's dinner," it checks
the resolver, finds `gift-advisor`, reads the SKILL.md, and executes. Same
result. Zero wasted tokens on the other 84 turns where gifts aren't relevant.
### Tier C: dormant (~180 skills)
Built-in OpenClaw skills that aren't in active rotation (1Password, Discord,
Notion, Trello, integrations you haven't wired up yet) plus specialized
skills that almost never fire. They're explicitly disabled in the config
with `enabled: false`. They exist on disk as documentation and potential.
Flip one boolean to wake them up. Zero tokens contributed to every prompt
until then.
### The numbers
Before tiering, on Garry's 306-skill OpenClaw:
| Metric | Before |
|---|---|
| Skills in system prompt | 306 |
| Skill-description tokens per turn | ~25,000 |
| Skill routing accuracy | degrading |
| Session startup | slow |
After tiering:
| Metric | After |
|---|---|
| Skills in system prompt (Tier A) | 35 |
| Skill-description tokens per turn | ~4,000 |
| Skills still accessible (A + B + C) | 301 |
| Capability loss | zero |
| **Tokens freed per turn** | **~21,000** |
21K tokens per turn is not a small optimization. It's the difference between
the model having room to think and the model being squeezed. It's the
difference between carrying 3 pages of conversation history and carrying 15.
## What the resolver actually does
The resolver is cheaper than the manifest. That's the load-bearing insight.
OpenClaw's native skill manifest puts ~80 tokens per skill into the system
prompt (name + description + location). At 300 skills that's 24,000 tokens
spent every turn whether the model needs the catalog or not.
The resolver puts ~15 tokens per skill into a compact markdown list. At
300 skills that's 4,500 tokens. But it only fires when the model checks
it, which is only when the request doesn't match a Tier A skill. Most
turns, the resolver costs zero tokens because the Tier A match handles it.
This is the routing-table pattern but applied to the skill manifest itself.
The resolver routes to skills, but it also routes around skills, keeping
them out of the context window until they're needed.
GBrain ships with a [bundled `skills/RESOLVER.md`](../../skills/RESOLVER.md)
you can use as a reference shape. The skillpack story for distributing
your own resolvers across machines is covered in
[skillpacks as scaffolding](skillpacks-as-scaffolding.md).
## The compact list format (v0.41.7.0)
GBrain's resolver parser used to require markdown tables:
```markdown
| Trigger | Skill |
|---------|-------|
| "gift idea" | `skills/gift-advisor/SKILL.md` |
```
That's fine when you have 20 entries. It gets unwieldy at 200, and at 300
it's unreadable. OpenClaw deployments quietly evolved a compact list
format that scales better:
```markdown
- **gift-advisor**: gift idea | what should I bring | birthday gift
- **flight-tracker**: track my flight | flight status | when does my flight land
```
Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a
306-skill compact-format resolver, the doctor reported every skill as
unreachable: **238 FAIL errors on every doctor run**. The parser was
silently treating the compact dialect as zero skills.
v0.41.7.0 ships dual-format support. The same `parseResolverEntries`
function reads both table rows and list rows in the same file, with the
v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
`../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor`
and the 238 FAILs collapse to 0.
### The list-format contract
A few rules to keep the parser unambiguous:
- **Skill names must be kebab-lowercase.** `gift-advisor`, `flight-tracker`,
`email-triage`. Names that start with an uppercase letter (`MyTool`,
`Note`, `Convention`) are deliberately ignored. This is what stops prose
bullets like `- **Note**: see [link]` from being mis-parsed as skill
rows in real-world AGENTS.md files.
- **The path always resolves to `skills/<name>/SKILL.md`.** An optional
`→ \`skills/path\`` (or ASCII `->`) suffix is allowed for readability,
but the parser strips it. For non-conventional paths (skills under
nested directories, references into `conventions/`, anything that
isn't `skills/<name>/SKILL.md`), use the table format.
- **Triggers separate with `|`.** Empty pieces and the literal `...`
placeholder are dropped. Each trigger becomes its own resolver entry,
all pointing at the same skill.
- **Bold or plain.** `- **name**: triggers` is preferred. `- name: triggers`
works as a fallback.
You can mix table and list rows in the same file. Useful when a brain
inherits a table-format `RESOLVER.md` from gbrain and a list-format
`../AGENTS.md` from OpenClaw.
## The doctor safety net
The danger with tiering is invisible skill loss. You disable a skill from
native scanning, forget to add it to the resolver, and now the agent can't
do something it used to do. You won't notice until the moment you need it.
`gbrain doctor` walks every skill on disk and verifies it's reachable,
either through native scanning (Tier A) or through the resolver (Tier B
and C). On Garry's setup, the first run after tiering found 63 unreachable
skills. Sixty-three capabilities that existed on disk but had no routing
path. Fixed in an hour by adding resolver entries.
Run it after every skill change:
```bash
gbrain doctor
```
For CI gates, use the JSON-emitting variant:
```bash
gbrain check-resolvable --json
gbrain check-resolvable --strict # warnings fail too
```
If a skill is unreachable, the output tells you which one and suggests
the fix. The resolver is a document. Documents are cheap to fix.
## Implementation walkthrough
Three changes. Total time about 45 minutes once you've decided which
skills go in which tier.
### 1. Audit and tier your skills
Walk through every skill. Ask: does this need to fire on every turn?
- If yes → Tier A.
- If it fires weekly or less but is real → Tier B.
- If you don't use it → Tier C.
### 2. Disable Tier B and C in your agent's config
For OpenClaw, the file is `openclaw.json`. Add an entry per disabled skill:
```json
{
"skills": {
"entries": {
"gift-advisor": { "enabled": false },
"flight-tracker": { "enabled": false },
"1password": { "enabled": false }
}
}
}
```
The exact config shape depends on which agent runtime you use. The point
is the same in all of them: tell the runtime not to inject this skill into
the system prompt. The file stays on disk; only the prompt injection stops.
### 3. Write the resolver
One line per Tier B and Tier C skill. Trigger phrases that match how you
actually ask for things:
```markdown
- **gift-advisor**: gift idea | what should I bring | birthday gift
- **flight-tracker**: track my flight | flight status | when do I land
- **investor-update-ingest**: investor update | portfolio update | company metrics
```
That's it. The model handles the rest. When a request doesn't match Tier A,
it checks the resolver, reads the matching SKILL.md, and executes.
### 4. Run `gbrain doctor` and fix any unreachable skills
The doctor sweep tells you which skills don't have a routing path. Add a
resolver entry for each one, re-run, repeat until the count is zero.
## A lesson from the first version
I initially converted my resolver from a clean list format to a table
format because the validator only spoke tables. That was wrong. When a
tool fails against valid data, the right move is to fix the tool, not
reshape the data. The list format was correct, compact, readable, easy
to maintain. The parser needed to support both shapes. v0.41.7.0 is
that fix.
The same principle applies everywhere in agent systems. Your SKILL.md is
the source of truth. Your AGENTS.md is the source of truth. Your resolver
is the source of truth. When tooling disagrees with your configuration,
the tooling is wrong. Fix the tooling.
## The scaling curve
At 50 skills, you don't need any of this. Just load everything.
At 100, you start feeling the drag but can push through.
At 200, routing accuracy drops and sessions get noticeably slower. This
is where most people stop adding skills, which means their agent stops
getting more capable. Bad trade.
At 300+, tiering is mandatory. But with tiering, there's no ceiling.
1,000 skills with 35 in the hot path and 965 in the resolver is the same
per-turn cost as 35 skills with no resolver. The cost stays flat.
Capabilities compound.
The architecture that gets you from 50 to 300 is different from the
architecture that gets you from 10 to 50. That's normal. Systems that
scale change shape. The important thing is that each tier preserves full
capability. You're organizing, not deleting.
## Related
- [Skill development cycle](skill-development.md) — the 5-step loop for
turning a repeated task into a real skill.
- [Skillpacks as scaffolding](skillpacks-as-scaffolding.md) — how to
distribute a coherent set of skills across machines and agents.
- [Sub-agent routing](sub-agent-routing.md) — when to delegate to a
sub-agent vs handle in-line, and the model routing table for each path.
GBrain: [github.com/garrytan/gbrain](https://github.com/garrytan/gbrain).
The `parseResolverEntries` parser lives at
[`src/core/check-resolvable.ts`](../../src/core/check-resolvable.ts);
the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md).
-145
View File
@@ -1,145 +0,0 @@
# `gbrain skillopt` — Self-evolving skills
Treat your `SKILL.md` files as the trainable parameters of an agent that
itself never changes. Write a benchmark of realistic tasks; SkillOpt watches
the agent run them, proposes specific edits, re-tests, and only keeps changes
that measurably improve the score.
Based on [SkillOpt](https://arxiv.org/abs/2605.23904) (Microsoft Research,
May 2026).
> **New to this?** Start with the hands-on tutorial:
> [Auto-improve a skill with `gbrain skillopt`](../tutorials/improving-skills-with-skillopt.md).
> It walks you from "I have a skill" to "I accepted a measurably better version"
> in ~20 minutes, including how to write your first benchmark. This page is the
> reference — flags, exit codes, cost model, safety guards.
## The 30-second pitch
```bash
# 1. Generate a starter benchmark from the skill itself (no routing-eval needed)
gbrain skillopt my-skill --bootstrap-from-skill
# 2. Review the benchmark — STRENGTHEN the generated judges (they're weak drafts),
# then delete the trailing `# BOOTSTRAP_PENDING_REVIEW` line
# 3. Run the optimizer (--split 1:1:1 is required for a ~15-task starter)
gbrain skillopt my-skill --bootstrap-reviewed --split 1:1:1
```
That's the entire workflow. (Already have a `routing-eval.jsonl`? Swap step 1 for
`--bootstrap-from-routing` — but routing tasks test dispatch, not output quality.)
## What's in the box
```
skills/my-skill/
SKILL.md ← what gets optimized (body only; D5)
skillopt-benchmark.jsonl ← what success looks like
skillopt/
best.md ← current best version
versions/
v0001_e1_s1.md ← per-step snapshots
v0002_e1_s2.md
...
history.json ← append-only run record (D8)
rejected.json ← bounded LRU of rejected edits
```
The audit trail lives at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl`
(ISO-week rotated; honors `GBRAIN_AUDIT_DIR`).
## How the loop works
For each step:
1. **Forward pass.** Run the candidate skill against a batch from `D_train`.
2. **Backward pass.** Two reflect calls (failures + successes per D7) propose
edits to address what worked / didn't work.
3. **Rank + clip.** Top-N edits within the LR budget (cosine schedule by
default; D10 has the ASCII curve in `orchestrator.ts`).
4. **Apply.** D9 tagged-result patches the body (frontmatter forbidden per
D5; ambiguous anchors rejected to the rejected-buffer).
5. **Validation gate.** D12 median-of-3 + epsilon=0.05: every sel-task runs
the judge 3 times, takes the median; only accepts if median > best by
more than 0.05.
6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe.
After each epoch with no improvement: D6 slow-update fires one meta-edit
proposal (this lives in v0.42 follow-up; v1 emits the audit event).
## Flags
| Flag | Default | Purpose |
|---|---|---|
| `--benchmark <path>` | `skills/<n>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
| `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) |
| `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) |
| `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl |
| `--bootstrap-reviewed` | off | Required after human-reviewing bootstrap output |
| `--epochs N` | 4 | Outer-loop iterations |
| `--batch-size N` | 8 | Tasks per inner step |
| `--lr N` | 4 | Max edits per step |
| `--lr-schedule cosine\|linear\|constant` | cosine | Edit-budget decay |
| `--split TRAIN:SEL:TEST` | 4:1:5 | Ratio; refuses if D_sel < 5 |
| `--optimizer-model MODEL` | tier.deep | Reflects + proposes |
| `--target-model MODEL` | tier.subagent | Executes the skill |
| `--judge-model MODEL` | tier.reasoning | Scores rollouts |
| `--patch \| --rewrite` | patch | Edit ops only vs. full rewrites |
| `--dry-run` | off | Cost preview, no LLM calls |
| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md |
| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills |
| `--max-cost-usd N` | 5.00 | Hard cap; preflight refuses if exceeded |
| `--max-runtime-min N` | 30 | Wall-clock cap |
| `--force` | off | Bypass dirty-working-tree refusal |
| `--resume <run-id>` | off | Resume a prior interrupted run |
| `--json` | off | Machine-readable stdout |
## Exit codes
| Code | Meaning |
|---|---|
| 0 | Improved + accepted (or `--no-mutate` proposed.md written) |
| 1 | No improvement; best skill unchanged |
| 2 | Aborted by gate (dirty tree, over budget, bench validation, etc.) |
## Cost model
A typical 20-task benchmark with defaults costs ~$0.90 per run:
- 32 rollouts × Sonnet ($0.009 each) ≈ $0.29
- 8 reflect calls × Opus (cached) ≈ $0.25
- 24 sel-judges × Sonnet (cached) ≈ $0.10
- Final test eval ≈ $0.07
- **Total ≈ $0.71**
For a 100-task benchmark: ~$5.00 (right at the default cap). Preflight
refuses to start when the estimate exceeds `--max-cost-usd`.
## Safety guards (the cathedral)
| Guard | Decision | What it prevents |
|---|---|---|
| Validation gate is mandatory | D12 (paper) | Accepting LLM judge noise as improvement |
| Frontmatter mutation forbidden | D5 | Routing surface drift (`check-resolvable` regression) |
| Per-skill DB lock | D14 | Two concurrent runs corrupting history/versions |
| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain |
| Bootstrap review sentinel | D15 | Self-referential benchmark gaming |
| Read-only tool sandbox in rollouts | D13 | Optimization runs writing junk pages to your brain |
| History-intent-first atomic commit | D8 | Half-written SKILL.md on crash |
| Cost preflight | D3 | Surprise mid-run budget exhaustion |
| Dirty-tree refusal | dry-fix pattern | Overwriting your uncommitted changes |
## When NOT to use SkillOpt
- **No benchmark.** Optimizing against guesses is worse than not optimizing.
- **Write-flavored skills.** Skills whose job is to `put_page` heavily can't
use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up.
- **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful
validation needs ≥20 tasks total per the paper.
## Related skills
- `gbrain skillify scaffold <name>` — create a new skill (use BEFORE skillopt)
- `gbrain skillpack-check <name>` — audit conformance + skillopt status
- `gbrain check-resolvable` — routing MECE validation (NOT mutated by skillopt)
-208
View File
@@ -1,208 +0,0 @@
# Skillpacks as scaffolding, not amber
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
scaffold + reference library. This guide explains the model and the
workflow.
## Why we changed it
Pre-v0.33 (the "amber" model):
- `gbrain skillpack install <name>` copied bundled skills into your
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
`AGENTS.md` with a `cumulative-slugs="..."` receipt.
- Subsequent installs hash-checked every file and refused to overwrite
local edits unless you passed `--overwrite-local`.
- `gbrain skillpack uninstall` had its own data-loss safeguards (D8
receipt gate + D11 content-hash pre-scan) and rebuilt the fence.
It worked, but it treated personal-AI skills like vendor packages.
Users couldn't cleanly fork a skill without the next install fighting
them. Every release re-litigated the same managed block. The test
surface alone for the managed block was ~1000 lines.
Skills aren't vendor packages. They're first-class code in your agent
repo. You scaffold once, you own them, you fork and edit freely. When
gbrain ships a new version, you ask "what changed?" — the agent reads
the diff and decides what (if anything) to integrate.
## The five commands
### `gbrain skillpack scaffold <name> [--workspace PATH]`
One-time, additive copy of a bundled skill into your repo. Refuses to
overwrite any file that exists. Routing comes from each skill's
frontmatter `triggers:` array — gbrain does NOT touch your `RESOLVER.md`
or `AGENTS.md` (see "How agents discover scaffolded skills" below).
```bash
cd ~/git/your-agent-repo
gbrain skillpack scaffold book-mirror
# files in skills/book-mirror/ + (if the skill declares paired source)
# src/commands/book-mirror.ts land in your workspace
```
`scaffold --all` copies every bundled skill that's missing. Never
prunes.
If a skill's frontmatter declares paired source files (`sources: [...]`
in the SKILL.md YAML head), scaffold copies them too. The partial-state
policy handles "skill shipped earlier, gained a paired source later" —
scaffold copies the new paired file even when the skill dir already
exists.
### `gbrain skillpack reference <name> [--workspace PATH] [--apply-clean-hunks] [--json]`
Read-only update lens. Diffs gbrain's bundle against your local copy
and emits per-file status (`identical` / `differs` / `missing`) plus
unified diffs for any `differs` entries.
```bash
gbrain skillpack reference book-mirror
# These files live at <gbrain-path> as reference. Read them and
# decide what (if anything) to integrate into your local skills/.
# Your local edits are intentional — do not blindly overwrite.
#
# reference: identical:14 differs:1 missing:0
#
# differs /your/workspace/skills/book-mirror/SKILL.md
# --- a/skills/book-mirror/SKILL.md
# +++ b/skills/book-mirror/SKILL.md
# @@ -10,3 +10,5 @@
# ... unified diff ...
```
`reference --all` sweeps the whole bundle (one-line-per-skill summary).
`reference <name> --apply-clean-hunks` is the auto-apply path. It
parses the diff between gbrain's bundle and your local copy, applies
every hunk whose pre-change context matches uniquely. **Two-way merge
limitation**: without scaffold-time base tracking (intentionally
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
from "you changed X." Applied hunks align everything to gbrain. Use
`--dry-run` first to preview, or run plain `reference` to inspect the
diff before letting auto-apply touch anything.
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
One-shot conversion for workspaces on the pre-v0.33 managed-block
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
markers and the manifest receipt comment from your resolver file.
**Preserves every row inside the fence verbatim.** Those rows become
user-owned routing the agent can still see during the transition to
frontmatter-based discovery.
```bash
cd ~/git/your-agent-repo
gbrain skillpack migrate-fence
# migrate-fence: fence_stripped
# resolver: /your/workspace/skills/RESOLVER.md
# fenced slugs: alpha, beta, gamma
# already present: alpha, beta
# skills copied: gamma (additive — beta and alpha kept their local edits)
```
Idempotent. Re-running after migration finds no fence and exits 0.
### `gbrain skillpack scrub-legacy-fence-rows [--workspace PATH] [--dry-run]`
Opt-in cleanup. Once you've confirmed your agent walks frontmatter
`triggers:` for routing, this command removes the legacy rows that
`migrate-fence` left behind.
**Two-condition gate** (both must hold for a row to be removed):
1. `skills/<slug>/` exists on host (it was a real scaffold).
2. That skill's frontmatter declares non-empty `triggers:` (proof
that frontmatter discovery covers this skill).
Rows whose slug fails either gate are preserved — user-owned routing
the migration shouldn't touch.
### `gbrain skillpack harvest <slug> --from <host-repo-root> [--no-lint] [--dry-run]`
Inverse of scaffold: lifts a proven skill from your host repo back
into gbrain so other clients can scaffold it. Default behavior:
- Symlinks in the host skill dir are rejected (canonical-path
confinement).
- Privacy linter scans the harvested files against
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
to walk the genericization checklist before running the CLI.
## How agents discover scaffolded skills
Routing under the new model lives entirely in each skill's frontmatter:
```yaml
---
name: book-mirror
triggers:
- "personalized version of this book"
- "mirror this book"
- "two-column book analysis"
---
```
Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
frontmatter, and match the user's intent against every skill's
`triggers:` array. When a match scores high enough, invoke that skill.
This replaces the v0.32 model where `gbrain skillpack install` wrote
table rows into your `RESOLVER.md`. Rows are gone (or, for users
migrating from the old model, preserved transitionally by
`migrate-fence` until they run `scrub-legacy-fence-rows`).
If you're a downstream agent author updating to this model:
1. On startup, scan `skills/*/SKILL.md` for frontmatter.
2. Build an in-memory routing table from each skill's `triggers:`
array.
3. On every user message, match against this table — either by
substring containment, semantic similarity, or whatever your
downstream agent already does for intent classification.
## Removing a scaffolded skill
There's no `gbrain skillpack uninstall` command in v0.33. The files
in your `skills/<slug>/` are first-class members of your repo —
delete them like any other code:
```bash
rm -rf skills/book-mirror
# if the skill declared paired source files:
rm src/commands/book-mirror.ts
# (consult the skill's frontmatter `sources:` array for the full list)
# if no other scaffolded skill needs them, you can also remove the
# shared deps that scaffold drops in:
rm skills/_brain-filing-rules.md
rm -rf skills/conventions/
rm skills/_output-rules.md
```
You own the files. There's no manifest to update, no fence to rebuild.
## When to use which command (quick decision tree)
- **New host repo, want a gbrain skill**`scaffold`
- **gbrain shipped a new version, want to see what's changed**
`reference` (read-only) or `reference --apply-clean-hunks` (auto)
- **Upgrading from v0.32 or earlier**`migrate-fence` (one-shot)
- **Cleanup after `migrate-fence`**`scrub-legacy-fence-rows`
- **Lift your fork's skill back into gbrain**`harvest` + the
`skillpack-harvest` editorial skill
## What about `install` and `uninstall`?
Both are removed in v0.33. Running either prints an error pointing at
the replacement command. No deprecated alias — this is a clean break.
If you have existing scripts referencing the old names, update them
once and move on.
@@ -1,265 +0,0 @@
# Incident Report: LSD Brainstorm 53× Cost Overrun
**Date:** 2026-05-20
**Severity:** High (financial — $50.71 actual vs $0.96 estimated)
**Component:** `gbrain lsd` / `gbrain brainstorm`
**Brain size:** 13,690 pages, 16,314 links, ~2,000 unique directory prefixes
**Version:** v0.37.1.0 (first release of brainstorm/lsd)
## What Happened
A user ran `gbrain lsd "what story should Garry's List write next" --yes` on a 13,690-page brain. The command:
1. **Estimated cost: $0.96** (2×12 = 24 crosses × 4 ideas + judge)
2. **Actual cost: $50.71** — 53× over estimate
3. **Token usage:** 4,906,011 input + 2,399,239 output = 7.3M total tokens
4. **Far set pulled 1,985 pages** instead of the configured 12
5. **Generated 15,868 raw ideas** across the crosses (vs expected ~96)
6. **Judge phase failed:** 2,989,338 tokens exceeded Claude Sonnet's 1M context limit
7. **Zero ideas surfaced to the user** — complete failure
A retry with `--limit 12` explicit:
- Far set correctly returned 12 pages, cost was $0.39
- But judge still failed: `parseJudgeJSON: no strategy produced valid JSON`
- Again, 0 ideas survived to output (96 generated, 0 scored)
## Root Causes
### RC1: Far Set Explosion (caused the $50 bill)
**File:** `src/core/brainstorm/domain-bank.ts``fetchFar()``listPrefixSampledPages()`
The domain bank samples pages by directory prefix to get diversity. `listPrefixSampledPages` returns **one page per prefix passed in**. On a 13K-page brain with ~2,000 unique prefixes (books/, civic/bundles/, civic/gl-article-*, people/, concepts/, etc.), passing all prefixes produces ~2,000 rows — not the configured `m=12`.
The cost estimator uses `m` (12) to predict crosses and cost. But the actual cross phase receives 1,985 far-set pages, producing `2 × 1985 = 3,970` crosses at 4 ideas each = 15,868 ideas.
**The estimate formula is correct for the intended behavior; the far set selection is what diverged.**
### RC2: No Cost Circuit Breaker
There is no mechanism to:
- Abort if estimated cost exceeds a threshold
- Abort mid-run if actual spend diverges from estimate
- Cap the far set size regardless of prefix count
- Warn the user that a run will be expensive before proceeding
The `--yes` flag skips the 10-second cost preview wait, removing even the manual inspection opportunity.
### RC3: Judge Context Overflow
The judge receives ALL ideas in a single prompt. With 15,868 ideas at ~350 tokens each, that's ~5.5M tokens — well beyond any model's context window.
Even on the retry with only 96 ideas, the judge failed with JSON parsing errors, suggesting the judge prompt/response format is fragile.
### RC4: Unpaired UTF-16 Surrogates in Page Content
Two crosses failed with: `The request body is not valid JSON: no low surrogate in string`
Some pages (likely OCR imports or web scrapes) contain unpaired UTF-16 surrogates. When these get serialized into the JSON request body for the LLM API, the JSON encoder produces invalid JSON.
### RC5: No Timeout on Individual Crosses
One cross timed out with no specific timeout configured. The default HTTP timeout allowed it to hang for an extended period before failing, consuming tokens on the API side.
## Observed Token Flow
```
Configured: 2 close × 12 far = 24 crosses × 4 ideas = 96 ideas + 1 judge call
Actual: 2 close × 1985 far = 3970 crosses × 4 ideas = 15,868 ideas + 1 judge call (failed)
Per-cross tokens (estimated): ~1,200 in + 600 out
Actual total: 4,906,011 in + 2,399,239 out
The judge call alone would have been:
15,868 ideas × ~350 tokens = ~5.5M tokens (prompt)
Model limit: 1M tokens (Sonnet)
Overflow: 5.5× context limit
```
## Proposed Fixes
### P1: Far Set Cap (Critical — prevents cost explosion)
`fetchFar()` must cap the number of prefixes BEFORE calling `listPrefixSampledPages`. The cap should be `max(m * 4, 50)` to allow some diversity headroom while preventing runaway growth. Final selection trimmed to `m` by distance score.
**Status:** Implemented in `dc080ac2`.
### P2: Cost Guardrails (Critical — defense in depth)
New flags for `brainstorm` and `lsd` commands:
- `--max-cost <usd>` (default $5): hard-abort if pre-run estimate exceeds
- `--strict-budget`: abort mid-run if running cost exceeds 5× estimate
- `--max-far-set <n>` (default 50): explicit far set size cap
**Status:** Implemented in `dc080ac2`.
### P3: Judge Chunking (Critical — prevents context overflow)
Split ideas into batches of ~100 before calling the judge LLM. Each batch is a separate API call; results concatenated. This bounds per-call token usage to ~35K regardless of total idea count.
**Status:** Implemented in `dc080ac2`.
### P4: Unicode Sanitization (Medium — prevents cross failures)
Strip unpaired UTF-16 surrogates from page content before building cross prompts. This is a general problem for any gbrain function that serializes user-generated page content into JSON for API calls.
**Status:** Implemented in `dc080ac2`.
### P5: Global Token & Time Budgets for All Analysis Functions (Proposed)
**This is the bigger architectural ask.** Every gbrain command that makes LLM calls should respect configurable budgets:
```yaml
# Proposed config additions to ~/.gbrain/config.json
budgets:
# Global defaults
default:
max_input_tokens: 500_000 # per-command input token cap
max_output_tokens: 200_000 # per-command output token cap
max_cost_usd: 5.00 # per-command dollar cap
max_runtime_seconds: 300 # 5-minute wall-clock cap
# Per-command overrides
brainstorm:
max_cost_usd: 2.00
max_runtime_seconds: 120
lsd:
max_cost_usd: 5.00
max_runtime_seconds: 300
dream:
max_cost_usd: 10.00
max_runtime_seconds: 600
extract:
max_input_tokens: 1_000_000
max_runtime_seconds: 900
enrich:
max_cost_usd: 3.00
max_runtime_seconds: 180
```
**Commands affected:**
- `brainstorm` / `lsd` — bisociation crosses + judge (this incident)
- `dream` — dream cycle phases (enrichment, emotional weight, etc.)
- `extract all` — link + timeline extraction across all pages
- `enrich` — per-page deep enrichment with web research
- `eval` — evaluation runs (suspected-contradictions, retrieval drift)
- `integrity auto` — automated content repair
- `doctor --remediate` — autonomous self-healing via Minions
**Implementation approach:**
1. Add a `BudgetTracker` class that wraps LLM calls with token/cost/time accounting
2. Every analysis function receives a budget context
3. On budget exhaustion: save partial results, emit a structured warning, exit cleanly
4. CLI flags (`--max-cost`, `--max-tokens`, `--timeout`) override config defaults
5. `--no-budget` escape hatch for power users who know what they're doing
### P6: Diarization / Summarization for Oversized Payloads (Proposed)
When a judge or analysis phase receives more content than fits in context:
1. **Estimate tokens** before calling the LLM
2. If over budget, **diarize**: summarize/compress the content to fit
3. For the judge specifically: rank ideas by a cheap heuristic first (keyword overlap, novelty score), then send only top-N to the LLM judge
4. For other analysis: progressive summarization — chunk → summarize → merge summaries → final analysis
This is effectively a **token budget allocator** that decides how to spend a fixed token budget across variable-length inputs.
```
Example: 15,868 ideas need judging, context limit 900K tokens
Step 1: Cheap pre-filter (keyword dedup, obvious duplicates) → 8,000 unique ideas
Step 2: Batch into 80 chunks of 100 ideas each
Step 3: Judge each chunk → 80 calls × ~35K tokens = 2.8M total (spread across calls)
Step 4: Merge top ideas from each chunk → final ranking
Total cost: ~$2-3 instead of $50
```
### P7: Structured Error Recovery (Proposed)
When a cross or judge call fails:
- Save the partial results immediately (don't wait for the full run)
- Emit a machine-readable error event (not just a log warning)
- Support `--retry-failed` to re-run only the failed crosses without repeating successful ones
- Checkpoint progress to disk so interrupted runs can resume
## Impact
- **Financial:** $50.71 wasted on a single failed run
- **User trust:** Zero ideas delivered despite ~7M tokens processed
- **Time:** ~15 minutes of compute time, plus overnight delay in reporting results
## Lessons
1. **First run of any new feature on a large brain should be dry-run or capped.** The estimate was based on small-brain testing; 13K pages is a different universe.
2. **Cost estimators must account for actual data cardinality, not just configured parameters.** The estimate used `m=12` but the real far set was `|prefixes|`.
3. **Every LLM-calling function needs a budget.** This isn't just a brainstorm problem — it's an architectural gap in any system that makes variable numbers of LLM calls based on data size.
4. **JSON serialization of user content is a landmine.** Any page could contain invalid Unicode. Sanitize at the serialization boundary, not per-feature.
## Shipped in v0.37.x (the budget cathedral wave)
P1-P4 already shipped via PR #1234 (the first fix wave). P5-P7 plus a few
architectural rounds shipped in the budget-cathedral wave that followed:
- **P1 (far set cap):** `fetchFar()` in `src/core/brainstorm/domain-bank.ts`
caps prefix sampling to `max(m*4, 50)` and trims final pages to `m` by
distance. The 2K-prefix explosion class is closed.
- **P2 (cost guardrails):** `--max-cost`, `--max-far-set`, `--strict-budget`,
`--judge-model`, `--max-ideas-per-judge-call` flags on brainstorm + lsd.
Pre-flight estimate refusal, mid-run cost-ceiling abort.
- **P3 (judge chunking):** `runJudge` in `src/core/brainstorm/judges.ts`
auto-chunks at 100 ideas/call. Context-window overflow is structurally
prevented.
- **P4 (unicode sanitization):** `sanitizeUnicode` in
`src/core/brainstorm/orchestrator.ts` strips unpaired surrogates before
serialization.
- **P5 (BudgetTracker at the gateway layer):** new
`src/core/budget/budget-tracker.ts` is the canonical primitive. The
gateway's `withBudgetTracker(tracker, fn)` composes via
`AsyncLocalStorage<BudgetTracker>` so every gateway-routed LLM call
inside the scope auto-records. `BudgetExhausted` is a typed error with
`reason: 'cost' | 'runtime' | 'no_pricing'`. `record()` throws when
cumulative spend exceeds the cap (TX1). `reserve()` hard-fails on
`no_pricing` when the cap is set + model missing from pricing maps (TX2).
- **P6 (payload-fitter):** `src/core/diarize/payload-fitter.ts` with
`'batch'` and `'summarize'` strategies. Summarize embed-clusters
(k=ceil(items/4)), Haiku-summarizes each cluster in parallel via
`Promise.allSettled` at parallelism=4. Surfaces `degraded: true` flag
when success ratio < 0.75 so callers decide whether to surface a partial
result or abort.
- **P7 (brainstorm checkpoint + --resume):**
`src/core/brainstorm/checkpoint.ts` persists FULL idea bodies (not just
counts — TX3 load-bearing). One `--resume <run_id>` flag covers both
failed and never-attempted crosses (TX4). `run_id` formula uses NO
embedding bits so the identity is stable across embedding-model swaps
(A5 amended). 7-day mtime-based GC wired into the cycle purge phase.
`--list-runs` lists saved checkpoints. `--force-resume` bypasses the 7d
staleness gate.
Also shipped alongside the wave (folded inline):
- **doctor --remediate --resume:** A4 amended. The mid-run cap is now a
real ceiling; `--max-cost` is an alias for `--max-usd`. On
BudgetExhausted, the orchestrator persists a checkpoint at
`~/.gbrain/remediation/<plan_hash>.json` and tells the user the exact
`gbrain doctor --remediate --resume` command. The resumed run skips
already-completed steps.
- **Audit-week-file consolidation (Q1):** four call sites
(shell-jobs / phantoms / slug-fallback / dream-budget) now share one
ISO-week filename helper. Year-boundary correctness pinned by tests.
- **eval-contradictions tracker telemetry:** the existing CostTracker
stays for the report shape; the runner additionally installs a
withBudgetTracker scope for the gateway-layer telemetry path.
What did NOT make this wave (filed in TODOS for a follow-up):
- The schema fix for `page_links` on PGLite. The brainstorm domain-bank
queries reference `page_links` but the embedded schema only defines
`links`; the E2E works around this with a view in test setup, but
real PGLite users currently can't run `gbrain brainstorm`. Schema fix
needed.
- `--max-cost` flag on `extract`, `enrich`, `integrity auto`. The
gateway-layer enforcement covers them when wrapped at the entrypoint,
but the CLI flag wiring is deferred.
- Async-batched audit writes. Sync `appendFileSync` is fine at typical
volumes; revisit if profiling shows it dominates.
- Multi-day brainstorm resume (>7d). The `--force-resume` flag is the
operator escape hatch for now.
-187
View File
@@ -1,187 +0,0 @@
# Embedding providers
GBrain ships with 16 embedding-provider recipes covering OpenAI, ZeroEntropy, Voyage, OpenRouter (single key, many hosted models), the major hosted alternatives, three local options, and a universal escape hatch (LiteLLM proxy). Run `gbrain providers list` to see the live registry; `gbrain providers explain --json` emits a machine-readable matrix for agents.
This page is the human-readable counterpart: capability per provider, env-var setup, dimensions, cost, and known constraints.
## Quick start
```
gbrain providers list # see all providers
gbrain providers env <provider-id> # see required env vars
gbrain providers test --model openai:text-embedding-3-large # smoke-test
gbrain init --pglite --model voyage # use a non-default provider
```
## Init resolves your provider from env keys
As of v0.37, `gbrain init --pglite` auto-detects which provider to use from your env vars. With `OPENAI_API_KEY` set, you get OpenAI. With `ZEROENTROPY_API_KEY` set, you get ZeroEntropy. If multiple provider keys are set, init fires an interactive picker. If no provider keys are set in a non-TTY context (CI, Docker build), init exits 1 with a paste-ready setup hint. Explicit flags (`--embedding-model`, `--no-embedding`) always win over env detection.
The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atomically, so subsequent runs are deterministic across releases.
## TL;DR table
| Provider | env vars | default dims | cost ($/1M tokens) | local? | multimodal? |
|---|---|---|---|---|---|
| `zeroentropyai` | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no |
| `openai` | `OPENAI_API_KEY` | 1536 | 0.13 | no | no |
| `openrouter` | `OPENROUTER_API_KEY` | 1536 | 0.02 | no | model-dependent |
| `voyage` | `VOYAGE_API_KEY` | 1024 | 0.18 | no | yes (`voyage-multimodal-3`) |
| `google` | `GOOGLE_GENERATIVE_AI_API_KEY` | 768 | 0.025 | no | no |
| `azure-openai` | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` | 1536 | 0.13 | no | no |
| `minimax` | `MINIMAX_API_KEY` | 1536 | 0.07 | no | no |
| `dashscope` | `DASHSCOPE_API_KEY` | 1024 | varies | no | no |
| `zhipu` | `ZHIPUAI_API_KEY` | 1024 | varies | no | no |
| `ollama` | (none — runs locally) | 768 | 0 | yes | no |
| `llama-server` | (none — runs locally) | user-set | 0 | yes | no |
| `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | no |
| `together` | `TOGETHER_API_KEY` | 768 | varies | no | no |
| `anthropic` | (no embedding model — chat only) | — | — | — | — |
| `deepseek` | (no embedding model — chat only) | — | — | — | — |
| `groq` | (no embedding model — chat only) | — | — | — | — |
**Note on local providers.** Ollama and llama-server have no required API key, so they don't show up in env-detection auto-pick. Pick them explicitly with `--embedding-model ollama:<model>` to avoid silently routing to a daemon that may not be running.
## If first import fails
If `gbrain import` fails with `expected N dimensions, not M`, run `gbrain doctor`. The output will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. **You should not need to delete `~/.gbrain`.** The bug-class that historically forced `rm -rf` recoveries is closed as of v0.37.
The doctor distinguishes two repair paths:
- **Empty brain** (no embedded chunks yet) — drop and re-init at the right dim:
```
gbrain init --force --pglite --embedding-model <provider>:<model> --embedding-dimensions <N>
```
- **Non-empty brain** — migrate cleanly with the supported reindex path:
```
gbrain retrieval-upgrade --to <provider>:<model> --reindex
```
## Decision tree
- **Cost-sensitive, English-only**: Ollama (free, local) or Voyage (paid, best quality per dollar).
- **Quality-first**: Voyage `voyage-4-large` (1024-2048 dims, ~3-4× more dense tokens than OpenAI tiktoken).
- **Code-heavy brain (gstack per-worktree, source repos)**: Voyage `voyage-code-3` (1024 default; supports 256/512/1024/2048). Tuned on programming languages. Voyage publishes head-to-head numbers showing it outperforms their general flagships on code retrieval ([voyageai.com/blog](https://voyageai.com/blog)). For gstack's per-worktree pglite-backed code brain, this is the right default — see Topology 3 in `docs/architecture/topologies.md`.
- **Reranking pair**: ZeroEntropy `zerank-2` is the hosted default in `tokenmax` mode (see [`docs/ai-providers/zeroentropy.md`](../ai-providers/zeroentropy.md)). Voyage `rerank-2.5` pairs cleanly with Voyage embeddings.
- **Local reranking (no API spend)**: `llama-server-reranker` recipe (v0.40.6.1) — point gbrain at your own `llama-server --reranking` instance running Qwen3-Reranker or self-hosted ZeroEntropy weights. Same `gateway.rerank()` seam, $0 per call. Walkthrough in [`docs/ai-providers/llama-server-reranker.md`](../ai-providers/llama-server-reranker.md).
- **One key for many hosted models**: OpenRouter. Set `OPENROUTER_API_KEY` and use `openrouter:<provider>/<model>` for chat against GPT-5.2, Claude 4.x, Gemini 3, DeepSeek, and dozens more without juggling per-provider keys. Embedding catalog includes OpenAI, Google, Qwen, BGE-M3.
- **Enterprise compliance**: Azure OpenAI (data residency + private endpoints) or self-hosted via llama-server / Ollama.
- **China region**: DashScope (Alibaba) or Zhipu (BigModel). DashScope's international endpoint at `dashscope-intl.aliyuncs.com`; override `provider_base_urls.dashscope` for the China endpoint.
- **OSS local, full control**: llama-server (`llama.cpp`) for any GGUF model; Ollama for the curated catalog.
- **Anything else**: LiteLLM proxy. Run LiteLLM in front of any provider (Bedrock, Vertex, Cohere, Jina, Fireworks, etc.) and point gbrain at it via `LITELLM_BASE_URL`.
## Per-provider details
### OpenAI
Default. Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 default), `text-embedding-3-small` (1536). Matryoshka via the `dimensions` field — gbrain pins it from `embedding_dimensions` config so existing 1536-dim brains stay aligned across SDK upgrades.
### Voyage AI
Best-in-class quality on the Voyage 4 family (Jan 2026 release). Set `VOYAGE_API_KEY`. Models: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-4-nano`, `voyage-3.5`, `voyage-code-3` (code-tuned), `voyage-finance-2`, `voyage-law-2`, `voyage-multimodal-3` (text + image).
Voyage 4 family shares an embedding space across all variants, so you can index with `voyage-4-large` and query with `voyage-4-lite` without reindexing. Dims: 256, 512, 1024, 2048. **2048 exceeds pgvector's HNSW cap of 2000** — those brains fall back to exact vector scans (still correct, just slower).
**For brains that index source code** (gstack's per-worktree pglite-backed code brain — see Topology 3 in `docs/architecture/topologies.md`), prefer `voyage-code-3` over `voyage-4-large`. Voyage tunes it on programming languages and publishes head-to-head numbers vs their general flagships on code retrieval. Configure at install time:
```bash
gbrain init --pglite --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024
```
To switch an existing brain, use `gbrain reinit-pglite --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024` (PGLite) or follow `docs/embedding-migrations.md` (Postgres). `gbrain config set embedding_model` is refused — the schema column has to resize.
`gbrain reindex --code` will print a recommendation when run against a brain whose configured embedding model isn't code-tuned; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1` if you've intentionally chosen another model (single-vendor procurement, compliance, etc.).
### Google Gemini
Set `GOOGLE_GENERATIVE_AI_API_KEY` (the AI Studio public API key). Model: `gemini-embedding-001`. Default 768 dims; Matryoshka up to 3072. Cheap.
For GCP service-account / Vertex AI auth (production deployments), see the v0.32.x follow-up — Vertex ADC is on the roadmap.
### OpenRouter
Single OpenAI-compatible API for fan-out to OpenAI, Anthropic, Google, DeepSeek, Meta Llama, Qwen, and dozens of other hosted providers. One key, many models. Set `OPENROUTER_API_KEY` and use `openrouter:<provider>/<model>` (e.g. `openrouter:openai/gpt-5.2`, `openrouter:anthropic/claude-sonnet-4.6`).
**Embedding**: `openai/text-embedding-3-small` (1536d default, Matryoshka shrink to 512/768/1024). OR's embedding catalog also includes `text-embedding-3-large`, `google/gemini-embedding-2-preview`, `qwen/qwen3-embedding-8b`, `bge-m3` — opt in via `--embedding-model openrouter:<id>`. Pricing matches the upstream provider (OR adds a small markup).
**Chat**: every chat model OR proxies works through `/v1/chat/completions`. The recipe lists 8 curated entry points (GPT-5.2 family, Claude 4.5/4.6/4.7, Gemini 3 Flash Preview, DeepSeek); any other OR catalog ID also works. Tool-calling envelope is supported by the OR endpoint, but per-model capability varies — check https://openrouter.ai/models before counting on tools for a specific slug.
**Optional env**:
- `OPENROUTER_BASE_URL` — point at a self-hosted OR-compatible proxy.
- `OPENROUTER_REFERER` (default `https://gbrain.ai`) and `OPENROUTER_TITLE` (default `gbrain`) — attribution headers for OR's leaderboard. Forks running gbrain inside a different agent stack (OpenClaw deployments etc.) should set these so their traffic gets attributed to them, not gbrain.
**Subagent loops**: gbrain's subagent infrastructure hard-pins to Anthropic-direct (stable `tool_use_id` across crashes/replays). OR-routed Anthropic is rejected at submit time regardless of the recipe flag. If you want the price/availability story OR offers for tool-calling, use it for chat only and keep an Anthropic key for subagent work.
### Azure OpenAI
Enterprise OpenAI behind Azure tenancy. Required env: `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` (e.g. `https://my-resource.openai.azure.com`), `AZURE_OPENAI_DEPLOYMENT` (the deployment name from your Azure portal). Optional: `AZURE_OPENAI_API_VERSION` (defaults to `2024-10-21`).
Unlike vanilla OpenAI, Azure uses `api-key:` header (not `Authorization: Bearer`) and a templated URL with `?api-version=` query param — gbrain handles both via the recipe's resolveAuth + resolveOpenAICompatConfig overrides.
Models: `text-embedding-3-large`, `text-embedding-3-small`, `text-embedding-ada-002` (your Azure deployment must serve the requested model).
### MiniMax (海螺AI)
Set `MINIMAX_API_KEY`. Optional `MINIMAX_GROUP_ID` for org-scoped accounts. Model: `embo-01` (1536 dims).
MiniMax's API takes a `type: 'db' | 'query'` field for asymmetric retrieval. v0.32 routes everything as `type='db'` (symmetric retrieval — same vector space for indexing and queries). Asymmetric query support is a v0.32.x follow-up.
### DashScope (Alibaba)
Set `DASHSCOPE_API_KEY`. International endpoint at `dashscope-intl.aliyuncs.com` by default; override `provider_base_urls.dashscope` for the China endpoint. Models: `text-embedding-v3` (current; Matryoshka 64-1024 dims), `text-embedding-v2`.
CJK-dominant content tokenizes denser than OpenAI tiktoken; gbrain declares `chars_per_token: 2` so the batch pre-split leaves headroom.
### Zhipu AI (BigModel)
Set `ZHIPUAI_API_KEY`. Models: `embedding-3` (current; Matryoshka 256-2048 dims), `embedding-2`. v0.32 default is 1024 (HNSW-compatible). The 2048-dim option works but falls into the exact-scan branch (see Voyage 4 Large note above).
### Ollama (local)
No env required — Ollama runs unauthenticated locally. Optional `OLLAMA_BASE_URL` (default `http://localhost:11434/v1`) and `OLLAMA_API_KEY` (for auth-enabled deployments).
Recipe ships with `nomic-embed-text` (768d, recommended), `mxbai-embed-large` (1024d), `all-minilm` (384d). `gbrain providers test --model ollama:nomic-embed-text` smoke-tests the local install.
### llama-server (local, llama.cpp)
`llama.cpp`'s `llama-server --embeddings` endpoint. No env required. Optional `LLAMA_SERVER_BASE_URL` (default `http://localhost:8080/v1`) and `LLAMA_SERVER_API_KEY`.
User-driven models: launch llama-server with `--model <gguf-path> --embeddings`, then run `gbrain init --embedding-model llama-server:<your-id> --embedding-dimensions <N>`. The recipe refuses the implicit shorthand `--model llama-server` because there's no canonical first model.
### LiteLLM proxy (universal escape hatch)
Run [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start) in front of any provider — Bedrock, Vertex, Cohere, Jina, Fireworks, OctoAI, etc. The proxy normalizes everything to the OpenAI-compatible API; gbrain points at the proxy via `LITELLM_BASE_URL` and proxies the call.
This is the catch-all for "my provider isn't in the list above." Set up LiteLLM, then `gbrain init --embedding-model litellm:<your-model-id> --embedding-dimensions <N>`.
## Choosing dimensions
Three numbers matter:
1. **Provider's native dims**: each model has a "true" output dim (e.g. OpenAI `text-embedding-3-large` is 3072 native).
2. **Matryoshka reductions**: most modern providers let you request a smaller vector via the `dimensions` field.
3. **HNSW cap**: pgvector's HNSW index supports up to 2000 dims. Brains above that fall back to exact vector scans (slower but correct; gbrain handles the SQL automatically via `chunkEmbeddingIndexSql` in `src/core/vector-index.ts`).
For most users: **stay at 1024 or 1536**. Bigger isn't better below the noise floor; smaller saves disk + RAM with marginal recall loss on Matryoshka providers.
## My provider isn't listed
Four options:
1. **Use OpenRouter** when the provider/model is available through OR's OpenAI-compatible API (covers most hosted chat models + a growing embedding catalog).
2. **Use LiteLLM proxy** (above) — the universal escape hatch. Works for 100+ providers.
3. **Open a feature request** at [github.com/garrytan/gbrain/issues](https://github.com/garrytan/gbrain/issues) with the provider's API docs URL and a setup snippet. Recipes are ~30-40 lines of TypeScript.
4. **Submit a recipe**: clone, copy `src/core/ai/recipes/voyage.ts` as the gold-standard openai-compat template, register in `src/core/ai/recipes/index.ts`, add a per-recipe smoke test under `test/ai/recipe-<name>.test.ts`. The recipe contract test (`test/ai/recipes-contract.test.ts`) and IRON RULE regression test pin the structural invariants.
## Switching providers on an existing brain
Embedding dimensions are baked into the schema at `gbrain init` time. As of v0.37.11.0, `gbrain config set embedding_model` and `gbrain config set embedding_dimensions` are refused — the schema column has to resize alongside the config, and `config set` only touches the config row.
The supported paths:
- **PGLite (default install):** `gbrain reinit-pglite --embedding-model <provider>:<model> --embedding-dimensions <N>` — one-command wipe-and-reinit that preserves every other config field (chat model, expansion model, API keys), backs up the prior brain to `<path>.bak`, runs `gbrain init` with the new flags, and re-syncs your brain repo. Add `--no-sync` to skip the resync, `--yes` to skip the TTY confirmation, `--json` for scripts.
- **Postgres (Supabase / self-hosted):** follow the SQL recipe in `docs/embedding-migrations.md` (drop the HNSW index, ALTER COLUMN TYPE, clear stale embeddings, recreate the index conditionally, then `gbrain init --supabase --embedding-model X --embedding-dimensions N` to update the file plane and re-embed).
`gbrain doctor` 8c "alternative_providers" surfaces unconfigured providers whose env is already set — useful when you've configured OpenAI but also have e.g. `VOYAGE_API_KEY` exported and want to know you can switch without extra setup.
`gbrain doctor` 8c "alternative_providers" surfaces unconfigured providers whose env is already set — useful when you've configured OpenAI but also have e.g. `VOYAGE_API_KEY` exported and want to know you can switch without extra setup.
+1 -1
View File
@@ -73,7 +73,7 @@ hook resumes blocking malformed pages.
## For downstream agent forks
If your OpenClaw wraps gbrain in a host repo
If your fork (Wintermute, Hermes, OpenClaw) wraps gbrain in a host repo
that's not the brain repo itself, you may want a separate hook strategy:
- **Brain repo IS the host repo** (gbrain skills + brain pages in one repo):
-164
View File
@@ -1,164 +0,0 @@
# Cross-Modal Search: Text↔Image Retrieval
## Summary
gbrain has a working multimodal embedding pipeline (Voyage multimodal-3, `embedding_image` column, 11K image chunks indexed) but search is siloed: text queries only search text embeddings, image queries don't exist. This proposal adds cross-modal query routing so text queries can surface images and image queries can surface text, using Voyage multimodal-3's shared embedding space.
## Problem
**What the user sees:** You can't search "photos from the hackathon" and get actual images. You can't upload a photo and ask "what do we know about this person?" Text search returns text. Image embeddings sit unused except via explicit `embeddingColumn: 'embedding_image'` override, which no user-facing path triggers.
**What the system does:**
- Text queries embed through the configured text model (OpenAI/ZE) and search the text column
- The `embedding_image` column exists (Voyage multimodal-3, 1024d) with 11,204 embedded chunks and a valid 83 MB HNSW index
- `postgres-engine.ts:searchVector()` supports `embeddingColumn: 'embedding_image'` but the query vector must come from a compatible model (Voyage multimodal, 1024d)
- Currently, `embedQuery()` always uses the text embedding model, producing a 1536d or 2560d vector that can't query the 1024d image column
**What it should do:**
1. Detect cross-modal intent in a search query ("show me photos of...", "find images from...", or explicit image search flag)
2. Embed the text query through Voyage multimodal-3 (same model used for image embeddings)
3. Search the `embedding_image` column with the multimodal query vector
4. Return image results alongside or instead of text results
5. Support image-as-query: accept an image input, embed it through Voyage multimodal-3, search text embeddings (if a shared multimodal column exists) or the image column
## Evidence
### Image embeddings exist and are indexed
```sql
-- Production state (May 2026)
SELECT COUNT(*) FROM content_chunks WHERE embedding_image IS NOT NULL;
-- 11,204
SELECT indexrelid::regclass, indisvalid, pg_size_pretty(pg_relation_size(indexrelid))
FROM pg_index WHERE indrelid = 'content_chunks'::regclass
AND indexrelid::regclass::text LIKE '%image%';
-- idx_chunks_embedding_image | true | 83 MB
```
### Modality metadata is broken
```sql
SELECT COUNT(*) FROM content_chunks WHERE modality = 'image';
-- 10 (should be ~11,204)
```
Most image chunks have `embedding_image IS NOT NULL` but `modality` is not set to `'image'`. This is a backfill gap from the v0.27.1 migration.
### Voyage multimodal-3 is cross-modal by design
From Voyage docs: voyage-multimodal-3 encodes text, images, and interleaved text+image into the same 1024-dimensional vector space. A text query embedded through this model can find relevant images, and vice versa. gbrain already uses it for the image column but never for query embedding.
### Search routing is text-only
`hybrid.ts` line ~414:
```typescript
const embeddings = await Promise.all(queries.map(q => embedQuery(q)));
```
`embedQuery()` always uses the global text model. No path exists to embed a text query through the multimodal model for cross-modal search.
## Proposed Fix
### Phase 1: Text → Image Search
**1. Cross-modal intent detection** (new file: `src/core/search/cross-modal.ts`)
Add a lightweight intent classifier that detects when a query is looking for images:
```typescript
function detectCrossModalIntent(query: string): 'text' | 'image' | 'both' {
// Explicit image patterns
const imagePatterns = [
/\b(show|find|get)\s+(me\s+)?(photos?|images?|pictures?|screenshots?)/i,
/\bwhat\s+does\s+.+\s+look\s+like/i,
/\b(whiteboard|diagram|slide|screenshot)\b/i,
/\bphoto(s)?\s+(of|from|at|with)\b/i,
];
if (imagePatterns.some(p => p.test(query))) return 'image';
return 'text'; // Default: text-only
}
```
**2. Multimodal query embedding** (extend `embedding.ts`)
Add `embedQueryMultimodal(text: string): Promise<Float32Array>` that routes through the configured multimodal model (Voyage multimodal-3) instead of the text model.
```typescript
export async function embedQueryMultimodal(text: string): Promise<Float32Array> {
// Use the multimodal provider, not the text provider
return gatewayEmbedQuery(text, { provider: cfg.embedding_multimodal_model });
}
```
**3. Hybrid search routing** (extend `hybrid.ts`)
When cross-modal intent is detected:
- Embed query through multimodal model (Voyage multimodal-3, 1024d)
- Search `embedding_image` column
- Return results with a `modality: 'image'` tag
- If intent is `'both'`: run text search AND image search, merge with RRF
**4. SearchOpts extension** (extend `types.ts`)
```typescript
interface SearchOpts {
// ... existing fields
crossModal?: 'text' | 'image' | 'both' | 'auto'; // Default: 'auto' (intent detection)
}
```
### Phase 2: Image → Text Search (future)
Accept an image buffer/URL as search input. Embed through Voyage multimodal-3. Search text embeddings. This requires a new search entry point (`searchByImage`) and MCP tool exposure. Defer to a follow-up PR.
### Phase 3: Unified Multimodal Column (future)
Embed ALL content (text + images) through Voyage multimodal-3 into a single column. This creates a truly unified search space but doubles embedding costs and requires re-embedding all text. Evaluate after Phase 1 results.
## Backfill: Fix modality metadata
Before cross-modal search is useful, fix the modality column:
```sql
-- Chunks with image embeddings but wrong modality
UPDATE content_chunks
SET modality = 'image'
WHERE embedding_image IS NOT NULL AND (modality IS NULL OR modality != 'image');
```
This is a prerequisite for Phase 1 since result display needs to know which chunks are images.
## Test Guidance
### Red tests (should fail before fix, pass after)
1. **Intent detection:** `detectCrossModalIntent("show me photos from the hackathon")` returns `'image'`.
2. **Intent detection negative:** `detectCrossModalIntent("what is founder mode?")` returns `'text'`.
3. **Multimodal embed routing:** `embedQueryMultimodal("hackathon")` returns a 1024d vector (Voyage multimodal dims), not 1536d or 2560d.
4. **Cross-modal search:** `hybridSearch("show me hackathon photos", { crossModal: 'image' })` returns results from the `embedding_image` column.
5. **Default behavior unchanged:** `hybridSearch("what is founder mode?")` returns text results as before (no cross-modal unless detected).
6. **Explicit override:** `hybridSearch("anything", { crossModal: 'image' })` forces image search regardless of intent detection.
### Edge cases
- Query matches image intent but no image embeddings exist for the topic: return empty image results, fall back to text.
- Multimodal model not configured: skip cross-modal, log warning, return text results.
- Mixed results ('both' mode): text and image results merged, each tagged with modality for display.
## Related Context
- PR #1106 adds dynamic text embedding column selection (prerequisite: the `embedding_columns` registry and provider routing from that PR make this easier to implement)
- v0.27.1 introduced the dual-column schema (`embedding` + `embedding_image`)
- `importImageFile` in `postgres-engine.ts` handles image ingestion and multimodal embedding
- Voyage multimodal-3 is already configured as `embedding_multimodal_model` in gbrain config
- The image OCR pipeline (`embedding_image_ocr: true`) extracts text from images before embedding, so image chunks have both visual and text representation
## Phasing
| Phase | Scope | Effort | Value |
|---|---|---|---|
| **1 (this PR)** | Text → Image search with intent detection | Medium | High — unlocks "find photos" queries |
| 2 | Image → Text search (upload photo, find related text) | Medium | Medium — cool but niche use case |
| 3 | Unified multimodal column (everything in one space) | Large | High — but expensive and requires re-embedding |
| Prereq | Fix modality column backfill | Small | Required for Phase 1 |
-224
View File
@@ -1,224 +0,0 @@
# Doctor Auto-Heal and Scoring Improvements
## Summary
The `gbrain doctor` health score system has several false-positive patterns and missing auto-heal capabilities. After the crash classification fix (shipped in this PR), these are the remaining improvements ranked by impact.
---
## 1. Frontmatter severity levels
### Problem
`NESTED_QUOTES` warnings dominate the frontmatter check (6,900+ of ~7,100 total issues). These are cosmetic YAML style issues — values like `title: "foo"` where the quotes are technically unnecessary. They don't affect sync, search, embedding, or any functionality.
By counting them the same as `YAML_PARSE` (actual parse failures) or `MISSING_OPEN` (missing frontmatter delimiters), the frontmatter check is perpetually WARN and the real issues are lost.
### Evidence
```
frontmatter_integrity: 7131 issues across 3 sources
default: 7012 (NESTED_QUOTES=6922, YAML_PARSE=90)
media-corpus: 16 (MISSING_OPEN=15, YAML_PARSE=1)
zion-brain: 103 (MISSING_OPEN=14, NESTED_QUOTES=89)
```
Only 280 of 7,131 issues are real problems. 96% are cosmetic noise.
### Proposed Fix
- Introduce severity levels: `error` (YAML_PARSE, MISSING_OPEN) vs `info` (NESTED_QUOTES)
- Doctor WARN/FAIL only on error-level issues
- Report info-level in the message text but don't affect check status
- Optional `--pedantic` flag includes info-level in status
### Test Cases
| Frontmatter issues | Severity breakdown | Expected status |
|---|---|---|
| 0 issues | n/a | OK |
| 50 NESTED_QUOTES only | 0 error, 50 info | OK (with note) |
| 3 YAML_PARSE | 3 error | WARN |
| 6900 NESTED_QUOTES + 3 YAML_PARSE | 3 error, 6900 info | WARN (mentions 3 errors) |
---
## 2. Temporal contradiction awareness
### Problem
The contradiction probe flags temporal evolutions as contradictions. Example:
- Page A (April): "Considering option X"
- Page B (May): "Decided on option Y"
These aren't contradictions — they're the same topic evolving over time. The probe has no time awareness.
### Evidence
From a probe run on 50 queries with top-k=15:
- 120 contradictions detected (112 high, 8 medium)
- After manual review: ~60% were temporal evolutions, not real conflicts
- Pages have `effective_date` or `created` timestamps that could disambiguate
### Proposed Fix
- Pass `effective_date` / `created` to the judge prompt
- Add verdict: `temporal_supersession` (later claim supersedes earlier)
- When both pages have dates and claims overlap, bias toward temporal interpretation
- Already designed in PR #993
### Test Cases
| Page A date | Page A claim | Page B date | Page B claim | Expected verdict |
|---|---|---|---|---|
| 2026-04 | "Considering X" | 2026-05 | "Chose Y" | temporal_supersession |
| 2026-04 | "Revenue is $1M" | 2026-04 | "Revenue is $500K" | contradiction |
| null | "X is true" | null | "X is false" | contradiction |
| 2025-01 | "CEO of Company" | 2026-01 | "Former CEO" | temporal_supersession |
---
## 3. Multi-source drift baseline
### Problem
4,791 pages show "multi-source drift" due to a pre-v0.30.3 `putPage` routing bug. These pages exist at the `default` source but should be at a named source. The `sources rehome` command to fix this hasn't shipped yet.
Every doctor run shows WARN for ~4,800 pages nobody can fix.
### Proposed Fix
Allow `doctor.baselines` config to acknowledge known-unfixable counts:
```yaml
doctor:
baselines:
multi_source_drift: 4800
```
When actual drift ≤ baseline: OK. When drift exceeds baseline: WARN (new drift).
Store in `.gbrain/doctor-baselines.json` so it works without config too:
```json
{
"multi_source_drift": { "count": 4800, "acknowledged_at": "2026-05-15", "reason": "pre-v0.30.3 putPage misroutes" }
}
```
### Test Cases
| Actual drift | Baseline | Expected |
|---|---|---|
| 4791 | 4800 | OK |
| 4900 | 4800 | WARN ("100 new drift beyond baseline") |
| 4791 | 0 (no baseline) | WARN (current behavior) |
---
## 4. Image assets acknowledgment
### Problem
When image files are missing from disk (stored externally, purged from git), the check permanently warns. No way to say "these are intentionally external."
### Proposed Fix
- `doctor --acknowledge image_assets` marks current missing count as accepted
- Stored in `.gbrain/doctor-baselines.json`
- WARN only for NEW missing images beyond acknowledged count
- Optional `image_assets.external_storage: true` config to skip disk check entirely
---
## 5. Auto-heal mode
### Problem
Many doctor warnings have known fixes that are safe to auto-apply:
| Warning | Auto-fix |
|---|---|
| Supervisor not running | Start supervisor |
| Stale embeddings | Submit `embed --stale` job |
| Extract coverage < 70% | Submit `extract all --skip-existing` job |
| Stale sync | Submit sync job |
| Effective date drift | Run `reindex-frontmatter` |
### Proposed Fix
`doctor --auto-heal` mode:
1. Run all checks
2. For fixable WARNs: submit fix as a job (not inline — via job queue)
3. Report what was fixed vs needs manual attention
4. Idempotent: check queue first, don't submit duplicates
5. Safety gate: never auto-heals FAILs, only WARNs
Config:
```yaml
doctor:
autoHeal:
enabled: true
minInterval: "6h"
skip:
- image_assets
- multi_source_drift
```
### Test Cases
| Check status | Auto-heal enabled | Job already queued | Expected |
|---|---|---|---|
| WARN: stale embeds | yes | no | Submit embed job |
| WARN: stale embeds | yes | yes | Skip (idempotent) |
| FAIL: max_crashes | yes | n/a | Don't auto-fix FAILs |
| WARN: stale embeds | no | n/a | Report only |
| WARN: image_assets | yes (but skipped) | n/a | Report only |
---
## 6. Score delta tracking
### Problem
No history — each `doctor` run is a snapshot. Can't tell if score is improving or degrading.
### Proposed Fix
- Write each run to `.gbrain/doctor-history.jsonl`:
```json
{"ts":"2026-05-15T12:00:00Z","score":60,"brain_score":79,"checks":{"supervisor":"ok","embeddings":"ok",...}}
```
- `doctor --trend` shows last N scores with deltas
- `doctor --json` includes `previous_score` and `delta` fields
---
## 7. Weighted scoring
### Problem
Going from 99% → 100% embed coverage weighs the same as 50% → 51%. But the last percent is the hardest (oversized pages, rate limits).
### Proposed Fix
Threshold-based scoring:
- 100% = full points
- ≥95% = 90% of points
- ≥80% = 70% of points
- <80% = proportional
---
## Priority Order
1. Frontmatter severity levels (highest noise reduction)
2. Temporal contradiction awareness (highest false positive reduction, already designed)
3. Auto-heal mode (biggest long-term value)
4. Score delta tracking (enables monitoring)
5. Multi-source drift baseline (quality of life)
6. Image assets acknowledgment (quality of life)
7. Weighted scoring (nice to have)
-103
View File
@@ -1,103 +0,0 @@
# Connect GBrain to ChatGPT
**Status (v0.26.0):** Unblocked. GBrain's `gbrain serve --http` ships OAuth 2.1
with PKCE, which is the ChatGPT MCP connector's hard requirement. Before v1.0,
this was a P0 TODO — the only major AI client that could not connect.
ChatGPT does not support bearer-token MCP servers. You must use the OAuth 2.1
HTTP server.
## Setup
### 1. Start the HTTP server
```bash
gbrain serve --http --port 3131
```
Save the admin bootstrap token printed on stderr. Open
`http://localhost:3131/admin` and paste it to access the dashboard.
### 2. Register a ChatGPT client
ChatGPT uses the authorization code flow with PKCE (browser-based OAuth).
Register from the `/admin` dashboard:
1. Click **Register client**.
2. Name: `chatgpt`.
3. Grant type: `authorization_code`.
4. Scopes: `read`, `write` (leave `admin` unchecked for ChatGPT).
5. Redirect URI: ChatGPT's OAuth redirect (copy it from the ChatGPT
connector setup screen — something like
`https://chat.openai.com/connector_platform_oauth_redirect`).
6. Hit **Register**. The credential-reveal modal shows the `client_id` once
with Copy and Download JSON buttons. There is no client secret for
PKCE-based public clients.
Host-repo wrappers can register programmatically:
```ts
await oauthProvider.registerClientManual(
'chatgpt',
['authorization_code'],
'read write',
['https://chat.openai.com/connector_platform_oauth_redirect'],
);
```
### 3. Expose the server publicly
```bash
brew install ngrok
ngrok http 3131 --url your-brain.ngrok.app
```
Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. ChatGPT's
connector auto-discovers the spec-compliant endpoint at
`/.well-known/oauth-authorization-server`.
### 4. Add the connector in ChatGPT
1. Open ChatGPT > Settings > Connectors.
2. Click **Add connector**.
3. MCP server URL: `https://your-brain.ngrok.app/mcp`.
4. Client ID: the `client_id` you saved in step 2.
5. Click **Connect**. ChatGPT opens the OAuth consent page, you approve, and
the connector is live.
Start a new conversation and ask ChatGPT to search your brain. The MCP tool
calls show up in the admin dashboard's live SSE feed in real time.
## Scopes
ChatGPT clients can request any combination of `read`, `write`, `admin`. The
scopes granted at consent time are enforced on every tool call. Four
operations are `localOnly` and rejected over HTTP regardless of scope:
`sync_brain`, `file_upload`, `file_list`, `file_url`. The HTTP server fails
closed for any attempt to reach local filesystem surface area.
Recommended ChatGPT scope: `read write`. Leave `admin` for your local CLI
and the admin dashboard.
## Troubleshooting
**"Invalid redirect_uri" during the ChatGPT connector OAuth handshake**
The registered `redirect-uri` must match ChatGPT's exactly. If ChatGPT
rejects your server, check the admin dashboard's **Agents** table for the
client, confirm the redirect URI matches what the error page shows, and
re-register with the correct URI.
**ChatGPT shows an MCP connection error after approval**
Open `/admin`, watch the SSE feed, and try again. If no request arrives, the
connector isn't reaching your ngrok URL. If a request arrives but fails,
the Request Log tab shows the exact error.
**"Unsupported grant_type" on the token endpoint**
ChatGPT uses `authorization_code`, which the MCP SDK supports natively.
If you see this error, verify the client was registered with
`--grant-types authorization_code` and not `client_credentials`.
## See also
- [DEPLOY.md](DEPLOY.md) — full OAuth 2.1 setup reference
- [ALTERNATIVES.md](ALTERNATIVES.md) — tunnel options (ngrok, Tailscale, Fly)
+5 -55
View File
@@ -1,10 +1,5 @@
# Connect GBrain to Claude Code
> New to this? The [Give your coding agent a memory](../tutorials/connect-coding-agent.md)
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
> end to end, plus the brain-first protocol that makes it worth it. This page is
> the connection reference.
## Option 1: Local (recommended, zero server needed)
```bash
@@ -14,44 +9,10 @@ claude mcp add gbrain -- gbrain serve
That's it. Claude Code spawns `gbrain serve` as a stdio subprocess. No server, no
tunnel, no token needed. Works with both PGLite and Supabase engines.
## Option 2: Remote, one command (fastest from a bearer token)
## Option 2: Remote (access from any machine)
If GBrain is running somewhere as an HTTP server (`gbrain serve --http`, see the
[ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md)) and you have a bearer token,
let `gbrain connect` generate the wire-up for you.
On the host (or anywhere `gbrain` is installed), mint a token and print the block:
```bash
gbrain auth create "claude-code"
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --token gbrain_xxx
```
`gbrain connect` prints a short, copy-paste block. Paste it into Claude Code — it
runs the `claude mcp add` for you and tells the agent to call `get_brain_identity`
and `list_skills` so it immediately knows what the brain can do.
Already on the machine you want to wire up? Skip the copy-paste and let `connect`
do it directly, with a built-in token smoke-test:
```bash
gbrain connect https://YOUR-DOMAIN.ngrok.app --token gbrain_xxx --install
```
(`--install` runs `claude mcp add`, then verifies the token by calling
`get_brain_identity` — so a wrong or expired token fails now, not silently on the
agent's first request. The URL is normalized: a bare host without `/mcp` gets it
appended; pass an explicit `https://` scheme.)
Pipe-friendly machine output (token redacted unless `--show-token`):
```bash
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --token gbrain_xxx --json
```
## Option 3: Remote, manual `claude mcp add`
Equivalent to what `gbrain connect` generates, if you'd rather run it yourself:
If you have GBrain running on a server with a public tunnel (see
[ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md)):
```bash
claude mcp add gbrain -t http \
@@ -59,12 +20,8 @@ claude mcp add gbrain -t http \
-H "Authorization: Bearer YOUR_TOKEN"
```
Replace `YOUR-DOMAIN` with your ngrok domain and `YOUR_TOKEN` with a token from
`gbrain auth create "claude-code"`.
> A `gbrain auth create` token is a long-lived, full-access secret. Keep it
> private (it lands in `~/.claude.json`), and prefer a scoped/short-lived token
> where your host supports one.
Replace `YOUR-DOMAIN` with your ngrok domain and `YOUR_TOKEN` with a token
from `gbrain auth create "claude-code"`.
## Verify
@@ -76,13 +33,6 @@ search for [any topic in your brain]
You should see results from your GBrain knowledge base.
> **`list_skills` returns nothing?** Skill discovery is gated by `mcp.publish_skills`
> on the host. New brains from `gbrain init` default it ON; brains upgraded from an
> older release stay OFF until you opt in. Enable it on the host with
> `gbrain config set mcp.publish_skills true`. The core tools (search, query,
> get_page, put_page, think, find_experts) work regardless. Note: `capture` is a
> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`.
## Remove
```bash
-71
View File
@@ -1,71 +0,0 @@
# Connect GBrain to Codex
> New to this? The [Give your coding agent a memory](../tutorials/connect-coding-agent.md)
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
> end to end, plus the brain-first protocol that makes it worth it. This page is
> the connection reference.
Codex CLI (`@openai/codex`, v0.130+) supports remote streamable-HTTP MCP servers
with a bearer token read from an environment variable. The token lives in your
shell env, not in Codex's config file.
## Fastest path: `gbrain connect`
Run anywhere `gbrain` is installed (mint a token on the brain host first):
```bash
gbrain auth create "codex"
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --token gbrain_xxx --agent codex
```
This prints a copy-paste block. Or wire it up directly and smoke-test the token:
```bash
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --token gbrain_xxx --agent codex --install
```
`--install` runs `codex mcp add` for you, then makes one real call to the brain so
a wrong/expired token fails right away. Because Codex reads the token from the env
var at runtime, keep `GBRAIN_REMOTE_TOKEN` exported in your shell profile.
## Manual setup
```bash
export GBRAIN_REMOTE_TOKEN=gbrain_xxx
codex mcp add gbrain --url https://YOUR-DOMAIN.ngrok.app/mcp \
--bearer-token-env-var GBRAIN_REMOTE_TOKEN
```
Codex stores the env-var *name* (`GBRAIN_REMOTE_TOKEN`), not the token itself, and
reads the value when it launches the MCP server. Add the `export` line to your
`~/.zshrc` / `~/.bashrc` so it's set in every session.
## Verify
In Codex, ask it to use the brain:
```
Call get_brain_identity, then search my brain for [topic].
```
`get_brain_identity` confirms whose brain you're connected to; `list_skills` shows
everything it can do.
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host (default
> ON for `gbrain init` brains, OFF for brains upgraded from older releases). Enable
> it on the host: `gbrain config set mcp.publish_skills true`. The core tools
> (search, query, get_page, put_page, think, find_experts) work regardless.
> `capture` is CLI-only, not an MCP tool — write over MCP with `put_page`.
## Remove
```bash
codex mcp remove gbrain
```
## Notes
- The token is a long-lived, full-access secret. Keep `GBRAIN_REMOTE_TOKEN` out of
version control and prefer a scoped token if your host supports one.
- Local stdio also works if you run the brain on the same machine:
`codex mcp add gbrain -- gbrain serve`.
+15 -170
View File
@@ -1,21 +1,17 @@
# Deploy GBrain Remote MCP Server
> **v0.26.0+:** `gbrain serve --http` ships full OAuth 2.1 (client credentials,
> auth code + PKCE, refresh rotation, optional DCR), an embedded React admin
> dashboard at `/admin`, scoped operations, and a live SSE activity feed.
> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back
> to the `access_tokens` table and grandfathers tokens to `read+write+admin`.
> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only);
> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md)
> for env vars and tunable defaults.
> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in
> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and
> per-request audit log. **Postgres-only** (PGLite is local-only by design).
> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults.
Access your brain from any device, any AI client. GBrain ships two transports:
`gbrain serve` (stdio) for local agents, and `gbrain serve --http` (v0.26.0+)
for remote clients over OAuth 2.1.
Access your brain from any device, any AI client. GBrain's MCP server runs locally
via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP
transport behind a public tunnel.
## Three Paths
## Two Paths
### Local stdio (zero setup)
### Local (zero setup)
```bash
gbrain serve
@@ -24,30 +20,7 @@ gbrain serve
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
### Remote over OAuth 2.1 (recommended, v0.26.0+)
```bash
gbrain serve --http --port 3131
ngrok http 3131 --url your-brain.ngrok.app
gbrain serve --http --port 3131 --public-url https://your-brain.ngrok.app
```
Built-in HTTP transport with OAuth 2.1, scoped operations, an admin dashboard
at `/admin`, and a live SSE activity feed. Zero external dependencies. This is
the only path that works with ChatGPT (OAuth 2.1 + PKCE is required by the
ChatGPT MCP connector). Pass `--public-url` whenever the server is reachable
at anything other than `http://localhost:<port>` so the OAuth issuer in
discovery metadata matches what clients hit (RFC 8414 §3.3).
Supported clients:
- **ChatGPT** — requires OAuth 2.1 + PKCE. Works natively with `--http`.
- **Claude Desktop / Cowork** — OAuth 2.1 or legacy bearer tokens.
- **Perplexity** — OAuth 2.1 client credentials grant.
- **Claude Code, Cursor, Windsurf** — can use OAuth or legacy bearer.
See the [OAuth 2.1 setup](#oauth-21-setup-v100) section below.
### Remote with legacy bearer tokens (pre-v0.26 deployments) — Postgres only
### Remote (any device, any AI client) — Postgres only
```
Your AI client (Claude Desktop, Perplexity, etc.)
@@ -63,133 +36,7 @@ This requires:
3. A public tunnel (ngrok, Tailscale, or cloud host)
4. A bearer token created via `gbrain auth create <name>`
Pre-v1.0 tokens are grandfathered as `read+write+admin` scopes when you upgrade
to the HTTP server, so no migration is required.
## OAuth 2.1 Setup (v0.26.0+)
### 1. Start the HTTP server
```bash
gbrain serve --http --port 3131
```
On first start, the server prints an **admin bootstrap token** to stderr:
```
Admin bootstrap token: 3a1f9c...
Open http://localhost:3131/admin and paste it to log in.
```
Save this token. Open `http://localhost:3131/admin` and paste it to access the
dashboard. The dashboard shows live activity, registered clients, request logs,
and per-client config export.
> **v0.26.9+:** `mcp_request_log.params` and the live SSE activity feed default
> to a redacted summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`.
> Declared param keys are kept (intersected against the operation's spec); unknown
> keys are counted but never named, and byte sizes round up to 1KB so size-probe
> attacks can't binary-search secret content. Operators on a personal laptop who
> want raw payloads back can pass `gbrain serve --http --log-full-params` (loud
> stderr warning fires at startup). Multi-tenant deployments should leave it on
> the redacted default.
### 2. Register OAuth clients
Register clients from the **`/admin` dashboard**:
1. Click **Register client**.
2. Enter a name (e.g. `perplexity`, `chatgpt`).
3. Pick scopes: `read`, `write`, `admin` (checkboxes).
4. Pick grant type: `client_credentials` for machine-to-machine (Perplexity,
Claude Desktop bearer mode) or `authorization_code` for browser-based
clients with PKCE (ChatGPT).
5. For `authorization_code` clients, paste the redirect URI.
6. Hit **Register**. The credential-reveal modal shows the `client_id` (and
`client_secret` for confidential clients) once. Copy or Download JSON
immediately — secrets are hashed on storage and never shown again.
Or from the CLI — faster for scripting:
```bash
gbrain auth register-client perplexity \
--grant-types client_credentials \
--scopes "read write"
```
**v0.34 — source-scoped clients.** Multi-source brains can scope a client's
write authority to one source and its read scope to a curated set with the
new `--source` and `--federated-read` flags:
```bash
gbrain auth register-client dept-x-agent \
--grant-types client_credentials \
--scopes "read write" \
--source dept-x \
--federated-read dept-x,shared,parent-canon
```
`--source` controls the write authority — `put_page` / `add_link` / etc only
land in `dept-x`. `--federated-read` controls the read axis independently;
queries return rows from any of the listed sources. Omit both flags for the
v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to
`source_id='default'` on `gbrain upgrade`.
Host-repo wrappers can register programmatically:
```ts
await oauthProvider.registerClientManual(
'perplexity',
['client_credentials'],
'read write',
[], // redirect_uris, empty for CC
);
```
For self-service client registration (Dynamic Client Registration, RFC 7591),
start the server with `--enable-dcr`. DCR is off by default.
### 3. Expose the server
**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
To accept connections from the ngrok tunnel (or any non-loopback source),
restart with `--bind`:
```bash
gbrain serve --http --port 3131 --bind 0.0.0.0 --public-url https://your-brain.ngrok.app
```
When `--public-url` is set without `--bind`, a stderr WARN fires at
startup so the misconfiguration ("the tunnel is up but my agent gets
ECONNREFUSED") is loud.
```bash
brew install ngrok
ngrok config add-authtoken YOUR_TOKEN
ngrok http 3131 --url your-brain.ngrok.app
```
Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. The MCP SDK's
router exposes the spec-compliant discovery endpoint at
`/.well-known/oauth-authorization-server`.
### 4. Scopes and localOnly
Every operation is tagged `read | write | admin`. Four operations are
`localOnly` and rejected over HTTP regardless of scope: `sync_brain`,
`file_upload`, `file_list`, `file_url`. Remote agents cannot reach local
filesystem surface area.
| Scope | What it allows |
|-------|---------------|
| `read` | `search`, `query`, `get_page`, `list_pages`, graph traversal |
| `write` | `put_page`, `delete_page`, `add_link`, `add_timeline_entry` |
| `admin` | Client management, token revocation, sweep, local-only ops |
## Legacy Bearer Token Setup
Keep using pre-v0.26 bearer tokens if you aren't ready to migrate. They
grandfather to `read+write+admin` scopes on the HTTP server.
## Remote Setup
### 1. Set up the tunnel
@@ -220,7 +67,6 @@ if compromised. Tokens are stored SHA-256 hashed in your database.
### 3. Connect your AI client
- **ChatGPT:** [setup guide](CHATGPT.md) (OAuth 2.1 + PKCE, requires `gbrain serve --http`)
- **Claude Code:** [setup guide](CLAUDE_CODE.md)
- **Claude Desktop:** [setup guide](CLAUDE_DESKTOP.md) (must use GUI, not JSON config)
- **Claude Cowork:** [setup guide](CLAUDE_COWORK.md)
@@ -277,8 +123,7 @@ Remote servers must be added via Settings > Integrations, NOT
| put_page | 100-500ms | Write + trigger search_vector update |
| get_stats | < 100ms | Aggregate query |
**Note:** `gbrain serve --http` shipped in v0.26.0 with OAuth 2.1 + admin
dashboard baked into the binary. The custom HTTP wrapper pattern (see
[voice recipe](../../recipes/twilio-voice-brain.md)) is still supported for
teams that need bespoke middleware, but for most remote deployments the
built-in server is the recommended path.
**Note:** `gbrain serve --http` (built-in HTTP transport) is planned but not yet
implemented. Currently, remote MCP requires a custom HTTP wrapper. See the
production deployment pattern in the [voice recipe](../../recipes/twilio-voice-brain.md)
for a reference implementation.
+14 -83
View File
@@ -1,83 +1,20 @@
# Connect GBrain to Perplexity Computer
Perplexity Computer connects as a **remote** MCP client, so GBrain must be served
over HTTP and reachable at a public HTTPS URL. Perplexity does not run
`gbrain serve` (stdio) the way Claude Code does — it needs a reachable endpoint:
Perplexity Computer supports remote MCP servers with bearer token authentication.
```
Perplexity Computer
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app/mcp)
→ gbrain serve --http (built-in OAuth 2.1 transport)
→ Postgres / PGLite
```
## Setup
## 1. Serve GBrain over HTTP (host side)
```bash
gbrain serve --http --port 3131 --bind 0.0.0.0 \
--public-url https://YOUR-DOMAIN.ngrok.app
```
- **`--bind 0.0.0.0` is required.** Since v0.34, `--http` defaults to
`127.0.0.1`, so without it the tunnel reaches the server but the connection is
refused (`ECONNREFUSED`).
- **`--public-url` must match the tunnel.** The OAuth issuer in the discovery
metadata has to line up with the URL Perplexity actually hits (RFC 8414 §3.3),
or OAuth client-credentials auth fails.
## 2. Expose it with a tunnel
```bash
ngrok http 3131 --url YOUR-DOMAIN.ngrok.app
```
See the [ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md) for a persistent
tunnel.
## 3. Create credentials
Two supported auth paths.
**OAuth 2.1 client credentials (recommended, v0.26.0+).** Perplexity is a cloud
service, so it holds whatever credential you give it. OAuth is the correct choice:
least-privilege scopes + short-lived rotating access tokens instead of a
long-lived full-access secret. Mint a client and print the connector fields in
one step (on the brain host):
```bash
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --agent perplexity --oauth --register
```
Or register separately and pass the creds (works anywhere, no DB needed):
```bash
gbrain auth register-client perplexity --grant-types client_credentials --scopes "read write"
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --agent perplexity --oauth \
--client-id gbrain_cl_xxx --client-secret gbrain_cs_xxx
```
`connect --oauth` prints the **Issuer URL + Client ID + Client Secret** to paste
in step 4.
**Legacy bearer token (simplest, best for local/personal):**
```bash
gbrain auth create "perplexity"
gbrain connect https://YOUR-DOMAIN.ngrok.app/mcp --token gbrain_xxx --agent perplexity
```
(Perplexity is a GUI connector, so there's no `--install``connect` prints the
exact values to paste in step 4.)
## 4. Add the connector in Perplexity
1. Open Perplexity (requires Pro subscription).
2. Go to **Settings → Connectors** (or **MCP Servers**).
1. Open Perplexity (requires Pro subscription)
2. Go to **Settings > Connectors** (or **MCP Servers**)
3. Add a new remote connector:
- **URL:** `https://YOUR-DOMAIN.ngrok.app/mcp`
- **Authentication:** API Key / Bearer Token, or OAuth client credentials
- Paste the token (bearer) or `client_id` + `client_secret` (OAuth).
4. Save.
- **Authentication:** API Key / Bearer Token
- **Token:** your GBrain access token
(create one with `gbrain auth create "perplexity"`)
4. Save
Replace `YOUR-DOMAIN` with your ngrok domain (see
[ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md) for setup).
## Verify
@@ -87,14 +24,8 @@ In a Perplexity conversation, ask it to use your brain:
Use my GBrain to search for [topic]
```
Have it call `get_brain_identity` (whose brain this is), then `list_skills`
(everything it can do).
## Notes
- Perplexity Computer is available to Pro subscribers; both the Mac app and web
version support remote MCP connectors.
- The Mac app can also use a local MCP server (`gbrain serve` stdio) if you'd
rather not expose an HTTP endpoint.
- A `gbrain auth create` token is a long-lived, full-access secret. Keep it
private and prefer a scoped token where possible.
- Perplexity Computer is available to Pro subscribers
- Both the Perplexity Mac app and web version support MCP connectors
- The Mac app also supports local MCP servers if you prefer `gbrain serve` (stdio)
@@ -1,153 +0,0 @@
# Migrating your OpenClaw brain to gbrain v0.41.2.0 (greenfield)
The v0.41.2.0 lens packs ship a one-shot importer that re-ingests your
existing OpenClaw brain (`~/git/brain/atoms/`, `concepts/`, `ideas/`)
through the new ingestion cathedral. Pages land in gbrain with an
`imported_from: markdown-greenfield` frontmatter marker so the new
extract_atoms + synthesize_concepts cycle phases skip them (lossless
import with provenance).
## Before you migrate
1. **Upgrade gbrain to v0.41.2.0+:**
```bash
gbrain upgrade
gbrain --version # should be 0.41.2.0 or later
```
2. **Activate the creator pack** (or gbrain-everything if you also
want investor + engineer lenses on the same brain):
```bash
gbrain config set schema_pack gbrain-creator
# OR
gbrain config set schema_pack gbrain-everything
```
3. **Apply schema migration v94** (take_domain_assignments table):
```bash
gbrain apply-migrations --yes
```
## The dry-run pass
Always start with `--dry-run` to see what the importer would do
without writing anything:
```bash
gbrain capture --source markdown-greenfield \
--repo ~/git/brain \
--dry-run \
--limit 100
```
The output reports:
- `emitted` — atoms/concepts/ideas that would import cleanly
- `skipped_no_type` — files without a `type:` frontmatter (counted
as benign skips; no audit appended)
- `skipped_invalid` — files that failed validation (these append to
`~/.gbrain/audit/markdown-greenfield-failures-YYYY-Www.jsonl`)
Inspect the audit JSONL:
```bash
ls ~/.gbrain/audit/markdown-greenfield-failures-*.jsonl
cat ~/.gbrain/audit/markdown-greenfield-failures-*.jsonl | jq .
```
Common failures:
- **Empty frontmatter** — file has `---` but no fields. Not a real
brain page; safe to leave skipped.
- **Malformed YAML** — fix the file in your OpenClaw then re-run.
- **Missing required field** — usually means the original OpenClaw
skill output a partial page; check whether the file is worth
preserving.
## The actual import
When the dry-run looks clean, drop the `--dry-run` and `--limit`
flags:
```bash
gbrain capture --source markdown-greenfield --repo ~/git/brain
```
Expect ~30-60 minutes for the full 24K-page set (atoms + concepts +
ideas). The importer:
1. Walks `atoms/{YYYY-MM-DD}/*.md`, `concepts/*.md`, `ideas/*.md` in
deterministic alphabetical order so partial-run resumes pick up
where they left off.
2. Stamps `imported_from: markdown-greenfield` + `imported_at:
<ISO timestamp>` on every page's frontmatter, preserving ALL
original fields verbatim under `metadata.original_frontmatter`.
3. Emits each as an IngestionEvent with `mode: 'migration'` (T2),
which bypasses the daemon's 24h DedupWindow. The importer owns
its own permanent slug-keyed idempotency.
4. Routes through `put_page` so pages land with proper FK chains and
embedding eligibility.
## After the import
Verify counts match:
```bash
gbrain stats
# Should show ~24K new pages with type=atom/concept/idea
```
The next `gbrain dream` cycle will:
- Run `extract_atoms` on NEW transcripts (skips pages with the
`imported_from` marker — your historical atoms are frozen, not
re-extracted).
- Run `synthesize_concepts` on NEW atoms (skips imported concepts
for the same reason).
- Run `extract_facts` over the imported pages — facts fences in
imported atoms/concepts populate the facts table normally.
## Retiring your OpenClaw's parallel crons
After verifying the import, retire your OpenClaw's parallel atom
pipeline cron entries. In `~/git/your-openclaw/workspace/cron.json`:
- Remove `atom-pipeline-coordinator` (every-30-min cron)
- Remove `atom-backfill-coordinator` (every-10-min cron)
Replace with nothing — gbrain's autopilot already runs extract_atoms +
synthesize_concepts inside every dream cycle when gbrain-creator (or
gbrain-everything) is the active pack.
The OpenClaw skills themselves shrink to thin wrappers:
- `content-atom-extractor` → calls `gbrain dream --phase extract_atoms`
- `concept-synthesis` → calls `gbrain dream --phase synthesize_concepts`
- `atom-backfill-coordinator` → DELETED (backfill is now part of
extract_atoms via the Source Quote + lesson enrichment in one
Haiku call per transcript)
## Rolling back
The import is fully reversible:
```bash
# Soft-delete every page with the marker (recoverable for 72h)
gbrain query "imported_from:markdown-greenfield" --type atom --json | \
jq -r '.[].slug' | xargs -I{} gbrain pages delete {}
# OR hard-delete past the soft-delete window
gbrain pages purge-deleted --older-than 0h
```
Your OpenClaw's `~/git/brain/atoms/` + `concepts/` + `ideas/` directories
are untouched by the importer — they remain the source of truth for
rollback. The greenfield importer only READS from them.
## Re-running after partial failures
The importer is idempotent at the page-slug level: re-running on the
same `--repo` produces zero net-new pages (every page either lands
fresh or matches an existing slug). If you fix some validation
failures in your OpenClaw and want to retry just those:
```bash
gbrain capture --source markdown-greenfield --repo ~/git/brain
```
Already-imported pages stay; previously-failed pages get a fresh
attempt; the audit JSONL accumulates per-week (ISO week file rotation).
-87
View File
@@ -1,87 +0,0 @@
# Headless install: Docker, CI, postinstall
As of v0.37, `gbrain init --pglite` in a non-TTY context (Docker `RUN`, CI step, postinstall hook) exits 1 when no embedding-provider API key is present in the environment. This is a deliberate fail-loud — the alternative was the v0.36 silent-broken-state class where init succeeded with a default that didn't match any real key.
Two patterns work for headless installs. Pick whichever fits your image lifecycle.
## Pattern 1: Provider key available at image build time
If your CI / Docker pipeline can inject the API key as a build-time env var, set it before `gbrain init`:
```dockerfile
# Multi-stage Dockerfile sketch
FROM oven/bun:1 AS builder
# Inject key at build via --build-arg or `--env` from CI.
ARG OPENAI_API_KEY
ENV OPENAI_API_KEY=$OPENAI_API_KEY
RUN bun install -g github:garrytan/gbrain
RUN gbrain init --pglite # auto-picks OpenAI, persists config
```
```yaml
# GitHub Actions equivalent
- name: Initialize gbrain
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
bun install -g github:garrytan/gbrain
gbrain init --pglite
```
Init writes `~/.gbrain/config.json` with the resolved `embedding_model` + `embedding_dimensions`. Subsequent runs (in the same image / runner) read from that config and don't re-resolve.
## Pattern 2: Provider key only at runtime (deferred-setup)
If the API key is a runtime secret (Kubernetes secret, runtime env injection, end-user-supplied), use `--no-embedding` at build time and configure the provider when the container actually runs:
```dockerfile
FROM oven/bun:1
RUN bun install -g github:garrytan/gbrain
# Build the brain shape without a provider — schema lands at the default
# width, but no embed callsite will actually run until runtime config.
RUN gbrain init --pglite --no-embedding
# At container start (entrypoint), provide the real provider:
ENTRYPOINT ["/bin/sh", "-c", "\
gbrain config set embedding_model openai:text-embedding-3-large \
&& gbrain init --force --pglite \
&& exec gbrain serve"]
```
The `gbrain init --no-embedding` opt-in writes `embedding_disabled: true` to config. Every embed callsite (`gbrain import`, `gbrain embed`, the `runEmbedCore` library entry point) checks this and refuses cleanly with a `gbrain config set embedding_model <id>` hint rather than proceeding with a silent default.
The runtime `gbrain init --force` re-runs the init flow against the now-populated env, which:
- Removes `embedding_disabled` from config.
- Resolves the provider via env detection.
- Re-templates the PGLite schema if dim differs from the build-time default.
## What WON'T work
```dockerfile
# Don't do this — silent default leaves you with vector(1280) ZE column
# and 1536d OpenAI provider at runtime, mismatched.
RUN gbrain init --pglite
```
If you upgrade from a pre-v0.37 image that used this pattern, `gbrain doctor` will surface the mismatch on first run after upgrade and print a paste-ready repair command (`gbrain init --force --embedding-model …` for empty brains, `gbrain retrieval-upgrade --reindex` for non-empty).
## Verifying a headless install
After init, run `gbrain doctor --json` to verify state:
```bash
gbrain doctor --json | jq '.checks[] | select(.name=="embedding_provider")'
```
The `embedding_provider` check returns `status: 'ok'` when:
- Config has a persisted `embedding_model`.
- Config has a persisted `embedding_dimensions`.
- Live provider probe returns the configured dim.
- DB column width matches.
If you used Pattern 2's deferred-setup path, the check shows `Skipped (no provider credentials)` until the runtime config is populated. That's expected.
@@ -1,213 +0,0 @@
# Proposal: Temporal Axis for Contradiction Probe
**Status:** Report / RFC
**Date:** 2026-05-14
**Context:** A large production run of `gbrain eval suspected-contradictions` surfaced ~115 HIGH findings. Walking through them by hand exposed a structural limitation in the probe.
## The Problem
The contradiction probe (`gbrain eval suspected-contradictions`) treats all claims as timeless. When two chunks make conflicting statements, the judge flags a contradiction regardless of whether both statements were true at their respective points in time.
This worked fine when the brain was mostly static wiki pages. It breaks now that the brain contains:
- Conversation transcripts with claims that were true when spoken
- Meeting pages capturing what people said on specific dates
- Takes that evolve (a founder's ARR claim in January vs. July)
- Status records that supersede each other (a state moves from "trial" to "confirmed")
The probe can't distinguish "this changed" from "this is wrong."
## Bug-class examples (synthetic placeholders)
### 1. Temporal Evolution (False Positive)
```
Finding: HIGH
A: [daily/transcripts/2026/2026-04-28] "status: trial"
B: [meetings/2026-05-07-session] "status: confirmed"
Axis: Whether status is trial or confirmed
```
Both are correct as of their respective dates. April 28: trial. May 7: confirmed. The probe flags this because it has no concept of "this claim was valid from X until Y." The May 7 record didn't make the April 28 transcript wrong; it recorded a change.
### 2. Negation Parsing (False Positive)
```
Finding: HIGH
A: [people/alice-example] "person traveled to city-a for alice-example's event — NOT bob-example's event"
B: [meetings/2026-05-11-context] mentions of bob-example's event in city-b
Axis: Whose event the city-a trip was for
```
The disambiguation fact contains "NOT bob-example's event" as an explicit negation. The judge reads "bob-example's event" as a positive claim and flags it against the alice-example context. The data is correct; the probe can't parse negation.
### 3. Role Changes (True Positive That Needs Time Awareness)
```
Finding: HIGH
A: [sources/notes/2017-03-28] advisor-example: "Partner, venture-firm-a"
B: [people/advisor-example] advisor-example: "Senior Policy Advisor, gov-org-b"
```
Both true at their respective times. 2017: partner at venture-firm-a. 2025: gov-org-b advisor. The current probe correctly flags this as a contradiction, but the resolution should be "superseded by time" not "one side is wrong." The 2017 note isn't wrong; it's a historical record.
## Scenario #1: Founder Tracking (the big one)
This is the use case that makes a time axis transformative rather than incremental.
The brain holds hundreds of company pages and thousands of meeting pages. Founders make claims:
- "We're at $50K MRR" (January OH)
- "We hit $200K MRR" (April OH)
- "We're at $150K MRR" (July OH — what happened?)
Today the probe would flag January vs. April as a contradiction. The real signal is April vs. July: **a claimed metric went backwards.** That's not a data quality issue; that's intelligence.
What a time-aware probe could surface:
**Claim trajectory tracking:**
```
Company: Acme Corp
2026-01: "$50K MRR" (source: OH transcript)
2026-04: "$200K MRR" (source: OH transcript)
2026-07: "$150K MRR" (source: OH transcript) ← REGRESSION DETECTED
2026-07: "$2M ARR" (source: investor update) ← INCONSISTENT WITH MRR
```
**Prediction vs. outcome:**
```
Founder: Jane Doe (Acme Corp)
2026-01: "We'll hit $1M ARR by June" (source: batch kickoff)
2026-06: Actual ARR: $400K (source: investor update)
→ Prediction accuracy: 40%
→ Pattern: consistently 2-3x optimistic on timeline
```
**Narrative consistency:**
```
Founder: John Smith (WidgetCo)
2026-01: "Our moat is proprietary data" (source: interview)
2026-03: "We're pivoting to an API-first model" (source: OH)
2026-06: "Our moat is network effects" (source: Demo Day)
→ Moat narrative changed 3x in 6 months — flag for review
```
This isn't adversarial. It's the kind of pattern an experienced operator notices intuitively across hundreds of conversations. GBrain can make it systematic.
## Scenario #2: Event Disambiguation
Two distinct events within a short window can conflate during ingestion because the probe has no temporal frame to say "event A is a different event from event B."
Time-aware facts would store (synthetic placeholders):
```
fact: "alice-example milestone" valid_from: 2026-04-15 valid_until: 2026-04-15
fact: "alice-example event in city-a" valid_from: 2026-04-17 valid_until: 2026-04-19
fact: "bob-example milestone" valid_from: 2026-05-04 valid_until: 2026-05-04
fact: "bob-example event in city-b" valid_from: 2026-05-12 valid_until: 2026-05-12
```
The probe should recognize these as two distinct events with non-overlapping time windows, not as contradictions about "whose event."
## Scenario #3: Role and Status Changes
People change roles. Companies change status. The brain records history. Synthetic examples representative of the cases observed in production:
- advisor-example: venture-firm-a partner (2019) → gov-org-b advisor (2025)
- investor-example: fund-a partner → fund-b CEO (2023)
- agent-fork: provider restriction event (2026-04-04) ≠ shutdown
- fund-c: "interesting fund" (early) → "declined" (later) → "losing confidence" (latest)
All of these are correct historical records. The probe should classify them as **temporal supersession** rather than **contradiction.**
## Scenario #4: Decision Tracking
Multi-step decisions that supersede earlier framings example (synthetic):
```
2026-04-24: "status: trial" (initial framing)
2026-04-25: "status: in progress" (confirmed, no longer "trial")
2026-05-07: "status: finalized" (session record)
2026-05-11: follow-up actions taken
```
Each step supersedes the previous. A time-aware probe would show the **evolution chain** rather than flagging each pair as a contradiction.
## What Exists Today
The probe already has some temporal infrastructure:
1. **`date-filter.ts`** — `shouldSkipForDateMismatch()` pre-filters pairs, but only checks whether dates are "too far apart" (a coarse heuristic). It doesn't reason about which claim is newer or whether one supersedes the other.
2. **`auto-supersession.ts`** — proposes resolution commands, checks `since_date` on takes. But this is post-hoc (after the judge flags a contradiction). The judge itself doesn't see dates.
3. **Facts table** has `valid_from` and `valid_until` columns. These exist but are sparsely populated and not used by the probe.
4. **Takes table** has `since_date`. Also sparsely populated.
## What Would Need to Change
### Phase 1: Judge prompt enhancement (smallest change, biggest impact)
Pass the source dates to the judge. The current judge prompt shows two text chunks and asks "are these contradictory?" If it also showed:
```
Statement A (from: 2026-04-28):
"status: trial"
Statement B (from: 2026-05-07):
"status: confirmed"
```
The judge could output a `temporal_supersession` verdict instead of `contradiction`. New verdict taxonomy:
- `no_contradiction` — statements are compatible
- `contradiction` — genuinely conflicting claims at the same point in time
- `temporal_supersession` — newer claim updates/replaces older claim (not an error)
- `temporal_regression` — a metric or status went backwards (potential signal)
- `temporal_evolution` — legitimate change over time, neither supersession nor regression
- `negation_artifact` — one side contains an explicit negation the judge misread
### Phase 2: Claim trajectory view (new command)
```bash
gbrain eval trajectory "Acme Corp MRR"
gbrain eval trajectory "advisor-example role"
gbrain eval trajectory "deal-x status"
```
Pull all time-stamped claims about an entity+attribute, sort chronologically, detect:
- Regressions (metric went down)
- Contradictions within the same time window
- Prediction vs. outcome gaps
- Narrative drift (moat story changed 3x)
### Phase 3: Automatic `valid_from`/`valid_until` population
During `extract_facts`, infer temporal bounds from source context:
- Meeting page dated 2026-04-28 → claims valid_from 2026-04-28
- Takes from transcripts → valid_from = transcript date
- Imported notes → valid_from = note date
- Entity pages with no date → valid_from = page created date (weakest signal)
### Phase 4: Founder scorecard
For founders specifically, a temporal probe could generate:
- **Claim accuracy score** — what they predicted vs. what happened
- **Consistency score** — how stable their narrative is over time
- **Growth trajectory** — whether the numbers are actually moving
- **Red flag detector** — metrics going backwards, story changing, timeline slipping
## Recommendation
Start with Phase 1. The judge prompt change is small. It immediately eliminates the temporal false positives (which were a majority of the residual HIGH findings in the production audit) and gives the probe a new vocabulary for time-aware reasoning.
Phase 2 (trajectory view) is the one that would change how operators use the brain for founder evaluation. Worth scoping as a standalone feature.
Phases 34 are downstream and can wait.
## Appendix: Production probe stats (2026-05-14)
- ~107K pages, ~257K chunks
- Previous run: ~115 HIGH findings across 50 queries
- After manual resolution: ~25 residual findings
- Of those ~25: roughly two-thirds temporal false positives, the remainder probe artifacts (self-contradiction, negation parsing)
- 0 genuine data contradictions remained on the queries tested
- Fresh targeted probe on a representative entity-role query: 0 contradictions (was 14+ before fixes)
-224
View File
@@ -1,224 +0,0 @@
# Tutorial: Build your first schema pack
You'll fork the bundled `gbrain-base` pack, add a custom `researcher` page type, import a handful of placeholder researcher pages, backfill their `page.type` column with one command, then prove the wiring works by running `gbrain whoknows` and seeing your new type surface in results. End state: a forked-and-active pack on disk, ~5 pages typed as `researcher`, and a query that proves the pack-aware routing fires end-to-end.
**Want the WHY before the HOW?** Read [`what-schemas-unlock.md`](what-schemas-unlock.md) first — 7 concrete use cases (4000 invisible meetings, the founder ops brain, the research brain, the legal brain, the team brain, agent-as-co-curator) plus the structural argument for why types matter at query time. Then come back here for the 5-minute walkthrough.
The whole walkthrough takes about 5 minutes. You'll see something working by step 3.
## What you'll need
- gbrain v0.40.7.0 or later (`gbrain --version` to check)
- A brain that's been initialized (`gbrain init` already run; either PGLite or Postgres is fine)
- A terminal you can paste commands into
That's it. No API keys required for this tutorial — every step works against the bundled pack and local-only commands.
## Step 1: See what pack is active today
```bash
gbrain schema active --json
```
You'll see something like:
```json
{
"pack_name": "gbrain-base",
"version": "1.0.0",
"sha8": "...",
"page_types_count": 22,
"source_tier": "default"
}
```
`source_tier: "default"` means you haven't customized anything — you're on the bundled pack. `page_types_count: 22` is the universal starter (person, company, meeting, note, etc.).
**You can't mutate bundled packs directly.** Step 2 forks it so you have something writable.
## Step 2: Fork the bundled pack
```bash
gbrain schema fork gbrain-base mine
```
Output: `Forked 'gbrain-base' → 'mine' at ~/.gbrain/schema-packs/mine/pack.json`.
The fork is a byte-for-byte copy of `gbrain-base` living at `~/.gbrain/schema-packs/mine/pack.json`. Now you have a writable pack you can mutate.
## Step 3: Activate the fork
```bash
gbrain schema use mine
```
Output: `Pack: mine (json) ... Active.`
Run `gbrain schema active --json` again to confirm `pack_name` is now `mine` and `source_tier` is `home-config` (read from `~/.gbrain/config.json`).
**You've already accomplished something visible** — the active pack changed, and any future query will route through your fork. The next four steps add a custom type and prove it works.
## Step 4: Add a researcher type
```bash
gbrain schema add-type researcher \
--primitive entity \
--prefix people/researchers/ \
--extractable \
--expert
```
Output: `Pack: mine (json)` + `Sha8: <prev> → <new>`.
What just happened:
- The mutation went through `withMutation`'s 8-step skeleton: bundled-guard → per-pack lock → read → mutate → file-plane lint validation → atomic write → audit log → cache invalidation.
- The pack now declares `researcher` as an entity primitive bound to `people/researchers/`, marked `extractable: true` (eligible for facts extraction) and `expert_routing: true` (surfaces in `whoknows` queries).
- An audit row landed in `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl` with your type name SHA-8-redacted and the prefix's first segment only (`people`) for privacy.
Verify the type is in the pack:
```bash
gbrain schema explain researcher
```
You'll see the resolved settings printed back.
## Step 5: Import some placeholder researcher pages
You need pages under `people/researchers/` for the next step to do anything. If your brain repo already has them, skip ahead. If not, drop 3-5 placeholder markdown files into `<your-brain-repo>/people/researchers/` and import:
```bash
mkdir -p people/researchers
cat > people/researchers/alice-example.md <<'EOF'
---
title: Alice Example
---
ML researcher at Example Lab. Works on contrastive embeddings.
EOF
cat > people/researchers/bob-example.md <<'EOF'
---
title: Bob Example
---
Vision researcher at Widget University. Recent paper on diffusion models.
EOF
cat > people/researchers/charlie-example.md <<'EOF'
---
title: Charlie Example
---
RL researcher at Acme Research. Focus on inverse reinforcement learning.
EOF
gbrain sync
```
The sync imports the new files. They'll be stored in the database but their `type` column will still be empty — the new type was added to the pack AFTER these pages already existed (the typical real-world scenario for an agent walking into an existing brain).
## Step 6: See the gap with `stats`
```bash
gbrain schema stats --json | jq '.aggregate, .dead_prefixes'
```
You'll see `untyped_pages: 3` (or however many you just imported) and `dead_prefixes: []` — your new prefix has 3 matching pages, so it's not dead.
The 3 researcher pages are "orphaned" by type even though they live in the right directory. The next step backfills them.
## Step 7: Backfill with `sync --apply`
First dry-run to see what would happen:
```bash
gbrain schema sync --json
```
You'll see something like:
```json
{
"schema_version": 1,
"apply": false,
"per_prefix": [
{
"type": "researcher",
"prefix": "people/researchers/",
"would_apply": 3,
"sample_slugs": ["people/researchers/alice-example", "people/researchers/bob-example", "people/researchers/charlie-example"],
"applied": 0
}
],
"total_would_apply": 3,
"total_applied": 0
}
```
`would_apply: 3` is what you'd touch. `sample_slugs` is the agent's drilldown signal — if those slugs look wrong, abort. They look right, so apply:
```bash
gbrain schema sync --apply
```
You'll see per-batch progress lines on stderr and a final `total_applied: 3`. The UPDATE ran in chunks of 1000 (yours fit in one chunk) and never wedged any concurrent writer.
## Step 8: Prove the wiring works
```bash
gbrain whoknows "machine learning"
```
If your researcher pages contain ML-related content, they'll surface in the ranked results — even though they're typed `researcher`, not `person` or `company`.
**This is the load-bearing demonstration of T1.5 wiring.** Pre-v0.40.7.0, `whoknows` hardcoded `['person', 'company']` as the eligible types and would have ignored your `researcher` pages entirely. The v0.40.7.0 wiring consults the active pack's `expert_routing: true` types via `expertTypesFromPack(pack.manifest)`, so your custom type now routes through expert search.
## What you built
You now have:
- A fork of `gbrain-base` named `mine` at `~/.gbrain/schema-packs/mine/pack.json`, active in your brain via `~/.gbrain/config.json`.
- A `researcher` page type registered in the pack with `entity` primitive, `people/researchers/` prefix, `extractable: true`, `expert_routing: true`.
- 3 pages typed as `researcher` (backfilled from disk via `gbrain schema sync --apply`).
- A query path that routes through the new type: `gbrain whoknows` reads the pack and includes `researcher` in its type filter.
You also exercised the full mutation skeleton: bundled-pack guard, per-pack lock, validation gate, atomic write, audit log, cache invalidation. Every step was idempotent — re-running any of them is a no-op.
## Next steps
**Add a link verb.** A `researcher` can `author` a `paper`. To model that:
```bash
gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable
gbrain schema add-link-type authored --page-type researcher --target-type paper
gbrain schema graph
```
The graph now shows `researcher --(authored)--> paper`.
**Add aliases for query closure.** If you want `gbrain query researcher` to also surface `person` rows (because researchers ARE people):
```bash
gbrain schema add-alias researcher person
```
Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) for the decision tree on when to add types vs aliases vs prefixes. The short version: <20 pages → don't pack-codify; 20-100 → alias on existing type; 100+ → first-class type.
**Lint your pack before shipping.** The 11-rule lint surface (with the optional `--with-db` flag for DB-aware checks) catches dangling references, prefix collisions, and dead-corpus warnings:
```bash
gbrain schema lint --with-db
```
**Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal).
**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher.
**Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`.
## Related docs
- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture.
- **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit).
- **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix.
- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private).
-112
View File
@@ -1,112 +0,0 @@
# Skillpack anatomy
The canonical one-page reference for what a third-party gbrain skillpack
looks like. The reference pack at `examples/skillpack-reference/` is the
live artifact this page describes; clone its tree and you have a 10/10
starting point.
## Tree
```
my-skillpack/
├── skillpack.json # manifest (cathedral fields declared)
├── skills/
│ └── <skill-slug>/
│ ├── SKILL.md # frontmatter + body, agent-readable
│ └── routing-eval.jsonl # >= 5 intents pinning trigger -> skill
├── runbooks/
│ └── bootstrap.md # post-scaffold display (NOT an executor)
├── test/
│ └── *.test.ts # bun:test unit tests
├── e2e/
│ └── *.test.ts # integration tests, gated on DATABASE_URL
├── evals/
│ └── *.judge.json # LLM-judge eval configs (>= 3 cases each)
├── CHANGELOG.md # Keep-a-Changelog shape
├── LICENSE # SPDX-matching text
├── README.md
└── .gitignore
```
`gbrain skillpack init <name>` scaffolds this exact tree, pre-filled
with stubs that score 10/10 on `gbrain skillpack doctor . --quick`
immediately. Replace the stubs with real content, run the doctor
between edits, and `gbrain skillpack pack` produces a deterministic
`<name>-<version>.tgz` ready to publish to the registry.
## How the agent uses a scaffolded pack
After `gbrain skillpack scaffold <source>` lands the files:
1. The user's agent walks `skills/*/SKILL.md` frontmatter and reads
each pack's `triggers:` array on startup or per-message.
2. When a user phrasing matches a trigger, the agent reads that
SKILL.md body top-to-bottom as in-context instructions.
3. gbrain DISPLAYS `runbooks/bootstrap.md` once after the scaffold
but does NOT auto-execute it. The agent decides whether to walk
the steps. This is the codex T1 supply-chain hardening: an
auto-walker would let a malicious pack mutate the user's brain
on install, which is how npm postinstall attacks happen.
## How the doctor scores a pack
Ten binary dimensions. Each is checked by a pure function in
`src/core/skillpack/rubric.ts` and returns `{passed, detail, fix_hint}`.
The doctor walks them in order and prints the score + per-dimension
status + paste-ready fix for every failure.
<!-- BEGIN auto-generated:rubric -->
### Core dimensions (5; must all pass to publish at any tier)
| # | Name | Description | Auto-fixable |
|---|------|-------------|--------------|
| 1 | `manifest_valid` | skillpack.json passes the v1 schema validator | no |
| 2 | `skills_have_skill_md` | every listed skill has SKILL.md with valid frontmatter (name, description, triggers) | no |
| 3 | `routing_evals_present` | every skill has routing-eval.jsonl with >= 5 intents | yes |
| 4 | `skills_have_unique_triggers` | no two skills in this pack share an exact trigger phrase (MECE) | no |
| 5 | `changelog_present_and_current` | CHANGELOG.md present and contains an entry for the current version | yes |
### Quality badges (5; earn for tier eligibility)
| # | Name | Description | Auto-fixable |
|---|------|-------------|--------------|
| 6 | `unit_tests_present` | pack declares unit_tests[] with at least one matching test file | yes |
| 7 | `e2e_tests_present` | pack declares e2e_tests[] with at least one matching test file | yes |
| 8 | `llm_eval_present` | pack declares llm_evals[] with >= 1 file containing >= 3 cases | yes |
| 9 | `bootstrap_runbook_present` | pack declares runbooks.bootstrap and the file is non-empty | yes |
| 10 | `license_present` | LICENSE file exists at the pack root (informational badge) | yes |
_Generated from `src/core/skillpack/rubric.ts` by `bun run scripts/build-skillpack-anatomy.ts`._
<!-- END auto-generated:rubric -->
## Tier eligibility
| Tier | Requirement |
|------|-------------|
| `endorsed` | All 5 core + all 5 badges, plus Garry's `endorsements.json` overlay in the registry repo |
| `community` | All 5 core + >= 3 of 5 badges. Default tier on PR merge. |
| `experimental` | All 5 core + < 3 badges |
| `blocked` | Any core dimension fails |
## CLI reference (third-party path)
```bash
# Publisher side
gbrain skillpack init my-pack # scaffold the tree
gbrain skillpack doctor my-pack # see the score + fix hints
gbrain skillpack doctor my-pack --fix --yes # auto-scaffold missing pieces
gbrain skillpack pack my-pack # deterministic tarball + SHA-256
# Consumer side
gbrain skillpack search <query> # browse the registry
gbrain skillpack info <name> # show full pack metadata
gbrain skillpack scaffold <source> # owner/repo, https, ./dir, ./*.tgz
gbrain skillpack registry --url X # point at a custom registry
```
## See also
- `examples/skillpack-reference/` — the live 10/10 reference pack
- `docs/designs/SKILLPACK_REGISTRY_V1_SPEC.md` — strategic spec + decisions
- `docs/guides/skillpacks-as-scaffolding.md` — v0.36 scaffold/reference model
-93
View File
@@ -1,93 +0,0 @@
# Takes vs Facts — Architectural Distinction
gbrain has two epistemological storage layers that serve different purposes.
**Never conflate them.**
## Takes (cold storage — `takes` table)
The epistemological layer. WHO believes WHAT, with confidence weight and time.
- **Source:** Extracted from brain pages (markdown) by LLM analysis
- **Scope:** Multi-holder — captures beliefs from *any* speaker, not just the brain owner
- **Kinds:** `take` (opinion), `fact` (verifiable), `bet` (prediction), `hunch` (intuition)
- **Lifecycle:** Cold storage, retrospective. Updated when pages change or re-extraction runs.
- **Scale:** 100K+ rows across thousands of holders in a mature brain
**Example takes:**
- `holder=people/garry-tan kind=bet` "AI will replace 50% of coding by 2030" (w=0.75)
- `holder=people/jared-friedman kind=take` "Momo has strong retention" (w=0.80)
- `holder=world kind=fact` "Clipboard raised $100M Series C" (w=1.0)
- `holder=brain kind=hunch` "Garry has a hero/rescuer pattern" (w=0.70)
**Query surface:** `gbrain takes list`, `gbrain takes search`, `gbrain think`
## Facts (hot memory — `facts` table, v0.31)
Personal knowledge from the brain owner's conversations. Real-time capture.
- **Source:** Extracted per-turn from conversation by the facts hook (Haiku)
- **Scope:** Single-user — only the brain owner's stated knowledge
- **Kinds:** `event`, `preference`, `commitment`, `belief`, `fact`
- **Lifecycle:** Hot storage, real-time. Captured as conversations happen.
- **Bridge:** Dream cycle `consolidate` phase promotes hot facts → cold takes nightly
**Example facts:**
- `kind=event` "I have a meeting with Brian tomorrow"
- `kind=preference` "I don't drink coffee"
- `kind=commitment` "We decided on nesting custody"
- `kind=belief` "I think the market is overheated"
**Query surface:** `gbrain recall`, MCP `_meta.brain_hot_memory`
## The Category Error
**Never dump takes into the facts table.** Takes include other people's attributed
beliefs (Jared's assessment of a company, PG's view on schools, a founder's
revenue claims). These are NOT the brain owner's personal facts.
**Never dump facts into the takes table without transformation.** Facts are
scoped to what the owner said in conversation. They become takes only through
the dream cycle's consolidate phase, which adds proper attribution, deduplication,
and temporal reasoning.
## The Bridge
The dream cycle's `consolidate` phase (v0.31) is the one-way bridge:
```
hot facts → [dream consolidate] → cold takes
```
Facts flow in ONE direction. The consolidate phase:
1. Groups related facts by entity
2. Deduplicates against existing takes
3. Promotes durable facts to takes with proper holder/weight
4. Marks consolidated facts with `consolidated_at` + `consolidated_into`
## Production Extraction Data (2026-05-10)
First full takes extraction run on a ~100K-page brain:
- **Model:** Azure GPT-5.5 (ties Opus quality at 1/8th cost — $0.033 vs $0.260/page)
- **Result:** 100,720 takes from 28,256 on-disk pages, $361.49, 83 errors (0.3%)
- **Breakdown:** 70,960 takes / 24,342 facts / 2,875 bets / 2,649 hunches
- **Holders:** 6,239 unique holders
- **Cross-modal eval:** 6.8/10 overall (GPT-5.5 + Opus 4.6 scored independently)
### Eval Dimensions
| Dimension | Score | Notes |
|-----------|-------|-------|
| Accuracy | 7.5 | Claims faithfully represent sources |
| Attribution | 6.5 | Holder/subject confusion was #1 issue |
| Weight calibration | 7.0 | Good range usage, some false precision |
| Kind classification | 6.5 | Occasional fact/take misclassification |
| Signal density | 6.5 | Some trivial extractions pass through |
### Key Learnings for Extraction Prompts
1. **Holder ≠ subject.** "Garry has a hero/rescuer pattern" → holder=brain, NOT people/garry-tan
2. **Atomic claims.** Split compound claims into separate rows
3. **Amplification ≠ endorsement.** Retweet-only → max weight 0.55
4. **Self-reported ≠ verified.** "Reports 7 figures" → holder=person, weight=0.75, NOT world/1.0
5. **No false precision.** Use 0.05 increments (0.35, 0.55, 0.75), not 0.74 or 0.82
6. **"So what" test.** Skip Twitter handles, follower counts, obvious metadata
-36
View File
@@ -1,36 +0,0 @@
# Tutorials
Step-by-step walkthroughs that take you from zero to a working outcome. Concrete commands, real numbers, no abstraction-first jargon. Each tutorial assumes no prior GBrain knowledge.
## Shipped
- [**Set up your personal AI agent + brain from zero**](personal-brain.md) — the canonical solo install. Two GitHub repos, a Telegram bot, AlphaClaw on Render, OpenClaw + GBrain + Supabase. End-to-end in about 2 hours; about $100 to $150 a month sustained. The full-stack install I'd run today.
- [**Set up GBrain as your company brain**](company-brain.md) — federated, multi-user, OAuth-scoped institutional memory for a 10-50 person team. Three sources (shared / customers / internal-only), per-user scope, first synthesized query as a teammate. About 90 minutes end-to-end, about $5 in API calls for the demo, under $100 a month sustained for a 25-person company.
- [**Auto-improve a skill with `gbrain skillopt`**](improving-skills-with-skillopt.md) — treat a `SKILL.md` as the trainable parameter of a frozen agent. Write your first benchmark from scratch (the part everyone gets stuck on), preview the cost, run the optimizer, read accepted vs no_improvement vs aborted, and accept a measurably better skill. About 20 minutes, about $1 in API calls. Reference: [`../guides/skillopt.md`](../guides/skillopt.md).
- [**Give your coding agent a memory: GBrain + Claude Code / Codex**](connect-coding-agent.md) — the two-funnel walkthrough for coding-agent users. Path A: connect Claude Code / Codex to a brain you already run (OpenClaw, Hermes, any `gbrain serve --http`). Path B: start from nothing with a 2-second local PGLite brain. Both end with the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits (brain-first lookup, ambient capture, briefing-from-your-brain, whoknows) that make it worth it. About 10 minutes.
## In progress
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect``suggest``review-candidates` so the brain learns your shape instead of forcing you to learn its.
- **Index your codebase as a code brain** — for developers. Initialize a brain in a code repo, swap to `voyage-code-3` for embeddings, use `gbrain code-def` / `gbrain code-refs` / `gbrain code-callers` to navigate the codebase semantically from any MCP-aware editor.
- **Run GBrain fully local with Ollama or llama.cpp** — for privacy-first deployments. No cloud calls, no API keys, no telemetry. Trades some retrieval quality for full local control. Useful for regulated industries, air-gapped environments, or just paranoia.
- **Set up the dream cycle** — the overnight enrichment daemon that makes the brain self-maintaining. Fixes citations, dedupes people pages, surfaces contradictions, generates founder scorecards on the schedule you configure. The piece that turns a static knowledge base into a brain that gets smarter while you sleep.
## Want to write one?
Tutorials follow the [Diataxis](https://diataxis.fr/) tutorial pattern: learning-oriented, walks a learner from zero to a working result in one session, every step produces a visible change. If you've used GBrain for something interesting and want to write the walkthrough, the existing [`company-brain.md`](company-brain.md) is the model. Open a PR.
## Related documentation
- **Reference:** [`docs/architecture/`](../architecture/) — system design, topologies, retrieval theory
- **How-to:** [`docs/guides/`](../guides/) — task-oriented runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion). Highlight: [scaling skills past 300](../guides/scaling-skills.md) — the three-tier architecture for agents that have outgrown the always-loaded skill manifest.
- **Integrations:** [`docs/integrations/`](../integrations/) — connecting external data sources (voice, email, calendar, embedding providers)
- **MCP setup:** [`docs/mcp/`](../mcp/) — per-client setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
- **Install paths:** [`docs/INSTALL.md`](../INSTALL.md) — every install path, end to end
-557
View File
@@ -1,557 +0,0 @@
# Tutorial: Extend your personal brain into a company brain
This tutorial picks up where the [personal brain tutorial](personal-brain.md) leaves off. You already have a working agent (OpenClaw on Render, talking to you on Telegram, with GBrain as memory and Supabase storing embeddings). Now you want your whole team to use it as shared institutional memory, with each person seeing only what they're allowed to see.
**Time:** about 90 more minutes on top of the personal-brain install.
**Cost:** under $100 a month sustained for a 25-person company.
If you haven't done the personal-brain install yet, [start there first](personal-brain.md). Come back when you've got the agent responding to you on Telegram. This tutorial assumes that's already working.
I'm Garry Tan. I built GBrain to run my own AI agents at Y Combinator. After a couple of months of multi-user features landing (parallel sync across team sources, per-user OAuth scoping, leak-free isolation across every read path), it's finally usable as a company brain too. This is the recipe I'd run if I were standing it up for a 10-50 person company today.
---
## Part 1: The mental model
### What changes when you go from personal to company
The personal brain you built is a single-user system: one git repo, one agent, your stuff. The company brain is the same architecture with three additions:
1. **Multiple sources** inside the same brain. Your meeting notes are one source. Each teammate's customer notebook is another. The shared company wiki is a third. They live in the same database but stay independent.
2. **Per-user logins** with scopes. Each teammate gets their own OAuth credential. The credential decides which sources they can read and write to. Alice writes to her customer source, reads hers plus the shared one. Bob writes to internal-ops, reads his plus the shared one. Neither can see the other's writes.
3. **Per-person folders, crons, and skills.** The shared brain has shared structure, but each teammate gets their own subfolder for their own work, their own scheduled tasks (weekly digest, customer follow-ups), and their own scoped skills.
### What this is NOT
It is **not** a different install. The agent runtime, Supabase backend, GBrain CLI, and AlphaClaw harness from the personal brain stay exactly as you set them up. We're adding to that stack, not replacing it.
It is also **not** a thin-client-everywhere setup. Your personal agent stays as it is (OpenClaw + Telegram). Each teammate adds their own client of choice (Claude Code, Cursor, Claude Desktop, their own OpenClaw, whatever) and points it at the brain.
### What you get that one person's brain doesn't
- **Shared memory.** The whole team queries the same brain. The contract notes that Alice wrote on Tuesday show up when Bob asks about that customer on Friday, with citations back to Alice's notes.
- **Scoped privacy.** Performance reviews don't leak into customer queries. Legal docs don't leak into sales searches. We fuzz-tested this across every read path and got zero leaks.
- **One sync pipeline.** Your brain git repo (or several if you want them isolated per team) feeds the brain. Everyone sees the latest.
- **One operating burden.** One server to monitor, not one per user.
---
## Part 2: Switch the brain backend to multi-user Postgres
The personal-brain install uses Supabase as the embeddings layer but the GBrain runtime itself might be using PGLite (single-machine) depending on which path you took. For a company brain, you want a real Postgres for the runtime too. If your personal-brain install is already on Postgres or Supabase end-to-end, skip to Part 3.
If you're on PGLite, migrate:
```bash
gbrain migrate --to supabase
```
This copies every page, chunk, embedding, link, and config over to your Supabase project. Run from the agent host machine, same one you set up in the personal-brain tutorial. Takes a few minutes per 10K pages.
Verify:
```bash
gbrain doctor
gbrain stats
```
Page count and chunk count should match what you had on PGLite.
---
## Part 3: Carve up the brain into sources
The personal brain has one source (called `default`) holding everything. For a company brain we want multiple. The right shape depends on your org. Here's a typical starting point for a 10-50 person company:
```bash
# A shared all-hands source for content everyone reads
gbrain sources add shared --path /srv/brain-repos/shared --name "Shared company wiki"
# A scoped source for sales/customer notes
gbrain sources add customers --path /srv/brain-repos/customers --name "Customer notes"
# A scoped source for internal-only docs (legal, HR, performance, board)
gbrain sources add internal --path /srv/brain-repos/internal --name "Internal-only"
```
Each `--path` is a directory on disk where you've checked out a git repo. Create them:
```bash
sudo mkdir -p /srv/brain-repos
sudo chown $USER /srv/brain-repos
cd /srv/brain-repos
git clone git@github.com:your-org/shared-wiki.git shared
git clone git@github.com:your-org/customers.git customers
git clone git@github.com:your-org/internal-docs.git internal
```
You can also keep the existing personal-brain repo as one of the sources. Just pick the role it plays (probably `shared` if it's already org-wide content).
### Two scoping models (pick the one that matches your shape)
There are two ways to scope teammates' access. They suit different deployment shapes.
**Model A: separate sources with OAuth scoping (recommended for true multi-user with different AI clients).** What this tutorial walks you through. Each teammate gets their own OAuth client, which carries `--source` + `--federated-read` flags. The brain refuses cross-source reads at the SQL layer; isolation is database-enforced. Each teammate can run their own MCP-aware client (Claude Code, Cursor, their own OpenClaw, etc.) and the scoping holds.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. There's no OAuth-enforced isolation; the agent itself enforces "Alice's writes go to her partners/ subdir." This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth, but the scoping is convention-only.
For most company-brain installs (10+ teammates each with their own AI client), Model A is the right starting point. If you're running the fat-agent-serves-everyone pattern from the personal-brain tutorial, Model B is genuinely simpler. You can also mix: separate sources for the obviously-different ones (customer notes vs internal-only) AND a `partners/<slug>/` convention inside the shared source for per-person workspace.
### Per-person folder structure inside each source
Inside each source, give each teammate their own subfolder. This is the structure I run:
```
customers/
├── alice-example/ ← Alice's customer notebook
│ ├── customers/
│ │ ├── acme-co.md
│ │ └── widget-systems.md
│ └── meetings/
│ └── 2026-05-21-acme-renewal.md
├── bob-example/ ← Bob's customer notebook
│ └── customers/
│ └── orbit-bio.md
└── shared-customers/ ← things both can see
└── all-active-deals.md
```
Two things this structure buys you:
1. **Each teammate's writes go to their own folder** even though they're in the same source. No accidental overwrites.
2. **You can later split a person's folder into its own source** (if Alice leaves and a new person takes her accounts, you can move `alice-example/` to a new source named after the new person and adjust scoping accordingly).
Same shape for `internal/`: `internal/alice-example/` for her HR docs, `internal/bob-example/` for his, `internal/legal/` for legal docs everyone can read, etc.
Now sync everything:
```bash
gbrain sync --all
```
Each source syncs in parallel under its own lock so they don't step on each other. Output looks like:
```
[shared] 100/100 pages
[customers] 240/240 pages
[internal] 85/85 pages
✓ all sources synced
```
Check the dashboard:
```bash
gbrain sources status
```
You should see all three sources with recent sync timestamps and page counts.
---
## Part 4: Expose the brain over HTTP MCP with OAuth
The personal brain talks to you through the AlphaClaw harness over Telegram. For a company brain we need a path that each teammate's AI client can hit independently. The HTTP MCP server is that path.
```bash
gbrain serve --http --port 3131 --bind 0.0.0.0
```
The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface.
The server prints an admin bootstrap token to stderr on first start. Save it. You'll use it once for the admin dashboard.
For development, tunnel the local server out via ngrok:
```bash
ngrok http 3131 --domain your-brain.ngrok.app
```
For production, put your server behind a real hostname with a real TLS certificate. Let's call your final URL `https://brain.acme-co.com` for the rest of this tutorial.
Re-run the server with the public URL so the OAuth discovery metadata matches what clients hit:
```bash
gbrain serve --http --port 3131 --bind 0.0.0.0 --public-url https://brain.acme-co.com
```
You should be able to hit `https://brain.acme-co.com/health` and get `{"status":"ok"}` back.
---
## Part 5: Register one OAuth client per teammate
Each teammate (or each AI agent for a teammate) gets their own OAuth client. The client controls what they can write and what they can read.
```bash
# Alice (sales): writes customers/alice-example, reads customers + shared
gbrain auth register-client alice-example \
--grant-types client_credentials \
--scopes read,write \
--source customers \
--federated-read customers,shared
# Bob (ops): writes internal/bob-example, reads internal + shared
gbrain auth register-client bob-example \
--grant-types client_credentials \
--scopes read,write \
--source internal \
--federated-read internal,shared
# Carol (legal): writes shared/legal, reads all three
gbrain auth register-client carol-example \
--grant-types client_credentials \
--scopes read,write \
--source shared \
--federated-read shared,customers,internal
```
Each `register-client` command prints a `client_id` and a `client_secret`. Save both for each teammate. They go into the teammate's local agent config.
A note on the flags:
- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder.
- `--federated-read` controls read scope. A client can read from one or more sources.
### Verify the scoping actually scopes
Before you hand the brain to teammates, verify isolation. Two terminal windows on your local machine using each client's credentials:
```bash
# Terminal 1, as Alice
export GBRAIN_REMOTE_CLIENT_ID=<Alice's client_id>
export GBRAIN_REMOTE_CLIENT_SECRET=<Alice's client_secret>
export GBRAIN_REMOTE_MCP_URL=https://brain.acme-co.com/mcp
gbrain search "performance review" --remote
```
Alice should see results only from `customers` and `shared`. The performance-review notes live in `internal`, which she's not scoped to read. She shouldn't see them.
```bash
# Terminal 2, as Bob (export his credentials similarly)
gbrain search "performance review" --remote
```
Bob should see the performance-review notes from `internal`, plus anything related from `shared`. He shouldn't see anything that lives only in `customers`.
If both queries return correctly scoped results, isolation is working.
---
## Part 6: Set up per-person crons
The personal-brain install runs the dream cycle (overnight enrichment) once per night for one user. A company brain needs per-person crons because each teammate has their own context: Alice wants a 7am customer-pipeline digest, Bob wants a 9am ops-status report, Carol wants a contract-compliance check every Monday.
Each cron is just a scheduled `gbrain agent run` call scoped to the teammate's client credentials. The schedule lives in the workspace repo (the one AlphaClaw deployed in the personal-brain tutorial), in a `crons/` directory. A typical layout:
```
your-org/myagent/
└── crons/
├── alice-example/
│ └── 07am-customer-digest.md
├── bob-example/
│ └── 09am-ops-status.md
└── carol-example/
└── monday-contract-compliance.md
```
Each cron file declares its schedule and the prompt that the agent runs:
```markdown
---
schedule: "0 7 * * *"
client: alice-example
---
# Customer pipeline digest
Pull every customer page in customers/alice-example/ that had activity in
the last 7 days. For each, summarize what changed and what the next action
is. Output as a markdown digest, post to Slack #alice-customers, save a
copy to customers/alice-example/digests/YYYY-MM-DD-pipeline.md.
```
The `client:` field tells the cron runner which OAuth client to use, which enforces the scoping. Alice's cron can only read Alice's sources and write to Alice's folder. It cannot accidentally touch Bob's customer notes.
To install the cron schedule, commit the file to the workspace repo and let AlphaClaw pick it up on next deploy. The cron-scheduler skill (one of the 60 that GBrain installed) handles the dispatch.
---
## Part 7: Add per-person skills
The 60+ skills GBrain installs are generic. Your team probably wants a few that are specific to them. Examples:
- `onboarding-new-hire`. Only Carol (HR) runs this. Walks through generating a welcome packet, scheduling intro meetings, provisioning accounts.
- `customer-success-followup`. Only Alice (sales) runs this. Pulls latest customer page, drafts a follow-up email, posts to her review queue.
- `weekly-team-digest`. Only you (admin) run this. Aggregates everyone's published pages into one weekly summary.
Skills are just markdown files in the workspace repo's `skills/` directory. The shape:
```
your-org/myagent/
└── skills/
├── onboarding-new-hire/
│ └── SKILL.md
├── customer-success-followup/
│ └── SKILL.md
└── weekly-team-digest/
└── SKILL.md
```
Each `SKILL.md` declares the trigger (verbs in plain English the agent listens for) and the procedure. Use the `gbrain skillify scaffold <name>` command to generate the boilerplate:
```bash
gbrain skillify scaffold onboarding-new-hire
```
That creates the directory + SKILL.md + routing entry. Edit the SKILL.md to describe the procedure, commit, deploy. The agent picks up the new skill on next request.
Per-person scoping for skills is handled at the routing layer: a skill can declare `allowed_clients: [carol-example]` in its frontmatter. If Alice asks her agent to run that skill, the agent refuses with "this skill is scoped to carol-example."
### Shared rule files at the skills root
Alongside individual skill directories, drop a few flat `_*-rules.md` files at the root of `skills/`. These are conventions that EVERY skill reads. The ones I run in production:
- `_brain-filing-rules.md`. the iron-rule decision tree for "where does this new page belong?" Numbered first-match-wins rules (people go in `people/`, companies in `companies/`, meetings in `meetings/`, etc.). Every ingest skill consults this before creating a page.
- `_output-rules.md`. output quality standards (deterministic links built from API data not LLM-composed strings, exact-phrasing requirements for citations, no AI-slop vocabulary).
- `_excluded-people.md`. a privacy gate. Names that must never be referenced or attributed in the brain even if they appear in source material. Re-attribute or discard. This is the file that prevents your agent from accidentally publishing things about people you've decided aren't fair game.
- `_operating-rules.md`. operational conventions (when to write to brain vs scratchpad, when to ask for confirmation, when to fire a notification).
- `_x-ingestion-rules.md`, `_x-api-rules.md`. per-source rules for specific integrations (Twitter, in this case).
These files turn into the de facto company policy for the agent. Edit one, and every skill that reads it picks up the new rule on the next request. Versioned in git, reviewable in PR.
---
## Part 8: Wire Slack carefully
Slack is the integration most teams want first, and it has enough sharp edges to deserve its own callout. The conventions I run:
**Two crons, two jobs.** One scan cron that runs every 5-15 minutes and surfaces signals (new threads in channels you care about, mentions of your teammates, decisions). One archive cron that runs nightly and stores the full conversation history. Splitting them this way means urgent signals get acted on fast while the slow archive work doesn't crowd the live channel.
**Channel-to-task-ID mapping.** Don't have your agent reference Slack channels by their actual channel IDs (`C03A8...`). Build a `topic-registry.json` (or similar) that maps each channel ID to a friendly task name (`acme-co-customer-success`, `engineering-standup`). Crons and skills reference channels by friendly name; the registry translates to IDs at runtime. This is the file you edit when a channel gets renamed or replaced.
**Deterministic links only.** When your agent writes a brain page that cites a Slack message, the link MUST be built from API data (workspace ID + channel ID + message timestamp), never composed by the LLM. LLMs hallucinate Slack URLs constantly. The convention lives in `_output-rules.md`; every skill that touches Slack inherits it.
**Dismissed-items state.** The scan cron remembers what it has already surfaced. If a channel had a thread on Tuesday that turned out to be noise, the dismissed-items file records it so the Wednesday scan doesn't surface it again. Without this, re-scans become a flood of repeat signals.
**Per-channel scoping mirrors per-person scoping.** Sensitive channels (#executive, #legal, #performance) should be scoped to teammates with the appropriate `--federated-read`. The brain stores everything, but who can query for it is gated by the same OAuth client model from Part 5.
The actual skills that implement this in production are named `slack`, `slack-scan`, `slack-archive`. Scaffold equivalents in your workspace with `gbrain skillify scaffold slack-scan`, then edit the generated SKILL.md to declare your channel mapping and triggers.
---
## Part 9: Onboard each teammate yourself (the botmaster pattern)
This is the part that decides whether your company brain actually gets adopted or sits unused.
**Do not just hand a new teammate their OAuth credential and tell them to "try it out."** They'll send one query, get a result that doesn't feel personal yet (because their slice is empty), conclude it's not useful, and never come back.
What works instead: I personally onboard each new teammate myself. The flow looks like this.
### Step 1: Pre-populate their slice
Before they ever log in, I seed their `partners/<their-slug>/` directory (or their dedicated source) with the context they need to feel like the brain already knows them:
- `partners/alice-example/USER.md`. a one-page profile: role, focus areas, current top 3 priorities, the kind of questions they tend to ask, the kind of writing they prefer (terse vs detailed, casual vs formal).
- `partners/alice-example/concepts/`. 5-10 frameworks or recurring themes that are specifically THEIRS. If Alice runs sales, that's "pipeline stage definitions," "ICP criteria," "objection-handling playbooks."
- `partners/alice-example/sources/`. links to the documents they care about (their team's shared docs, their inbox conventions, the dashboards they check).
- 2-3 example brain entries that demonstrate the shape: a customer page they'd recognize, a meeting note from a recent meeting they attended, an idea they've shared with the team.
Takes me maybe 20 minutes per teammate. The payoff: the moment they run their first query, the brain answers with their context, not a generic response. That's the difference between "this is a cool tool" and "this knows me."
### Step 2: Walk them through 2-3 wow flows
Before letting them DM the agent freely, I personally walk them through 2-3 specific flows that I know will land:
1. A query that demonstrates synthesis: "ask the brain about [a customer they know well]. Notice how it pulls together pages from three sources into one answer with citations." This shows the brain layer in action.
2. A query that demonstrates gap analysis: "ask the brain about [something it doesn't know yet]. Notice how it tells you what's missing instead of making it up." This builds trust.
3. A write-back flow: "tell the brain about [a meeting they just had]. Notice how it auto-files, links to the other people who were there, and surfaces related history." This shows the agent's value as a capture tool, not just a query tool.
These three flows take maybe 15 minutes total. By the end, the teammate has seen the brain do something they couldn't have done themselves in that time. They feel powerful.
### Step 3: Graduate to DM only after the wow moment lands
After the walkthrough, I give them their OAuth credential and the agent's DM (Telegram, Slack DM, whatever your interface is). I explicitly say "now you can ask it anything, write to it anytime, and it'll keep learning from you."
The order matters. If you give them DM access first and expect them to discover the wow moments themselves, most won't. They'll send one generic query, get a generic answer, and bounce. The botmaster pattern (pre-populate → walk through → graduate to DM) flips the conversion rate.
Repeat this flow for every new teammate. About 45 minutes per person, total. Compared to the cost of an unadopted internal tool, it's the best 45 minutes you'll spend.
---
## Part 10: Connect each teammate's AI client
Each teammate runs their AI client (Claude Code, Cursor, Claude Desktop, OpenClaw, Hermes, whatever) configured to point at your brain server through their OAuth credentials.
Recommended path for each teammate: the thin-client install. On their machine:
```bash
curl -fsSL https://bun.sh/install | bash
bun install -g github:garrytan/gbrain
gbrain init --mcp-only \
--issuer-url https://brain.acme-co.com \
--mcp-url https://brain.acme-co.com/mcp \
--oauth-client-id <their client_id> \
--oauth-client-secret <their client_secret>
```
The thin-client install creates a local config that knows how to talk to your brain but never opens its own database. Most CLI commands route through the remote server transparently.
Now they configure their AI client. For Claude Desktop, the teammate adds an MCP server entry in `~/Library/Application Support/Claude/claude_desktop_config.json`:
```jsonc
{
"mcpServers": {
"company-brain": {
"command": "gbrain",
"args": ["serve"]
}
}
}
```
When Claude Desktop launches, it talks to the local `gbrain serve` stdio bridge, which forwards every request to your remote brain over HTTPS with their OAuth token attached. From Claude Desktop's perspective it's just one MCP server.
For Claude Code, Cursor, OpenClaw, Hermes, and other clients, per-client setup steps live in [`docs/mcp/`](../mcp/). They all follow the same shape: point the agent at the local `gbrain serve` bridge, which knows about the remote.
---
## Part 11: First real query as a teammate
Have Alice run a real query from her machine. The interesting verb is `gbrain think`, which gives back a synthesized answer instead of raw pages.
```bash
gbrain think "What's the latest update from acme-co? When did we last talk to them?"
```
What Alice gets back, assuming the brain has been syncing for a week and her sources contain a customer page for acme-co and several meeting notes:
```
## Answer
The most recent customer contact with acme-co was a renewal-discussion
meeting on 2026-05-18, attended by alice-example and acme-co's CTO. Key
points discussed [customers/alice-example/meetings/2026-05-18-acme-renewal]:
- They are upgrading their plan from team to enterprise.
- Annual contract value is moving from $48K to $180K.
- Decision driver: a new compliance requirement they have to meet by Q3.
Prior contact was a quarterly check-in on 2026-04-03 [customers/alice-example/meetings/2026-04-03-acme-q2-checkin].
**Gap noted:** No customer-success notes have been filed since the
2026-05-18 renewal meeting. If a follow-up has happened, it's not in
the brain yet.
```
Three things to notice:
1. **Sourced.** Every claim cites the meeting note it came from.
2. **Synthesized.** Alice didn't read three pages and stitch them together. The brain did.
3. **Honest about gaps.** The brain knows what it doesn't know and says so, instead of inventing a follow-up that didn't happen.
That last part is the gap analysis. It's the part of the brain layer that nobody else ships.
Bob asking the same question would get nothing about acme-co. He's not scoped to read `customers`. He'd see his own internal-ops content if he asked something relevant to that. Carol asking would see both, because she's scoped to read all three sources.
---
## Part 12: Operating the company brain
Three commands do most of the operational work.
### Background daemon: `gbrain autopilot`
The personal-brain install already turned this on. For a company brain, the same autopilot covers all your sources because they live in one database. It runs every five minutes; on a healthy brain (health score 95+) it sleeps; on a brain that's drifting it submits targeted maintenance jobs.
### Self-healing: `gbrain doctor --remediate`
```bash
gbrain doctor --remediate --yes --target-score 90 --max-usd 5
```
Computes a dependency-ordered plan of maintenance jobs that would raise the brain's health score to the `--target-score`, runs the plan, refuses to spend past the `--max-usd` cap. Safe to cron.
### Monitoring: `gbrain sources status` and the admin dashboard
```bash
gbrain sources status
```
Returns a per-source dashboard: when each source last synced, how many pages, how many embedded, how many unacked sync failures. The at-a-glance health check.
The admin dashboard at `https://brain.acme-co.com/admin` shows live request volume, registered OAuth clients, recent activity, and brain stats. Use the admin bootstrap token from Part 4 to log in the first time, then register additional admin users from inside the dashboard.
---
## Part 13: Cost and speed expectations
Real numbers from the published benchmark, running the default stack (GBrain with ZeroEntropy for embedding + reranker):
- **Embedding cost:** $0.05 per million tokens. For comparison, GBrain configured with OpenAI is $0.13 (2.6× more expensive), Voyage is $0.18 (3.6× more).
- **Ingest speed:** about 22 seconds for a small test corpus of 164 pages on the host machine. For a 10K-page corpus, expect about 20 minutes the first time, then most syncs are incremental and finish in seconds.
- **Query latency:** about 122 ms median for a `gbrain search`. For comparison, the same query through GBrain with OpenAI takes about 282 ms.
- **Synthesized-answer latency:** a few seconds, dominated by the Anthropic API.
- **Retrieval quality:** on the public LongMemEval benchmark, GBrain hits 97.60% recall at the top 5 retrieved sessions, beating the previous published state of the art at 96.6%. On the in-house BrainBench corpus of relational queries, GBrain beats commodity vector retrieval by 38 percentage points, because the graph layer surfaces relationships that vector similarity alone misses.
Full methodology and per-run receipt JSONs live in [the gbrain-evals repo](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-05-23-v0.40.6.0-snapshot.md).
For a 25-person company at sustained use, expect about $35 a month in embeddings (ZeroEntropy at $0.05/million tokens), $50 a month in Anthropic calls for the synthesized-answer queries, plus your hosting bill. Under $100 a month for the AI side at most companies your size.
---
## Part 14: Common gotchas
### "My teammate can't see anything"
Check `gbrain auth list` on the host and confirm their client has `--source` set to a source that actually exists. Empty or null `--source` means the client falls through to the `default` source, which probably has no content if you set up three named sources.
### "Sync is slow and feels stuck"
The first sync embeds every page, which takes time. Check `gbrain sources status` for the live page count. If it's climbing you're not stuck, you're just embedding. If you've got a 10K-page corpus and ZeroEntropy is being throttled, the per-source parallel sync looks like progress on three sources at once rather than one source moving fast.
### "I see a page I shouldn't see"
This shouldn't happen, but if you suspect it, run `gbrain search <query> --remote --json` as the constrained client and inspect the `source_id` field on every returned result. Every row should be in the client's `--federated-read` set. If one isn't, file an issue with the exact slug and source IDs.
### "The synthesized answer is wrong"
The brain layer is grounded in the retrieved pages. If the retrieved pages contain bad information, the answer will too. The gap-analysis note often catches this: if the answer says "based on retrieved pages from date X" and date X is six months ago, the brain is telling you the information is stale. Run `gbrain sync --all` to refresh and try again.
### "OAuth `/token` endpoint returns 401 for my client"
Verify the client secret matches what was printed at register-client time. The server stores only a SHA-256 hash; if you lost the original, you have to revoke the client and re-register. Use `gbrain auth revoke-client <client_id>` and re-run `register-client`.
### "Postgres connection is exhausting"
Each parallel sync worker opens its own pool. With three sources and the default four workers per source, you can hit your Postgres connection limit if it's set low. Either reduce the worker count with `gbrain sync --all --parallel 2 --workers 2`, or raise your Postgres `max_connections` to at least 100. Supabase's free tier defaults to 60, which is tight.
### "I want to add a fourth teammate but they need access to all three sources"
```bash
gbrain auth register-client diana-example \
--grant-types client_credentials \
--scopes read,write \
--source shared \
--federated-read shared,customers,internal
```
That's it. Add or rotate teammates as the org grows.
---
## What you built
You now have the personal-brain agent from the previous tutorial, plus a multi-user shared layer on top: three federated sources holding shared, customer, and internal-only content; per-person folders inside each source so teammates' writes don't collide; per-person OAuth clients with scoped read and write; per-person crons that run on each teammate's own schedule with their own scoping; per-person skills the agent only runs for the right person. Each teammate queries the brain in plain English through their AI client and gets back synthesized, sourced answers that are correctly scoped.
What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
Questions, gotchas, or wins worth sharing? Open an issue at [github.com/garrytan/gbrain](https://github.com/garrytan/gbrain/issues).
-235
View File
@@ -1,235 +0,0 @@
# Give your coding agent a memory: GBrain + Claude Code / Codex
Coding agents got very good at code. They're still amnesiac about everything
else. Claude Code and Codex forget your last conversation, can't tell you what
you decided three meetings ago, and re-derive context you already have written
down somewhere. GBrain is the retrieval layer that fixes that: search, synthesis,
and a self-wiring knowledge graph, wired into your agent over MCP.
There are two ways to do this. Pick the one that matches where you are:
- **Path A — I already run a brain** (OpenClaw, Hermes, or any `gbrain serve`
host) and I want my Claude Code / Codex to reach the same brain. → [jump to Path A](#path-a-connect-an-agent-to-a-brain-you-already-have)
- **Path B — I have nothing yet.** Spin up a local brain in 2 seconds and wire it
into my coding agent. → [jump to Path B](#path-b-start-from-nothing-local-brain-local-agent)
Both end in the same place: an agent that searches your brain before it answers,
and writes new knowledge back as you work. The last section,
[Now make it actually useful](#now-make-it-actually-useful), is the same for both
and is the part that changes how you work.
Prerequisite for either path: `bun install -g github:garrytan/gbrain`.
---
## Path A: connect an agent to a brain you already have
You already have a populated brain (the OpenClaw / Hermes case: it's on your
agent host, full of meetings, people, and ideas). You want Claude Code on your
laptop, and Codex too, to query it. This is the remote path: the host serves
HTTP, your laptop agents connect with a token.
### A1. On the host: serve over HTTP
If your host isn't already serving HTTP MCP, start it:
```bash
gbrain serve --http --bind 0.0.0.0 --public-url https://your-host.example.com
```
Two flags matter and people skip them:
- **`--bind 0.0.0.0`** — the default bind is `127.0.0.1` (loopback only), which
silently refuses every remote connection. If your agent "can't reach the
brain" and you didn't pass this, that's why. `gbrain serve --http` warns you at
startup when `--public-url` is set without `--bind`.
- **`--public-url`** — the externally reachable HTTPS URL (your Render/Railway
URL, ngrok domain, Tailscale Funnel, etc.). It's the issuer the OAuth/MCP
layer advertises.
Watch the startup banner. It now prints a `Skills:` line:
```
║ Skills: published ║
```
If it says `not published`, your connected agents will be able to search and
write but won't see your skill catalog (the OpenClaw skills that make your setup
special). Turn it on:
```bash
gbrain config set mcp.publish_skills true
```
(New brains from `gbrain init` default this ON. Brains upgraded from before
v0.41.36 stay OFF until you opt in, so this is the common gotcha for existing
OpenClaw users.)
### A2. On the host: mint a token
```bash
gbrain auth create "laptop-agents"
```
Copy the `gbrain_…` token it prints. It's a long-lived, full-access secret. Treat
it like a password; prefer a scoped OAuth client for anything cloud-hosted (see
[DEPLOY.md](../mcp/DEPLOY.md)).
### A3. On the laptop: one command per agent
```bash
# Claude Code
gbrain connect https://your-host.example.com/mcp --token gbrain_xxx --install
# Codex
gbrain connect https://your-host.example.com/mcp --token gbrain_xxx --agent codex --install
```
`--install` runs the agent's `mcp add` for you AND smoke-tests the token: it
actually calls `get_brain_identity` before handing off, so a wrong or expired
token fails right now, not silently on the agent's first request. You'll see:
```
Added MCP server 'gbrain' -> https://your-host.example.com/mcp.
Verified: {"version":"0.42.x","engine":"postgres","page_count":146646,...}
```
Drop `--install` to print a paste-ready block instead (useful when the host and
the agent are different machines, or you want to read before you run). Codex
reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands
in Codex's config file. Keep that variable exported in your shell profile.
### A4. Verify
In the agent: *"Call get_brain_identity, then search my brain for [a topic you
know is in there]."* You should get your own pages back. Done.
Full per-client detail: [Claude Code](../mcp/CLAUDE_CODE.md),
[Codex](../mcp/CODEX.md), [Perplexity](../mcp/PERPLEXITY.md).
---
## Path B: start from nothing (local brain, local agent)
No OpenClaw, no server, no token. The lowest-friction path in the whole product:
a local PGLite brain in the same process your agent spawns. Zero server, zero
tunnel.
### B1. Create a local brain
```bash
gbrain init --pglite # 2 seconds; embedded Postgres via WASM, no Docker
```
### B2. Put something in it
A brain with nothing in it answers nothing, so an empty brain on day one feels
broken. Two ways to fill it:
```bash
# Bulk-import a folder of markdown you already have:
gbrain import ~/notes/
# Or capture as you go (one thought at a time):
gbrain capture "Decided to use PGLite as the default engine: zero-config beats Postgres for <1000 files."
```
You don't have to import everything up front. The capture-as-you-go habit (see
the next section) means the brain fills with the decisions and context you
generate while working, and is genuinely useful by day two.
### B3. Wire it into your coding agent
```bash
# Claude Code
claude mcp add gbrain -- gbrain serve
# Codex
codex mcp add gbrain -- gbrain serve
```
That's the whole wire-up. No token, no URL, no tunnel. The agent spawns
`gbrain serve` as a stdio subprocess and talks to your local brain directly.
### B4. Verify
In the agent: *"search my brain for PGLite"* (or whatever you just captured). You
get the page back. The same brain is now query-able from the CLI
(`gbrain query "..."`) and from your agent.
---
## Now make it actually useful
Connecting is the easy part. The value comes from teaching your agent a few
habits. These are the patterns that turn a coding agent into a knowledge-aware
one. Paste the protocol below into your agent's instructions file
(`CLAUDE.md` for Claude Code, `AGENTS.md` for Codex / Cursor / others), then lean
on the patterns.
### The brain-first protocol (paste this in)
```markdown
## Brain-first protocol
You have a knowledge brain connected over MCP. Before answering any question
about people, companies, decisions, projects, or past context:
1. **Search first.** Call `search` (or `query` for a synthesized answer) against
the brain BEFORE answering from memory or asking me. If the brain has the
answer, use it. Never ask "who is X?" or "what did we decide about Y?" before
searching — the brain probably already knows.
2. **Write back.** When I make a decision, mention a new person/company, or land
on an idea worth keeping, write it to the brain with `put_page` (entity pages
under people/, companies/; decisions under decisions/ or notes/). One insight,
one page, linked.
3. **Cite.** When you answer from the brain, name the page you used.
```
### The four patterns worth stealing
These come straight from a production OpenClaw setup. They translate directly to
any coding agent with GBrain connected:
**1. Brain-first lookup (never ask what you can retrieve).** The single highest-
value habit. Before the agent asks you "which repo?" or "who owns this?", it
searches. Try: *"What did we decide about the auth rewrite?"* and watch it pull
the decision page instead of asking you to re-explain.
**2. Ambient capture (your brain as a side effect of working).** Don't make
saving a separate chore. Tell the agent: *"As we work, capture any decision or
new idea to the brain without interrupting."* After a month of this, you have
hundreds of linked pages and patterns you didn't know were there.
**3. Briefing from your brain (not from the internet).** *"What do I need to know
before my 2pm with the Acme team?"* pulls your meeting history, the people,
what's still open, what the brain doesn't know yet. The agent does your prep
because it read your context. (`query` gives you the synthesized answer with
citations; this is the example on the [README](../../README.md).)
**4. whoknows (expertise routing).** *"Who do I know who's shipped a rate
limiter in Postgres?"* The `find_experts` tool ranks people in your brain by
relevance + recency. Useful the moment your brain has more than a handful of
people in it.
That's the spine of it. Two commands to connect, one protocol to paste, four
habits to build. Your agent stops being amnesiac.
---
## Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Agent "can't reach the brain" (Path A) | `gbrain serve --http` bound to loopback | Restart with `--bind 0.0.0.0` |
| `list_skills` returns nothing / errors | Skill publishing OFF on the host | `gbrain config set mcp.publish_skills true` |
| Token rejected on first call | Wrong/expired token | Re-mint with `gbrain auth create`; `--install` smoke-tests it for you |
| `unknown tool: capture` | `capture` is CLI-only, not an MCP tool | Use `put_page` over MCP; `capture` only on the CLI |
| Empty results (Path B) | Brain has nothing in it yet | `gbrain import ~/notes/` or `gbrain capture "..."` |
## Next steps
- Go full autonomous: the overnight enrichment daemon ([dream cycle](../../CHANGELOG.md)) fixes citations, dedupes people, builds scorecards while you sleep. See `gbrain autopilot --install`.
- Run a real agent platform on top: [personal-brain tutorial](personal-brain.md).
- Scale to a team: [company-brain tutorial](company-brain.md).
- Every MCP client's exact setup: [`docs/mcp/`](../mcp/).
@@ -1,288 +0,0 @@
# Auto-improve a skill with `gbrain skillopt`
You have a `SKILL.md`. Sometimes the agent following it does a great job, sometimes
it forgets a step or pads the output. This tutorial takes you from that skill to a
measurably better version of it, in one session, without you hand-editing the
prose. By the end you'll have written your first benchmark, watched the optimizer
propose and test edits, and accepted an improvement that actually scored higher.
Time: ~20 minutes. Cost: ~$1 in API calls for the worked example.
Based on [SkillOpt](https://arxiv.org/abs/2605.23904) (Microsoft Research, May 2026).
## The mental model (two sentences)
Your `SKILL.md` is the trainable parameter; the agent that reads it never changes.
SkillOpt runs the agent against a benchmark of realistic tasks, proposes specific
edits to the skill body, re-tests, and keeps a change **only when it measurably
beats the current version** on a held-out slice.
That's the whole idea. The benchmark is how "better" gets defined — which is why
writing it is the one part you can't skip. Everything else is mechanical.
## The easiest path: generate a starter, then strengthen it
You don't start from a blank file. One command reads the SKILL.md and writes a
full starter benchmark for you:
```bash
gbrain skillopt meeting-prep --bootstrap-from-skill
```
It infers what the skill produces, writes ~15 tasks (each with rule judges) to
`skills/meeting-prep/skillopt-benchmark.jsonl`, and appends a
`# BOOTSTRAP_PENDING_REVIEW` sentinel so nothing runs until a human has looked.
Then you **review and strengthen the judges** (the generated checks are weak
drafts), delete the sentinel line, and run:
```bash
gbrain skillopt meeting-prep --bootstrap-reviewed --split 1:1:1
```
If you run an agent over this brain (OpenClaw, Claude Code, Cursor, any MCP client
with the gbrain skills installed), it does this for you: just say "improve my
meeting-prep skill." It runs `--bootstrap-from-skill`, strengthens the judges,
dry-runs for cost, runs the optimizer, and reports the diff + score delta back.
You keep or discard.
**Read the rest of this tutorial to understand what that command produces** — the
benchmark format, how to strengthen a draft (or write one by hand), how to read
the outcome, and where the output lands.
## What you'll need
- `gbrain` installed and a brain initialized (`gbrain --version` works).
- One embedding/chat provider configured. SkillOpt makes real LLM calls.
`gbrain models doctor` should show at least one reachable chat model.
- A skill you want to improve, living at `skills/<name>/SKILL.md`. This tutorial
uses a skill called `meeting-prep` — substitute your own name everywhere.
- A clean git working tree for that skill file (SkillOpt refuses to run over
uncommitted changes so it can never clobber your edits; `--force` overrides).
If you don't have a skill yet, scaffold one first:
```bash
gbrain skillify scaffold meeting-prep
```
## Step 1: Get a benchmark — generated or hand-written
A benchmark is a `.jsonl` file — **one JSON object per line** — where each line is
a task plus a way to score the agent's answer. It's the crux: the benchmark IS
your definition of "better."
**The recommended way is to generate a starter** (the section above):
`gbrain skillopt meeting-prep --bootstrap-from-skill` writes the file for you, then
you strengthen the judges. The format below is exactly what it produces, so this
section doubles as your guide to reviewing and sharpening a generated draft.
**To follow this tutorial verbatim** (or to hand-curate from scratch), paste this
complete 15-task starter. It's deliberately generic — once you've seen the loop
work, **replace these tasks with your skill's real cases** (that's Step 6):
```bash
cat > skills/meeting-prep/skillopt-benchmark.jsonl <<'EOF'
{"task_id":"mp-001","task":"Prep me for a 1:1 with a direct report I haven't met with in 3 weeks.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"agenda"},{"op":"contains","arg":"follow-up"}]}}
{"task_id":"mp-002","task":"Prep me for a first sales call with a company I know nothing about.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"company"},{"op":"min_citations","arg":1}]}}
{"task_id":"mp-003","task":"Prep me for a board meeting where I present the quarterly numbers.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"metric"}]}}
{"task_id":"mp-004","task":"Prep me for a performance review I'm giving to an underperformer.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"example"}]}}
{"task_id":"mp-005","task":"Prep me for a candidate interview for a senior backend role.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"question"}]}}
{"task_id":"mp-006","task":"Prep me for a vendor renewal negotiation where I want a discount.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"leverage"}]}}
{"task_id":"mp-007","task":"Prep me for a kickoff with a new cross-functional project team.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"goal"},{"op":"contains","arg":"owner"}]}}
{"task_id":"mp-008","task":"Prep me for a difficult conversation about a missed deadline.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"impact"}]}}
{"task_id":"mp-009","task":"Prep me for an investor update call after a flat quarter.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"metric"},{"op":"min_citations","arg":1}]}}
{"task_id":"mp-010","task":"Prep me for a skip-level with someone two reports below me.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"question"}]}}
{"task_id":"mp-011","task":"Prep me for a customer escalation call after an outage.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"timeline"}]}}
{"task_id":"mp-012","task":"Prep me for a partnership exploration call with a competitor-adjacent company.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"company"},{"op":"min_citations","arg":1}]}}
{"task_id":"mp-013","task":"Prep me for a sprint retro where morale has been low.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"action"}]}}
{"task_id":"mp-014","task":"Prep me for a salary negotiation a report initiated.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"market"}]}}
{"task_id":"mp-015","task":"Prep me for an all-hands where I announce a reorg.","judge":{"kind":"rule","checks":[{"op":"max_chars","arg":1800},{"op":"contains","arg":"why"}]}}
EOF
```
Each line has three fields:
- `task_id` — a unique label. Anything; you'll see it in the audit trail.
- `task` — the prompt the agent gets, exactly as a user would phrase it.
- `judge` — how the answer is scored. `kind: "rule"` is deterministic and **free**
(no LLM call): it runs a list of `checks`, and the task's score is the fraction
that pass.
The rule checks you can use:
| `op` | `arg` | Passes when the agent's answer… |
|---|---|---|
| `contains` | string | includes that substring |
| `regex` | string | matches that regex (multiline) |
| `section_present` | heading text | has a markdown heading with that text |
| `max_chars` | number | is at most that many characters (punishes padding) |
| `min_citations` | number | has at least N citations (markdown links, `wiki/…` refs, `[1]` footnotes) |
| `tool_called` | tool name | the agent called that tool during the rollout |
| `tool_not_called` | tool name | the agent did NOT call that tool |
Rule judges are the right place to start. They're free, deterministic, and they
force you to say concretely what a good answer looks like. (`judge.kind` can also
be `"llm"` with a rubric, or `"qrels"` for retrieval tasks — see the
[reference guide](../guides/skillopt.md) once you outgrow rules.)
### The one gotcha: how many tasks you need
SkillOpt splits your benchmark three ways — **train** (propose edits against),
**sel** (the held-out gate that decides accept/reject), and **test** (final
score). The sel slice must have **at least 5 tasks** or the run refuses, so noise
can't masquerade as improvement.
The default split is `4:1:5`, which means sel is 1/10th of your tasks — so the
default needs **~50 tasks** before it'll run. That's too many for a first
benchmark, which is why every command below passes `--split 1:1:1`: with the
15-task starter that's a clean **5 train / 5 sel / 5 test**, and sel hits the
floor exactly.
```bash
# 15 tasks + --split 1:1:1 → 5 train / 5 sel / 5 test
gbrain skillopt meeting-prep --split 1:1:1
```
If you ever see `D_sel has N task(s) after split (need >=5)`, you either added
fewer than 15 tasks or used a split whose middle number is too small a share.
`--split 1:1:1` on 15+ tasks is the simplest thing that works.
> When you swap in your own tasks (Step 6), keep at least 15 and cover the boring
> middle, not just the edge cases. The benchmark IS your definition of quality;
> a thin benchmark optimizes for a thin definition.
## Step 2: Preview the cost (dry run)
Before spending anything, see what the run will cost:
```bash
gbrain skillopt meeting-prep --split 1:1:1 --dry-run
```
This makes **zero LLM calls** — it just prints the plan and the cost estimate.
A ~15-task benchmark with defaults runs around $0.70$1.00. The preflight refuses
to start a real run whose estimate exceeds `--max-cost-usd` (default $5.00), so
you can't get surprise-billed mid-run.
> `--dry-run` exits with code **2** ("aborted"). That's the convention for "did
> not run the optimization," not a failure. The cost line is what you came for.
## Step 3: Run it for real
```bash
gbrain skillopt meeting-prep --split 1:1:1
```
You'll watch it work: a baseline eval to set the bar, then per-step forward passes
(run the skill), backward passes (propose edits), and a validation gate that
runs each sel task's judge 3 times and takes the median — accepting only if the
median beats the current best by more than 0.05.
When it finishes, the last lines tell you everything:
```
[skillopt] Outcome: accepted
[skillopt] Best sel-score: 0.840
[skillopt] Final cost: $0.71
[skillopt] SKILL.md rewritten with 6 optimization steps.
```
### Reading the outcome
| Outcome | Exit code | What it means | What to do |
|---|---|---|---|
| `accepted` | 0 | A candidate beat the baseline. SKILL.md was rewritten (or a proposed file written — see Step 5). | Review the diff, keep it. |
| `no_improvement` | 1 | Nothing cleared the gate. Your skill is already good, or the benchmark can't tell good from bad. | Strengthen the benchmark (Step 6) or stop. |
| `aborted` | 2 | A gate stopped it: dirty working tree, over budget, `D_sel < 5`, or `--dry-run`. | Read the message — it names the gate. |
`no_improvement` is not a failure. It's the gate doing its job: it would rather
keep your known-good skill than accept a change it can't prove is better.
## Step 4: See what changed
The optimizer leaves a full audit trail under the skill:
```bash
ls skills/meeting-prep/skillopt/
```
```
best.md ← the current winning version (== SKILL.md when accepted)
versions/
v0001_e1_s1.md ← every step's candidate, so you can diff any of them
v0002_e1_s2.md
...
history.json ← append-only record of every accept/reject + scores
rejected.json ← edits that were tried and didn't help (so it won't retry them)
```
The actual change to your skill is a normal git diff:
```bash
git diff skills/meeting-prep/SKILL.md
```
Run-level events (cost, model, scores per run) also land in the rotating audit
log at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl`.
## Step 5: Accept or reject — and the bundled-skill rule
**For a skill you own** (your own `skills/` dir): an `accepted` run rewrites
`SKILL.md` in place. It's already a git diff — review it, then `git commit` to
keep it or `git checkout` to throw it away. Nothing is committed for you.
**For a skill that ships with gbrain** (anything under the gbrain repo's own
`skills/`): SkillOpt refuses to overwrite it by default and writes the winner to
`skills/<name>/skillopt/best.md` instead, so an optimization pass can never
silently mutate a skill other people depend on. Two ways to handle that:
```bash
# See the proposed improvement without touching SKILL.md (works for ANY skill):
gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
# → writes skills/meeting-prep/skillopt/best.md, prints its path. Copy what you want.
# Actually rewrite a bundled skill (explicit opt-in):
gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled
```
Rule of thumb: `--no-mutate` when you want to read the diff before trusting it;
`--allow-mutate-bundled` only when you intend to commit a change to a shared skill.
## Step 6: Iterate
The loop that actually makes skills better:
1. Run it. If `no_improvement`, the benchmark probably can't distinguish good
from bad yet.
2. Add tasks that capture what you wish the skill did differently. Saw the agent
skip citations? Add `{"op":"min_citations","arg":2}`. Saw it ramble? Tighten
`max_chars`.
3. Re-run. A sharper benchmark gives the optimizer a real gradient to climb.
4. When a run lands `accepted`, read the diff, commit it, and bank the win.
The skill you ship gets better every time the benchmark gets sharper. That's the
whole game: you're not editing prose, you're improving the definition of done and
letting the optimizer chase it.
## What you built
You wrote a benchmark that encodes what "good" means for one skill, previewed the
cost, ran the optimizer, and either accepted a measurably better skill or learned
your benchmark needs sharpening. Same loop scales to every skill you own — and
`gbrain skillopt --all` runs it across every skill that has a benchmark, under a
brain-wide cost cap.
## Where to go next
- **Full flag + exit-code reference, cost model, safety guards:**
[`docs/guides/skillopt.md`](../guides/skillopt.md)
- **Every flag inline:** `gbrain skillopt --help`
- **Batch + fleet + background runs** (`--all`, `--target-models`, `--background`),
**LLM and qrels judges**, **held-out test sets**, and **resume after a crash**
(`--resume <run-id>`): all in the reference guide above.
- **Generate a starter benchmark from the SKILL.md** (the recommended way to start):
`gbrain skillopt <name> --bootstrap-from-skill` → review + strengthen the judges →
delete the sentinel → `--bootstrap-reviewed --split 1:1:1`. Tune the count with
`--bootstrap-tasks N` (max 50).
- **Bootstrap from existing routing fixtures** instead: `gbrain skillopt <name>
--bootstrap-from-routing` (routing tasks test dispatch, not quality — tighten them).
-257
View File
@@ -1,257 +0,0 @@
# Tutorial: Set up your personal AI agent + brain from zero
By the end of this tutorial you'll have your own AI agent running on a server you control, talking to you over Telegram, with a brain that remembers everything you tell it. About two hours end-to-end, $100 to $150 a month sustained.
This is the install I'd run if I were setting up the whole stack from scratch today. I documented it live during a setup session with a collaborator (we used Granola to capture the screen because "this is already too complicated for an archetypical person"). The tutorial is the cleaned-up version of that session.
> "This is the Apple I, we're just soldering breadboards over here."
If you only want the **brain layer** (no agent, no Telegram, just gbrain as memory for an MCP client you already use), skip to the [CLI standalone install](../INSTALL.md#2-cli-standalone) in INSTALL.md. If you want the whole agent **shared with a team**, read the [company brain tutorial](company-brain.md) instead. This tutorial is the solo, full-stack, talk-to-it-on-Telegram path.
---
## What you're building
A personal AI agent with four pieces:
- **A brain** (git repo). Your knowledge base, constantly ingesting and growing.
- **A harness** (OpenClaw via AlphaClaw). The runtime that gives the LLM tools, memory, and integrations.
- **A chat interface** (Telegram). How you talk to it.
- **Skills** (60+ installed via GBrain). Reusable capabilities the agent can invoke.
Architecture:
```
Telegram → AlphaClaw (harness) → OpenClaw (agent) → GBrain (knowledge/skills) → Supabase (embeddings/search)
```
Git repo is the system of record. The whole thing is multiplayer by default: any agent that hooks into the repo works. Conflicts resolve through git.
---
## Prerequisites
| Requirement | Why |
|---|---|
| GitHub account (org or personal) | For the two repos that store the agent + brain |
| Render account | For hosting the agent runtime |
| Telegram account | For talking to your agent |
| API keys: OpenAI, Anthropic at minimum | Embeddings + the Claude model |
| About $100 to $150 a month | Render Pro + Supabase + API usage |
---
## Step 1: Create two GitHub repos
You need two repos, not one.
1. **Workspace repo.** Agent configuration, skills, memory, crons. Example name: `your-org/myagent`. Private.
2. **Brain repo.** Knowledge base, people pages, meeting notes, all the content the agent reads and writes. Example name: `your-org/myagent-brain`. Private.
```
GitHub → New Repository → your-org/myagent (workspace)
GitHub → New Repository → your-org/myagent-brain (brain)
```
Both repos start empty. GBrain will populate the brain repo with its default structure on first install.
---
## Step 2: Generate a fine-grained Personal Access Token
GitHub → Settings → Developer Settings → Personal Access Tokens → Fine-grained tokens.
- **Name:** `myagent-token`
- **Expiration:** 1 year (or no expiration if available)
- **Repository access:** select both repos only
- **Permissions:** Read AND Write access to both repos (Contents, Metadata, Pull requests)
GitHub's fine-grained PAT UI is painful. You may need to reload the page after creating repos before they appear in the selector. This is the worst part of the whole setup. Push through.
Save this token. You'll need it for the AlphaClaw setup.
---
## Step 3: Create a Telegram bot
1. Open Telegram, message [@BotFather](https://t.me/BotFather)
2. Send `/newbot`
3. Name your bot (whatever you want)
4. Get the bot token
5. Save it. You'll need it for the AlphaClaw setup.
---
## Step 4: Deploy via AlphaClaw on Render
AlphaClaw is the setup harness that manages OpenClaw deployment.
1. Go to [alphaclaw.com](https://alphaclaw.com)
2. Enter your **workspace repo** (not the brain repo): `your-org/myagent`
3. Select "Use existing" if the repo already exists
4. Enter your GitHub PAT from Step 2
5. Enter your Telegram bot token from Step 3
6. Deploy
Render will build a Docker container with the harness. First deploy takes about 5 minutes.
**Memory matters.** If the instance runs out of memory during install, upgrade to Render Pro. The base tier is too small for GBrain + OpenClaw together. My production instance runs 48 cores and 64GB RAM (about $1,500 a month) but that's overkill for a new setup. Pro tier ($85 a month) is the minimum viable.
---
## Step 5: Add provider API keys
In the AlphaClaw UI (Providers tab):
- **OpenAI API Key.** Required for embeddings if you use the OpenAI provider.
- **Anthropic API Key.** Required for Claude (the main model the agent talks through).
- **Perplexity API Key.** Optional, for web search.
- **Voyage API Key.** Optional, alternative to OpenAI for embeddings.
- **ZeroEntropy API Key.** Recommended. GBrain ships with ZeroEntropy as the default embedder + reranker because it's about 2× faster than OpenAI and about 2.6× cheaper.
You can use the same keys across multiple agents.
---
## Step 6: Install GBrain
Once OpenClaw is running:
```bash
gbrain install
```
This installs:
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
---
## Step 7: Set up Supabase (embeddings and search)
GBrain uses Supabase for vector embeddings and full-text search at scale. There are three setup gotchas I hit the hard way. Walk through them in this order.
### 7a. Create the project and turn on pgvector
1. Create a Supabase project at [supabase.com](https://supabase.com). Pick a region close to where your Render host runs.
2. In the Supabase dashboard, go to **Database → Extensions**.
3. Find `vector` (the pgvector extension) and toggle it on.
Skip this and every embed write fails with "type vector does not exist" the moment GBrain tries to create its schema. pgvector is what stores the embeddings; the schema migrations refuse to run without it. Five seconds in the UI; an hour of debugging if you forget.
### 7b. Get the CONNECTION POOLER connection string, not the direct one
In **Project Settings → Database → Connection string**, Supabase shows you two options. They look almost identical. Use the right one.
- **Direct connection** (port 5432). Talks straight to the Postgres instance. IPv6-only. Will fail if your Render host doesn't have IPv6 outbound (most don't by default).
- **Connection pooler** (port 6543, hostname starts with `aws-0-...pooler.supabase.com`). Talks through Supabase's pgbouncer. Works over IPv4. Survives connection storms from parallel workers.
You want the **connection pooler** string. Format looks like:
```
postgresql://postgres.YOUR-PROJECT:YOUR-PASSWORD@aws-0-us-west-1.pooler.supabase.com:6543/postgres
```
Configure it via:
```bash
gbrain config set database_url "postgresql://postgres.YOUR-PROJECT:YOUR-PASSWORD@aws-0-us-west-1.pooler.supabase.com:6543/postgres"
```
### 7c. Buy the IPv4 add-on if your host is IPv4-only
Even with the pooler, some Supabase regions and some Render plans hit IPv6 resolution snags. If your `gbrain doctor` shows connection failures and the error mentions "network unreachable" or hangs forever on connect, you need Supabase's **IPv4 add-on**.
In the Supabase dashboard, **Project Settings → Add-ons → IPv4 address**. About $4 a month. Toggle on, wait a minute, retry the connection. This bit me on multiple installs before I learned to just buy it up front.
### 7d. Verify the connection
```bash
gbrain doctor
```
Green checks on schema, connectivity, pgvector extension, embedding provider. If any of those are yellow, the message will tell you which gotcha you hit (and which of 7a / 7b / 7c to revisit).
### Operating note
Supabase is usually the scaling bottleneck, not CPU or LLM calls. If you're doing heavy ingestion (emails, calendar, Slack streaming in), upgrade from small to large DB instance early. Don't wait for the small instance to choke; the symptoms (silent failed inserts, sync timeouts, embedding backfill stalls) all look like different bugs but are the same bug.
---
## Step 8: Verify and chat
1. Open Telegram
2. Message your bot
3. It should respond using OpenClaw + GBrain
Send a test message. If it responds with context-awareness and can search the brain, you're live.
---
## Architecture notes
### Git as system of record
The brain repo IS the brain. Any agent that can read and write to the git repo can participate. This makes the architecture inherently multiplayer: multiple agents can share a brain, work on different parts, and resolve conflicts through git.
### Thin client vs fat client
- **Fat client** (my production setup). OpenClaw + AlphaClaw + GBrain + 200 crons + email processing + Slack + calendar. About $1,500 a month. Processes everything in real time.
- **Thin client** (what this tutorial builds). OpenClaw + GBrain + Telegram. About $85 a month. Chat-driven, on-demand.
The goal for GBrain is to make the thin client as awesome as the fat client. Most users will start thin and grow.
### MCP server
GBrain exposes a Model Context Protocol server that enables inter-agent communication and integration with external systems. This is how you add read and write access to your product's API, databases, or other services.
### Brain sharing
Brains share through git. My main agent can populate another agent's brain by pushing content to its repo. The MCP layer enables cross-agent brain queries. Just push to the git repo and the other agent picks it up on next sync.
---
## What this costs
| Component | Monthly cost |
|-----------|-------------|
| Render Pro (minimum viable) | about $85 |
| Supabase (small) | free to $25 |
| OpenAI API (embeddings) | $5 to $20 (much less if you use ZeroEntropy as the default) |
| Anthropic API (Claude) | $50 to $500 (usage dependent) |
| **Total minimum** | **about $100 to $150 a month** |
My production setup is about $10,000 a month, but that's 10 instances, 200 crons, processing email and Slack and calendar in real time, running sub-agents. Not what you need on day one.
> "Next year it's not going to cost $10,000 a month. It'll cost $1,000 a month. And then the year after that, it'll be $100 a month, and then everyone will have it."
---
## Common issues
1. **Render runs out of memory during install.** Upgrade to Pro tier.
2. **GitHub PAT can't see the repos.** Reload the page after creating repos. Make sure the fine-grained token has the correct repo selection.
3. **Telegram bot doesn't respond.** Check the bot token in AlphaClaw. Make sure the Render instance is actually running.
4. **Supabase bottleneck on heavy ingestion.** Upgrade the DB instance size before the small one chokes.
5. **GBrain.io provisioning fails.** The hosted instance may need Pro tier. Check the machine allocation in the AlphaClaw UI.
---
## What you built
You now have a personal AI agent running on Render, talking to you on Telegram, with a brain that ingests and remembers everything you tell it. Every conversation gets indexed, every new entity (person, company, deal, concept) gets its own page, the overnight enrichment daemon dedupes and consolidates while you sleep. You wake up with a smarter agent than the one you went to bed with.
Where to go next:
- **Wire ingestion** from external systems. Email, calendar, voice calls, tweets, Slack. The skills are already installed; you just configure the credentials. See [`docs/integrations/`](../integrations/) for per-source recipes.
- **Connect your existing AI client** (Claude Code, Cursor, Claude Desktop) to the same brain. See [`docs/mcp/`](../mcp/) for per-client setup.
- **Set up the dream cycle** properly. The autopilot daemon runs overnight enrichment by default but you can tune what it does. See [`docs/architecture/`](../architecture/) for the full cycle reference.
- **Add a teammate to your brain**, or stand the whole thing up as a company brain. See the [company brain tutorial](company-brain.md) for the multi-user walkthrough.
Questions, gotchas, or wins worth sharing? Open an issue at [github.com/garrytan/gbrain](https://github.com/garrytan/gbrain/issues).
-329
View File
@@ -1,329 +0,0 @@
# v0.38.0.0 Smoke Test Report
> **Editor's Note (v0.39.3.0):** This report was contributed verbatim from
> PR #1299 (`garrytan-agents`). Two findings were re-diagnosed during the
> v0.39.3.0 wave; see `CHANGELOG.md` for corrections:
>
> - **BUG-2 location:** the empty-body crash site is the `else` branch at
> `src/commands/serve-http.ts:1594-1597` (`Buffer.from(JSON.stringify(req.body), 'utf8')`
> throws when `req.body === undefined` because `JSON.stringify(undefined) === undefined`),
> not line 1508 as originally reported. The empty-Buffer guard at `:1600`
> correctly fires for empty Buffers but never reaches the undefined case.
>
> - **WARN-5 root cause:** `gbrain capture --help` is minimal because
> `capture` is missing from the `CLI_ONLY_SELF_HELP` set at `src/cli.ts:34-53`.
> The detailed `HELP` constant at `src/commands/capture.ts:90-113` is
> correct and comprehensive but unreachable; the dispatcher's generic
> short-circuit at `:95` fires `printCliOnlyHelp(command)` first.
> `brainstorm` and `lsd` are in the self-help set, which is why their help works.
>
> All 2 bugs and 10 warnings are addressed in v0.39.3.0. The report below
> is preserved as the historical record of what production looked like on
> 2026-05-22.
Production smoke test of the v0.38.0.0 ingestion cathedral release on a live
Postgres-backed server (Supabase, pgvector, PgBouncer transaction mode).
**Test date:** 2026-05-22
**Server:** MCP HTTP server v0.38.0.0 on port 3131, Postgres engine
**Prior version:** v0.37.9.0
---
## Summary
| Category | Pass | Warn | Fail |
|----------|------|------|------|
| `gbrain capture` — basic | 7 | 0 | 0 |
| `gbrain capture` — edge cases | 3 | 3 | 1 |
| Dedup behavior | 0 | 2 | 0 |
| POST /ingest webhook | 5 | 1 | 1 |
| Provenance columns | 2 | 1 | 0 |
| `brainstorm` / `lsd` | 2 | 1 | 0 |
| CLI help & discoverability | 1 | 2 | 0 |
| MCP server health | 1 | 0 | 0 |
| **Total** | **21** | **10** | **2** |
---
## 🐛 Bugs (2)
### BUG-1: `capture --file` doubles frontmatter on files with existing frontmatter
**Severity:** Medium
**Repro:**
```bash
cat > /tmp/test.md << 'EOF'
---
title: Pre-existing Title
tags: [test, frontmatter]
---
# Pre-existing content
This file already has frontmatter.
EOF
gbrain capture --file /tmp/test.md
gbrain get inbox/2026-05-22-XXXXXX
```
**Observed:** The page on disk gets TWO frontmatter blocks. The outer block
has `title: '---'` (it parsed the inner `---` delimiter as the title), and
the original frontmatter is preserved verbatim inside the body:
```yaml
---
type: note
title: '---'
captured_at: '2026-05-22T16:06:11.334Z'
captured_via: capture-cli
ingested_via: put_page
ingested_at: '2026-05-22T16:06:13.038Z'
source_kind: put_page
---
---
title: Pre-existing Title
tags: [test, frontmatter]
---
# Pre-existing content
```
**Expected:** `buildContent` should detect existing frontmatter (the commit
message says it has a "looks like markdown" heuristic for first-line heading
or frontmatter delimiter) and not double-wrap. The inner frontmatter fields
should merge with capture's fields.
**Location:** `src/commands/brainstorm.ts``buildContent` function (shared
with capture).
---
### BUG-2: POST /ingest crashes with unhandled TypeError on empty body
**Severity:** Medium
**Repro:**
```bash
# With valid bearer token:
curl -X POST localhost:3131/ingest \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: text/plain"
```
**Observed:** Returns a 500 HTML error page with stack trace:
```
TypeError: The first argument must be of type string, Buffer, ArrayBuffer,
Array, or Array-like Object. Received undefined
at serve-http.ts:1508:23
```
**Expected:** The route already has an `empty_body` check (documented in the
code and tested in the E2E suite), but the body-parser middleware for
`/ingest` uses `express.raw()` which returns `undefined` for an empty POST
(no Content-Length, no body). The `empty_body` guard fires AFTER the
`computeContentHash(body)` call, which crashes on `undefined`.
**Fix:** Move the null/undefined/empty-buffer check before the content-hash
computation, or add a guard at the top of the route handler:
```typescript
if (!req.body || req.body.length === 0) {
return res.status(400).json({ error: 'empty_body', message: '...' });
}
```
---
## ⚠️ Warnings (10)
### WARN-1: Dedup does not actually deduplicate identical captures
**Observed:** Capturing identical text twice produces the same slug but
different `content_hash` values:
```
Run 1: slug=inbox/2026-05-22-3d5b671a, hash=1faf3166...
Run 2: slug=inbox/2026-05-22-3d5b671a, hash=f6ee8098...
```
**Root cause:** The content hash includes the full serialized page with
frontmatter, and `captured_at` changes between runs (it's timestamped).
The slug is deterministic (derived from content text), so it correctly
maps to the same page — but the hash changes every time.
**Impact:** The "24h content-hash LRU dedup" in the daemon layer won't
catch duplicate captures because the hash differs. The `put_page`
upsert catches it at the slug level (overwrites), so no duplicate pages
are created — but the content_hash-based dedup layer is effectively
bypassed for CLI captures.
**Suggestion:** Compute content_hash from the user's input text before
adding frontmatter, or exclude `captured_at` from the hash computation.
---
### WARN-2: Same text with different `--type` flags overwrites the previous capture
Same slug is generated for identical text regardless of `--type`. The
second capture with `--type observation` silently overwrites the first
with `--type idea`. This is correct behavior (slug = content hash), but
may surprise users who expect type changes to produce distinct pages.
---
### WARN-3: `--source` flag crashes with raw FK violation
```
gbrain capture: put_page failed: insert or update on table "pages" violates
foreign key constraint "pages_source_id_fk"
```
The error message exposes a raw Postgres FK violation. Users have no way
to know what to do. The capture command should catch this error and print
a human-friendly message:
```
Error: source 'my-source' is not registered. Register it first:
gbrain sources add my-source --path /path/to/source
```
---
### WARN-4: `facts:absorb` connection error after every capture
Every capture logs:
```
[facts:absorb] failed to log gateway_error for inbox/...: No database
connection: connect() has not been called. Fix: Run gbrain init --supabase
```
Non-fatal (exit 0), but noisy. The facts subsystem tries to open a
separate connection that the CLI capture path doesn't initialize.
---
### WARN-5: `capture --help` is minimal — doesn't show flags
```
$ gbrain capture --help
Usage: gbrain capture
gbrain capture - run gbrain --help for the full command list.
```
Compare with `brainstorm --help` which has a full Options section,
examples, and cost info. The capture help should document `--type`,
`--file`, `--source`, `--json`, `--stdin`, `--slug`, etc.
---
### WARN-6: `capture`, `brainstorm`, `lsd` missing from `gbrain --help`
None of these three commands appear in the main help text. They work when
invoked directly, but users can't discover them from the help output.
`gbrain --help` lists every other command (get, put, search, etc.) but
the v0.37/v0.38 commands are absent.
---
### WARN-7: Binary file capture succeeds silently
```bash
head -c 256 /dev/urandom > /tmp/binary.bin
gbrain capture --file /tmp/binary.bin
```
This succeeds, creating a page with binary garbage as content. Should
either reject non-text files or at minimum warn.
---
### WARN-8: Provenance columns not populated by `capture`
The capture CLI reports `"source_kind": "capture-cli"` in its JSON output,
but the actual database row has:
```json
{ "source_id": "default", "source_uri": null, "source_kind": null }
```
The provenance fields from capture's output don't round-trip into the
`put_page` call that persists the page.
---
### WARN-9: Admin register-client ignores scope parameter
Registering a client via `/admin/api/register-client` with
`"scope": "read write"` creates a client with scope `"read"` only.
The admin endpoint appears to ignore or default the scope field. Required
manual DB update to get `write` scope for webhook testing.
---
### WARN-10: `brainstorm` / `lsd` may hang or timeout on PgBouncer
In transaction-mode PgBouncer environments, `brainstorm` consistently
times out after the cost estimate with `canceling statement due to
statement timeout`. The hybrid search + domain-bank phase appears to
hit PgBouncer's statement timeout. This may be environmental, but the
command should handle the timeout gracefully and report what failed
rather than silently producing no output.
---
## ✅ What Works Well
- **`gbrain capture "text"`** — works perfectly for the simple case.
Frontmatter stamps, JSON output, disk write, immediate searchability.
- **`gbrain capture --file`** — works for plain text and simple markdown
(frontmatter doubling bug only affects files with existing frontmatter).
- **`gbrain capture ""`** and no-args — clean error messages:
`"provide content positionally, --file PATH, or --stdin"`.
- **`gbrain capture --file /nonexistent`** — clean ENOENT error.
- **Unicode/emoji**`gbrain capture "测试 🧠🔥 émojis"` works perfectly.
- **Long text** — 2500+ character capture works, correctly produces 2 chunks.
- **POST /ingest auth gate** — properly rejects missing auth (401), invalid
tokens (401), and insufficient scope (403).
- **POST /ingest content-type validation** — properly rejects image/png and
application/pdf with a helpful error pointing to skillpack processors.
- **POST /ingest accepted types** — text/plain, text/markdown, text/html,
application/json all accepted and queued.
- **X-Gbrain-Source-Id header** — custom source ID flows through correctly.
- **MCP health endpoint** — returns clean JSON with version and engine.
- **OAuth client_credentials flow** — works correctly once the client has
proper scope.
- **Content-hash dedup on webhook** — verified via response `content_hash`.
- **Migration v81** — provenance columns added cleanly, nullable, no
disruption to existing pages. 290K+ existing pages unaffected.
- **`brainstorm --help` / `lsd --help`** — excellent help text with
examples, cost estimates, and cross-references.
- **Soft delete + recovery** — cleanup via `gbrain delete` works with
72h recovery window.
- **`--type` flag** — `note`, `idea`, `observation` all work correctly.
---
## 💡 Suggestions
1. **`capture --help` parity** — give it the same quality help text as
`brainstorm --help`. Document every flag.
2. **Main help completeness** — add `capture`, `brainstorm`, and `lsd` to
the `gbrain --help` command listing. These are user-facing features that
can't be discovered.
3. **Content-hash stability** — consider computing the dedup hash from
user input text only (before frontmatter injection) so identical
captures are properly deduped at the daemon layer.
4. **Provenance write-through**`capture` already knows it's
`source_kind: "capture-cli"`. Pass this through to `put_page` so the
DB columns are populated.
5. **Binary file guard** — reject or warn on non-text input in
`capture --file`. Check file content or extension before proceeding.
6. **`--source` error UX** — catch the FK violation and print a
human-friendly hint about registering sources.
-177
View File
@@ -1,177 +0,0 @@
# What schemas unlock
Most note-taking apps treat every page the same. You write something, it goes in a pile, you search the pile with text matching. Tags help, but tags are flat. After a few thousand pages, the pile gets noisy and the search gets stupid.
Schemas are how gbrain stops being a pile of notes and becomes something with structure. A schema declares what KINDS of things live in your brain (`person`, `company`, `meeting`, `researcher`, `case`, `lab-result`), what they link to (`attended`, `authored`, `prescribed-by`), what facts the system should extract automatically (`mrr=50000`, `damages=5000000`), and which types route through expert search vs general search.
The default schema (`gbrain-base`) ships with 22 page types covering the universal shapes — people, companies, meetings, notes, daily, calendar events. That's enough to start. But your brain is yours, and your brain's shape is not the default shape. A research brain needs `researcher` and `paper` as first-class types. A founder brain needs `lead`, `investor`, `portco`, `deal-stage`. A lawyer brain needs `case`, `motion`, `deposition`, `precedent`. Same engine, totally different shape.
v0.40.7.0 made it possible for AGENTS to author that shape for you. Not just "the user manually edits YAML in `~/.gbrain/schema-packs/mine/pack.yaml`" but "your agent sees the corpus, proposes a type, asks for approval, applies it atomically with a full audit trail, then backfills 4000 existing pages with one chunked SQL command." That's the new thing.
This doc is the WHY. The [tutorial](schema-author-tutorial.md) is the HOW.
## Killer use cases
### 1. The 4000 invisible pages
You have 4000 markdown files under `meetings/` going back two years. The default schema doesn't have a `meeting` type, so all 4000 are typed `note` (the catchall). When you run:
```bash
gbrain whoknows "Q3 roadmap discussion"
```
You get the top 10 text matches, ranked by raw relevance. The brain has no idea these are meetings. It can't route to attendees. It can't pull dates. It can't surface "this conversation came up again with the same people three weeks later."
Add a `meeting` type:
```bash
gbrain schema add-type meeting --primitive temporal --prefix meetings/ --extractable
gbrain schema sync --apply
```
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
### 2. The founder ops brain
You're a founder or investor with ~500 markdown files mixing leads, portfolio companies, deal notes, intros, and follow-ups. You've been writing freely; you have no system. Your queries are all "wait, who introduced me to that fintech founder again?" and you scroll Notion for 20 minutes.
Add the founder shape:
```bash
gbrain schema fork gbrain-base mine
gbrain schema use mine
# Types
gbrain schema add-type lead --primitive entity --prefix people/leads/ --expert
gbrain schema add-type investor --primitive entity --prefix people/investors/ --expert --extractable
gbrain schema add-type portco --primitive entity --prefix companies/portco/ --expert --extractable
gbrain schema add-type deal --primitive entity --prefix companies/deals/ --extractable
# Link verbs
gbrain schema add-link-type invested-in --page-type investor --target-type portco
gbrain schema add-link-type intro-from --page-type lead --target-type lead
gbrain schema add-link-type passed-on --page-type investor --target-type deal
gbrain schema add-link-type led-by --page-type deal --target-type investor
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
### 3. The research brain
Replace "founder" with "PhD student" and the same pattern applies with different types: `researcher`, `paper`, `lab`, `grant`, `dataset` + `authored`, `cites`, `funded-by`, `uses-dataset`.
```bash
gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable
gbrain schema add-link-type authored --page-type researcher --target-type paper
gbrain schema add-link-type cites --page-type paper --target-type paper
gbrain schema add-link-type uses --page-type paper --target-type dataset
```
Suddenly "show me papers that cite this work AND use the same dataset" is a `gbrain graph-query` traversal, not 30 minutes in Google Scholar. The fact extraction picks up `arxiv_id=2402.04253`, `cited_by_count=140`, `published_date=2026-02-15` automatically. Your reading-list-as-markdown turns into a queryable research graph that knows who works on what and what's connected to what.
### 4. The legal brain (or any domain where claims have numbers)
Lawyers, medical providers, accountants, anyone working in a domain where the meaning of a number depends on its type. A "judgment of $5M" against a "$2M case strategy threshold" is a comparison the brain can do — but only if both numbers are typed.
```bash
gbrain schema add-type case --primitive entity --prefix legal/cases/ --extractable --expert
gbrain schema add-type motion --primitive annotation --prefix legal/motions/ --extractable
gbrain schema add-type deposition --primitive annotation --prefix legal/depositions/ --extractable
gbrain schema add-link-type filed-in --page-type motion --target-type case
gbrain schema add-link-type cites --page-type motion --target-type precedent
```
Now `## Facts` fences in your case notes can carry typed claims (`damages=5000000`, `filed_date=2026-05-23`, `judge=jane-doe`) that gbrain stores as first-class columns. `gbrain eval trajectory legal/cases/acme-v-widget` prints the case history with regressions flagged. `gbrain founder scorecard` (renamed for legal: roll up plaintiff success rate, average damages, settlement-vs-trial ratio) gives you a structured view of how your practice is performing.
This isn't possible without typed page kinds. You can write the same prose in any note-taking app. Only gbrain treats the numbers as comparable across pages of the same type.
### 5. The team brain
`gbrain mounts add` lets you stack additional brains alongside your personal one. Each mounted brain has its OWN schema pack. The eng team's brain has `incident`, `runbook`, `service`, `oncall-rotation`. The design team's brain has `component`, `experiment`, `ab-test`, `figma-link`. The legal team's brain has cases and depositions.
When you query, the schema pack governs how each source's content is routed. An eng query against the mounted eng brain knows that `incidents/2026-05-23-db-outage.md` is an `incident` page with `severity=p0`, `mttr=47min`, `on_call=alice-example` — extractable typed facts. Your personal query against the same brain still works, but the routing is sharper because the eng team has invested in their ontology.
The schema is the team's tribal knowledge made explicit. Two engineers on different teams searching the same brain get DIFFERENT routing because their personal packs declare different expert types.
### 6. The "agent co-curates your ontology" pattern (the new thing)
This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for.
Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes:
> You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page.
You approve once. The agent calls `schema_apply_mutations` over MCP with a batch:
```json
{
"pack": "mine",
"mutations": [
{"op": "add_type", "name": "yc-w24-company", "primitive": "entity", "prefix": "companies/yc-w24/", "extractable": true, "expert_routing": true},
{"op": "add_alias", "type": "yc-w24-company", "alias": "company"}
]
}
```
All inside ONE `withPackLock` scope, atomic, audited (the agent's `client_id` captured in the audit log as `actor: mcp:<clientId8>`). Cache invalidated cross-process. Sync backfills the 47 pages. The brain learned a new category of thing without you having to think about it.
The next time you query "YC W24 companies in fintech", the brain routes through the new type. Six months later when you forget the pattern entirely, the agent reminds you it's there and offers to consolidate it with the W25 batch.
The brain learns. The agent is the curator. You approve, the agent does the work.
### 7. The before-vs-after benchmark
If you want to FEEL the difference without buying the pitch:
Pick a real corpus you have. Run `gbrain whoknows` on a topic that should match. Note the top-3 results.
Then run `gbrain schema review-orphans --limit 50 --json` and look at the untyped pages. If 10+ of them share an obvious prefix that should be a real type, add the type + sync.
Re-run the same `whoknows` query. Top-3 should shift, because the new type is now routing through expert ranking instead of being lumped into the catchall. The numerical delta IS the win. You can run a tutorial in 5 minutes; this experiment proves it matters on your actual content.
## Why this matters
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees.
## What changed in v0.40.7.0 specifically
v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pack and edit `pack.yaml` by hand. What you couldn't do was let an agent author it safely — there were no atomic file locks, no audit log, no MCP exposure, no pack-aware wiring in the query path. The cathedral was built but unreachable from the outside.
v0.40.7.0 closed those gaps:
- **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race.
- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking."
- **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`.
- **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:<clientId8>`).
- **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`.
- **Cross-process invalidation** via stat-mtime TTL gate inside `loadActivePack`. Operator runs `gbrain schema add-type` from a terminal; the autopilot daemon picks up the new type within 1 second without a restart.
The cumulative effect: an agent can safely co-curate your ontology with a complete forensic trail. That's the new thing.
## Where to start
- **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end.
- **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity.
- **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types.
- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it.
- **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes.
The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app.
That's what we built. Try it on a corpus you actually have and the numbers go up.
-40
View File
@@ -1,40 +0,0 @@
{
"version": 1,
"description": "Embedding provider smoke test — verifies semantic search returns expected results for known brain content. Run after any embedding model change or migration.",
"queries": [
{
"id": "yc-labs-strategy",
"query": "YC Labs strategy and product team",
"relevant": [
"originals/yc-labs-internal-team",
"originals/harj-yc-labs-strategy-2026-05"
]
},
{
"id": "garry-tan-person",
"query": "Who is Garry Tan",
"relevant": [
"people/garry-tan"
]
},
{
"id": "gstack-project",
"query": "GStack open source AI coding framework",
"relevant": [
"projects/gstack/gstackbrain"
]
},
{
"id": "yc-carry-compensation",
"query": "GP carry and compensation structure at YC",
"relevant": [
"originals/harj-yc-labs-strategy-2026-05"
]
},
{
"id": "meeting-search",
"query": "recent office hours meeting notes",
"relevant": []
}
]
}
@@ -1,2 +0,0 @@
# Per-run output JSONLs land here; only baseline-runs/<date>-<model>.jsonl is canonical.
run-*.jsonl

Some files were not shown because too many files have changed in this diff Show More