mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
124
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Garry TanandClaude Fable 5
|
d35c9c9e44 | ||
|
|
fc310db3ea | ||
|
Masa
|
52306ed438 | ||
|
Masa
|
a2521d0a32 | ||
|
Garry Tan
|
75fae742d5 | ||
|
MasaandClaude Opus 5
|
a13ecb1f0a | ||
|
MasaandClaude Opus 5
|
e2ff128e35 | ||
|
MasaandClaude Opus 5
|
69472c24e7 | ||
|
MasaandClaude Opus 5
|
ea17f5911c | ||
|
MasaandClaude Opus 5
|
d050969da6 | ||
|
MasaandClaude Opus 5
|
7c65bcee9b | ||
|
MasaandClaude Opus 5
|
9afbf4a09a | ||
|
MasaandClaude Opus 5
|
e11e407752 | ||
|
Garry TanandClaude Fable 5
|
c5952b8714 | ||
|
Garry TanandClaude Fable 5
|
130d321d23 | ||
|
MasaandClaude Fable 5
|
485c4773ca | ||
|
MasaandClaude Fable 5
|
f8878be54e | ||
|
MasaandClaude Fable 5
|
0df56f1efa | ||
|
Ziyang Guo
|
01d9639204 | ||
|
Garry TanandClaude Fable 5
|
f15480b9d0 | ||
|
Ziyang Guo
|
0b47afbf40 | ||
|
Ziyang Guo
|
9f81e1d8b2 | ||
|
Ziyang Guo
|
86555e39bb | ||
|
zay
|
3257758492 | ||
|
Garry TanandClaude Fable 5
|
a948dfd6e2 | ||
|
|
15b9863d13 | ||
|
|
aecb33e795 | ||
|
Garry Tan
|
e5dee4fb78 | ||
|
Garry Tan
|
3c26f2eaf2 | ||
|
Garry Tan
|
615c33e5b5 | ||
|
|
40d1d4cabc | ||
|
Garry TanandClaude Fable 5
|
d610a845a8 | ||
|
Garry TanandClaude Fable 5
|
f87488ff36 | ||
|
Garry Tan
|
ddb39df23a | ||
|
Garry TanandClaude Fable 5
|
88731d8cf3 | ||
|
Garry TanandClaude Fable 5
|
4a00c31b12 | ||
|
Garry TanandClaude Fable 5
|
94662eb1e1 | ||
|
Garry TanandClaude Fable 5
|
2f65ed8da6 | ||
|
Garry TanandClaude Fable 5
|
ca1eed3e95 | ||
|
Sean Gearin
|
82fe0216ff | ||
|
Masa
|
fa313f2969 | ||
|
|
2e554500e2 | ||
|
Masa
|
630786dc65 | ||
|
Masa
|
d92f0dc86f | ||
|
|
a68379050e | ||
|
|
6d1232d5a6 | ||
|
Sina Matian
|
aff34a428a | ||
|
Garry Tan
|
f07e9dff11 | ||
|
Garry Tan
|
72ec53bb49 | ||
|
Garry Tan
|
0244104b8d | ||
|
Garry TanandTime Attakc
|
48618bd4bf | ||
|
Garry Tan
|
dcaea7fdbb | ||
|
Garry Tan
|
c6bfab582e | ||
|
Garry Tan
|
c5ac3efe9f | ||
|
Garry Tan
|
0a6b697070 | ||
|
Garry Tan
|
873587f831 | ||
|
Garry Tan
|
78391e8b64 | ||
|
Garry TanandTime Attakc
|
f08a51d9de | ||
|
Garry TanandTime Attakc
|
7cbb99ffef | ||
|
Garry TanandTime Attakc
|
aa5b9e6e2d | ||
|
Garry Tan
|
8e3699156f | ||
|
Garry Tan
|
cc1783c0e4 | ||
|
Garry TanandDiego
|
e77a15f0d8 | ||
|
Garry TanandYMYD
|
522cfb032a | ||
|
Garry Tananddialthewolff
|
273bd0e2be | ||
|
Garry Tan
|
7522b90db9 | ||
|
Garry TanandTime Attakc
|
42f810c406 | ||
|
Garry Tan
|
1ca1a14f24 | ||
|
Garry TanandDiego
|
64c191b1d7 | ||
|
Garry TanandDiego
|
d49ea83db4 | ||
|
Garry Tan
|
d71d50503d | ||
|
Garry TanandDiego
|
a228776b8f | ||
|
Garry TanandMammad M.
|
c0b28f104d | ||
|
Garry Tanandcvillarroel2
|
61ef727710 | ||
|
Garry Tan
|
532e591655 | ||
|
Garry Tan
|
3acd511b80 | ||
|
Garry Tan
|
25e4c0c3b1 | ||
|
Garry Tan
|
ba27a186ec | ||
|
Garry TanandChristoph
|
4bb313cd80 | ||
|
Garry Tanand老赵
|
f39b059ad8 | ||
|
Garry TanandKyzcreig
|
42c2d56df3 | ||
|
Garry TanandVyacheslav Zakharov
|
e3806cf46f | ||
|
Garry TanandAlexey
|
a82a83dbc3 | ||
|
Garry TanandBen Young
|
03de3246f3 | ||
|
Garry TanandAustin Wilhite
|
37ad1d2104 | ||
|
Garry Tanandzenspam
|
aeb75dc839 | ||
|
Garry TanandTime Attakc
|
298b6b01b8 | ||
|
Garry Tan
|
d6f929bfbd | ||
|
Garry TanandRyan Ayers
|
9c1a4b8fce | ||
|
Garry Tan
|
3523d8fd7e | ||
|
Garry TanandTime Attakc
|
a3000d4630 | ||
|
Garry TanandBrett
|
64ad743d98 | ||
|
Garry TanandTime Attakc
|
9ada48e600 | ||
|
Garry TanandTime Attakc
|
aa05255887 | ||
|
Garry TanandTime Attakc
|
241603aab8 | ||
|
Garry TanandTime Attakc
|
69be8bb707 | ||
|
Garry TanandTime Attakc
|
1116a95926 | ||
|
Garry TanandTime Attakc
|
c8ea38421a | ||
|
Garry TanandTime Attakc
|
163a83baa3 | ||
|
Garry TanandTime Attakc
|
f7295e3308 | ||
|
|
f84bfb57f2 | ||
|
|
ad7114f0ad | ||
|
|
23003a2163 | ||
|
|
4c0ec60275 | ||
|
Masa
|
13d95ba0ab | ||
|
Time Attakc
|
022a443e9b | ||
|
Masa
|
e7439828f1 | ||
|
Time Attakc
|
5773736c63 | ||
|
cybernaut6404
|
56454c6ba8 | ||
|
alexey-metaengage
|
63e79838b9 | ||
|
Masa
|
3062859420 | ||
|
Masa
|
addf03119d | ||
|
daragao3
|
dba0ae7b1e | ||
|
Masa
|
7376c0266e | ||
|
Sean Gearin
|
002ac8050f | ||
|
Javier Aldape
|
f75dbb4ed6 | ||
|
Masa
|
437889c0bd | ||
|
Time Attakc
|
335e470394 | ||
|
Masa
|
7e21e47c15 | ||
|
Paolo Belcastro
|
bf7d706bb4 | ||
|
Masa
|
bb69aa8b65 | ||
|
paul-0320
|
b4a9c7683d | ||
|
Time Attakc
|
3c61e25503 | ||
|
|
c6dc0adf26 |
@@ -14,3 +14,16 @@
|
||||
# through bash. `eol=lf` pins the checkout regardless of the user's
|
||||
# core.autocrlf setting.
|
||||
*.sh text eol=lf
|
||||
|
||||
# Markdown gets the same pin, for a different failure mode: the frontmatter
|
||||
# parsers anchor on LF. Under a CRLF checkout the opening fence becomes
|
||||
# "---\r\n", which an LF-only /^---\n/ (or a startsWith("---\n")) does not
|
||||
# match, so a well-formed document silently parses as having no frontmatter.
|
||||
# There is no error -- the field just comes back empty. That has surfaced as
|
||||
# blank skill descriptions, a fixer inserting its banner above the
|
||||
# frontmatter instead of below it, resolver trigger extraction dropping
|
||||
# entries, and a generated-doc freshness check reporting every line as
|
||||
# drifted. The parsers stay CR-tolerant on their own merits (gbrain reads
|
||||
# Markdown it does not own), but pinning this repo's own .md checkout to LF
|
||||
# removes the whole class for anyone working here.
|
||||
*.md text eol=lf
|
||||
|
||||
@@ -66,6 +66,20 @@ jobs:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
run: bun run test:heavy
|
||||
|
||||
# Agent-bootstrap offline Docker e2e [A7] — placeholder registration.
|
||||
# The networkless cold-machine container test (interview → render →
|
||||
# verify with a fake gh) lands with the bootstrap doors task at
|
||||
# tests/docker/bootstrap-e2e.sh. Gated on file existence so this stays
|
||||
# a visible no-op until then. It runs at heavy cadence (nightly +
|
||||
# `heavy-tests` label) by design — never in the PR shard matrix.
|
||||
- name: Bootstrap offline Docker e2e (placeholder)
|
||||
run: |
|
||||
if [ -f tests/docker/bootstrap-e2e.sh ]; then
|
||||
bash tests/docker/bootstrap-e2e.sh
|
||||
else
|
||||
echo "SKIP: tests/docker/bootstrap-e2e.sh not present yet — placeholder until the bootstrap doors task lands."
|
||||
fi
|
||||
|
||||
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
|
||||
# run. Upload those + the rss workload JSON on failure for triage
|
||||
# without re-running locally.
|
||||
@@ -87,3 +101,46 @@ jobs:
|
||||
path: heavy-artifacts/
|
||||
retention-days: 14
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Real-agent door e2e: drives the ACTUAL `claude` + `codex` binaries (no PATH
|
||||
# shims) against a real gbrain over MCP. These pay real API cost and need the
|
||||
# binaries installed + authed, which a stock GitHub runner does NOT have — so
|
||||
# both tests self-SKIP (describe.skipIf on binary/auth) and the job is a clean
|
||||
# no-op here. It exists so a self-hosted / manually-provisioned runner WITH
|
||||
# authed claude/codex (and ANTHROPIC/OPENAI creds) actually exercises the real
|
||||
# binaries. Heavy cadence only (nightly + `real-agent-e2e` label + dispatch);
|
||||
# NEVER the PR shard matrix.
|
||||
real-agent-e2e:
|
||||
name: Real-agent door e2e (skips without authed binaries)
|
||||
if: |
|
||||
github.event_name != 'pull_request' ||
|
||||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
|
||||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
|
||||
# Reference both door tests; run only the ones present (the claude door
|
||||
# may land in a sibling PR). Missing binary/auth → the file self-skips, so
|
||||
# a stock runner reports a green no-op rather than failing.
|
||||
- name: Run real-agent door tests
|
||||
run: |
|
||||
files=()
|
||||
for f in \
|
||||
test/e2e/bootstrap-real-claude.serial.test.ts \
|
||||
test/e2e/bootstrap-real-codex.serial.test.ts; do
|
||||
[ -f "$f" ] && files+=("$f")
|
||||
done
|
||||
if [ "${#files[@]}" -eq 0 ]; then
|
||||
echo "SKIP: no real-agent door test files present yet."
|
||||
exit 0
|
||||
fi
|
||||
echo "Running: ${files[*]}"
|
||||
# --timeout: real-agent turns are slow (live claude/codex); the door
|
||||
# tests self-skip without authed binaries so this is a no-op elsewhere.
|
||||
bun test --timeout=600000 "${files[@]}"
|
||||
|
||||
@@ -1,14 +1,75 @@
|
||||
name: Release
|
||||
|
||||
# Publishes a GitHub release for every VERSION bump that lands on master:
|
||||
# tag + title `v<VERSION>`, notes from that version's CHANGELOG.md entry,
|
||||
# compiled binaries attached (#3521).
|
||||
#
|
||||
# Why every bump: `gbrain check-update` resolves the latest version from the
|
||||
# VERSION file on master, but binary self-update
|
||||
# (src/core/binary-self-update.ts) downloads assets from `releases/latest`.
|
||||
# If releases lag VERSION, binary installs are told an upgrade exists that
|
||||
# self-update cannot apply. Keeping releases/latest == VERSION closes that gap.
|
||||
#
|
||||
# Idempotent: the `version` job skips build+release when a release for
|
||||
# v<VERSION> already exists WITH all expected assets. A half-published release
|
||||
# (tag exists / assets incomplete) is repaired on the next run — softprops
|
||||
# updates the existing release in place. Historical 3-segment tags are never
|
||||
# touched; a new 4-segment VERSION always mints a new tag.
|
||||
#
|
||||
# The asset names are a contract with expectedAssetName() in
|
||||
# src/core/binary-self-update.ts, pinned by test/release-workflow.test.ts.
|
||||
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
branches: [master]
|
||||
paths: [VERSION]
|
||||
workflow_dispatch: {} # manual first run / backfill of the current VERSION
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: release
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
version:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.v.outputs.version }}
|
||||
exists: ${{ steps.v.outputs.exists }}
|
||||
template_tree_hash: ${{ steps.v.outputs.template_tree_hash }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- id: v
|
||||
name: Read VERSION and check for an existing complete release
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
version="$(tr -d '[:space:]' < VERSION)"
|
||||
echo "version=$version" >> "$GITHUB_OUTPUT"
|
||||
# [C3] Record the vendored template tree's git hash alongside the
|
||||
# asset completeness check. The publish-template job embeds it in
|
||||
# the template repo's commit message, so template-repo HEAD can be
|
||||
# audited back to the exact reviewed tree in THIS repo. Empty until
|
||||
# templates/bootstrap/template-repo/ is vendored.
|
||||
tree_hash="$(git rev-parse "HEAD:templates/bootstrap/template-repo" 2>/dev/null || true)"
|
||||
echo "template_tree_hash=$tree_hash" >> "$GITHUB_OUTPUT"
|
||||
# Complete = release exists AND carries every asset the self-updater
|
||||
# can request. A partial release must NOT short-circuit, so a re-run
|
||||
# can repair it.
|
||||
assets="$(gh release view "v$version" --repo "$GITHUB_REPOSITORY" \
|
||||
--json assets --jq '[.assets[].name] | sort | join(",")' 2>/dev/null || true)"
|
||||
if [ "$assets" = "gbrain-darwin-arm64,gbrain-linux-x64" ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Release v$version already published with all assets — nothing to do."
|
||||
else
|
||||
echo "exists=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
build:
|
||||
needs: version
|
||||
if: needs.version.outputs.exists == 'false'
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
@@ -29,11 +90,19 @@ jobs:
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
# --timeout matches every scripts/ runner and covers hook budgets too
|
||||
# (bunfig.toml's timeout key is ignored by bun; hooks default to 5s).
|
||||
- run: bun test --timeout=60000
|
||||
- run: bun run verify
|
||||
# No test re-run here: the Test workflow already gated this exact SHA at
|
||||
# merge (10 shards + E2E). Re-running the whole suite serially on the
|
||||
# release runner is a flakier duplicate gate — it blocked the first
|
||||
# release on ambient-env tests (run 30698650484). The build job's gate
|
||||
# is the artifact itself: compile, then smoke-test the binary.
|
||||
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
|
||||
- name: Smoke-test the compiled binary
|
||||
run: |
|
||||
chmod +x bin/${{ matrix.artifact }}
|
||||
out="$(./bin/${{ matrix.artifact }} --version)"
|
||||
echo "binary reports: $out"
|
||||
v="$(tr -d '[:space:]' < VERSION)"
|
||||
case "$out" in *"$v"*) echo "version matches VERSION file" ;; *) echo "binary version '$out' does not contain '$v'" >&2; exit 1 ;; esac
|
||||
- name: Attest build provenance
|
||||
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
|
||||
with:
|
||||
@@ -44,16 +113,137 @@ jobs:
|
||||
path: bin/${{ matrix.artifact }}
|
||||
|
||||
release:
|
||||
needs: build
|
||||
needs: [version, build]
|
||||
if: needs.version.outputs.exists == 'false'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write # create the tag + release (scoped to this job only)
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
|
||||
with:
|
||||
path: artifacts
|
||||
- name: Extract CHANGELOG entry for release notes
|
||||
# env-bound, not inlined into the script: VERSION comes from master so
|
||||
# it isn't attacker-reachable today, but a `${{ }}` inside `run:` is
|
||||
# shell injection by construction if that ever changes.
|
||||
env:
|
||||
RELEASE_VERSION: ${{ needs.version.outputs.version }}
|
||||
run: |
|
||||
v="$RELEASE_VERSION"
|
||||
if ! bash scripts/changelog-entry.sh "$v" > /tmp/release-notes.md || ! [ -s /tmp/release-notes.md ]; then
|
||||
echo "See [CHANGELOG.md](https://github.com/${GITHUB_REPOSITORY}/blob/master/CHANGELOG.md) for v$v." > /tmp/release-notes.md
|
||||
fi
|
||||
- name: Create release
|
||||
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
|
||||
with:
|
||||
tag_name: v${{ needs.version.outputs.version }}
|
||||
name: v${{ needs.version.outputs.version }}
|
||||
target_commitish: ${{ github.sha }}
|
||||
body_path: /tmp/release-notes.md
|
||||
fail_on_unmatched_files: true
|
||||
files: |
|
||||
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
|
||||
artifacts/gbrain-linux-x64/gbrain-linux-x64
|
||||
generate_release_notes: true
|
||||
# [C1 = D6-A] `latest-stable` is the single sanctioned distribution ref:
|
||||
# the README paste block, the BOOTSTRAP_FOR_AGENTS.md fetch URL, and
|
||||
# `bun install -g github:garrytan/gbrain#latest-stable` all reference it
|
||||
# permanently, so paste blocks copied into the wild never rot and there
|
||||
# is no 404 window between VERSION landing and assets publishing. It is
|
||||
# a maintainer-controlled tag advanced ONLY here — the FINAL step of the
|
||||
# release job, after binaries + provenance attestation have fully
|
||||
# published — so a half-built release never moves it. The force-move
|
||||
# (`+`) is intentional: latest-stable tracks the newest verified release;
|
||||
# per-release history lives in the immutable v<VERSION> tags.
|
||||
# scripts/check-bootstrap-tag.sh keeps the entry docs pinned to this ref.
|
||||
#
|
||||
# If this step ALONE fails, re-advance by hand (a full re-run would skip:
|
||||
# the release already exists with all assets):
|
||||
# git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
|
||||
- name: Advance latest-stable to this release commit
|
||||
run: git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"
|
||||
|
||||
# [G7/S3#4] Publishes the rendered agent-workspace template repo (the
|
||||
# GitHub "Use this template" door) from CI ONLY — no human pushes it by
|
||||
# hand, so what adopters clone is exactly what this repo reviewed. Guarded
|
||||
# three ways:
|
||||
# 1. The release above fully published (needs: release + the exists gate).
|
||||
# 2. The vendored tree templates/bootstrap/template-repo/ exists — the
|
||||
# generator/doors work may not have landed yet; skip, never fail.
|
||||
# 3. The TEMPLATE_REPO_PAT secret is configured. Secrets are not readable
|
||||
# in job-level `if:` expressions, so the secret is bound to env (the
|
||||
# same env-indirection pattern as the release-notes step) and checked
|
||||
# by the gate step's shell.
|
||||
#
|
||||
# TEMPLATE_REPO_PAT scope: a fine-grained PAT with `contents: write` on the
|
||||
# template repository ONLY — no other repositories, no other permissions.
|
||||
# Documented in docs/RELEASING.md.
|
||||
publish-template:
|
||||
needs: [version, release]
|
||||
if: needs.version.outputs.exists == 'false'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
env:
|
||||
TEMPLATE_REPO_PAT: ${{ secrets.TEMPLATE_REPO_PAT }}
|
||||
# owner/name of the template repo; override via repository variable.
|
||||
TEMPLATE_REPO: ${{ vars.TEMPLATE_REPO || 'garrytan/gbrain-agent-template' }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- id: gate
|
||||
name: Gate on PAT + vendored template tree
|
||||
run: |
|
||||
if [ -z "$TEMPLATE_REPO_PAT" ]; then
|
||||
echo "publish=false" >> "$GITHUB_OUTPUT"
|
||||
echo "SKIP: TEMPLATE_REPO_PAT secret not configured — template-repo publishing disabled."
|
||||
elif [ ! -d templates/bootstrap/template-repo ]; then
|
||||
echo "publish=false" >> "$GITHUB_OUTPUT"
|
||||
echo "SKIP: templates/bootstrap/template-repo/ not vendored yet — nothing to publish."
|
||||
else
|
||||
echo "publish=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
run: bun install
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
name: Generate template tree and byte-diff against the vendored copy
|
||||
run: |
|
||||
bun run scripts/generate-template-repo.ts --out /tmp/template-tree
|
||||
# [A1] Publish gate: fresh generator output must equal the vendored
|
||||
# tree byte-for-byte. A mismatch means the vendored tree is stale —
|
||||
# regenerate + commit it (scripts/check-bootstrap-templates.sh runs
|
||||
# this same diff offline in `bun run verify`).
|
||||
diff -r /tmp/template-tree templates/bootstrap/template-repo
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
name: Force-push the template repo
|
||||
env:
|
||||
RELEASE_VERSION: ${{ needs.version.outputs.version }}
|
||||
TEMPLATE_TREE_HASH: ${{ needs.version.outputs.template_tree_hash }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
cd /tmp/template-tree
|
||||
git init -q -b main
|
||||
git config user.name "gbrain-release-bot"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add -A
|
||||
# History-less by design: each release force-publishes one commit
|
||||
# whose message binds (gbrain version, vendored tree hash) [C3].
|
||||
git commit -q -m "gbrain v${RELEASE_VERSION} template (tree ${TEMPLATE_TREE_HASH:-unvendored})"
|
||||
# Out-of-band credential: a one-shot GIT_ASKPASS script reads the
|
||||
# PAT from env at prompt time, so the token never rides argv (where
|
||||
# `ps`, runner traces, and error messages echoing the remote URL
|
||||
# could surface it).
|
||||
ASKPASS="$(mktemp)"
|
||||
# shellcheck disable=SC2016 # $1/$TEMPLATE_REPO_PAT are literal on
|
||||
# purpose — they must expand when /bin/sh runs the askpass script at
|
||||
# git's credential prompt, not when this outer shell writes the file.
|
||||
printf '%s\n' '#!/bin/sh' \
|
||||
'case "$1" in Username*) printf "x-access-token\n" ;; *) printf "%s\n" "$TEMPLATE_REPO_PAT" ;; esac' \
|
||||
> "$ASKPASS"
|
||||
chmod +x "$ASKPASS"
|
||||
GIT_ASKPASS="$ASKPASS" GIT_TERMINAL_PROMPT=0 \
|
||||
git push --force "https://github.com/${TEMPLATE_REPO}.git" HEAD:main
|
||||
rm -f "$ASKPASS"
|
||||
|
||||
@@ -165,6 +165,40 @@ jobs:
|
||||
- run: bun install
|
||||
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
|
||||
|
||||
brainbench:
|
||||
# BrainBench memory-conformance gate (Cathedral 2). Hermetic: in-memory
|
||||
# PGLite, zero API keys, ~15s for the full 141-fixture × 3-harness run.
|
||||
# Governance (decision 4): compares HEAD's run against MAIN's committed
|
||||
# baseline via `git show origin/master:evals/brainbench/baselines/main.json`
|
||||
# — a PR cannot rewrite the thing it's compared against. Exit 1 blocks the
|
||||
# merge until the regression is fixed or blessed (justification in the
|
||||
# updated baseline / fixture diff in corpus-bless mode).
|
||||
needs: cache-check
|
||||
if: needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10 # ~15s hermetic run; matches the per-job-timeout hardening (#2254)
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
with:
|
||||
fetch-depth: 0 # the gate needs origin/master's baseline
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bash scripts/ci-brainbench-gate.sh
|
||||
env:
|
||||
BRAINBENCH_OUT: ${{ runner.temp }}/brainbench-result.json
|
||||
- name: BrainBench delta → step summary
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f "${{ runner.temp }}/brainbench-result.json" ]; then
|
||||
bun scripts/render-brainbench-delta.ts "${{ runner.temp }}/brainbench-result.json" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
slow-entity-resolve-perf:
|
||||
# Dedicated runner for the entity-resolve perf test (~159s, single perf
|
||||
# describe with one test that builds 5000+ pages and asserts the NEW
|
||||
@@ -187,6 +221,19 @@ jobs:
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
|
||||
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
|
||||
# (20K-page corpus + ratio guard) shares this runner — same perf-job
|
||||
# shape, runs in parallel with the matrix.
|
||||
- run: bun test test/entity-card-perf.slow.test.ts --timeout=300000
|
||||
# Protocol self-certification: init a scratch brain and run the
|
||||
# conformance kit against gbrain's own stdio server. --synthesize is
|
||||
# safe here: no LLM key in CI, so it asserts the clean `unavailable`
|
||||
# protocol error instead of spending tokens.
|
||||
- name: MEMORY_VERBS conformance (self-certify, stdio)
|
||||
run: |
|
||||
export GBRAIN_HOME="$RUNNER_TEMP/gbrain-conformance"
|
||||
bun run src/cli.ts init --pglite --no-embedding --non-interactive
|
||||
bun run src/cli.ts protocol conformance --synthesize
|
||||
|
||||
test:
|
||||
# Pure matrix shard — no verify, no serial. Each shard runs its slice
|
||||
@@ -242,7 +289,7 @@ jobs:
|
||||
# entry written when tests hadn't actually passed).
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
cache-write:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
|
||||
if: success() && needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
@@ -265,7 +312,7 @@ jobs:
|
||||
# Branch protection (when configured) gates on this single job name.
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
test-status:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
@@ -278,15 +325,16 @@ jobs:
|
||||
SERIAL="${{ needs.serial-tests.result }}"
|
||||
SLOW_EVAL="${{ needs.slow-eval-longmemeval.result }}"
|
||||
SLOW_PERF="${{ needs.slow-entity-resolve-perf.result }}"
|
||||
BRAINBENCH="${{ needs.brainbench.result }}"
|
||||
TEST="${{ needs.test.result }}"
|
||||
echo "cache-check.hit=$HIT"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF test=$TEST"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF brainbench=$BRAINBENCH test=$TEST"
|
||||
if [ "$HIT" = "true" ]; then
|
||||
echo "✓ cache HIT for hash ${{ needs.cache-check.outputs.hash }} — CI green"
|
||||
exit 0
|
||||
fi
|
||||
# Cache miss: every gated job must have succeeded.
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$TEST"; do
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$BRAINBENCH" "$TEST"; do
|
||||
if [ "$r" != "success" ]; then
|
||||
echo "✗ gated job did not succeed (got $r) — CI fail"
|
||||
exit 1
|
||||
|
||||
@@ -4,8 +4,16 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom
|
||||
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
|
||||
start here.
|
||||
|
||||
> **Becoming someone's persistent personal agent** (identity + memory + private repo)?
|
||||
> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap`
|
||||
> flow — instead of the plain install below, then come back here for the operating
|
||||
> protocol. Connecting to an EXISTING remote brain from a laptop agent?
|
||||
> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP
|
||||
> table in [`README.md`](./README.md)).
|
||||
|
||||
## Install (5 min)
|
||||
|
||||
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
|
||||
1. Install gbrain via Bun (the canonical path):
|
||||
```bash
|
||||
curl -fsSL https://bun.sh/install | bash
|
||||
@@ -26,8 +34,8 @@ start here.
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step
|
||||
flow (API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
@@ -69,10 +77,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
|
||||
and `gbrain eval replay --against base.ndjson`. For public benchmark
|
||||
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
|
||||
<dataset.jsonl>` runs against an isolated in-memory PGLite
|
||||
per question — your `~/.gbrain` is never opened. Full guide:
|
||||
[`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
|
||||
- **Drive the brain to a target health score:** the one-command
|
||||
loop. `gbrain doctor --remediation-plan --json` previews what would be
|
||||
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
|
||||
walks a dependency-ordered plan (sync before extract, embed after
|
||||
@@ -81,22 +89,20 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
keys hit a `max_reachable_score` ceiling and bail with what's missing.
|
||||
Three phase handlers (synthesize / patterns / consolidate) are
|
||||
PROTECTED — only trusted local callers can submit them; MCP cannot.
|
||||
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
|
||||
and the CHANGELOG entry for v0.36.4.0.
|
||||
- **Track a founder/company over time (v0.35.7):** when an entity has
|
||||
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md).
|
||||
- **Track a founder/company over time:** when an entity has
|
||||
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
|
||||
`unit: USD`, `period: monthly` columns), run
|
||||
`gbrain eval trajectory <entity-slug>` for the chronological history
|
||||
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
|
||||
for a four-signal JSON rollup (claim_accuracy / consistency /
|
||||
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
|
||||
same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:**
|
||||
`gbrain think` now uses this substrate automatically on temporal /
|
||||
same data — read scope, visibility-filtered for remote callers.
|
||||
`gbrain think` uses this substrate automatically on temporal /
|
||||
knowledge_update intent (default ON; flip `think.trajectory_enabled=false`
|
||||
to opt out). Migration v82 added `facts.event_type` so non-metric event
|
||||
rows (`meeting`, `job_change`, `location_change`) ride through the same
|
||||
pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query
|
||||
them.
|
||||
to opt out). Non-metric event rows (`meeting`, `job_change`,
|
||||
`location_change`) ride through the same pipeline via `facts.event_type`;
|
||||
pass `kind: 'event'` or `'all'` to `find_trajectory` to query them.
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
<!-- gbrain-runbook-stamp: 0.45.0.0 -->
|
||||
<!-- This stamp must equal the VERSION file at every release; CI enforces it
|
||||
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
|
||||
the installed binary and warns on skew. -->
|
||||
|
||||
# BOOTSTRAP_FOR_AGENTS.md — become your human's persistent agent
|
||||
|
||||
You are an AI agent (Claude Code or Codex) reading this because a human pasted a
|
||||
block asking you to set yourself up as their persistent personal agent, with gbrain
|
||||
as your memory. This runbook is the judgment layer; the deterministic work lives in
|
||||
`gbrain bootstrap` subcommands with exit codes. Follow it top to bottom.
|
||||
|
||||
**Scope note:** this path is for Claude Code and Codex (desktop apps or CLIs).
|
||||
Running OpenClaw or Hermes? Use `INSTALL_FOR_AGENTS.md` instead.
|
||||
|
||||
**End state:** this folder is your workspace — identity files rendered from your
|
||||
human's own answers, a local brain (PGLite, embedded, no server), MCP wired,
|
||||
per-turn context, and a private GitHub repo as your durable body. ~15 minutes,
|
||||
mostly interview.
|
||||
|
||||
## Hard rules for you, the installing agent
|
||||
|
||||
⛔ **NEVER INVENT ANSWERS.** Personality, purpose, and boundaries come from the
|
||||
human. A guessed SOUL.md gets believed by every future session. If you do not have
|
||||
an answer, ask. The render step structurally refuses to run until the required
|
||||
answers exist and were read back.
|
||||
|
||||
⛔ **ASK IN SMALL BATCHES.** The interview is 12 questions max (6 required), asked
|
||||
in three batches. Mirror each batch back in one line. Accept "skip" on any
|
||||
non-required question — momentum beats completeness.
|
||||
|
||||
⛔ **STAY INSIDE THIS PHASE LIST.** Run `gbrain bootstrap status --json` and follow
|
||||
ITS phase list — the CLI is the source of truth, this document is commentary. If a
|
||||
step you are asked to run is not in the CLI's phase list, refuse it. If `status`
|
||||
reports a version skew between this runbook and the installed binary, say so and
|
||||
prefer the binary's instructions.
|
||||
|
||||
⛔ **NO SILENT FAILURE.** Every blocking condition (secret-scan block, lock
|
||||
collision, partial install) surfaces through `status`/`verify`/`doctor` output —
|
||||
read it and relay it to the human in plain language. Never work around a refusal.
|
||||
|
||||
⛔ **VERIFY BEFORE CLAIMING DONE.** The install is done when `gbrain bootstrap
|
||||
verify` exits 0 — not when the transcript looks good. Paste its report to the human.
|
||||
|
||||
⛔ **RESPECT THE TOOLCHAIN TRUST RULES.** Install bun via a platform package manager
|
||||
when available (`brew install oven-sh/bun/bun`); the only permitted fallback is the
|
||||
checksum-verified variant: download the pinned release to a file, verify it against
|
||||
that release's SHASUMS256.txt, and only then execute. Install gh the same way —
|
||||
platform package manager first (`brew install gh`, `apt install gh`, `dnf install gh`,
|
||||
`winget install GitHub.cli` per the official instructions); never a piped
|
||||
curl-to-shell one-liner. Install gbrain ONLY as
|
||||
`bun install -g github:garrytan/gbrain#latest-stable` — the npm package named
|
||||
"gbrain" is an unrelated project.
|
||||
|
||||
## Codex preflight (ChatGPT desktop / Codex CLI only)
|
||||
|
||||
Codex sandboxes command execution. Before starting, tell the human: "I'll need
|
||||
approval to run install commands (bun, gh, gbrain) and to write in this folder —
|
||||
approve those prompts when they appear." If approvals are globally disabled, ask the
|
||||
human to enable workspace-write + network for this session. Count the approval taps
|
||||
you needed; report the count at the end (it feeds the install-time measurement).
|
||||
|
||||
## Phase walkthrough (commentary — the CLI's list wins)
|
||||
|
||||
1. **Preflight.** `git`, `bun`, `gh` present. Install what's missing per the trust
|
||||
rules above: bun via a platform package manager or the checksum-verified download
|
||||
— the checksum-verified install is the ONLY permitted non-package-manager
|
||||
variant; gh via the platform package manager. (On a clean Mac, `git` may trigger
|
||||
the Xcode tools dialog — that download does not count against the 15 minutes,
|
||||
tell the human to let it run.)
|
||||
`gh auth status` — if logged out, the human's ONE manual step:
|
||||
`gh auth login -h github.com -p https -w` (you run it; they click Authorize).
|
||||
Then `gbrain bootstrap status` — it is idempotent and resume-aware; after any
|
||||
partial failure, re-run it and continue where it points.
|
||||
2. **Engine.** `gbrain init --pglite` (2 seconds, no server). Search mode defaults
|
||||
to balanced silently — do NOT ask; the human can change it any time with
|
||||
`gbrain search modes`. The one thing to raise here is the OPTIONAL provider
|
||||
key — with no key you run keyless: keyword search plus memory you author
|
||||
yourself through the write tools; everything works, one key upgrades search to
|
||||
semantic and enables automatic fact extraction. Never pressure for a key. If the
|
||||
human provides one, pass it to the CLI prompt — it goes to the 0600 config file,
|
||||
never into the interview answers, never into chat logs you keep.
|
||||
3. **Interview.** `gbrain bootstrap interview --init`, then ask the questions from
|
||||
the bank (the CLI prints them) in three batches, recording each answer verbatim
|
||||
with `--set KEY "value"`. Push once on vague answers to the required questions.
|
||||
After the last batch: read ALL answers back in one compact block, ask "Is this
|
||||
the thing you want in the room?", and only then run
|
||||
`gbrain bootstrap interview --confirm <hash>` with the hash `--status` printed
|
||||
for the read-back set. The gate fails if you confirm a set the human never saw.
|
||||
4. **Render.** `gbrain bootstrap render` — identity files appear. Show the human
|
||||
SOUL.md. Existing files are never overwritten (re-runs are safe; `--force`
|
||||
backs up first).
|
||||
5. **Skills + brain wiring.** The CLI scaffolds the skill set and registers
|
||||
`brain/` as the workspace source. Nothing to judge here; relay the output.
|
||||
6. **Wire the harness.** `gbrain bootstrap hooks --harness <detected>`:
|
||||
- Claude Code: installs per-turn hooks ON by default — do NOT ask; loading the
|
||||
brain every turn is the whole point of installing gbrain for your agent. Tell
|
||||
the human it is on and how to turn it off (`GBRAIN_HOOKS=0`, or re-run with
|
||||
`--no-hooks`, or `gbrain bootstrap uninstall`). The ONE consent to actually
|
||||
ask in this phase is MCP scope: project (recommended — any other repo you open
|
||||
cannot read your brain) vs user (your agent everywhere, but any repo you open
|
||||
can query it, and two open sessions contend for the database).
|
||||
- Codex: registers MCP (`codex mcp add`) and relies on the AGENTS.md protocol —
|
||||
say plainly that Codex gets pull-based context, not per-turn push.
|
||||
7. **Private repo.** `gbrain bootstrap repo` — creates a PRIVATE GitHub repo from
|
||||
the workspace, verifies the privacy bit through the API, pushes. Asks the
|
||||
background-persistence consent (15-minute scan-gated push job; declining still
|
||||
persists at session end). If the human has no GitHub or declines: local-only
|
||||
mode with an honest warning; `bootstrap repo` can run any time later.
|
||||
8. **Verify.** `gbrain bootstrap verify` — the whole contract: brain round-trip
|
||||
through the real write path, graph floor, token sweep, secret scan, repo
|
||||
privacy, hooks smoke, capability report (keyless or keyed). Exit 0 or it is not
|
||||
done. Paste the report. Then relay the first-run tour it prints (three prompts
|
||||
the human should try, starting with restarting the session).
|
||||
|
||||
## Machine two
|
||||
|
||||
If this workspace was cloned from an existing agent repo (agent.json says
|
||||
initialized), run `gbrain bootstrap attach` instead of the interview/render/repo
|
||||
phases — it wires this machine (source, hooks, MCP) and verifies. If agent.json
|
||||
says it is an uninitialized template, proceed with the normal flow from phase 1.
|
||||
|
||||
## Failure modes, and what they actually mean
|
||||
|
||||
| Symptom | Real cause | Fix |
|
||||
|---|---|---|
|
||||
| `interview --status` exits nonzero forever | A required answer is genuinely missing | Ask the human. Do not default it. |
|
||||
| Render refuses with unresolved tokens | Interview incomplete or a template edit broke a token | Finish the interview; `status` names the tokens. |
|
||||
| `verify` fails the magic-moment check | The fact never landed (keyless: the Facts fence was not written) | Re-run the write step it names; check `gbrain doctor`. |
|
||||
| Secret-scan block on push | A credential-shaped string in a tracked file | Fix or allowlist deliberately (`.gbrain-scan-allow`); never force. |
|
||||
| "bootstrap already running (pid N)" | A concurrent bootstrap holds the lock | Wait or investigate that pid; the lock self-clears when stale. |
|
||||
| Brain tools fail with a lock error | Another live session's serve owns the database | Close the other session; sequential use is the v1 contract. |
|
||||
| Hook reports "brain context unavailable" | serve not running or degraded | `gbrain doctor` names it; hooks fail open by design. |
|
||||
|
||||
## Hand off
|
||||
|
||||
Finish by telling the human: the private repo URL (or the local-only status), the
|
||||
capability mode (keyless vs keyed), the three commands they will actually reuse
|
||||
(`gbrain doctor`, `gbrain bootstrap verify`, `gbrain sources push`), and the
|
||||
first-run tour. Then delete nothing — this runbook was fetched, not installed.
|
||||
+526
@@ -2,6 +2,532 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.45.0.0] - 2026-08-10
|
||||
|
||||
**Your coding agent can now become your personal agent.** Paste one block into Codex or Claude Code and it sets itself up as a persistent agent with a memory that survives across sessions: it interviews you, writes its own identity files from your answers, spins up a local brain, and keeps a private GitHub repo as its durable body. Close the laptop and reopen it tomorrow, and it still knows who you are, who you're talking to, and what you told it last time. This is the OpenClaw/Hermes experience — identity, memory, schedules, persistence — running on the subscription you already pay for, with nothing to deploy.
|
||||
|
||||
**Start with Codex.** It runs on your ChatGPT subscription, takes about fifteen minutes, and deploys nothing. Claude Code is the same install. OpenClaw and Hermes are still the way to run GBrain exactly as designed — always on, enriching around the clock — at real server and API cost; Codex is the recommended first step for anyone new to GBrain.
|
||||
|
||||
### How to use it
|
||||
|
||||
Pick the folder that will become your agent's home and paste (Codex shown; the Claude Code block is identical):
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
The agent runs `gbrain bootstrap` — a new command family (`status`, `interview`, `render`, `repo`, `hooks`, `verify`, `attach`, `uninstall`) that drives the whole install. It works with **zero API keys**: your harness's model is the LLM, so the agent authors memory directly and search runs keyword-only; add one optional key (OpenAI, Anthropic, or Voyage) to unlock semantic search and automatic fact extraction. Everything is consent-gated — hooks, background push, MCP scope — and nothing runs while your harness is closed (the honest desktop contract; true 24/7 is what a hosted brain adds).
|
||||
|
||||
### What you get
|
||||
|
||||
- **A memory that compounds.** On Claude Code, session hooks inject relevant brain context at the start of each prompt and ingest the transcript when the session ends; on Codex, the rendered `AGENTS.md` carries the same pull protocol. Facts you state get written back through the brain's own tools and resurface next session.
|
||||
- **A portable body.** Your workspace is a private GitHub repo (privacy verified via the API before anything is pushed, secret-scanned before every commit). Clone it on a second machine and run `gbrain bootstrap attach`. It mounts anywhere GBrain runs — the exit plan if a provider ever changes its terms.
|
||||
- **Multi-model install.** Works in the ChatGPT desktop app, the Codex CLI, Claude Code desktop, and the Claude Code CLI. Same body format, two doors.
|
||||
|
||||
### What to know after upgrading
|
||||
|
||||
Existing installs are untouched — this is a new, opt-in surface. If you want it, run the paste block in a fresh folder. `gbrain doctor` gained bootstrap health checks (hook heartbeat, push staleness, serve/lock collisions, runbook skew); if anything looks off after an install, `gbrain doctor` names it and prints the fix.
|
||||
|
||||
### What we caught and fixed before merging
|
||||
|
||||
This shipped through a full review pass (architecture, security, adversarial, cross-model). Highlights, in plain terms: the credential scanner now recognizes today's API-key formats and reads the exact bytes being committed rather than a stale snapshot, so a key can't slip through a timing gap; private-key material is redacted whole, not just its header; and the scanner fails safe — an unreadable or oversized file blocks the push instead of passing unscanned. The GitHub repo is confirmed to belong to you and to be private before any content leaves the machine.
|
||||
|
||||
## To take advantage of v0.45.0.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
|
||||
warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **To try the new agent-bootstrap surface,** paste the block above into Codex
|
||||
or Claude Code in a fresh folder, or read
|
||||
[`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) for the full contract.
|
||||
Existing brains need no action — bootstrap is an opt-in new surface, not a
|
||||
change to how your current install works.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`
|
||||
and `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
## [0.44.1.0] - 2026-08-11
|
||||
|
||||
**Any current model works now. gbrain stops rejecting model ids it hasn't heard of.**
|
||||
|
||||
New models ship every week. Until now, gbrain kept a built-in list of "known" models for each major provider (Anthropic, OpenAI, Google), and if you configured a model that list hadn't learned yet, gbrain refused to run it, even when the provider was already serving that model to everyone else. In practice that meant a brand-new model like `openai:gpt-5.6-sol` read as "not available" on an install whose binary shipped three weeks earlier, and the error message steered people toward older models instead. Worse, the workaround was inconsistent: the same model id worked when set as `models.default` but was rejected when set as `models.think`, for no reason a user could see.
|
||||
|
||||
That whole class of failure is gone. gbrain now checks only what a provider can do (Anthropic has no embedding models, Voyage has no chat models). Which model id you use is your call. If you name one that does not exist, the provider says so at call time, in its own words, and gbrain shows you that message instead of guessing.
|
||||
|
||||
How to use it:
|
||||
|
||||
```bash
|
||||
gbrain config set models.default openai:gpt-5.6-sol # any current model id
|
||||
gbrain config set models.think google:gemini-3.6-flash # per-task keys work identically now
|
||||
gbrain models doctor # pre-flight: probes your configured models live
|
||||
```
|
||||
|
||||
What changed in each case:
|
||||
|
||||
| You do | Before | Now |
|
||||
|---|---|---|
|
||||
| Configure a model newer than your gbrain binary | Rejected: "not listed... Known models: ..." | Runs |
|
||||
| Set that model via `models.think` / `models.dream.*` | Rejected even when `models.default` worked | Identical behavior on every key |
|
||||
| Typo a model id | Caught instantly, locally | Fails at the provider with the provider's own message; `gbrain models doctor` still catches it pre-flight without spending tokens |
|
||||
| Use a chat model from an embeddings-only provider | Rejected | Still rejected (that check is about the provider, not the model) |
|
||||
|
||||
Things to watch: a typo'd model id now costs one failed provider call instead of failing free and instantly. Run `gbrain models doctor` after changing model config if you want the old fail-fast feel. `gbrain think`'s fallback answer now carries the provider's actual error text, so a bad model id no longer reads as an API-key problem.
|
||||
|
||||
## To take advantage of v0.44.1.0
|
||||
|
||||
No migration, no schema change. `gbrain upgrade` is enough.
|
||||
|
||||
1. **Set any current model:**
|
||||
```bash
|
||||
gbrain config set models.default <provider>:<model>
|
||||
```
|
||||
2. **Verify:**
|
||||
```bash
|
||||
gbrain models doctor
|
||||
```
|
||||
3. **If a model you know is real still fails,** the error now comes from the provider; check the id spelling and your key, then file an issue at https://github.com/garrytan/gbrain/issues with the `gbrain models doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/ai/model-resolver.ts` — `assertTouchpoint(recipe, touchpoint, modelId)` checks provider touchpoint capability only; the native-recipe model allowlist throw is removed. Recipe `models:` arrays remain as data: `models[0]` default selection for `--model <provider>` shorthand and env-ready pickers, guard-test fixtures for the repo's own hardcoded defaults, and `gbrain providers list` display.
|
||||
- `src/core/ai/gateway.ts` — the extended-models registry (`_extendedModels`, `registerExtendedModel`, `registerConfigSelectedChatModel`, both registration loops, and the tier-resolution loop that fed them) is deleted. Every per-task model key (`models.think`, `models.dream.*`, `facts.extraction_model`, ...) now behaves exactly like `models.default`. `gateway.rerank()` keeps its own model-list check: each listed reranker id maps to a known request/response wire shape.
|
||||
- `src/core/think/index.ts` — the graceful "no LLM available" sentinel surfaces the thrown `AIConfigError`'s own message and fix (which key is missing, or what the provider rejected) instead of generic key advice.
|
||||
- `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — drops the now-dead chat-model registration call.
|
||||
- Tests — `test/gateway-tier-extended-models.test.ts` deleted (pinned the removed machinery); rejection tests across `test/ai/`, `test/think-*`, and `test/cycle/` now pin the pass-through contract; `unknown_model` still fires for providers lacking the touchpoint, so every probe reason stays reachable; new test pins the sentinel carrying the provider's error text.
|
||||
- Docs — `docs/architecture/KEY_FILES.md` entries for the resolver, gateway, and reindex handler updated to the new contract.
|
||||
|
||||
## [0.44.0.0] - 2026-06-12
|
||||
|
||||
**BrainBench: agent memory now has a scorecard.** `gbrain eval brainbench` is a public, reproducible, cross-harness conformance suite for the four ways agent memory fails — and from this release forward, every memory PR must hold or move its numbers against a committed baseline that CI compares against master's own copy.
|
||||
|
||||
Four suites, scored per harness seam:
|
||||
|
||||
- **know-to-ask** — does memory volunteer the right context unprompted, and stay silent when it should? (`know_to_ask_failure_rate` + the anti-gaming `false_fire_rate` — "always inject" can't win both)
|
||||
- **push precision / recall** — when context is volunteered, was it the right context, within each harness's injection budget?
|
||||
- **write-back fidelity** — do conversation facts survive the PRODUCTION conversation→memory pipeline with correct provenance? (graded through a new injectable-extractor seam on the conversation-facts pipeline, so CI executes the shipped segmentation/insertion/dedup code with zero LLM calls; `--llm` opts into the real extractor under a run-scoped budget)
|
||||
- **continuity** — a decision recorded in one session, recalled by a different harness on the same brain
|
||||
|
||||
Plus two cross-cutting measures: `source_isolation_violations` (gates at zero, every run — cross-source leakage is the data-leak invariant) and `avg_injected_tokens` (the intrusion budget, reported per harness).
|
||||
|
||||
Every scoreboard row carries a `seam` label: the `openclaw` row exercises the shipped context-engine pipeline byte-for-byte (`production`); the `claude-code` and `codex` rows grade the same gbrain primitives through those harnesses' injection-shape contracts (`contract`) — the exported wire types are the contracts future integrations implement, and the rows flip to `production` with continuous numbers when they land. At v0.44.0.0 the production seam reads: know-to-ask failure 0.15 (the documented v1 reflex limits, now measured), push recall 0.81 at precision 1.0, write-back fidelity 1.0, continuity 1.0, zero isolation violations.
|
||||
|
||||
### Added
|
||||
|
||||
- `gbrain eval brainbench` — hermetic by default (in-memory PGLite, no keys, no LLM, ~7s for the full corpus), with `--harness`, `--suite`, `--fixtures DIR --gold DIR` (the foreign-runner surface), `--json`, `--out` (the canonical CI artifact), `--compare`, `--update-baseline`, `--justification`, `--allow-regression`, `--include-holdout`, and `--llm --budget-usd`. Exit codes are the contract: 0 pass, 1 regression, 2 error/inconclusive.
|
||||
- A 141-fixture / 241-gold-turn benchmark corpus across 7 stratified categories, generated deterministically (seed 42) over a whole-cloth fictional universe with SEALED gold (a `gold` key inside a fixture is a validation error; adapters only ever see sanitized turns), a ~15% holdout split, and a blind double-label validation receipt (96.4% agreement) recorded in the corpus ledger. Rebuild byte-identically with `bun evals/brainbench/generator/gen.ts`.
|
||||
- Published interchange contract for foreign runners: JSON Schemas for fixtures, gold, results, and baselines under `evals/brainbench/schema/` — point the CLI at any conforming corpus and parse the schema'd receipt back.
|
||||
- A CI gate (`brainbench` job + `scripts/ci-brainbench-gate.sh`) hardened through four adversarial review rounds: it compares against MASTER's committed baseline (a PR cannot rewrite the thing it's graded by), any baseline edit without a fixture change must byte-match the actual run, corpus changes route through a bless mode where shrinking gold coverage or regressing a metric requires a written justification visible in the PR diff, baselines bind their run config, and the gate fails hard — never silently open — on broken refs or deleted baselines.
|
||||
- Eleven new metrics in the metric glossary (plain-English definitions in `docs/eval/METRIC_GLOSSARY.md`); every JSON response carries the standard `_meta.metric_glossary` block.
|
||||
- Methodology doc at `docs/eval/BRAINBENCH.md`: seam disclosure, formulas, pre-registered expectations, determinism posture, gate governance, gold methodology, and accepted residuals.
|
||||
|
||||
### Changed
|
||||
|
||||
- `gbrain eval run-all --suites brainbench` now actually runs the suite in-process (closing the long-standing orchestrator-stub follow-up) and records one result per sweep under `EvalRunRecord` schema v3 with `mode: 'n/a'` — benchmark records no longer fabricate a search mode.
|
||||
- The conversation-facts pipeline accepts an injectable per-segment extractor (default unchanged — the LLM path); this is the seam the write-back suite grades production code through.
|
||||
- The synthetic-corpus privacy guard now scans the BrainBench corpus dirs (fixtures AND gold) alongside the calibration corpus, and the fixture-authoring guide adds scenario-level privacy rules: scenarios are invented whole-cloth, never anonymized from real situations.
|
||||
- The CI aggregate (`test-status`) now checks the brainbench job's result explicitly — a failing memory gate fails the branch-protection check.
|
||||
|
||||
### Fixed
|
||||
|
||||
- CLI exit codes for the new command route through the shared write-fence + aliveness-grace exit seam, so PGLite's WASM exit-code stomping and Bun's exit-time stdout discard can't corrupt the CI contract.
|
||||
|
||||
To take advantage of v0.44.0.0: run `gbrain eval brainbench` — no setup, no keys, no brain required. If it ever reports something broken after an upgrade, `bun evals/brainbench/generator/gen.ts` rebuilds the corpus byte-identically and `gbrain eval brainbench --update-baseline` re-derives the baseline from an actual run; both are safe to re-run any time.
|
||||
## [0.43.0.0] - 2026-08-08
|
||||
|
||||
**Your agent now has five memory verbs it can actually reach.** Cathedral 1 freezes
|
||||
a stable, versioned memory protocol — `recall`, `remember`, `entity`, `synthesize`,
|
||||
`forget` — over the brain's operation catalog, the way Postgres speaks one wire
|
||||
protocol to every client. Point any MCP harness at it (`claude mcp add gbrain --
|
||||
gbrain serve --surface verbs`, or the Codex/OpenClaw equivalents) and the agent sees
|
||||
exactly five self-describing tools instead of a wall of internal ops. Every response
|
||||
carries what it is, why it matched, where it came from, and what it spent (the token
|
||||
budget on `recall`, latency on `entity`, the full cost block on `synthesize`) — and the
|
||||
contract never breaks: v1 field names and meanings are frozen, changes are
|
||||
additive-forever.
|
||||
|
||||
What you can do now that you couldn't before:
|
||||
|
||||
- **Remember a fact once, recall it in a fresh session — in any harness.** `remember`
|
||||
takes mandatory provenance (where the fact came from) and an optional expiry, dedupes
|
||||
against what's already known, and supersedes the old fact when it changes
|
||||
("X joined acme-example" → "X left acme-example" — the outdated fact expires, the
|
||||
history stays). `recall` retrieves saved facts and,
|
||||
with a query, budget-packed page snippets — the server enforces the token budget and
|
||||
tells you what it dropped instead of trusting the client to trim.
|
||||
- **Look up one person/company/project as a compact card in well under 100ms, zero LLM
|
||||
calls.** `entity` resolves a name to a privacy-safe card (who/what, aliases,
|
||||
last-touched, open threads, top typed edges) — or, on a miss, near-miss suggestions
|
||||
instead of a dead end.
|
||||
- **Reason across pages when you actually need it.** `synthesize` is the explicitly
|
||||
expensive verb (it says so in its own description), with a best-effort cost block so
|
||||
agents choose it deliberately. No API key configured? It says `unavailable` with a
|
||||
fix, never a fake answer.
|
||||
- **Certify any memory server against the contract.** `gbrain protocol --json` publishes
|
||||
the machine-readable spec; `gbrain protocol conformance [--target <endpoint>]` runs the
|
||||
frozen-contract test suite against gbrain's own server or any MCP endpoint — and
|
||||
provably fails servers that don't comply. `gbrain protocol stats` shows per-verb
|
||||
adoption and your real time-to-first-use, all from a local file that never leaves
|
||||
your machine.
|
||||
|
||||
`gbrain serve` keeps every operation by default (`--surface full`) — existing installs
|
||||
are unchanged. The new `--surface verbs` is the quickstart surface for agents. No schema
|
||||
migration; the verbs ride the existing facts, pages, and typed-graph tables. The full
|
||||
contract, per-harness install blocks, and the additive-forever versioning policy live in
|
||||
[docs/protocol/MEMORY_VERBS_v1.md](docs/protocol/MEMORY_VERBS_v1.md).
|
||||
|
||||
**This release also fixes which search verb your agent reaches for (#2416).** The
|
||||
`search` and `query` tool descriptions, the mandatory lookup chain, and the search
|
||||
guides had drifted from reality: they described `search` as keyword-only (it has been
|
||||
cheap hybrid — vector + keyword, no LLM expansion — for many releases) and told agents
|
||||
to try `search` first for everything, falling back to `query` only when results looked
|
||||
"thin". For concept questions ("all the X that do Y", "the ecosystem around Z") that
|
||||
fallback never fired, and synonym-phrased matches dropped silently.
|
||||
|
||||
- **Concept questions now route to `query` by default.** The tool descriptions and the
|
||||
lookup-chain convention are intent-driven: exact known tokens → `search` (cheaper, no
|
||||
expansion call); concept / synonym / exhaustive-set questions → `query` first. Verified
|
||||
with a live LLM routing eval: all concept phrasings route to `query`, and personal
|
||||
questions still route to the salience ops.
|
||||
- **"Got results" is no longer treated as "got everything."** The descriptions, docs, and
|
||||
conventions now say it plainly: a populated `search` result set is not proof of
|
||||
coverage, and `query` is still top-K — literal exhaustive enumeration belongs to
|
||||
`list_pages` pagination.
|
||||
- **The CLI nudges you when it can help.** A concept-shaped `gbrain search "..."` prints a
|
||||
one-line hint on stderr suggesting the equivalent `gbrain query` call (results stay
|
||||
clean on stdout; `--quiet` silences it; never auto-reroutes).
|
||||
- **`think` cost accounting reads consistently.** When no LLM call ran (stubbed or no
|
||||
API key), `usage` is now uniformly `null` in `--json` output rather than sometimes
|
||||
missing.
|
||||
|
||||
To take advantage of v0.43.0.0: re-run `gbrain serve` with `--surface verbs` to give
|
||||
your coding agent the five-verb memory protocol (or keep `--surface full` for the
|
||||
complete operation catalog — both speak the verbs). Run `gbrain protocol conformance`
|
||||
to self-certify, and `gbrain protocol stats` to watch adoption. Memories your agent
|
||||
saves are readable by every agent connected to the brain by default; pass
|
||||
`visibility: "private"` for local-only facts. If your agent instructions or skill
|
||||
files copy the old "search first, query if thin" rule, refresh them from
|
||||
`skills/conventions/brain-first.md` — the shipped skillpack carries the corrected
|
||||
routing.
|
||||
## [0.42.76.0] - 2026-08-08
|
||||
|
||||
**Mistyped or unsupported flags now fail loudly instead of being silently ignored — including the ones that were supposed to make a command safe.**
|
||||
|
||||
**Strict flag validation, CLI-wide.** Every gbrain command now rejects a flag it does not understand, with a clear error naming the flag and the command, before any work runs. Before, commands read their flags ad hoc and ignored the rest — so `gbrain post-upgrade --dry-run` accepted the flag, ignored it, and applied migrations for real. That class is gone: the legal flags for every command are derived from each command's own source into a generated registry, checked before dispatch, and a command may only advertise a safety flag like `--dry-run` if its code actually reads it. On commands routed through the operations contract, a trailing `--dry-run` is now a real rehearsal switch rather than a no-op. `--json` invocations get the same error as a structured payload, so scripts fail cleanly too.
|
||||
|
||||
**A word of warning (intentional breaking change):** cron jobs or scripts that pass stray, misspelled, or long-removed flags have been running on luck — the flag did nothing. Those invocations now exit with an error naming the flag. That is the point: fix the invocation once and it means what it says forever. Everything after `--` is passthrough and remains untouched.
|
||||
|
||||
**Upgrades can't wedge on forward-referenced columns anymore — as a class.** The v0.42.56.0-era startup wedge (a schema blob referencing a column that pre-existing brains didn't have yet) had two more latent instances waiting in the jobs table. Both are now probed and healed at startup, and the schema coverage guard was rewritten to cross-reference every column referenced by the embedded schema against the set of columns any migration has ever added — so a new forward reference cannot ship without its startup probe. A recovery test walks the exact journey an affected brain takes: failed upgrade, retry on the fixed binary, converge with no leftover state blocking the way.
|
||||
|
||||
**Remote agents get more, within the same fences.** The `think` operation is now available to remote MCP callers as a read-only synthesis — the local CLI can still persist results, while remote callers are forced read-only. Chunk reads now resolve through the same source-scope rules as page reads, so a federated grant that can open a page can also read that page's chunks, and a caller without the grant cannot reach chunks outside its own floor. Chunk payloads also stop carrying raw embedding vectors over the wire — noticeably smaller responses with no behavior change, since no consumer ever read them. Two internal call sites that forward caller identity now treat anything ambiguous as untrusted, matching the fail-closed rule the rest of the codebase already follows.
|
||||
|
||||
**Source-bound clients can be minted over HTTP.** The `/admin/api/register-client` endpoint now accepts `source` and `federatedRead` bindings, mirroring the CLI's `--source` / `--federated-read` flags — so an admin UI or provisioning proxy can create a client confined to a specific brain source without shelling out to the CLI. Omitting both preserves the historical default, and invalid source ids get a structured 400.
|
||||
|
||||
**`gbrain doctor` and `repair-jsonb` see further and misfire less.** The double-encoded-JSON scan now covers the subagent execution columns, and the damage test requires the stored text to actually parse as JSON before flagging it — a legitimate string value that merely starts with `[` or `{` (a log line, a code snippet) is no longer misclassified, and a repair pass can no longer corrupt it. One damaged table no longer aborts the scan of the rest.
|
||||
|
||||
### To take advantage of v0.42.76.0
|
||||
## [0.42.75.0] - 2026-08-08
|
||||
|
||||
**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.**
|
||||
|
||||
The dreaded `RuntimeError: Aborted()` at startup — the one that made zero-config brains unusable after a macOS upgrade and pushed people onto Homebrew Postgres — was never a macOS or WASM bug. An unclean shutdown (typically the upgrade reboot) tears the write-ahead log inside the data dir, and every open after that dies replaying it. gbrain now detects that failure on any command, backs up the WAL state to a sibling directory, resets it in place (the pg_resetwal recovery Postgres has shipped for decades, ported to run against PGLite data dirs), and reopens your brain — pages, embeddings, and history intact. Transactions that never reached a checkpoint may be lost; that is the standard trade for a database that would otherwise not open at all.
|
||||
|
||||
### Added
|
||||
- **Automatic WAL repair on startup.** A torn-WAL abort self-heals on the next gbrain command: backup → in-place reset → retry, with a loud notice naming the backup and recommending `gbrain doctor`. Disable with `GBRAIN_PGLITE_WAL_REPAIR=off`.
|
||||
- **`gbrain pglite-repair`** — the deliberate version: `--dry-run` gives a read-only diagnosis of the data dir; `--yes` runs the same in-place repair manually. Refuses to operate while any live process holds the brain, and never force-removes another process's lock.
|
||||
- **`gbrain doctor` diagnoses unopenable PGLite brains.** A new `pglite_data_dir` check reads the data dir from disk when connect fails, names the right recovery rung (repair vs rebuild), inventories repair backups, and escalates when repairs keep recurring — the signal that something is still killing gbrain mid-write.
|
||||
- **Recovery guardrails throughout:** repair runs only under a cleanly-acquired lock (never after taking over another process's lock, with a quarantine window when a lock's holder couldn't be verified); a live database — including a native Postgres one — is refused by a `postmaster.pid` liveness check; repeated attempts inside one corruption episode reuse one backup instead of stacking copies (newest three episodes retained); a cooldown stops repair loops from silently eating data on machines where crashes keep recurring; and every restore path reports honestly whether your original files are back in place or waiting in the backup.
|
||||
|
||||
### Changed
|
||||
- **`gbrain reinit-pglite` works bare.** The embedding model and dimensions now default from your config file, so the rebuild rung of the recovery ladder is one command mid-outage (explicit flags still win; environment overrides are deliberately ignored so a stale shell export can't change the rebuild target).
|
||||
- **Honest error messages.** The startup-abort hint now names the real cause (torn WAL after an unclean shutdown), states exactly what auto-repair did or why it stood down, and lays out the full ladder: repair → rebuild → engine switch. The docs that claimed PGLite is "incompatible with macOS 26.x" have been rewritten (README, INSTALL, ENGINES) — thanks @roysaurav for the original native-Postgres walkthrough, which remains the engine-switch rung.
|
||||
- Message-less WASM error objects no longer surface as `[object Object]`.
|
||||
|
||||
### Fixed
|
||||
- The classifier that routes startup failures now matches the abort message PGLite actually produces (it previously fell through to a generic hint), while catalog corruption keeps routing to rebuild — WAL repair is never suggested for damage it cannot fix.
|
||||
- Lock-file reads can no longer misclassify a healthy live holder as corrupt (writes are atomic now), a holder owned by another user is treated as alive, and an in-flight acquisition is no longer mistaken for a corrupt lock.
|
||||
|
||||
Credit where due: @yang1996202-cpu (#2575), @AndreLYL (#223), and @roysaurav (#1670) for reports and diagnosis, the #223 thread contributors whose recoveries proved the root cause, and @yestheboxer, whose rejected upstream recovery PR (electric-sql/pglite#994) this port builds on.
|
||||
|
||||
### To take advantage of v0.42.75.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to configure. If a cron job or script starts failing with `unknown flag`, that invocation was passing a flag that did nothing — remove or fix the flag and it will not regress silently again.
|
||||
|
||||
### For contributors
|
||||
|
||||
Community fixes absorbed with credit: @colinagent (#2598 think read-scope; the upgrade-rewind e2e pattern from #2623), @guim4dev (#2016 register-client source bindings), @vinsew (#597 repair-jsonb coverage extension), @javieraldape (#2494/#2531 output-correctness class — BigInt-safe local rendering and the search `--json` regression pin land here; parts of both PRs shipped earlier from master). Thank you — superseded PRs are being closed with notes.
|
||||
|
||||
The unit-test runner is now memory-safe on machines running multiple workspaces: shard concurrency adapts to actually-available memory, and a serial rescue lane re-runs files that died to OOM or external kills before calling them failures — a red suite now means real failures, not memory pressure. The flag registry regenerates via `bun run build:flag-registry` and is pinned by freshness, drift, and consumption-evidence guards.
|
||||
If your brain currently won't open, that's it — the next command repairs it. If you'd rather look first: `gbrain pglite-repair --dry-run`.
|
||||
|
||||
## [0.42.74.0] - 2026-08-07
|
||||
|
||||
**Two fixes for agents that reach a brain over the network: takes-holder visibility now works the way you set it, and the voice recipe is safe by default.**
|
||||
|
||||
Legacy bearer tokens served over `gbrain serve --http` now honor the takes-holder allow-list you set with `gbrain auth permissions <token> set-takes-holders`. Before, that setting was read on one serving path but silently ignored on the other, so a remote agent saw only world-held takes no matter what you granted — a token you widened to see brain-held takes saw none of them, and a token you narrowed still saw the public ones. Both directions now behave as configured, an empty grant means "no takes" (not "the default set"), and the two serving paths decode and apply the grant through one shared piece of code so they cannot drift apart again. Tokens with no grant continue to fall back to public-only, so nothing widens on upgrade.
|
||||
|
||||
The bundled voice-agent recipe (`recipes/agent-voice`) ships secure by default. Its reference server now refuses cross-origin browser requests unless you name the origins in `AGENT_VOICE_CORS_ORIGIN`, gates the endpoints that spend your OpenAI key or read your brain so a stray web page can't trigger them, and listens on loopback only until you set `HOST` to expose it. The voice page you run locally is unaffected. Because this recipe is copied into your own repo at install time, `gbrain integrations install agent-voice --refresh` picks up the hardened version.
|
||||
|
||||
### To take advantage of v0.42.74.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Then, if you serve a brain to remote agents, set each token's takes-holder scope with `gbrain auth permissions <token> set-takes-holders world,brain` (or your desired holders). Voice-recipe operators run `gbrain integrations install agent-voice --refresh --target <your-host-repo>`, then set `AGENT_VOICE_CORS_ORIGIN` if a browser on another origin needs access and `HOST=0.0.0.0` only if the server must listen beyond loopback.
|
||||
|
||||
### For contributors
|
||||
|
||||
Both issues were reported by external security researchers who supplied fixes. Ship-stage adversarial review hardened two more spots: the two serving paths now share one permissions-decode helper (not just the allow-list parser) so a malformed double-encoded row can't make them disagree, and the hot-memory cache key encodes the allow-list collision-free so the empty-vs-absent distinction holds for every holder value. Credit @Derek95king (takes-holder threading) and @sebastiondev (voice-recipe CORS).
|
||||
|
||||
## [0.42.73.2] - 2026-08-05
|
||||
|
||||
**A write that deduplication redirects onto an existing page is now checked against the write scope of whoever asked for it.** When the same content arrives under a new slug, gbrain recognises it and points the write at the page that already holds it. That redirected target is now tested against the caller's own scope — under whichever mechanism confines that caller. One of the two mechanisms was consulted at that point; both are now.
|
||||
|
||||
Nothing changes for local CLI use, or for clients that hold unrestricted write access — neither was ever scope-confined. A confined caller whose write dedups onto a page **inside** its own scope keeps working exactly as before; that redirect is a feature and it is preserved, with a regression test to keep it that way. A confined caller whose write dedups onto a page **outside** its scope now gets `permission_denied`, with the remedy in the message: drop the `id:` frontmatter field, or change the content, to write a new page under your own prefix. The denial does not name the page the write resolved to.
|
||||
|
||||
Recommended for any brain served over HTTP to scope-restricted clients.
|
||||
|
||||
### To take advantage of v0.42.73.2
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to configure. Existing clients keep their scopes unchanged, and no re-registration is needed.
|
||||
|
||||
### For contributors
|
||||
|
||||
Reported privately by an external security researcher, who supplied a fix and a regression test with it. The version that shipped composes the two existing scope-matching rules into a single predicate rather than restating either one, so the check at the door and the check after a redirect cannot drift apart; the audit the report prompted closed the same gap on one further caller path.
|
||||
|
||||
## [0.42.73.1] - 2026-08-05
|
||||
|
||||
**Removes the PR gate that v0.42.73.0 added, and reverts the v0.42.72.1 contribution-policy change it enforced.** The gate cannot function on this repository, and it caused a real incident before that was understood.
|
||||
|
||||
The gate needed two things this repository does not grant it: an `ANTHROPIC_API_KEY` Actions secret for its verdict, and read-write workflow permissions to post a comment or set a label. Without them it can only skip. Worse, on its first live runs a read-only token turned every API call into a 403, the code treated that as a crash, and the check went red on an outside contributor's pull request four times with no comment explaining why. That was fixed in v0.42.73.0, but a check that runs on every pull request and can never reach a verdict does not earn its place in the repository.
|
||||
|
||||
The v0.42.72.1 contribution policy is also withdrawn: the human-written intent paragraph and gbrain-in-use screenshot are no longer required on issues and pull requests. `CONTRIBUTING.md`, both issue templates, and the pull-request template return to their pre-2026-08-02 state, and issues and PRs are reviewed on their content by maintainers, as before.
|
||||
|
||||
The code is preserved in git history at v0.42.73.0 and can be restored if the repository ever grants those permissions. If it is restored, the mechanical half — the intent and screenshot check, the version-first title rule, the red flags — should render to the Actions job summary instead of a comment, because that needs no token permission and no API key.
|
||||
|
||||
### To take advantage of v0.42.73.1
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to change. Everything else v0.42.73.0 shipped — the five contributed correctness fixes, `slug_filter`, and the four dependency pins that cleared six CVEs — is unaffected and stays.
|
||||
|
||||
## [0.42.73.0] - 2026-08-04
|
||||
|
||||
**Every incoming pull request now gets a verdict before anyone reads it — and five contributed fixes for silent wrong answers.**
|
||||
|
||||
**The PR gate.** Open a pull request against gbrain and an automated check now posts a single verdict comment within a minute: **merge-lane**, **close-lane**, or **needs-maintainer**, with its reasons and a checklist of what a human reviewer should verify for that specific diff. It also checks mechanically that the description carries the human-written intent paragraph and the screenshot of gbrain in use that `CONTRIBUTING.md` requires, and that the title leads with its version.
|
||||
|
||||
It is deliberately **advisory** — a triage signal and a reviewer checklist, not an authorization boundary. A green verdict is not permission to merge; a maintainer still decides. Pull-request code is never checked out or executed: the verdict comes from the description and the diff read through the API. Maintainer, bot, and draft pull requests are exempt from the intent-and-screenshot floor only (release automation cannot screenshot itself); they still receive the full verdict. Where the rubric can be argued with, the decision is taken away from it: a merge-lane recommendation is downgraded automatically when a diff adds a dependency, a new provider recipe, or new config keys, edits workflows, deletes a test, exceeds 40 files or 400 net source lines, or changes `src/` without touching a single test.
|
||||
|
||||
**Your import output parses again.** `gbrain import <dir> --json` printed five informational lines to stdout ahead of the JSON payload, so anything parsing that output read zero imports while its own bookkeeping recorded the files as ingested — and the next run skipped them permanently. Those lines now go to stderr under `--json`; human output is byte-for-byte unchanged.
|
||||
|
||||
**`sources harden --dry-run` no longer changes anything.** It reset the helper's executable bit before reaching the dry-run check, so a documented preview quietly mutated permissions.
|
||||
|
||||
**Telemetry records the model that actually ran.** Two nightly-cycle phases wrote a hardcoded or unrelated model name into their verdict cache, evidence signature, and spend metering while the gateway ran whatever chat model you configured. On any brain with a non-default model, the recorded history was fiction.
|
||||
|
||||
**`gbrain integrity` stops contradicting itself.** Dead-link findings were counted in the "Review queue" total but written to a different file, so `integrity review` disagreed with `integrity auto`'s own summary. They now get their own line.
|
||||
|
||||
**Retype rules can address API-ingested pages.** Mapping rules could only filter on a file path, which is empty for every page written through `put_page` — so no rule could target that whole class. A new `slug_filter` filters on the slug instead, and combines with the path filter when both are given.
|
||||
|
||||
Also: the `integrity` source comment no longer documents a `--dry-run` subcommand form that exits with an error.
|
||||
|
||||
### To take advantage of v0.42.73.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain import <dir> --json | jq . # now parses
|
||||
gbrain integrity auto # dead links reported separately
|
||||
```
|
||||
|
||||
Nothing to configure for the gate — it runs on pull requests to this repository. If you maintain a fork and want it, the workflow needs an `ANTHROPIC_API_KEY` secret; without one it skips loudly rather than blocking anyone.
|
||||
|
||||
### For contributors
|
||||
|
||||
The gate went through six rounds against two independent blind reviewers, each judging cold. The findings that changed the design most were not exploits but false positives: a code fence that swallowed the rest of a description, an explanation written as bullet points scoring zero words, a word floor stricter than the published policy, and a comment telling contributors to reopen a pull request that was never closed. Those four descriptions are now permanent regression fixtures — a gate that insults a first-time contributor is worse than no gate. Two properties are deliberate and documented rather than fixed: the mechanical floor is a floor (a determined author clears it in seconds), and a bare URL in a cited reason still autolinks.
|
||||
|
||||
Contributed by @YiconZiwei (#2655), @time-attack (#3764, #3759, #3726, #3751, #3739, and the gate groundwork in #3573/#3698).
|
||||
|
||||
## [0.42.72.1] - 2026-08-02
|
||||
|
||||
**Every issue and pull request now needs a human-written paragraph and a screenshot of gbrain actually being used.**
|
||||
|
||||
Effective immediately, opening an issue or a PR requires two things from you personally: a paragraph you wrote yourself saying why you're opening it — what you were doing, what went wrong or what you needed, why it matters — and a screenshot of your terminal, agent session, or logs showing the real situation. Rough grammar is fine and preferred over polish. AI-generated or AI-polished intent text is not accepted; the paragraph is the human part. AI assistance for the *code* is still welcome.
|
||||
|
||||
Issues and PRs missing either are closed without review, and can be reopened once both are added. Scrub private names, companies, keys, and brain contents from screenshots before attaching — a redacted screenshot is fine, a missing one is not.
|
||||
|
||||
The requirement is stated in `CONTRIBUTING.md` and pre-filled in the bug-report and feature-request issue templates plus a new pull-request template, so the fields are in front of you when you open one.
|
||||
|
||||
## [0.42.72.0] - 2026-08-01
|
||||
|
||||
**Per-person write isolation inside a shared source, and a guide for putting gbrain behind a multi-user agent harness.**
|
||||
|
||||
Until now, `--source` was the only write boundary: a client could write anywhere inside the source it was scoped to, and keeping each person in their own folder was a convention the agent had to honor by itself. Registering a client with `--bound-slug-prefixes` now makes that boundary real. Writes outside the bound prefixes are refused by the server, on every op that can name a page.
|
||||
|
||||
**Adding a binding to an existing client narrows it on purpose.** Ops that write by something other than a page slug can't be confined to a prefix, so a bound client is refused them outright rather than left with an unfenced path: `extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`, and `POST /ingest`. `put_page`'s automatic fact extraction is skipped for the same reason — it writes to entity pages the caller never named. Reads are unaffected, and unbound clients behave exactly as before. The gate keys on "anything that is not a plain read", so an op added in a future release is refused to bound clients until it is explicitly fenced.
|
||||
|
||||
Both prefix spellings work: the `wiki/agents/alice/*` glob that `submit_agent` bindings already use, and the plainer `emp-alice/` form. Change a binding in place with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>` — existing tokens pick it up on their next request, so no secret rotation is needed when someone joins or leaves a team.
|
||||
|
||||
**New guide: [gbrain as the company brain for a qm deployment](docs/integrations/qm-harness.md).** qm is a multiplayer agent harness where each employee and each channel gets an isolated agent scope. The guide covers the whole path — one central `gbrain serve --http`, the thin-client binary baked into the sandbox image, one OAuth client per scope, and a roster-driven provisioning script that converges the brain to a list of people and channels. It also states plainly what the model does *not* give you: within a shared source, reads stay source-granular, so prefix isolation is a write boundary, not a privacy boundary.
|
||||
gbrain upgrade # or: bun install -g gbrain@0.42.72.0
|
||||
gbrain apply-migrations --yes # required: the fence refuses writes it cannot evaluate
|
||||
```
|
||||
|
||||
To fence an existing client to a folder:
|
||||
|
||||
```bash
|
||||
gbrain auth rescope-client <client_id> --bound-slug-prefixes partners/alice-example/
|
||||
gbrain auth rescope-client <client_id> --bound-slug-prefixes none # undo
|
||||
```
|
||||
|
||||
Verify it took, from a client holding that credential — the first write should succeed and the second should be refused:
|
||||
|
||||
```bash
|
||||
gbrain put partners/alice-example/notes/test --content "mine"
|
||||
gbrain put partners/bob-example/notes/test --content "not mine"
|
||||
```
|
||||
|
||||
## [0.42.71.0] - 2026-08-01
|
||||
|
||||
**GBrain now publishes real releases. Every version bump from here on lands on the [Releases page](https://github.com/garrytan/gbrain/releases) with organized notes and downloadable binaries — and binary self-update finally works.**
|
||||
|
||||
Until now the repo had no releases at all: `gbrain check-update` could tell you a new version existed, but `gbrain self-upgrade` downloaded from an empty releases API and failed every time, and anyone trying to follow what shipped had to read raw commit history. That's what people have been (rightly) complaining about.
|
||||
|
||||
From this release forward, every version bump automatically:
|
||||
|
||||
- **Tags the commit** (`v0.42.71.0`) so versions are addressable in git.
|
||||
- **Publishes a GitHub Release** whose notes are that version's CHANGELOG entry — the same organized, user-facing writeup, not a commit dump.
|
||||
- **Attaches compiled binaries** for macOS (arm64) and Linux (x64), so `gbrain self-upgrade` and fresh binary installs work without a toolchain.
|
||||
|
||||
The pipeline is idempotent: a partial release (tag exists, assets incomplete) is repaired on the next run instead of wedging. It runs post-merge, so a flaky release build can never turn master red. Releases for today's two fix waves (v0.42.69.0 and v0.42.70.0) have been backfilled with their CHANGELOG notes so the Releases page tells the whole story of the day; binaries attach from v0.42.71.0 onward.
|
||||
|
||||
### To take advantage of v0.42.71.0
|
||||
|
||||
```bash
|
||||
gbrain check-update # now resolves against real releases
|
||||
gbrain self-upgrade # now actually downloads a binary
|
||||
```
|
||||
|
||||
Or browse https://github.com/garrytan/gbrain/releases for organized per-version notes.
|
||||
|
||||
### For contributors
|
||||
|
||||
`docs/RELEASING.md` gains the release-publication section; `scripts/changelog-entry.sh` extracts a version's CHANGELOG section (used for release notes — keep entries under the standard `## [X.Y.Z.W]` headers and they publish verbatim). The workflow keeps all actions SHA-pinned, tightens top-level permissions to `contents: read` with write scoped to the release job only, and env-binds all interpolations.
|
||||
|
||||
Contributed by @time-attack (#3573, closing #3521).
|
||||
|
||||
## [0.42.70.0] - 2026-08-01
|
||||
|
||||
**Community fix wave two: 18 contributed fixes. The headline: several things you asked gbrain to do were being quietly ignored — and now they aren't.**
|
||||
|
||||
**`--brain` now actually routes.** The documented `gbrain query "X" --brain media-team` parsed the flag and then ran against your host brain anyway. It now routes to the named brain, and an unknown brain name fails loudly instead of silently answering from the wrong database.
|
||||
|
||||
**`sync --dry-run` no longer touches anything.** A dry run could pull from the remote and — if your sync strategy had changed — delete indexed pages before the "dry run" early-return was reached. Previews are now read-only, full stop.
|
||||
|
||||
**`apply-migrations --yes` applies.** It previously warned that your schema was behind and then printed "All migrations up to date" with exit 0. If you have wedged brains that upgrade never healed, this was why.
|
||||
|
||||
**Links between your pages resolve the way you write them.** Dir-qualified wikilinks with raw Obsidian names (`[[wiki/entities/AI 3.0]]`) now resolve to the sync-slugified page; references in non-whitelisted directories are no longer silently dropped; and a scan bug that could add an edge to a *parent* page you never referenced was caught in the wave's composite review and fixed before shipping.
|
||||
|
||||
**Windows and self-hosters.** Markdown files keep LF endings so frontmatter parsers stop mis-reading on Windows checkouts; the archive-crawler path gate no longer denies every real Windows path (and no longer fail-opens on NTFS case-insensitivity); a chat-synopsis tier that was hardcoded to one provider now follows your configured models; vector search asks the index for as many candidates as it was told to consider.
|
||||
|
||||
**Quieter, more honest infrastructure.** `serve --http` no longer leaves an orphan holding the database lock after Ctrl-C; a minion child that fails to launch settles immediately instead of hanging its slot; doctor gains checks for content-hash duplicates, undeclared database-only pages, stale heartbeats, and a tamper-evident manifest for the skills directory; federated reads respect per-source isolation settings in two more paths; and the security docs were rewritten to describe fixes without cataloguing attack surface.
|
||||
gbrain upgrade
|
||||
gbrain extract --stale # re-extracts links under the fixed resolver
|
||||
gbrain doctor # includes the new silent-failure checks
|
||||
```
|
||||
|
||||
If your brain uses `link_resolution.global_basename` and was populated before this release, a small number of superseded `wikilink_basename` edges can linger beside their newer typed replacements after re-extraction (edge writes are append-only by design). `gbrain reconcile-links` cleans them up; they are harmless to queries that dedup on target.
|
||||
|
||||
### For contributors
|
||||
|
||||
The composite review of this wave (two independent max-effort review passes over the combined branch) caught two interaction defects that per-PR review could not: the ungated bare-path scanner reading inside wikilink spans, and an extraction watermark set to a date that same-day stamps would already outrun. Both were fixed in the wave with discriminating tests. One reviewed-and-approved PR was deliberately held out: it conflicts semantically with its author's own sibling PR in this wave, and choosing between their two path-resolution mechanisms is the author's call.
|
||||
|
||||
Contributed by @time-attack (#3618, #3085, #3539, #3576, #3533, #3453, #3457, #3560, #3161), @daragao3 (#3619, #3536, #3517, #3578), @paul-0320 (#3613, #3564), @cvillarroel2 (#3678), @mamedov (#3624), @dialthewolff (#3550).
|
||||
|
||||
## [0.42.69.0] - 2026-08-01
|
||||
|
||||
**A community fix wave: 22 contributed fixes, most of them for work your brain was quietly not doing.**
|
||||
|
||||
The theme of this release is silent failure. A nightly cycle that reported `ok` while extracting nothing. An `embed` run that left a whole page unsearchable because one chunk in it failed, then exited 0. A health metric that recommended the same step forever because it counted one thing and the fix measured another. None of these looked broken from the outside, which is exactly why they lasted.
|
||||
|
||||
**If you run a local or non-Anthropic model, atom extraction was doing nothing.** With a cost cap set, any model absent from the pricing tables made the first work item hard-fail, which latched a budget flag and skipped every remaining item — while the phase still reported success. Local models (`ollama`, `llama-server`) now price at $0, because local inference costs electricity rather than tokens, so their caps stay enforceable. Genuinely unpriced paid providers still skip, but loudly now instead of silently.
|
||||
|
||||
**`gbrain embed` no longer lets one bad chunk darken an entire page,** and it exits non-zero when embeddings actually fail. If you have a cron wrapping `gbrain embed`, a brain holding permanently un-embeddable content will now turn that cron red. That is the intended change — it was previously green while silently incomplete.
|
||||
|
||||
**Non-Latin and diacritic names now survive mention extraction.** The by-mention tokenizer matched ASCII letters and digits only, so `Đà Nẵng` shredded into one- and two-character fragments and never matched anything. Names in Vietnamese, and any script outside ASCII, are now tokenized properly.
|
||||
|
||||
**Self-hosted embedding backends work.** Fixed-dimension OpenAI-compatible servers that reject an explicit `dimensions` parameter no longer get sent one when the requested width already matches the model's native width. A vector search on the embedded database also now asks the index for as many candidates as it was told to consider, instead of silently truncating the pool to the driver default.
|
||||
|
||||
**Multi-source brains route correctly in two more places.** A programmatic `sync_brain` call now syncs the source it was handed rather than the global default, and entity slug resolution keeps its path separators instead of flattening `people/alice-example` into an id no page can hold.
|
||||
|
||||
**Safer default on a destructive migration.** Submitting the type-unification job without an explicit `apply` now previews instead of applying. If you have that command in a runbook, add `"apply":true` — the playbooks and README were updated to show it.
|
||||
|
||||
Also: interrupted imports keep their tail instead of losing progress below the next 100-file boundary; `gbrain init --help` prints its own help instead of a stub; `doctor` stops reporting Windows drive paths as missing files under WSL and bounds its embedding health probe instead of retrying a permanent auth failure three times; cycle lock-release and stamp-write failures are visible instead of swallowed; and references to a `gbrain install` command that never existed are gone from the docs.
|
||||
|
||||
### To take advantage of v0.42.69.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade # or: bun install -g gbrain@0.42.69.0
|
||||
gbrain doctor # confirms the health metric now converges
|
||||
gbrain embed --stale # exits non-zero if anything is genuinely un-embeddable
|
||||
```
|
||||
|
||||
If you use a local chat model for the nightly cycle, re-run it once and check that atoms actually land:
|
||||
|
||||
```bash
|
||||
gbrain dream --json | jq '.phases[] | select(.name=="extract_atoms")'
|
||||
```
|
||||
|
||||
If you have `gbrain jobs submit unify-types` in a runbook or script, add `"apply":true` to its `--params` or it will now preview only.
|
||||
|
||||
### For contributors
|
||||
|
||||
Two defects existed only in the *combination* of otherwise-sound fixes, and were caught by reviewing the composed branch rather than the individual changes:
|
||||
|
||||
- `isModelPriceable` was introduced with a test asserting `llama-server` is unpriced, while a second fix in the same wave priced `llama-server` at $0. Together the assertion inverted. Reconciled by using a genuinely unpriced provider in the test and pinning the positive case: free local providers are priceable at $0, so their caps stay enforced.
|
||||
- The type-unification default flipped to dry-run, but three agent-facing playbooks still presented a bare submit as the apply step. Because a second fix in the same wave also edited one of those files, each change looked self-consistent alone. Skills ship downstream via the skillpack, so this would have propagated a playbook whose apply step silently did nothing.
|
||||
|
||||
One reviewed fix was deliberately held back: extending the inline subagent drain to Postgres composes badly with this wave's minion connection-recovery work, since the drain calls the same queue operations without the new recovery path and can strand a child job in a per-run queue no worker will claim.
|
||||
|
||||
Contributed by @alexey-metaengage (#3652), @time-attack (#3568, #3572, #3567, #3555, #3523, #3144, #3574, #3532, #3545), @brettdavies (#3552, #3553), @mattchronicle (#3364), @rayers (#3589), @zenspam (#3699), @awilhite (#3691), @Grimnoth (#3541), @georgell-ceo (#3634), @Vyacheslav-Zakharov (#3631), @Kyzcreig (#3585), @HammerTech-Z (#3581), @cfeddersen (#3563).
|
||||
|
||||
## [0.42.68.1] - 2026-07-30
|
||||
|
||||
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
|
||||
|
||||
The embedded database allows one process at a time, and holds a lock to enforce it. These two commands opened a second connection to the same database from inside the process that already held that lock, then waited for a lock that could never be released — because the thing holding it was the waiting process itself. The wait ran its full 30 seconds and the command exited with an error naming a blocking process that was, in fact, itself. Both commands now reuse the connection that is already open.
|
||||
|
||||
Nothing changes for brains on Postgres, where a second connection was always allowed.
|
||||
|
||||
## To take advantage of v0.42.68.1
|
||||
|
||||
Nothing to undo — the commands failed without writing anything. Just run whichever you needed:
|
||||
```bash
|
||||
gbrain reindex-frontmatter
|
||||
```
|
||||
|
||||
## [0.42.67.0] - 2026-07-28
|
||||
|
||||
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
|
||||
|
||||
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md` — and the five frozen MEMORY_VERBS `recall`/`remember`/`entity`/`synthesize`/`forget`, servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -119,12 +119,14 @@ detail on demand.)
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
| memory verbs / MCP tool surface (`--surface`) / conformance | `docs/protocol/MEMORY_VERBS_v1.md` + the `verbs*`/`surface.ts`/`protocol.ts` entries in `KEY_FILES.md` |
|
||||
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
|
||||
| running or writing tests | `docs/TESTING.md` |
|
||||
| bulk-command progress wiring | `docs/progress-events.md` |
|
||||
| eval methodology / metrics | `docs/eval/` |
|
||||
| brains vs sources / topology | `docs/architecture/brains-and-sources.md`, `topologies.md` |
|
||||
| skill routing | `skills/RESOLVER.md` |
|
||||
| agent bootstrap (paste-in install, hooks, `gbrain bootstrap`, sweep, keyless) | `docs/guides/bootstrap.md` + `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + the KEY_FILES bootstrap cluster |
|
||||
| shipping a release / CHANGELOG / PR conventions | `docs/RELEASING.md` (ship IRON RULES stay inline below) |
|
||||
|
||||
The per-file index (`## Key files`), the thin-client routing seam, and the testing
|
||||
@@ -190,9 +192,10 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix
|
||||
has three verbatim homes: this section, the `gbrain init` picker copy
|
||||
(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step
|
||||
3.5** — update all three when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
@@ -272,8 +275,9 @@ audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills
|
||||
(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md`
|
||||
(`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
briefing, migrate, setup, publish.
|
||||
|
||||
+80
-82
@@ -19,14 +19,20 @@ The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
|
||||
correct with no extra steps.
|
||||
|
||||
If you cloned before that pin existed, your working copy still has the old
|
||||
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
|
||||
it once, from the repository root:
|
||||
`.gitattributes` pins `*.md text eol=lf` for the same reason. The frontmatter
|
||||
readers anchor on a `---` fence followed by a Unix line ending, so a CRLF
|
||||
checkout makes a well-formed document parse as having no frontmatter. That
|
||||
failure is silent: no error, the field just comes back empty.
|
||||
|
||||
If you cloned before either pin existed, your working copy still has the old
|
||||
Windows line endings. Bash will fail with `$'\r': command not found`, and
|
||||
frontmatter will read as absent. Refresh it once, from the repository root:
|
||||
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
|
||||
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
|
||||
git ls-files --eol -- '*.md' | grep -c w/crlf # 0 means Markdown is clean
|
||||
```
|
||||
|
||||
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
|
||||
@@ -42,7 +48,9 @@ src/
|
||||
core/
|
||||
operations.ts Contract-first operation definitions (the foundation)
|
||||
engine.ts BrainEngine interface
|
||||
postgres-engine.ts Postgres implementation
|
||||
engine-factory.ts Engine factory (dynamic import of the configured engine)
|
||||
postgres-engine.ts Postgres + pgvector implementation
|
||||
pglite-engine.ts PGLite (embedded Postgres via WASM) implementation
|
||||
db.ts Connection management + schema loader
|
||||
import-file.ts Import pipeline (chunk + embed + tags)
|
||||
types.ts TypeScript types
|
||||
@@ -53,12 +61,16 @@ src/
|
||||
supabase-admin.ts Supabase admin API
|
||||
file-resolver.ts MIME detection + content hashing
|
||||
migrate.ts Migration helpers
|
||||
bootstrap/ Agent-bootstrap flow (interview, hooks, repo, verify)
|
||||
yaml-lite.ts Lightweight YAML parser
|
||||
chunkers/ 3-tier chunking (recursive, semantic, llm)
|
||||
search/ Hybrid search (vector, keyword, hybrid, expansion, dedup)
|
||||
embedding.ts OpenAI embedding service
|
||||
embedding.ts Embedding service (provider-routed; ZeroEntropy default)
|
||||
mcp/
|
||||
server.ts MCP stdio server (generated from operations)
|
||||
http-transport.ts HTTP MCP transport (OAuth, body caps)
|
||||
dispatch.ts Op dispatch + scope enforcement + param redaction
|
||||
rate-limit.ts Rate limiting
|
||||
schema.sql Postgres DDL
|
||||
skills/ Fat markdown skills for AI agents
|
||||
test/ Unit tests (bun test, no DB required)
|
||||
@@ -71,15 +83,21 @@ test/e2e/ E2E tests (requires DATABASE_URL, real Postgres+pgvect
|
||||
docs/ Architecture docs
|
||||
```
|
||||
|
||||
Per-file invariants live in `docs/architecture/KEY_FILES.md` — read a file's entry
|
||||
before editing it.
|
||||
|
||||
## Running tests
|
||||
|
||||
The canonical reference for test tiers, isolation rules, timing, and the E2E
|
||||
lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version:
|
||||
|
||||
```bash
|
||||
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
|
||||
bun run test # parallel 8-shard fan-out + serial post-pass
|
||||
# Inner edit loop (~85s on a Mac dev box)
|
||||
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
|
||||
bun test test/markdown.test.ts # specific unit test
|
||||
|
||||
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
|
||||
bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck
|
||||
# Pre-push gate (19+ parallel checks + typecheck)
|
||||
bun run verify
|
||||
|
||||
# Pre-merge sanity (everything CI runs)
|
||||
bun run test:full # verify + parallel unit + slow + smart e2e
|
||||
@@ -97,92 +115,54 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
|
||||
DATABASE_URL=postgresql://... bun run test:e2e
|
||||
```
|
||||
|
||||
Use `bun run verify` before pushing. The guard chain catches: banned fork-name
|
||||
leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
|
||||
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
|
||||
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
|
||||
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
|
||||
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
|
||||
(`scripts/check-progress-to-stdout.sh`), test-isolation rule violations
|
||||
(`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel
|
||||
loop" below), silent fallback to recursive chunking in the compiled binary
|
||||
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
|
||||
(`scripts/check-admin-build.sh`), and resolver drift on bundled skills
|
||||
(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any
|
||||
warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift
|
||||
before merge). `bun run check:all` runs the full historical sweep including the
|
||||
trailing-newline and exports-count checks.
|
||||
(`scripts/check-admin-build.sh`), resolver drift on bundled skills
|
||||
(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full
|
||||
historical sweep including the trailing-newline and exports-count checks.
|
||||
|
||||
### Writing tests that survive the parallel loop
|
||||
|
||||
`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the
|
||||
same shard share a process, so process-global state leaks between them. Four
|
||||
lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
|
||||
`bun run test` shards 1000+ unit-test files across up to 4 worker processes,
|
||||
capping total concurrency (shards × intra-shard files) to available memory and
|
||||
re-running OOM-killed or externally-killed files serially before calling them
|
||||
failures (see `docs/TESTING.md` for the rescue-pass details and knobs). Files
|
||||
in the same shard share a process, so process-global state leaks between them.
|
||||
Four lint rules (`scripts/check-test-isolation.sh`, R1–R4) enforce isolation:
|
||||
no direct `process.env` mutation (use `withEnv()` from
|
||||
`test/helpers/with-env.ts`), no `mock.module(...)` outside `*.serial.test.ts`,
|
||||
and every `new PGLiteEngine(` goes inside the canonical `beforeAll` block with
|
||||
a paired `afterAll(disconnect)`.
|
||||
|
||||
| Rule | What it bans | Fix |
|
||||
|---|---|---|
|
||||
| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` |
|
||||
| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` |
|
||||
| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) |
|
||||
| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` |
|
||||
**The full rules, the canonical PGLite block, the `withEnv` pattern, and the
|
||||
`*.serial.test.ts` quarantine policy live in
|
||||
[`docs/TESTING.md`](docs/TESTING.md#test-isolation-lint-and-helpers)
|
||||
— read that before writing a new test file.** Files that predate the rules are
|
||||
listed in `scripts/check-test-isolation.allowlist`; the allow-list MUST shrink
|
||||
over time — never add new entries.
|
||||
|
||||
Canonical PGLite block (R3 + R4 compliant — paste this verbatim):
|
||||
|
||||
```ts
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
afterAll(async () => { await engine.disconnect(); });
|
||||
beforeEach(async () => { await resetPgliteState(engine); });
|
||||
```
|
||||
|
||||
Env-touching tests:
|
||||
|
||||
```ts
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
test('reads OPENAI_API_KEY', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
|
||||
expect(loadConfig().openai_key).toBe('sk-test');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
`withEnv` saves and restores keys via try/finally including when the callback
|
||||
throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is
|
||||
process-global). Files using `withEnv` stay outside the future
|
||||
`test.concurrent()` codemod's eligibility filter.
|
||||
|
||||
When to quarantine instead of fix: rename to `*.serial.test.ts` if the file
|
||||
uses `mock.module(...)`, is genuinely env-coupled (module-load env readers +
|
||||
ESM caching defeat dynamic-import-after-env tricks), or intentionally shares
|
||||
state across `it()` boundaries. Quarantine count cap: 10 (informational).
|
||||
|
||||
Files that violated these rules at the v0.26.7 baseline are listed in
|
||||
`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over
|
||||
time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep
|
||||
+ codemod) remove entries as files get fixed.
|
||||
|
||||
### Local CI gate (recommended before pushing, v0.23.1+)
|
||||
### Local CI gate (recommended before pushing)
|
||||
|
||||
```bash
|
||||
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
|
||||
bun run ci:local # full gate: gitleaks + guards/typecheck + 4-shard parallel unit + E2E
|
||||
bun run ci:local:diff # gate with diff-aware E2E selector
|
||||
bun run ci:select-e2e # print which E2E files the selector would run
|
||||
```
|
||||
|
||||
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
|
||||
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
|
||||
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
|
||||
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
|
||||
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
|
||||
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
`ci:local` spins up four pgvector services plus a transaction-mode PgBouncer via
|
||||
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite
|
||||
sharded 4 ways in parallel, then tears down. Named volumes keep the install warm
|
||||
across runs. Requires Docker (Docker Desktop, OrbStack, or Colima) and `gitleaks`
|
||||
on host (`brew install gitleaks`). Override the postgres host port with
|
||||
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
|
||||
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
|
||||
Fail-closed selector: an unmapped `src/` change runs ALL E2E files. Hand-tune
|
||||
narrower mappings via `scripts/e2e-test-map.ts`.
|
||||
|
||||
### PR-side security checks
|
||||
@@ -211,6 +191,15 @@ automatically appears in the CLI, MCP server, and tools-json:
|
||||
For CLI-only commands (init, upgrade, import, export, files, embed, doctor, sync):
|
||||
1. Create `src/commands/mycommand.ts`
|
||||
2. Add the case to `src/cli.ts`
|
||||
3. Regenerate the flag registry: `bun run build:flag-registry`. The CLI rejects
|
||||
unknown flags before dispatch; each CLI-only command's legal flag set is
|
||||
derived from its source into `src/core/cli-flag-registry.generated.ts`.
|
||||
`test/cli-flag-validation.test.ts` pins registry freshness, drift, and
|
||||
consumption evidence (a safety flag like `--dry-run` may only be advertised
|
||||
if the command's code actually reads it), so a stale registry fails the
|
||||
build. At runtime a missing registry entry fails open — a forgotten regen
|
||||
never bricks a command. Rerun the regen whenever you add or remove a flag
|
||||
on an existing command, too.
|
||||
|
||||
Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
|
||||
|
||||
@@ -219,7 +208,7 @@ Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
|
||||
See `docs/ENGINES.md` for the full guide. In short:
|
||||
|
||||
1. Create `src/core/myengine-engine.ts` implementing `BrainEngine`
|
||||
2. Add to engine factory in `src/core/engine.ts`
|
||||
2. Add to the engine factory in `src/core/engine-factory.ts`
|
||||
3. Run the test suite against your engine
|
||||
4. Document in `docs/`
|
||||
|
||||
@@ -304,7 +293,7 @@ NDJSON wire format is documented in
|
||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||
|
||||
For public benchmark coverage on top of replay, `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
|
||||
<dataset.jsonl>` runs LongMemEval against gbrain's hybrid
|
||||
retrieval. One in-memory PGLite per question, runtime-enumerated
|
||||
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
|
||||
published `evaluate_qa.py`. Use it alongside replay when changes affect
|
||||
@@ -313,9 +302,18 @@ regressions on YOUR queries, LongMemEval catches them on a public set the
|
||||
benchmark community already cites. See the "Public benchmarks: LongMemEval"
|
||||
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
|
||||
## Shipping
|
||||
|
||||
Releases go through the `/ship` skill, never hand-rolled. The full release +
|
||||
contributor process (CHANGELOG voice, version-locations sync, PR conventions,
|
||||
community-PR-wave workflow) lives in [`docs/RELEASING.md`](docs/RELEASING.md).
|
||||
Community PRs are batched into release waves rather than merged one-by-one;
|
||||
contributor attribution stays attached via `Co-Authored-By:` trailers and every
|
||||
accepted contribution is credited in `CHANGELOG.md`.
|
||||
|
||||
## Welcome PRs
|
||||
|
||||
- SQLite engine implementation
|
||||
- Additional engine implementations (see [`docs/ENGINES.md`](docs/ENGINES.md))
|
||||
- Docker Compose for self-hosted Postgres
|
||||
- Additional migration sources
|
||||
- New enrichment API integrations
|
||||
|
||||
+30
-5
@@ -3,6 +3,11 @@
|
||||
Read this entire file, then follow the steps. Ask the user for API keys when needed.
|
||||
Target: ~30 minutes to a fully working brain.
|
||||
|
||||
> **Scope note:** this guide installs the BRAIN (for agent platforms like OpenClaw
|
||||
> and Hermes, or a standalone brain). If you are Claude Code or Codex and the human
|
||||
> asked you to become their persistent personal agent — identity + memory + private
|
||||
> repo — follow `BOOTSTRAP_FOR_AGENTS.md` instead.
|
||||
|
||||
## Step 0: If you are not Claude Code
|
||||
|
||||
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
|
||||
@@ -16,6 +21,16 @@ If you fetched this file by URL without cloning yet, the companion files live at
|
||||
|
||||
## Step 1: Install GBrain
|
||||
|
||||
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
|
||||
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
|
||||
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
|
||||
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
|
||||
> supported sources are `github:garrytan/gbrain` (optionally pinned as
|
||||
> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a
|
||||
> git clone, exactly as shown below.
|
||||
> If an unrelated npm install is already present, remove it first
|
||||
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
|
||||
|
||||
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
|
||||
|
||||
```bash
|
||||
@@ -81,6 +96,7 @@ default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
<!-- Cost matrix: three verbatim homes — CLAUDE.md "Search Mode", src/commands/init-mode-picker.ts, and this block. Sync all three when refreshing. -->
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
@@ -191,13 +207,13 @@ scaffold the bundled skills into it:
|
||||
|
||||
```bash
|
||||
cd /path/to/agent/workspace
|
||||
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
|
||||
gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md
|
||||
```
|
||||
|
||||
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
|
||||
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
|
||||
diff against gbrain's bundle when you want upstream improvements. (The legacy
|
||||
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
|
||||
`gbrain skillpack install` managed-block model was removed in v0.33 — run
|
||||
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
|
||||
|
||||
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
|
||||
@@ -254,8 +270,17 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
|
||||
|
||||
## Step 9: Verify
|
||||
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
|
||||
actually works) is the most important.
|
||||
Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4
|
||||
(live sync actually works) is the most important.
|
||||
|
||||
Once verification passes and the brain has content, run the activation probe:
|
||||
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
|
||||
See "The onboard surface" below for what the recommendations mean and the
|
||||
consent gates around unattended remediation.
|
||||
|
||||
## Upgrade
|
||||
|
||||
@@ -298,7 +323,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
|
||||
## v0.42.0+ onboard surface (NEW)
|
||||
## The onboard surface
|
||||
|
||||
`gbrain onboard` is the activation surface gbrain did not have before.
|
||||
Once your brain has any content, run `gbrain onboard --check --json` to
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Search gives you raw pages. GBrain gives you the answer.** It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box. Run a full autonomous agent on top of it, or just wire it into Claude Code or Codex as a supercharged retrieval layer in one command; either way your coding agent stops being amnesiac about everything that isn't code.
|
||||
|
||||
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
|
||||
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **155,795 pages, 24,589 people, 5,340 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
|
||||
|
||||
**And now it works as a company brain too.** Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the [company-brain](https://www.ycombinator.com/rfs#company-brain) shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. **[Tutorial: set up GBrain as your company brain →](docs/tutorials/company-brain.md)**
|
||||
|
||||
@@ -11,7 +11,7 @@ Lots of personal-knowledge systems give you keyword matching and grep in a box.
|
||||
- **A synthesis layer that gives you the actual answer.** Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
|
||||
- **A self-wiring knowledge graph.** Every page write extracts entity refs and creates typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, **+31.4 points P@5** over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
|
||||
The point of building a 150K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
|
||||
|
||||
It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.
|
||||
|
||||
@@ -65,11 +65,51 @@ This is the difference between a search engine and a brain. Search finds the pag
|
||||
|
||||
## Install
|
||||
|
||||
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
|
||||
> [!WARNING]
|
||||
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
|
||||
> package with no connection to this project. Do not run `npm install -g gbrain` or
|
||||
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
|
||||
> your PATH. Install and upgrade ONLY via the documented paths below
|
||||
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
|
||||
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
|
||||
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
|
||||
> shadowing npm install and prints the fix.
|
||||
|
||||
### Have your agent install it (recommended)
|
||||
GBrain is designed to be installed and operated by an AI agent. **New to GBrain? Start with Codex** — it runs on the ChatGPT subscription you already have, takes ~15 minutes, and deploys nothing. Already living in Claude Code? Its path is identical. Want GBrain running the way it was designed to run — always on, enriching your brain around the clock? That's OpenClaw or Hermes, at real server + API cost. Each path below is complete on its own. (Wiring it up by hand instead? Jump to [CLI standalone](#cli-standalone-no-agent) or the [MCP table](#connect-gbrain-to-your-ai-client-mcp).)
|
||||
|
||||
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
|
||||
### For Codex — the recommended first step
|
||||
|
||||
Turn Codex into your persistent personal agent. Works in the **ChatGPT desktop app** (open Codex on a folder) and in the **Codex CLI** (`codex` in a terminal) — same install, same result. Pick the folder that will become your agent's home, and paste:
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
Codex will ask for command approvals during the install — approving them is the sandbox working as intended. What you get, in about 15 minutes: a short interview (6 required questions) → your agent's identity (SOUL.md, USER.md, MEMORY.md) rendered from your own answers, never invented → a local PGLite brain (2 seconds, no server, no Docker) → MCP wired so every session can search and write memory → a **private** GitHub repo, created and privacy-verified, as your agent's durable body. Works with **zero API keys** — keyword search plus memory your agent writes itself; one optional key (OpenAI, Anthropic, or Voyage) upgrades to semantic search and automatic fact extraction. Codex reads brain context through its tools each turn (pull-based).
|
||||
|
||||
### For Claude Code — turn it into your persistent personal agent
|
||||
|
||||
Works in the **desktop app** and in the **CLI** (`claude` in a terminal) — identical harness, identical result. Open Claude Code in the folder that will become your agent's home, and paste the same block:
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
Everything from the Codex path applies — interview, identity from your own answers, local brain, private repo, keyless mode — plus Claude Code gets **per-turn context hooks** (consent-gated): your brain loads automatically into every prompt, and each session persists itself to your private repo at exit. Restart the session after install and ask "what did I tell you my top jobs were?" — that's the moment it clicks. Full contract, security posture, and uninstall: [docs/guides/bootstrap.md](docs/guides/bootstrap.md).
|
||||
|
||||
### For OpenClaw or Hermes — GBrain as intended, always on
|
||||
|
||||
This is GBrain used the way it was designed to be used: a server-hosted agent with 24/7 crons, continuous ingestion, and the overnight dream cycle that enriches your brain while you sleep — your agent works whether your laptop is open or not. It's also the highest-cost path: a deployed server (8GB+ RAM) plus raw API token usage that scales with how hard your agent runs, well beyond a chat subscription. Start here if you want the full experience from day one; start with Codex above if you want to feel it first. If you don't have a platform running yet, both deploy in one click:
|
||||
|
||||
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
|
||||
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
|
||||
@@ -81,21 +121,21 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
|
||||
The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
|
||||
|
||||
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
|
||||
|
||||
### Quick start: Claude Code or Codex
|
||||
### Lighter ways in
|
||||
|
||||
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
|
||||
|
||||
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel:
|
||||
**Just want a memory for your coding agent — no identity, no repo.** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full tool wall; drop the flag for every operation:
|
||||
|
||||
```bash
|
||||
gbrain init --pglite # 2-second local brain (no Docker)
|
||||
claude mcp add gbrain -- gbrain serve # or: codex mcp add gbrain -- gbrain serve
|
||||
gbrain init --pglite # 2-second local brain (no Docker)
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs # or: codex mcp add gbrain -- gbrain serve --surface verbs
|
||||
```
|
||||
|
||||
If `claude` is not found, install Claude Code first — or use the per-harness blocks in the [protocol doc](docs/protocol/MEMORY_VERBS_v1.md). Heads-up: memories agents save default to brain-wide visibility (every connected agent can recall them); pass `visibility: "private"` for local-only facts.
|
||||
|
||||
**Already have a brain on a remote host** (OpenClaw, Hermes, or any `gbrain serve --http`)? Point your laptop agents at it with one command each — `--install` wires it up and smoke-tests the token before handoff:
|
||||
|
||||
```bash
|
||||
@@ -103,18 +143,9 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --install
|
||||
gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install # Codex
|
||||
```
|
||||
|
||||
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — both paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
|
||||
**Brain-only install into another coding agent** (Cursor, Claude Cowork, or anything that can fetch a URL and run shell commands) — paste the OpenClaw/Hermes block above (`INSTALL_FOR_AGENTS.md`); it installs the brain, skills, and dream cycle without the personal-agent identity layer. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
|
||||
|
||||
### Install the full autonomous setup into your existing agent
|
||||
|
||||
Want the whole thing — local brain, 43 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
|
||||
|
||||
```
|
||||
Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
|
||||
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — the memory-only paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
|
||||
|
||||
### CLI standalone (no agent)
|
||||
|
||||
@@ -130,7 +161,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
|
||||
|
||||
### Connect GBrain to your AI client (MCP)
|
||||
|
||||
GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:
|
||||
GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side) — or exactly the five memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
|
||||
|
||||
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
|
||||
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
|
||||
@@ -166,7 +197,7 @@ gbrain think "who's working on AI agents at portfolio companies?"
|
||||
|
||||
**`gbrain think`** runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.
|
||||
|
||||
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.
|
||||
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 150K-page brain is worth the effort.
|
||||
|
||||
`gbrain agent run "..."` exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.
|
||||
|
||||
@@ -206,7 +237,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
|
||||
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
|
||||
|
||||
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
|
||||
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
|
||||
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
|
||||
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
|
||||
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
|
||||
|
||||
@@ -252,7 +283,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
## Capabilities
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
@@ -276,9 +307,9 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. `gbrain eval brainbench` runs the cross-harness memory conformance suite: know-to-ask, push precision/recall, write-back fidelity, and cross-session continuity, scored per harness seam (your OpenClaw's production pipeline plus Claude Code and Codex injection contracts) against a committed 141-fixture synthetic corpus — hermetic by default (in-memory PGLite, no keys, seconds), and CI gates every PR against master's committed baseline. Methodology in [`docs/eval/BRAINBENCH.md`](docs/eval/BRAINBENCH.md); search-mode methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
|
||||
|
||||
@@ -290,14 +321,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
|
||||
|
||||
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
|
||||
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
|
||||
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
|
||||
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
|
||||
- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
|
||||
- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
|
||||
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
|
||||
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
|
||||
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source.
|
||||
|
||||
**Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md).
|
||||
|
||||
@@ -307,14 +338,13 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
|
||||
**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
|
||||
|
||||
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
|
||||
|
||||
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
|
||||
two flags + a recommended pattern. Switch your cron to a per-source loop
|
||||
with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating
|
||||
gracefully half-a-minute earlier:
|
||||
**Hourly cron sync keeps timing out on a federated brain?** Switch your
|
||||
cron to a per-source loop with shell `timeout(1)` doing the OS-level kill
|
||||
and gbrain self-terminating gracefully half-a-minute earlier:
|
||||
|
||||
```bash
|
||||
gbrain sync --break-lock --all --max-age 1800
|
||||
@@ -327,19 +357,17 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status
|
||||
`partial` and `last_commit` UNCHANGED — the next run re-walks the same
|
||||
diff and `content_hash` short-circuits already-imported files. The
|
||||
`--max-age 1800` first command self-heals any wedged-but-alive locks
|
||||
left by a hung previous run, using the v98 `last_refreshed_at` semantic
|
||||
(NOT `acquired_at`) so healthy long-running holders are safe by
|
||||
construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md)
|
||||
for the honest scope notes (extract + embed phases run to completion;
|
||||
30-min rollout window for `--max-age` post-migration v98; full-sync
|
||||
triggers deferred to v0.42+).
|
||||
left by a hung previous run, keyed on the lock's last refresh time
|
||||
(NOT when it was acquired) so healthy long-running holders are safe by
|
||||
construction. Scope note: the extract + embed phases still run to
|
||||
completion once started; `--timeout` interrupts the import walk only.
|
||||
|
||||
**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes
|
||||
the bug class structurally. The engine now self-retries every bulk batch
|
||||
write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on
|
||||
Supavisor pooler blips, with a 12s worst-case wait that covers the full
|
||||
5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents
|
||||
via the new `batch_retry_health` check (reads the last 24h of
|
||||
**Dream cycle silently losing wiki links on Supabase?** The engine
|
||||
self-retries every bulk batch write (`addLinksBatch` /
|
||||
`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips,
|
||||
with a 12s worst-case wait that covers the full 5-10s circuit-breaker
|
||||
recovery window. `gbrain doctor` surfaces incidents via the
|
||||
`batch_retry_health` check (reads the last 24h of
|
||||
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually
|
||||
slow pooler:
|
||||
|
||||
@@ -357,34 +385,33 @@ retry wrap is engine-level, but PGLite has no pooler so retries never
|
||||
fire in practice.
|
||||
|
||||
**Dream cycle losing ~150 link rows per run with `'No database
|
||||
connection: connect() has not been called'` errors in the log?** v0.41.27.0
|
||||
makes the retry layer self-heal on a nulled-out database singleton. A
|
||||
new `reconnect` callback on `withRetry` rebuilds the connection between
|
||||
attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()`
|
||||
connection: connect() has not been called'` errors in the log?** The
|
||||
retry layer self-heals on a nulled-out database singleton: a
|
||||
`reconnect` callback on `withRetry` rebuilds the connection between
|
||||
attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()`
|
||||
so engine-level batch writes survive a mid-cycle disconnect by something
|
||||
else in the same process. Same release: `gbrain capture` no longer trails
|
||||
a `'No database connection'` stderr line from a background facts:absorb
|
||||
worker firing after CLI exit — the op-dispatch finally block awaits
|
||||
else in the same process. `gbrain capture` also no longer trails a
|
||||
`'No database connection'` stderr line from a background facts:absorb
|
||||
worker firing after CLI exit — op dispatch awaits
|
||||
`getFactsQueue().drainPending({timeout: 1000})` before
|
||||
`engine.disconnect()`. To find which code path is still calling
|
||||
disconnect mid-process, run `gbrain doctor --json | jq '.checks[] |
|
||||
select(.id=="batch_retry_health")'`; the extended check now surfaces
|
||||
24h disconnect-call count and the most-recent caller frame from a new
|
||||
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.)
|
||||
select(.id=="batch_retry_health")'`; the check surfaces the
|
||||
24h disconnect-call count and the most-recent caller frame from the
|
||||
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit.
|
||||
|
||||
**`gbrain brainstorm` returning `judge_failed: true` with 0 scored
|
||||
ideas?** v0.41.21.0 closes the two bugs that caused it. The judge
|
||||
hard-coded a 4K-token output cap; for any run past ~40 ideas the call
|
||||
truncated mid-JSON and the parser threw. Same release closes a slash-
|
||||
form pricing miss: `gbrain brainstorm --judge-model
|
||||
anthropic/claude-sonnet-4-6 --max-cost 5` failed with
|
||||
`BudgetExhausted reason=no_pricing` because every pricing site only
|
||||
matched the colon form. Both shapes work now. No config change, no
|
||||
schema migration — `gbrain upgrade` is the whole fix.
|
||||
ideas?** Two historical bugs caused it, both fixed: the judge
|
||||
hard-coded a 4K-token output cap (any run past ~40 ideas truncated
|
||||
mid-JSON and the parser threw), and slash-form model ids
|
||||
(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
|
||||
--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because
|
||||
pricing lookups only matched the colon form. Both shapes work now. No
|
||||
config change, no schema migration — `gbrain upgrade` is the whole fix.
|
||||
|
||||
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
|
||||
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
|
||||
`reindex --markdown` now ADD current frontmatter tags and never delete,
|
||||
tags?** Upgrade — tag reconciliation is add-only now. Re-import and
|
||||
`reindex --markdown` ADD current frontmatter tags and never delete,
|
||||
so enrichment tags written to the DB (auto-tag, dream synthesize,
|
||||
signal-detector) survive a re-chunk. The reindex DB-only fallback also
|
||||
reconstructs the full markdown (frontmatter + body + timeline) before
|
||||
@@ -392,10 +419,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter,
|
||||
title, and timeline instead of getting overwritten with empty
|
||||
frontmatter. Trade-off: removing a tag from a page's frontmatter no
|
||||
longer removes it from the DB on the next sync (frontmatter-tag removal
|
||||
needs a provenance column, deferred). (Closes #1621.)
|
||||
needs a provenance column, deferred).
|
||||
|
||||
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
|
||||
v0.41.37.0 ships three things. First, name the stalling file:
|
||||
Three tools. First, name the stalling file:
|
||||
|
||||
```bash
|
||||
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
|
||||
@@ -410,28 +437,28 @@ the sync with the pack disabled and re-run extraction later:
|
||||
gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
|
||||
`gbrain schema lint` warns on the classic nested-quantifier ReDoS
|
||||
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
|
||||
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
|
||||
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
|
||||
PGLite is single-writer and a live MCP server contends for the write
|
||||
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
|
||||
for the full triage. (Closes #1569.)
|
||||
for the full triage.
|
||||
|
||||
**`gbrain init --migrate-only` / a schema migration fails on Windows
|
||||
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
|
||||
with `getaddrinfo ENOTFOUND`?** Upgrade — schema bring-up now runs its
|
||||
phases in-process instead of spawning a child `gbrain init
|
||||
--migrate-only` per phase. The spawned child died on
|
||||
Windows + bun + Supabase pooler with a DNS-resolution failure even
|
||||
though the parent connected fine; running in-process removes the spawn
|
||||
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
|
||||
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
|
||||
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
|
||||
completes in ~1-2 seconds. (Closes #1605, #1581.)
|
||||
entirely. The grandfather migration that used to hang 70+ minutes on an
|
||||
80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on
|
||||
the page PK, soft-delete-filtered, source-safe) and completes in seconds.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
|
||||
- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall)
|
||||
- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
|
||||
- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows`
|
||||
- [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory
|
||||
|
||||
+18
-23
@@ -34,15 +34,18 @@ enforced structurally by actionlint on every workflow change.
|
||||
|
||||
## Remote MCP Security
|
||||
|
||||
### ⚠️ Do NOT use open OAuth client registration for remote MCP
|
||||
### Keep dynamic client registration disabled unless explicitly needed
|
||||
|
||||
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
|
||||
support, **never allow unauthenticated client registration**. An attacker
|
||||
who discovers your server URL can:
|
||||
GBrain disables Dynamic Client Registration (DCR) by default. Keep that
|
||||
default for internet-reachable deployments and pre-register trusted clients
|
||||
with operator-approved scopes and source access. Enabling DCR lets network
|
||||
callers create OAuth client records, so use it only when the deployment's
|
||||
trust model requires self-service registration and browser approval remains
|
||||
part of the authorization flow.
|
||||
|
||||
1. Register a new OAuth client via `POST /register`
|
||||
2. Use `client_credentials` grant to obtain a bearer token
|
||||
3. Access all brain data via the MCP tools
|
||||
Do not enable `--enable-dcr-insecure` on an untrusted network. That option is
|
||||
reserved for deployments that intentionally allow self-registered
|
||||
machine-to-machine clients without browser approval.
|
||||
|
||||
### Recommended: `gbrain serve --http`
|
||||
|
||||
@@ -104,12 +107,10 @@ Auth methods (`--token-endpoint-auth-method`):
|
||||
- `none` — public PKCE-only client (no secret minted; ChatGPT custom
|
||||
connector, Claude Code, Cursor)
|
||||
|
||||
The validator rejects unknown methods at the registration boundary, and
|
||||
the same gate applies to the admin endpoint `POST /admin/api/register-client`
|
||||
and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
|
||||
`redirect_uris = []` and `token_endpoint_auth_method = NULL`, forcing
|
||||
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
|
||||
without `--enable-dcr`. That footgun is gone.
|
||||
The same validator applies to CLI, admin, and DCR registration paths, so
|
||||
unknown authentication methods are rejected consistently. Browser-based
|
||||
clients can be configured entirely through the supported CLI flags; operators
|
||||
do not need to edit OAuth database rows by hand.
|
||||
|
||||
### DCR consent default (v0.42.55+)
|
||||
|
||||
@@ -186,16 +187,10 @@ When the request `Origin` matches the allowlist, the server echoes it
|
||||
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
|
||||
CORS header is sent and the browser blocks the request.
|
||||
|
||||
**v0.41.3:** the same allowlist now gates every OAuth endpoint (`/mcp`,
|
||||
`/token`, `/authorize`, `/register`, `/revoke`). Pre-v0.41.3 these used
|
||||
default-wide-open `cors()` middleware, leaking
|
||||
`Access-Control-Allow-Origin: *` on every response — any web origin could
|
||||
complete a token exchange from a logged-in operator's browser. The CORS
|
||||
preflight handler in the legacy bearer transport was also asymmetric
|
||||
(actual-request path correctly default-deny, but OPTIONS preflight leaked
|
||||
`Access-Control-Allow-Methods` + `Access-Control-Allow-Headers` to every
|
||||
Origin); both are now consolidated through a single allowlist-gated path.
|
||||
A startup stderr WARN fires when `--bind 0.0.0.0` is set without
|
||||
The same allowlist gates the complete MCP and OAuth HTTP surface. Actual
|
||||
requests and browser preflight requests use one allowlist-gated policy, so
|
||||
unlisted origins receive no cross-origin authorization. A startup stderr
|
||||
warning fires when `--bind 0.0.0.0` is set without
|
||||
`GBRAIN_HTTP_CORS_ORIGIN`, surfacing the default-deny posture before the
|
||||
first request.
|
||||
|
||||
|
||||
@@ -1,5 +1,211 @@
|
||||
# TODOS
|
||||
|
||||
## BrainBench follow-ups (filed v0.44.0.0, Cathedral 2)
|
||||
|
||||
Deferred from the BrainBench wave (eng-reviewed; plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-bright-firefly.md`).
|
||||
|
||||
- [ ] **`--live` agent-in-the-loop know-to-ask.** Replay fixtures with a real model deciding whether to issue retrieval calls; grade the agent, not just the deterministic reflex. Pre-registered in `docs/eval/BRAINBENCH.md` (the v1 metric grades the injection decision, which IS the shipped mechanism). Needs: seeded N-repeat methodology for model stochasticity + budget rails. Priority: P2.
|
||||
- [ ] **Intrusion-budget gating calibration.** `avg_injected_tokens` is reported, non-gating (decision 18) — a wrong threshold is worse than none. After a few weeks of scoreboard data across PRs, pick calibrated per-seam thresholds and promote it to a gated metric. Priority: P2.
|
||||
- [ ] **Flip contract adapters to production when real integrations land.** `adapters/claude-code.ts` exports the UserPromptSubmit hook wire types; the real hook swaps the in-process transport for an exec of the hook script and flips `seam: 'contract'` → `'production'` with continuous bench numbers. Same for codex fragments. This is the integration PR's checklist item — without it the seam disclosure goes stale. Priority: P1 (attached to the harness-integration PR, not standalone).
|
||||
- [ ] **Cathedral 1 conformance-kit fixture import.** The memory-verbs conformance scenarios convert to BrainBench fixtures via the published `evals/brainbench/schema/fixture.schema.json` once `garrytan/cathedral-1` merges ("conformance tests double as BrainBench seed fixtures", decision log 2026-06-12). Free corpus growth from already-reviewed scenarios. Blocked by: cathedral-1 on master. Priority: P2.
|
||||
- [ ] **Live-embeddings fidelity mode (`--embeddings`).** Hermetic CI grades the keyword/alias arms only (disclosed); an opt-in mode seeding real embeddings would grade write-back/continuity retrieval through the vector path. Same budget rails as `--llm`. Priority: P3.
|
||||
- [ ] **Community fixture intake + competitor adapters.** The TD1 remainder after the generated corpus absorbed in-PR growth: an `external-authors/`-style intake path for contributed fixtures (validator + privacy guard already gate them) and adapters for non-gbrain memory systems against the published schemas, enabling true head-to-head rows in the gbrain-evals scorecard. Priority: P3.
|
||||
- [ ] **JSON-Schema ↔ validator parity test.** `evals/brainbench/schema/*.schema.json` is the foreign-runner contract but only existence/parse is pinned — the schemas could drift from `fixtures.ts`'s validator silently. Needs a JSON-schema validator dep (ajv) or a hand-rolled subset checker; deferred to avoid a new dependency mid-wave. Priority: P3.
|
||||
- [ ] **`eval compare` markdown rendering for `mode: 'n/a'` rows.** EvalRunRecord v3 records brainbench under 'n/a'; the markdown renderer iterates SEARCH_MODES only, so those rows surface in `--json` output but not the mode table (documented in the v3 docstring). Add an un-grouped section to `renderMarkdown`. Priority: P3.
|
||||
- [ ] **Periodic re-baselining (the ratchet doesn't auto-tighten).** Improvements aren't banked into master's baseline until a PR updates it, so a regression back to a stale baseline level passes. Documented as an accepted residual in `docs/eval/BRAINBENCH.md`; the fix is an operator habit or a scheduled job that re-runs `--update-baseline` after metric-improving merges. Priority: P3.
|
||||
|
||||
- [ ] **Hermetic-ize the 7 env-sensitive LLM-availability tests.** `test/think-gateway-adapter.test.ts`, `test/conversation-parser/llm-base.test.ts`/`llm-fallback.test.ts`, `test/doctor-ze-checks.test.ts` assert behavior "when ANTHROPIC_API_KEY is unset" by reading the live process env — they fail on any dev shell that exports provider keys (verified failing on clean master in such a shell; green in keyless CI). Stub/save-restore the env per test so local runs match CI. Priority: P2.
|
||||
## #2416 follow-ups (query-steering wave)
|
||||
|
||||
- [ ] **P2 — MCP-envelope `hint` field for concept-shaped `search` calls.**
|
||||
**What:** surface the concept→query nudge to remote/MCP agent callers, not
|
||||
just the CLI. **Why:** MCP agents are the primary misrouting class the
|
||||
#2416 issue describes; the shipped CLI stderr nudge covers the caller class
|
||||
*least* at risk. **Context:** the `search` op returns a bare
|
||||
`SearchResult[]` (`src/core/operations.ts` — both return sites), so a hint
|
||||
needs an envelope change that ripples into `formatResult`, MCP
|
||||
serialization, and array-shape tests — deliberately kept out of the atomic
|
||||
#2416 commit. The pure classifier already exists
|
||||
(`looksConceptShaped`/`conceptNudge` in `src/core/search/query-intent.ts`);
|
||||
only the transport is missing. Consider a sibling metadata channel (like
|
||||
`_meta.metric_glossary`) rather than changing the array shape.
|
||||
**Depends on:** agreeing an envelope pattern that doesn't break existing
|
||||
MCP consumers.
|
||||
|
||||
## MEMORY_VERBS v1 follow-ups (filed v0.43.0.0 — Cathedral 1)
|
||||
|
||||
Deferred from the Cathedral 1 ship (CEO review, EXPANSION mode). Both are
|
||||
additive to the frozen v1 contract — neither breaks it. See plan + GSTACK
|
||||
REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-agile-iverson.md`
|
||||
and the scope record at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-06-12-memory-verbs-protocol.md`.
|
||||
|
||||
- [ ] **P3 — external-implementation certification PROGRAM.** The conformance
|
||||
TOOLING shipped (`gbrain protocol conformance --target <endpoint>`); the
|
||||
PROGRAM around it (badges, a registry of conformant implementations, listed
|
||||
third-party servers) waits for a second implementation to exist. **Why:** the
|
||||
protocol-not-product thesis only pays off once someone else implements
|
||||
MEMORY_VERBS; until then a certification program certifies an empty set.
|
||||
**Where:** new — would build on `src/commands/protocol.ts` conformance output.
|
||||
- [ ] **P3 — persistent open-threads model for the entity card.** v1 derives
|
||||
`entity.open_threads` from active commitment-kind facts + recent timeline
|
||||
entries (best-effort, possibly empty). A richer model (a real threads table:
|
||||
conversation id, opened/closed state, last activity) would make open-threads
|
||||
authoritative. **Why:** the card's open-threads field is the weakest signal
|
||||
in v1; a first-class threads store would make it load-bearing. **Where:**
|
||||
`src/core/verbs/entity-card.ts` open-threads assembly + a new schema table
|
||||
(additive — the card field already exists, so this is a quality upgrade, not
|
||||
a contract change).
|
||||
- [ ] **P2 — `recall` filter composition vs the spec (found by the v0.43.0.0
|
||||
cross-model doc review).** The handler dispatch is first-match
|
||||
(`supersessions` > `entity` > `session_id` > `since`), so `since` is
|
||||
silently ignored when `entity`/`session_id` is supplied, and `limit` has no
|
||||
server-side cap. Either compose the filters (additive — the spec's "filters
|
||||
the FACTS arm" wording already reads that way) or spell the precedence out
|
||||
in `docs/protocol/MEMORY_VERBS_v1.md`. **Where:** the `recall` handler in
|
||||
`src/core/operations.ts`.
|
||||
- [ ] **P3 — widen `synthesize`'s `unavailable` mapping.** Only the
|
||||
missing-key gateway warning maps to the `unavailable` error today; other
|
||||
no-usable-model failures can surface as `internal` (contract-legal but less
|
||||
actionable) or, worst case, a stubbed success. Audit the gateway failure
|
||||
modes and map every model-unusable path to `unavailable` with a fix
|
||||
suggestion. **Where:** the `synthesize` handler in `src/core/verbs.ts`.
|
||||
## Fix-wave 1 follow-ups (upgrade-wedge + trust-seam wave, 2026-08)
|
||||
|
||||
Deferred from the un-wedge-v121 hotfix wave (eng review + codex outside voice
|
||||
CLEARED; every item an explicit review decision). Waves 2–6 of the sequence are
|
||||
planned separately (provider-compat rescue is next; its original 2026-07-24
|
||||
DeepSeek-deprecation deadline has now PASSED — re-verify each cluster against
|
||||
master before starting, several fixes landed independently).
|
||||
|
||||
- [ ] **P2 — Shared strict `parseFlags` helper as the #2185 end-state (eng
|
||||
review 2B).** This wave ships the generated known-flags registry +
|
||||
pre-dispatch validator (parser and registry can drift only until the
|
||||
freshness guard fires). The structural end-state migrates commands onto one
|
||||
shared strict parser so parser == registry by construction; mechanical but
|
||||
touches 60+ command files — its own PR. Where: `src/commands/*.ts`,
|
||||
`src/cli.ts`, `scripts/generate-flag-registry.ts` (retires).
|
||||
- [ ] **P2 — `whoknows` CLI routing (surfaced by the #2035-class sweep).**
|
||||
`handleCliOnly`'s `whoknows` case (the dedicated CLI renderer with
|
||||
thin-client routing) is dead code — the command resolves via the
|
||||
`find_experts` op alias, and adding it to CLI_ONLY trips the alias-collision
|
||||
guard. Decide the intended surface alongside PR #2509 (whoknows --explain
|
||||
per-result factor breakdown) and delete whichever lane loses. Where:
|
||||
`src/cli.ts`, `src/commands/whoknows.ts`, PR #2509.
|
||||
- [ ] **P3 — #2544 second half: per-put_page `getAllSlugs` full scan.** The
|
||||
getChunks egress half shipped in this wave (explicit non-vector column
|
||||
list). The remaining Postgres-egress cost is put_page's per-call
|
||||
`getAllSlugs` table scan — needs a targeted existence probe or cached slug
|
||||
set. Where: `src/core/operations.ts` put_page path, both engines.
|
||||
- [ ] **P3 — #1558 admin-UI register form.** The `/admin/api/register-client`
|
||||
API now accepts `source` + `federatedRead` (this wave, PR #2016 absorbed);
|
||||
the admin SPA form fields + `/admin/api/sources` picker are the UI layer.
|
||||
Where: `src/commands/serve-http.ts` admin SPA blob.
|
||||
- [ ] **P3 — jsonb-integrity surfaces: batch + share (ship-review follow-up).**
|
||||
doctor's jsonbIntegrityCheck runs 2 queries per target (16 round-trips) and
|
||||
duplicates the TARGETS table with repair-jsonb (already drifted once on the
|
||||
jsonPayloadOnly predicate before being mirrored by hand). Batch the counts
|
||||
into one UNION ALL query and extract a shared targets constant
|
||||
(src/core/jsonb-integrity-targets.ts) consumed by both. Where:
|
||||
`src/commands/doctor.ts` jsonbIntegrityCheck, `src/commands/repair-jsonb.ts`.
|
||||
- [ ] **P3 — register-client HTTP-level e2e (ship-review follow-up).** The
|
||||
source/federatedRead lane is covered by unit normalizers + a structural
|
||||
route pin; a DATABASE_URL-gated serve-http e2e (register with bindings →
|
||||
assert stored client via /admin/api/agents; invalid source → 400
|
||||
invalid_source) closes the wire-level gap. Where:
|
||||
`test/e2e/serve-http-oauth.test.ts`.
|
||||
- [ ] **P3 — get_chunks `__all__` sentinel narrows to 'default' (red-team,
|
||||
Wave 3 territory).** `sourceScopeOpts` returns `{}` for a trusted local
|
||||
`--source __all__` caller (documented "spans the brain"), but both engines'
|
||||
getChunks map empty scope to the 'default' floor — the one read op where
|
||||
`{}` is reinterpreted. Fold into the Wave 3 source-federation cluster's
|
||||
`__all__` work (an explicit unscoped signal in the engine signature, or
|
||||
handler-side expansion for trusted callers). Where: `src/core/operations.ts`
|
||||
get_chunks, both engines' getChunks.
|
||||
- [ ] **P3 — #2536 wedged-migration diagnostics.** The v121 wedge aborted
|
||||
initSchema BEFORE runMigrations, so the wedged-migration diagnostics row was
|
||||
never written — operators got a bare SQL error with no remediation hint.
|
||||
Write the diagnostics row (or a stderr remediation block) from the blob-replay
|
||||
catch path too. Where: `src/core/migrate.ts`, `src/commands/apply-migrations.ts`.
|
||||
## WAL-repair wave follow-ups (#223/#1670/#2575)
|
||||
|
||||
- [ ] **P2 — gate auto-repair on an unclean-shutdown marker (adversarial F7).** The classifier
|
||||
deliberately over-matches (`RuntimeError`/`unreachable` → `wasm-abort`). If an unclean
|
||||
shutdown leaves a REPLAYABLE WAL tail (normal crash recovery would restore those committed
|
||||
txns) and the reopen then fails on a transient WASM error (OOM), auto-repair fires, layout
|
||||
validation can't tell torn from replayable, and resetWal discards the tail while the notice
|
||||
says "data preserved." Bounded today (backup always taken + restore + honest failure + repeated
|
||||
attempts capped), but a false-positive-with-successful-retry silently drops committed data.
|
||||
Fix direction (probe-verified): PGLite removes `postmaster.pid` on clean close, so gate AUTO
|
||||
repair (not the manual command) on `postmaster.pid` presence — a clean dir that aborts is not
|
||||
torn-WAL. Requires making the serial regression test stamp a `postmaster.pid` before corrupting
|
||||
(it currently clean-disconnects then corrupts, which the red-team flagged as unfaithful anyway).
|
||||
Needs a recall/precision call before landing.
|
||||
- [ ] **P3 — live non-gbrain PGLite consumer not caught by the postmaster.pid liveness guard
|
||||
(adversarial F8).** PGLite writes a sentinel `postmaster.pid` of `-42`; the liveness refusal in
|
||||
`validateWalRepairTarget` requires `pid > 0`, so it protects native Postgres dirs but not a
|
||||
non-gbrain pglite app that has the dir open (such an app writes no `.gbrain-lock`). Deliberate
|
||||
misuse of `pglite-repair --path <foreign pglite dir>` required. Option: refuse when
|
||||
postmaster.pid holds pid ≤ 0 with a very recent mtime, or document the boundary.
|
||||
- [ ] **P3 — mixed-version torn-lock read (adversarial F10 residual).** The heartbeat + initial
|
||||
lock writes are atomic (tmp+rename) now, but an OLD gbrain binary writing heartbeats IN PLACE
|
||||
while a NEW binary poll-reads can still catch a torn read → corrupt-lock verdict → a live
|
||||
holder's lock reaped → two writers (the #2348 class, version-skew-triggered). The reap marker
|
||||
quarantines repair, not the concurrent open. Cheap hardening: double-read the lock file (~50ms
|
||||
apart) before declaring it corrupt.
|
||||
|
||||
|
||||
- [ ] **P2 — graceful PGLite close on SIGTERM for the remaining long-running paths.**
|
||||
The torn-WAL genesis this wave repairs is an unclean shutdown: `src/core/process-cleanup.ts`
|
||||
releases locks on SIGTERM but never closes the PGlite handle, so `serve` / `jobs work` /
|
||||
`sync` killed mid-write (macOS-upgrade reboot, `systemctl stop`) leave the WAL torn.
|
||||
Autopilot already ships the pattern (d2fd1f29, #3178/#1872: `registerCleanup('autopilot-engine-close', ...)`
|
||||
— abort in-flight work → ≤2s bounded wait inside the 3s cleanup deadline →
|
||||
`engine.disconnect()`, double-call safe; rationale comment at autopilot.ts:438-452).
|
||||
Extend that exact pattern to the remaining long-running PGLite paths (register in
|
||||
connect()/command scope; dedupe so autopilot doesn't double-close), pinned by a serial
|
||||
lifecycle test. Interacts with #2084 exitCode containment + #1337 close ordering — read
|
||||
those comments in pglite-engine.ts first. Auto-repair makes recurrence self-healing
|
||||
meanwhile, so this is prevention, not recovery.
|
||||
- [ ] **P3 — pglite upgrade blocker tracker.** Two couplings make a "routine" pglite bump a
|
||||
breaking change: (a) pglite ≥0.5 removes the `@electric-sql/pglite/vector` export that
|
||||
`pglite-engine.ts` imports (verified against npm); (b) the pg_resetwal port
|
||||
(`src/core/pglite-resetwal.ts`) is coupled to the PG17 pg_control layout
|
||||
(`PG_CONTROL_VERSION` 1700 — guarded at runtime by `WalResetUnsupportedError`, so a
|
||||
mismatched bump makes the repair tool refuse every dir rather than corrupt, but it still
|
||||
means the repair feature silently dies). Any future pglite upgrade wave must revisit BOTH
|
||||
together and re-derive the ControlFileData offset table for the new PG major.
|
||||
|
||||
## serve --http takes-holders + agent-voice hardening follow-ups (filed v0.42.74.0)
|
||||
|
||||
Deferred from the #2529/#2477 security-fix wave (plan-eng-review + codex outside
|
||||
voice CLEARED). None block the wave.
|
||||
|
||||
- [ ] **P2 — Per-OAuth-client `takes_holders` storage (#2529 follow-up).** Legacy
|
||||
bearer tokens honor `access_tokens.permissions.takes_holders` through
|
||||
`verifyAccessToken`; OAuth clients have no equivalent column on `oauth_clients`,
|
||||
so OAuth-minted tokens fail closed to `['world']`. Needs a schema migration
|
||||
(`oauth_clients.takes_holders` JSONB or TEXT[]) + a `register-client` flag +
|
||||
the `verifyAccessToken` JOIN projection. Include surfacing the EFFECTIVE
|
||||
takes-holder scope in `whoami` output as part of this follow-up, so operators
|
||||
can self-diagnose the legacy-vs-OAuth semantic split instead of reading docs.
|
||||
Where: `src/schema.sql`, `src/core/migrate.ts`, `src/core/oauth-provider.ts`,
|
||||
`src/commands/auth.ts`, `src/core/operations.ts` (whoami).
|
||||
- [ ] **P3 — agent-voice Host-header allowlist (DNS-rebinding hardening).** The
|
||||
#2477 fix ships default-deny CORS + an Origin gate on `/session`/`/tool`, but
|
||||
the gate derives self-origin from the `Host` header, so a DNS-rebound page
|
||||
(attacker origin whose host resolves to the operator's loopback) still passes.
|
||||
Validate `Host` against `localhost`/`127.0.0.1`/operator-configured hosts and
|
||||
403 otherwise; slots beside `originAllowed()` in the router. Issue #2477
|
||||
explicitly deferred this. Where: `recipes/agent-voice/code/server.mjs`.
|
||||
- [ ] **P3 — Debounce `last_used_at` in the oauth-provider legacy path.** The
|
||||
legacy branch of `verifyAccessToken` fires an unconditional
|
||||
`UPDATE access_tokens SET last_used_at = now()` on EVERY request, while the
|
||||
legacy HTTP transport debounces the same write to once per 60s via a WHERE
|
||||
clause (`src/mcp/http-transport.ts` validateToken). Apply the same pattern —
|
||||
one fewer write per request on the `serve --http` hot path.
|
||||
Where: `src/core/oauth-provider.ts`.
|
||||
|
||||
## v0.42.67.0 follow-ups (Windows build tooling)
|
||||
|
||||
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
|
||||
@@ -2383,8 +2589,9 @@ contributor traps.
|
||||
- [ ] **P2: Document `FREE_LOCAL_RERANK_PROVIDERS` invariant.** `src/core/budget/budget-tracker.ts:lookupPricing`
|
||||
returns `{input:0, output:0}` for any model id under the `llama-server-reranker:`
|
||||
provider on the rerank kind. The contract relies on all callers going through
|
||||
`gateway.rerank()`'s `assertTouchpoint`-with-extended-models check (which validates
|
||||
the model exists before pricing fires). Theoretical bypass: a future caller that
|
||||
`gateway.rerank()`'s own model-list check (rerank-specific; it validates the
|
||||
model exists before pricing fires — note this was never `assertTouchpoint`,
|
||||
which checks provider touchpoints only). Theoretical bypass: a future caller that
|
||||
reserves directly against BudgetTracker with `kind: 'rerank'` and an arbitrary
|
||||
`llama-server-reranker:<anything>` model id gets free pricing. Fix: code comment
|
||||
documenting the invariant, OR move the freeness check to gateway.rerank() where
|
||||
@@ -2580,7 +2787,7 @@ contributor traps.
|
||||
|
||||
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
|
||||
|
||||
- [ ] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
|
||||
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
|
||||
|
||||
|
||||
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
|
||||
@@ -4808,3 +5015,151 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
+ the models-doctor tests.
|
||||
|
||||
**Depends on:** nothing.
|
||||
|
||||
## Agent-bootstrap wave follow-ups (filed at build time)
|
||||
|
||||
- [x] **P2 — compiled `gbrain` binary can now `serve` a PGLite brain.** FIXED:
|
||||
`src/core/pglite-embedded-assets.ts` embeds PGLite's runtime payload
|
||||
(`pglite.wasm`, `initdb.wasm`, `pglite.data`, `vector.tar.gz`,
|
||||
`pg_trgm.tar.gz`) with Bun's `import ... with { type: 'file' }` and hands them
|
||||
to PGLite via `PGliteOptions` (`pgliteWasmModule` / `initdbWasmModule` /
|
||||
`fsBundle` + custom `vector`/`pg_trgm` extensions whose `setup()` returns a
|
||||
`bundlePath` pointing at the embedded tarball, materialized to a temp file
|
||||
because `createReadStream` — unlike `readFileSync` — can't read `/$bunfs`
|
||||
paths). `src/core/pglite-engine.ts` spreads `getEmbeddedPgliteOptions()` at
|
||||
both `PGlite.create()` sites, so the Bun-vfs #1340 ENOENT no longer fires for
|
||||
a correctly-built binary. Guarded by `scripts/check-pglite-embedded.sh`
|
||||
(compiles a smoketest and asserts a real PGLite query round-trips), wired into
|
||||
`bun run verify` + `check:all` + `check:pglite-embedded`. The real-agent e2e
|
||||
harness (`test/helpers/agent-harness.ts`) now resolves to the fast compiled
|
||||
MCP server; the `bun run` fallback stays as a safety net. Upstream Bun issue
|
||||
(still open since Nov 2024) is now moot for gbrain.
|
||||
- [ ] **P1 — `--background --follow` spawns a nonexistent subcommand.**
|
||||
`src/core/cli-options.ts` (~:391) spawns `gbrain jobs follow <id>` after a
|
||||
background submit, but jobs.ts has no `follow` subcommand (`jobs watch
|
||||
--follow` exists and takes no id; `jobs submit --follow` is inline-only).
|
||||
The spawned child hits the unknown-subcommand path, so `--background
|
||||
--follow` submits fine but the live stream never attaches. Fix: implement
|
||||
`jobs follow <id>` (poll get_job + stream progress) or retarget the spawn.
|
||||
Found during the v0.45.0.0 doc audit; KEY_FILES documents current behavior.
|
||||
- [ ] **P1 — bootstrap status is not workspace-scoped.** status.ts reads the
|
||||
newest GLOBAL verify snapshot + global push-status, so a green verify in
|
||||
workspace A can report workspace B as verified/healthy. Key the verify
|
||||
snapshots + push-status by workspace path (or store them under the
|
||||
workspace's own state dir) so status reflects the workspace it runs in.
|
||||
Found by the v0.45.0.0 adversarial pass (Codex P2, raised to P1 — multi-
|
||||
workspace is the graduation/multi-device story).
|
||||
- [ ] **P2 — mkdir-lock ABA steal race (folds into the shared-primitive TODO).**
|
||||
Both acquirePushLock (workspace-push.ts) and acquireBootstrapLock (lock.ts)
|
||||
steal a dead+stale lock as rmSync-then-mkdir with no re-verify between, so
|
||||
two simultaneous stealers can both believe they hold it (same known class as
|
||||
pglite-lock; git index.lock bounds the push damage). When extracting the
|
||||
shared mkdir-lock primitive, harden the steal: re-read + token-compare the
|
||||
stale owner immediately before rmSync, or use an atomic rename-based steal.
|
||||
Cross-model finding (Claude + Codex) v0.45.0.0 adversarial.
|
||||
- [ ] **P2 — uninstall TOCTOU on a live PGLite DB.** uninstall probes the serve
|
||||
lock then recursively deletes later; a serve starting in between wins the
|
||||
race. Gated today by the lock probe + refuse-on-live-serve, so this is the
|
||||
residual known lock-class window — close it when the shared lock primitive
|
||||
lands (hold the lock across the probe→delete span). Codex P1 v0.45.0.0.
|
||||
- [ ] **P2 — sweep fairness: recency-only selection starves old pages.**
|
||||
sweep.ts pass 2 repeatedly takes the newest batchLimit pages with no
|
||||
extraction-watermark filter, so frequently-updated pages monopolize every
|
||||
sweep and older pages never get link/timeline extraction. Add an
|
||||
extracted-watermark (or round-robin) so the backlog drains. Codex P2.
|
||||
- [ ] **P2 — untracked deny-glob exclusion is invisible under detached push.**
|
||||
A first-time deny-glob file (e.g. a fresh .env) is dropped from staging and
|
||||
reported only via the logger callback + excludedUntracked — but session-end/
|
||||
session-start auto-push runs via spawnDetachedPush with stdio:'ignore', so
|
||||
the warning is discarded. Fail-safe (never leaves the machine) but the user
|
||||
gets a false "synced" impression. Surface excluded deny-matches via the
|
||||
heartbeat/push-status file so doctor + status can report them. Claude
|
||||
adversarial (minor) v0.45.0.0.
|
||||
|
||||
- [ ] **P2 — shared receipt upsert helper.** InstallReceipt construct/merge
|
||||
logic is quadruplicated (bootstrap.ts writeRenderReceipt +
|
||||
appendReceiptRegistration, attach.ts, repo.ts recordRepoInReceipt) with
|
||||
slightly different defaults; guardReceiptOverwrite is wired at each site
|
||||
individually. One `upsertReceipt(home, ws, patch)` in format.ts.
|
||||
- [ ] **P2 — promote the atomic tmp+rename write idiom to one core helper.**
|
||||
~7 hand-rolled copies across format.ts/interview.ts/lock.ts/render.ts/
|
||||
hook.ts/workspace-push.ts/verify.ts; hooks.ts already has a private
|
||||
atomicWriteJson to promote (plus a text variant).
|
||||
- [ ] **P2 — extract a shared mkdir-lock primitive.** bootstrap/lock.ts and
|
||||
workspace-push.ts both implement atomic-mkdir + PID/age/token steal rules
|
||||
citing the same pglite-lock learnings (#2058/#2348); three parallel
|
||||
implementations counting pglite-lock. One primitive with injectable stale
|
||||
window + throw-vs-result adapters.
|
||||
- [ ] **P2 — move the hook heartbeat read surface into core.** core/bootstrap/
|
||||
status.ts and core/bootstrap/verify.ts dynamic-import readHeartbeatTail
|
||||
from commands/hook.ts (core→commands inversion). A core/hooks-telemetry.ts
|
||||
owning the read/write surface removes the reach-ins.
|
||||
- [ ] **P2 — consolidate the two GitHub remote parsers + privacy probes.**
|
||||
workspace-push.ts parseGithubOwnerRepo (accepts ssh:// + trailing /) vs
|
||||
repo.ts parseGithubRemote (scp-form only), and verifyRemotePrivacy (gh repo
|
||||
view) vs verifyRepoPrivate (gh api): grammars/failure classes can drift.
|
||||
One exported parser + one privacy-probe core returning the verdict union.
|
||||
- [ ] **P2 — connector-ingest capability probe (with v1.1 connector ingest).**
|
||||
The plan's build-order-2 probe (email/calendar connector availability wired
|
||||
into install output) deferred with the feature it gates; the capability
|
||||
report today covers embeddings + extraction only.
|
||||
- [ ] **P2 — G15 retention warnings in doctor.** MEMORY.md size cap + corpus
|
||||
retention + orphaned stop-hook buffers are enforced/pruned but doctor never
|
||||
warns when they accumulate; add the three checks to the bootstrap group.
|
||||
- [ ] **P2 — NER-based graph floor in verify.** The graph-floor check asserts
|
||||
wikilink-edge/backlink link-table rows; a true entity-extraction floor
|
||||
(≥1 NER entity + one edge-only query op) needs the extraction path or a
|
||||
keyed provider, so it rides with a keyed-mode verify extension.
|
||||
- [ ] **P2 — `--isolated` flag productization.** GBRAIN_HOME threading through
|
||||
MCP registration env, hooks, and uninstall guards ships in v1; the
|
||||
documented `bootstrap --isolated` one-flag wrapper (set env + assert
|
||||
database_path containment + gitignore check, per D2=C) is not yet a flag.
|
||||
- [ ] **P2 — sweep N+1 page fetch.** The links/timeline pass getPage-per-slug
|
||||
loop is bounded (batchLimit 20) but serializes round-trips on the live
|
||||
serve connection; add a getPagesBySlugs batch read to both engines (engine
|
||||
parity + bootstrap-probe obligations) and use it.
|
||||
|
||||
- [ ] **P2 — `gbrain quota` meter command.** Productize the release-gate quota
|
||||
measurement (script+doc ship with the bootstrap wave) into a command that reports
|
||||
a session/day's token burn against the harness subscription allowance. Blocked on
|
||||
a proven per-harness token-count method — an inaccurate meter erodes the trust it
|
||||
exists to build (CEO review D3.4; not promoted by a passing quota gate, by
|
||||
decision). Start: the spike's measurement scripts under docs/designs/.
|
||||
- [ ] **P2 — Networked Docker paste-flow e2e.** The offline container e2e (interview
|
||||
→ render → verify with fake gh) ships with the wave; the full networked flow (bun
|
||||
install, gh auth, repo create, MCP registration) stays manual. Unblocks after 10
|
||||
consecutive green offline runs in heavy-tests (CEO review D3.3b). Start:
|
||||
tests/docker/ harness + the fake-gh recording shim.
|
||||
- [ ] **P2 — Real-agent door e2e needs a provisioned runner.** The real-binary door
|
||||
tests (`test/e2e/bootstrap-real-{claude,codex}.serial.test.ts`) drive the ACTUAL
|
||||
`claude`/`codex` binaries against a real gbrain and pay API cost. They self-SKIP
|
||||
(`describe.skipIf` on binary/auth) everywhere else, so the `real-agent-e2e` job in
|
||||
`.github/workflows/heavy-tests.yml` is a green no-op on stock GitHub runners. To
|
||||
actually EXERCISE them we need a self-hosted / manually-provisioned runner with
|
||||
authed `claude` + `codex` and provider creds exported
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`). Until then they
|
||||
run locally on an operator machine only. Start: stand up a labeled runner with the
|
||||
binaries pre-authed, or a scheduled self-hosted lane.
|
||||
|
||||
- [ ] **P1 — unit-shard exit hang: bun test process leaks a ref'd handle and never
|
||||
exits after all tests pass.** Probabilistic (scales with file count/duration),
|
||||
PRE-EXISTING: reproduces on origin/master with the same 274-file combination
|
||||
(`bun test $(cat list) --max-concurrency=2` — shard-2 composition from the
|
||||
agent-bootstrap wave's reshuffled split; solo halves/eighths pass, full list
|
||||
hangs on both branches; a repeat of the same 68-file quarter passed, so it is a
|
||||
race, not a single file). Forensics: the "[process-cleanup] db-lock:test:r1:
|
||||
PGlite is closed" tail line is printed BY the SIGTERM cleanup at kill time, not
|
||||
pre-hang; db-lock refresh + process-watchdog intervals are already unref'd —
|
||||
the leak is elsewhere (candidate class: a ref'd handle in a PGLite/worker
|
||||
teardown race). run-unit-parallel.sh now classifies the exact signature
|
||||
(killed at cap + all assigned files started + zero fail markers) as a loud
|
||||
warn-pass so it can't red-X unrelated work; fixing the leak removes the
|
||||
classifier's reason to exist. Start: reproduce with `--inspect` /
|
||||
Bun.unsafe process handle dumps on the full shard-2 list.
|
||||
- [ ] **P2 — WAL-repair unit tests flake under parallel shard load.**
|
||||
`test/pglite-repair.test.ts` (`repairPgliteWal rename-based backup`,
|
||||
`inspectPgliteDataDir verdicts`) failed once in a full 4-shard run and pass
|
||||
44/44 standalone — load-sensitive, from the v0.42.75.0 WAL auto-repair wave
|
||||
(pre-existing on master; observed during the agent-bootstrap gate runs).
|
||||
Start: run the file under `--max-concurrency=4` alongside PGLite-heavy
|
||||
neighbors to reproduce; suspect tmp-dir or timing assumptions.
|
||||
|
||||
+4
-3
@@ -19,7 +19,8 @@
|
||||
},
|
||||
"overrides": {
|
||||
"@babel/core": "^7.29.6",
|
||||
"postcss": "^8.5.10",
|
||||
"nanoid": "^3.3.17",
|
||||
"postcss": "^8.5.23",
|
||||
},
|
||||
"packages": {
|
||||
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
|
||||
@@ -224,7 +225,7 @@
|
||||
|
||||
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
|
||||
|
||||
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
|
||||
"nanoid": ["nanoid@3.3.18", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w=="],
|
||||
|
||||
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
|
||||
|
||||
@@ -232,7 +233,7 @@
|
||||
|
||||
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
|
||||
|
||||
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
|
||||
"postcss": ["postcss@8.5.25", "", { "dependencies": { "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw=="],
|
||||
|
||||
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
|
||||
|
||||
|
||||
+2
-1
@@ -20,6 +20,7 @@
|
||||
},
|
||||
"overrides": {
|
||||
"@babel/core": "^7.29.6",
|
||||
"postcss": "^8.5.10"
|
||||
"postcss": "^8.5.23",
|
||||
"nanoid": "^3.3.17"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
"express-rate-limit": "^7.5.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"heic-decode": "^2.1.0",
|
||||
"js-yaml": "^3.15.0",
|
||||
"js-yaml": "^3.15.1",
|
||||
"marked": "^18.0.2",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
@@ -53,13 +53,13 @@
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"body-parser": "^2.3.0",
|
||||
"fast-uri": "^3.1.4",
|
||||
"fast-uri": "^3.1.5",
|
||||
"fast-xml-builder": "^1.1.7",
|
||||
"fast-xml-parser": "^5.7.0",
|
||||
"form-data": "^4.0.6",
|
||||
"hono": "^4.12.25",
|
||||
"ip-address": "^10.1.1",
|
||||
"js-yaml": "^3.15.0",
|
||||
"hono": "^4.12.34",
|
||||
"ip-address": "^10.3.1",
|
||||
"js-yaml": "^3.15.1",
|
||||
"qs": "^6.15.2",
|
||||
},
|
||||
"packages": {
|
||||
@@ -163,7 +163,7 @@
|
||||
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@2.0.11", "", { "peerDependencies": { "hono": "^4" } }, "sha512-bjD221KPLoJTWUwso1J6fGKiTXEUFedG/s0visavY4zakFPkeGURMRNly+FhBHs7T8Dz4qHaZIMX9ZoJHSJtKA=="],
|
||||
"@hono/node-server": ["@hono/node-server@2.1.0", "", { "peerDependencies": { "hono": "^4" } }, "sha512-XovyyCCnBzW+zKu+z/zq8hwNs4KOR5rEMAOxo2f40Q5xoOI37IMm6MIg2COOUtUApo0i6850MTBKH2u4QLGIqg=="],
|
||||
|
||||
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
|
||||
|
||||
@@ -401,7 +401,7 @@
|
||||
|
||||
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
|
||||
|
||||
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
|
||||
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
|
||||
|
||||
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
|
||||
|
||||
@@ -437,7 +437,7 @@
|
||||
|
||||
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
|
||||
|
||||
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
|
||||
"hono": ["hono@4.13.0", "", {}, "sha512-jhunvfHWxd7J5EFfSgH4xsYJzSe/lfqbUCxiyyeaQasUsXeEHXtzVid+7EOGByc5JnFa23SSFL3Y2RV/z1T+eQ=="],
|
||||
|
||||
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
|
||||
|
||||
@@ -447,7 +447,7 @@
|
||||
|
||||
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
|
||||
|
||||
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
|
||||
"ip-address": ["ip-address@10.4.0", "", {}, "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ=="],
|
||||
|
||||
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
|
||||
|
||||
@@ -461,7 +461,7 @@
|
||||
|
||||
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
|
||||
|
||||
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
|
||||
"js-yaml": ["js-yaml@3.15.1", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag=="],
|
||||
|
||||
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
|
||||
|
||||
|
||||
+104
-82
@@ -4,7 +4,7 @@
|
||||
|
||||
Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else.
|
||||
|
||||
v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
|
||||
Two engines ship today: `PGLiteEngine` — embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
|
||||
|
||||
## Why this matters
|
||||
|
||||
@@ -12,7 +12,7 @@ Different users have different constraints:
|
||||
|
||||
| User | Needs | Best engine |
|
||||
|------|-------|-------------|
|
||||
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) |
|
||||
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) |
|
||||
| Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase |
|
||||
| Open source hacker | Single file, no server, git-friendly | PGLiteEngine |
|
||||
| Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted |
|
||||
@@ -23,72 +23,30 @@ The engine interface means we don't have to choose. PGLite is the zero-friction
|
||||
|
||||
## The interface
|
||||
|
||||
```typescript
|
||||
// src/core/engine.ts
|
||||
**The single source of truth is `export interface BrainEngine` in
|
||||
`src/core/engine.ts`.** It is large (100+ methods) and grows with every
|
||||
feature wave — do NOT work from any snapshot of it, including an old copy of
|
||||
this doc. Read the interface itself, and let
|
||||
`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you
|
||||
whether both engines agree.
|
||||
|
||||
export interface BrainEngine {
|
||||
// Lifecycle
|
||||
connect(config: EngineConfig): Promise<void>;
|
||||
disconnect(): Promise<void>;
|
||||
initSchema(): Promise<void>;
|
||||
transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T>;
|
||||
The method families, to orient you before opening the file:
|
||||
|
||||
// Pages CRUD
|
||||
getPage(slug: string): Promise<Page | null>;
|
||||
putPage(slug: string, page: PageInput): Promise<Page>;
|
||||
deletePage(slug: string): Promise<void>;
|
||||
listPages(filters: PageFilters): Promise<Page[]>;
|
||||
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
|
||||
// Chunks
|
||||
upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void>;
|
||||
getChunks(slug: string): Promise<Chunk[]>;
|
||||
|
||||
// Links
|
||||
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
|
||||
removeLink(from: string, to: string): Promise<void>;
|
||||
getLinks(slug: string): Promise<Link[]>;
|
||||
getBacklinks(slug: string): Promise<Link[]>;
|
||||
traverseGraph(slug: string, depth?: number): Promise<GraphNode[]>;
|
||||
|
||||
// Tags
|
||||
addTag(slug: string, tag: string): Promise<void>;
|
||||
removeTag(slug: string, tag: string): Promise<void>;
|
||||
getTags(slug: string): Promise<string[]>;
|
||||
|
||||
// Timeline
|
||||
addTimelineEntry(slug: string, entry: TimelineInput): Promise<void>;
|
||||
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
|
||||
|
||||
// Raw data
|
||||
putRawData(slug: string, source: string, data: object): Promise<void>;
|
||||
getRawData(slug: string, source?: string): Promise<RawData[]>;
|
||||
|
||||
// Versions
|
||||
createVersion(slug: string): Promise<PageVersion>;
|
||||
getVersions(slug: string): Promise<PageVersion[]>;
|
||||
revertToVersion(slug: string, versionId: number): Promise<void>;
|
||||
|
||||
// Stats + health
|
||||
getStats(): Promise<BrainStats>;
|
||||
getHealth(): Promise<BrainHealth>;
|
||||
|
||||
// Ingest log
|
||||
logIngest(entry: IngestLogInput): Promise<void>;
|
||||
getIngestLog(opts?: IngestLogOpts): Promise<IngestLogEntry[]>;
|
||||
|
||||
// Config
|
||||
getConfig(key: string): Promise<string | null>;
|
||||
setConfig(key: string, value: string): Promise<void>;
|
||||
|
||||
// Migration + advanced (added v0.7)
|
||||
runMigration(sql: string): Promise<void>;
|
||||
getChunksWithEmbeddings(slug: string): Promise<ChunkWithEmbedding[]>;
|
||||
}
|
||||
```
|
||||
- **Lifecycle + identity** — `connect` / `disconnect` / `reconnect`,
|
||||
`initSchema`, `transaction`, `withReservedConnection`, and the `kind`
|
||||
discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch.
|
||||
- **Pages CRUD** — `getPage`, `putPage`, `deletePage`, `listPages`, slug
|
||||
resolution.
|
||||
- **Search** — `searchKeyword`, `searchVector`, chunk-level variants, takes
|
||||
search (keyword + vector), and `relationalFanout` (the typed-edge recall
|
||||
arm).
|
||||
- **Chunks + embeddings** — upsert/get, embedding-bearing variants.
|
||||
- **Graph** — links (single + batch writers), backlinks, `traverseGraph`,
|
||||
`traversePaths`.
|
||||
- **Tags, timeline (single + batch), raw data, versions.**
|
||||
- **Takes / facts / eval / salience** — the epistemological layer and the
|
||||
instruments over it.
|
||||
- **Stats, health, ingest log, config, migrations.**
|
||||
|
||||
### Key design choices
|
||||
|
||||
@@ -131,7 +89,7 @@ export interface BrainEngine {
|
||||
|
||||
RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific.
|
||||
|
||||
## PostgresEngine (v0, ships)
|
||||
## PostgresEngine
|
||||
|
||||
**Dependencies:** `postgres` (porsager/postgres), `pgvector`
|
||||
|
||||
@@ -144,9 +102,7 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
- JSONB for frontmatter with GIN index
|
||||
- Connection pooling via Supabase Supavisor (port 6543)
|
||||
|
||||
**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in.
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
@@ -193,17 +149,17 @@ run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
## PGLiteEngine
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
**Dependencies:** `@electric-sql/pglite`
|
||||
|
||||
**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented.
|
||||
**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep.
|
||||
|
||||
**PGLite-specific details:**
|
||||
- Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes)
|
||||
- Parameterized queries throughout (shared utilities in `src/core/utils.ts`)
|
||||
- `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set
|
||||
- Data stored at `~/.gbrain/brain.db` (configurable)
|
||||
- Data stored at `~/.gbrain/brain.pglite` (configurable)
|
||||
- pgvector HNSW index for cosine similarity vector search (same as Postgres)
|
||||
- tsvector + ts_rank for full-text search (same as Postgres)
|
||||
- pg_trgm for fuzzy slug resolution (same as Postgres)
|
||||
@@ -221,6 +177,66 @@ live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
|
||||
|
||||
**Symptom:** every PGLite-touching command dies at startup with
|
||||
`PGLite failed to initialize its WASM runtime … Aborted(). Build with
|
||||
-sASSERTIONS for more info.` — commonly first seen right after a macOS
|
||||
upgrade.
|
||||
|
||||
**Real root cause:** corrupt WAL/checkpoint state in the data dir after an
|
||||
unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the
|
||||
write-ahead log; every subsequent open fails WAL replay inside WASM and
|
||||
Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM
|
||||
incompatibility — the same signature reproduces across macOS versions and on
|
||||
Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun
|
||||
version bump changes it.
|
||||
|
||||
**Recovery ladder** (top rung first):
|
||||
|
||||
1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort,
|
||||
backs up `pg_wal/` + `pg_control` into a sibling
|
||||
`<dataDir>.wal-repair-backup-<ts>/` dir, resets the WAL in place
|
||||
(pg_resetwal semantics — data files preserved; transactions not
|
||||
checkpointed before the corruption may be lost), and retries once. On
|
||||
success it prints a loud stderr notice naming the backup and recommending
|
||||
`gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired
|
||||
data-dir lock (never after reaping another process's lock), skips for a
|
||||
cooldown window after a failed attempt
|
||||
(`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one
|
||||
backup per corruption episode (newest 3 episodes retained), and restores
|
||||
the original files if the retry still fails. Kill-switch:
|
||||
`GBRAIN_PGLITE_WAL_REPAIR=off`.
|
||||
2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir
|
||||
(read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset
|
||||
deliberately. Refuses when another gbrain process holds the brain (a live
|
||||
`gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`.
|
||||
3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default
|
||||
from your config) wipes and re-creates the brain from your brain repo, or
|
||||
manually: back up `~/.gbrain`, move `brain.pglite` aside,
|
||||
`gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`.
|
||||
Required for *catalog* corruption (58P01 / pgvector load failure) — WAL
|
||||
repair cannot fix that class.
|
||||
4. **Switch engines.** `gbrain init --supabase`, or native Postgres +
|
||||
pgvector (recipe below, contributed by @roysaurav):
|
||||
|
||||
```bash
|
||||
brew install postgresql@17
|
||||
brew services start postgresql@17
|
||||
createdb gbrain
|
||||
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
|
||||
cd pgvector && make && make install
|
||||
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
# ~/.gbrain/config.json: { "engine": "postgres",
|
||||
# "database_url": "postgresql://localhost:5432/gbrain" }
|
||||
gbrain apply-migrations --yes && gbrain doctor
|
||||
```
|
||||
|
||||
`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails
|
||||
to connect: it diagnoses the dir from disk, names the repair command, reports
|
||||
retained repair backups, and escalates when repairs keep recurring (that
|
||||
means the unclean-shutdown genesis is still active — see the ladder's rung 4).
|
||||
|
||||
## JSONB writes: never double-encode (the #2339 trap)
|
||||
|
||||
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
|
||||
@@ -259,16 +275,22 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make.
|
||||
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
|
||||
2. Add to engine factory in `src/core/engine-factory.ts`:
|
||||
```typescript
|
||||
export function createEngine(type: string): BrainEngine {
|
||||
switch (type) {
|
||||
case 'pglite': return new PGLiteEngine();
|
||||
case 'postgres': return new PostgresEngine();
|
||||
case 'myengine': return new MyEngine();
|
||||
default: throw new Error(`Unknown engine: ${type}`);
|
||||
export async function createEngine(config: EngineConfig): Promise<BrainEngine> {
|
||||
switch (config.engine || 'postgres') {
|
||||
case 'pglite': {
|
||||
const { PGLiteEngine } = await import('./pglite-engine.ts');
|
||||
return new PGLiteEngine();
|
||||
}
|
||||
case 'myengine': {
|
||||
const { MyEngine } = await import('./my-engine.ts');
|
||||
return new MyEngine();
|
||||
}
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
The factory uses dynamic imports so engines are only loaded when selected.
|
||||
The factory uses dynamic imports so an engine's dependencies (e.g. the
|
||||
PGLite WASM blob) are only loaded when that engine is selected.
|
||||
3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }`
|
||||
4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor.
|
||||
5. Document in this file + add a design doc in `docs/`
|
||||
@@ -299,7 +321,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu
|
||||
| JSONB queries | GIN index | GIN index | Identical |
|
||||
| Concurrent access | Connection pooling | Single process | PGLite limitation |
|
||||
| Hosting | Supabase, self-hosted, Docker | Local file | |
|
||||
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 |
|
||||
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical |
|
||||
|
||||
## Future engine ideas
|
||||
|
||||
|
||||
@@ -6,6 +6,14 @@ A system prompt for any AI agent that wants to build and maintain a personal kno
|
||||
|
||||
Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest.
|
||||
|
||||
> **Relationship to schema packs:** this document is the prose, paste-in
|
||||
> version of the schema pattern. gbrain also ships a machine-enforced
|
||||
> counterpart — schema packs (`gbrain schema`, typed pages, extraction,
|
||||
> aliases, lint) — documented in `docs/architecture/schema-packs.md` and
|
||||
> `docs/schema-author-tutorial.md`. The prose schema here and the active
|
||||
> schema pack should describe the same brain; when you evolve one, evolve
|
||||
> the other.
|
||||
|
||||
---
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
<!-- skillpack-version: 0.7.0 -->
|
||||
<!-- source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md -->
|
||||
# GBrain Skillpack: Reference Architecture for AI Agents
|
||||
|
||||
This is a reference architecture for how a production AI agent uses gbrain as its
|
||||
knowledge backbone. Based on patterns from a real deployment with 14,700+ brain
|
||||
files, 40+ skills, and 20+ cron jobs running continuously.
|
||||
files, the 50+ bundled skills (`skills/manifest.json`), and 20+ cron jobs running
|
||||
continuously.
|
||||
|
||||
**The memex vision, realized.** Vannevar Bush imagined a device where an individual
|
||||
stores everything, mechanized so it may be consulted with exceeding speed. GBrain is
|
||||
@@ -25,6 +25,7 @@ The foundational read-write loop and data model.
|
||||
| [Entity Detection](guides/entity-detection.md) | Run it on every message. Capture original thinking + entity mentions |
|
||||
| [The Originals Folder](guides/originals-folder.md) | Capturing WHAT YOU THINK, not just what you found |
|
||||
| [Brain-First Lookup](guides/brain-first-lookup.md) | Check the brain before calling any external API |
|
||||
| [Push-Based Context](guides/push-context.md) | volunteer_context: the brain volunteers relevant pages instead of waiting to be asked |
|
||||
| [Compiled Truth + Timeline](guides/compiled-truth.md) | Above the line: current synthesis. Below: append-only evidence |
|
||||
| [Source Attribution](guides/source-attribution.md) | Every fact needs a citation. Format and hierarchy |
|
||||
|
||||
@@ -99,6 +100,7 @@ Keeping it running and up to date.
|
||||
|
||||
| Guide | What It Covers |
|
||||
|-------|---------------|
|
||||
| [Agent Bootstrap](guides/bootstrap.md) | The paste-in install: `gbrain bootstrap`, hooks, `bootstrap verify`, keyless mode |
|
||||
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
|
||||
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
|
||||
|
||||
|
||||
+28
-14
@@ -1,5 +1,13 @@
|
||||
# GBrain Installation Verification Runbook
|
||||
|
||||
> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install
|
||||
> contract (round-trip, graph floor, and more) automatically and exits non-zero
|
||||
> on failure — it is the modern first thing to run after any install. See
|
||||
> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the
|
||||
> **manual, deep-verification** companion: use it when `bootstrap verify` fails
|
||||
> and you need to isolate which layer broke, or when you want to understand
|
||||
> what "healthy" looks like check by check.
|
||||
|
||||
Run these checks after install to confirm every part of GBrain is working.
|
||||
Each check includes the command, expected output, and what to do if it fails.
|
||||
|
||||
@@ -20,7 +28,8 @@ gbrain doctor --json
|
||||
**Expected:** All checks return `"ok"`:
|
||||
- `connection`: connected, N pages
|
||||
- `pgvector`: extension installed
|
||||
- `rls`: enabled on all tables
|
||||
- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains
|
||||
skip this check; the embedded engine has no remote surface)
|
||||
- `schema_version`: current
|
||||
- `embeddings`: coverage percentage
|
||||
|
||||
@@ -33,12 +42,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table.
|
||||
|
||||
**Check:** Ask the agent: "What is the brain-agent loop?"
|
||||
|
||||
**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes
|
||||
the read-write cycle: detect entities, read brain, respond with context, write
|
||||
brain, sync.
|
||||
**Expected:** The agent describes the read-write cycle documented in
|
||||
[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities,
|
||||
read brain, respond with context, write brain, sync.
|
||||
|
||||
**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the
|
||||
install paste (read `docs/GBRAIN_SKILLPACK.md`).
|
||||
**If it fails:** The agent hasn't loaded the skillpack. Have it read
|
||||
`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links.
|
||||
|
||||
---
|
||||
|
||||
@@ -53,8 +62,8 @@ gbrain check-update --json
|
||||
**Expected:** Returns JSON with `current_version`, `latest_version`,
|
||||
`update_available` (boolean). The cron `gbrain-update-check` is registered.
|
||||
|
||||
**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md
|
||||
Section 17.
|
||||
**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md)
|
||||
for how to register the update-check cron.
|
||||
|
||||
---
|
||||
|
||||
@@ -88,8 +97,9 @@ find /data/brain -name '*.md' \
|
||||
Some difference is normal (files added since last sync), but if page count is
|
||||
less than half the file count, sync is silently skipping pages.
|
||||
|
||||
**If page count is way too low:** The #1 cause is an unreachable direct
|
||||
connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543)
|
||||
**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an
|
||||
unreachable direct connection on an IPv4-only host. (PGLite brains have no
|
||||
network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543)
|
||||
for reads, but routes migrations, DDL, and sync transactions to a derived direct
|
||||
connection (`db.<ref>.supabase.co:5432`), which is IPv6-only.
|
||||
- On an IPv4-only host, reads work but sync transactions fail and silently skip
|
||||
@@ -122,7 +132,7 @@ This is the real test. Edit a brain page, push, wait, search.
|
||||
1. Edit a page in the brain repo (e.g., correct a fact on a person's page):
|
||||
|
||||
```bash
|
||||
# Example: fix a line in Gustaf's page
|
||||
# Example: fix a line in alice-example's page
|
||||
cd /data/brain
|
||||
# Make a small edit to any .md file
|
||||
git add -A && git commit -m "test: verify live sync" && git push
|
||||
@@ -253,19 +263,23 @@ gbrain repair-jsonb
|
||||
|
||||
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
|
||||
|
||||
**Bonus check** — frontmatter-keyed queries actually resolve:
|
||||
**Bonus check** — the doctor's dedicated JSONB scan agrees:
|
||||
|
||||
```bash
|
||||
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
|
||||
gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*'
|
||||
```
|
||||
|
||||
If this returns rows on a brain with person pages, the JSONB path is healthy.
|
||||
**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store
|
||||
objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`.
|
||||
|
||||
---
|
||||
|
||||
## Quick Verification (all checks in one pass)
|
||||
|
||||
```bash
|
||||
# 0. The one-command contract check (exits non-zero on failure)
|
||||
gbrain bootstrap verify
|
||||
|
||||
# 1. Schema
|
||||
gbrain doctor --json
|
||||
|
||||
|
||||
+35
-38
@@ -1,15 +1,24 @@
|
||||
# Install
|
||||
|
||||
Three install paths. Pick one. Mix later if needed.
|
||||
**Recommended door: the agent bootstrap.** Open your agent (Codex, Claude Code,
|
||||
or any harness) in the folder that will become its home and paste the block
|
||||
from the [README's install section](../README.md) — the agent fetches
|
||||
`BOOTSTRAP_FOR_AGENTS.md` from the `latest-stable` tag, installs the CLI,
|
||||
initializes a local PGLite brain, wires MCP, and isn't done until
|
||||
`gbrain bootstrap verify` exits 0. Full contract, security posture, and
|
||||
uninstall: [docs/guides/bootstrap.md](guides/bootstrap.md).
|
||||
|
||||
## 1. Run with an agent platform (recommended)
|
||||
The paths below are the manual equivalents and deep-dive detail. Pick one.
|
||||
Mix later if needed.
|
||||
|
||||
## 1. Run with an agent platform
|
||||
|
||||
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
bun install -g github:garrytan/gbrain#latest-stable
|
||||
gbrain init --pglite # 2 seconds; no server
|
||||
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
|
||||
gbrain skillpack scaffold --all # scaffolds every bundled skill (skills/manifest.json) into your agent workspace
|
||||
gbrain doctor # green checks all the way down
|
||||
```
|
||||
|
||||
@@ -24,7 +33,7 @@ To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts
|
||||
No agent platform, just shell + MCP-aware editor.
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
bun install -g github:garrytan/gbrain#latest-stable
|
||||
gbrain init --pglite
|
||||
```
|
||||
|
||||
@@ -58,16 +67,17 @@ gbrain autopilot --install # background daemon for nightly enrichment
|
||||
**Wire this same local brain into your coding agent** — zero server, zero token:
|
||||
|
||||
```bash
|
||||
claude mcp add gbrain -- gbrain serve # Claude Code
|
||||
codex mcp add gbrain -- gbrain serve # Codex
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs # Claude Code
|
||||
codex mcp add gbrain -- gbrain serve --surface verbs # Codex
|
||||
```
|
||||
|
||||
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
|
||||
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. `--surface verbs` gives the agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](protocol/MEMORY_VERBS_v1.md)) instead of the full tool catalog; drop the flag (default `full`) for every operation. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
|
||||
|
||||
## 3. MCP server (any MCP client)
|
||||
|
||||
```bash
|
||||
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
|
||||
gbrain serve --surface verbs # stdio MCP, just the 5 memory verbs (quickstart)
|
||||
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
|
||||
```
|
||||
|
||||
@@ -106,44 +116,31 @@ Useful for: team mounts, brain-as-a-service deployments, dev machines without di
|
||||
## Verifying the install
|
||||
|
||||
```bash
|
||||
gbrain bootstrap verify # the whole install contract; exits non-zero on failure
|
||||
gbrain doctor --json # full health check
|
||||
gbrain models # which AI models are configured for what
|
||||
gbrain models doctor # 1-token probe per configured model
|
||||
```
|
||||
|
||||
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
|
||||
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). For the manual check-by-check runbook, see [docs/GBRAIN_VERIFY.md](GBRAIN_VERIFY.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### PGLite crashes on macOS 26.x (Tahoe)
|
||||
### PGLite crashes at startup (`RuntimeError: Aborted()`)
|
||||
|
||||
PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL:
|
||||
This crash (typically first seen after a macOS upgrade) is **not** a
|
||||
macOS/WASM incompatibility — an unclean shutdown tore the data dir's
|
||||
write-ahead log, and every subsequent open fails WAL replay. The short
|
||||
version of the recovery ladder:
|
||||
|
||||
```bash
|
||||
# Install PostgreSQL + pgvector
|
||||
brew install postgresql@17
|
||||
brew services start postgresql@17
|
||||
createdb gbrain
|
||||
1. **Auto-repair (default):** run any gbrain command — gbrain detects the
|
||||
abort, resets the WAL in place (data preserved, backup kept), and
|
||||
continues. Then run `gbrain doctor`.
|
||||
2. **Manual repair:** `gbrain pglite-repair --dry-run`, then
|
||||
`gbrain pglite-repair --yes`.
|
||||
3. **Rebuild:** `gbrain reinit-pglite`.
|
||||
4. **Switch engines:** Supabase or native Homebrew Postgres + pgvector.
|
||||
|
||||
# Build pgvector from source (required for vector search)
|
||||
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
|
||||
cd pgvector && make && make install
|
||||
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
|
||||
# Point gbrain at your local Postgres
|
||||
cat > ~/.gbrain/config.json << 'EOF'
|
||||
{
|
||||
"engine": "postgres",
|
||||
"database_url": "postgresql://localhost:5432/gbrain",
|
||||
"schema_pack": "gbrain-base-v2"
|
||||
}
|
||||
EOF
|
||||
|
||||
# Run migrations and verify
|
||||
gbrain apply-migrations --yes
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend.
|
||||
|
||||
> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again.
|
||||
The full ladder — safety bounds, kill-switches, when WAL repair can't help,
|
||||
and the Homebrew Postgres recipe — lives in
|
||||
[docs/ENGINES.md](ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted).
|
||||
|
||||
+90
-11
@@ -26,19 +26,21 @@ Two equivalent paths:
|
||||
|
||||
**Path B — manual lifecycle (still supported):**
|
||||
- `bun test` — unit tests (no database required)
|
||||
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
|
||||
run `bun run test:e2e`, then tear it down.
|
||||
- Follow the "E2E test DB lifecycle" steps in
|
||||
[docs/TESTING.md](TESTING.md) to spin up the test DB, run
|
||||
`bun run test:e2e`, then tear it down.
|
||||
|
||||
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
|
||||
|
||||
**Always run typecheck before pushing.** `bun test` (the bun runner)
|
||||
skips TypeScript type checking — it only enforces runtime behavior.
|
||||
**Always run typecheck before pushing.** Neither `bun test` (the bun runner)
|
||||
nor `bun run test` gates on types — `bun run test` is just
|
||||
`bash scripts/run-unit-parallel.sh` (the sharded unit runner; no typecheck,
|
||||
no shell pre-checks — see the test-tier table in [docs/TESTING.md](TESTING.md)).
|
||||
Three ways to actually gate on types:
|
||||
|
||||
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
|
||||
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
|
||||
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
|
||||
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
|
||||
1. `bun run verify` — runs the shell guard checks (privacy, jsonb, source-id,
|
||||
progress-to-stdout, …) plus `bun run typecheck` in parallel
|
||||
(`scripts/run-verify-parallel.sh`). Use this mid-branch.
|
||||
2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
|
||||
3. `bun run ci:local` — the full local CI gate from Path A.
|
||||
|
||||
@@ -290,7 +292,8 @@ matter" with BrainBench-style before/after table, "what this means" closer, then
|
||||
|
||||
Create a migration file at `skills/migrations/v[version].md` when a release
|
||||
includes changes that existing users need to act on. The auto-update agent
|
||||
reads these files post-upgrade (Section 17, Step 4) and executes them.
|
||||
reads these files post-upgrade (see `docs/guides/upgrades-auto-update.md`)
|
||||
and executes them.
|
||||
|
||||
**You need a migration file when:**
|
||||
- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
|
||||
@@ -343,8 +346,8 @@ canonical.
|
||||
## Schema state tracking
|
||||
|
||||
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
|
||||
adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
|
||||
reads this during upgrades to suggest new schema additions without re-suggesting
|
||||
adopted, declined, or added custom. The auto-update agent
|
||||
(`docs/guides/upgrades-auto-update.md`) reads this during upgrades to suggest new schema additions without re-suggesting
|
||||
things the user already declined. The setup skill writes the initial state during
|
||||
Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
|
||||
|
||||
@@ -362,6 +365,82 @@ done
|
||||
|
||||
If any SHA differs from what's in the workflow files, update the pin and version comment.
|
||||
|
||||
## GitHub releases (binary assets + self-update) — #3521
|
||||
|
||||
`.github/workflows/release.yml` publishes a GitHub release automatically for
|
||||
**every VERSION bump that lands on master** (trigger: push to master touching
|
||||
`VERSION`, plus `workflow_dispatch` for a manual first run or repair). No
|
||||
manual tag push is part of the ship flow — the workflow reads `VERSION` (the
|
||||
single source of truth), mints tag `v<VERSION>` at the pushed commit, titles
|
||||
the release the same, uses that version's `CHANGELOG.md` entry as the notes
|
||||
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
|
||||
missing), and attaches the compiled binaries.
|
||||
|
||||
### The `latest-stable` tag
|
||||
|
||||
The **final step of the release job** force-advances the `latest-stable` tag to
|
||||
the release commit (`git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"`).
|
||||
`latest-stable` is the single sanctioned distribution ref: the README paste
|
||||
block, the `BOOTSTRAP_FOR_AGENTS.md` fetch URL, and
|
||||
`bun install -g github:garrytan/gbrain#latest-stable` all reference it
|
||||
permanently, so paste blocks copied into the wild never rot and there is no 404
|
||||
window between VERSION landing and assets publishing.
|
||||
`scripts/check-bootstrap-tag.sh` keeps the entry docs pinned to this ref.
|
||||
|
||||
Because it moves ONLY after binaries + provenance attestation have fully
|
||||
published, a half-built release never advances it. If the tag-advance step
|
||||
alone fails, re-advance by hand (a full workflow re-run would skip — the
|
||||
release already exists with all assets):
|
||||
|
||||
```bash
|
||||
git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
|
||||
```
|
||||
|
||||
### The `publish-template` job
|
||||
|
||||
After the release job, a `publish-template` job force-pushes the rendered
|
||||
agent-workspace template repo (the GitHub "Use this template" door,
|
||||
`vars.TEMPLATE_REPO`, default `garrytan/gbrain-agent-template`) from CI only —
|
||||
no human pushes it by hand, so what adopters clone is exactly what this repo
|
||||
reviewed. It is guarded three ways: the release above fully published; the
|
||||
vendored tree `templates/bootstrap/template-repo/` exists (skip, never fail,
|
||||
if not); and the `TEMPLATE_REPO_PAT` secret is configured (skip if not).
|
||||
Before pushing, it regenerates the template tree
|
||||
(`bun run scripts/generate-template-repo.ts`) and byte-diffs it against the
|
||||
vendored copy — a mismatch fails the job; regenerate + commit the vendored
|
||||
tree (`scripts/check-bootstrap-templates.sh` runs the same diff offline in
|
||||
`bun run verify`).
|
||||
|
||||
**`TEMPLATE_REPO_PAT` scope:** a fine-grained PAT with `contents: write` on
|
||||
the template repository ONLY — no other repositories, no other permissions.
|
||||
Configure it as a repo secret; when absent, template publishing is disabled
|
||||
and the job skips cleanly.
|
||||
|
||||
Why every bump, not selective: `gbrain check-update` resolves the latest
|
||||
version from `VERSION` on master, while binary self-update
|
||||
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
|
||||
Any release that lags `VERSION` tells binary installs an upgrade exists that
|
||||
self-update cannot apply. `releases/latest` must track `VERSION`.
|
||||
|
||||
Invariants:
|
||||
|
||||
- **Asset names are a contract.** The build matrix's `artifact:` names must
|
||||
equal what `expectedAssetName()` in `src/core/binary-self-update.ts`
|
||||
returns (`gbrain-darwin-arm64`, `gbrain-linux-x64` today). Adding a
|
||||
platform means updating BOTH plus the version job's completeness check;
|
||||
`test/release-workflow.test.ts` pins all of it.
|
||||
- **Idempotent + self-repairing.** The version job skips when a release for
|
||||
`v<VERSION>` already exists with all expected assets; a partial release
|
||||
(tag but no release, or missing assets) is completed on re-run. Racing
|
||||
master pushes queue via the `release` concurrency group — a skipped
|
||||
intermediate version is fine, latest is what matters.
|
||||
- **Historical tags are never rewritten.** Old 3-segment versions keep their
|
||||
history; every new 4-segment `VERSION` mints a fresh tag.
|
||||
- **Permissions stay scoped.** `contents: write` lives on the release job
|
||||
only; everything else runs read-only.
|
||||
- **Never advance `latest-stable` on a partial release.** The tag moves only
|
||||
as the final release-job step, after every asset has published. Manual
|
||||
re-advances must point at a fully published `v<VERSION>` release.
|
||||
|
||||
## PR descriptions cover the whole branch
|
||||
|
||||
|
||||
+31
-10
@@ -11,13 +11,13 @@ Seven test command tiers, each with a clear scope:
|
||||
|
||||
| Command | What it runs | Wallclock | When to use |
|
||||
|---|---|---|---|
|
||||
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
@@ -32,9 +32,12 @@ CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygw
|
||||
bash that ships with Git for Windows tolerates it, so a green local run is not by
|
||||
itself evidence that a script is CRLF-clean.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
|
||||
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
|
||||
pick it up; see the Windows section of `CONTRIBUTING.md`.
|
||||
`core.autocrlf=true` default that Git for Windows installs. It pins `*.md` the
|
||||
same way, because the frontmatter readers anchor on a `---` fence followed by a
|
||||
Unix line ending and a CRLF checkout makes a document parse as having no
|
||||
frontmatter, silently. Working copies cloned
|
||||
before those pins need a one-time `git rm --cached -r . -q && git reset --hard` to
|
||||
pick them up; see the Windows section of `CONTRIBUTING.md`.
|
||||
|
||||
Wallclock figures in the table above are from a Mac dev box. Windows is
|
||||
substantially slower because each check pays full process-creation cost, and three
|
||||
@@ -44,10 +47,10 @@ there even though they pass on Linux and macOS.
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. CI is the ground truth for "did everything pass."
|
||||
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
|
||||
|
||||
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
|
||||
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
|
||||
|
||||
### Failure-first logging
|
||||
|
||||
@@ -58,19 +61,35 @@ When `bun run test` finds any failure, the wrapper:
|
||||
3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
|
||||
4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
|
||||
|
||||
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
|
||||
If a shard hits the per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap (default 3000s — sized so the heaviest count-balanced shard finishes under 4-way contention; `GBRAIN_TEST_SHARD_KILL_AFTER` sets the grace after TERM before KILL, default 30s), the wrapper classifies the kill one of two ways:
|
||||
|
||||
- **EXIT-HANG → warn-pass.** If the shard's log had been silent for ≥300s at kill time AND shows zero `(fail)` markers, the shard finished all its work, leaked a handle, and never exited (a pre-existing, master-reproducible PGLite-adjacent leak — see TODOS.md "unit-shard exit hang"). The wrapper prints a `⚠️ shard N/M: EXIT-HANG ... Treating as pass-with-warning` banner, writes `EXIT-HANG (idle Ns, 0 fails) ... warn-pass` to the summary, and does NOT fail the run. Its pass counts are undercounted (bun never printed its final summary). Bun's per-test `--timeout` turns a genuinely hung TEST into a printed `(fail)` — new output — so this classification cannot mask a hung test; the residual maskable case is a file-level import hang in the very last file, which the banner keeps visible.
|
||||
- **WEDGED → hard failure.** Anything else (failures present, or the log was still growing) writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log with the last 50 lines of the shard log, marks the run failed, and proceeds with other shards' results.
|
||||
|
||||
Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT a test failure — don't burn time bisecting it; a `WEDGED` line is.
|
||||
|
||||
### File taxonomy
|
||||
|
||||
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
|
||||
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
|
||||
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
|
||||
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
|
||||
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
|
||||
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
|
||||
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
|
||||
|
||||
### Skills-manifest freshness guard
|
||||
|
||||
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
|
||||
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
|
||||
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
|
||||
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
|
||||
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
|
||||
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
|
||||
|
||||
### Test-isolation lint and helpers
|
||||
|
||||
**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules.
|
||||
|
||||
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
|
||||
|
||||
| Rule | What it bans | Fix |
|
||||
@@ -189,6 +208,8 @@ Unit tests and what they cover:
|
||||
- `test/dedup.test.ts` — source-aware dedup, compiled truth guarantee, layer interactions.
|
||||
- `test/intent.test.ts` — query intent classification: entity/temporal/event/general.
|
||||
- `test/eval.test.ts` — retrieval metrics: `precisionAtK`, `recallAtK`, `mrr`, `ndcgAtK`, `parseQrels`.
|
||||
- `test/brainbench-fixtures.test.ts` / `test/brainbench-generator.test.ts` / `test/brainbench-metrics.test.ts` / `test/brainbench-continuity.test.ts` / `test/brainbench-writeback.test.ts` / `test/brainbench-adapters.test.ts` / `test/brainbench-scoreboard.test.ts` — the BrainBench memory-conformance unit suites (`src/eval/brainbench/`): fixture loader/validator + the sealed-gold seal (a `gold` key inside a fixture must reject) and committed-corpus integrity; generator determinism (the committed corpus is exactly what `gen.ts` produces, holdout discipline, category counts); metric formulas over hand-built turn rows (zero should-retrieve turns, empty injections, acceptable-vs-gold asymmetry, micro-averaging); cross-harness continuity (writer's decision persists through the production write-back pipeline, reader recalls on the SAME brain); write-back grading the PRODUCTION conversation→facts pipeline via the injected gold extractor; adapter seam contracts over hermetic PGLite (budget caps, suppression modes); scoreboard + gate governance (baseline determinism, count-aware gating, corpus-bless modes, justification flow, isolation gates-at-zero).
|
||||
- `test/eval-brainbench-e2e.test.ts` — BrainBench CLI end-to-end via subprocess against a small tmp corpus: the literal exit codes (0 pass / 1 regression / 2 error-or-inconclusive — the CI product), `--out` artifact validity incl. `_meta.metric_glossary`, byte-deterministic `--update-baseline`, anti-vacuous-pass, and the `eval run-all` in-process wiring.
|
||||
- `test/check-resolvable.test.ts` — resolver reachability, MECE overlap, gap detection, proximity-based DRY detection, `extractDelegationTargets` coverage.
|
||||
- `test/dry-fix.test.ts` — auto-fix: three shape-aware expander pure-function tests; five guards (working-tree-dirty, no-git-backup, inside-code-fence, already-delegated within 40 lines, ambiguous-multi-match, block-is-callout).
|
||||
- `test/doctor-fix.test.ts` — `gbrain doctor --fix` CLI integration: dry-run preview, apply path, JSON output shape.
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# Upgrading Downstream Agents
|
||||
|
||||
> **Currency note:** this file is an append-only historical log and stopped
|
||||
> receiving new sections after v0.36.5.0. **The canonical, maintained upgrade
|
||||
> channel is `skills/migrations/v*.md`** (the agent-executed migration files
|
||||
> that `gbrain upgrade` / `gbrain post-upgrade` route through), plus
|
||||
> `CHANGELOG.md` for what each release changed. Use this file only to catch a
|
||||
> long-diverged fork up through the versions it covers; for anything after
|
||||
> v0.36.5.0, walk the migration files and CHANGELOG instead.
|
||||
|
||||
GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments,
|
||||
agent forks of any kind) often **copy** these skill files into their own workspace and
|
||||
diverge over time — adding agent-specific phases, removing irrelevant ones, tightening
|
||||
|
||||
+131
-125
File diff suppressed because one or more lines are too long
@@ -4,14 +4,14 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
|
||||
|
||||
## The four strategies in concert
|
||||
|
||||
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
|
||||
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at acme-example?" → pages mentioning "alice-example + retrieval" even when the user never typed "acme".
|
||||
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
|
||||
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
|
||||
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
|
||||
|
||||
## Why each one alone fails
|
||||
|
||||
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
|
||||
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in alice-example's portfolio" returns essays about portfolios, not company pages.
|
||||
|
||||
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
|
||||
|
||||
@@ -21,7 +21,7 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
|
||||
|
||||
## The benchmark
|
||||
|
||||
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus.
|
||||
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus. (This is the retrieval-ranking benchmark; the in-repo `gbrain eval brainbench` suite — [`docs/eval/BRAINBENCH.md`](../eval/BRAINBENCH.md) — gates the memory behaviors *above* retrieval: unprompted context push, write-back fidelity, cross-session continuity.)
|
||||
|
||||
| Strategy | P@5 | R@5 | Notes |
|
||||
|---|---|---|---|
|
||||
@@ -36,8 +36,8 @@ BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/ga
|
||||
|
||||
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
|
||||
|
||||
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
|
||||
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
|
||||
- Standard markdown links: `[Alice Example](wiki/people/alice-example)`
|
||||
- Obsidian wikilinks: `[[wiki/people/alice-example|Alice Example]]`
|
||||
- Typed-link blockquotes: `> **Convention:** see [path](path).`
|
||||
|
||||
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') JOIN pages ON CONFLICT DO NOTHING RETURNING 1` (free-text-safe; the prior `unnest(${arr}::text[])` form crashed on calendar/Zoom context per gbrain#1861). The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
|
||||
@@ -46,7 +46,7 @@ Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`,
|
||||
|
||||
## ZeroEntropy as reranker: 60% top-1 reshuffle
|
||||
|
||||
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
|
||||
ZeroEntropy's `zerank-2` is the default reranker (on for the `balanced` and `tokenmax` mode bundles, off for `conservative`). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
|
||||
|
||||
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
|
||||
|
||||
@@ -62,8 +62,9 @@ The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `Sea
|
||||
|
||||
## Named-thing retrieval (per-page pool + title + alias + evidence)
|
||||
|
||||
A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than
|
||||
embedding proximity. Four layers, added after the incident in
|
||||
A brain organized around *chosen names* (project codenames, place nicknames —
|
||||
say a project named "Helios" whose page is also known as "the Sun Room") needs
|
||||
more than embedding proximity. Four layers, added after the incident in
|
||||
[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md):
|
||||
|
||||
- **Per-page max-pool** — `searchVector` (both engines) collapses chunk-grain
|
||||
@@ -79,7 +80,7 @@ embedding proximity. Four layers, added after the incident in
|
||||
`page_aliases` table (separate from the `slug_aliases` wikilink redirect) and
|
||||
consulted at query time: a full normalized-query match injects/boosts the
|
||||
canonical page (`applyAliasHop`). The only layer that bridges true synonyms
|
||||
with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill
|
||||
with zero surface overlap ("the Sun Room" → the Helios page). Backfill
|
||||
existing pages with `gbrain reindex --aliases`.
|
||||
- **Evidence contract** — every result carries `evidence`
|
||||
(`alias_hit | exact_title_match | high_vector_match | keyword_exact |
|
||||
@@ -97,13 +98,17 @@ Promote or reject them via `gbrain extraction-pending` / `gbrain
|
||||
extraction-review`.
|
||||
|
||||
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
|
||||
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
|
||||
title + alias, expansion off); `query` is the full-control variant. Route
|
||||
concept / landscape / "all-of-X" questions to `query` — expansion recovers
|
||||
synonym-phrased matches `search` can miss, and a populated `search` result set
|
||||
is not proof of coverage (both are top-K; exhaustive enumeration belongs to
|
||||
`list_pages`). NamedThingBench
|
||||
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
|
||||
specific miss with `gbrain search diagnose "<q>" --target <slug>`.
|
||||
|
||||
## Intent-aware query rewriting
|
||||
|
||||
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
|
||||
`src/core/search/query-intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
|
||||
|
||||
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
|
||||
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
|
||||
@@ -116,42 +121,68 @@ The classifier is deterministic (no LLM call). Wrong classification degrades gra
|
||||
|
||||
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
|
||||
|
||||
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
|
||||
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale. The `query` op is the exception: it defaults `expand: true` per call (pass `expand: false` to opt out) — expansion-by-default is what makes it the concept/landscape verb.
|
||||
|
||||
## Putting it together
|
||||
|
||||
The full pipeline for a `query` op:
|
||||
|
||||
```
|
||||
intent classify
|
||||
intent classify (query-intent.ts — deterministic, no LLM)
|
||||
│
|
||||
▼
|
||||
expansion (if enabled)
|
||||
expansion (if enabled — tokenmax only by default)
|
||||
│
|
||||
▼
|
||||
hybrid search:
|
||||
├── vector (HNSW on chunk embeddings)
|
||||
hybrid recall + fusion:
|
||||
├── vector (HNSW on chunk embeddings, per-page max-pool)
|
||||
├── keyword (BM25 via tsvector)
|
||||
├── relational (v0.42.34.0: typed-edge recall arm — relational queries only)
|
||||
├── title-phrase arm
|
||||
├── relational (typed-edge recall arm — relational queries only)
|
||||
├── source-aware re-rank (CASE in SQL)
|
||||
└── RRF fusion → top 30
|
||||
└── RRF fusion → cosine re-score → post-fusion boosts
|
||||
(backlink / salience / recency / graph signals / exact-match)
|
||||
│
|
||||
▼
|
||||
graph augment (typed-edge traversal from any seed)
|
||||
graph augment (optional two-pass structural expansion — walkDepth > 0)
|
||||
│
|
||||
▼
|
||||
reranker (zerank-2 cross-encoder, top 30 → reordered)
|
||||
deduplication (4-layer: per-page cap, Jaccard, type diversity)
|
||||
│
|
||||
▼
|
||||
token-budget enforcement (per mode bundle)
|
||||
reranker (zerank-2 cross-encoder — balanced/tokenmax; fail-open)
|
||||
│
|
||||
▼
|
||||
deduplication (same slug, different chunks → keep best)
|
||||
alias hop (exact alias match injects/boosts the canonical page)
|
||||
│
|
||||
▼
|
||||
evidence stamp → adaptive return (opt-in) → autocut (reranked modes)
|
||||
│
|
||||
▼
|
||||
limit slice → token-budget enforcement (per mode bundle)
|
||||
│
|
||||
▼
|
||||
results
|
||||
```
|
||||
|
||||
The stage order is pinned by `hybridSearch` in `src/core/search/hybrid.ts`:
|
||||
dedup runs BEFORE the reranker (so the reranker sees a diverse candidate pool,
|
||||
capped by its own `topNIn`), the alias hop runs AFTER the reranker (so a query
|
||||
that is a page's declared name reliably surfaces that page regardless of how
|
||||
the reranker scored body chunks), and the token budget is enforced last, on
|
||||
the final slice.
|
||||
|
||||
### Autocut: score-discontinuity result-sizing
|
||||
|
||||
Default-on for `balanced` and `tokenmax` (off for `conservative`, which has no
|
||||
reranker and therefore no trustworthy cliff signal). `applyAutocut`
|
||||
(`src/core/search/autocut.ts`) cuts the ranked set at the largest
|
||||
cross-encoder rerank-score cliff, before the limit slice, first page only.
|
||||
Never-empty failsafe (`minKeep`), no-op when fewer than 2 results carry a
|
||||
finite rerank score (covers the fail-open reranker path), and alias-hop exact
|
||||
matches are preserved through the cut. Knobs: per-call `SearchOpts.autocut` →
|
||||
`search.autocut` / `search.autocut_jump` config → mode bundle.
|
||||
|
||||
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
|
||||
|
||||
## How to verify on your own brain
|
||||
|
||||
@@ -19,18 +19,18 @@ need to understand both of them, or queries misroute silently.
|
||||
A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase.
|
||||
Each brain has:
|
||||
- Its own `pages` table, `chunks` table, `embeddings`, etc.
|
||||
- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2).
|
||||
- Its own OAuth surface if served over HTTP MCP.
|
||||
- Its own separate lifecycle, backup, access control.
|
||||
|
||||
Brains are enumerated by:
|
||||
- **host** — your default brain, configured in `~/.gbrain/config.json`.
|
||||
- **mounts** — additional brains registered in `~/.gbrain/mounts.json` via
|
||||
`gbrain mounts add <id>` (v0.19+).
|
||||
`gbrain mounts add <id>`.
|
||||
|
||||
Routing: `--brain <id>`, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or
|
||||
longest-path match against registered mount paths. Falls back to `host`.
|
||||
|
||||
### Sources (the repo axis, v0.18.0+)
|
||||
### Sources (the repo axis)
|
||||
|
||||
A **source** is a named content repo *inside* one brain. Every `pages` row
|
||||
carries a `source_id`. Slugs are unique per source, not globally.
|
||||
@@ -142,7 +142,7 @@ Use this topology when:
|
||||
|
||||
You're senior enough to sit across multiple teams. You maintain your personal
|
||||
brain (with N sources inside) AND mount several work team brains. Each team
|
||||
brain is itself a multi-source brain in the v0.18.0 sense — organized
|
||||
brain is itself a multi-source brain — organized
|
||||
internally however the team owner chose.
|
||||
|
||||
```
|
||||
@@ -181,7 +181,7 @@ Use this topology when:
|
||||
- You need latent-space federation (agent decides when to query across
|
||||
brains), not SQL federation.
|
||||
|
||||
Cross-brain queries are **not deterministic** in v0.19. The agent sees the
|
||||
Cross-brain queries are **not deterministic**. The agent sees the
|
||||
brain list and re-queries as needed. That's the feature — it keeps debugging
|
||||
sane and access control clean.
|
||||
|
||||
@@ -202,6 +202,13 @@ WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)?
|
||||
Both axes follow the same layered pattern on purpose. If you know one, you
|
||||
know the other.
|
||||
|
||||
One addition on the source axis for remote (MCP/OAuth) callers: a client
|
||||
registered with federated reads carries `ctx.auth.allowedSources` — an
|
||||
ARRAY of readable sources that takes precedence over the scalar
|
||||
`ctx.sourceId` on every read path (`sourceScopeOpts(ctx)` in the
|
||||
operations layer). Local CLI callers never set it; the scalar chain above
|
||||
is the whole story for them.
|
||||
|
||||
---
|
||||
|
||||
## For agents reading this
|
||||
@@ -236,7 +243,7 @@ know the other.
|
||||
|
||||
## Further reading
|
||||
|
||||
- v0.18.0 CHANGELOG — introduced `sources` primitive.
|
||||
- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`.
|
||||
- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain
|
||||
publisher, not just the subscriber.
|
||||
- [`topologies.md`](./topologies.md) — where the DB lives (operator recipes
|
||||
for each deployment shape).
|
||||
- `skills/conventions/brain-routing.md` — the agent-facing decision table.
|
||||
- `CHANGELOG.md` — release history for the `sources` and `mounts` primitives.
|
||||
|
||||
@@ -10,14 +10,16 @@
|
||||
> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80
|
||||
> during successive master merges — v0.37.0.0's autonomous-remediation wave
|
||||
> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79.
|
||||
> - **Follow-up minor** (forthcoming): falsifiability + category extraction at
|
||||
> `propose_takes`, SQL-side grade gate, per-category calibration scorecards,
|
||||
> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation
|
||||
> against the v0.36.1.0 fixtures.
|
||||
> - **Follow-up minor — NEVER IMPLEMENTED.** The falsifiability + category
|
||||
> extraction at `propose_takes`, SQL-side grade gate, per-category
|
||||
> calibration scorecards, and pg_trgm-based proposal dedup described in the
|
||||
> sections below remain UNSHIPPED design. Do not read §§1–4 as current
|
||||
> behavior; only the `unresolvable` hotfix above landed.
|
||||
>
|
||||
> Preserved here per the hotfix plan's PR #1191 close protocol so the
|
||||
> production context (96K-page brain, 6.8% falsifiability rate, category
|
||||
> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation.
|
||||
> production context (falsifiability rate + category breakdown observed on a
|
||||
> large real brain) doesn't get lost in the CHANGELOG → release-notes
|
||||
> condensation.
|
||||
|
||||
## Problem
|
||||
|
||||
|
||||
@@ -68,11 +68,13 @@ existing.
|
||||
## Migration shape
|
||||
|
||||
```ts
|
||||
// src/core/migrate.ts — append after the v80 entry
|
||||
// src/core/migrate.ts — append after the CURRENT last entry in the
|
||||
// MIGRATIONS array (take the next unused version number at implementation
|
||||
// time; the numbers below are placeholders, not a reserved slot)
|
||||
const migrations = [
|
||||
// ...existing v1-v80...
|
||||
// ...existing entries...
|
||||
{
|
||||
version: 81,
|
||||
version: NEXT_VERSION, // next unused number in the MIGRATIONS array
|
||||
name: 'frontmatter_scan_state',
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
|
||||
@@ -194,7 +196,7 @@ stale data as authoritative.
|
||||
```
|
||||
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
|
||||
Design lives at docs/architecture/frontmatter-scan-incremental.md.
|
||||
Schema migration v81 + sync-side UPSERT + incremental scan command
|
||||
New schema migration + sync-side UPSERT + incremental scan command
|
||||
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
|
||||
table + writes first; flip the reader one release later.
|
||||
```
|
||||
|
||||
@@ -1,105 +1,33 @@
|
||||
# GBrain Infrastructure Layer
|
||||
# GBrain Infrastructure Layer (orientation pointer)
|
||||
|
||||
The shared foundation that all skills, recipes, and integrations build on.
|
||||
This page is a router — the detailed, current-state references live in the
|
||||
docs below (this file once carried its own copies of the pipeline and schema;
|
||||
those rotted, so each concept now has exactly one home).
|
||||
|
||||
## Data Pipeline
|
||||
## Where things live
|
||||
|
||||
```
|
||||
INPUT (markdown files, git repo)
|
||||
↓
|
||||
FILE RESOLUTION (local → .redirect → .supabase → error)
|
||||
↓
|
||||
MARKDOWN PARSER (gray-matter frontmatter + body)
|
||||
→ compiled_truth + timeline separation
|
||||
↓
|
||||
CONTENT HASH (SHA-256 idempotency check — skip if unchanged)
|
||||
↓
|
||||
CHUNKING (3 strategies, configurable)
|
||||
├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy
|
||||
├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing
|
||||
└── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates
|
||||
↓
|
||||
EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions)
|
||||
→ batch 100, exponential backoff, non-fatal if fails
|
||||
↓
|
||||
DATABASE TRANSACTION (atomic: page + chunks + tags + version)
|
||||
↓
|
||||
SEARCH (hybrid, available immediately)
|
||||
```
|
||||
|
||||
## Search Architecture
|
||||
|
||||
GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search:
|
||||
|
||||
```
|
||||
User Query
|
||||
↓
|
||||
EXPANSION (optional: Claude Haiku generates 2 alternative phrasings)
|
||||
↓
|
||||
├── VECTOR SEARCH (pgvector HNSW, cosine distance)
|
||||
│ → 2x limit results per query variant
|
||||
│
|
||||
└── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank)
|
||||
→ 2x limit results
|
||||
↓
|
||||
RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly)
|
||||
↓
|
||||
4-LAYER DEDUP
|
||||
├── Best 3 chunks per page (source dedup)
|
||||
├── Jaccard similarity > 0.85 (text dedup)
|
||||
├── No type exceeds 60% (diversity)
|
||||
└── Max 2 chunks per page (page cap)
|
||||
↓
|
||||
TOP N RESULTS (default 20)
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) |
|
||||
| `src/core/postgres-engine.ts` | Postgres + pgvector implementation |
|
||||
| `src/core/import-file.ts` | importFromFile + importFromContent pipeline |
|
||||
| `src/core/sync.ts` | Git-based incremental change detection |
|
||||
| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing |
|
||||
| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff |
|
||||
| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) |
|
||||
| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection |
|
||||
| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking |
|
||||
| `src/core/search/hybrid.ts` | RRF merge of vector + keyword |
|
||||
| `src/core/search/dedup.ts` | 4-layer result deduplication |
|
||||
| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku |
|
||||
| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) |
|
||||
| `src/core/operations.ts` | Contract-first operation definitions (31 ops) |
|
||||
| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) |
|
||||
|
||||
## Schema Overview
|
||||
|
||||
10 tables in Postgres:
|
||||
|
||||
- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB)
|
||||
- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline)
|
||||
- **links** — typed edges (knows, works_at, invested_in, founded, etc.)
|
||||
- **tags** — many-to-many page tagging
|
||||
- **timeline_entries** — structured events (date, source, summary, detail)
|
||||
- **page_versions** — snapshot history for diff/revert
|
||||
- **raw_data** — sidecar JSON from external APIs (preserves provenance)
|
||||
- **files** — binary attachments in storage backend
|
||||
- **ingest_log** — audit trail of import operations
|
||||
- **config** — brain-level settings (version, embedding model, chunk strategy)
|
||||
|
||||
Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C).
|
||||
Vector search uses HNSW index with cosine distance on content_chunks.embedding.
|
||||
| Topic | Home |
|
||||
|---|---|
|
||||
| Ingest pipeline (file resolution → frontmatter parse → content-hash idempotency → chunking → embedding → atomic write) | per-file entries in [`KEY_FILES.md`](./KEY_FILES.md): `src/core/import-file.ts`, `src/core/sync.ts`, `src/core/markdown.ts`, `src/core/embedding.ts`, `src/core/chunkers/*` |
|
||||
| Chunking strategies (recursive / semantic / LLM-guided) | `src/core/chunkers/{recursive,semantic,llm}.ts` entries in [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Search pipeline (hybrid RRF, graph, reranker, autocut, dedup, budgets) | [`RETRIEVAL.md`](./RETRIEVAL.md) |
|
||||
| Search modes + cost knobs | `docs/guides/search-modes.md` + the CLAUDE.md Search Mode table |
|
||||
| Per-file index of `src/` (what each file does + its invariants) | [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Schema DDL | the `MIGRATIONS` array in `src/core/migrate.ts` (source of truth) + `src/schema.sql`; per-table classification in [`system-of-record.md`](./system-of-record.md) |
|
||||
| Engines (PGLite vs Postgres, parity rules) | `docs/ENGINES.md` + the engine entries in [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Operations contract (CLI + MCP generated from one source) | `src/core/operations.ts` (100+ operations; run `gbrain --tools-json` for the live list) |
|
||||
| Brains vs sources (which database vs which repo inside it) | [`brains-and-sources.md`](./brains-and-sources.md) |
|
||||
|
||||
## The Thin Harness Principle
|
||||
|
||||
GBrain is the deterministic layer. Skills and recipes are the latent space layer.
|
||||
GBrain is the deterministic layer. Skills and recipes are the latent-space layer.
|
||||
|
||||
See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full
|
||||
architecture philosophy.
|
||||
|
||||
- **GBrain CLI** = thin harness (same input → same output)
|
||||
- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills
|
||||
- **Skills** (the bundled set routed by `skills/RESOLVER.md`) = fat skills
|
||||
- **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure
|
||||
|
||||
The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Lens packs (v0.41.2.0)
|
||||
# Lens packs
|
||||
|
||||
Four bundled schema packs that turn the gbrain dream cycle into a multi-lens
|
||||
brain. Activate one with `gbrain config set schema_pack <name>` and the cycle
|
||||
@@ -7,7 +7,7 @@ picks up the pack's declared phases on the next `gbrain dream` run.
|
||||
## The four packs
|
||||
|
||||
```
|
||||
gbrain-base (shipped v0.38)
|
||||
gbrain-base
|
||||
▲
|
||||
│ extends
|
||||
┌──────────────┼──────────────────────┐
|
||||
@@ -60,37 +60,33 @@ conviction so high-stakes misses cost more).
|
||||
|
||||
### gbrain-engineer
|
||||
Bridge-only pack. Declares `learning` page type + reuses base `code`.
|
||||
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource
|
||||
(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
|
||||
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
|
||||
each JSONL line as a `learning` page when this pack is active. Three
|
||||
calibration domains: `architecture_calls` (scalar_brier),
|
||||
`effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier).
|
||||
|
||||
Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred
|
||||
to v0.42+ — they'll ship when a real user authors the first one (D8).
|
||||
Speculative ADR/postmortem/refactor_thesis/tech_debt types are
|
||||
deferred — they'll ship when a real user authors the first one.
|
||||
|
||||
### gbrain-everything
|
||||
Meta-pack stacking creator + investor + engineer via the v0.38
|
||||
Meta-pack stacking creator + investor + engineer via the
|
||||
`extends` + `borrow_from` chain. Single-active-pack constraint
|
||||
preserved — this IS the active pack; the registry walks extends +
|
||||
borrow to materialize the merged view.
|
||||
|
||||
**Merge contract (T20 / #1749).** `resolvePack` merges parent → child
|
||||
(child-wins) for the six ingest/query-shaping fields: `page_types`,
|
||||
`link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`,
|
||||
and `takes_kinds` (unioned — a child cannot narrow it). `phases` and
|
||||
`calibration_domains` are **NOT** inherited: they gate cycle execution,
|
||||
so each pack must declare its own participation explicitly. That is why
|
||||
`gbrain-everything` re-declares all its phases and all 7
|
||||
`calibration_domains` — inheritance does not carry them.
|
||||
**Merge contract.** The full `extends` + `borrow_from` merge rules live in
|
||||
[`schema-packs.md` § Merge contract](./schema-packs.md#merge-contract-extends--borrow_from).
|
||||
The one rule that matters here: `phases` and `calibration_domains` are
|
||||
**NOT** inherited (they gate cycle execution, so each pack must declare
|
||||
its own participation explicitly) — which is why `gbrain-everything`
|
||||
re-declares all its phases and all 7 `calibration_domains`.
|
||||
|
||||
Activate via `gbrain config set schema_pack gbrain-everything` and
|
||||
calibration_profile produces all 7 domain scorecards in one JSONB.
|
||||
|
||||
## Calibration profile widening (T10)
|
||||
## Calibration profile domains
|
||||
|
||||
Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a
|
||||
`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared
|
||||
Each declared
|
||||
domain produces a `{n, brier, accuracy, aggregator, page_types,
|
||||
extras}` entry. Four aggregator algorithms (closed enum):
|
||||
|
||||
@@ -109,20 +105,20 @@ Domain names are OPEN (third-party packs can declare new domain labels
|
||||
without a gbrain release). Aggregator algorithms are CLOSED (safe SQL
|
||||
stays in code, validated at pack-load).
|
||||
|
||||
## take_domain_assignments table (T1)
|
||||
## take_domain_assignments table
|
||||
|
||||
New JOIN table (migration v94):
|
||||
JOIN table (migration v94):
|
||||
`take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT,
|
||||
source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id,
|
||||
domain))`. Multi-domain assignment honest — a take about "Sequoia's
|
||||
investment in Anthropic" can land in BOTH `deal_success` AND
|
||||
domain))`. Multi-domain assignment honest — a take about "fund-a's
|
||||
investment in acme-example" can land in BOTH `deal_success` AND
|
||||
`market_call` rather than being force-bucketed.
|
||||
|
||||
## What this enables for the user
|
||||
|
||||
- **Atoms + concepts ship in the binary.** Your OpenClaw's parallel
|
||||
atom-pipeline-coordinator + atom-backfill-coordinator + concept-
|
||||
synthesis crons can retire (T12 follow-up). One `gbrain dream` cron
|
||||
synthesis crons can retire. One `gbrain dream` cron
|
||||
covers everything.
|
||||
- **gstack learnings reach gbrain.** Engineer-pack-active brains
|
||||
surface every gstack-logged learning as a queryable page within
|
||||
@@ -131,22 +127,22 @@ investment in Anthropic" can land in BOTH `deal_success` AND
|
||||
often you're wrong on deals AND market calls AND architecture
|
||||
AND effort estimates in one `gbrain calibration --json` call.
|
||||
- **Lossless OpenClaw migration.** The `markdown-greenfield`
|
||||
importer (T7, mode='migration') re-ingests existing OpenClaw
|
||||
importer (mode='migration') re-ingests existing OpenClaw
|
||||
pages with permanent slug-keyed idempotency + per-row JSONL audit
|
||||
+ the `imported_from` marker so extract_atoms + synthesize_concepts
|
||||
don't re-extract already-atomized material.
|
||||
|
||||
## v0.41.2.1 follow-ups (filed in plan)
|
||||
## Known gaps / deferred follow-ups
|
||||
|
||||
- Per-page-type `frontmatter_validators` on PageTypeSchema so the
|
||||
atom_type enum (currently hardcoded in extract_atoms.ts) reads from
|
||||
the active pack manifest at runtime per D11.
|
||||
the active pack manifest at runtime.
|
||||
- 3-check quality gate (truism / punchline / entity-page reject) as
|
||||
a multi-pass extract_atoms refinement.
|
||||
- Embedding-similarity dedup in synthesize_concepts (currently
|
||||
exact-string concept ref match only).
|
||||
- Voice gate integration for T1 Canon narratives.
|
||||
- Voice gate integration for concept narratives.
|
||||
- op_checkpoint resumability for cross-cycle continuation in both
|
||||
phases.
|
||||
- Parity-baseline eval gates against your OpenClaw's existing 13K atoms
|
||||
+ 11K concepts on a 500-page sample subset.
|
||||
- Parity-baseline eval gates against a pre-existing downstream
|
||||
atom/concept corpus on a sample subset.
|
||||
|
||||
@@ -56,7 +56,8 @@ that tuple lights up the `pack_upgrade_available` onboard check.
|
||||
│ gbrain onboard --check --explain shows per-cluster narrative │
|
||||
│ User reviews; if OK, runs: │
|
||||
│ gbrain jobs submit unify-types --allow-protected \ │
|
||||
│ --params '{"target_pack":"gbrain-base-v2"}' │
|
||||
│ --params '{"target_pack":"gbrain-base-v2","apply":true}' │
|
||||
│ (omit "apply":true for a dry-run; that is the default) │
|
||||
│ (Autopilot never auto-fires this; manual_only) │
|
||||
└──────────────────────────┬─────────────────────────────────────┘
|
||||
↓
|
||||
@@ -126,10 +127,10 @@ candidate ≠ the active pack name, loads the manifest via
|
||||
migration_from.version)`. Returns matching packs sorted by version
|
||||
descending.
|
||||
|
||||
v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed
|
||||
packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the
|
||||
filesystem-scan cost needs the cache invalidation strategy from
|
||||
`registry.ts`).
|
||||
Successor detection covers bundled packs only. Future work: enumerate
|
||||
user-installed packs at `~/.gbrain/schema-packs/*/pack.yaml` (deferred
|
||||
because the filesystem-scan cost needs the cache invalidation strategy
|
||||
from `registry.ts`).
|
||||
|
||||
## The manual_only apply policy
|
||||
|
||||
@@ -172,8 +173,8 @@ migration_from:
|
||||
version: "1.x"
|
||||
|
||||
page_types:
|
||||
# Inherit gbrain-base-v2's 15 types here (or use extends to merge
|
||||
# automatically once v0.43+ extends-chain composition lands)
|
||||
# Inherit gbrain-base-v2's 15 types here (or declare `extends:
|
||||
# gbrain-base-v2` and let the merge contract in schema-packs.md merge them)
|
||||
- { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true }
|
||||
- { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true }
|
||||
# ... all 13 other v2 canonicals ...
|
||||
@@ -221,17 +222,17 @@ Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl`
|
||||
identities (before + after), per-phase counts (would_apply + applied),
|
||||
warnings, completion timestamp. Privacy: page slugs are NOT logged in
|
||||
bulk (only the per-rule sample_slugs[≤10]); for forensic debugging
|
||||
add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired.
|
||||
|
||||
## What's NOT yet supported
|
||||
|
||||
- Subprocess sandbox for the publish-gate (v0.43+ TODO)
|
||||
- Subprocess sandbox for the publish-gate
|
||||
- Per-source pack-upgrade (the handler accepts `sourceId` but
|
||||
`findPackSuccessors` doesn't yet pass it through)
|
||||
- Cross-brain federated mounts that disagree on canonical packs
|
||||
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
|
||||
- LLM-assisted mapping_rules codegen from production data (`gbrain
|
||||
schema detect-mappings`; deferred to v0.43+)
|
||||
- Automatic rollback (today: manual SQL or `gbrain restore`)
|
||||
- LLM-assisted mapping_rules codegen from production data (a proposed
|
||||
`gbrain schema detect-mappings`)
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -241,6 +242,6 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
- Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable`
|
||||
- Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS`
|
||||
- Handler: `src/core/schema-pack/unify-types-handler.ts`
|
||||
- Migration: `src/core/migrate.ts:105` (slug_aliases table)
|
||||
- Migration: the `slug_aliases` entry in `src/core/migrate.ts`'s `MIGRATIONS` array
|
||||
- Type taxonomy doc: `docs/architecture/type-taxonomy.md`
|
||||
- Skill: `skills/schema-unify/SKILL.md`
|
||||
|
||||
@@ -7,20 +7,25 @@ paths, and which link verbs connect what to what. The schema pack is the
|
||||
querying, or routing experts. It is the single source of truth for
|
||||
"what's in your brain."
|
||||
|
||||
The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the
|
||||
user-facing reference; for implementation details see
|
||||
`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in
|
||||
`src/core/schema-pack/`.
|
||||
This doc is the user-facing reference; for implementation details see
|
||||
`docs/designs/V038_SCHEMA_PACKS.md` (the original design) and the engine
|
||||
layer in `src/core/schema-pack/`.
|
||||
|
||||
## What ships in the box
|
||||
|
||||
Two bundled packs:
|
||||
Seven bundled packs (`src/core/schema-pack/base/`):
|
||||
|
||||
- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior
|
||||
byte-for-byte. Existing brains see zero behavior change after upgrade.
|
||||
Covers: person, company, deal, meeting, project, place, concept, writing,
|
||||
analysis, guide, hardware, architecture, etc. (the original
|
||||
`ALL_PAGE_TYPES` list).
|
||||
- **`gbrain-base-v2`** — the 15-type canonical taxonomy. Fresh installs
|
||||
(`gbrain init`) activate this by default. See
|
||||
[`type-taxonomy.md`](./type-taxonomy.md) for the full type list and the
|
||||
upgrade path from `gbrain-base`.
|
||||
|
||||
- **`gbrain-base`** — the original hardcoded behavior, byte-for-byte
|
||||
(person, company, deal, meeting, project, place, concept, writing,
|
||||
analysis, guide, hardware, architecture, etc. — the original
|
||||
`ALL_PAGE_TYPES` list). Still the resolution-chain fallback (tier 7)
|
||||
for brains with no pack configured anywhere, so pre-existing brains see
|
||||
zero behavior change until they opt in to something newer.
|
||||
|
||||
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional
|
||||
directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal,
|
||||
@@ -32,12 +37,17 @@ Two bundled packs:
|
||||
gbrain schema use gbrain-recommended
|
||||
```
|
||||
|
||||
- **`gbrain-creator`**, **`gbrain-investor`**, **`gbrain-engineer`**,
|
||||
**`gbrain-everything`** — the lens packs, which add cycle phases and
|
||||
calibration domains on top of the base taxonomy. See
|
||||
[`lens-packs.md`](./lens-packs.md).
|
||||
|
||||
Plus user-installed packs at `~/.gbrain/schema-packs/<name>/pack.yaml`
|
||||
that you author with `gbrain schema init` or `gbrain schema fork`.
|
||||
|
||||
## CLI surface
|
||||
|
||||
Five inspection verbs (shipped in v0.38):
|
||||
Inspection verbs:
|
||||
|
||||
```bash
|
||||
gbrain schema active # show resolved pack + which tier set it
|
||||
@@ -47,7 +57,7 @@ gbrain schema validate # validate a manifest's shape
|
||||
gbrain schema use <pack> # activate a pack (writes ~/.gbrain/config.json)
|
||||
```
|
||||
|
||||
Eight authoring + discovery verbs (shipped in v0.39):
|
||||
Authoring + discovery verbs:
|
||||
|
||||
```bash
|
||||
gbrain schema detect # propose types matching brain shape
|
||||
@@ -62,12 +72,12 @@ gbrain schema graph # ASCII type listing (experimental)
|
||||
gbrain schema lint # flag duplicates + missing prefixes
|
||||
gbrain schema explain <type> # plain-English type description (experimental)
|
||||
gbrain schema downgrade --to <p> # restore previous pack (recovery)
|
||||
gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry)
|
||||
gbrain schema usage --since 30d # per-verb invocation counts (telemetry)
|
||||
```
|
||||
|
||||
The verbs marked `experimental` are demand-gated per D14: their usage is
|
||||
tracked via T15's schema-events audit, and v0.40+ retro decides whether
|
||||
to deprecate any that stay <5% usage.
|
||||
The verbs marked `experimental` are demand-gated: usage is tracked via the
|
||||
schema-events audit (`gbrain schema usage`), which informs whether
|
||||
rarely-used verbs get deprecated.
|
||||
|
||||
## Resolution chain (7 tiers)
|
||||
|
||||
@@ -78,10 +88,10 @@ this chain top-down. First match wins.
|
||||
|------|--------|-------|
|
||||
| 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. |
|
||||
| 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. |
|
||||
| 3 | Per-source DB config key `schema_pack:source:<id>` | New in v0.38. |
|
||||
| 3 | Per-source DB config key `schema_pack:source:<id>` | |
|
||||
| 4 | Brain-wide DB config key `schema_pack` | |
|
||||
| 5 | `gbrain.yml schema:` section | Repo-checked. |
|
||||
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. |
|
||||
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` (and `gbrain init`, which sets `gbrain-base-v2`) writes. |
|
||||
| 7 | Default: `gbrain-base` | Always present. |
|
||||
|
||||
## How the agent uses the active pack
|
||||
@@ -97,18 +107,18 @@ Every read + write path consults the active pack at runtime:
|
||||
- **`extract_facts`** runs only on `extractable: true` types.
|
||||
- **`enrichment-service`** routes person/company enrichment based on the
|
||||
pack's primitive declarations.
|
||||
- **Search hybrid cache** (`knobsHash`) folds in pack name + version
|
||||
(v0.39 T21). A cache row written under pack A is unreachable when pack
|
||||
- **Search hybrid cache** (`knobsHash`) folds in pack name + version.
|
||||
A cache row written under pack A is unreachable when pack
|
||||
B is active. Cross-pack contamination is structurally impossible.
|
||||
|
||||
## The magical moment (T2-T4 + T10)
|
||||
## The magical moment
|
||||
|
||||
Persona A (Notion refugee) installs gbrain, imports her exports, and the
|
||||
brain looks unfamiliar — the default `gbrain-base` pack expects
|
||||
`people/`, `companies/`, etc., but her files live under `Projects/`,
|
||||
`Reading/`, `Daily Notes/`. The friction signal fires in two places:
|
||||
|
||||
1. **Import warn (T7):** the end of `gbrain import` prints
|
||||
1. **Import warn:** the end of `gbrain import` prints
|
||||
`[schema] X of Y pages (Z%) have no type matching the active schema
|
||||
pack. Run gbrain schema detect to propose a pack matching your
|
||||
content shape.`
|
||||
@@ -124,7 +134,7 @@ gbrain schema review-candidates # human gate on promotion
|
||||
gbrain schema review-candidates --apply Projects/ # accept
|
||||
```
|
||||
|
||||
The agent (via the new EIIRP skill) automates phases 1-3 of this for any
|
||||
The agent (via the EIIRP skill, `skills/eiirp/SKILL.md`) automates phases 1-3 of this for any
|
||||
significant work session. The brain's schema becomes a living artifact
|
||||
the agent maintains, not a hardcoded ceremony the user authors.
|
||||
|
||||
@@ -172,9 +182,10 @@ filing_rules: []
|
||||
|
||||
## Merge contract (`extends` + `borrow_from`)
|
||||
|
||||
This section is the single home for the merge rules (other docs link here).
|
||||
`resolvePack` composes a pack against its `extends` chain (and any
|
||||
`borrow_from` targets) into the `resolved.manifest` every consumer reads
|
||||
(T20 / #1749). The rules:
|
||||
`borrow_from` targets) into the `resolved.manifest` every consumer reads.
|
||||
The rules:
|
||||
|
||||
- **Six fields inherit, child-wins:** `page_types`, `link_types`,
|
||||
`frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`.
|
||||
@@ -200,9 +211,8 @@ filing_rules: []
|
||||
|
||||
## Recovery + revert
|
||||
|
||||
The single-PR cathedral is hard to revert atomically. Per codex finding
|
||||
#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore
|
||||
the active-pack config field:
|
||||
A pack activation is config, not code, so reverting code alone doesn't
|
||||
undo it. `gbrain schema downgrade` restores the active-pack config field:
|
||||
|
||||
```bash
|
||||
gbrain schema downgrade --to gbrain-base
|
||||
@@ -214,19 +224,19 @@ gbrain schema downgrade
|
||||
|
||||
1. `git revert <merge-commit>` — restores the code.
|
||||
2. `gbrain schema downgrade --to gbrain-base` — restores config.
|
||||
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
|
||||
v0.39-typed pages that no longer have a matching type in the active
|
||||
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — hard-deletes
|
||||
soft-deleted pages that no longer have a matching type in the active
|
||||
pack.
|
||||
|
||||
The cache + eval rows that pack-aware code wrote are isolated by the
|
||||
`knobsHash` pack-folding (T21) — they become unreachable under the
|
||||
`knobsHash` pack-folding — they become unreachable under the
|
||||
restored pack so no eviction is needed.
|
||||
|
||||
## Distribution
|
||||
|
||||
`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as
|
||||
`.gbrain-skillpack` tarballs (T14 artifact abstraction). The
|
||||
discriminator is `api_version` in the manifest:
|
||||
`.gbrain-schema` tarballs ride the same distribution pipeline as
|
||||
`.gbrain-skillpack` tarballs. The discriminator is `api_version` in the
|
||||
manifest:
|
||||
|
||||
- `gbrain-schema-pack-v1` → schemapack
|
||||
- `gbrain-skillpack-v1` → skillpack
|
||||
@@ -237,22 +247,17 @@ respectively.
|
||||
|
||||
Publication to the public registries (`garrytan/gbrain-schema-registry`,
|
||||
`garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR
|
||||
workflow as v0.37 skillpack publishing.
|
||||
workflow as skillpack publishing.
|
||||
|
||||
## What's deferred to v0.40+
|
||||
## Known limits / deferred work
|
||||
|
||||
- **Per-source pack federation across mounts.** A query crossing multiple
|
||||
sources currently rejects with `permission_denied` when those sources
|
||||
have divergent active packs (T19 + codex finding #2). The v0.40+ work
|
||||
computes a true per-source closure via the existing
|
||||
`buildSourceClosureCte` engine surface.
|
||||
- **`extends` chain semver compatibility checks** between pack versions.
|
||||
- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack
|
||||
can declare "I work best with these primitives present in your pack."
|
||||
- **Live schema migration helpers** — when you add a type, auto-suggest
|
||||
backfill of existing pages.
|
||||
- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the
|
||||
full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+
|
||||
retro reads T23 usage telemetry to decide which to deprecate.
|
||||
sources rejects with `permission_denied` when those sources have
|
||||
divergent active packs (`src/core/schema-pack/op-trust-gate.ts`). A true
|
||||
per-source closure via the existing `buildSourceClosureCte` engine
|
||||
surface remains future work.
|
||||
- **Pack version upgrades** (e.g. `gbrain-base` → `gbrain-base-v2`) are
|
||||
handled by the successor-detection + unify-types mechanism — see
|
||||
[`pack-upgrade-mechanism.md`](./pack-upgrade-mechanism.md).
|
||||
|
||||
See `TODOS.md` v0.40+ section for the full deferred list.
|
||||
The live deferred list is in `TODOS.md`.
|
||||
|
||||
@@ -52,3 +52,13 @@ gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
|
||||
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
|
||||
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
|
||||
|
||||
The manual diagnosis above has an automated cousin: the progress-aware stall
|
||||
watchdog. If the import drain makes no forward progress for
|
||||
`GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900; keyed on file-import
|
||||
progress, not the lock heartbeat), the run aborts with
|
||||
`reason: 'stall_timeout'` and releases the per-source lock so the next
|
||||
`gbrain sync` resumes from the checkpoint. It fires BETWEEN files — a hang
|
||||
inside one file's import runs until the wall-clock hard deadline. `0`
|
||||
disables it. The full sync-resumability knob table lives in CLAUDE.md
|
||||
("Sync resumability + lock tuning").
|
||||
|
||||
@@ -19,11 +19,13 @@ entire DB from scratch.
|
||||
|
||||
This means:
|
||||
|
||||
- **Disaster recovery is one command.** If your DB volume corrupts, if
|
||||
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
|
||||
a backup. You wipe the DB, re-import from your brain repo, and the
|
||||
derived state regenerates. v0.32.3 ships `gbrain rebuild
|
||||
--confirm-destructive` as the documented one-liner.
|
||||
- **Disaster recovery is a short, boring sequence.** If your DB volume
|
||||
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
|
||||
don't need a backup. You wipe the derived tables (on PGLite,
|
||||
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
|
||||
your brain repo with `gbrain sync`, and `gbrain extract all`
|
||||
regenerates the derived state. See "Disaster recovery" below for the
|
||||
exact commands.
|
||||
- **Multi-machine sync is git.** Your brain is a repo. Push from one
|
||||
machine, pull from another, and the second machine's DB rebuilds on
|
||||
its next sync. No "back up the database" step.
|
||||
@@ -86,6 +88,7 @@ the repo. The architectural rule still holds — these aren't
|
||||
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
|
||||
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
|
||||
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
|
||||
| `op_checkpoint_paths` | Sync-resume checkpoint. Append-only progress banking; a completed sync makes it irrelevant. |
|
||||
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
|
||||
|
||||
A new derived table that holds user-knowledge MUST land FS-first.
|
||||
@@ -146,11 +149,9 @@ The promise the rule makes:
|
||||
# Snapshot what's there
|
||||
gbrain stats > /tmp/before.txt
|
||||
|
||||
# Wipe and rebuild
|
||||
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
|
||||
# (pages + content_chunks survive
|
||||
# the CASCADE-safe design)
|
||||
# OR manually for v0.32.2:
|
||||
# Wipe and rebuild — delete the derived tables (pages + content_chunks
|
||||
# survive the CASCADE-safe design), then re-derive from the repo.
|
||||
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
|
||||
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
|
||||
gbrain sync
|
||||
gbrain extract all
|
||||
@@ -189,9 +190,6 @@ reconciler / migration layer without the explicit allow-list comment.
|
||||
|
||||
## Related
|
||||
|
||||
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
|
||||
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
|
||||
and round 2 finds)
|
||||
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
|
||||
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
|
||||
- `scripts/check-system-of-record.sh` — the CI gate that enforces
|
||||
|
||||
@@ -3,68 +3,63 @@
|
||||
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
|
||||
only; release history lives in `CHANGELOG.md` + git.
|
||||
|
||||
`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local
|
||||
brain content, just an OAuth client pointing at a remote `gbrain serve --http`.
|
||||
v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25
|
||||
silently fell through to `connectEngine()` and opened the empty local PGLite,
|
||||
returning "No results." against a populated remote brain. v0.31.1 fixes the
|
||||
silent-empty-results bug class for every operation surface.
|
||||
`gbrain init --mcp-only` sets up a thin-client install: no local brain content,
|
||||
just an OAuth client pointing at a remote `gbrain serve --http`. Every operation
|
||||
surface routes through the remote brain — a thin-client install never opens the
|
||||
empty local PGLite, so a populated remote brain can't silently return
|
||||
"No results." Local-only commands refuse with a pinpoint hint instead of
|
||||
falling through.
|
||||
|
||||
Key files:
|
||||
Key files (per-file detail lives in each file's `KEY_FILES.md` entry; this doc
|
||||
carries the routing-seam picture):
|
||||
|
||||
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no
|
||||
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (no
|
||||
parallel `src/core/thin-client/` module; routing is a ~80-line conditional
|
||||
in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine`
|
||||
so thin-client installs never open the empty PGLite. localOnly ops on
|
||||
thin-client refuse via `refuseThinClient` (with pinpoint hint table
|
||||
`THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort`
|
||||
`THIN_CLIENT_REFUSE_HINTS`, which covers the full DB-bound command surface —
|
||||
sync, embed, extract, migrate, enrich, dream, jobs, sources, pages, files,
|
||||
eval, code-*, and more). Banner via `printIdentityBannerBestEffort`
|
||||
before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`,
|
||||
non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason`
|
||||
for canned, actionable error messages. ENG-2 renderer parity: local-engine
|
||||
for canned, actionable error messages. Renderer parity: the local-engine
|
||||
path runs `JSON.parse(JSON.stringify(result))` so renderers see the same
|
||||
shape on both paths (kills Date/bigint/Buffer drift class).
|
||||
- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`.
|
||||
Hardened in v0.31.1 (CDX-4): all transport errors normalized to
|
||||
`RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions
|
||||
{timeoutMs, signal}`; `buildAbortController` composes external signal with
|
||||
timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind`
|
||||
('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code`
|
||||
field carrying server-supplied error codes (e.g. `missing_scope`).
|
||||
`extractToolErrorCode` parses JSON envelopes first, falls back to substring
|
||||
detection for legacy server messages. `unpackToolResult<T>(res)` unchanged
|
||||
(parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape.
|
||||
- `src/core/cli-options.ts` — `parseGlobalFlags` adds `--timeout=Ns` (accepts
|
||||
`30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s
|
||||
for most ops, 180s for `think`). `parseTimeout(s)` exported helper.
|
||||
- `src/core/doctor-remote.ts` — `gbrain remote doctor` adds the
|
||||
`oauth_client_scopes_probe` check (CDX-5). Probes the read tier via
|
||||
`get_brain_identity` and admin tier via `get_health`; reports per-tier
|
||||
shape on both paths (kills the Date/bigint/Buffer drift class).
|
||||
- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`,
|
||||
the transport under the routing seam. All transport errors normalize to
|
||||
`RemoteMcpError` via the `toRemoteMcpError` funnel, with a stable
|
||||
`RemoteMcpErrorReason` union the dispatcher's `never` switch keys off.
|
||||
Full symbol-level detail: the `src/core/mcp-client.ts` entry in
|
||||
[`KEY_FILES.md`](./KEY_FILES.md).
|
||||
- `src/core/cli-options.ts` — `parseGlobalFlags` supports `--timeout=Ns`
|
||||
(accepts `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command
|
||||
default (30s for most ops, 180s for `think`). `parseTimeout(s)` exported
|
||||
helper.
|
||||
- `src/core/doctor-remote.ts` — `gbrain remote doctor` includes the
|
||||
`oauth_client_scopes_probe` check. Probes the read tier via
|
||||
`get_brain_identity` and the admin tier via `get_health`; reports per-tier
|
||||
status with pinpoint remediation when admin is missing. `buildScopeCheck`
|
||||
+ `ScopeProbeResult` exported for test access. Skippable via
|
||||
`GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC
|
||||
initialize level only (MCP SDK Client hangs on shape mismatch).
|
||||
- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
|
||||
- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries.
|
||||
- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller).
|
||||
- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality` → `'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error.
|
||||
- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).
|
||||
- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex).
|
||||
- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed.
|
||||
- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on).
|
||||
- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command).
|
||||
- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows.
|
||||
- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)` — `date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query.
|
||||
- `src/core/operations.ts` — `get_brain_identity` op (read scope, no params,
|
||||
banner-only): cheap counter packet `{version, engine, page_count,
|
||||
chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses
|
||||
`engine.getStats()`; banner's 60s client-side TTL bounds frequency to
|
||||
≤1/60s per CLI process (well below the Fly.io health-check cadence that
|
||||
motivated the original `getStats` cost warning).
|
||||
`engine.getStats()`; the banner's 60s client-side TTL bounds frequency to
|
||||
≤1/60s per CLI process.
|
||||
- `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command
|
||||
thin-client routing branches. These commands bypass the operation-layer
|
||||
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
|
||||
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
|
||||
to op params. `think` is a special case: the server's `think` op
|
||||
intentionally disables `--save`/`--take` for remote callers
|
||||
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
|
||||
loudly when those flags are set.
|
||||
to op params. `think` is a special case: the server's `think` op is
|
||||
read-scoped for OAuth/MCP and intentionally disables `--save`/`--take` for
|
||||
remote callers (the `safeSave`/`safeTake` trust-boundary gate in the `think`
|
||||
handler in `operations.ts`); thin-client `think` warns loudly when those
|
||||
flags are set.
|
||||
|
||||
Cross-modal search files (image query, SSRF-guarded image loading, spend
|
||||
tracking, multimodal reindex) are indexed per-file in
|
||||
[`KEY_FILES.md`](./KEY_FILES.md) and described behaviorally in
|
||||
[`RETRIEVAL.md`](./RETRIEVAL.md) — they are not part of the thin-client
|
||||
routing seam.
|
||||
|
||||
@@ -108,11 +108,13 @@ instead of a local DB connection:
|
||||
}
|
||||
```
|
||||
|
||||
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
|
||||
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
|
||||
`integrity`, `serve`) on a thin-client install with a clear error pointing
|
||||
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
|
||||
(OAuth discovery, token round-trip, MCP smoke).
|
||||
The CLI dispatch guard refuses every DB-bound command (`sync`, `embed`,
|
||||
`extract`, `migrate`, `serve`, `enrich`, `jobs`, `sources`, `pages`,
|
||||
`files`, `eval`, and the rest of the local-only surface — the full hint
|
||||
table is `THIN_CLIENT_REFUSE_HINTS` in `src/cli.ts`) on a thin-client
|
||||
install with a clear error pointing at the remote host. `gbrain doctor`
|
||||
runs a dedicated thin-client check set (OAuth discovery, token round-trip,
|
||||
MCP smoke). See [`thin-client.md`](./thin-client.md) for the routing seam.
|
||||
|
||||
### Setup
|
||||
|
||||
@@ -394,6 +396,9 @@ simultaneously — that's by design.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/guides/bootstrap.md` — `gbrain bootstrap`, the paved-road paste-in
|
||||
install for Topology 1 with a desktop coding agent (interview, hooks,
|
||||
MCP registration, verify).
|
||||
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
|
||||
vs sources axes).
|
||||
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Type Taxonomy (v0.41.22: gbrain-base-v2)
|
||||
# Type Taxonomy (gbrain-base-v2)
|
||||
|
||||
> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor
|
||||
> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs
|
||||
> The 14-canonical-type DRY/MECE taxonomy. Predecessor
|
||||
> `gbrain-base` (24 types) stays bundled for back-compat; fresh installs
|
||||
> default to `gbrain-base-v2`.
|
||||
|
||||
## Why
|
||||
@@ -76,9 +76,10 @@ gbrain onboard --check --explain # per-cluster narrative dry-run
|
||||
↓
|
||||
gbrain jobs submit unify-types \ # PROTECTED + manual_only
|
||||
--allow-protected \
|
||||
--params '{"target_pack":"gbrain-base-v2"}'
|
||||
--params '{"target_pack":"gbrain-base-v2","apply":true}'
|
||||
# omit "apply":true → dry-run (default)
|
||||
↓
|
||||
Handler runs 4 phases:
|
||||
Handler runs 8 phases:
|
||||
┌─────────────────────────────────────┐
|
||||
│ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL)
|
||||
├─────────────────────────────────────┤
|
||||
@@ -108,8 +109,8 @@ Every primitive ships with a documented rollback:
|
||||
| Operation | Rollback |
|
||||
|-----------|----------|
|
||||
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
|
||||
|
||||
## What if my brain doesn't fit?
|
||||
@@ -127,7 +128,8 @@ For brains with substantial custom types that deserve their own canonical
|
||||
2. Edit your fork to add page_types + mapping_rules covering your
|
||||
custom domain.
|
||||
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
|
||||
--params '{"target_pack":"my-pack"}'`
|
||||
--params '{"target_pack":"my-pack","apply":true}'` (omit `"apply":true`
|
||||
for a dry-run preview — that is the default)
|
||||
|
||||
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
|
||||
version: "1.x"}` to register itself as a successor — future agents
|
||||
@@ -163,8 +165,10 @@ explicitly disambiguated this as canonical, so it should outrank fuzzy
|
||||
matches that hit aliases by accident."
|
||||
|
||||
`SearchResult.alias_resolved_boost` is stamped on touched results for
|
||||
`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to
|
||||
invalidate pre-v0.42 cache rows that don't reflect the new stage.
|
||||
`--explain` formatter visibility. The stage participates in the search
|
||||
cache key (`KNOBS_HASH_VERSION` in `src/core/search/mode.ts` is the
|
||||
single source of truth for the current cache-key version), so cache rows
|
||||
written before the stage existed are unreachable.
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -174,4 +178,3 @@ invalidate pre-v0.42 cache rows that don't reflect the new stage.
|
||||
- Migration handler: `src/core/schema-pack/unify-types-handler.ts`
|
||||
- Onboard checks: `src/core/onboard/checks.ts`
|
||||
- Skill: `skills/schema-unify/SKILL.md`
|
||||
- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md`
|
||||
|
||||
+21
-19
@@ -1,4 +1,4 @@
|
||||
# gbrain eval suspected-contradictions (v0.32.6)
|
||||
# gbrain eval suspected-contradictions
|
||||
|
||||
The contradiction probe samples retrieval results, asks an LLM judge whether
|
||||
any pair contradicts on a factual claim relevant to the user's query, and
|
||||
@@ -142,25 +142,27 @@ pay near-zero on re-runs (until you bump PROMPT_VERSION).
|
||||
gate makes accidental private-data commits hard, but the operator MUST
|
||||
inspect every redaction before commit.
|
||||
|
||||
## Temporal axis
|
||||
|
||||
The judge distinguishes real contradictions from legitimate change-over-time.
|
||||
The verdict enum has six members (`no_contradiction | contradiction |
|
||||
temporal_supersession | temporal_regression | temporal_evolution |
|
||||
negation_artifact`), and `pages.effective_date` is threaded into the judge
|
||||
prompt so the probe doesn't cry wolf on facts that simply changed.
|
||||
|
||||
The trajectory substrate builds on the same signal:
|
||||
`gbrain eval trajectory <entity>` shows the chronological typed-claim
|
||||
history with regressions flagged inline; `gbrain founder scorecard
|
||||
<entity>` rolls up four signals (accuracy, consistency, growth
|
||||
direction, red flags) into a stable JSON contract. MCP op
|
||||
`find_trajectory` (read scope, visibility-filtered for remote callers)
|
||||
exposes the same data to agents. The probe's `temporal_supersession`
|
||||
verdict and the consolidate phase's `valid_until` writeback both
|
||||
preserve the `auto-supersession.ts` "NEVER auto-applies" invariant
|
||||
— the probe only emits paste-ready commands; only `consolidate`
|
||||
writes `valid_until` (a grep guard pins this).
|
||||
|
||||
## See also
|
||||
|
||||
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
|
||||
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
|
||||
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
|
||||
+ trend-tracking workflow.
|
||||
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
|
||||
six-member verdict enum (`no_contradiction | contradiction |
|
||||
temporal_supersession | temporal_regression | temporal_evolution |
|
||||
negation_artifact`) and threaded `pages.effective_date` into the judge
|
||||
prompt so the probe stops crying wolf on legitimate change-over-time.
|
||||
v0.35.7 lands the trajectory substrate the probe pointed at:
|
||||
`gbrain eval trajectory <entity>` shows the chronological typed-claim
|
||||
history with regressions flagged inline; `gbrain founder scorecard
|
||||
<entity>` rolls up four signals (accuracy, consistency, growth
|
||||
direction, red flags) into a stable JSON contract. MCP op
|
||||
`find_trajectory` (read scope, visibility-filtered for remote callers)
|
||||
exposes the same data to agents. The probe's `temporal_supersession`
|
||||
verdict and the consolidate phase's `valid_until` writeback both
|
||||
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
|
||||
— the probe still emits paste-ready commands, only `consolidate`
|
||||
writes `valid_until` (R1+R8 grep guard pins this).
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# Agent Bootstrap — Product Design (normative for scope & sequencing)
|
||||
|
||||
**Status:** APPROVED (product review, 3 adversarial rounds; CEO review; eng review —
|
||||
0 unresolved decisions). Implementation source of truth:
|
||||
[AGENT_BOOTSTRAP_PLAN.md](AGENT_BOOTSTRAP_PLAN.md).
|
||||
|
||||
## Problem
|
||||
|
||||
Agent platforms (OpenClaw, Hermes) deliver the full personal-agent experience —
|
||||
scheduled work, skill files, SOUL.md identity, persistent memory — but setting one up
|
||||
means deploying a server, wiring API keys, and paying raw token prices. Meanwhile
|
||||
nearly everyone already runs Claude Code or Codex, both subsidized by subscriptions,
|
||||
both capable of executing an install protocol themselves.
|
||||
|
||||
**The feature:** a paste-in install that adds memory + skills + identity +
|
||||
session-triggered schedules + knowledge persistence to a private GitHub repo,
|
||||
mimicking the agent-platform experience on the desktop apps people already have.
|
||||
"Just use your local harness as your agent."
|
||||
|
||||
## Target surfaces & order
|
||||
|
||||
| Surface | In v1? | Order | Per-turn context seam | Persistence (write) seam |
|
||||
|---|---|---|---|---|
|
||||
| Codex in ChatGPT desktop | YES | 1st | spike exit question #2 | spike exit question #1 (blocking); fallback: mandated MCP writes + end-of-session sweep |
|
||||
| Codex CLI | YES | with 1st | AGENTS.md pull protocol + `volunteer_context` | mandated MCP write ops |
|
||||
| Claude Code desktop | YES | 2nd | hooks: SessionStart / UserPromptSubmit via serve IPC | SessionEnd hook: transcript ingest + scan-gated push |
|
||||
| Claude Code CLI | YES | with 2nd | same hooks | same hooks |
|
||||
|
||||
The wedge is NOT platform feature-parity on day one. It is: paste → interview → an
|
||||
agent that knows who it is and who you are → recalls what you told it in the next
|
||||
session → visibly compounds during week one.
|
||||
|
||||
## The repo format (the contract)
|
||||
|
||||
A private GitHub repo is the product artifact — the agent's portable body:
|
||||
|
||||
- **Manifest:** `agent.json` (`format_version: 1`, provisional; `initialized`
|
||||
sentinel distinguishes a template clone from a bootstrapped workspace).
|
||||
- **Identity:** SOUL.md, USER.md, MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md,
|
||||
ACCESS_POLICY.md, GITHUB.md — rendered ONLY from interview answers, never invented.
|
||||
- **Content:** `brain/` (the gbrain source), `memory/` (daily notes), `skills/`,
|
||||
`state/` (committed: interview.json, portable mcp.json; local-only files gitignored).
|
||||
- **Compatibility promise:** hosted gbrain mounts `format_version: 1` repos natively;
|
||||
compatibility is a test against this spec.
|
||||
|
||||
## Premises (all settled)
|
||||
|
||||
1. **Free desktop tier of the hosted ladder** — the desktop ceiling (laptop asleep =
|
||||
agent asleep; data outgrows the disk) is the graduation mechanic to hosted gbrain.
|
||||
2. **One agent-body format, two doors.** Portability up the ladder is moat #1.
|
||||
3. **Awake-when-you-are is the honest desktop contract.** Session-triggered schedules
|
||||
(jobs run at turn/session boundaries while the harness is open); true 24/7 crons
|
||||
are hosted-tier, stated in-product.
|
||||
4. **Day-one-empty-brain is the #1 churn risk.** Magic moment with zero corpus
|
||||
(interview → next-session recall) + fast ramp (file import; connector ingest v1.1).
|
||||
5. **Wire-level truth before build:** a clean-machine spike gates door 1 (write-seam
|
||||
pass bar: 0 durable-write failures in 20 sessions over ≥3 days, else extend to 50).
|
||||
6. **The graph is moat #2:** v1 exercises entity extraction, backlinks, and
|
||||
graph-aware recall; verify enforces a graph floor.
|
||||
7. **Keyless mode is first-class:** the harness agent IS the subsidized LLM — with
|
||||
zero API keys, memory is agent-authored through write ops, search is keyword-only,
|
||||
and the magic moment still passes. One optional key unlocks embeddings +
|
||||
auto-extraction.
|
||||
|
||||
## Build order (one cathedral PR; size trip-wire: split at build order 2 if PR is
|
||||
open >10 days from first code commit)
|
||||
|
||||
0. **Spike + quota gate** (manual, gates door-1 ship; per-harness quota measured; a
|
||||
p90 day must fit ≤10% of weekly subscription quota or schedule scope is cut).
|
||||
1. **Shared body + engine machinery:** `gbrain bootstrap` family, templates, format
|
||||
spec, secret-scan-gated persistence, verify, uninstall (v1 via CEO-review
|
||||
expansion; receipt-keyed scope per the PLAN's CX2-12).
|
||||
2. **Codex door ships first** (runbook variant + approvals preflight + capability
|
||||
probe; CLI path not spike-gated).
|
||||
3. **Claude Code door:** hooks, IPC turn_context, transcript ingestion, greeting
|
||||
digest, schedule mechanism.
|
||||
4. **Graduation seam, desktop half:** format spec + documented upgrade path (advisor
|
||||
nudge ships with the hosted mount in v1.1).
|
||||
|
||||
## Out of scope for v1
|
||||
|
||||
| Deferred | Lands | Why |
|
||||
|---|---|---|
|
||||
| Connector-driven ingest (email/calendar) | v1.1 (keyed to probe) | unverified host capability |
|
||||
| Hosted mount + "outgrowing this laptop" nudge | v1.1 together | never point at a destination that can't accept the repo |
|
||||
| `serve --attach` (simultaneous multi-harness) | fast-follow | v1 documents one-live-serve politely |
|
||||
| Codex `notify` transcript sweeper | fast-follow | mandated MCP writes cover v1 |
|
||||
| Windows (named-pipe IPC) | deferred | v1 = macOS + Linux |
|
||||
| Networked Docker paste-flow e2e | fast-follow | offline container e2e covers 80% at 20% of the flake |
|
||||
| `gbrain quota` meter command | TODOS | measurement ships as script+doc; productize when per-harness token counting is proven |
|
||||
| True 24/7 crons on desktop | never | hosted-tier by design |
|
||||
|
||||
## Success criteria
|
||||
|
||||
- **TTFM ≤15 min** paste→verified install, excluding first-run toolchain downloads
|
||||
(published separately); every human action counted.
|
||||
- **Magic moment, deterministically:** verify asserts an interview fact is retrievable
|
||||
through the agent's own MCP path; fresh-session end-to-end is a scripted human
|
||||
confirmation. Must pass keyless.
|
||||
- **Pilot tests:** a non-developer pilot completes the door-1 install unaided and
|
||||
still uses it in week two; a developer completes the door-2 README install ≤10 min
|
||||
with a week-two recall check.
|
||||
- **Graph floor:** ≥1 entity extracted, ≥1 backlink resolved, one edge-only query
|
||||
answered — via the real MCP write path.
|
||||
- **Ladder proof:** a desktop-grown repo validates against `format_version: 1`.
|
||||
- **Honesty checks:** quota number published; desktop contract stated in-product;
|
||||
door 1 demotes to documented-beta on its trigger rather than shipping flaky.
|
||||
|
||||
## Distribution
|
||||
|
||||
Paste block + tag-pinned runbook (`BOOTSTRAP_FOR_AGENTS.md`, fetched at the
|
||||
`latest-stable` ref — advanced by the release job only after assets publish, so
|
||||
published copies never rot); optional GitHub template repo (generated at release from
|
||||
the same renderer); binary via `bun install -g github:garrytan/gbrain#latest-stable`
|
||||
(never npm). The paste block lives in the README's `## Install` section, as
|
||||
per-harness subsections ordered "For Codex — the recommended first step" → "For
|
||||
Claude Code" → "For OpenClaw or Hermes" (the 2026-08-09 ordering decision, recorded
|
||||
in the PLAN's artifact table). `INSTALL_FOR_AGENTS.md` remains the paste path for
|
||||
agent platforms and lives inside the OpenClaw/Hermes subsection.
|
||||
|
||||
## Threat model (v1 summary)
|
||||
|
||||
Tag-pinned fetch + version-stamp skew check + runbook phase allowlist; secret-scan
|
||||
gates every commit AND corpus write (loud block, per-finding override); repo privacy
|
||||
verified via API after create; hooks in gitignored local settings with a kill switch;
|
||||
interview answers rendered as fenced data (never instructions) with escaping and
|
||||
caps; uninstall keyed to a machine-local receipt, never deletes a brain it didn't
|
||||
create; provider-policy drift acknowledged as residual risk — posture: measured
|
||||
sustainable load, no absent-user background burn, portable body as the exit plan.
|
||||
Full posture: `docs/guides/bootstrap.md`.
|
||||
@@ -0,0 +1,817 @@
|
||||
# Agent Bootstrap — Implementation Plan (normative)
|
||||
|
||||
The engineering source of truth for `gbrain bootstrap` — the paste-in install that
|
||||
turns Claude Code / Codex desktop apps into a persistent personal agent (identity +
|
||||
memory + skills + session-triggered schedules + private-repo persistence + local
|
||||
PGLite brain). Product scope/sequencing source of truth:
|
||||
[AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md). Where they disagree, the
|
||||
design doc wins on scope; this doc wins on implementation.
|
||||
|
||||
Reviews absorbed: 3-designer panel + adversarial critique; product design review (3
|
||||
rounds); CEO review (SELECTIVE EXPANSION, ~55 findings); outside voice x2 (37
|
||||
findings, 35 accepted); eng review (32 findings). All accepted fixes are inlined
|
||||
below with their finding IDs. 0 unresolved decisions.
|
||||
|
||||
---
|
||||
|
||||
## As-shipped deltas (read this first — where the code moved after the plan froze)
|
||||
|
||||
This plan is layered: later absorption sections (the post-design-review deltas, the
|
||||
CX2 series) override earlier prose, and THIS section overrides everything below it.
|
||||
The shipped implementation matches the plan except for these deltas:
|
||||
|
||||
1. **Verify runs LAST, not before host registration.** [CX2-5]'s determinism goal
|
||||
survived, but the shipped phase order (single TS source:
|
||||
`src/core/bootstrap/status.ts` `PHASES`) is
|
||||
preflight → engine → interview → render → skills → wire → repo → **verify**,
|
||||
and verify runs in-process on the caller-held engine, calling
|
||||
`runMaintenanceSweep` directly — no transient serve. It works pre-registration
|
||||
AND as the weekly re-run (`src/core/bootstrap/verify.ts`).
|
||||
2. **Uninstall scope: [CX2-12] wins over the CEO-expansion bullet.** `~/.gbrain` is
|
||||
NEVER deleted wholesale — only receipt-enumerated bootstrap-created state
|
||||
(`src/core/bootstrap/uninstall.ts`).
|
||||
3. **Module naming/layout:** `private-repo.ts` shipped as `repo.ts`; additional
|
||||
shipped modules the artifact table doesn't list: `attach.ts, assets.ts,
|
||||
format.ts, host-specs.ts, hooks.ts, lock.ts, status.ts, template-repo.ts,
|
||||
uninstall.ts`.
|
||||
4. **Templates layout:** all bootstrap templates live under `templates/bootstrap/`
|
||||
(not at `templates/` root).
|
||||
5. **Test filenames:** `test/hook-command.serial.test.ts` and
|
||||
`test/e2e/bootstrap-*.serial.test.ts` — the `.serial` variants the plan's own
|
||||
[A7] mandated; the artifact table predates that.
|
||||
6. **README ordering:** the D5 placement was superseded by the 2026-08-09 user
|
||||
decision — per-harness `## Install` sections ordered Codex → Claude Code →
|
||||
OpenClaw/Hermes, with `INSTALL_FOR_AGENTS.md` living inside the OpenClaw/Hermes
|
||||
section (annotated in the artifact table; the D5 prose at the bottom is stale).
|
||||
|
||||
## Post-design-review deltas (2026-08-07, /office-hours APPROVED — these override below)
|
||||
|
||||
Product: **"GBrain for Codex" + "GBrain for Claude Code"** (names contingent on
|
||||
trademark review) — the free desktop tier of the hosted-gbrain ladder (the hosted service is the
|
||||
graduation path; the desktop ceiling is the mechanic, not a bug). Target surfaces are
|
||||
the DESKTOP APPS; **Codex door ships first** (named pilot user is a non-developer
|
||||
ChatGPT-app user). CLIs come along via shared machinery.
|
||||
|
||||
**Build order (replaces the PR1/FF phasing below where they conflict):**
|
||||
0. **Spike (gates door-1 ship; starts immediately, before build):** clean-machine
|
||||
reproduction of Garry's working Codex-in-ChatGPT + gbrain prototype. Exit questions
|
||||
with pass bars: (#1 blocking) reliable per-turn/per-session WRITE trigger — 0
|
||||
durable-write failures in 20 sessions across ≥3 days on a non-founder machine, else
|
||||
extend to 50; causes logged. (#2) per-turn READ seam — context block present at turn
|
||||
start or degraded pull-mode documented. Also: folder access, MCP registration path,
|
||||
approval-tap count, connector availability (yes/no/degraded). PLUS in parallel: the
|
||||
**quota release gate** measured against Garry's production usage — pass: p90 day
|
||||
≤10% of weekly Max/Pro quota; failure = scope change to "session-agent, no schedules
|
||||
by default" BEFORE schedule build effort. Codex CLI is NOT spike-gated.
|
||||
1. **Shared body + engine machinery (PR1):** everything in this plan's PR1 (bootstrap
|
||||
family, gbrain hook + IPC turn_context, templates, question bank, secret scan,
|
||||
sources push, skillpack scaffold wiring, doctor checks) PLUS: `agent.json` repo
|
||||
manifest with `format_version: 1` (the ladder contract — the hosted gbrain service mounts this
|
||||
format; compatibility is a test, not a hope); wire EXISTING import + entity
|
||||
extraction + backlinks + graph-aware query into install/verify (funds design
|
||||
Premises 4/6 — new verify check: ≥1 entity extracted, ≥1 backlink resolved, one
|
||||
edge-only query answered); byte floors scaled to answered-question count (they catch
|
||||
skipped interviews, NOT invention — never pressure padding); commit this plan to
|
||||
docs/designs/AGENT_BOOTSTRAP_PLAN.md.
|
||||
2. **Codex door (ships first):** runbook variant for Codex desktop approvals/sandbox
|
||||
(seam per spike; fallback = mandated MCP writes via AGENTS.md gates + end-of-session
|
||||
sweep, shipped only with measured reliability) + Codex CLI path (AGENTS.md pull
|
||||
protocol; session-start gate includes "run due jobs"). Connector-ingest capability
|
||||
probe wired into install output; connector ingest itself is v1.1.
|
||||
3. **Claude Code door:** hooks per this plan (SessionStart/UserPromptSubmit/SessionEnd
|
||||
via serve IPC), MCP project-scope default, transcript ingestion. Includes the v1
|
||||
schedule mechanism: hook layer checks HEARTBEAT.md due-job list at session start +
|
||||
turn boundaries. If the spike finds no turn-boundary trigger on ChatGPT desktop,
|
||||
schedules are Claude-Code-only in v1 and per-surface copy says so.
|
||||
4. **Graduation seam, desktop half only:** format spec + documented upgrade path +
|
||||
honest-contract copy. The "outgrowing this laptop" advisor nudge moves to v1.1
|
||||
WITH the hosted mount (never point at a destination that can't accept the repo).
|
||||
|
||||
**New v1 requirements from the design review (fold into implementation):**
|
||||
- Binary install is tag-pinned: `bun install -g github:garrytan/gbrain#<release-tag>`,
|
||||
same stamp as the runbook, so `bootstrap status` skew check is meaningful.
|
||||
- Non-terminal-buyer error channel: every blocking condition (scan block, lock
|
||||
collision, partial install, verify fail) surfaces through the agent's reply channel
|
||||
(agent-readable status/verify/doctor output the runbook + AGENTS.md instruct the
|
||||
agent to relay; hook failures write a status file surfaced at session start).
|
||||
- Both doors, one machine: shared brain, one live serve; installer detects + reuses
|
||||
existing registration; simultaneous-session collision fails politely, doctor names
|
||||
it, agent relays it. serve --attach lifts the limit later.
|
||||
- state/ boundary: interview.json + mcp.json COMMITTED (machine-2 re-render must not
|
||||
hard-fail); *.local/caches/heartbeat files gitignored; DB never committed.
|
||||
- OS scope v1: macOS + Linux (unix-socket IPC). Windows deferred (named-pipe = spike
|
||||
question if prioritized).
|
||||
- Success criteria additions: TTFM ≤15 min EXCLUDING first-run toolchain downloads
|
||||
(published separately); door-2 developer test (README paste → ≤10 min install,
|
||||
week-two recall check); graph floor verify check; magic moment = deterministic
|
||||
retrieval assertion + scripted human confirmation (never fake-automated).
|
||||
- Provider-policy drift acknowledged as unmitigable external risk; posture = quota
|
||||
gate + no absent-user background burn + portable body as exit plan.
|
||||
- README: bootstrap paste block is the "full agent" option INSIDE the existing "Quick
|
||||
start: Claude Code or Codex" section; the 2-line memory-only quick start STAYS,
|
||||
labeled "just memory, no agent"; INSTALL_FOR_AGENTS.md remains the platform headline.
|
||||
- Consolidated v1 non-goals (from design doc, amended by CEO review): connector ingest
|
||||
(v1.1), hosted mount + advisor nudge (v1.1), serve --attach, notify sweeper, Windows,
|
||||
networked-paste Docker e2e (fast-follows); 24/7 crons never on desktop.
|
||||
|
||||
**CEO-review accepted expansions (2026-08-07, SELECTIVE EXPANSION — now v1 scope):**
|
||||
- **GitHub template repo** ("Use this template") as a second distribution artifact for
|
||||
the ChatGPT door (resolves design OQ2 = yes). Same rendered file set, published;
|
||||
kept in sync with templates/ by extending scripts/check-bootstrap-templates.sh to
|
||||
diff the template repo content. Build order 2.
|
||||
- **`gbrain bootstrap uninstall`** in v1 (was fast-follow): removes MCP registration +
|
||||
hooks + bootstrap-created state (confirm-gated), leaves the repo ("the body remains
|
||||
yours"). [Scope superseded by CX2-12 + as-shipped delta 2: `~/.gbrain` is never
|
||||
deleted wholesale — only receipt-enumerated bootstrap-created state.]
|
||||
- **Docker cold-machine e2e (offline parts) in CI** in v1: networkless read-only
|
||||
container running interview → render → verify with fake gh (codex-as-agent
|
||||
tests/docker shape). The full networked paste flow stays a fast-follow (flake).
|
||||
- **Hot-memory greeting digest in SessionStart**: top facts via IPC turn_context when
|
||||
serve is up; file-plane MEMORY.md digest fallback when not (session start often
|
||||
precedes serve spawn); fail-open. Claude door in v1; door 1 iff spike finds a read
|
||||
seam.
|
||||
- **First-run tour**: `bootstrap verify` success output ends with three scripted magic
|
||||
prompts (who am I to you / remember X then restart / what do you know about
|
||||
<project>) — makes week-one compounding visible (design Premise 4).
|
||||
- **Deferred to TODOS.md in the PR** (must land as TODOS entries with the cathedral
|
||||
PR): `gbrain quota` meter command (release-gate measurement ships as script+doc
|
||||
first; productize once the per-harness token-count method is proven); networked
|
||||
Docker paste-flow e2e.
|
||||
|
||||
## Deep-review hardening (2026-08-07 CEO review sections 1–9 — ALL ACCEPTED per
|
||||
## Garry's standing directive; IDs trace to the review record. These are v1 scope.)
|
||||
|
||||
**Architecture (S1):**
|
||||
- [D4] The release-time template-repo generator IS `core/bootstrap/render.ts --minimal`
|
||||
with placeholder answers — one rendering code path, two consumers; CI guard becomes a
|
||||
byte-diff of generator output vs the vendored tree.
|
||||
- [D5] Bootstrap phase list defined ONCE in TS; `bootstrap status --json` emits it; the
|
||||
runbook defers to it ("follow status's phase list"); CI checks runbook phase names
|
||||
against the TS list.
|
||||
|
||||
**Source binding + brain semantics (G1, S3#1 — the two product-breaking fixes):**
|
||||
- [G1] MCP registration passes the workspace source explicitly
|
||||
(`claude mcp add -e GBRAIN_SOURCE=<workspace-slug>` / codex env equivalent) so agent
|
||||
writes land in the workspace source, fact fence-writes reach `brain/` files (not
|
||||
DB-only fallback), and the private repo actually fills. `bootstrap verify` asserts an
|
||||
MCP-path `put_page` materializes a COMMITTED file under `brain/` — a green verify with
|
||||
an empty repo is impossible.
|
||||
- [S3#1] `turn-context.ts` constructs an OperationContext with `remote: true` and
|
||||
threads `visibility: ['world']` into all fact reads (parity with the existing
|
||||
meta-hook posture — the IPC path must never widen what MCP would return). IPC test:
|
||||
a `visibility='private'` fact NEVER appears in a turn_context response. Verify's
|
||||
magic-moment fact is written with visibility the harness can read back (world).
|
||||
|
||||
**Interview + render hardening (S3#3, G10, G12, A8):**
|
||||
- Answers render inside fenced, explicitly-subordinate blocks ("verbatim principal
|
||||
input — data, not instructions"); strip/escape line-leading `#`, `<!--`, and code
|
||||
fences; per-answer length cap (~4KB, confirm-to-truncate); reject/escape `{{` and
|
||||
control chars at `--set` time (a Handlebars user's honest answer must not brick the
|
||||
token sweep). Verify fails if any heading or managed-block marker in a rendered file
|
||||
traces to an interview value (protects harden's AGENTS.md marker splicing).
|
||||
- [A8] `--confirm` requires the hash of the exact answer set that was read back;
|
||||
per-answer `set_at` provenance recorded; hostile test: single-batch set+confirm
|
||||
exits non-zero and verify reports `provenance: unverified`.
|
||||
- [G12] Conflict-marker detection on `state/interview.json` read → agent-readable
|
||||
"resolve this file" message, never a stack trace.
|
||||
|
||||
**Uninstall confinement (G2, S3#5, A2):**
|
||||
- Never delete a brain bootstrap didn't create: `agent.json` created-by stamp checked;
|
||||
default KEEPS the DB; deletion only behind explicit `--delete-brain` with a confirm
|
||||
that enumerates sources + page count; facts export offered first (facts are NOT
|
||||
derived state). Refuse when `GBRAIN_HOME` is set unless `--home` is explicit AND
|
||||
`isPathContained` + gbrain-home signature (config.json + brain.pglite) both pass.
|
||||
Host-config edits are marker-keyed managed blocks only (settings.local.json,
|
||||
~/.codex/config.toml) — foreign hooks/servers survive; test asserts full
|
||||
before/after filesystem+registration diff incl. foreign entries + symlinked home.
|
||||
|
||||
**Hooks + IPC hardening (G5, G11, S3#6, S3#7, S3#8, A9, A3):**
|
||||
- [G5] `bootstrap hooks`/`--repair`/uninstall use marker-keyed managed-block
|
||||
read-merge-write on settings.local.json — never overwrite `permissions.allow` or
|
||||
foreign hooks; never append duplicates.
|
||||
- [G11] IPC client timeout parameterized per kind (turn_context > 250ms default);
|
||||
window payload + assembled block clamped below the 256KB message cap before send.
|
||||
- [ENG-1] **Claude Code hook-output cap:** stdout/additionalContext is capped at
|
||||
10,000 chars by the harness (overflow is diverted to a file and NOT injected) —
|
||||
the assembled turn_context block is budgeted to ≤8KB (pointers + facts trimmed by
|
||||
confidence to fit), asserted in the hook snapshot test. Docs: code.claude.com hooks
|
||||
reference (verified 2026-08-08; the hook writers are dated spec-targets, so this cap
|
||||
lives with the settings-shape module).
|
||||
|
||||
## Eng-review hardening (2026-08-08 /plan-eng-review — ALL ACCEPTED per standing
|
||||
## directive; seams verified against source with file:line quotes)
|
||||
|
||||
- [ENG-2] (9/10) **cli.ts registration is THREE touchpoints, not one:** `CLI_ONLY` set
|
||||
(cli.ts:58), the engine-free if-chain inside `handleCliOnly` (add before the
|
||||
`connectEngine` terminator at cli.ts:1840), and `CLI_ONLY_SELF_HELP` (cli.ts:61 —
|
||||
omit it and the subcommands' `--help` is dead code, the documented init.ts:117 trap).
|
||||
`bootstrap`/`hook` must NOT enter `THIN_CLIENT_REFUSED_COMMANDS`. Membership test per
|
||||
the #2035 precedent (test/cli-bigint-normalize.test.ts:46 shape).
|
||||
- [ENG-3] (9/10) **IPC widening = handler restructure, not a field add.** Correct path
|
||||
is `src/core/context/resolve-ipc.ts` (constants at :26-28). `ResolveHandler` is a
|
||||
single-function type (:42) and the server handler dispatches unconditionally
|
||||
(:127-128) — turn_context needs a discriminated-union request + handler map
|
||||
(restructuring the closure at src/mcp/server.ts:90-108), NAMED response types
|
||||
(today's responses are inline literals), and per-kind MAX_MSG_BYTES/CLIENT_TIMEOUT.
|
||||
Absent `kind` defaults to 'resolve'; old-serve grace confirmed (malformed → client
|
||||
IPC_UNAVAILABLE fail-soft, resolve-ipc.ts:80-84).
|
||||
- [ENG-4] (8/10) **Postgres-hook silent degrade on unmigrated brains:** volunteer.ts
|
||||
(:247-249) and retrieval-reflex.ts (:180-183) swallow missing-table errors
|
||||
(pre-v110/v117), so a direct-engine hook returns empty, not an error. Fix: hook
|
||||
heartbeat records `degraded_reason: 'schema_pre_vNNN'`; doctor pairs
|
||||
hook-in-use + unmigrated-brain into a named warning.
|
||||
- [ENG-5] (8/10) **Sweep layer ownership decided:** startup sweep attaches in
|
||||
src/mcp/server.ts immediately after `server.connect` (:77) in the same best-effort
|
||||
try/catch shape as the resolve-IPC block (:85-112), with cleanup added to shutdown
|
||||
(:122); the idle sweep lives in src/commands/serve.ts reusing the `armIdle` pattern
|
||||
(:437-451) through the injectable `deps.setInterval` seam (:281,297), every timer
|
||||
`unref()`d (the :424/:444 convention) so the sweep can never hold the process open.
|
||||
- [ENG-6] (9/10) **Compiled-binary asset rule:** `bin/gbrain` ships via
|
||||
`bun build --compile`; `dirname(dirname(__dirname))` template resolution
|
||||
(init.ts:1514 pattern) breaks in the binary. Templates + questions.json + runbook
|
||||
stamps are STATICALLY IMPORTED (bundled) — same mechanism as the existing
|
||||
`skills/_brain-filing-rules.json` static import in brain-repo-durability.ts. A
|
||||
compiled-binary e2e asserts `bootstrap render` works with NO repo checkout present.
|
||||
- [ENG-7] (7/10) **Host-format module precedents named:** registration strings follow
|
||||
connect.ts `AGENT_SPECS` + argv builders (:61-67, :262-267); file-writing hook/
|
||||
settings writers follow the integrations.ts / frontmatter-install-hook.ts idiom
|
||||
(backup + marker + restore). The dated-spec-target scaffold is imported from
|
||||
codex-as-agent (greenfield here — docs/plans/ has n=1 file); it becomes
|
||||
`src/core/bootstrap/host-specs.ts` with TARGETS entries carrying id/status/
|
||||
verifiedAt/references.
|
||||
- [ENG-8] (9/10) **Facts visibility knob = ONE resolver helper.** The 'private'
|
||||
default is duplicated at backstop.ts:185, :352, operations.ts:4468, :5812 — and the
|
||||
:4468 ternary coerces any non-'world' to 'private', so a config default needs an
|
||||
explicit caller-unset check. Implement `resolveDefaultVisibility(engine)` (reads
|
||||
`facts.default_visibility` via the getConfig precedent, extract.ts:44) feeding
|
||||
ctx.visibility at ALL FOUR sites; no schema change (CHECK already permits 'world',
|
||||
migrate.ts:2319). Documented as security-relevant: it widens what remote/MCP
|
||||
callers read via meta-hook.ts:66 — the intended effect, stated as such.
|
||||
- [ENG-9] (8/10) **Secret-scan module reuses residents:** seed exclusion list from
|
||||
`.gitleaks.toml` allowlist paths (test/, skills/, .claude/skills/) so the scanner
|
||||
doesn't fire on fixtures CI already ignores; findings render through
|
||||
`redactSecretsInText` (shell-redact.ts:36) for consistent `<REDACTED:name>` output.
|
||||
- [ENG-10] (7/10) **Renderer must not eat intentional literals:** skillpack scaffolds
|
||||
carry a literal `{{output-from-skill}}` token (init-scaffold.ts:269) that must
|
||||
survive to disk — the renderer is never pointed at skillpack scaffold paths, pinned
|
||||
by a negative test.
|
||||
- [ENG-11] (7/10) **turn_context reuses the existing hot-memory cache** (30s TTL keyed
|
||||
on source+session, meta-hook.ts) instead of a fresh facts query per turn — the
|
||||
per-turn cost profile is then identical to what every MCP tool call already pays.
|
||||
- [ENG-tests] New tests from this review: CLI_ONLY membership (bootstrap, hook);
|
||||
compiled-binary render e2e (no repo checkout); Postgres-hook degraded_reason;
|
||||
resolveDefaultVisibility across all 4 call sites (unset/world/private × config);
|
||||
IPC handler-map dispatch (kind absent/resolve/turn_context/unknown); renderer
|
||||
negative test on scaffold literals; 8KB block-budget snapshot assert.
|
||||
- [ENG-tests-2] Coverage-trace gaps (Section 3): **attach-mode e2e** (clone a
|
||||
bootstrap-created fixture repo on "machine 2" → attach → hooks repair → verify);
|
||||
**sweep tests** (bounded batch per idle tick, spend-gate off ⇒ no LLM calls,
|
||||
unref/shutdown never held open, corpus file marked processed exactly once);
|
||||
**keyless-mode e2e** (zero API keys: install → agent-authored fact via ops → BM25
|
||||
recall → magic-moment passes → capability report says keyless);
|
||||
**decline-everything e2e** (no gh, no keys, hooks declined: install completes
|
||||
local-only with honest warnings, verify exits 0-with-warnings, nothing silently
|
||||
broken).
|
||||
|
||||
## Eng outside-voice absorption (Codex round 2, 2026-08-08 — 17/17 ACCEPTED,
|
||||
## all with file:line evidence; CX2 ids)
|
||||
|
||||
- [CX2-1 P0] **Template-vs-attach discriminator:** agent.json presence cannot
|
||||
discriminate a template clone from a machine-2 clone. The template ships
|
||||
`agent.json` with `initialized: false`; `bootstrap render` flips it true atomically
|
||||
AND writes a machine-local install receipt (`~/.gbrain/bootstrap/receipt.json`);
|
||||
`attach` requires `initialized: true`. [pairs with CX2-12]
|
||||
- [CX2-2 P0] **One-live-serve wording clarified:** each door spawns its OWN stdio
|
||||
serve via its MCP registration (process-bound transport, server.ts:76); "reuse
|
||||
registration" means config, never process. v1 contract restated: one live serve at
|
||||
a time per brain; sequential across doors works; simultaneous fails politely.
|
||||
- [CX2-3 P0] **Durability must be parent-repo-aware:** write-through targets
|
||||
`repo/brain` but hardening asserts `.git` in that exact dir (sources-harden.ts:83,
|
||||
brain-repo-durability.ts:621) — would fail on the workspace layout. Fix: durability
|
||||
resolves the repo root via `git rev-parse --show-toplevel` (the sync.ts:1002
|
||||
precedent); commit/push operate on the parent repo; integration test on the
|
||||
workspace fixture.
|
||||
- [CX2-4 P0] **Keyless facts ingestion made deterministic:** put_page only queues
|
||||
extraction (operations.ts:1337); facts-fence reconciliation lives in the cycle
|
||||
extractor (extract-facts.ts:337). The serve sweep INCLUDES the zero-LLM facts-fence
|
||||
reconciliation pass, so agent-authored `## Facts` fences populate the facts table
|
||||
with no API key. Keyless e2e asserts it.
|
||||
- [CX2-5 P0] **Graph-floor verify made deterministic:** verify cannot command the
|
||||
host's serve (stdio owned by the desktop app). Fix: the sweep gets a trusted
|
||||
local-only CLI entry (`gbrain sweep --once`, CLI_ONLY, never over MCP), and
|
||||
`bootstrap verify` runs BEFORE host registration on its own transient serve/engine:
|
||||
write via op → `sweep --once` → edge query. No timing nondeterminism.
|
||||
[Sequencing superseded by as-shipped delta 1: verify shipped as the LAST phase,
|
||||
in-process on the caller-held engine; the determinism goal is unchanged.]
|
||||
- [CX2-6 P1] **Cross-platform lock replaces flock dependence:** flock(1) absent ⇒
|
||||
locking silently disabled (brain-repo-durability.ts:137) — macOS is the v1 target.
|
||||
One cross-platform lock (atomic mkdir/lockfile with PID+age+token semantics) spans
|
||||
scan → stage → commit → pull → push, coordinated with the post-commit hook.
|
||||
[As-shipped delta: a TOCTOU fix reordered the scanned phase to stage FIRST, then
|
||||
secret-scan the STAGED index blobs (`git cat-file`), so scanned bytes == committed
|
||||
bytes; unscannable staged blobs fail closed (`blocked_unscannable`). Lock span
|
||||
otherwise unchanged.]
|
||||
- [CX2-7 P1] **Push ordering pinned:** commit FIRST, then divergenceSafePull, then
|
||||
push (the existing durability ordering, brain-repo-durability.ts:200 —
|
||||
divergenceSafePull returns skipped_dirty on a dirty tree, git-remote.ts:489); test:
|
||||
dirty local + advanced remote.
|
||||
- [CX2-8 P1] **GBRAIN_HOME dual semantics normalized:** config appends `.gbrain`
|
||||
(config.ts:1210) while durability uses the value directly
|
||||
(brain-repo-durability.ts:95) — the S3#10 `ensureGbrainHome()` choke point is also
|
||||
the single semantic resolver; the --isolated e2e asserts the credential store's
|
||||
ACTUAL location is gitignored, not just the expected path.
|
||||
- [CX2-9 P1] **Provider-key resolution re-specified honestly:** env legitimately
|
||||
overrides file config (config.ts:568) — the CX-P1.4 claim "never from shell env" is
|
||||
wrong. Real contract: interview-provided keys go to the 0600 config file so
|
||||
GUI-spawned serves (which lack shell env) find them; normal env>file precedence
|
||||
stands; test = GUI-launch simulation with empty env + file key.
|
||||
- [CX2-10 P1] **IPC authorization, not just authentication:** turn_context binds
|
||||
server-side to the registered GBRAIN_SOURCE and rejects caller-supplied cross-source
|
||||
requests (existing handler accepts caller sourceId, server.ts:89); cross-source
|
||||
rejection test.
|
||||
- [CX2-11 P1] **Hot-memory cache session key made real:** meta-hook reads an ad-hoc
|
||||
`source_session` that dispatch never sets (meta-hook.ts:49, dispatch.ts:208) — all
|
||||
callers collapse to the null-session cache key today. Session identity becomes
|
||||
typed OperationContext state set from MCP `_meta.session_id`; two-session isolation
|
||||
test.
|
||||
- [CX2-12 P1] **Uninstall ownership = machine-local receipt** (never the repo-carried
|
||||
agent.json, which template/attach clones inherit); uninstall stops/refuses a live
|
||||
serve before touching state; `~/.gbrain` global config/sources/clones are NEVER
|
||||
deleted wholesale — only receipt-enumerated bootstrap-created state.
|
||||
- [CX2-13 P1] **Committed-state hygiene:** the optional interview API key bypasses
|
||||
answers/hashes/provenance/logs entirely → written only to the 0600 config sink
|
||||
(config.ts:1138); committed `state/mcp.json` is the PORTABLE snippet (no absolute
|
||||
paths, no machine GBRAIN_HOME) — machine-specific wiring lives in local state and
|
||||
is regenerated by attach/`hooks --repair`.
|
||||
- [CX2-14 P1] **Generator determinism:** template-repo renders use canonical
|
||||
placeholder provenance (frozen timestamps), exclude runtime state; guard test =
|
||||
two independent renders are byte-identical.
|
||||
- [CX2-15 P1] **Two scan policies, not one:** the `.gitleaks.toml` allowlist is a CI
|
||||
fixture policy for a PUBLIC repo — importing it into the personal-repo runtime
|
||||
scanner creates blind spots (skills/ trees). Runtime scanner ships its own minimal
|
||||
allowlist + per-finding override; ENG-9 amended.
|
||||
- [CX2-16 P1] **Bootstrap lock done properly:** atomic acquisition + PID liveness +
|
||||
age guard + ownership token (the pid-reuse learning applied); kill→immediate-rerun
|
||||
recovery test; export `LiveServeLockError` (currently unexported, pglite-lock.ts:28)
|
||||
so the planned class assertion can import it.
|
||||
- [CX2-17 P2] **Format-aware host-config writers:** JSON has no comment-marker
|
||||
boundary — settings.local.json gets a structural JSON merger (gbrain entries keyed
|
||||
by a `_gbrain` marker property, semantic dup detection surviving reordering);
|
||||
config.toml gets a TOML-aware writer; both atomic write+backup; the
|
||||
frontmatter-install-hook replace/backup idiom applies only to whole-file targets.
|
||||
G5 amended.
|
||||
- [S3#6] Socket binds in a 0700 dir with mode set BEFORE exposure; turn_context
|
||||
requires a shared secret from a 0600 file in the data dir; heartbeat counts
|
||||
turn_context serves so doctor can flag unexplained callers.
|
||||
- [S3#7] Heartbeat JSONL schema pinned to counters + durations + error codes — NO
|
||||
prompt/fact/slug text; dir 0700; line cap; CI test greps fixture for keys outside
|
||||
the allowlist.
|
||||
- [S3#8] `transcript_path` confined: `isPathContained(path, ~/.claude/projects)`,
|
||||
`.jsonl` + valid envelope on line 1, lstat-reject symlinks, byte cap.
|
||||
- [A9] IPC request carries `protocol: 2`; hook treats a response lacking the protocol
|
||||
echo as "stale serve" and degrades LOUDLY (heartbeat entry + doctor warn); live test
|
||||
against a v1-shaped server.
|
||||
- [A3] Session-start digest: explicit list of digest-eligible MEMORY.md sections
|
||||
(respecting the template's own security-boundary note); three-case test (socket up /
|
||||
socket absent / malformed MEMORY.md).
|
||||
|
||||
**Persistence + sync (G4, G6, G8, G9, G13, G14/A5, G15, S3#2, S3#10):**
|
||||
- [G4] `hook session-start` checks for unpushed commits / dirty tree from crashed
|
||||
sessions, pushes, and names it in the greeting digest (SessionEnd is not the only
|
||||
persistence path anymore).
|
||||
- [G6] Verify + every push gate run `git ls-files` against a deny-glob list
|
||||
(`*.pglite`, `.env*`, keys) — a truncated or pre-existing .gitignore can't leak.
|
||||
- [G8] `bootstrap repo` refuses any pre-existing `origin` (always creates a dedicated
|
||||
repo); "couldn't verify visibility" is refuse-and-name-the-reason, never fail-open;
|
||||
idempotency keys off the remote URL, not the name probe.
|
||||
- [G9] Workspace lockfile (pid+timestamp) makes concurrent `bootstrap` runs impossible;
|
||||
second run exits "bootstrap already running (pid N)".
|
||||
- [G13] Fixed verify probe slug; sweep any prior probe before writing; excluded from
|
||||
retrieval; delete failure reported as a verify warning.
|
||||
- [G14/A5] The ENTIRE `sources push` (add+commit+push) runs under the existing
|
||||
durability flock; single-flight test: N concurrent pushes → 1 winner, N-1 clean
|
||||
"skipped, push in flight" exits, zero leftover locks.
|
||||
- [G15] Retention policy everywhere: MEMORY.md size cap in the template contract with
|
||||
rotation into `memory/reference/`; corpus pruned via
|
||||
`dream.synthesize.corpus_retention_days` (default 30); orphaned stop-hook buffers
|
||||
GC'd; doctor warns on all three.
|
||||
- [S3#2] Secret scan runs at corpus-WRITE time (redact matched span + log redaction);
|
||||
bootstrap adds one consent line naming the extraction/embedding provider that will
|
||||
see session text.
|
||||
- [S3#10] One `ensureGbrainHome()` choke point creates ~/.gbrain 0700 (all callers);
|
||||
brain-push.log 0600 + rotated; `--isolated` e2e asserts `git check-ignore
|
||||
.gbrain/git-credentials` passes and push never stages it.
|
||||
|
||||
**Transcripts (G3, A6):**
|
||||
- Parser registered as a dated spec-target (same pattern as host-format writers);
|
||||
heartbeat logs parsed-turns/bytes; `bytes>0 && turns==0` raises a LOUD status-file
|
||||
failure ("the agent stopped learning" is never silent); scrubbed
|
||||
`claude-code.jsonl` fixture (tool_use/tool_result/thinking/image/sidechain/summary/
|
||||
compact-boundary shapes) added to `test/fixtures/conversation-formats/` and wired
|
||||
into `check:conversation-parser` + fixture-privacy guard; session-id-keyed corpus
|
||||
filenames prevent double-ingest on resume.
|
||||
|
||||
**Template repo (G7, S3#4, A1, C3):**
|
||||
- Published ONLY from a CI release job (branch-protected, no direct pushes) behind the
|
||||
placeholder assertion + secret scan + privacy scripts run against the RENDERED
|
||||
artifact; fine-grained PAT secret scoped `contents:write` to the template repo only,
|
||||
documented in docs/RELEASING.md; version-job completeness check verifies template
|
||||
repo HEAD tree hash == vendored tree. Vendored rendered tree lives at
|
||||
`templates/bootstrap/template-repo/` — the CI guard diffs generator↔vendored OFFLINE
|
||||
(network comparison happens only in the release workflow). Template's first runbook
|
||||
step is `bootstrap status`, which hard-fails on a public `origin` before any
|
||||
identity file lands (closes the no-privacy-gate-on-this-door hole).
|
||||
|
||||
**Toolchain trust (S3#9):**
|
||||
- Prefer platform package managers for bun/gh; curl fallback downloads to a file,
|
||||
verifies against the pinned bun release's SHASUMS256.txt, then executes; the
|
||||
runbook's phase allowlist names the verified form as the only permitted variant.
|
||||
|
||||
**Observability (B1–B5):**
|
||||
- [B1] Append-only `~/.gbrain/bootstrap/install.jsonl` ({ts, phase, outcome,
|
||||
duration_ms, binary_version, harness, workspace}) written by every subcommand;
|
||||
`status` renders the tail.
|
||||
- [B2] Every verify run persisted to `~/.gbrain/bootstrap/verify-<ts>.json` (keep 5);
|
||||
doctor reports last verify timestamp/status/deltas.
|
||||
- [B3] Hook heartbeat records outcome + reason on EVERY invocation; trailing-20
|
||||
failure rate over threshold → one visible line inside the injected context block
|
||||
("brain context unavailable for the last N turns — run `gbrain doctor`").
|
||||
- [B4] `~/.gbrain/bootstrap/push-status.json`; doctor fails when last successful push
|
||||
>48h old with a dirty tree; surfaced in the SessionStart digest.
|
||||
- [B5] `bootstrap status --json` emits a support blob (workspace, binary version,
|
||||
harness, engine, last verify, last push, hook failure rate); AGENTS.md instructs
|
||||
the agent to relay it verbatim on any "something's broken" report.
|
||||
|
||||
**Tests (A1–A9 not already covered above):**
|
||||
- [A4] First-run tour prompts pinned in the questions.json↔template bijection guard +
|
||||
verify success-output snapshot test.
|
||||
- [A7] Flake pinning: hook deadline asserted against an injected slow-IPC stub (real
|
||||
latency kept as non-gating benchmark); kill-mid-phase via deterministic
|
||||
`GBRAIN_BOOTSTRAP_ABORT_AFTER=<phase>` injection; lock-contention asserts on
|
||||
LiveServeLockError class; all subprocess tests named `*.serial.test.ts` with
|
||||
explicit --timeout; offline Docker e2e runs from heavy-tests.yml (nightly + label),
|
||||
not the PR shard matrix.
|
||||
|
||||
**Release mechanics (C1 RESOLVED + C2, C4, C5, C6, C8):**
|
||||
- [C1 = D6-A, decided by Garry] **Distribution ref: single `latest-stable` ref.** The
|
||||
release job force-updates `latest-stable` to the just-verified release commit as its
|
||||
FINAL step (after binary assets publish + provenance attestation). README paste
|
||||
block, runbook URL, and `bun install -g github:garrytan/gbrain#latest-stable` all
|
||||
reference it permanently — no per-release tag pins, no 404 window, no per-ship
|
||||
README edits. The fetched runbook embeds the concrete VERSION it was cut from;
|
||||
`bootstrap status` compares that stamp to the installed binary (skew check intact).
|
||||
New guard `scripts/check-bootstrap-tag.sh`: README/runbook reference ONLY the
|
||||
sanctioned ref AND the runbook's embedded stamp equals VERSION. Modeled on
|
||||
test/release-workflow.test.ts.
|
||||
- [C2] README.md + BOOTSTRAP_FOR_AGENTS.md re-admitted to `scripts/ci-cache-hash.sh`
|
||||
ALLOW_PATTERNS (a README-only paste-block change must never skip CI green).
|
||||
- [C4/C5] `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + a SCRUBBED
|
||||
`docs/designs/AGENT_BOOTSTRAP_DESIGN.md` (banned names → capability-class phrasing,
|
||||
founder quotes/pilot identifiers/pricing-funnel strategy removed) land in the SAME
|
||||
commit; normativity paragraphs point in-repo; `bash scripts/check-privacy.sh` run
|
||||
against the staged index before that commit.
|
||||
- [C6] CHANGELOG states capabilities functionally ("installs hooks that run on each
|
||||
prompt; installs an opt-in background push job") linking to docs/guides/bootstrap.md
|
||||
which owns the full security/consent posture together with the rendered
|
||||
ACCESS_POLICY.md; zero plan IDs / review-round references.
|
||||
- [C7] Public product names ("GBrain for Codex"/"GBrain for Claude Code") are a
|
||||
pre-merge checklist item owned by Garry (trademark review); all committed copy uses
|
||||
the neutral `gbrain bootstrap` verb until sign-off.
|
||||
- [C8] Zero-migration release confirmed (facts/context_volunteer_events already
|
||||
exist); ALL new bootstrap telemetry stays on the filesystem — the moment it moves
|
||||
into a table it becomes migration v126 + bootstrap-coverage + engine-parity
|
||||
obligations.
|
||||
- [G16] User-scope consent question names the project-scope tradeoff explicitly;
|
||||
CLAUDE.md renders a one-line "this agent lives in <dir>" note.
|
||||
|
||||
## Outside-voice absorption (Codex, 2026-08-07 — dispositions under Garry's
|
||||
## auto-accept directive; CX ids trace to the codex output)
|
||||
|
||||
**ACCEPTED — product-critical (the two the whole review chain missed):**
|
||||
- [CX-P0.5 **Keyless mode is a first-class design requirement.**] The named pilot user
|
||||
has ChatGPT Pro but NO API keys — and gbrain's embeddings + extract_facts (Haiku)
|
||||
are API-metered. Bootstrap MUST work with zero API keys: the HARNESS AGENT is the
|
||||
subsidized LLM, so AGENTS.md instructs it to author facts/timeline/links explicitly
|
||||
through write ops (zero API cost); search degrades to BM25 keyword (no embeddings);
|
||||
verify prints an honest capability report ("keyless mode: keyword search, agent-
|
||||
authored memory; add ONE key to unlock embeddings + auto-extraction") and the
|
||||
interview offers the optional key question. The magic-moment check must pass in
|
||||
keyless mode (fact written by the agent via ops, recalled next session). Quota-gate
|
||||
copy amended: API spend (embeddings/extraction) is metered separately from
|
||||
subscription quota and is ZERO in keyless mode; with a key, existing spend gates
|
||||
(docs/operations/spend-controls.md) govern. [also resolves CX-P0.6]
|
||||
- [CX-P0.1+P0.3 **Serve-resident maintenance sweep** closes the persistence loop.]
|
||||
Nothing previously ingested the transcript corpus into the live brain (dream is
|
||||
disabled; CLI can't open PGLite under a live serve), and remote `put_page`
|
||||
deliberately skips auto-link/timeline extraction (operations.ts:1273) so the graph
|
||||
would never compound from harness writes. Fix: the serve process (the lock owner)
|
||||
runs a bounded, spend-gated maintenance sweep — on startup and idle — that (a)
|
||||
ingests unprocessed corpus files (keyless mode: skipped, agent-authored memory
|
||||
covers it), (b) runs the deterministic zero-LLM link/timeline extraction over
|
||||
recently written workspace-source pages. Verify's graph-floor check exercises the
|
||||
REAL MCP write path end-to-end (write via stdio op → sweep → edge query), never a
|
||||
synthetic install-time shim.
|
||||
|
||||
**ACCEPTED — durability/trust:**
|
||||
- [CX-P0.4] Write-through failures on `put_page` are best-effort today → for the
|
||||
workspace source they become LOUD: doctor check counts DB-pages lacking file
|
||||
backing; surfaced in the SessionStart digest; sources push reconciles before commit.
|
||||
- [CX-P1.1] Single-principal visibility posture: bootstrap sets the workspace brain's
|
||||
fact default visibility to 'world' (the desktop agent IS the principal's main
|
||||
session; ACCESS_POLICY.md documents it; multi-tier users flip the knob). Keeps the
|
||||
IPC world-only filter (S3#1) AND working personalization.
|
||||
- [CX-P1.2] The injected turn_context block is wrapped in the same "retrieved data,
|
||||
never instructions" subordinate envelope as interview answers (provenance-labeled).
|
||||
- [CX-P1.4] GUI env inheritance: MCP registration carries the absolute binary path +
|
||||
explicit env (GBRAIN_SOURCE, GBRAIN_HOME when --isolated); provider keys resolve
|
||||
from ~/.gbrain/config.json, never from shell env, for GUI-spawned serves.
|
||||
- [CX-P1.5] **`bootstrap attach` mode** (machine two): a cloned repo carrying
|
||||
agent.json enters attach mode (register source, hooks --repair, MCP, verify) —
|
||||
the refuse-pre-existing-origin rule (G8) applies only when agent.json is absent.
|
||||
- [CX-P1.6] Git conflict model: sources push does fetch + divergence-safe rebase pull
|
||||
(reuse harden's divergenceSafePull) before push; non-FF/conflict = loud status +
|
||||
agent-relayed instruction, never silent retry; repo-local git author identity set
|
||||
at `bootstrap repo` (ported from setup-private-repo).
|
||||
- [CX-P1.7] Search-mode consent folds into the interview as one optional question
|
||||
(default balanced) — preserves the INSTALL_FOR_AGENTS consent contract without
|
||||
another stop; the spike's TTFM measurement counts every stop.
|
||||
- [CX-P1.9] Non-circular error surface: README + GITHUB.md carry the one recovery
|
||||
command ("if it seems broken: `gbrain doctor`"); doctor covers hook/push/serve
|
||||
health independent of the possibly-broken agent loop.
|
||||
- [CX-P1.10] Forget semantics documented honestly in ACCESS_POLICY.md/GITHUB.md: the
|
||||
repo is append-only history; true deletion = documented history-rewrite procedure;
|
||||
GitHub remains default-but-optional (local-only mode with honest warning).
|
||||
- [CX-P1.11] Template repo README embeds the same VERSION stamp as the runbook;
|
||||
`bootstrap status` validates it identically (closes the adopter-skew window).
|
||||
- [CX-P1.12] agent.json spec labeled **provisional-v1**: desktop-side validation
|
||||
only, consumers tolerate unknown fields, hosted mount may bump to v2 with a
|
||||
migration note — a version contract, not a frozen promise.
|
||||
- [CX-P0.7+P1.13] Spike/pilot instrument upgraded from binary pass/fail to product
|
||||
metrics: per-turn write-attempt vs durable-write precision, correct-recall rate,
|
||||
false-memory incidents, correction round-trips — measured through the 2-week pilot
|
||||
(the pilot IS the extended sample); the 0-failures-in-20 bar remains only the
|
||||
minimum to START the pilot.
|
||||
|
||||
**REJECTED (with reasons, recorded):**
|
||||
- [CX-P0.2] "Two desktop apps cannot share the brain" — TRUE for simultaneous
|
||||
sessions and already the documented v1 limit (polite collision + doctor + attach
|
||||
proxy fast-follow); sequential use works. No change beyond what's accepted.
|
||||
- [CX-P1.8] "Networked paste flow untested" — known; manual clean-machine acceptance
|
||||
in v1 + networked Docker e2e deliberately deferred (D3.3b). Stands.
|
||||
|
||||
**CROSS-MODEL TENSION (recorded, not re-litigated):** Codex's "fundamental
|
||||
simplification" (Codex-only, 3-question interview, no hooks/template/GitHub, 2-week
|
||||
manual pilot before building) is a REDUCTION-mode argument against the settled
|
||||
cathedral decision (D1-A) and accepted Approach B scope. Disposition: rejected as
|
||||
sequencing (Garry chose the cathedral 30 minutes prior, with the two-wave case
|
||||
recorded for revisit + a size trip-wire), but its measurement substance was absorbed
|
||||
via CX-P0.7/P1.13, and the spike + pilot ARE the "prove the loop" instrument, run
|
||||
before the doors ship rather than instead of building them.
|
||||
|
||||
|
||||
## Design (synthesized)
|
||||
|
||||
Synthesis of three independent designs (DX lens, runtime-parity lens, architecture lens) +
|
||||
an adversarial critique that verified the load-bearing claims against both repos. Where the
|
||||
designs disagreed, the critique's evidence-checked winner is taken.
|
||||
|
||||
### The experience (end state)
|
||||
|
||||
One paste block (README + tweet) → the agent fetches `BOOTSTRAP_FOR_AGENTS.md` (raw GitHub
|
||||
URL **pinned to a release tag**, same mechanism as INSTALL_FOR_AGENTS.md) and drives:
|
||||
|
||||
preflight → interview (chat, 12Q/6-required, hard gate) → render identity files → skills →
|
||||
MCP + hooks wiring → sources add/sync/embed → private GitHub repo (created, privacy-verified,
|
||||
pushed) → `gbrain bootstrap verify` (exit-code contract) → completion manifest.
|
||||
|
||||
Human actions: paste, answer interview in chat, `gh auth login` if needed, ~2 consent
|
||||
questions. Target ≤15 min. Definition of done includes the magic moment: after restart, the
|
||||
agent recalls a fact the human said during the interview.
|
||||
|
||||
Second session: SessionStart hook injects identity digest + "since last time"; every prompt
|
||||
gets Live Context + ≤3 volunteered brain pointers + hot memory injected via hook; the agent
|
||||
writes facts back same-turn through MCP ops; SessionEnd ingests the transcript into the dream
|
||||
corpus and fires a scan-gated commit+push. "Your local harness IS your agent."
|
||||
|
||||
### Decisions (settled by panel + critique)
|
||||
|
||||
- **D1 Topology: single private repo.** Workspace root = identity files (SOUL.md, USER.md,
|
||||
MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md, GITHUB.md, ACCESS_POLICY.md) + `skills/` +
|
||||
`memory/` (daily notes + README) + `state/` + **`brain/`** (people/ companies/ meetings/
|
||||
concepts/ daily/). **Only `brain/` is registered as the gbrain source** (critique: indexing
|
||||
the whole repo puts the contract files into the retrieval corpus — noise + self-referential
|
||||
injection surface). `.gbrain-source` dotfile at root routes CLI calls. PGLite DB is NEVER
|
||||
in the repo (`.gitignore` renders first: `*.pglite`, `.env*`, `state/*.local`, corpus paths,
|
||||
keys/PEM). Two-repo model stays documented as the graduation path in GITHUB.md/docs.
|
||||
- **D2 Workspace = the cwd the user pasted in.** Guard: if cwd is an existing code project
|
||||
(tracked files/remote), ask ONE relocation question. DB default: global `~/.gbrain` host
|
||||
brain (workspace is a source); `--isolated` escape hatch per resolved D2.
|
||||
- **D3 Delivery: hybrid.** New `gbrain bootstrap` command family (deterministic, idempotent,
|
||||
exit-coded) + fetched runbook (judgment: interview wording, read-backs, consent). Bootstrap
|
||||
is **CLI-only like init/connect — NOT an operation**; zero new `ctx.remote` surface (a
|
||||
remote-triggerable `gh repo create` + filesystem render is forbidden by the trust model).
|
||||
Subcommands: `status` (resume entrypoint; also verifies runbook version stamp vs binary —
|
||||
supply-chain skew check), `interview --init|--set K V|--skip K|--status|--confirm|--show`,
|
||||
`render [--force] [--only F] [--minimal]`, `repo`, `hooks --harness claude-code|codex
|
||||
[--repair]`, `verify [--json]`.
|
||||
- **D4 Interview: one question bank, two entry points.** Port codex-as-agent's
|
||||
`questions.json` (scrubbed) to `templates/bootstrap/questions.json`: 12 asked / 6 required
|
||||
(AGENT_NAME, PRINCIPAL_NAME, AGENT_PURPOSE, AGENT_TOP_JOBS, PRINCIPAL_CONTEXT,
|
||||
VOICE_REGISTER — the wince question and vibe samples from soul-audit Phase 2 fold in as
|
||||
optional keys). Answers → `<ws>/state/interview.json` (committed; identity source of truth,
|
||||
same sensitivity as the rendered USER.md; makes re-render-on-new-machine work).
|
||||
`--status` exits non-zero until required present; `render` additionally refuses until
|
||||
`--confirm` (set only after the read-back "Is this the thing you want in the room?").
|
||||
Hard rules verbatim in runbook: NEVER INVENT ANSWERS, accept "skip", quote literal words.
|
||||
`skills/soul-audit/SKILL.md` becomes the re-run/deepen surface over the same bank
|
||||
(`interview --set` + `render --only SOUL.md`) — one bank, CI-guarded against drift.
|
||||
- **D5 Per-turn context (critique-decided; the PGLite lock is the constraint).**
|
||||
`gbrain serve` holds the PGLite single-writer lock for its lifetime (#2348: live holders
|
||||
are never stolen) — so hooks must NEVER open the engine on PGLite. New engine-free
|
||||
`gbrain hook <event>` command (no-engine dispatch branch in cli.ts) talks ONLY to serve's
|
||||
existing resolve-IPC unix socket (`src/core/context/resolve-ipc.ts`), widened with
|
||||
back-compat `kind: 'resolve' | 'turn_context'`. `turn_context` request carries
|
||||
{window, priorContextText, sourceId}; serve assembles server-side (new
|
||||
`src/core/context/turn-context.ts`): reflex pointers + volunteerContext pages (≤3) +
|
||||
hot-memory facts (the same content `_meta.brain_hot_memory` carries — this ROUTES AROUND
|
||||
the _meta invisibility rather than fixing the harness). Postgres fallback: hook opens the
|
||||
engine directly (multi-connection safe) since the IPC socket only exists on PGLite.
|
||||
- Claude Code hooks (written to **`.claude/settings.local.json`** — gitignored; committed
|
||||
hooks with absolute paths are a portability trap + persistence-of-execution surface;
|
||||
`bootstrap hooks --repair` re-renders on a new machine):
|
||||
- SessionStart → `gbrain hook session-start`: zero-DB file reads (date/tz, MEMORY.md
|
||||
open-commitments, ops/tasks.md if present) + best-effort socket warm probe. ≤1.5s.
|
||||
- UserPromptSubmit → `gbrain hook user-prompt`: stdin hook JSON → tail last 4 turns from
|
||||
transcript_path → socket turn_context → stdout additionalContext block.
|
||||
**800ms hard self-deadline**, fail-open (exit 0, empty stdout), heartbeat JSONL at
|
||||
`~/.gbrain/integrations/hooks/` for doctor.
|
||||
- Stop → `gbrain hook stop`: append turn to live per-session buffer (pure file append).
|
||||
- SessionEnd → `gbrain hook session-end`: parse full transcript .jsonl → corpus .txt in
|
||||
`dream.synthesize.session_corpus_dir` (default `~/.gbrain/transcripts/corpus/`, 0700,
|
||||
never in the repo) + fire scan-gated commit+push (D6). Closes the transcript gap in PR1.
|
||||
- Absolute binary path everywhere; `GBRAIN_HOOKS=0` kill switch.
|
||||
- Codex (no hooks): honest pull model. Rendered AGENTS.md carries adapted per-message
|
||||
gates 0–7 (entity lookup = "call recall/volunteer_context with recent window";
|
||||
receipts; WRITE IT DOWN same turn via extract_facts/put_page) +
|
||||
`codex mcp add gbrain -- gbrain serve`. FF2: `notify` hook in ~/.codex/config.toml as
|
||||
transcript sweeper (validate the event semantics first). Runbook/verify state the
|
||||
degradation plainly: Claude Code = push-on-hook, Codex = pull-on-protocol.
|
||||
- **Write-path rule rendered into AGENTS.md** (critique hole 2): on PGLite, durable
|
||||
knowledge is written through MCP ops (put_page/extract_facts/add_timeline_entry), never
|
||||
by editing brain/ files directly — file edits are invisible to retrieval until a sync
|
||||
can run, and sync can't run while serve holds the lock. MEMORY.md/memory/ file edits are
|
||||
fine (file-plane, loaded by path not retrieval).
|
||||
- **D6 Private repo + persistence.** `gbrain bootstrap repo` = TS port of
|
||||
setup-private-repo.mjs: gh-auth exit-2 gate (only human step: `gh auth login`), slugified
|
||||
`<agent-name>-workspace` collision probe, `gh repo create --private --source . --push`,
|
||||
**privacy verified via `gh api ... --jq .private` (hard fail)**. Sync: extract secret scan
|
||||
to `src/core/secret-scan.ts` (sk-/gh[pousr]_/github_pat_/xox[baprs]-/PEM; blocks commit);
|
||||
new `gbrain sources push [<id>|--path]` = scan-gated add+commit+push, refuses public
|
||||
remotes, pushes even on clean tree. `hardenBrainRepo` gains the scan as a step (its
|
||||
existing post-commit hook + cron machinery is reused, NOT a parallel sync system).
|
||||
Cadence per resolved D3: 15-min cron installed after an explicit consent question;
|
||||
SessionEnd-hook push always on as the no-daemon backstop/fallback. GITHUB.md persistence
|
||||
contract rendered.
|
||||
- **D7 Rendered files.** AGENTS.md (adapted gates + hard gates: WRITE IT DOWN, NO SILENT
|
||||
FAILURE, VERIFY BEFORE CLAIMING DONE, RED LINES, PRIVATE REPO PERSISTENCE + brain-first
|
||||
protocol from docs/tutorials/connect-coding-agent.md + brain filing contract rendered from
|
||||
skills/_brain-filing-rules.md), CLAUDE.md (thin: @AGENTS.md @SOUL.md @USER.md @MEMORY.md +
|
||||
hooks note), SOUL.md (codex-as-agent section skeleton: Identity/Mission/Worldview/The
|
||||
Standard/Honesty/Voice+wince/Good vs bad output/High agency/Never — IDENTITY.md merged in),
|
||||
USER.md ("their literal words are ground truth"), **MEMORY.md (new template**: hot state,
|
||||
corrections format `- YYYY-MM-DD — rule (Bug: ...)`, open commitments, security-boundary
|
||||
note), ACCESS_POLICY.md, HEARTBEAT.md (quiet hours, verify-time-first, silence contract,
|
||||
jobs disabled), GITHUB.md, memory/README.md, brain/ dirs + READMEs, .gitignore,
|
||||
state/interview.json, state/mcp.json (portable snippet). Skills via existing
|
||||
`gbrain skillpack scaffold --all` (resolved D4) + inert-skill report (skillpack check
|
||||
wired into verify AND into the completion manifest's next-steps block with the exact API
|
||||
keys to add). ALL templates REWRITTEN generic (privacy IRON RULE —
|
||||
never copy codex-as-agent prose verbatim; CI placeholder assertion). TOOLS.md deferred
|
||||
(essential lines fold into AGENTS.md). `installDefaultTemplates` (init.ts:1513) finally
|
||||
gets its caller via `render --minimal`.
|
||||
- **D8 Verify (union).** (1) doctor green; (2) DB round-trip: put_page → get → query-with-
|
||||
score → DELETE probe page; (3) MCP registered (`claude|codex mcp list`) + probeBrainIdentity
|
||||
smoke; (4) {{TOKEN}} sweep hard-fail; (5) byte floors (SOUL.md ≥3000B, USER.md ≥1000B);
|
||||
(6) secret scan clean; (7) repo private via API; (8) **hooks smoke UNDER LIVE SERVE**
|
||||
(spawn real serve, pipe fixture UserPromptSubmit stdin, assert non-empty block + <800ms +
|
||||
never acquires the lock — bootstrap-time-only smoke is a false green); (9) one manual
|
||||
sources push succeeded; (10) inert-skill report; (11) transcript parser dry-run on fixture.
|
||||
Prints ranked completion manifest. Re-runnable weekly as the workspace rot self-check.
|
||||
- **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
|
||||
daemon). OPT-IN: 15-min harden cron. Autopilot NOT default on PGLite (verified: its
|
||||
sync/embed children would contend with every live serve for the single-writer lock, and
|
||||
nothing handles LiveServeLockError politely today) — recommended on Postgres; any future
|
||||
scheduled job must treat lock-held as skip-silently-and-log. LLM crons (briefing, dream via
|
||||
`claude -p`/`codex exec`) ship rendered-but-disabled with the enable-one-at-a-time ritual.
|
||||
- **D10 Phasing.**
|
||||
- **PR1 (complete usable experience):** BOOTSTRAP_FOR_AGENTS.md + README paste block;
|
||||
`gbrain bootstrap` family; `gbrain hook` + IPC v2 turn_context; new templates + question
|
||||
bank; secret-scan + `sources push` + harden integration; SessionEnd transcript ingest
|
||||
(Claude Code .jsonl parser); skillpack scaffold wiring; soul-audit re-run update; doctor
|
||||
checks (hooks heartbeat, dual-serve report, sync check); docs; unit + e2e (incl.
|
||||
hook-under-serve + lock-contention pins); CI guards.
|
||||
- **FF2:** Codex notify sweeper + ~/.codex/sessions parser. **FF3:** `gbrain serve
|
||||
--attach` stdio proxy (two harnesses fully concurrent on one PGLite brain). **FF4:**
|
||||
cron fleet + cron-doctor port + heartbeat activation. **FF5:** Docker fresh-machine e2e
|
||||
in CI, _meta surfacing experiments, per-turn-context BrainBench eval, upgrade re-render
|
||||
nudge via runPostUpgrade (`bootstrap render --diff`).
|
||||
|
||||
### Security/trust invariants (critique holes, addressed)
|
||||
|
||||
1. Paste block pinned to release tag; runbook version-stamped; `bootstrap status` compares
|
||||
stamp vs binary and warns; runbook instructs refusing steps outside its phase list.
|
||||
2. Write-through-ops rule in AGENTS.md (above).
|
||||
3. MCP scope: project scope default, user-scope opt-in (resolved D1); threat named in
|
||||
ACCESS_POLICY.md either way.
|
||||
4. Codex sandbox reality: runbook carries a Codex-specific preflight (approval mode /
|
||||
workspace-write + network consent) — the paste block warns the human they'll be asked.
|
||||
5. Hooks in settings.local.json (gitignored) + --repair.
|
||||
6. Upgrade story: FF5 re-render nudge; render never clobbers (backup on --force).
|
||||
7. Transcript corpus + answers privacy: corpus 0700 outside repo; interview.json committed
|
||||
(same sensitivity as rendered USER.md, which is committed); responsible-disclosure
|
||||
phrasing in CHANGELOG (functional, no attack-surface enumeration).
|
||||
8. Process invariants: CLAUDE.md dispatcher row edit → `bun run build:llms` same commit;
|
||||
KEY_FILES.md entries current-state prose; version-first PR title; ship via /ship;
|
||||
/document-release after.
|
||||
|
||||
### New/changed artifacts (paths)
|
||||
|
||||
| Path | New/changed |
|
||||
|---|---|
|
||||
| `BOOTSTRAP_FOR_AGENTS.md` | NEW root runbook (fetched by paste block) |
|
||||
| `README.md` | dedicated "For Codex" / "For Claude Code" paste-block sections, ordered Codex → Claude Code → OpenClaw/Hermes at equal weight (user decision 2026-08-09, supersedes D5's ordering; both platform paths preserved) |
|
||||
| `src/commands/bootstrap.ts` (+ `src/commands/bootstrap/*.ts`) | NEW dispatcher + subcommands |
|
||||
| `src/commands/hook.ts` | NEW `gbrain hook session-start|user-prompt|stop|session-end` |
|
||||
| `src/cli.ts` | CHANGED: `bootstrap`+`hook` in no-engine dispatch branch |
|
||||
| `src/core/bootstrap/{interview,render,private-repo,verify}.ts` | NEW (TS ports) |
|
||||
| `src/core/secret-scan.ts` | NEW (shared: sources push, harden, verify) |
|
||||
| `src/commands/sources.ts` + `src/core/brain-repo-durability.ts` | CHANGED: `sources push`, scan-gated hook/cron |
|
||||
| `src/core/context/resolve-ipc.ts` + `src/mcp/server.ts` | CHANGED: IPC v2 `turn_context` (back-compat) |
|
||||
| `src/core/context/turn-context.ts` | NEW server-side block assembly |
|
||||
| `src/core/transcripts/claude-code-jsonl.ts` | NEW parser + corpus writer |
|
||||
| `templates/{MEMORY,AGENTS,CLAUDE,GITHUB,memory-README}.md.template` + `gitignore.template` + enriched SOUL/USER/HEARTBEAT/ACCESS_POLICY | NEW/CHANGED (generic, scrubbed) |
|
||||
| `templates/bootstrap/questions.json` | NEW shared question bank |
|
||||
| `skills/soul-audit/SKILL.md` | CHANGED: re-run surface over the bank |
|
||||
| `src/commands/doctor.ts` | CHANGED: hooks heartbeat, dual-serve, sync checks |
|
||||
| `docs/guides/bootstrap.md` + docs/mcp/ + connect-coding-agent.md cross-links | NEW/CHANGED |
|
||||
| `scripts/check-bootstrap-templates.sh` | NEW CI guard (token↔bank bijection + placeholder-only assertion) |
|
||||
| `docs/architecture/KEY_FILES.md`, `CLAUDE.md` (+build:llms) | CHANGED |
|
||||
| `test/bootstrap-*.test.ts`, `test/hook-command.test.ts`, `test/secret-scan.test.ts`, `test/e2e/bootstrap-lifecycle.test.ts`, IPC back-compat tests | NEW |
|
||||
|
||||
### Port map (codex-as-agent → gbrain)
|
||||
|
||||
interview.mjs → core/bootstrap/interview.ts · render-templates.mjs → core/bootstrap/render.ts
|
||||
· setup-private-repo.mjs → core/bootstrap/private-repo.ts · verify-install.mjs (+
|
||||
install-gbrain.mjs round-trip) → core/bootstrap/verify.ts · git-sync.mjs → secret-scan.ts +
|
||||
sources push · questions.json → templates/bootstrap/questions.json (scrubbed) · AGENTS.md
|
||||
gates 0–7 / SOUL/USER/MEMORY/HEARTBEAT/GITHUB templates → REFERENCE structure, REWRITE
|
||||
content (privacy rule) · install-skills.mjs → REFERENCE (skillpack scaffold exists) ·
|
||||
cron fleet/cron-doctor → FF4 · codex-plugin-spec one-file-owns-format pattern → governs the
|
||||
settings.local.json + config.toml writers (single module owns each host format).
|
||||
|
||||
### Verification (how we know it works end-to-end)
|
||||
|
||||
- Unit: interview gate exit codes; render token hard-fail/no-clobber/backup; secret-scan
|
||||
fixture corpus (positives + benign lookalikes); hook stdin→JSON contract; IPC v1↔v2
|
||||
back-compat both directions; questions.json↔template token bijection.
|
||||
- E2E: full lifecycle in temp dir (sandboxed GBRAIN_HOME, PATH-shimmed fake gh/claude/codex
|
||||
recording invocations): render → repo → verify exit 0; idempotency (second run no-op);
|
||||
kill-mid-phase → `status` resumes. **hook-under-serve** and **lock-contention** pins
|
||||
(permanent). Engine parity for turn-context on both engines.
|
||||
- Manual acceptance at ship: fresh macOS account, real paste, both harnesses, timed
|
||||
(≤15 min, ≤3 human actions).
|
||||
|
||||
### Resolved user decisions (Garry, 2026-08-07)
|
||||
|
||||
- **D1 = A. MCP scope: project scope default, user-scope opt-in** (consent question during
|
||||
bootstrap; threat named in ACCESS_POLICY.md).
|
||||
- **D2 = C. DB: global `~/.gbrain` default + documented `--isolated` escape hatch.**
|
||||
`gbrain bootstrap --isolated` threads `GBRAIN_HOME=<workspace>/.gbrain` through init, MCP
|
||||
registration env (`claude mcp add -e` / codex config env), and the hook commands in
|
||||
settings.local.json. Port install-gbrain.mjs's guards: GBRAIN_HOME does NOT isolate
|
||||
`sync.repo_path` (set it explicitly), strip ambient GBRAIN_DATABASE_URL/DATABASE_URL/
|
||||
GBRAIN_BRAIN_ID, assert database_path is inside the workspace. `.gbrain/` already
|
||||
gitignored by the rendered template.
|
||||
- **D3 = B with consent. 15-min scan-gated commit+push cron is the default posture, but the
|
||||
runbook ASKS PERMISSION before installing it** ("Enable background persistence? Installs a
|
||||
15-min launchd/cron job that commits + pushes this workspace, secret-scan-gated"). The
|
||||
SessionEnd-hook push stays as the always-available, no-daemon backstop and is the fallback
|
||||
when the cron is declined — persistence never silently disappears. Autopilot posture
|
||||
unchanged: off on PGLite, recommended on Postgres.
|
||||
- **D4 = B. ALL bundled skills scaffold in** (`skillpack scaffold --all`). The onboarding
|
||||
ENDS with a next-steps block in the completion manifest: the inert-skill report (which
|
||||
installed skills are dormant for lack of which API key), the exact keys to add and where,
|
||||
and pointers to soul-audit (deepen identity this week) + cold-start (fill the brain with
|
||||
your data). The routing-table-size concern is mitigated by frontmatter-trigger routing
|
||||
(authoritative since v0.36) and noted for a future curated-profile fast-follow if dispatch
|
||||
accuracy suffers in practice.
|
||||
- **D5 = Codex/Claude-Code-scoped placement.** [Superseded by the 2026-08-09 user
|
||||
decision — see as-shipped delta 6 and the artifact table's README row.] This is NOT the new headline install —
|
||||
most users still use GBrain with OpenClaw/Hermes, so `INSTALL_FOR_AGENTS.md` remains the
|
||||
primary paste path at the top of the README. The bootstrap paste block becomes the
|
||||
flagship "For Codex" / "For Claude Code" README sections, ahead of the OpenClaw/Hermes path at equal weight (and
|
||||
docs/tutorials/connect-coding-agent.md cross-links it). Command name stays
|
||||
`gbrain bootstrap`; paste block is pinned to a release tag (supply-chain integrity).
|
||||
BOOTSTRAP_FOR_AGENTS.md opens with a scope note: "For Claude Code / Codex. Running
|
||||
OpenClaw or Hermes? Use INSTALL_FOR_AGENTS.md instead."
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Agent Bootstrap — Spike Instrument (build order 0)
|
||||
|
||||
Manual validation that gates door-1 (Codex/ChatGPT desktop) ship. Run on a machine
|
||||
the maintainer does not own, with fresh accounts. Owner: the maintainer. Timebox:
|
||||
~1 week wall-clock. Outcomes feed the design doc's gate
|
||||
([AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md)).
|
||||
|
||||
## Exit questions and pass bars
|
||||
|
||||
**#1 (blocking) — the write seam.** What reliably persists memory per turn/session
|
||||
on the ChatGPT-desktop surface?
|
||||
- Protocol: run 20 sessions across ≥3 days of ordinary use. Each session must
|
||||
produce at least one durable write (a page/fact retrievable in the NEXT session).
|
||||
- Pass: 0 durable-write failures in 20 sessions; else extend to 50 and log every
|
||||
failure's cause (crash, sleep, approval friction, format drift, model forgot).
|
||||
- Below the bar → door 1 demotes to documented beta (Codex CLI unaffected).
|
||||
|
||||
**#2 — the read seam.** Is injected/pulled context demonstrably present at turn
|
||||
start? Pass: context block present (or the degraded pull-mode documented as
|
||||
door-1's v1 behavior). Pass expands the greeting digest to door 1.
|
||||
|
||||
**#3 — capability surface.** Record, with screenshots: local folder access
|
||||
(yes/no/how), MCP registration path (config file? `codex mcp add`? UI?), approval
|
||||
taps for each toolchain step (count them), connector availability for
|
||||
email/calendar (yes/no/degraded).
|
||||
|
||||
**#4 — quota.** Per harness the doors run on (Claude Code: Max plan; Codex: the
|
||||
ChatGPT plan): log each day's usage-meter readings during the pilot.
|
||||
- Load model: ordinary sessions + hooks + one session-triggered schedule.
|
||||
- Pass: a p90 day consumes ≤10% of the weekly allowance (per-door; one harness
|
||||
failing cuts schedule scope for that door only).
|
||||
- Measurement: the harness's own usage UI (screenshot at day start/end) + a tally
|
||||
of sessions/turns from the transcript dir. No telemetry — this is a manual
|
||||
instrument by design.
|
||||
|
||||
**#5 — TTFM baseline.** One full paste-to-verified install, timed. Count every
|
||||
human action (paste / auth click / interview answers / consents / approval taps).
|
||||
Toolchain download time recorded separately (excluded from the 15-minute target).
|
||||
|
||||
## Pilot metrics (continue for 2 weeks after the spike)
|
||||
|
||||
Per week, from session review (screen recordings + self-report — no telemetry):
|
||||
correct-write rate (things worth remembering that got written), correct-recall
|
||||
rate (recalls that were right), false-memory incidents (recalled things that were
|
||||
wrong), correction round-trips (corrections that stuck as standing rules). The
|
||||
0-failures-in-20 bar is the minimum to START the pilot, not the proof — the pilot
|
||||
is the sample.
|
||||
|
||||
## Log template (one row per session)
|
||||
|
||||
| # | date | door | duration | writes attempted | writes durable | recalls right/wrong | approvals | notes |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Deliverable
|
||||
|
||||
A filled copy of this doc committed as `AGENT_BOOTSTRAP_SPIKE_RESULTS.md`
|
||||
(scrubbed: no real names beyond the maintainer, no account identifiers), plus the
|
||||
gate decision recorded in the design doc: door-1 ships full / ships as documented
|
||||
beta / schedule scope cut per quota.
|
||||
|
||||
**Gate status:** not yet run — no `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` is committed,
|
||||
so no gate decision is recorded and door 1 has not been promoted past the
|
||||
documented-beta bar by this instrument. Update this line when the results land.
|
||||
@@ -154,10 +154,11 @@ these are the densest source of real bugs in the whole backlog.
|
||||
`aliases:`), first-H1-title, and basename fallback resolution (path-equality-only gives
|
||||
~5.5% edge recall on real vaults). Master shipped global-basename (#1388); the alias/
|
||||
title fallbacks are the still-novel part.
|
||||
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **OPEN, high.** The
|
||||
link extractor's `DIR_PATTERN` is a frozen 16-prefix const that ignores pack-declared
|
||||
`path_prefixes`, so default-pack installs silently lose wikilinks to `person/`,
|
||||
`writing/`, `wiki/*`. Resolve prefixes from the active pack.
|
||||
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **RESOLVED via #2576.**
|
||||
The extractor no longer gates on the frozen `DIR_PATTERN` whitelist: any dir-shaped
|
||||
path produces a candidate and the persist paths' page-existence checks decide, so
|
||||
pack-declared directories (`person/`, `writing/`, `wiki/*`, `ops/`) link without a
|
||||
prefix registry.
|
||||
- **DB-source extraction** (#1539, @afshaker) — **OPEN, high.** The cycle's extract phase
|
||||
only walks the filesystem, so DB-resident pages (imported transcripts, remote-DB brains)
|
||||
never get links/timeline and `brain_score` is capped. Thread `source:'db'`.
|
||||
|
||||
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
|
||||
|
||||
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
|
||||
|
||||
`gbrain install voice-agent`
|
||||
`gbrain skillpack scaffold voice-agent`
|
||||
|
||||
That's it.
|
||||
|
||||
+31
-29
@@ -8,11 +8,19 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
|
||||
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
||||
that lives on top of that format.
|
||||
|
||||
## v0.41 update — the LOOP is now real
|
||||
If you're touching **memory behavior** rather than retrieval ranking — the
|
||||
Retrieval Reflex push path, conversation→facts write-back, cross-session
|
||||
continuity, source isolation — the gate for that layer is **BrainBench**
|
||||
(`gbrain eval brainbench`): see [`eval/BRAINBENCH.md`](./eval/BRAINBENCH.md).
|
||||
The two stack: this doc's capture→baseline→replay loop gates query-level
|
||||
result sets; BrainBench gates the memory behaviors above them, with its own
|
||||
committed baseline (`evals/brainbench/baselines/main.json`) compared against
|
||||
MAIN's copy in CI so a PR can't self-approve a regression.
|
||||
|
||||
Before v0.41, you could capture eval rows and replay them but nothing
|
||||
stitched them into a gate. `gbrain bench publish` + `gbrain eval gate`
|
||||
close the loop. Two gates:
|
||||
## The eval gate loop
|
||||
|
||||
`gbrain bench publish` + `gbrain eval gate` stitch captured eval rows into
|
||||
a pass/fail gate. Two gates:
|
||||
|
||||
- **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline
|
||||
you captured against your current brain. Catches: "did my refactor break
|
||||
@@ -39,7 +47,7 @@ gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/person
|
||||
gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson
|
||||
```
|
||||
|
||||
### Privacy posture (D9)
|
||||
### Privacy posture
|
||||
|
||||
**Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real
|
||||
user captures stay local in `~/.gbrain/baselines/`. The boundary is
|
||||
@@ -131,14 +139,9 @@ gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
|
||||
```
|
||||
|
||||
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": true}} // force on
|
||||
{"eval": {"capture": false}} // force off
|
||||
```
|
||||
|
||||
Explicit config beats the env var both directions.
|
||||
The full on/off resolution order (config beats env var, both directions) is
|
||||
documented once in [`eval-capture.md`](./eval-capture.md) — that file is the
|
||||
capture contract.
|
||||
|
||||
## The 4-command loop
|
||||
|
||||
@@ -205,7 +208,7 @@ retrieval, and which queries did it move most?"
|
||||
|
||||
For a third evaluation axis — public benchmark, ground-truth labels, full
|
||||
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
|
||||
<dataset.jsonl>` runs the LongMemEval benchmark against gbrain's
|
||||
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
|
||||
imported, the question asked, the hypothesis emitted as JSONL — exactly the
|
||||
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
|
||||
@@ -337,7 +340,7 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
|
||||
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
|
||||
|
||||
## Public benchmarks: LongMemEval (v0.28.8)
|
||||
## Public benchmarks: LongMemEval
|
||||
|
||||
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
|
||||
benchmark directly against gbrain's hybrid retrieval. Different evaluation
|
||||
@@ -398,7 +401,7 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
|
||||
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
|
||||
LLM latency.
|
||||
|
||||
## Measuring brain consistency over time (v0.32.6)
|
||||
## Measuring brain consistency over time
|
||||
|
||||
`gbrain eval suspected-contradictions` is a complementary measurement
|
||||
instrument: it samples retrieval results for unmarked semantic
|
||||
@@ -435,20 +438,19 @@ commands per high-severity finding.
|
||||
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
|
||||
decision criteria gated on Wilson CI lower-bound.
|
||||
|
||||
## v0.40.1.0 Track D — Eval infrastructure
|
||||
## Eval infrastructure: by-type breakdowns, the hermetic gate, batch scoring
|
||||
|
||||
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
|
||||
covers the dev loop that uses them and the gates they enforce.
|
||||
Three further eval surfaces, and the dev loop that uses them.
|
||||
|
||||
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
|
||||
|
||||
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
|
||||
surfaces it in machine-readable form. Two additive changes:
|
||||
LongMemEval computes per-question-type recall internally, and surfaces it in
|
||||
machine-readable form:
|
||||
|
||||
1. Every per-question JSONL row now includes a `question: string` field so the
|
||||
1. Every per-question JSONL row includes a `question: string` field so the
|
||||
`gbrain eval cross-modal --batch` consumer (below) can read it without
|
||||
joining back against the source dataset.
|
||||
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
|
||||
2. The `--by-type` flag emits a final aggregate line keyed by `question_type`:
|
||||
|
||||
```json
|
||||
{"schema_version": 1, "kind": "by_type_summary",
|
||||
@@ -480,11 +482,11 @@ echo "exit=$?" # 1 if any type fell below 0.80
|
||||
|
||||
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
|
||||
|
||||
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
|
||||
silently regress retrieval." Replaces the original "replay against captured
|
||||
eval_candidates" design (which Codex caught as non-functional in CI — see
|
||||
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
|
||||
real-query version).
|
||||
The structural fix for "PRs touching `src/core/search/` silently regress
|
||||
retrieval." A "replay against captured eval_candidates" design can't work in
|
||||
CI (CI has no captured production queries), so the gate is hermetic; see the
|
||||
`contributor-mode CI capture` TODO in `TODOS.md` for the deferred
|
||||
real-query version.
|
||||
|
||||
How it works:
|
||||
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
|
||||
@@ -499,7 +501,7 @@ How it works:
|
||||
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
|
||||
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
|
||||
|
||||
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
|
||||
#### Refreshing the qrels fixture (the `Why:` discipline)
|
||||
|
||||
When CI fails because a legitimate ranking change moved expected slugs, the
|
||||
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
|
||||
|
||||
@@ -28,10 +28,10 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|---|---|---|
|
||||
| `--limit N` | 100 | Random sample of N takes from the brain. |
|
||||
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
|
||||
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
|
||||
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud rather than silently blowing the budget. |
|
||||
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
|
||||
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
|
||||
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
|
||||
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
|
||||
| `--json` | off | Emit the full receipt to stdout. |
|
||||
|
||||
## Receipt JSON shape (`schema_version: 1`)
|
||||
@@ -50,7 +50,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
},
|
||||
"prompt_sha8": "abcd1234",
|
||||
"models_sha8": "abcd1234",
|
||||
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
|
||||
"models": ["openai:gpt-5.2", "anthropic:claude-opus-4-7", "google:gemini-2.0-flash"],
|
||||
"cycles_run": 3,
|
||||
"successes_per_cycle": [3, 3, 2],
|
||||
"verdict": "pass",
|
||||
@@ -73,8 +73,8 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|
||||
- `schema_version` — locks the contract. Adding optional fields is additive
|
||||
and compatible. Renaming, removing, or changing semantics bumps the version.
|
||||
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
|
||||
(codex review #3). When the rubric definition changes, both fields update,
|
||||
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch.
|
||||
When the rubric definition changes, both fields update,
|
||||
and trend mode groups runs accordingly so a stricter rubric doesn't
|
||||
silently look like a quality drop.
|
||||
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
|
||||
@@ -83,7 +83,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
models in `--models` doesn't change the sha (sort is stable).
|
||||
- `successes_per_cycle` — count of contributing models per cycle. A model
|
||||
contributes when (a) its JSON parsed AND (b) every declared rubric dim
|
||||
has a finite score (codex review #5 — missing-dim drops the contribution).
|
||||
has a finite score (a missing dim drops the whole contribution).
|
||||
- `verdict` — `pass` if every dim mean >= 7 AND every dim min across
|
||||
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
|
||||
2/3 models contributed complete scores.
|
||||
@@ -93,11 +93,11 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|
||||
## Receipt persistence
|
||||
|
||||
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
|
||||
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
|
||||
Receipts persist to **`eval_takes_quality_runs`** (the DB is authoritative)
|
||||
AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
|
||||
as a best-effort artifact. The DB row carries the full receipt JSON in the
|
||||
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
|
||||
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
|
||||
can still reconstruct via `loadReceiptFromDb`.
|
||||
|
||||
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
|
||||
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
# BrainBench — cross-harness memory conformance methodology
|
||||
|
||||
BrainBench generalizes gbrain's internal eval surface into a reproducible,
|
||||
cross-harness benchmark for agent memory. It operationalizes the four failure
|
||||
modes of the agent-memory thesis: **know-to-ask** (nobody has a push path),
|
||||
**push precision/recall** (the intrusion budget must be enforced),
|
||||
**write-back fidelity** (memory write is even less solved than read), and
|
||||
**cross-session continuity** (continuity that survives the harness hop).
|
||||
Every subsequent memory PR must move — or hold, with a recorded justification —
|
||||
a BrainBench number to merge.
|
||||
|
||||
Operator quickstart, corpus layout, and fixture-authoring rules live in
|
||||
[`evals/brainbench/README.md`](../../evals/brainbench/README.md). This document
|
||||
is the methodology: what the numbers mean, what they deliberately do not mean,
|
||||
and how the gate governs change.
|
||||
|
||||
## Seam disclosure (read this before comparing rows)
|
||||
|
||||
Every scoreboard row carries a `seam` column:
|
||||
|
||||
| Harness | Seam | What the row actually measures |
|
||||
|---|---|---|
|
||||
| `openclaw` | **production** | The shipped OpenClaw context-engine pipeline, byte-for-byte (`extractCandidates` → `resolveEntitiesToPointers`, 3-pointer budget, prior-context suppression, markdown pointer block). |
|
||||
| `claude-code` | **contract** | gbrain's memory primitives driven through the UserPromptSubmit hook wire contract (`{prompt, session_id, cwd}` in → `{hookSpecificOutput.additionalContext}` out, exported from `src/eval/brainbench/adapters/claude-code.ts`). 2-pointer budget; NO conversation memory — a hook sees only the current prompt, so suppression is off and the re-injection cost is visible as `false_fire_rate`. |
|
||||
| `codex` | **contract** | The fragments model: a static entity-index preamble (computed once, slugs not counted as injections) + at most ONE per-turn fragment. Measures how much push quality degrades when injection is mostly static. |
|
||||
|
||||
**Contract rows do NOT measure third-party harness behavior.** They measure
|
||||
gbrain's primitives under each harness's injection-shape constraints. The rows
|
||||
are comparable because fixtures, brain, and gold are identical — only the seam
|
||||
contract varies. When a real integration lands (the hooks/fragments PR), its
|
||||
adapter swaps transport (exec the real hook) and flips to `production` with
|
||||
continuous numbers. Also not graded, by design: the production orchestrator's
|
||||
config gate, integration heartbeat, and 1500 ms timeout wrapper.
|
||||
|
||||
All three adapters drive ONE shared pipeline (`adapters/shared.ts`) with
|
||||
declarative configs — comparability is structural, not disciplined.
|
||||
|
||||
## Metrics (formulas)
|
||||
|
||||
All micro-averaged per (harness × suite) cell; registered in
|
||||
`src/core/eval/metric-glossary.ts` (plain-English in
|
||||
[`METRIC_GLOSSARY.md`](METRIC_GLOSSARY.md)); JSON output carries one
|
||||
`_meta.metric_glossary` block.
|
||||
|
||||
- `know_to_ask_failure_rate` = |should-retrieve turns where injected ∩ (gold ∪ acceptable) = ∅| / |should-retrieve turns|. Lower better.
|
||||
- `false_fire_rate` = |stay-silent turns with any injection| / |stay-silent turns|. Lower better. Anti-gaming companion: "always inject" cannot win both.
|
||||
- `push_precision` = Σ|injected ∩ (gold ∪ acceptable)| / Σ|injected| over turns with injection. `acceptable_slugs` count for precision, not recall.
|
||||
- `push_recall` = Σ|injected ∩ gold| / Σ|gold| over should-retrieve turns. Pointer budgets cap this by design.
|
||||
- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
|
||||
- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
|
||||
- `continuity_rate` = |decision probes recalled by the reader| / |probes|, per READER harness. The writer fixture's decisions persist through the production write-back pipeline — which is harness-INDEPENDENT in v1 — so each pair preps once and every harness replays the read-only reader against the same persisted state (an ordered writer×reader sweep would rebuild byte-identical brains for identical scores). A probe succeeds via pointer injection or stored-fact keyword lookup. The per-writer axis activates when harness-specific write paths land.
|
||||
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant. Granularity disclosure: detection is slug-keyed, so it catches injection of slugs seeded ONLY in a foreign source; a same-slug cross-source CONTENT leak would require the engine's source-scoped SQL itself to fail, which the engine-layer source-isolation fuzz (gbrain-evals Cat 22) covers directly.
|
||||
- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
|
||||
- `extraction_recall` / `extraction_precision` — `--llm` runs only: the real extractor's output vs gold keyword probes.
|
||||
|
||||
### What know-to-ask deliberately means in v1
|
||||
|
||||
It grades the **deterministic injection decision** — the Reflex pipeline that
|
||||
ships at the seam. The agent never "knows to ask"; the reflex pushes. An
|
||||
agent-LLM-in-the-loop replay (did the *model* issue a retrieval call when the
|
||||
reflex stayed silent?) is **pre-registered as the `--live` extension**:
|
||||
fixture-compatible, seeded, N-repeat methodology — and unimplemented. No LLM
|
||||
grading is faked in v1.
|
||||
|
||||
### Difficulty is stratified on purpose
|
||||
|
||||
Several know-to-ask variants exercise documented v1 reflex limits (lowercase
|
||||
mentions, surname-only references — `src/core/context/entity-salience.ts`).
|
||||
Gold records what SHOULD happen; the committed baseline records what the
|
||||
current system does (`know_to_ask_failure_rate` ≈ 0.15 at v1). The gap is the
|
||||
measured roadmap, not a bug in the bench.
|
||||
|
||||
## Pre-registered expectations (v1, recorded before the first published run)
|
||||
|
||||
1. The production seam (openclaw) leads `push_recall` strictly: 3-pointer > 2-pointer > 1-fragment budgets. *(Observed at landing: 0.81 / 0.65 / 0.45.)*
|
||||
2. The no-suppression contract (claude-code) is the only seam with `false_fire_rate` > 0. *(Observed: 0.02–0.03.)*
|
||||
3. `write_back_fidelity` = 1.0 and `provenance_accuracy` = 1.0 in deterministic mode — the production pipeline must not lose or mis-attribute gold facts it was handed. Anything below 1.0 is a pipeline bug, not benchmark noise.
|
||||
4. `source_isolation_violations` = 0 everywhere.
|
||||
5. `push_precision` = 1.0 at v1 (exact-match resolution arms cannot inject an irrelevant page on this corpus); expected to dip below 1.0 when fuzzy/semantic resolution lands — that dip is the precision/recall trade made visible.
|
||||
|
||||
## Determinism & statistical posture
|
||||
|
||||
The harness is deterministic end-to-end: regex extraction + SQL resolution
|
||||
(zero LLM, zero embeddings — facts seed with NULL embeddings; keyword/alias
|
||||
arms carry retrieval), seeded PRNG corpus, one in-memory PGLite reset between
|
||||
fixtures. Two runs produce identical metrics, so N-repeat error bars are
|
||||
meaningless here (stddev = 0 by construction, the gbrain-evals "deterministic
|
||||
adapters" convention) and the gate can be exact: **any flipped gold item is a
|
||||
real behavior change.** Bootstrap/CI discipline applies to the future `--live`
|
||||
and `--llm` published runs, which are model-stochastic.
|
||||
|
||||
## Gate governance (decision 4 — why a PR can't self-approve)
|
||||
|
||||
CI (`.github/workflows/test.yml` `brainbench` job, local parity
|
||||
`scripts/ci-brainbench-gate.sh`) fetches the baseline **from main**
|
||||
(`git show origin/master:evals/brainbench/baselines/main.json`) and compares
|
||||
HEAD's fresh run against it:
|
||||
|
||||
- **Same `fixtures_hash`** → count-aware gate: any newly-failed gold item, any
|
||||
adverse gated-metric move, or any isolation violation fails (exit 1).
|
||||
- **Different hash** (the PR changed fixtures) → **corpus-bless mode**: the
|
||||
PR's committed baseline must EXACTLY match HEAD's actual run (the file
|
||||
cannot lie; exit 2 until `--update-baseline` is re-run), and any adverse
|
||||
move vs main's baseline requires a `justification` string in the committed
|
||||
baseline — visible in the PR diff, judged by the reviewer.
|
||||
- `--allow-regression "reason"` is the local one-off escape hatch; the reason
|
||||
is recorded in the run output. It is not available to CI.
|
||||
|
||||
The committed baseline is diff-stable by construction (metrics rounded to 4
|
||||
decimals, keys sorted, receipts excluded; the run CONFIG — holdout/llm/
|
||||
harness/suite sets — is bound into it, and comparisons across mismatched
|
||||
configs are inconclusive). Same-hash hardening: any committed-baseline edit
|
||||
without a fixture change must byte-match the actual run (receipts-backed), a
|
||||
regressing receipts-backed update still needs a `justification`, gold_total
|
||||
may not move at all under an unchanged corpus, and the CI script refuses a
|
||||
working-tree baseline deletion. Holdout fixtures (~15%) are excluded from the
|
||||
gate and scored only in published runs (`--include-holdout`).
|
||||
|
||||
Accepted residuals (review-enforced, by design): a `justification` string is
|
||||
judged by the human reviewer, not parsed; count-preserving corpus dilution
|
||||
(replacing hard fixtures with easy ones at equal gold_total) is visible only
|
||||
in the fixture diff; and the ratchet does not auto-tighten — improvements
|
||||
aren't banked into main's baseline until a PR updates it (a regression back
|
||||
to the stale baseline level passes; periodic re-baselining is the operator's
|
||||
job, filed as a TODO).
|
||||
|
||||
## Gold methodology
|
||||
|
||||
Gold derives from the corpus generator (the same PRNG step that authors a turn
|
||||
authors its annotation, so gold-vs-text drift is structurally impossible for
|
||||
generated fixtures), plus hand-authored spike fixtures that froze the schema.
|
||||
A 10% double-label validation pass (independent agent review of fixture text vs
|
||||
gold, blind to the generator's intent) is run at corpus-change time; its
|
||||
receipt is recorded in the corpus `_ledger.json` and any disagreement is a
|
||||
fixture bug to fix, not a tolerance to average over.
|
||||
|
||||
## Interop
|
||||
|
||||
- **Foreign runners (gbrain-evals):** the subprocess contract is
|
||||
`gbrain eval brainbench --fixtures DIR --gold DIR --json --out FILE`;
|
||||
schemas in `evals/brainbench/schema/`. The sibling gbrain-evals repo wires
|
||||
this as `eval/runner/brainbench-memory.ts` with a published scorecard.
|
||||
- **Memory-verbs conformance kit (Cathedral 1):** conformance scenarios
|
||||
convert to BrainBench fixtures via the published fixture schema
|
||||
(`schema_version` 1) once that wave lands — the conversion path is the
|
||||
schema itself; no bespoke importer is required.
|
||||
- **Naming note:** "BrainBench" historically also names the in-house
|
||||
retrieval corpus in the sibling gbrain-evals repo (the 145-query relational
|
||||
suite, Cat taxonomy) and `test/cathedral-ii-brainbench.test.ts` (v0.20.0
|
||||
code-graph recall pins). This suite — the cross-harness memory conformance
|
||||
bench — is the generalization the name now primarily refers to; the older
|
||||
references stand unchanged.
|
||||
|
||||
## Extends docs/eval-bench.md
|
||||
|
||||
The capture → baseline → replay loop in [`eval-bench.md`](../eval-bench.md)
|
||||
gates *retrieval result sets* at the query level. BrainBench gates the
|
||||
*memory behaviors* above them. The two share the receipts discipline and the
|
||||
.gbrain-evals run ledger (`EvalRunRecord` v3; brainbench records once per
|
||||
sweep under `mode: 'n/a'`).
|
||||
@@ -168,6 +168,96 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli
|
||||
|
||||
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
|
||||
|
||||
## BrainBench — Cross-Harness Memory Conformance
|
||||
|
||||
### Know-to-ask failure rate (BrainBench)
|
||||
|
||||
**Key:** `know_to_ask_failure_rate`
|
||||
|
||||
**Plain English:** Of the conversation turns where memory SHOULD have surfaced something unprompted, the fraction where nothing relevant was injected. This is the thesis failure mode every agent harness shares: the agent can't ask for what it doesn't know it forgot — the memory layer has to volunteer it.
|
||||
|
||||
**Range:** 0..1, LOWER is better. 0.15 means memory stayed silent on 15% of the turns where it had the answer.
|
||||
|
||||
### False-fire rate (BrainBench)
|
||||
|
||||
**Key:** `false_fire_rate`
|
||||
|
||||
**Plain English:** Of the turns where memory should have stayed SILENT, the fraction where it injected anyway. The anti-gaming companion to the know-to-ask rate — "always inject" would ace one and bomb the other. Silence beats noise.
|
||||
|
||||
**Range:** 0..1, LOWER is better.
|
||||
|
||||
### Push precision (BrainBench)
|
||||
|
||||
**Key:** `push_precision`
|
||||
|
||||
**Plain English:** Of everything the memory layer volunteered into context, what fraction was actually relevant to the turn? Micro-averaged over injected pointers, so a 3-pointer turn weighs three times a 1-pointer turn — the way a token budget experiences it.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Push recall (BrainBench)
|
||||
|
||||
**Key:** `push_recall`
|
||||
|
||||
**Plain English:** Of everything that SHOULD have been volunteered (the gold pointers), what fraction actually was? Pointer budgets cap this by design: a seam that may inject only 1 fragment cannot reach full recall on a 3-entity turn — that constraint is what the per-harness rows measure.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Write-back fidelity (BrainBench)
|
||||
|
||||
**Key:** `write_back_fidelity`
|
||||
|
||||
**Plain English:** Of the facts stated in a conversation, what fraction survived the PRODUCTION conversation→memory pipeline (segmentation, insertion, dedup) and are findable afterward with the right entity attached? Measures the write path users actually run, not a test-only insert.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Provenance accuracy (BrainBench)
|
||||
|
||||
**Key:** `provenance_accuracy`
|
||||
|
||||
**Plain English:** Of the facts that survived write-back, what fraction carry correct provenance — the right source tag, session id, and origin page? A fact you can't trace is a fact you can't trust, audit, or expire.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Cross-session continuity rate (BrainBench)
|
||||
|
||||
**Key:** `continuity_rate`
|
||||
|
||||
**Plain English:** A decision is recorded in one session and persisted through the production write path; a different harness asks about it later on the same brain. What fraction of those decision probes were recalled — by pointer injection or stored-fact lookup? This is the continuity-that-survives-the-harness-hop moat, measured.
|
||||
|
||||
**Range:** 0..1, higher is better. Scored per reader harness (the v1 write path is harness-independent, disclosed in docs/eval/BRAINBENCH.md).
|
||||
|
||||
### Source-isolation violations (BrainBench)
|
||||
|
||||
**Key:** `source_isolation_violations`
|
||||
|
||||
**Plain English:** Count of injected pointers that belong to a source other than the active one. Cross-source leakage is gbrain's must-never-violate invariant (a missed source filter is a data leak), so this gates at ZERO — any baseline, any run.
|
||||
|
||||
**Range:** 0..n, count. MUST be 0; any value above 0 fails the gate.
|
||||
|
||||
### Average injected tokens per turn (BrainBench)
|
||||
|
||||
**Key:** `avg_injected_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens of volunteered context per replayed turn (chars/4 heuristic). The intrusion-budget diagnostic: two seams with equal precision can differ 3x in how much context they spend to get it. Reported, not gated, until calibration data exists.
|
||||
|
||||
**Range:** 0..n tokens, judgment call — lower is cheaper, but starving the agent has its own cost. Non-gating.
|
||||
|
||||
### Extraction recall (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_recall`
|
||||
|
||||
**Plain English:** With the real LLM extractor running (instead of the deterministic gold extractor), what fraction of the gold facts did it actually extract and persist? Only scored in --llm runs — the hermetic CI gate never calls a model.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
### Extraction precision (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_precision`
|
||||
|
||||
**Plain English:** Of everything the real LLM extractor persisted, what fraction matches a gold fact? Low precision means the extractor invents or over-extracts — junk memory that pollutes future recall.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Search Mode Evaluation Methodology
|
||||
|
||||
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
|
||||
_How gbrain measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible — pinned datasets, recorded seeds, and the exact run commands below._
|
||||
|
||||
## 1. What this measures and what it doesn't
|
||||
|
||||
@@ -21,14 +21,15 @@ If you want to know how a mode behaves on YOUR brain, run `gbrain search stats -
|
||||
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
|
||||
- **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
|
||||
|
||||
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
No private brain content is used in any reported result. The NDJSON run records under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
|
||||
## 3. Sample selection
|
||||
|
||||
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
|
||||
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode bundle is the only independent variable. A mode Δ therefore measures the joint effect of every knob the bundles differ on — today that's `tokenBudget`, `expansion`, `relationalRetrieval` (the typed-edge fourth recall arm, ON for balanced/tokenmax, OFF for conservative), and `searchLimit`; the canonical diff is `MODE_BUNDLES` in `src/core/search/mode.ts`.
|
||||
- **Cache comparability across upgrades.** The query cache keys on a versioned knobs hash (`KNOBS_HASH_VERSION` in `mode.ts`) that folds in the active knob set + embedding column/provider, so one mode's cached results can't be served to another mode's queries — and a version bump makes prior rows unreachable (one-time miss spike). Cross-run comparisons that straddle a knobs-hash bump see a cold cache on the first re-run.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from a run's `--output` dumps.
|
||||
|
||||
## 4. Run procedure
|
||||
|
||||
@@ -46,14 +47,14 @@ gbrain eval run-all \
|
||||
--limit 500 \
|
||||
--budget-usd-retrieval 5 \
|
||||
--budget-usd-answer 20 \
|
||||
--output docs/eval/results/v0.32.3/
|
||||
--output docs/eval/results/<version>/
|
||||
|
||||
# Render the comparison.
|
||||
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
|
||||
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
|
||||
gbrain eval compare --md > docs/eval/results/<version>/README.md
|
||||
gbrain eval compare --json > docs/eval/results/<version>/comparison.json
|
||||
```
|
||||
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. When a release publishes eval numbers, the `--output` dumps under `docs/eval/results/<version>/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. **No dumps are committed in the repo right now** — reproduce by running the commands above; determinism (§3) means your re-run matches the reported orderings.
|
||||
|
||||
## 5. Threats to validity
|
||||
|
||||
@@ -68,7 +69,7 @@ Honest list. We name what would let a critic dismiss the numbers.
|
||||
|
||||
## 6. Per-question raw outputs
|
||||
|
||||
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
|
||||
Every reported metric is reproducible from the NDJSON dumps a run writes to its `--output` directory (`docs/eval/results/<version>/` when a release publishes numbers; none are committed right now — see §4). The commit SHA in the methodology footer pins the code version.
|
||||
|
||||
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
|
||||
|
||||
|
||||
@@ -12,13 +12,13 @@ surfaces**, and which one you pick depends on the operation.
|
||||
│ gbrain process │
|
||||
│ │
|
||||
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
|
||||
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
|
||||
│ │ (HTTP + OAuth) │ │ admin ops │ │
|
||||
openclaw, fork) ────┼──▶ MCP ops surface │ │ local-only │ │
|
||||
│ │ (HTTP + OAuth) │ │ commands │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ search, query, │ │ sync, embed, │ │
|
||||
│ │ put_page, │ │ dream, doctor,│ │
|
||||
│ │ get_page, │ │ autopilot, │ │
|
||||
│ │ find_experts, │ │ init, secrets │ │
|
||||
│ │ put_page, │ │ extract, │ │
|
||||
│ │ get_page, │ │ dream, │ │
|
||||
│ │ find_experts, │ │ enrich, ... │ │
|
||||
│ │ ... │ │ │ │
|
||||
│ └──────────────────┘ └────────────────┘ │
|
||||
│ ▲ ▲ │
|
||||
@@ -26,7 +26,7 @@ surfaces**, and which one you pick depends on the operation.
|
||||
│ │ │ │
|
||||
│ thin-client OAuth shell-job `inherit:`│
|
||||
│ (preferred for (only path for │
|
||||
│ MCP-equivalent ops) localOnly ops) │
|
||||
│ MCP-equivalent ops) local-only work) │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -45,7 +45,7 @@ the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
|
||||
The host runs gbrain as a long-lived HTTP server:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
The agent registers as an OAuth client (one-time):
|
||||
@@ -75,22 +75,27 @@ commands through the configured remote MCP. The agent can call
|
||||
agent to a specific source within a federated brain.
|
||||
- One audit surface (`mcp_request_log`) covers every op call uniformly.
|
||||
|
||||
## Surface 2 — localOnly admin ops via shell-job `inherit:`
|
||||
## Surface 2 — local-only work via shell-job `inherit:`
|
||||
|
||||
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
|
||||
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
|
||||
list (as of v0.36.5.0) includes:
|
||||
Two mechanisms keep local-only work off the remote surface, and they operate
|
||||
at different layers:
|
||||
|
||||
- `sync` (filesystem walks need local FS access)
|
||||
- `embed` (orchestrates the embed pipeline)
|
||||
- `extract` (walks markdown files)
|
||||
- `dream` (synthesis cycle)
|
||||
- `doctor` (filesystem hygiene checks)
|
||||
- `autopilot` (background daemon orchestration)
|
||||
- `init` (creates `~/.gbrain/`)
|
||||
- `secrets` (config management)
|
||||
- **Op layer:** operations flagged `localOnly: true` in
|
||||
`src/core/operations.ts` are filtered out of the HTTP MCP surface entirely
|
||||
— a remote caller never sees them.
|
||||
- **CLI layer:** on a thin-client install (remote MCP configured, no local
|
||||
engine), commands that require a local engine or the local filesystem are
|
||||
refused at dispatch with a pinpoint hint naming the closest alternative.
|
||||
The authoritative set is `THIN_CLIENT_REFUSED_COMMANDS` in `src/cli.ts` —
|
||||
read it there rather than trusting any list copied into a doc; it covers
|
||||
`sync`, `embed`, `extract`, `dream`, `enrich`, `serve`, `config`, and a
|
||||
couple dozen more.
|
||||
|
||||
For these, the agent cannot route through HTTP MCP. The only path is to run
|
||||
Notable non-members: `doctor` is NOT refused on a thin client — it reroutes
|
||||
to an outbound-HTTP probe set (`src/core/doctor-remote.ts`); `bootstrap` and
|
||||
`hook` are engine-free and work on any install shape.
|
||||
|
||||
For refused commands, the agent cannot route through HTTP MCP. The path is to run
|
||||
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
|
||||
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
|
||||
DLQ / audit trail all come for free.
|
||||
@@ -114,12 +119,11 @@ full validation rules and error catalog.
|
||||
|
||||
### Why this is preferred over writing secrets into `env:` per-job
|
||||
|
||||
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
|
||||
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
|
||||
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
|
||||
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
|
||||
validation. The error message names `inherit: ["database_url"]` as the
|
||||
replacement.
|
||||
- Passing `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` per job would
|
||||
land the URL plaintext in `minion_jobs.data` and the shell-audit JSONL —
|
||||
visible to anyone with brain-DB read access (or a brain dump, or a shared
|
||||
brain via mounts). Pre-enqueue validation rejects it; the error message
|
||||
names `inherit: ["database_url"]` as the replacement.
|
||||
|
||||
### Worker setup (one-time, per host)
|
||||
|
||||
@@ -146,11 +150,11 @@ proxy for worker env.
|
||||
| `get_page` / `list_pages` | HTTP MCP | Same. |
|
||||
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
|
||||
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
|
||||
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
|
||||
| `sync` / `embed` / `extract` | Shell job + `inherit:` | Thin-client refused; needs local engine + FS. |
|
||||
| `dream` | Shell job + `inherit:` | Thin-client refused; synthesis runs on the host. |
|
||||
| `doctor` | Run directly (any install) | Not refused: thin clients get the remote probe set. |
|
||||
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
|
||||
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
|
||||
| `init` / `config` | One-time host setup | Operator action, not agent action. |
|
||||
|
||||
## Recommended patterns
|
||||
|
||||
@@ -166,16 +170,19 @@ proxy for worker env.
|
||||
- **`env:` still works** for non-secret values, or for cases where you
|
||||
WANT the value in the row (e.g. an opaque correlation token your audit
|
||||
flow needs to read back later). The validator doesn't second-guess you.
|
||||
- **Never try to route a `localOnly` op through thin-client MCP.** It will
|
||||
fail with `localOnly op refused in thin-client mode`. Use shell-job +
|
||||
`inherit:` (for secrets) or `env:` (for non-secrets).
|
||||
- **Never try to route a refused command through a thin client.** The CLI
|
||||
refuses it at dispatch with a hint. Use shell-job + `inherit:` (for
|
||||
secrets) or `env:` (for non-secrets) on the host instead.
|
||||
- **Push-based context.** Beyond request/response ops, MCP clients can
|
||||
receive volunteered context via the `volunteer_context` op — see
|
||||
[push-context.md](./push-context.md).
|
||||
|
||||
## Migration: from pre-v0.36.5.0
|
||||
## Migration: from `env:`-passed secrets
|
||||
|
||||
If your agent submits shell jobs that pass secrets via `env:`:
|
||||
|
||||
```jsonc
|
||||
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
|
||||
// Rejected at submit: the URL would persist in minion_jobs.data plaintext.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
@@ -186,7 +193,7 @@ If your agent submits shell jobs that pass secrets via `env:`:
|
||||
Switch to (recommended):
|
||||
|
||||
```jsonc
|
||||
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
|
||||
// Name in row, value resolved at child-spawn from worker config.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
# GBrain Bootstrap — your harness as your agent
|
||||
|
||||
`gbrain bootstrap` turns a Claude Code or Codex session into a persistent personal
|
||||
agent: identity files rendered from your own answers, a local PGLite brain,
|
||||
per-turn context, session-triggered schedules, and a private GitHub repo as the
|
||||
agent's durable, portable body. This guide is the full contract — what gets
|
||||
installed, what runs when, what it can and cannot do, and how to undo all of it.
|
||||
|
||||
Normative design docs: [AGENT_BOOTSTRAP_DESIGN.md](../designs/AGENT_BOOTSTRAP_DESIGN.md)
|
||||
(scope) and [AGENT_BOOTSTRAP_PLAN.md](../designs/AGENT_BOOTSTRAP_PLAN.md)
|
||||
(implementation). The paste block lives in the README; the runbook your agent
|
||||
follows is `BOOTSTRAP_FOR_AGENTS.md` at the repo root, fetched at the
|
||||
`latest-stable` ref.
|
||||
|
||||
## What gets installed, exactly
|
||||
|
||||
| Piece | Where | Runs when |
|
||||
|---|---|---|
|
||||
| Identity files (SOUL/USER/MEMORY/AGENTS/CLAUDE/HEARTBEAT/ACCESS_POLICY/GITHUB) | your workspace folder | loaded at session start |
|
||||
| `agent.json` manifest + `brain/`, `memory/`, `skills/`, `state/` | workspace | — |
|
||||
| Local brain (PGLite) | `~/.gbrain/` (never in the repo) | while a session's MCP serve is open |
|
||||
| MCP registration (`gbrain serve`) | project scope by default | spawned by your harness per session |
|
||||
| Hooks (Claude Code, ON by default) | `.claude/settings.local.json` (gitignored) | each prompt; fail-open; `--no-hooks` opts out at install, `GBRAIN_HOOKS=0` disables at runtime |
|
||||
| Session persistence | SessionEnd hook → scan-gated commit+push | at session end |
|
||||
| Optional 15-min push job | launchd/cron (consent-gated) | while logged in |
|
||||
| Private GitHub repo | your account, created by `bootstrap repo` | privacy verified via API |
|
||||
| Machine receipt | `~/.gbrain/bootstrap/receipt.json` | uninstall is keyed to it |
|
||||
|
||||
**What does NOT run:** anything while the harness is closed. Session-triggered
|
||||
schedules fire at turn/session boundaries only. True 24/7 operation is what a
|
||||
hosted brain provides — this is the honest desktop contract.
|
||||
|
||||
## The awake-when-you-are contract
|
||||
|
||||
Your agent is awake when your harness is. Laptop asleep = agent asleep. What this
|
||||
buys you: no daemon fleet, no background token burn while you're away, and a load
|
||||
profile that fits inside a subscription plan. The measured sustainable load and the
|
||||
per-harness numbers are published with each release; if a provider changes quota or
|
||||
policy, the portable body (your repo) is the exit plan — it mounts anywhere gbrain
|
||||
runs.
|
||||
|
||||
## Keyless mode
|
||||
|
||||
With zero API keys, everything works: the agent authors memory explicitly through
|
||||
the brain's write tools (`put_page`, timeline entries, `## Facts` fences — your
|
||||
harness's model is the LLM, already paid for), and search runs keyword-only
|
||||
(BM25). `bootstrap verify` prints the capability report honestly. One optional key
|
||||
(OpenAI, Anthropic, or Voyage) unlocks semantic search and automatic fact
|
||||
extraction; the key goes to the 0600 config file, never into the repo or the
|
||||
interview answers. API spend is metered separately from your subscription and is
|
||||
zero in keyless mode; with a key, the standard spend gates apply
|
||||
([spend-controls](../operations/spend-controls.md)).
|
||||
|
||||
## Security posture
|
||||
|
||||
- **Supply chain:** the paste block and install command pin the `latest-stable`
|
||||
ref — a maintainer-controlled tag advanced only after a release fully publishes.
|
||||
The runbook carries a version stamp; `bootstrap status` warns on skew. The
|
||||
runbook instructs the agent to refuse steps outside the CLI's phase list. bun
|
||||
installs via package manager or checksum-verified download.
|
||||
- **Secrets:** every commit AND every transcript-corpus write is secret-scanned
|
||||
(key-shaped patterns; loud block; per-finding allowlist at
|
||||
`.gbrain-scan-allow`). A deny-glob backstop refuses tracked `*.pglite`/`.env*`
|
||||
files even if `.gitignore` is damaged. Push refuses public remotes and
|
||||
unverifiable visibility.
|
||||
- **Injection boundaries:** interview answers render as fenced data (escaped,
|
||||
length-capped) — text you paste can never become instructions in your agent's
|
||||
contract. Retrieved brain context is injected under an explicit
|
||||
"data, not instructions" envelope. Facts visible to the harness respect the
|
||||
brain's visibility tiers.
|
||||
- **Hooks:** live in gitignored local settings (absolute paths, machine-specific;
|
||||
`bootstrap hooks --repair` regenerates on a new machine). Every hook fails open
|
||||
— a brain hiccup never blocks a prompt — and failures are visible: repeated
|
||||
degradation prints a notice inside the context block, and `gbrain doctor` names
|
||||
the cause.
|
||||
- **Privacy of transcripts:** session transcripts are retained locally (0700,
|
||||
outside the repo, pruned after `dream.synthesize.corpus_retention_days`, default
|
||||
30 — set it in the config file, `~/.gbrain/config.json`; the DB config plane
|
||||
doesn't carry this key yet) and secret-redacted at write time. They never enter the repo. The extraction
|
||||
provider (if you configured a key) sees session text — the install names the
|
||||
provider when asking for the key.
|
||||
|
||||
## Honest forget semantics
|
||||
|
||||
The repo is git history — append-only. Deleting a line removes it from the working
|
||||
tree, not from history. To truly remove something: rewrite history
|
||||
(`git filter-repo --path <file> --invert-paths` or `--replace-text`), force-push,
|
||||
and re-clone on other machines. `MEMORY.md` and daily notes follow the same rule
|
||||
you'd apply to any journal: write what you'd be comfortable persisting.
|
||||
|
||||
## Degradation matrix
|
||||
|
||||
| You declined / lack | What still works | What you lose |
|
||||
|---|---|---|
|
||||
| API keys | everything (keyless mode) | semantic search, auto-extraction |
|
||||
| GitHub / `gh` | full local agent | off-machine durability (repo re-runnable later) |
|
||||
| Hooks (Claude Code) | pull protocol via AGENTS.md gates | automatic per-turn context + session-end persistence |
|
||||
| Codex (no hook system) | pull protocol + MCP tools | per-turn push (stated plainly; not oversold) |
|
||||
| Second simultaneous session | first session unaffected | second session's brain tools fail politely (one live serve per brain — v1 contract) |
|
||||
|
||||
## Multi-device
|
||||
|
||||
Clone your agent repo on machine two and run `gbrain bootstrap attach` — it
|
||||
validates the manifest, wires this machine (source registration, hooks repair,
|
||||
MCP), and verifies. The brain database is derived state, rebuilt from `brain/` +
|
||||
re-ingestion; hot facts extracted only on machine one arrive via the repo's pages
|
||||
and fences. Simultaneous editing from two machines is ordinary git conflict
|
||||
territory — `sources push` pulls divergence-safely (commit first, rebase pull,
|
||||
loud on conflicts).
|
||||
|
||||
## Uninstall
|
||||
|
||||
`gbrain bootstrap uninstall` removes exactly what this machine's install receipt
|
||||
records: hook wiring, MCP registrations (surgically — foreign servers and hooks
|
||||
survive), and bootstrap-created state. Your repo is never touched — the body
|
||||
remains yours. The brain database is KEPT by default; `--delete-brain` is offered
|
||||
only when bootstrap created the brain, offers a facts export first, and enumerates
|
||||
what it is about to remove. It refuses to run while a session's serve is live.
|
||||
|
||||
## If something seems broken
|
||||
|
||||
One command: `gbrain doctor`. It covers hook health, push staleness, serve/lock
|
||||
collisions, schema state, and prints fixes. `gbrain bootstrap status --json` emits
|
||||
a support blob (versions, harness, last verify/push, hook failure rate) your agent
|
||||
can relay verbatim when you report a problem.
|
||||
|
||||
## Real-agent e2e
|
||||
|
||||
Most bootstrap tests drive the dispatcher with PATH-shimmed `claude`/`codex`
|
||||
recorders — fast, hermetic, no API cost. Two additional "door" tests drive the
|
||||
ACTUAL binaries end to end so we catch real-world drift (a `codex mcp add` flag
|
||||
that changed shape, a harness that stopped calling our MCP server):
|
||||
|
||||
- `test/e2e/bootstrap-real-claude.serial.test.ts` — real `claude -p` over MCP.
|
||||
- `test/e2e/bootstrap-real-codex.serial.test.ts` — real `codex exec`. It runs the
|
||||
keyless-`init` → interview → render → `gbrain bootstrap hooks --harness codex`
|
||||
path (executing the real `codex mcp add` into a hermetic `~/.codex/config.toml`),
|
||||
asserts the rendered `AGENTS.md` carries the Gate-3 brain-first pull protocol
|
||||
(Codex has no hook system, so the pull protocol is its per-turn seam), then
|
||||
spends one live `codex exec` turn to prove real codex → gbrain MCP → brain →
|
||||
a seeded, brain-only fact (falling back to a shell `gbrain query` if headless
|
||||
stdio-MCP is unavailable).
|
||||
|
||||
These pay real API cost and take 30s–2min per turn, so they are NOT in the PR
|
||||
shard. Everything is hermetic (temp `HOME` / `CODEX_HOME` / `CLAUDE_CONFIG_DIR` /
|
||||
`GBRAIN_HOME` per test — the operator's real `~/.claude`, `~/.gbrain`, `~/.codex`
|
||||
are never touched; auth is copied read-only). Each file self-SKIPS via
|
||||
`describe.skipIf` when its binary or auth is absent, so on a machine without the
|
||||
tool it is a clean no-op that never fails. CI wires them into the `real-agent-e2e`
|
||||
job in `.github/workflows/heavy-tests.yml` (nightly + the `real-agent-e2e` /
|
||||
`heavy-tests` label); on a stock runner they self-skip. To actually exercise the
|
||||
binaries you need a runner with authed `claude`/`codex` and the provider creds
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`) exported.
|
||||
|
||||
Run locally (where both are installed + authed):
|
||||
|
||||
```bash
|
||||
bun test test/e2e/bootstrap-real-codex.serial.test.ts
|
||||
```
|
||||
@@ -27,6 +27,7 @@ READ: check brain FIRST (before responding)
|
||||
│ → gbrain search "{entity name}"
|
||||
│ → gbrain get {slug} (if you know it)
|
||||
│ → gbrain query "what do we know about {topic}"
|
||||
│ → full protocol: brain-first-lookup.md
|
||||
│
|
||||
▼
|
||||
RESPOND with brain context (every answer is better with context)
|
||||
@@ -100,12 +101,14 @@ on_message(text):
|
||||
Write immediately after the conversation, while the context is fresh.
|
||||
|
||||
3. **Sync after every write batch.** Without sync, the brain search index is
|
||||
stale. The next query won't find what you just wrote.
|
||||
stale. The next query won't find what you just wrote. On installs set up
|
||||
via `gbrain bootstrap`, per-turn context injection and session-end
|
||||
persistence hooks automate parts of this loop — see
|
||||
[bootstrap.md](bootstrap.md) and [push-context.md](push-context.md).
|
||||
|
||||
4. **External APIs are fallback, not primary.** `gbrain search` before
|
||||
Brave Search. `gbrain get` before Crustdata. The brain has relationship
|
||||
history, your own assessments, meeting transcripts, cross-references.
|
||||
No external API can provide that.
|
||||
4. **External APIs are fallback, not primary.** `gbrain search` before any
|
||||
web or enrichment API. The full brain-before-external protocol (and why)
|
||||
lives in [brain-first-lookup.md](brain-first-lookup.md).
|
||||
|
||||
## How to Verify It Works
|
||||
|
||||
|
||||
@@ -51,22 +51,25 @@ The brain has context no external API can provide:
|
||||
- Timeline (what changed recently, what's trending)
|
||||
|
||||
A LinkedIn scrape gives you their job title. The brain gives you: "co-founded
|
||||
Brex, you had coffee with him 3 times, last discussed the payments infrastructure
|
||||
thesis, he's interested in your take on AI agents."
|
||||
widget-co, you had coffee with her 3 times, last discussed the payments
|
||||
infrastructure thesis, she's interested in your take on AI agents."
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Try keyword first, then hybrid.** Keyword search works without embeddings
|
||||
(day one). Hybrid search needs embeddings but finds semantic matches. Try
|
||||
both in sequence.
|
||||
(day one — and it's ALL you get in keyless mode, see
|
||||
[bootstrap.md](bootstrap.md)). Hybrid search needs embeddings but finds
|
||||
semantic matches. Try both in sequence.
|
||||
|
||||
2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact
|
||||
slug doesn't exist, it suggests alternatives. Use this for name variants
|
||||
("Pedro" → "pedro-franceschi").
|
||||
2. **Fuzzy slug matching is opt-in.** Pass `--fuzzy` (the `fuzzy: true` param
|
||||
on `get_page`) and a near-miss slug resolves to the unique candidate, or
|
||||
returns an `ambiguous_slug` error listing the candidates. WITHOUT the flag
|
||||
a miss just throws `page_not_found` (with a hint to retry with
|
||||
`fuzzy: true`). Use it for name variants ("Alice" → "alice-example").
|
||||
|
||||
3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?"
|
||||
should check the brain first. The brain might have it, and the lookup adds
|
||||
no latency (< 100ms for keyword search).
|
||||
should check the brain first. The brain might have it, and a keyword
|
||||
lookup is fast enough to be effectively free.
|
||||
|
||||
4. **Load compiled truth + recent timeline.** The compiled truth gives you the
|
||||
state of play in 30 seconds. The timeline gives you what changed recently.
|
||||
|
||||
@@ -17,8 +17,8 @@ on new_information(info):
|
||||
# This is world knowledge -- facts about entities external to the agent
|
||||
gbrain put <slug> --content "..."
|
||||
# Examples:
|
||||
# "Pedro is CEO of Brex" -> gbrain (person page)
|
||||
# "Brex raised Series D at $12B" -> gbrain (company page)
|
||||
# "alice-example is CEO of widget-co" -> gbrain (person page)
|
||||
# "widget-co raised Series D at $12B" -> gbrain (company page)
|
||||
# "Tuesday's meeting covered Q2" -> gbrain (meeting page)
|
||||
# "The meatsuit maintenance tax" -> gbrain (originals page)
|
||||
|
||||
@@ -57,15 +57,16 @@ on user_asks(question):
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world.
|
||||
1. **Don't store people in agent memory.** "alice-example prefers email over Slack" feels like a preference, but it's a fact about Alice -- it goes in GBrain on her page. Agent memory is for the agent's own operational state, not facts about people in the world.
|
||||
2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration.
|
||||
3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory.
|
||||
4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything.
|
||||
On installs set up via `gbrain bootstrap`, "agent memory" has a concrete file form: MEMORY.md and the other identity files in the agent repo (see [bootstrap.md](bootstrap.md)). The routing rule is unchanged -- those files hold operational state and identity, not world knowledge.
|
||||
5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
|
||||
1. Ask the agent "Who is alice-example?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
|
||||
2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state.
|
||||
3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers.
|
||||
4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory).
|
||||
|
||||
@@ -44,12 +44,12 @@ Sharp technical leader. Under-appreciated internally. Watch for signs of burnout
|
||||
Ascending. Likely CTO track if the migration succeeds.
|
||||
|
||||
## Relationship
|
||||
Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis.
|
||||
Met through alice-example. Had coffee 3x. Last: discussed API architecture thesis.
|
||||
|
||||
## Contact
|
||||
sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
|
||||
|
||||
---
|
||||
<!-- timeline -->
|
||||
|
||||
## Timeline
|
||||
|
||||
@@ -58,7 +58,7 @@ sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
|
||||
[Source: Meeting notes, 2026-04-07 2:00 PM PT]
|
||||
- **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops.
|
||||
[Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT]
|
||||
- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background.
|
||||
- **2026-03-15** | First meeting. Intro from alice-example. Strong technical background.
|
||||
[Source: User, direct conversation, 2026-03-15 3:00 PM PT]
|
||||
```
|
||||
|
||||
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
|
||||
page = gbrain get {slug}
|
||||
|
||||
// TIMELINE: always APPEND (never edit existing entries)
|
||||
gbrain add_timeline_entry {slug} {
|
||||
gbrain timeline-add {slug} {
|
||||
date: today,
|
||||
summary: new_info.summary,
|
||||
detail: new_info.detail,
|
||||
@@ -113,9 +113,19 @@ support that claim.
|
||||
truth chunks with higher relevance than timeline chunks. This means the freshest
|
||||
synthesis surfaces first in search results.
|
||||
|
||||
4. **The --- separator matters.** GBrain uses the first standalone `---` after
|
||||
frontmatter to split compiled_truth from timeline. Everything above is compiled
|
||||
truth, everything below is timeline.
|
||||
4. **The timeline sentinel matters — and a bare `---` is NOT one.** GBrain
|
||||
splits compiled_truth from timeline at the first recognized sentinel, in
|
||||
order of precedence:
|
||||
1. `<!-- timeline -->` — preferred; unambiguous, and what GBrain itself
|
||||
emits when it writes a page.
|
||||
2. `--- timeline ---` — decorated separator.
|
||||
3. `---` ONLY when the next non-empty line is `## Timeline` or
|
||||
`## History` (backward-compat for older gbrain-written files).
|
||||
|
||||
A plain `---` line anywhere else is a markdown horizontal rule, not a
|
||||
separator. Author new pages with `<!-- timeline -->` (as in the example
|
||||
above); everything above it is compiled truth, everything below is
|
||||
timeline.
|
||||
|
||||
5. **Don't skip the Assessment section.** The assessment is the value. "Strong
|
||||
technical leader" is something no API can provide. It's YOUR read on this
|
||||
|
||||
@@ -8,6 +8,12 @@ Without this: media links are bookmarks that decay -- you remember watching a vi
|
||||
|
||||
## Implementation
|
||||
|
||||
gbrain's own media surfaces complement this pattern: the bundled
|
||||
`media-ingest` skill (`skills/media-ingest/`) ships the ingestion workflow,
|
||||
and `gbrain files` handles binary/file upload for attachments that should
|
||||
live alongside pages. For meeting recordings specifically, see
|
||||
[meeting-ingestion.md](meeting-ingestion.md).
|
||||
|
||||
```
|
||||
on user_shares_media(url_or_file):
|
||||
|
||||
@@ -46,10 +52,10 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Step 4: Extract and cross-reference entities
|
||||
for person in transcript.mentioned_people:
|
||||
gbrain add_link <slug> <person_slug>
|
||||
gbrain add_link <person_slug> <slug>
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Discussed in {video_title}: {what_was_said}" \
|
||||
gbrain link <slug> <person_slug>
|
||||
gbrain link <person_slug> <slug>
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Discussed in {video_title}: {what_was_said}" \
|
||||
--source "YouTube: {url}"
|
||||
|
||||
# PATTERN 2: Social Media Bundles
|
||||
@@ -80,8 +86,8 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Extract entities and cross-reference
|
||||
for entity in bundle.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# PATTERN 3: PDFs and Documents
|
||||
elif media.type == "pdf" or media.type == "document":
|
||||
@@ -109,8 +115,8 @@ on user_shares_media(url_or_file):
|
||||
"""
|
||||
|
||||
for entity in document.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# Always sync after ingestion
|
||||
gbrain sync
|
||||
@@ -119,7 +125,7 @@ on user_shares_media(url_or_file):
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it.
|
||||
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain.
|
||||
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "The speaker made a specific claim about compute scaling that contradicts what another researcher said in their NeurIPS talk -- see media/youtube/a-researcher-neurips-2025" is useful. The analysis connects the new media to the existing brain.
|
||||
3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page.
|
||||
4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry.
|
||||
5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power.
|
||||
@@ -127,7 +133,7 @@ on user_shares_media(url_or_file):
|
||||
## How to Verify
|
||||
|
||||
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
|
||||
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
|
||||
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
|
||||
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
|
||||
|
||||
@@ -26,6 +26,27 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
|
||||
| Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) |
|
||||
| Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below |
|
||||
|
||||
### Prefer gbrain's native schedulers where they fit
|
||||
|
||||
System cron is the lowest common denominator, but gbrain ships its own
|
||||
scheduling surfaces — reach for these first:
|
||||
|
||||
- **`gbrain dream`** — the shipped nightly maintenance cycle (lint,
|
||||
backlinks, extract, sync, embed, synthesize). Schedule THIS instead of
|
||||
hand-rolling the dream cycle below.
|
||||
- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry,
|
||||
backoff, and an audit trail. See the `minion-orchestrator` skill.
|
||||
- **`gbrain autopilot`** — the long-lived background daemon that runs cycles
|
||||
on its own cadence.
|
||||
- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to
|
||||
manage its harness's scheduler.
|
||||
- **Bootstrap session-triggered schedules** — `gbrain bootstrap` installs
|
||||
HEARTBEAT.md-driven schedules that fire on session activity; see
|
||||
[bootstrap.md](bootstrap.md).
|
||||
|
||||
For scheduling `sync` + `embed --stale` specifically, the home doc is
|
||||
[live-sync.md](live-sync.md).
|
||||
|
||||
## Implementation: Setting Up Cron Jobs
|
||||
|
||||
```bash
|
||||
@@ -50,18 +71,12 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
|
||||
|
||||
### Quiet Hours Gate (MANDATORY)
|
||||
|
||||
Every cron job that sends notifications MUST check quiet hours first.
|
||||
See [Quiet Hours](quiet-hours.md) for the full pattern.
|
||||
|
||||
```bash
|
||||
# In every cron script:
|
||||
if ! bash scripts/quiet-hours-gate.sh; then
|
||||
mkdir -p /tmp/cron-held
|
||||
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
|
||||
exit 0
|
||||
fi
|
||||
# Not quiet hours — send normally
|
||||
```
|
||||
Every cron job that sends notifications MUST check quiet hours first. The
|
||||
gate is a small script YOU create (it doesn't ship with gbrain) and call at
|
||||
the top of every notification-sending cron script; held output goes to a
|
||||
holding directory that the morning briefing drains. See
|
||||
[Quiet Hours](quiet-hours.md) for the gate script and the full pattern —
|
||||
don't copy a snippet from here, that page is the single home.
|
||||
|
||||
### Travel-Aware Timezone Handling
|
||||
|
||||
@@ -88,6 +103,12 @@ morning briefing. Zero config change needed.
|
||||
|
||||
The most important cron job. Runs while you sleep.
|
||||
|
||||
**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle
|
||||
(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule
|
||||
it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is
|
||||
covered. The pseudocode that follows is the harness-side variant for agents
|
||||
that also do LLM-driven entity sweeps and memory consolidation on top.
|
||||
|
||||
### What It Does
|
||||
|
||||
```
|
||||
@@ -150,11 +171,11 @@ echo "Dream cycle starting at $(date)"
|
||||
# Phase 1: Entity sweep (spawn sub-agent)
|
||||
# Read today's conversation logs, extract entities, update brain
|
||||
|
||||
# Phase 2: Citation hygiene
|
||||
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
|
||||
# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize)
|
||||
gbrain dream
|
||||
|
||||
# Phase 3: Embed any stale content
|
||||
gbrain embed --stale
|
||||
# Phase 3: Surface anything the cycle flagged
|
||||
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
|
||||
|
||||
echo "Dream cycle complete at $(date)"
|
||||
```
|
||||
|
||||
@@ -143,4 +143,7 @@ the same pass.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The
|
||||
[email-to-brain recipe](../../recipes/email-to-brain.md) implements this
|
||||
collector pattern end-to-end; [cron-schedule.md](cron-schedule.md) covers
|
||||
scheduling it.*
|
||||
|
||||
@@ -53,6 +53,8 @@ Write extracted content to `brain/diligence/{company}/{doc-name}.md`:
|
||||
**Step 5: Save Raw Files.**
|
||||
Copy original PDFs/files to `brain/diligence/{company}/.raw/`
|
||||
Preserve originals for reference. The diarized version is for search.
|
||||
This is safe by design: `.raw/` directories are excluded from sync, so the
|
||||
originals never enter the search index — only your extracted markdown does.
|
||||
|
||||
**Step 6: Create or Update index.md.**
|
||||
Every diligence directory needs an `index.md`:
|
||||
@@ -103,7 +105,8 @@ cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} in
|
||||
|
||||
**Step 9: Publish (if asked).**
|
||||
When the user wants a shareable brief, create a password-protected
|
||||
published version. Strip internal notes and raw assessment language.
|
||||
published version via the `publish` skill (`skills/publish/`). Strip
|
||||
internal notes and raw assessment language.
|
||||
|
||||
### Quality Bar
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
|
||||
but it is provider-agnostic: any configured `provider:model` works as a
|
||||
target.
|
||||
|
||||
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
|
||||
README reference).
|
||||
Also reachable as `gbrain retrieval-upgrade` — the alias that `gbrain doctor`
|
||||
repair hints and the README point at.
|
||||
|
||||
## Quick start
|
||||
|
||||
|
||||
@@ -8,6 +8,11 @@ Without this: brain pages are thin shells with only what the user manually typed
|
||||
|
||||
## Implementation
|
||||
|
||||
gbrain ships both halves of this: `gbrain enrich` is the batch enrichment
|
||||
primitive (finds thin pages and enriches at scale), and the `enrich` skill
|
||||
(`skills/enrich/`) is the agent-driven page-at-a-time workflow. The pipeline
|
||||
below is the pattern they implement — use it to customize or extend.
|
||||
|
||||
```
|
||||
on enrich(entity, trigger):
|
||||
# trigger: meeting mention, email thread, social interaction, user request
|
||||
@@ -49,28 +54,29 @@ on enrich(entity, trigger):
|
||||
data["contacts"] = google_contacts(entity.email) # Contact data
|
||||
|
||||
# Step 5: Store raw data (auditable, re-processable)
|
||||
gbrain put_raw_data <entity_slug> \
|
||||
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
|
||||
gbrain call put_raw_data \
|
||||
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
|
||||
# Overwrite on re-enrichment, don't append
|
||||
|
||||
# Step 6: Write to brain page
|
||||
if path == "CREATE":
|
||||
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
|
||||
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
|
||||
elif path == "UPDATE":
|
||||
# Append timeline, update compiled truth ONLY if materially new
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
|
||||
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
|
||||
# Flag contradictions -- don't silently resolve them
|
||||
|
||||
# Step 7: Cross-reference the graph
|
||||
gbrain add_link <person_slug> <company_slug> # person -> company
|
||||
gbrain add_link <company_slug> <person_slug> # company -> person
|
||||
gbrain add_link <person_slug> <deal_slug> # person -> deal
|
||||
gbrain link <person_slug> <company_slug> # person -> company
|
||||
gbrain link <company_slug> <person_slug> # company -> person
|
||||
gbrain link <person_slug> <deal_slug> # person -> deal
|
||||
# Every entity page links to every other entity page that references it
|
||||
|
||||
# People page sections (not a LinkedIn profile -- a living portrait):
|
||||
# Executive Summary, State, What They Believe, What They're Building,
|
||||
# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline
|
||||
# People page sections: use the person-page structure from compiled-truth.md
|
||||
# (Executive Summary, State, What They Believe, ... Timeline) -- that doc is
|
||||
# the single home for the section taxonomy. Enrichment can add texture
|
||||
# sections on top (What Motivates Them, Hobby Horses, Open Threads).
|
||||
# Facts are table stakes. TEXTURE is the value.
|
||||
|
||||
# Extract texture, not just facts:
|
||||
@@ -94,10 +100,10 @@ on enrich(entity, trigger):
|
||||
## How to Verify
|
||||
|
||||
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
|
||||
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
|
||||
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Compiled Truth](compiled-truth.md) for the person-page section taxonomy, [Spend Controls](../operations/spend-controls.md) for gbrain's own embedding/LLM spend gates.*
|
||||
|
||||
@@ -7,10 +7,10 @@ Every inbound message gets scanned for original thinking AND entity mentions so
|
||||
## What the User Gets
|
||||
|
||||
Without this: the agent answers questions but forgets everything. You mention
|
||||
Pedro in a meeting, and next week the agent doesn't know who Pedro is.
|
||||
Alice in a meeting, and next week the agent doesn't know who Alice is.
|
||||
|
||||
With this: every person, company, and idea mentioned in conversation gets a
|
||||
brain page. Next time Pedro comes up, the agent already has context. The
|
||||
brain page. Next time Alice comes up, the agent already has context. The
|
||||
brain compounds.
|
||||
|
||||
## Implementation
|
||||
@@ -18,6 +18,11 @@ brain compounds.
|
||||
Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it
|
||||
to finish before responding. It runs in parallel.
|
||||
|
||||
This pattern is harness-side by design, but gbrain ships help on both ends:
|
||||
the `signal-detector` skill (`skills/signal-detector/`) is the bundled
|
||||
version of this detection loop, and `gbrain extract` runs gbrain's own
|
||||
extraction machinery (entities, facts) over already-synced content.
|
||||
|
||||
```
|
||||
on_every_message(message_text, source_context):
|
||||
|
||||
@@ -109,6 +114,7 @@ is_notable(entity):
|
||||
| Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions |
|
||||
| Hot takes with reasoning | Routine operational messages |
|
||||
| Metaphors that reveal new angles | Requests without embedded insight |
|
||||
| Emotional/psychological insights about self or others | |
|
||||
|
||||
### Filing Rules
|
||||
|
||||
@@ -116,11 +122,17 @@ is_notable(entity):
|
||||
|--------|-------------|
|
||||
| User generated the idea | `brain/originals/{slug}.md` |
|
||||
| User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
|
||||
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
|
||||
| World concept someone else coined | `brain/concepts/{slug}.md` |
|
||||
| Product or business idea | `brain/ideas/{slug}.md` |
|
||||
| Person mentioned | `brain/people/{slug}.md` |
|
||||
| Company mentioned | `brain/companies/{slug}.md` |
|
||||
| Media referenced | `brain/media/{type}/{slug}.md` |
|
||||
| Article ABOUT the user | `brain/media/writings/{slug}.md` |
|
||||
|
||||
This table is the single home for the capture/filing taxonomy. Other guides
|
||||
([idea-capture](idea-capture.md) especially) link here rather than carrying
|
||||
their own copy.
|
||||
|
||||
### The Iron Law of Back-Linking
|
||||
|
||||
@@ -128,21 +140,21 @@ Every entity mention MUST create a back-link FROM the entity page TO the
|
||||
source. This is not optional.
|
||||
|
||||
```
|
||||
// When message mentions "Pedro" and creates a meeting page:
|
||||
// When message mentions "Alice" and creates a meeting page:
|
||||
|
||||
// 1. Update the meeting page (normal)
|
||||
brain/meetings/2026-04-10-board-sync.md:
|
||||
- Pedro presented Q1 numbers
|
||||
- Alice presented Q1 numbers
|
||||
|
||||
// 2. ALSO update Pedro's page (back-link)
|
||||
brain/people/pedro-franceschi.md:
|
||||
// 2. ALSO update Alice's page (back-link)
|
||||
brain/people/alice-example.md:
|
||||
## Timeline
|
||||
- **2026-04-10** | Presented Q1 numbers at board sync
|
||||
[Source: User, board meeting, 2026-04-10]
|
||||
```
|
||||
|
||||
Without back-links, you can't traverse the graph. "Show me everything related
|
||||
to Pedro" only works if Pedro's page links back to every mention.
|
||||
to Alice" only works if Alice's page links back to every mention.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
@@ -163,7 +175,7 @@ to Pedro" only works if Pedro's page links back to every mention.
|
||||
|
||||
5. **Dedup before creating.** Always `gbrain search` before creating a page.
|
||||
Variant spellings, nicknames, and company abbreviations cause duplicates.
|
||||
"Pedro Franceschi" and "Pedro" might be the same person.
|
||||
"Alice Example" and "Alice" might be the same person.
|
||||
|
||||
## How to Verify
|
||||
|
||||
@@ -182,7 +194,7 @@ to Pedro" only works if Pedro's page links back to every mention.
|
||||
4. **Send a boring message.** Say "ok sounds good." Verify: nothing was
|
||||
created. The detector should report "No signals detected."
|
||||
|
||||
5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi."
|
||||
5. **Check for duplicates.** Mention "Alice" then later "Alice Example."
|
||||
Verify: one page, not two.
|
||||
|
||||
---
|
||||
|
||||
@@ -8,6 +8,11 @@ Without this: the agent triages email mechanically ("you have 12 unread"), preps
|
||||
|
||||
## Implementation
|
||||
|
||||
Before hand-rolling these: gbrain bundles the morning-briefing half of this
|
||||
pattern as the `briefing` skill (`skills/briefing/`) and the task-prep half
|
||||
as `daily-task-prep` (`skills/daily-task-prep/`). Use the workflows below to
|
||||
extend or customize what those skills already ship.
|
||||
|
||||
```
|
||||
# WORKFLOW 1: Email Triage
|
||||
on email_batch(emails):
|
||||
@@ -53,28 +58,28 @@ on upcoming_meeting(meeting):
|
||||
"last_interaction": page.timeline[0], # most recent
|
||||
"open_threads": page.open_threads,
|
||||
"relationship_temperature": page.relationship,
|
||||
"relevant_deals": gbrain get_links <attendee_slug>,
|
||||
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
|
||||
}
|
||||
else:
|
||||
briefing[attendee] = "No brain page -- consider enriching"
|
||||
|
||||
# Surface: shared history, what to follow up on, what to watch for
|
||||
# "Last time you discussed the Series B timeline. Pedro was concerned
|
||||
# about burn rate. Here's the latest from his company page."
|
||||
# "Last time you discussed the Series B timeline. alice-example was
|
||||
# concerned about burn rate. Here's the latest from her company page."
|
||||
|
||||
# WORKFLOW 3: Post-Inbox Brain Updates
|
||||
on inbox_cleared():
|
||||
for email in processed_emails:
|
||||
if email.contained_new_information:
|
||||
# Update the sender's brain page with new signal
|
||||
gbrain add_timeline_entry <sender_slug> \
|
||||
--entry "Email re: {subject}. Key info: {extracted_signal}" \
|
||||
gbrain timeline-add <sender_slug> {date} \
|
||||
"Email re: {subject}. Key info: {extracted_signal}" \
|
||||
--source "email from {sender} re {subject}, {date}"
|
||||
|
||||
# Update any mentioned entity pages too
|
||||
for entity in email.mentioned_entities:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said_about_them}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said_about_them}" \
|
||||
--source "email from {sender}, {date}"
|
||||
|
||||
# WORKFLOW 4: Scheduling Nudges
|
||||
@@ -93,9 +98,9 @@ on schedule_request(meeting):
|
||||
|
||||
1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical.
|
||||
2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise.
|
||||
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate."
|
||||
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with alice-example -- last time you discussed the Series B, she was concerned about burn rate."
|
||||
4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip.
|
||||
5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
|
||||
5. **Scheduling nudges require timeline data.** "You haven't met with charlie-example in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
+13
-39
@@ -61,14 +61,13 @@ capture_idea(message_text, source_context):
|
||||
|
||||
### The Authorship Test
|
||||
|
||||
| Signal | Destination |
|
||||
|--------|-------------|
|
||||
| User generated the idea | `brain/originals/{slug}.md` |
|
||||
| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
|
||||
| World concept someone else coined | `brain/concepts/{slug}.md` |
|
||||
| Product or business idea | `brain/ideas/{slug}.md` |
|
||||
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
|
||||
| Article ABOUT user | `brain/media/writings/` |
|
||||
Who authored the idea determines where it files: user-generated ideas,
|
||||
syntheses, and ghostwritten work go to `brain/originals/`; borrowed world
|
||||
concepts to `brain/concepts/`; product ideas to `brain/ideas/`; articles
|
||||
ABOUT the user to `brain/media/writings/`. The full filing-rules table (and
|
||||
the what-counts-as-original-thinking criteria) lives in
|
||||
[entity-detection.md](entity-detection.md) — the single home for the capture
|
||||
taxonomy.
|
||||
|
||||
### Capture Standards
|
||||
|
||||
@@ -78,21 +77,6 @@ capture_idea(message_text, source_context):
|
||||
"tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase.
|
||||
The vivid version is the real version.
|
||||
|
||||
**What counts as worth capturing:**
|
||||
- Original observations about how the world works
|
||||
- Novel connections between disparate things
|
||||
- Frameworks and mental models
|
||||
- Pattern recognition moments ("I keep seeing X in every Y")
|
||||
- Hot takes with reasoning behind them
|
||||
- Metaphors that reveal new angles
|
||||
- Emotional/psychological insights about self or others
|
||||
|
||||
**What does NOT count:**
|
||||
- Routine operational messages ("ok", "do it")
|
||||
- Pure questions without embedded observations
|
||||
- Echoing back something the agent said
|
||||
- Acknowledgments and reactions
|
||||
|
||||
### The Depth Test
|
||||
|
||||
**Could someone unfamiliar with the user read this page and understand not
|
||||
@@ -137,21 +121,9 @@ Every original MUST link to:
|
||||
|
||||
### Notability Filtering
|
||||
|
||||
Before creating any entity page, check notability:
|
||||
|
||||
**Create a page for:**
|
||||
- People you know or discuss with specificity
|
||||
- Companies you're evaluating, working with, or investing in
|
||||
- Media you mention with personal reaction
|
||||
- Anyone you've explicitly engaged with
|
||||
|
||||
**Don't create pages for:**
|
||||
- Generic references or passing examples
|
||||
- Low-engagement accounts who mentioned you once
|
||||
- Pure metaphors ("like the Roman Empire...")
|
||||
- One-off encounters with no follow-up
|
||||
|
||||
**Decision:** If notable AND no page exists, create a full page with web
|
||||
Before creating any entity page, check notability — the full create/skip
|
||||
criteria live in [entity-detection.md](entity-detection.md#notability-filtering).
|
||||
The decision rule: if notable AND no page exists, create a FULL page with web
|
||||
search enrichment. No stubs. If you make a page, make it good.
|
||||
|
||||
## Tricky Spots
|
||||
@@ -187,4 +159,6 @@ search enrichment. No stubs. If you make a page, make it good.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The bundled
|
||||
`idea-ingest` skill (`skills/idea-ingest/`) ships this workflow. See also:
|
||||
[Entity Detection](entity-detection.md).*
|
||||
|
||||
@@ -43,7 +43,12 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
|
||||
|
||||
- `gbrain sync --repo <path>` -- one-shot incremental sync. Detects changes via
|
||||
`git diff`, imports only what changed. For small changesets (<= 100 files),
|
||||
embeddings are generated inline during import.
|
||||
embeddings are generated inline during import — unless the inline cost gate
|
||||
intervenes: when the estimated embedding spend crosses the configured floor
|
||||
in a non-interactive session (cron, `--json`), sync auto-defers embeds to a
|
||||
capped `embed-backfill` job instead of spending silently. Either way the
|
||||
chunks get embedded; a deferred run just finishes asynchronously. See
|
||||
[spend controls](../operations/spend-controls.md).
|
||||
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
|
||||
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
|
||||
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
|
||||
@@ -97,15 +102,27 @@ Triggers sync on push events for instant sync (<5s).
|
||||
### What Gets Synced
|
||||
|
||||
Sync only indexes "syncable" markdown files. These are excluded by design:
|
||||
- Hidden paths (`.git/`, `.raw/`, etc.)
|
||||
- The `ops/` directory
|
||||
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`
|
||||
- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees
|
||||
(`node_modules/`, `dist/`, `build/`, `venv/`)
|
||||
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md`
|
||||
|
||||
### Sync is Idempotent
|
||||
Everything else is ordinary synced content — including `ops/` (the bundled
|
||||
daily-task-manager skill files its canonical page under `ops/tasks`).
|
||||
|
||||
### Sync is Idempotent — and Resumable
|
||||
|
||||
Concurrent runs are safe. Two syncs on the same commit no-op because content
|
||||
hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
|
||||
Long syncs also survive being killed: progress checkpoints into the database
|
||||
as files drain, so a killed or aborted run resumes from where it stopped, and
|
||||
the sync bookmark only advances on true completion. A progress-aware stall
|
||||
watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts
|
||||
a run that stops making forward progress and releases the per-source lock so
|
||||
the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence
|
||||
and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env
|
||||
vars — incident-time escape hatches, not everyday knobs.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always chain sync + embed.** Running `gbrain sync` without
|
||||
|
||||
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
|
||||
for person in meeting.attendees + meeting.mentioned_people:
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
# Update their State section if new information surfaced
|
||||
# Update company pages for each person's company if relevant
|
||||
|
||||
for company in meeting.mentioned_companies:
|
||||
gbrain add_timeline_entry <company_slug> \
|
||||
--entry "Discussed in '{meeting.title}': {what_was_said}" \
|
||||
gbrain timeline-add <company_slug> {date} \
|
||||
"Discussed in '{meeting.title}': {what_was_said}" \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
|
||||
# Step 4: Extract action items
|
||||
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 5: Back-link everything (bidirectional graph)
|
||||
for entity in all_entities_mentioned:
|
||||
gbrain add_link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain add_link <entity_slug> <slug> # entity -> meeting
|
||||
gbrain link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain link <entity_slug> <slug> # entity -> meeting
|
||||
|
||||
# Step 6: Sync so new pages are immediately searchable
|
||||
gbrain sync
|
||||
@@ -58,23 +58,34 @@ on new_meeting_transcript(meeting):
|
||||
# Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings
|
||||
# Source: Circleback (https://circleback.ai) or any service with
|
||||
# speaker diarization + API/webhook access
|
||||
|
||||
# Automation: the built-in `extract-timeline-from-meetings` Minion job
|
||||
# automates step 3 (entity timeline propagation) for already-ingested
|
||||
# meeting pages: gbrain jobs submit extract-timeline-from-meetings --follow
|
||||
```
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth.
|
||||
2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation.
|
||||
3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room.
|
||||
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
|
||||
3. **Mentioned people are not just attendees.** If the meeting discussed "Alice's team at widget-co," then Alice's page AND widget-co's page need updates -- even though Alice wasn't in the room.
|
||||
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Alice pushed back on the burn rate, Charlie didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
|
||||
5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always.
|
||||
6. **`--source` on `timeline-add` is the citation text, not source routing.** Because the op declares its own `source` parameter, the CLI binds `--source` to it. To write into a different registered source, use the `.gbrain-source` dotfile or `GBRAIN_SOURCE` env for routing instead.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
|
||||
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
|
||||
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
|
||||
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
|
||||
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
|
||||
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/meeting-ingestion/SKILL.md` — the bundled, agent-executable skill
|
||||
for this workflow (the canonical step-by-step home; this guide is the
|
||||
pattern overview).
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -40,6 +40,10 @@ gbrain jobs supervisor status --json
|
||||
|
||||
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
|
||||
gbrain jobs supervisor stop
|
||||
|
||||
# Optional: cap worker memory in MB (--max-rss). Without the flag the RSS
|
||||
# watchdog is still on, at a RAM-relative auto-sized cap.
|
||||
gbrain jobs supervisor --concurrency 4 --max-rss 4096
|
||||
```
|
||||
|
||||
**Exit codes:**
|
||||
@@ -50,9 +54,11 @@ gbrain jobs supervisor stop
|
||||
| 1 | Max crashes exceeded (worker kept dying) |
|
||||
| 2 | Another supervisor holds the PID lock |
|
||||
| 3 | PID file unwritable (permission / path error) |
|
||||
| 4 | Queue-scoped DB lock lost mid-run (`LOCK_LOST` — exited rather than risk a split-brain) |
|
||||
|
||||
An agent seeing exit=2 can safely treat it as "one is already running";
|
||||
exit=1 should page a human.
|
||||
exit=4 as "restart me — the DB lock refresh failed"; exit=1 should page
|
||||
a human.
|
||||
|
||||
### Lowering scheduling priority (`--nice`)
|
||||
|
||||
@@ -233,7 +239,7 @@ use a dedicated queue name like `nightly-enrich` above.
|
||||
|
||||
## Upgrading from an older deployment
|
||||
|
||||
### From `minion-watchdog.sh` (pre-v0.20)
|
||||
### From `minion-watchdog.sh`
|
||||
|
||||
Earlier versions of this guide shipped a 68-line bash watchdog
|
||||
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
|
||||
@@ -270,10 +276,10 @@ Regardless of which deployment path you're upgrading from:
|
||||
in-flight job landing partial schema.
|
||||
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
|
||||
`gbrain doctor` reports any migration as `partial` or `pending`.
|
||||
3. **If you run shell jobs:** from v0.14 onward, pass
|
||||
`--allow-shell-jobs` to the supervisor (or keep
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
|
||||
need the flag; only the worker does.
|
||||
3. **If you run shell jobs:** pass `--allow-shell-jobs` to the
|
||||
supervisor (or keep `GBRAIN_ALLOW_SHELL_JOBS=1` in
|
||||
`/etc/gbrain.env`). Submitters don't need the flag; only the worker
|
||||
does.
|
||||
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
|
||||
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
|
||||
should show no unexplained growth in `dead` between pre- and
|
||||
@@ -283,29 +289,30 @@ Regardless of which deployment path you're upgrading from:
|
||||
|
||||
### Supabase connection drops
|
||||
|
||||
The worker uses a single Postgres connection. If Supabase drops it
|
||||
(maintenance, connection limits, network blip), lock renewal fails
|
||||
silently. The stall detector then dead-letters the job after
|
||||
`max_stalled` misses.
|
||||
If Supabase drops the worker's Postgres connection (maintenance,
|
||||
connection limits, network blip), this now self-heals under the
|
||||
supervisor: the worker's DB-liveness probe self-exits (`db_dead`) on a
|
||||
dead pool and the supervisor respawns it with a fresh pool, and the
|
||||
supervisor also restarts a worker that stops making progress while
|
||||
claimable work waits. The escalation commands and thresholds live in the
|
||||
[queue operations runbook](queue-operations-runbook.md) — that's the
|
||||
canonical home for wedge recovery.
|
||||
|
||||
**Current defaults that make this worse:**
|
||||
What can still bite: a *brief* blip during a long-running job can make
|
||||
lock renewal miss, and the stall detector dead-letters the job after
|
||||
`max_stalled` misses (schema column default 5; lock duration and stall
|
||||
check interval are both 30 s).
|
||||
|
||||
- `lockDuration: 30000` (30 s) — too short for long jobs during
|
||||
connection blips.
|
||||
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
|
||||
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
|
||||
- `stalledInterval: 30000` (30 s) — checks too aggressively.
|
||||
|
||||
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
|
||||
(since v0.13.1). These write onto the job row at submit time — which is
|
||||
what `handleStalled()` reads — so per-job tuning is the real knob today.
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags.
|
||||
These write onto the job row at submit time — which is what
|
||||
`handleStalled()` reads — so per-job tuning is the real knob.
|
||||
|
||||
### DO NOT pass `maxStalledCount` to `MinionWorker`
|
||||
|
||||
It's a no-op. The stall detector reads the row's `max_stalled` column
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts`.
|
||||
Use `gbrain jobs submit --max-stalled N` per-job instead.
|
||||
|
||||
### Zombie shell children
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Minions fix — repairing a half-migrated install
|
||||
|
||||
> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're
|
||||
> on any recent release, the canonical fix below (`gbrain apply-migrations
|
||||
> --yes`) is all you need; the stopgap sections exist for archaeology.
|
||||
|
||||
**tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially
|
||||
set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs
|
||||
still on `agentTurn`), run:
|
||||
@@ -34,17 +38,16 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`.
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
If the install is half-migrated, you'll see:
|
||||
If the install is half-migrated, you'll see the `minions_migration` check
|
||||
fail:
|
||||
|
||||
```
|
||||
[FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
(Missing `~/.gbrain/preferences.json` on a fresh install is a valid
|
||||
pre-`apply-migrations` state — doctor deliberately does NOT fail on that
|
||||
alone; the partial-migration record is the canonical half-migration signal.)
|
||||
|
||||
For a machine-readable report (cron-friendly):
|
||||
|
||||
|
||||
@@ -129,11 +129,10 @@ JSONL records the same. Pre-enqueue validation rejects the submission if the
|
||||
worker can't resolve the requested key, with a paste-ready
|
||||
`gbrain config set database_url <value>` hint.
|
||||
|
||||
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
|
||||
wrote things like:
|
||||
**Why not just write the URL into `env:` directly?** You *can*:
|
||||
|
||||
```jsonc
|
||||
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
|
||||
// ❌ Works, but plants the secret in the job row. Prefer inherit:.
|
||||
{
|
||||
"cmd": "gbrain stats",
|
||||
"cwd": "/data/gbrain",
|
||||
@@ -141,14 +140,14 @@ wrote things like:
|
||||
}
|
||||
```
|
||||
|
||||
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
|
||||
This plants plaintext secrets in `minion_jobs.data` (DB row) and in the
|
||||
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
|
||||
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
|
||||
forbid that pattern — the validator trusts the agent — but **prefer
|
||||
or a shared brain via the mounts feature) sees the URL. The validator
|
||||
doesn't forbid the pattern — it trusts the agent — but **prefer
|
||||
`inherit:`** for any secret you want kept out of the row. Names land in the
|
||||
row; values resolve at child-spawn from the worker's config.
|
||||
|
||||
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
|
||||
**Scope:** `inherit:` is **free-form**. Pass any snake_case
|
||||
config-key name and the worker resolves the value from `loadConfig()` at
|
||||
child-spawn time:
|
||||
|
||||
|
||||
@@ -59,6 +59,12 @@ streaming progress to stderr. It is idempotent: re-running with the same
|
||||
language produces identical vectors. `--json` prints a machine-readable
|
||||
result envelope but still requires `--yes` (or an interactive confirm).
|
||||
|
||||
No cache purge is needed. The resolved language is part of the query-cache
|
||||
key, so rows written under the previous language are unreachable after the
|
||||
switch — searches read the retokenized index immediately instead of being
|
||||
served pre-switch results for up to `search.cache.ttl_seconds`. Switching
|
||||
back reaches the original rows rather than rebuilding them.
|
||||
|
||||
## Recipe: accent-insensitive Portuguese (`pt_br`)
|
||||
|
||||
Brazilian Portuguese content often mixes accented and unaccented spellings
|
||||
|
||||
@@ -5,6 +5,11 @@ is a `source`: a logical brain-within-the-brain with its own slug
|
||||
namespace, its own sync state, and its own federation policy. The rest
|
||||
of this guide walks the three canonical scenarios.
|
||||
|
||||
(Sources are the *within-one-database* axis. If you want to connect a
|
||||
whole separate database — a team-published brain with its own access
|
||||
policy — that's the *brain* axis: `gbrain mounts add`. See
|
||||
`docs/architecture/brains-and-sources.md` for the two-axis topology.)
|
||||
|
||||
## The three scenarios
|
||||
|
||||
### 1. Unified knowledge recall (wiki + gstack)
|
||||
@@ -66,14 +71,14 @@ gbrain search "tech layoffs" --source yc-media,garrys-list
|
||||
### 3. Mixed (wiki federated + sessions isolated)
|
||||
|
||||
Your main wiki is federated with a few trusted sources. Your session
|
||||
transcripts (coming in v0.18) land in a separate isolated source so
|
||||
they don't dominate every search result.
|
||||
transcripts (`gbrain transcripts` ingests them) land in a separate
|
||||
isolated source so they don't dominate every search result.
|
||||
|
||||
```bash
|
||||
# Federated sources
|
||||
gbrain sources add gstack --path ~/.gstack --federated
|
||||
|
||||
# Isolated source (future v0.18 — sessions use this shape today for ingest)
|
||||
# Isolated source for session transcripts
|
||||
gbrain sources add sessions --path ~/.claude/sessions --no-federated
|
||||
```
|
||||
|
||||
@@ -91,7 +96,7 @@ first):
|
||||
6. The seeded `default` source.
|
||||
|
||||
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
|
||||
the `gstack` source. Outside any registered directory with no env/dotfile
|
||||
set, it writes to the default.
|
||||
|
||||
@@ -104,14 +109,16 @@ Every source row stores `config.federated: boolean` in its JSONB config.
|
||||
| `true` | Source participates in unqualified `gbrain search "X"` results. |
|
||||
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
|
||||
|
||||
The seeded `default` source is `federated=true` so pre-v0.17 brains
|
||||
behave exactly as before — every page appears in search.
|
||||
The seeded `default` source is `federated=true` so single-source brains
|
||||
behave as you'd expect — every page appears in search.
|
||||
|
||||
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
|
||||
|
||||
## Commands
|
||||
|
||||
Full subcommand reference:
|
||||
The most-used subcommands (run `gbrain sources --help` for the full,
|
||||
always-current reference — it also covers `status`, `current`,
|
||||
`set-cr-mode`, and the `push`/`pull` durability surface):
|
||||
|
||||
```
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
|
||||
@@ -119,9 +126,17 @@ gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--
|
||||
--path must be a git repo (or a subdirectory of one) — see
|
||||
"The git requirement for --path sources" below. --force
|
||||
skips that check to register before git-init exists.
|
||||
gbrain sources add <id> --url <git-url> [--pat-file <p>] [--clone-dir <path>] [--no-harden]
|
||||
Clone + register a remote repo in one step; auto-hardens
|
||||
for durability when a PAT is provided (see "Durability" below).
|
||||
gbrain sources list [--json] List all sources with page counts + federation state.
|
||||
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
Cascade-delete a source (pages, chunks, timeline).
|
||||
gbrain sources archive <id> Soft-delete: hide from search, keep data for a TTL
|
||||
grace window. Prefer this over `remove`.
|
||||
gbrain sources restore <id> Un-archive. `gbrain sources archived` lists expiries;
|
||||
`gbrain sources purge` permanently deletes expired archives.
|
||||
gbrain sources remove <id> [--confirm-destructive] [--dry-run]
|
||||
Permanently cascade-delete a source (pages, chunks,
|
||||
timeline). Shows an impact preview first.
|
||||
gbrain sources rename <id> <new-name>
|
||||
Change display name only; id is immutable.
|
||||
gbrain sources default <id> Set the brain-level default.
|
||||
@@ -188,10 +203,10 @@ citations keep working.
|
||||
|
||||
```bash
|
||||
# Pass --source explicitly
|
||||
gbrain put-page topics/ai ... --source wiki
|
||||
gbrain put topics/ai ... --source wiki
|
||||
|
||||
# Or rely on the dotfile / env / CWD match
|
||||
cd ~/.gstack && gbrain put-page plans/multi-repo ...
|
||||
cd ~/.gstack && gbrain put plans/multi-repo ...
|
||||
# → source auto-resolves to gstack
|
||||
```
|
||||
|
||||
@@ -253,8 +268,8 @@ reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
|
||||
|
||||
## Upgrading an existing brain
|
||||
|
||||
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
|
||||
existing pages all move under `source_id='default'`. Behavior is
|
||||
`gbrain upgrade` runs the needed schema migrations automatically. Your
|
||||
existing pages all live under `source_id='default'`. Behavior is
|
||||
unchanged until you add a second source.
|
||||
|
||||
To add one:
|
||||
@@ -266,13 +281,13 @@ cd ~/.gstack && gbrain sources attach gstack && gbrain sync
|
||||
|
||||
Two commands. The existing default source is untouched.
|
||||
|
||||
## Not in v0.18.0
|
||||
## Related features that build on sources
|
||||
|
||||
- Session transcript ingest (`.jsonl`, raised size cap, session
|
||||
PageType) — v0.18.
|
||||
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
|
||||
- ACL enforcement via caller-identity — v0.17.1.
|
||||
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
|
||||
release after the core plumbing stabilizes.
|
||||
|
||||
All of these build on the `sources` primitive shipped here.
|
||||
- **Session transcript ingest** — `gbrain transcripts` (server-private:
|
||||
raw chat exports stay on the host machine).
|
||||
- **Per-source retention** — `gbrain sources archive` / `archived` /
|
||||
`purge` (soft-delete with a TTL grace window).
|
||||
- **One-shot remote bootstrap** — `gbrain sources add <id> --url <git-url>`
|
||||
(clone + register + auto-harden).
|
||||
- **Access control across brains** — the *brain* axis (`gbrain mounts`);
|
||||
see `docs/architecture/brains-and-sources.md`.
|
||||
|
||||
@@ -20,8 +20,8 @@ on every_inbound_message(message):
|
||||
for entity in entities:
|
||||
existing = gbrain search "{entity.name}"
|
||||
if existing:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said}" \
|
||||
--source "User, direct message, {timestamp}"
|
||||
# else: flag for enrichment if important enough
|
||||
|
||||
@@ -43,13 +43,14 @@ on information_needed(topic):
|
||||
# An agent that reaches for the web before checking its own brain
|
||||
# is wasting money and giving worse answers.
|
||||
|
||||
# DISCIPLINE 3: Sync After Every Write (MANDATORY)
|
||||
on brain_write_complete():
|
||||
# DISCIPLINE 3: Sync After Every Repo Write (MANDATORY)
|
||||
on brain_repo_files_changed():
|
||||
gbrain sync
|
||||
# Without this, search results are stale.
|
||||
# The page you just wrote won't appear in gbrain search or gbrain query
|
||||
# until sync runs. Skipping this means the next lookup misses the
|
||||
# most recent data.
|
||||
# `gbrain put` indexes immediately -- pages written through the CLI/MCP
|
||||
# are searchable the moment the command returns. No sync needed there.
|
||||
# But files written DIRECTLY to the brain repo (an editor, a script,
|
||||
# another agent committing markdown) are invisible to search until
|
||||
# `gbrain sync` imports them. If anything touched repo files, sync.
|
||||
|
||||
# DISCIPLINE 4: Daily Heartbeat Check
|
||||
on daily_schedule("09:00"):
|
||||
@@ -62,15 +63,17 @@ on daily_schedule("09:00"):
|
||||
on nightly_schedule("02:00"):
|
||||
# The dream cycle is the most important discipline.
|
||||
# The brain COMPOUNDS overnight.
|
||||
# gbrain ships this as a first-class command: `gbrain dream`.
|
||||
# The pseudocode below is the shape of the work it does.
|
||||
|
||||
# 5a: Entity sweep -- find unlinked mentions
|
||||
pages = gbrain list_pages
|
||||
pages = gbrain list
|
||||
for page in pages:
|
||||
mentions = extract_entity_mentions(page.content)
|
||||
existing_links = gbrain get_links <page.slug>
|
||||
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
|
||||
for mention in mentions:
|
||||
if mention not in existing_links:
|
||||
gbrain add_link <page.slug> <mention_slug> # fix broken graph
|
||||
gbrain link <page.slug> <mention_slug> # fix broken graph
|
||||
|
||||
# 5b: Citation audit -- find facts without sources
|
||||
for page in pages:
|
||||
@@ -80,7 +83,7 @@ on nightly_schedule("02:00"):
|
||||
|
||||
# 5c: Memory consolidation -- update compiled truth from timeline
|
||||
for page in stale_pages(older_than="7d"):
|
||||
timeline = gbrain get_timeline <page.slug>
|
||||
timeline = gbrain timeline <page.slug>
|
||||
if timeline.has_new_entries_since_last_consolidation:
|
||||
# Re-synthesize compiled truth from accumulated timeline
|
||||
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
|
||||
@@ -103,18 +106,18 @@ on nightly_schedule("02:00"):
|
||||
## Tricky Spots
|
||||
|
||||
1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots.
|
||||
2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes.
|
||||
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
|
||||
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context.
|
||||
2. **Skipping Discipline 3 (sync after repo writes) means stale search results.** A file lands in the brain repo, then you search for it -- and get nothing back. The file exists but isn't imported. Always sync after repo-file writes. (`gbrain put` is exempt: it indexes on write.)
|
||||
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Alice yesterday about the board seat" in passing -- that's a timeline entry on Alice's page, a potential update to her State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
|
||||
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Alice Example" returns a LinkedIn profile. The brain returns the full picture including private context.
|
||||
5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
|
||||
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
|
||||
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
|
||||
3. Write a markdown file directly into the brain repo (not via `gbrain put`), run `gbrain sync`, then `gbrain search` for it. Confirm it appears in results (verifies the sync discipline). A `gbrain put` page should appear in search immediately, with no sync.
|
||||
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -47,8 +47,8 @@ on user_message(message):
|
||||
|
||||
# Step 3: Cross-link to everything that shaped the thinking
|
||||
for entity in idea.influences:
|
||||
gbrain add_link originals/{slug} <entity_slug>
|
||||
gbrain add_link <entity_slug> originals/{slug}
|
||||
gbrain link originals/{slug} <entity_slug>
|
||||
gbrain link <entity_slug> originals/{slug}
|
||||
|
||||
# Step 4: Sync
|
||||
gbrain sync
|
||||
@@ -79,7 +79,7 @@ on user_message(message):
|
||||
|
||||
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
|
||||
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
|
||||
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
|
||||
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Plugin authors guide (v0.15)
|
||||
# Plugin authors guide — subagent definitions
|
||||
|
||||
`gbrain` discovers subagent definitions from outside this repo via
|
||||
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
|
||||
@@ -7,6 +7,11 @@ subagents alongside it, drop a plugin directory on that env path.
|
||||
|
||||
This guide is for plugin authors. The CLI user doesn't need to read it.
|
||||
|
||||
> **Two plugin systems.** This doc covers *subagent definitions*
|
||||
> (markdown prompts the `subagent` job handler runs). Custom *job
|
||||
> handlers* (code the Minion worker executes) are a separate system —
|
||||
> see [plugin-handlers.md](plugin-handlers.md).
|
||||
|
||||
## Minimum viable plugin
|
||||
|
||||
```
|
||||
@@ -65,7 +70,7 @@ You control where your plugin lives on disk; `gbrain` doesn't guess.
|
||||
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
|
||||
with a warning naming both sources.
|
||||
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY:
|
||||
|
||||
- You **cannot** declare new tools.
|
||||
- You **cannot** extend the brain tool allow-list.
|
||||
@@ -76,8 +81,8 @@ with a warning naming both sources.
|
||||
your plugin gives you a loud startup error, not a silent "tool never
|
||||
fires" at 3am.
|
||||
|
||||
v0.16+ may open up plugin-declared tools with a separate contract. Don't
|
||||
expect it.
|
||||
Plugin-declared tools would require a new `plugin_version` contract;
|
||||
nothing under `gbrain-plugin-v1` opens that up.
|
||||
|
||||
## `gbrain.plugin.json`
|
||||
|
||||
@@ -85,9 +90,9 @@ expect it.
|
||||
|------------------|--------|----------|--------------------------------------------------------------------|
|
||||
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"`. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in future `gbrain plugin list`. |
|
||||
| `description` | string | no | Informational; appears in load/collision warnings. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
@@ -103,8 +108,7 @@ Recognized frontmatter fields:
|
||||
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
|
||||
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
|
||||
|
||||
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
|
||||
may consume more of them.
|
||||
Unknown frontmatter fields are preserved but ignored by the handler.
|
||||
|
||||
## Caveats that will bite you
|
||||
|
||||
@@ -115,8 +119,8 @@ may consume more of them.
|
||||
|
||||
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
|
||||
worker runs on a different host than the `gbrain agent logs` caller,
|
||||
the CLI won't see heartbeats from that worker. v0.16 will unify this;
|
||||
for now assume worker + CLI share a filesystem.
|
||||
the CLI won't see heartbeats from that worker. Assume worker + CLI
|
||||
share a filesystem.
|
||||
|
||||
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
|
||||
invocation. Tools that gate on `remote=true` (file_upload's strict
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
# Plugin handlers — registering host-specific Minion handlers
|
||||
|
||||
GBrain's Minion worker ships with seven built-in handlers: `sync`,
|
||||
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
|
||||
GBrain's Minion worker ships with a full set of built-in handlers,
|
||||
registered by `registerBuiltinHandlers` in `src/commands/jobs.ts` —
|
||||
that registry is the source of truth. Examples: `sync`, `embed`,
|
||||
`lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`, `shell`,
|
||||
`subagent`, `orphans`, `integrity`, plus dream-cycle phases and other
|
||||
maintenance jobs. Submitting an unknown job name with
|
||||
`gbrain jobs submit <name> --follow` prints the full registered list.
|
||||
These cover every background operation the gbrain CLI itself performs.
|
||||
|
||||
Host platforms (OpenClaw deployments, future hosts) register their own
|
||||
@@ -10,6 +15,11 @@ handlers via a plugin bootstrap that imports
|
||||
code, loaded by the worker, with the same trust model as any other
|
||||
code in the host's repo.
|
||||
|
||||
> **Two plugin systems.** This doc covers *job handlers* (code the Minion
|
||||
> worker runs). Custom *subagent definitions* (markdown prompts loaded via
|
||||
> `GBRAIN_PLUGIN_PATH`) are a separate system — see
|
||||
> [plugin-authors.md](plugin-authors.md).
|
||||
|
||||
## Why code, not data
|
||||
|
||||
An earlier design draft shipped `~/.claude/gbrain-handlers.json` where
|
||||
@@ -64,14 +74,23 @@ auto-loads on startup (configurable via a host-provided entry point).
|
||||
|
||||
## Handler contract
|
||||
|
||||
Every handler receives a `MinionJobContext`:
|
||||
Every handler receives a `MinionJobContext` (canonical definition:
|
||||
`src/core/minions/types.ts`). The load-bearing fields:
|
||||
|
||||
```ts
|
||||
interface MinionJobContext {
|
||||
data: Record<string, unknown>; // job params (whatever the cron submit passed)
|
||||
job: MinionJob; // full job row (id, queue, attempts, etc.)
|
||||
signal: AbortSignal; // set to aborted when the worker is shutting down
|
||||
inbox: MinionInbox; // read messages sent to this job while it runs
|
||||
id: number; // job id
|
||||
name: string; // job type
|
||||
data: Record<string, unknown>; // job params (whatever the cron submit passed)
|
||||
attempts_made: number;
|
||||
signal: AbortSignal; // fires on timeout, cancel, pause, or lock loss
|
||||
shutdownSignal: AbortSignal; // fires only on worker SIGTERM/SIGINT
|
||||
deadlineAtMs: number | null; // wall-clock deadline from timeout_at, if set
|
||||
updateProgress(progress: unknown): Promise<void>;
|
||||
updateTokens(tokens: TokenUpdate): Promise<void>;
|
||||
log(message: string | TranscriptEntry): Promise<void>;
|
||||
isActive(): Promise<boolean>; // is the job lock still held?
|
||||
readInbox(): Promise<InboxMessage[]>; // unread messages sent to this job
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Queue operations runbook
|
||||
|
||||
"My queue looks wedged — what do I run?" The commands below are in the order
|
||||
you probably want them. Shipped with v0.19.1 after a production incident
|
||||
where the queue held for 90+ minutes before the operator noticed.
|
||||
you probably want them. Born from a production incident where the queue held
|
||||
for 90+ minutes before the operator noticed.
|
||||
|
||||
## First signal: jobs aren't running
|
||||
|
||||
@@ -23,16 +23,18 @@ container health), but its DB connection died (common behind a transaction
|
||||
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
|
||||
pile up with **0 active**. Liveness checks all pass; nothing crashes.
|
||||
|
||||
As of v0.42.22.0 this self-heals — you usually won't have to do anything:
|
||||
This self-heals — you usually won't have to do anything:
|
||||
|
||||
- **The worker exits on its own dead pool.** Under a supervisor, the worker's
|
||||
DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the
|
||||
supervisor respawns it with a fresh pool.
|
||||
- **The supervisor restarts a worker that stops making progress.** If a queue
|
||||
has claimable work, **0 live-lock active jobs**, and no completions for 15
|
||||
minutes while the child is alive, the supervisor restarts it (covers stuck
|
||||
handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
|
||||
`--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables).
|
||||
minutes across 3 consecutive health checks while the child is alive, the
|
||||
supervisor restarts it (covers stuck handlers too, not just dead pools).
|
||||
These thresholds are built in — there are no CLI flags to tune them. A
|
||||
restart-loop breaker caps wedge restarts at 3 per 30-minute window, then
|
||||
switches to a one-shot `wedge_restart_loop` alert in the audit log.
|
||||
|
||||
The signal is loud now — check either:
|
||||
|
||||
@@ -45,8 +47,12 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
|
||||
stale completions). Manual fix if you ever need it:
|
||||
|
||||
```bash
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool
|
||||
gbrain jobs retry <id> # dead-lettered jobs
|
||||
# Restart the supervisor with a fresh pool. `start` alone runs in the
|
||||
# FOREGROUND (blocks); use --detach to get your shell back.
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start --detach --json
|
||||
|
||||
# Re-queue any jobs that were dead-lettered during the wedge.
|
||||
gbrain jobs retry <id>
|
||||
```
|
||||
|
||||
## Triage commands
|
||||
@@ -101,9 +107,9 @@ claiming. Start one:
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
## Follow-ups tracked for v0.20+
|
||||
## Related
|
||||
|
||||
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
|
||||
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
|
||||
subcheck both need this).
|
||||
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
|
||||
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
|
||||
exit codes, and per-platform deployment (systemd / Fly / Render).
|
||||
- [Minions shell jobs](minions-shell-jobs.md) — the `shell` job type's
|
||||
security model and error table.
|
||||
|
||||
@@ -113,6 +113,22 @@ fi
|
||||
send_notification "$OUTPUT"
|
||||
```
|
||||
|
||||
### GBrain-native hooks
|
||||
|
||||
Two places gbrain already understands quiet hours natively — use these
|
||||
before rolling your own gate for the same job:
|
||||
|
||||
- **Self-upgrade** — `auto` mode only applies upgrades during quiet hours,
|
||||
configured via `gbrain config set self_upgrade.quiet_hours
|
||||
'{"start":23,"end":8,"tz":"US/Pacific"}'`. See
|
||||
[upgrades-auto-update.md](upgrades-auto-update.md).
|
||||
- **Cron prompts** — schedule-driven notification jobs should carry the
|
||||
gate described in this doc; [cron-schedule.md](cron-schedule.md) covers
|
||||
the scheduling side.
|
||||
|
||||
The shell pattern below is for everything else: your own cron jobs,
|
||||
collectors, and notification paths that gbrain doesn't gate for you.
|
||||
|
||||
### Configurable Hours
|
||||
|
||||
Some users want different quiet hours. Store the config:
|
||||
@@ -140,7 +156,11 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring)
|
||||
skill reads and clears the held directory. Orphaned held files mean the
|
||||
pickup integration is broken.
|
||||
|
||||
3. **Timezone auto-detection is fragile.** Calendar-based timezone detection
|
||||
3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a
|
||||
held message must not be lost across a restart, use a durable held
|
||||
directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`.
|
||||
|
||||
4. **Timezone auto-detection is fragile.** Calendar-based timezone detection
|
||||
relies on the user having airline/hotel events with location data. If the
|
||||
user books travel without calendar entries, the system won't detect the
|
||||
move. Fall back to activity-hour analysis (responding at 3 AM PT = probably
|
||||
|
||||
@@ -9,8 +9,9 @@ Separate agent behavior (replaceable) from world knowledge (permanent) into two
|
||||
Without this: agent config and world knowledge are mixed together. Switch agents
|
||||
and you lose your knowledge. Switch knowledge tools and you lose your agent setup.
|
||||
|
||||
With this: your brain (14,700+ files of people, companies, meetings, ideas)
|
||||
survives any agent swap. Your agent config survives any knowledge tool swap.
|
||||
With this: your brain (thousands of files of people, companies, meetings,
|
||||
ideas) survives any agent swap. Your agent config survives any knowledge
|
||||
tool swap.
|
||||
|
||||
## Implementation
|
||||
|
||||
@@ -119,7 +120,15 @@ without losing your agent setup.
|
||||
notes). The agent repo contains operational config. Different access controls.
|
||||
|
||||
**GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep
|
||||
the search index current. The agent repo is never indexed by GBrain.
|
||||
the search index current. The agent repo is not indexed by default.
|
||||
|
||||
**Multi-source nuance.** With multi-source brains
|
||||
([multi-source-brains.md](multi-source-brains.md)), "the brain repo" means
|
||||
"each registered source." You CAN deliberately register a non-brain repo
|
||||
(e.g. `~/.gstack`) as its own isolated or federated source — that's a
|
||||
conscious registration with its own slug namespace, not a violation of the
|
||||
boundary. The rule below is about *unregistered, accidental* indexing of
|
||||
agent config.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
@@ -133,9 +142,12 @@ the search index current. The agent repo is never indexed by GBrain.
|
||||
belongs in the brain. Agent configs, skills, cron jobs, and operational
|
||||
state are replaceable. People, companies, ideas, and meetings are not.
|
||||
|
||||
3. **Don't index the agent repo.** GBrain indexes the brain repo only.
|
||||
Running `gbrain sync` against the agent repo pollutes search results
|
||||
with operational config instead of world knowledge.
|
||||
3. **Don't casually index the agent repo.** Running `gbrain sync` against
|
||||
the agent repo pollutes search results with operational config instead
|
||||
of world knowledge. (Registering it deliberately as an isolated source
|
||||
is different — see the multi-source nuance above. Pin the working
|
||||
directory to the right source with a `.gbrain-source` dotfile via
|
||||
`gbrain sources attach <id>`.)
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
+43
-20
@@ -34,35 +34,35 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
|
||||
|
||||
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
|
||||
|
||||
## v0.26.7 — auto-RLS event trigger and one-time backfill
|
||||
## Auto-RLS: the event trigger and the one-time backfill
|
||||
|
||||
Starting in v0.26.7 (migration v35), gbrain ships two changes that close the
|
||||
gap where a table could exist in your `public` schema without RLS for any
|
||||
amount of time at all.
|
||||
gbrain ships two mechanisms (schema migration v35) that close the gap where a
|
||||
table could exist in your `public` schema without RLS for any amount of time
|
||||
at all.
|
||||
|
||||
**1. The event trigger.** A Postgres DDL event trigger named
|
||||
`auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY`
|
||||
on every newly created `public.*` table. It covers `CREATE TABLE`,
|
||||
`CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres
|
||||
reports as a table-creation command. Tables created by gbrain itself, by
|
||||
your other apps sharing the same Supabase project (Baku, Hermes, anything),
|
||||
or by a human running raw SQL all get RLS enabled the moment they exist.
|
||||
any other app sharing the same Supabase project, or by a human running raw
|
||||
SQL all get RLS enabled the moment they exist.
|
||||
Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly
|
||||
ignored — Supabase manages those, and we should not touch them.
|
||||
|
||||
**2. The one-time backfill.** When you upgrade to v0.26.7, the migration
|
||||
**2. The one-time backfill.** The first upgrade that applies migration v35
|
||||
walks every existing `public.*` base table whose RLS is off and whose comment
|
||||
doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS
|
||||
on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op
|
||||
on every brain.
|
||||
|
||||
### Breaking change: read this before upgrading
|
||||
### Read this before upgrading a pre-auto-RLS brain
|
||||
|
||||
If you have public tables that are intentionally RLS-off and you want them
|
||||
to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before**
|
||||
running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public
|
||||
table that doesn't carry the exact comment contract documented below. There
|
||||
is no `--dry-run` flag on the migration.
|
||||
the upgrade that applies migration v35. The backfill flips RLS on for any
|
||||
public table that doesn't carry the exact comment contract documented below.
|
||||
There is no `--dry-run` flag on the migration.
|
||||
|
||||
The minimum cost of getting this wrong is one round-trip: the operator runs
|
||||
the SQL to enable RLS on a table that should have been exempt, then
|
||||
@@ -71,7 +71,7 @@ prevent a re-flip on a later doctor run. No data is lost.
|
||||
|
||||
### Cross-app implications
|
||||
|
||||
If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates
|
||||
If a non-gbrain app (a side project, a script you wrote, anything) creates
|
||||
tables in the same Supabase project, the trigger will enable RLS on those
|
||||
tables too. Two ways to handle that:
|
||||
|
||||
@@ -90,17 +90,40 @@ ship a policy.
|
||||
|
||||
### What if the trigger gets dropped?
|
||||
|
||||
`gbrain doctor` includes a new `rls_event_trigger` check that verifies the
|
||||
`gbrain doctor` includes an `rls_event_trigger` check that verifies the
|
||||
trigger is installed and enabled. If you drop it manually for any reason
|
||||
(debugging, migration testing, anything), doctor warns and gives you the
|
||||
recovery command:
|
||||
(debugging, migration testing, anything), doctor warns and points you here.
|
||||
|
||||
```
|
||||
gbrain apply-migrations --force-retry 35
|
||||
Recreate it by re-running the trigger DDL from migration v35 — idempotent
|
||||
(`CREATE OR REPLACE` + `DROP EVENT TRIGGER IF EXISTS`), safe to paste into
|
||||
psql as a BYPASSRLS role (e.g. `postgres`):
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION auto_enable_rls()
|
||||
RETURNS event_trigger AS $$
|
||||
DECLARE
|
||||
obj record;
|
||||
BEGIN
|
||||
FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands()
|
||||
WHERE object_type = 'table'
|
||||
AND schema_name = 'public'
|
||||
LOOP
|
||||
EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', obj.object_identity);
|
||||
END LOOP;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP EVENT TRIGGER IF EXISTS auto_rls_on_create_table;
|
||||
CREATE EVENT TRIGGER auto_rls_on_create_table
|
||||
ON ddl_command_end
|
||||
WHEN TAG IN ('CREATE TABLE', 'CREATE TABLE AS', 'SELECT INTO')
|
||||
EXECUTE FUNCTION auto_enable_rls();
|
||||
```
|
||||
|
||||
Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS`
|
||||
and recreates cleanly.
|
||||
(This is the same DDL migration v35 runs — the canonical copy lives in the
|
||||
`MIGRATIONS` array in `src/core/migrate.ts`. There's no CLI shortcut:
|
||||
`gbrain apply-migrations --force-retry` targets the vX.Y.Z orchestrator
|
||||
registry, not numeric schema migrations like v35.)
|
||||
|
||||
### Why no FORCE ROW LEVEL SECURITY?
|
||||
|
||||
@@ -147,7 +170,7 @@ Rules:
|
||||
```sql
|
||||
ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY;
|
||||
COMMENT ON TABLE public.expenses_ramp IS
|
||||
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22';
|
||||
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=you, 2026-04-22';
|
||||
```
|
||||
|
||||
After that, `gbrain doctor` reports:
|
||||
|
||||
@@ -127,9 +127,9 @@ you can use as a reference shape. The skillpack story for distributing
|
||||
your own resolvers across machines is covered in
|
||||
[skillpacks as scaffolding](skillpacks-as-scaffolding.md).
|
||||
|
||||
## The compact list format (v0.41.7.0)
|
||||
## The compact list format
|
||||
|
||||
GBrain's resolver parser used to require markdown tables:
|
||||
GBrain's resolver parser originally required markdown tables:
|
||||
|
||||
```markdown
|
||||
| Trigger | Skill |
|
||||
@@ -146,14 +146,14 @@ format that scales better:
|
||||
- **flight-tracker**: track my flight | flight status | when does my flight land
|
||||
```
|
||||
|
||||
Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a
|
||||
306-skill compact-format resolver, the doctor reported every skill as
|
||||
unreachable: **238 FAIL errors on every doctor run**. The parser was
|
||||
silently treating the compact dialect as zero skills.
|
||||
When `gbrain doctor` only spoke the table dialect, a 306-skill
|
||||
compact-format resolver reported every skill as unreachable: **238 FAIL
|
||||
errors on every doctor run**. The parser was silently treating the compact
|
||||
dialect as zero skills.
|
||||
|
||||
v0.41.7.0 ships dual-format support. The same `parseResolverEntries`
|
||||
function reads both table rows and list rows in the same file, with the
|
||||
v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
|
||||
Today the parser supports both. The same `parseResolverEntries`
|
||||
function reads table rows and list rows in the same file, with the
|
||||
multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
|
||||
`../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor`
|
||||
and the 238 FAILs collapse to 0.
|
||||
|
||||
@@ -267,8 +267,7 @@ I initially converted my resolver from a clean list format to a table
|
||||
format because the validator only spoke tables. That was wrong. When a
|
||||
tool fails against valid data, the right move is to fix the tool, not
|
||||
reshape the data. The list format was correct, compact, readable, easy
|
||||
to maintain. The parser needed to support both shapes. v0.41.7.0 is
|
||||
that fix.
|
||||
to maintain. The parser needed to support both shapes — and now it does.
|
||||
|
||||
The same principle applies everywhere in agent systems. Your SKILL.md is
|
||||
the source of truth. Your AGENTS.md is the source of truth. Your resolver
|
||||
|
||||
+128
-28
@@ -1,54 +1,152 @@
|
||||
# Search Modes
|
||||
|
||||
## Goal
|
||||
Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result.
|
||||
Two decisions shape every gbrain lookup, and this guide covers both:
|
||||
|
||||
## What the User Gets
|
||||
Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls.
|
||||
1. **Which mode bundle** your brain runs — `conservative` / `balanced` /
|
||||
`tokenmax`, the named cost-knob presets that control cache, token budget,
|
||||
query expansion, and result count. This is the config-level decision you
|
||||
make once (at `gbrain init` or via `gbrain config set search.mode`).
|
||||
2. **Which lookup verb** to use per call — `gbrain search` (keyword),
|
||||
`gbrain query` (hybrid), or `gbrain get` (direct). This is the
|
||||
per-lookup decision an agent makes on every question.
|
||||
|
||||
## Implementation
|
||||
## The three mode bundles
|
||||
|
||||
A search mode is a named preset that sets every search-cost knob at once.
|
||||
The bundles are frozen in `src/core/search/mode.ts` (`MODE_BUNDLES`):
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `relationalRetrieval` | false | **true** | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
- **`conservative`** — smallest payloads. Pairs naturally with a cheap
|
||||
downstream model (Haiku-class) or a high query volume.
|
||||
- **`balanced`** — the default and the fallback when no mode is set.
|
||||
- **`tokenmax`** — no token budget, LLM query expansion on, 50 results.
|
||||
Pairs with an expensive downstream model you want fully fed.
|
||||
|
||||
Two of the knobs deserve a sentence:
|
||||
|
||||
- **`expansion`** rewrites your query into multiple variants via a cheap
|
||||
LLM call per search (adds roughly $1.50 per 1K queries) — better recall,
|
||||
small extra cost.
|
||||
- **`relationalRetrieval`** adds a graph-walk recall arm for relational
|
||||
questions ("who invested in X", "what connects A and B"); it's a pure
|
||||
no-op for non-relational queries. The `query` op's `relational` flag
|
||||
forces it on/off per call.
|
||||
|
||||
### Setting and resolving the mode
|
||||
|
||||
```bash
|
||||
gbrain config set search.mode tokenmax
|
||||
```
|
||||
|
||||
Per-knob resolution (highest first):
|
||||
|
||||
per-call SearchOpts → per-key config override (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in bare `hybridSearch`, not just the cached wrapper,
|
||||
so eval replays test the same mode-affected behavior as the production
|
||||
`query` op. The query cache folds the active knobs into its key
|
||||
(`knobs_hash`), so switching modes never serves you a stale result set
|
||||
from a different configuration.
|
||||
|
||||
### Cost intuition
|
||||
|
||||
gbrain's own cost is rounding error; what the mode really controls is how
|
||||
many tokens your *downstream agent* pays to read per query. The
|
||||
corner-to-corner spread is ~25x once you pair mode with downstream model.
|
||||
Rough anchors at 10K queries/month, full payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku-class (\$1/M in) | Sonnet-class (\$3/M in) | Opus-class (\$5/M in) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K tok) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K tok) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K tok) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly with volume. Cache hits cut all numbers ~50%; disciplined
|
||||
prompt caching in the agent loop cuts further. Mismatched pairings waste
|
||||
capacity in both directions — a tokenmax payload overwhelms a cheap model,
|
||||
a conservative payload starves an expensive one. The full methodology and
|
||||
realistic-scale walkthrough live in
|
||||
[`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
### CLI surfaces
|
||||
|
||||
```bash
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
gbrain search diagnose "<query>" --target <slug>
|
||||
# trace where a page surfaces (or fails to)
|
||||
# across the keyword/vector/alias/hybrid layers
|
||||
```
|
||||
|
||||
The mode picker runs inside `gbrain init` (non-TTY auto-selects `balanced`).
|
||||
|
||||
## Choosing a lookup verb (search vs query vs get)
|
||||
|
||||
Independent of which bundle is active, every individual lookup should use
|
||||
the cheapest verb that answers the question.
|
||||
|
||||
```
|
||||
on user_asks_about(topic):
|
||||
# Decision tree: pick the right search mode
|
||||
# Decision tree: pick the right lookup verb
|
||||
|
||||
if know_exact_slug(topic):
|
||||
# MODE 3: Direct get -- instant, no search overhead
|
||||
# Direct get -- instant, no search overhead
|
||||
result = gbrain get <slug>
|
||||
# e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi
|
||||
# e.g., "Tell me about Alice" -> gbrain get alice-example
|
||||
# Returns the FULL page -- compiled truth + timeline
|
||||
|
||||
elif topic.is_exact_name or topic.is_keyword:
|
||||
# MODE 1: Keyword search -- fast, no embeddings needed, day-one ready
|
||||
# MODE 1: Cheap-hybrid search -- vector + keyword + RRF, NO LLM
|
||||
# expansion. Embeds the query when embeddings are configured; the
|
||||
# keyword arm still works day-one without them (keyword-only is
|
||||
# also available via the search.mcp_keyword_only opt-out).
|
||||
results = gbrain search "{name_or_keyword}"
|
||||
# e.g., "Find anything about Series A" -> gbrain search "Series A"
|
||||
# Returns CHUNKS, not full pages
|
||||
|
||||
# IMPORTANT: keyword search returns chunks
|
||||
# IMPORTANT: search returns chunks
|
||||
# If the chunk confirms relevance, THEN load the full page:
|
||||
if chunk.confirms_relevance:
|
||||
full_page = gbrain get <slug_from_chunk>
|
||||
|
||||
elif topic.is_semantic_question:
|
||||
# MODE 2: Hybrid search -- semantic + keyword, needs embeddings
|
||||
elif topic.is_semantic_question or topic.is_concept_or_landscape:
|
||||
# MODE 2: Full hybrid -- adds multi-query LLM expansion on top of
|
||||
# vector + keyword + RRF. Owns concept / landscape / "all-of-X"
|
||||
# questions: expansion recovers synonym- and outcome-phrased
|
||||
# matches a single embedding misses. Costs one LLM expansion call
|
||||
# per query -- worth it for these question shapes.
|
||||
results = gbrain query "{natural language question}"
|
||||
# e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts"
|
||||
# Returns ranked chunks via vector + keyword + RRF
|
||||
# e.g., "all the companies doing offshore wind" -> gbrain query "..."
|
||||
# Returns ranked chunks via vector + keyword + expansion + RRF
|
||||
|
||||
# Same rule: chunks first, then get full page if needed
|
||||
if chunk.confirms_relevance:
|
||||
full_page = gbrain get <slug_from_chunk>
|
||||
|
||||
# Quick reference:
|
||||
# | Mode | Command | Needs Embeddings | Speed | Best For |
|
||||
# |---------|----------------------|------------------|---------|---------------------------------|
|
||||
# | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches |
|
||||
# | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match |
|
||||
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
|
||||
# | Mode | Command | Needs Embeddings | Speed | Best For |
|
||||
# |-------------|----------------------|------------------|---------|-------------------------------------------|
|
||||
# | Cheap-hybrid| gbrain search "term" | Uses if present | Fastest | Known names, exact tokens |
|
||||
# | Full hybrid | gbrain query "..." | Yes | Fast | Concept / landscape / "all-of-X", synonyms |
|
||||
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
|
||||
|
||||
# Progression over time:
|
||||
# Day 1: keyword search (works without embeddings)
|
||||
# After first embed: hybrid search unlocked
|
||||
# Day 1: search (keyword arm works without embeddings)
|
||||
# After first embed: vector arm + full hybrid (query) unlocked
|
||||
# Once you know slugs: direct get for speed
|
||||
|
||||
# Precedence for conflicting information within a page:
|
||||
@@ -58,21 +156,23 @@ on user_asks_about(topic):
|
||||
# 4. External sources (web search, APIs)
|
||||
```
|
||||
|
||||
## Tricky Spots
|
||||
### Tricky Spots
|
||||
|
||||
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
|
||||
2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available.
|
||||
3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug.
|
||||
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page.
|
||||
5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
|
||||
2. **Search works without embeddings.** On day one before any embedding run, `gbrain search` still works (the keyword arm carries it; the vector arm joins once embeddings exist). Don't tell the user "search isn't available yet" -- search is always available.
|
||||
3. **Don't use full hybrid for known names.** `gbrain query "Alice Example"` wastes an LLM expansion call. Use `gbrain search "Alice Example"` or better yet `gbrain get alice-example` if you know the slug.
|
||||
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Alice" -- get the full page.
|
||||
5. **Full hybrid needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
|
||||
6. **A populated `gbrain search` result set is not proof you found everything.** Search runs without query expansion, so synonym- and outcome-phrased matches can be missed even when it returns plenty of hits. For "find every / all / the landscape of" questions, use `gbrain query`; for literal exhaustive enumeration ("list every page of type X"), use `list_pages` pagination. A nonzero count is not a completeness signal.
|
||||
|
||||
## How to Verify
|
||||
### How to Verify
|
||||
|
||||
1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references.
|
||||
1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references.
|
||||
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
|
||||
3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline.
|
||||
3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline.
|
||||
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
|
||||
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
|
||||
6. Run `gbrain search modes` -- confirm the active mode bundle and any per-key overrides are what you expect.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -40,7 +40,9 @@ Show the user the results. Get feedback.
|
||||
- Revise the process based on what you learned.
|
||||
|
||||
**Step 4: Codify into a Skill.**
|
||||
Write the SKILL.md. Either:
|
||||
Write the SKILL.md (`gbrain skillify scaffold <name>` scaffolds the tree for
|
||||
you; `gbrain skillopt` optimizes an existing skill against a benchmark).
|
||||
Either:
|
||||
- **New skill** -- genuinely new capability
|
||||
- **Add to existing skill** -- variation of something that exists (parameterize it)
|
||||
|
||||
@@ -63,7 +65,8 @@ Skills should be **Mutually Exclusive, Collectively Exhaustive**:
|
||||
- Each signal source has exactly ONE owner skill
|
||||
- Two skills creating the same brain page = MECE violation
|
||||
|
||||
**Example ownership (no overlap):**
|
||||
**Example ownership (no overlap — illustrative; your skill roster will
|
||||
differ):**
|
||||
|
||||
| Signal Source | Owner Skill | Creates |
|
||||
|--------------|-------------|---------|
|
||||
|
||||
@@ -65,14 +65,14 @@ For each step:
|
||||
more than 0.05.
|
||||
6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe.
|
||||
|
||||
After each epoch with no improvement: D6 slow-update fires one meta-edit
|
||||
proposal (this lives in v0.42 follow-up; v1 emits the audit event).
|
||||
After each epoch with no improvement: D6 slow-update fires. Today it emits
|
||||
the audit event only; the full meta-edit proposal is a tracked follow-up.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `--benchmark <path>` | `skills/<n>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
|
||||
| `--benchmark <path>` | `skills/<name>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
|
||||
| `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) |
|
||||
| `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) |
|
||||
| `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl |
|
||||
@@ -136,7 +136,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
|
||||
|
||||
- **No benchmark.** Optimizing against guesses is worse than not optimizing.
|
||||
- **Write-flavored skills.** Skills whose job is to `put_page` heavily can't
|
||||
use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up.
|
||||
use the read-only sandbox; mocked-write capture is a tracked follow-up.
|
||||
- **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful
|
||||
validation needs ≥20 tasks total per the paper.
|
||||
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
# Skillpacks as scaffolding, not amber
|
||||
|
||||
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
|
||||
scaffold + reference library. This guide explains the model and the
|
||||
workflow.
|
||||
`gbrain skillpack` is a scaffold + reference library, not a package
|
||||
manager. This guide explains the model and the workflow.
|
||||
|
||||
## Why we changed it
|
||||
## Why it works this way
|
||||
|
||||
Pre-v0.33 (the "amber" model):
|
||||
An earlier design (the "amber" model):
|
||||
|
||||
- `gbrain skillpack install <name>` copied bundled skills into your
|
||||
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
|
||||
@@ -26,7 +25,16 @@ repo. You scaffold once, you own them, you fork and edit freely. When
|
||||
gbrain ships a new version, you ask "what changed?" — the agent reads
|
||||
the diff and decides what (if anything) to integrate.
|
||||
|
||||
## The five commands
|
||||
## The core workflow commands
|
||||
|
||||
The five commands below are the scaffold-and-own workflow. The full
|
||||
`gbrain skillpack` surface is larger (`list`, `diff`, `check`, `search`,
|
||||
`info`, `registry`, `doctor`, `init`, `pack`, `endorse`, …) — run
|
||||
`gbrain skillpack --help` for the always-current list. One worth calling
|
||||
out here: **`gbrain skillpack init-brain-pack <name>`** scaffolds a
|
||||
*brain-resident* pack inside a brain/source repo (`brain_resident: true`
|
||||
plus a machine-parseable README) that connecting harnesses discover on
|
||||
`gbrain sources add`.
|
||||
|
||||
### `gbrain skillpack scaffold <name> [--workspace PATH]`
|
||||
|
||||
@@ -77,15 +85,15 @@ gbrain skillpack reference book-mirror
|
||||
`reference <name> --apply-clean-hunks` is the auto-apply path. It
|
||||
parses the diff between gbrain's bundle and your local copy, applies
|
||||
every hunk whose pre-change context matches uniquely. **Two-way merge
|
||||
limitation**: without scaffold-time base tracking (intentionally
|
||||
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
|
||||
limitation**: without scaffold-time base tracking (intentionally out of
|
||||
scope), this cannot distinguish "gbrain changed X"
|
||||
from "you changed X." Applied hunks align everything to gbrain. Use
|
||||
`--dry-run` first to preview, or run plain `reference` to inspect the
|
||||
diff before letting auto-apply touch anything.
|
||||
|
||||
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
|
||||
|
||||
One-shot conversion for workspaces on the pre-v0.33 managed-block
|
||||
One-shot conversion for workspaces still on the legacy managed-block
|
||||
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
|
||||
markers and the manifest receipt comment from your resolver file.
|
||||
|
||||
@@ -157,7 +165,7 @@ Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
|
||||
frontmatter, and match the user's intent against every skill's
|
||||
`triggers:` array. When a match scores high enough, invoke that skill.
|
||||
|
||||
This replaces the v0.32 model where `gbrain skillpack install` wrote
|
||||
This replaces the legacy model where `gbrain skillpack install` wrote
|
||||
table rows into your `RESOLVER.md`. Rows are gone (or, for users
|
||||
migrating from the old model, preserved transitionally by
|
||||
`migrate-fence` until they run `scrub-legacy-fence-rows`).
|
||||
@@ -173,7 +181,7 @@ If you're a downstream agent author updating to this model:
|
||||
|
||||
## Removing a scaffolded skill
|
||||
|
||||
There's no `gbrain skillpack uninstall` command in v0.33. The files
|
||||
There's no `gbrain skillpack uninstall` command. The files
|
||||
in your `skills/<slug>/` are first-class members of your repo —
|
||||
delete them like any other code:
|
||||
|
||||
@@ -195,16 +203,17 @@ You own the files. There's no manifest to update, no fence to rebuild.
|
||||
## When to use which command (quick decision tree)
|
||||
|
||||
- **New host repo, want a gbrain skill** → `scaffold`
|
||||
- **Shipping a pack from inside a brain/source repo** → `init-brain-pack`
|
||||
- **gbrain shipped a new version, want to see what's changed**
|
||||
→ `reference` (read-only) or `reference --apply-clean-hunks` (auto)
|
||||
- **Upgrading from v0.32 or earlier** → `migrate-fence` (one-shot)
|
||||
- **Upgrading from the legacy managed-block model** → `migrate-fence` (one-shot)
|
||||
- **Cleanup after `migrate-fence`** → `scrub-legacy-fence-rows`
|
||||
- **Lift your fork's skill back into gbrain** → `harvest` + the
|
||||
`skillpack-harvest` editorial skill
|
||||
|
||||
## What about `install` and `uninstall`?
|
||||
|
||||
Both are removed in v0.33. Running either prints an error pointing at
|
||||
the replacement command. No deprecated alias — this is a clean break.
|
||||
Both are removed. Running either prints an error pointing at the
|
||||
replacement command. No deprecated alias — this is a clean break.
|
||||
If you have existing scripts referencing the old names, update them
|
||||
once and move on.
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
Every fact in the brain traces to where it came from -- who said it, in what context, and when.
|
||||
|
||||
## What the User Gets
|
||||
Without this: six months from now, someone reads a brain page and has no idea if "Pedro co-founded Brex" came from Pedro himself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
|
||||
Without this: six months from now, someone reads a brain page and has no idea if "Alice co-founded widget-co" came from Alice herself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
|
||||
|
||||
## Implementation
|
||||
|
||||
@@ -23,11 +23,11 @@ on brain_write(page, fact):
|
||||
# [Source: Crustdata LinkedIn enrichment, 2026-04-07 12:35 PM PT]
|
||||
elif source.type == "social_media":
|
||||
# MUST include full URL -- not just @handle
|
||||
# [Source: X/@pedroh96 tweet, product launch, 2026-04-07](https://x.com/pedroh96/status/...)
|
||||
# [Source: X/@alice_example tweet, product launch, 2026-04-07](https://x.com/alice_example/status/...)
|
||||
elif source.type == "email":
|
||||
# [Source: email from Sarah Chen re Q2 board deck, 2026-04-05 2:30 PM PT]
|
||||
# [Source: email from Alice Example re Q2 board deck, 2026-04-05 2:30 PM PT]
|
||||
elif source.type == "workspace":
|
||||
# [Source: Slack #engineering, Keith re deploy schedule, 2026-04-06 11:45 AM PT]
|
||||
# [Source: Slack #engineering, Charlie re deploy schedule, 2026-04-06 11:45 AM PT]
|
||||
elif source.type == "web":
|
||||
# [Source: Happenstance research, 2026-04-07 12:35 PM PT]
|
||||
elif source.type == "published":
|
||||
@@ -57,7 +57,7 @@ SOURCE_PRIORITY = [
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Compiled truth is NOT exempt from citations.** "Pedro co-founded Brex" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
|
||||
1. **Compiled truth is NOT exempt from citations.** "Alice co-founded widget-co" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
|
||||
2. **Tweet URLs are mandatory.** `[Source: X/@handle tweet, topic, date]` without a URL is a broken citation. Hundreds of brain pages end up with unreachable tweet references when the URL is omitted. Always: `[Source: X/@handle tweet, topic, date](https://x.com/handle/status/ID)`.
|
||||
3. **"User said it" isn't enough.** WHERE, ABOUT WHAT, WHEN. `[Source: User, direct message, 2026-04-07 12:33 PM PT]` -- not just `[Source: User]`.
|
||||
4. **Don't silently resolve conflicts.** When the user says one thing and an API says another, note the contradiction in compiled truth with both citations. Let the reader decide.
|
||||
@@ -71,5 +71,13 @@ SOURCE_PRIORITY = [
|
||||
4. Check timeline entries on 3 random pages. Each entry should have a source citation with date and context.
|
||||
5. Look for a page where the user stated something that contradicts an API result. Confirm the contradiction is noted, not silently resolved.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/_brain-filing-rules.md` — the canonical citation-format rules every
|
||||
brain write follows (this guide is the narrative walkthrough of the same
|
||||
contract).
|
||||
- `skills/citation-fixer/SKILL.md` — audits and repairs existing pages
|
||||
against that format.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -2,19 +2,42 @@
|
||||
|
||||
## Goal
|
||||
|
||||
Route sub-agents to the cheapest model that can do the job, saving 10-40x on costs without sacrificing quality.
|
||||
Route sub-agents to the cheapest model that can do the job, saving large
|
||||
multiples on cost without sacrificing quality.
|
||||
|
||||
## What the User Gets
|
||||
|
||||
Without this: every sub-agent runs on Opus ($15/MTok). Entity detection on
|
||||
every message costs $3-5/day. Research tasks cost $10+ each.
|
||||
Without this: every sub-agent runs on your most expensive model. Entity
|
||||
detection fires on every message at top-tier rates; research tasks cost
|
||||
several dollars each.
|
||||
|
||||
With this: entity detection runs on Sonnet ($3/MTok, 5x cheaper). Research
|
||||
runs on DeepSeek ($0.50/MTok, 30x cheaper). Main session stays on Opus for
|
||||
quality. Total cost drops 70-80%.
|
||||
With this: entity detection runs on a cheap fast model, research execution
|
||||
runs on a budget model, and only planning/synthesis touch the expensive
|
||||
model. Total cost drops 70-80%.
|
||||
|
||||
(Illustrative input-token anchors from gbrain's canonical pricing table,
|
||||
`src/core/model-pricing.ts`: Opus-class $5/MTok, Sonnet-class $3/MTok,
|
||||
Haiku-class $1/MTok. Budget providers run well under $1/MTok. Prices
|
||||
drift — the pricing table is the source of truth, not this doc.)
|
||||
|
||||
## Implementation
|
||||
|
||||
### GBrain's native mechanism: model tiers
|
||||
|
||||
Before hardcoding vendors, use gbrain's tier routing. Every gbrain
|
||||
subagent/LLM call resolves through a named tier
|
||||
(`utility` / `reasoning` / `deep` / `subagent`), and you point each tier
|
||||
at whatever model you want once:
|
||||
|
||||
```bash
|
||||
gbrain config set models.tier.subagent anthropic:claude-haiku-4-5
|
||||
gbrain config set models.tier.deep anthropic:claude-opus-4-7
|
||||
```
|
||||
|
||||
Per-call override: `gbrain agent run --model <provider:model>`. The
|
||||
conventions file `skills/conventions/model-routing.md` is the canonical
|
||||
routing policy; this guide is the cost rationale behind it.
|
||||
|
||||
### Routing Table
|
||||
|
||||
| Task Type | Recommended Model | Why |
|
||||
@@ -44,14 +67,16 @@ on_every_message(text):
|
||||
3. FACTS: New info about existing entities -> update timeline
|
||||
4. CITATIONS: Every fact needs [Source: ...] attribution
|
||||
5. Sync changes to brain repo`,
|
||||
model: "sonnet-class", // fast + cheap
|
||||
model: "sonnet-class", // fast + cheap; haiku-class is cheaper still
|
||||
timeout: 120s
|
||||
})
|
||||
```
|
||||
|
||||
**Why Sonnet-class for detection:** Entity detection is pattern matching, not
|
||||
deep reasoning. Sonnet is 5-10x cheaper than Opus and fast enough for async
|
||||
detection. The main session continues on Opus while detection runs in parallel.
|
||||
**Why a cheaper class for detection:** Entity detection is pattern matching,
|
||||
not deep reasoning. Sonnet-class runs at a fraction of Opus-class cost, and
|
||||
Haiku-class at a fraction of that — both fast enough for async detection.
|
||||
The main session continues on your best model while detection runs in
|
||||
parallel.
|
||||
|
||||
### Research Pipeline Pattern
|
||||
|
||||
@@ -64,9 +89,9 @@ For research-heavy tasks, use a multi-model pipeline:
|
||||
```
|
||||
|
||||
**Why this works:** The planning and synthesis steps need taste and judgment
|
||||
(Opus). The execution step is mechanical data gathering (DeepSeek at 25-40x
|
||||
lower cost). You get Opus-quality output at DeepSeek-level cost for 80% of
|
||||
the work.
|
||||
(Opus-class). The execution step is mechanical data gathering (a budget
|
||||
model at a small fraction of the cost). You get top-tier output at
|
||||
budget-model cost for 80% of the work.
|
||||
|
||||
### When to Spawn Sub-Agents
|
||||
|
||||
@@ -82,15 +107,16 @@ the work.
|
||||
|
||||
The main session runs on your best model. Everything else runs on the
|
||||
cheapest model that can do the job. In practice, 60-70% of sub-agent
|
||||
work is entity detection (Sonnet) and research execution (DeepSeek),
|
||||
which are 10-40x cheaper than the main session model.
|
||||
work is entity detection and research execution, which run at a small
|
||||
fraction of the main session model's cost.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Sonnet, not Opus, for detection.** The most common mistake is running
|
||||
entity detection on Opus. Detection is pattern matching, not deep reasoning.
|
||||
Sonnet is 5-10x cheaper and fast enough. Reserve Opus for the main session
|
||||
where reasoning quality matters.
|
||||
1. **A cheap class, not Opus, for detection.** The most common mistake is
|
||||
running entity detection on Opus-class. Detection is pattern matching, not
|
||||
deep reasoning. Sonnet- or Haiku-class is several times cheaper and fast
|
||||
enough. Reserve Opus-class for the main session where reasoning quality
|
||||
matters.
|
||||
|
||||
2. **Don't block the main thread.** Sub-agents must run asynchronously. If the
|
||||
signal detector runs synchronously, the user waits 30-120 seconds for every
|
||||
@@ -98,10 +124,11 @@ which are 10-40x cheaper than the main session model.
|
||||
a response immediately.
|
||||
|
||||
3. **Cost optimization is multiplicative.** Entity detection runs on every
|
||||
single message. If you use Opus at $15/MTok for detection across 50
|
||||
messages/day, that's $3-5/day just for detection. Sonnet at $3/MTok brings
|
||||
that to $0.60-1.00/day. Over a month, the wrong model choice costs $100+
|
||||
more than necessary.
|
||||
single message, so the per-call price difference compounds across 50+
|
||||
messages/day. Routing detection from Opus-class ($5/MTok in) to
|
||||
Haiku-class ($1/MTok in) is a flat 5x cut on your highest-frequency LLM
|
||||
call — over a month, the wrong model choice for detection alone costs
|
||||
real money. (Current per-model rates: `src/core/model-pricing.ts`.)
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
@@ -16,9 +16,9 @@ benefit-focused bullets, waits for explicit permission, then runs the full
|
||||
upgrade flow including re-reading skills, running migrations, and syncing
|
||||
schema. The user gets new capabilities automatically.
|
||||
|
||||
## Self-upgrade modes (v0.42)
|
||||
## Self-upgrade modes
|
||||
|
||||
gbrain now stays current the way gstack does: it rides invocation frequency. A
|
||||
gbrain stays current the way gstack does: it rides invocation frequency. A
|
||||
throttled, cache-read-only check runs at the start of every `gbrain` invocation
|
||||
(CLI and MCP) and emits an `UPGRADE_AVAILABLE <old> <new>` marker on stderr. No
|
||||
host cron required — every agent kind (Claude Code, Codex, OpenClaw, Hermes, the
|
||||
@@ -44,6 +44,14 @@ execution. The trust model is TLS + GitHub (same as `gbrain upgrade`);
|
||||
signature verification is a tracked follow-up. Apply manually any time with
|
||||
`gbrain self-upgrade`.
|
||||
|
||||
The `auto` quiet-hours window is configured via the
|
||||
`self_upgrade.quiet_hours` config key
|
||||
(`gbrain config set self_upgrade.quiet_hours '{"start":23,"end":8,"tz":"US/Pacific"}'`).
|
||||
The quiet-hours *pattern* itself — gating any notification or background
|
||||
action on the user's local sleep window — is owned by
|
||||
[quiet-hours.md](quiet-hours.md); this doc only covers the self-upgrade
|
||||
hook into it.
|
||||
|
||||
## Implementation
|
||||
|
||||
### The Check (cron-initiated)
|
||||
@@ -163,15 +171,18 @@ Also persist in `~/.gbrain/update-state.json` so it survives agent context reset
|
||||
|
||||
If you loaded this SKILLPACK directly (copied or read from GitHub) without
|
||||
installing gbrain, you can still stay current. Both GBRAIN_SKILLPACK.md and
|
||||
GBRAIN_RECOMMENDED_SCHEMA.md have version markers:
|
||||
GBRAIN_RECOMMENDED_SCHEMA.md carry a `<!-- source: ... -->` header pointing
|
||||
at their canonical copies, and GBRAIN_RECOMMENDED_SCHEMA.md also carries a
|
||||
version marker:
|
||||
|
||||
```bash
|
||||
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md | head -1
|
||||
# Returns: <!-- skillpack-version: X.Y.Z -->
|
||||
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md | head -1
|
||||
# Returns: <!-- schema-version: X.Y.Z -->
|
||||
```
|
||||
|
||||
If the remote version is newer, fetch the full file and replace your local
|
||||
copy. Set up a weekly cron to check automatically.
|
||||
If the remote version is newer (or the remote SKILLPACK content differs from
|
||||
your local copy), fetch the full file and replace your local copy. Set up a
|
||||
weekly cron to check automatically.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
|
||||
@@ -44,6 +44,7 @@ These require manual setup (no self-installing recipe yet):
|
||||
|-------|-------------|
|
||||
| [Credential Gateway](credential-gateway.md) | Set up ClawVisor or Hermes for Gmail, Calendar, Contacts access |
|
||||
| [Meeting & Call Webhooks](meeting-webhooks.md) | Circleback meeting transcripts + Quo/OpenPhone SMS/calls |
|
||||
| [qm Harness](qm-harness.md) | gbrain as the company brain for a qm (multi-user agent harness) deployment — central HTTP MCP, per-scope clients, roster provisioning, write fencing |
|
||||
|
||||
## How to Read a Recipe
|
||||
|
||||
@@ -69,6 +70,12 @@ health_checks: # typed DSL to verify the integration is working
|
||||
auth_user: "$TWILIO_ACCOUNT_SID"
|
||||
auth_token: "$TWILIO_AUTH_TOKEN"
|
||||
label: "Twilio account"
|
||||
- type: heartbeat_max_age # staleness gate: FAILS `integrations doctor`
|
||||
max_age: 48h # when the newest heartbeat event is older.
|
||||
label: "Data freshness" # The other types are point-in-time and stay
|
||||
# green even when a sense stops producing data.
|
||||
output_paths: # repo-relative dirs the collector writes files to;
|
||||
- daily/voice/ # lets doctor/sync warn if one lands in db_only
|
||||
setup_time: 30 min # estimated time to complete setup
|
||||
---
|
||||
|
||||
@@ -86,7 +93,8 @@ a source install, or the global install copy) are trusted. Recipes discovered at
|
||||
runtime from `$GBRAIN_RECIPES_DIR` or a cwd-local `./recipes/` are marked untrusted:
|
||||
they cannot run `command` health checks, cannot run `http` health checks (SSRF
|
||||
defense), and cannot use the deprecated string health_check form. Untrusted recipes
|
||||
can still use `env_exists` and `any_of` compositions. To ship a recipe that runs
|
||||
can still use `env_exists`, `heartbeat_max_age` (reads only the local heartbeat
|
||||
file — no exec, no network), and `any_of` compositions. To ship a recipe that runs
|
||||
live checks, contribute it upstream so it becomes package-bundled.
|
||||
|
||||
## The Deterministic Collector Pattern
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: gbrain
|
||||
description: Search and write the company knowledge brain. Use for any question about the org, people, projects, decisions, or history, and to persist durable knowledge beyond this scope's notebook.
|
||||
---
|
||||
|
||||
# gbrain — the company brain
|
||||
|
||||
This sandbox has the `gbrain` CLI connected (thin-client) to the org's central
|
||||
brain. It is the deep, indexed, cross-source memory: org docs, shared channel
|
||||
knowledge, and every agent's durable notes. Your scope's own notebook stays the
|
||||
fast per-turn memory; the brain is where knowledge outlives a scope and becomes
|
||||
searchable by everyone entitled to it.
|
||||
|
||||
## First-run setup (once per sandbox — skip if `gbrain remote doctor` passes)
|
||||
|
||||
Your scope's brain credentials arrive via the deployment's secret handoff
|
||||
(keychain entry or one-time secret drop named `gbrain`). Then:
|
||||
|
||||
```bash
|
||||
gbrain init --mcp-only \
|
||||
--issuer-url "https://brain.<org>.com" \
|
||||
--mcp-url "https://brain.<org>.com/mcp" \
|
||||
--oauth-client-id "<client id from the handoff>" \
|
||||
--oauth-client-secret "<client secret from the handoff>"
|
||||
gbrain whoami # must succeed before using any other command
|
||||
```
|
||||
|
||||
Pass the secret with `--oauth-client-secret`, not via `GBRAIN_REMOTE_CLIENT_SECRET`:
|
||||
an env-sourced secret is deliberately NOT written to `~/.gbrain/config.json`, so
|
||||
every later command would fail with "No client_secret available" once the
|
||||
variable is out of scope. The flag persists it to the config file on this
|
||||
sandbox's durable disk, which is what the tool's credential capture expects.
|
||||
|
||||
Do not run `gbrain remote doctor` — it needs `admin` scope, which your client
|
||||
does not have (by design). `gbrain whoami` is the read-scope health check.
|
||||
|
||||
## Reading (do this liberally)
|
||||
|
||||
```bash
|
||||
gbrain search "who decided X and why" # hybrid semantic + keyword search
|
||||
gbrain get <slug> # read one page
|
||||
gbrain query "question" --json # search tuned for agent consumption
|
||||
```
|
||||
|
||||
You can read: the shared agent-memory source, org read-only sources (wiki,
|
||||
handbook), and everything under them. Reads are isolation-enforced server-side;
|
||||
you only ever see sources your client is entitled to.
|
||||
|
||||
## Writing (durable knowledge only, under YOUR prefixes)
|
||||
|
||||
Your client is write-fenced to slug prefixes — your own namespace plus the
|
||||
channels you belong to. Writes outside them are rejected server-side.
|
||||
|
||||
```bash
|
||||
# personal durable memory (your namespace):
|
||||
gbrain put emp-<your-slug>/people/jane-example --content "..."
|
||||
|
||||
# shared channel knowledge (channels you are in):
|
||||
gbrain put chan-eng/decisions/2026-08-database-choice --content "..."
|
||||
```
|
||||
|
||||
Conventions:
|
||||
- Write conclusions and durable facts, not chat transcripts. One page per
|
||||
entity/decision/topic; update the page rather than appending near-duplicates.
|
||||
- Markdown with YAML frontmatter; the brain chunks, embeds, and links it.
|
||||
- Cross-reference liberally: `gbrain link <from> <to>` (from must be in your
|
||||
namespace; linking TO any readable page is fine).
|
||||
- When you learn something channel-relevant in personal work, mirror the
|
||||
conclusion into the channel prefix with a `(said in <where>)` provenance
|
||||
note.
|
||||
|
||||
## When to reach for the brain
|
||||
|
||||
- Any question about the org, a person, a project, a decision, or history →
|
||||
`gbrain search` FIRST, then answer.
|
||||
- You produced knowledge with value beyond this conversation → `gbrain put`.
|
||||
- Something looks wrong (auth errors, empty results you don't expect) →
|
||||
`gbrain whoami` to confirm which client and scopes you're using, and report
|
||||
its output.
|
||||
+258
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env bash
|
||||
# provision-scopes.sh — roster-driven gbrain provisioning for a qm deployment
|
||||
# (or any multi-user agent harness with per-person + per-channel scopes).
|
||||
#
|
||||
# Reads a roster of channels + employees and converges the brain to it:
|
||||
# - ensures the shared agent-memory source exists (path-less: agents write
|
||||
# pages into it over MCP; `gbrain sync` skips it; if the brain host has
|
||||
# sync.repo_path configured, pages also write through to .sources/<id>/
|
||||
# on disk for git-backed durability)
|
||||
# - registers one OAuth client per employee, write-fenced via
|
||||
# bound_slug_prefixes to emp-<slug>/ plus chan-<c>/ for each channel
|
||||
# they are in, with federated reads over the memory source + any
|
||||
# read-only sources you pass
|
||||
# - re-running after roster edits rescopes existing clients IN PLACE
|
||||
# (client ids are remembered in the state file; secrets never rotate
|
||||
# unless you revoke + delete the state row)
|
||||
#
|
||||
# Usage:
|
||||
# provision-scopes.sh roster.tsv \
|
||||
# [--memory-source agents] [--read-sources org-wiki,handbook] \
|
||||
# [--budget-usd-per-day 5] [--state-file roster.state.tsv] \
|
||||
# [--secrets-out new-credentials.tsv] [--gbrain gbrain] [--dry-run]
|
||||
#
|
||||
# Roster format (one entry per line; '#' comments and blank lines ignored):
|
||||
# channel <slug>
|
||||
# employee <slug> [comma-separated channel slugs]
|
||||
#
|
||||
# SECURITY: --secrets-out receives client secrets for NEW registrations,
|
||||
# written exactly once (gbrain never re-shows them). Deliver each row to its
|
||||
# scope's sandbox (e.g. via the harness keychain or a one-time secret drop),
|
||||
# then delete the file.
|
||||
#
|
||||
# ponytail: sequential CLI loop, one gbrain invocation per roster row — fine
|
||||
# to hundreds of employees; batch via the admin API if that ever hurts.
|
||||
|
||||
# -f (noglob) is load-bearing, not stylistic: roster lines are word-split
|
||||
# unquoted below, so without it a line like `employee * eng` would expand
|
||||
# against the working directory and silently provision a filename as a
|
||||
# person — i.e. the wrong write fence. Nothing here needs globbing.
|
||||
set -euf -o pipefail
|
||||
|
||||
# Client secrets and the id state file are written by this script; 077 makes
|
||||
# them 0600 instead of the default 0644. Set before the first file is created.
|
||||
umask 077
|
||||
|
||||
die() { echo "ERROR: $*" >&2; exit 1; }
|
||||
|
||||
# Slugs become source ids, client names, AND slug-prefix write fences. The
|
||||
# fence list is comma-separated, so an unvalidated slug containing a comma
|
||||
# would inject an EXTRA prefix and hand the client write access to someone
|
||||
# else's namespace. Fail closed on anything that isn't plain kebab-case.
|
||||
valid_slug() {
|
||||
case "$1" in
|
||||
'') return 1 ;;
|
||||
-*|*-) return 1 ;;
|
||||
*[!a-z0-9-]*) return 1 ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
}
|
||||
require_slug() {
|
||||
valid_slug "$2" || die "roster: invalid $1 slug '$2' (allowed: lowercase a-z, 0-9, interior hyphens)"
|
||||
}
|
||||
|
||||
ROSTER="${1:-}"
|
||||
[ -n "$ROSTER" ] && [ -f "$ROSTER" ] || die "usage: provision-scopes.sh <roster-file> [flags] (roster not found: '$ROSTER')"
|
||||
shift
|
||||
|
||||
GBRAIN="${GBRAIN:-gbrain}"
|
||||
MEMORY_SOURCE="agents"
|
||||
READ_SOURCES=""
|
||||
BUDGET="5"
|
||||
STATE_FILE=""
|
||||
SECRETS_OUT=""
|
||||
DRY_RUN=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--memory-source) MEMORY_SOURCE="$2"; shift 2 ;;
|
||||
--read-sources) READ_SOURCES="$2"; shift 2 ;;
|
||||
--budget-usd-per-day) BUDGET="$2"; shift 2 ;;
|
||||
--state-file) STATE_FILE="$2"; shift 2 ;;
|
||||
--secrets-out) SECRETS_OUT="$2"; shift 2 ;;
|
||||
--gbrain) GBRAIN="$2"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
*) die "unknown flag: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
STATE_FILE="${STATE_FILE:-${ROSTER}.state.tsv}"
|
||||
SECRETS_OUT="${SECRETS_OUT:-${ROSTER}.new-credentials.tsv}"
|
||||
|
||||
# The roster usually lives in the deployment repo, so the default secrets and
|
||||
# state paths land there too — one `git add -A` from committing live
|
||||
# credentials. The STATE file matters as much as the secrets file: it maps
|
||||
# employee -> client_id, and this script feeds that id straight to
|
||||
# `rescope-client`, so whoever can write it decides which client receives a
|
||||
# given employee's write authority. Treat both as privileged infrastructure,
|
||||
# at the same trust level as the roster itself.
|
||||
for f in "$SECRETS_OUT" "$STATE_FILE"; do
|
||||
if git -C "$(dirname "$f")" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
|
||||
echo "WARN: $f is inside a git work tree. Never commit it;" >&2
|
||||
echo " gitignore it, or pass --secrets-out/--state-file outside the repo." >&2
|
||||
fi
|
||||
done
|
||||
|
||||
# A group/world-writable parent directory defeats the symlink and ownership
|
||||
# checks below: anyone with write access there can swap the file between our
|
||||
# check and our append. Refuse rather than pretend the checks hold.
|
||||
for d in "$(dirname "$SECRETS_OUT")" "$(dirname "$STATE_FILE")"; do
|
||||
perms=$(ls -ld "$d" | awk '{print $1}')
|
||||
case "$perms" in
|
||||
?????w*|????????w*) die "refusing to write credentials into a group/world-writable directory: $d ($perms)" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Secure the credential sinks BEFORE anything is appended. umask only governs
|
||||
# files this script creates; a pre-existing world-readable file would receive
|
||||
# secrets first and be chmod'ed only afterwards, and a symlink planted at
|
||||
# either path would redirect them entirely.
|
||||
for f in "$SECRETS_OUT" "$STATE_FILE"; do
|
||||
[ -L "$f" ] && die "refusing to write credentials through a symlink: $f"
|
||||
if [ -e "$f" ]; then
|
||||
[ -f "$f" ] || die "refusing to write credentials to a non-regular file: $f"
|
||||
[ -O "$f" ] || die "refusing to write credentials to a file owned by another user: $f"
|
||||
else
|
||||
: > "$f"
|
||||
fi
|
||||
chmod 600 "$f"
|
||||
done
|
||||
|
||||
run() {
|
||||
if [ "$DRY_RUN" = 1 ]; then echo "DRY-RUN: $GBRAIN $*" >&2; return 0; fi
|
||||
# shellcheck disable=SC2086 — $GBRAIN may carry args ("bun run src/cli.ts")
|
||||
$GBRAIN "$@"
|
||||
}
|
||||
|
||||
state_lookup() { # state_lookup <employee-slug> -> client_id or empty
|
||||
[ -f "$STATE_FILE" ] || return 0
|
||||
awk -F'\t' -v s="$1" '$1 == s { print $2; exit }' "$STATE_FILE"
|
||||
}
|
||||
|
||||
# ── Pass 1: parse roster, collect declared channels ─────────────────────────
|
||||
CHANNELS=""
|
||||
EMPLOYEES=""
|
||||
lineno=0
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
lineno=$((lineno + 1))
|
||||
line="${line%%#*}"
|
||||
line="${line%$'\r'}" # a CRLF roster would otherwise yield 'emp-alice\r/' prefixes that fence everything out
|
||||
[ -z "${line//[[:space:]]/}" ] && continue
|
||||
# shellcheck disable=SC2086 — deliberate word split; globbing is off (set -f above)
|
||||
set -- $line
|
||||
[ "$#" -le 3 ] || die "roster line $lineno: too many fields ('$line'). Channels are ONE comma-separated field with no spaces: 'employee alice eng,product'"
|
||||
case "$1" in
|
||||
channel)
|
||||
require_slug channel "${2:-}"
|
||||
CHANNELS="$CHANNELS $2"
|
||||
;;
|
||||
employee)
|
||||
require_slug employee "${2:-}"
|
||||
case " $EMPLOYEES " in *" $2:"*) die "roster line $lineno: employee '$2' listed twice" ;; esac
|
||||
if [ -n "${3:-}" ]; then
|
||||
for c in ${3//,/ }; do require_slug "channel-reference" "$c"; done
|
||||
fi
|
||||
EMPLOYEES="$EMPLOYEES $2:${3:-}"
|
||||
;;
|
||||
*) die "roster line $lineno: unknown entry type '$1' (expected 'channel' or 'employee')" ;;
|
||||
esac
|
||||
done < "$ROSTER"
|
||||
|
||||
# ── Pass 2: ensure the shared memory source exists (path-less) ──────────────
|
||||
if out=$(run sources add "$MEMORY_SOURCE" --name "agent memory ($MEMORY_SOURCE)" 2>&1); then
|
||||
echo "source '$MEMORY_SOURCE': created"
|
||||
else
|
||||
echo "$out" | grep -q "already registered" || die "sources add failed: $out"
|
||||
echo "source '$MEMORY_SOURCE': already exists"
|
||||
fi
|
||||
|
||||
# ── Pass 3: converge one client per employee ────────────────────────────────
|
||||
FED_READ="$MEMORY_SOURCE${READ_SOURCES:+,$READ_SOURCES}"
|
||||
new_secrets=0
|
||||
|
||||
for entry in $EMPLOYEES; do
|
||||
slug="${entry%%:*}"
|
||||
chans="${entry#*:}"
|
||||
|
||||
prefixes="emp-$slug/"
|
||||
if [ -n "$chans" ]; then
|
||||
for c in ${chans//,/ }; do
|
||||
echo " $CHANNELS " | grep -q " $c " || echo "WARN: employee '$slug' references undeclared channel '$c'" >&2
|
||||
prefixes="$prefixes,chan-$c/"
|
||||
done
|
||||
fi
|
||||
|
||||
client_id="$(state_lookup "$slug")"
|
||||
if [ -n "$client_id" ]; then
|
||||
# The state file usually sits in the deployment repo, so anyone who can
|
||||
# edit it could otherwise retarget this privileged rescope at an arbitrary
|
||||
# client id (e.g. point alice's row at an admin client). Shape-check it.
|
||||
case "$client_id" in
|
||||
gbrain_cl_) die "state file: empty client id for '$slug'" ;;
|
||||
gbrain_cl_*[!a-zA-Z0-9_]*) die "state file: malformed client id for '$slug': $client_id" ;;
|
||||
gbrain_cl_*) ;;
|
||||
*) die "state file: client id for '$slug' does not look like a gbrain client: $client_id" ;;
|
||||
esac
|
||||
# --source too, so a re-run actually CONVERGES the client to the roster:
|
||||
# without it, changing --memory-source (or inheriting a state row written
|
||||
# against an older one) silently leaves the old write source in place
|
||||
# while the script reports success.
|
||||
run auth rescope-client "$client_id" --source "$MEMORY_SOURCE" \
|
||||
--federated-read "$FED_READ" --bound-slug-prefixes "$prefixes" >/dev/null
|
||||
echo "employee '$slug': rescoped $client_id [write: $prefixes]"
|
||||
elif [ "$DRY_RUN" = 1 ]; then
|
||||
echo "employee '$slug': WOULD register qm-emp-$slug [write: $prefixes] [read: $FED_READ]"
|
||||
continue
|
||||
else
|
||||
out=$(run auth register-client "qm-emp-$slug" \
|
||||
--grant-types client_credentials --scopes "read write" \
|
||||
--source "$MEMORY_SOURCE" --federated-read "$FED_READ" \
|
||||
--bound-slug-prefixes "$prefixes" --budget-usd-per-day "$BUDGET" 2>&1) \
|
||||
|| die "register-client failed for '$slug' (output withheld: it can contain a secret). Re-run the command by hand to see why."
|
||||
client_id=$(echo "$out" | sed -n 's/.*Client ID:[[:space:]]*\(gbrain_cl_[^[:space:]]*\).*/\1/p' | head -1)
|
||||
secret=$(echo "$out" | sed -n 's/.*Client Secret:[[:space:]]*\(gbrain_cs_[^[:space:]]*\).*/\1/p' | head -1)
|
||||
if [ -z "$client_id" ] || [ -z "$secret" ]; then
|
||||
# The client may well have been created — dying silently would strand a
|
||||
# live credential nobody can find. Say so WITHOUT echoing the captured
|
||||
# output: it contains the freshly minted secret, and this path ends up
|
||||
# in CI logs.
|
||||
die "could not parse client id/secret for '$slug' from register-client output (output withheld: it contains a secret). A client MAY have been created; check \`gbrain auth list\` and revoke any stray 'qm-emp-$slug'."
|
||||
fi
|
||||
printf '%s\t%s\n' "$slug" "$client_id" >> "$STATE_FILE"
|
||||
printf '%s\t%s\t%s\n' "$slug" "$client_id" "$secret" >> "$SECRETS_OUT"
|
||||
chmod 600 "$STATE_FILE" "$SECRETS_OUT" 2>/dev/null || true # umask covers new files; this covers pre-existing ones
|
||||
new_secrets=$((new_secrets + 1))
|
||||
echo "employee '$slug': registered $client_id [write: $prefixes]"
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Pass 4: flag offboarded employees ───────────────────────────────────────
|
||||
# Removing someone from the roster is the highest-stakes edit there is, and
|
||||
# this script cannot safely revoke on its own (a typo'd roster would nuke live
|
||||
# credentials). Report instead, with the exact command.
|
||||
if [ -f "$STATE_FILE" ]; then
|
||||
while IFS=$'\t' read -r st_slug st_client _rest; do
|
||||
[ -n "${st_slug:-}" ] || continue
|
||||
case " $EMPLOYEES " in
|
||||
*" $st_slug:"*) ;;
|
||||
*) echo "STALE: '$st_slug' ($st_client) is no longer in the roster but its credentials still work." >&2
|
||||
echo " Revoke with: $GBRAIN auth revoke-client $st_client" >&2 ;;
|
||||
esac
|
||||
done < "$STATE_FILE"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "Done. State: $STATE_FILE"
|
||||
if [ "$new_secrets" -gt 0 ]; then
|
||||
echo "$new_secrets NEW client secret(s) written to $SECRETS_OUT — deliver to each scope's sandbox, then DELETE the file."
|
||||
fi
|
||||
@@ -0,0 +1,10 @@
|
||||
# Roster for provision-scopes.sh — one line per channel / employee.
|
||||
# channel <slug>
|
||||
# employee <slug> [comma-separated channels they belong to]
|
||||
|
||||
channel eng
|
||||
channel product
|
||||
|
||||
employee alice-example eng,product
|
||||
employee bob-example eng
|
||||
employee carol-example
|
||||
|
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"id": "gbrain",
|
||||
"label": "gbrain company brain",
|
||||
"advertise": "gbrain",
|
||||
"hints": [
|
||||
"Company knowledge brain: searchable, cross-source, persistent.",
|
||||
"Search it BEFORE answering questions about the org, people, projects, decisions, or history: `gbrain search \"<question>\"`.",
|
||||
"Write durable knowledge with `gbrain put <slug> --content ...`, only under your own slug prefixes.",
|
||||
"See the gbrain skill for slug conventions and first-run setup."
|
||||
],
|
||||
"auth": {
|
||||
"check": "gbrain whoami",
|
||||
"reauth": "gbrain init --mcp-only --force --issuer-url \"$GBRAIN_ISSUER_URL\" --mcp-url \"$GBRAIN_MCP_URL\" --oauth-client-id \"$GBRAIN_CLIENT_ID\" --oauth-client-secret \"$GBRAIN_CLIENT_SECRET\"",
|
||||
"credentialPaths": [
|
||||
{ "path": ".gbrain/config.json", "kind": "file" }
|
||||
]
|
||||
},
|
||||
"install": { "binary": "gbrain" }
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
# qm (multi-user agent harness) — gbrain as the company brain
|
||||
|
||||
Connect gbrain to [qm](https://github.com/yc-software/qm) — the multiplayer
|
||||
agent harness where each employee and each channel gets an isolated agent
|
||||
scope — so every scope's agent can search and write one shared, indexed,
|
||||
isolation-enforced company brain. The same recipe fits any harness with
|
||||
per-person sandboxes that can run a CLI.
|
||||
|
||||
**Shape:** one central `gbrain serve --http` (OAuth 2.1) next to qm's core;
|
||||
the `gbrain` binary baked into qm's sandbox image as a thin client; one OAuth
|
||||
client per employee, read-fenced by source federation and write-fenced by
|
||||
`bound_slug_prefixes`. Zero qm code changes — everything lives in the qm
|
||||
*deployment directory*.
|
||||
|
||||
qm's native memory (per-scope notebook) stays as-is for fast per-turn recall.
|
||||
gbrain adds what qm doesn't have: semantic + hybrid search, cross-scope
|
||||
knowledge, entity graphs, and durable memory that outlives a scope.
|
||||
|
||||
## Topology
|
||||
|
||||
| gbrain concept | qm concept |
|
||||
|---|---|
|
||||
| one brain (one Postgres/Supabase DB) | the org |
|
||||
| source `agents` (path-less, shared) | all agent-written memory |
|
||||
| slug prefix `emp-<slug>/` in `agents` | an employee's personal scope |
|
||||
| slug prefix `chan-<slug>/` in `agents` | a channel/room scope |
|
||||
| source `org-wiki` (git-backed, read-only) | company docs |
|
||||
| OAuth client `qm-emp-<slug>` | one employee's agent identity |
|
||||
|
||||
Isolation model:
|
||||
|
||||
- **Reads** are source-granular, SQL-enforced (`federated_read`): every
|
||||
employee client reads `agents` + the read-only sources you grant.
|
||||
- **Writes** are slug-prefix-granular, server-enforced (`bound_slug_prefixes`,
|
||||
v0.42.72.0+): a client can only mutate pages under its own `emp-<slug>/`
|
||||
and its channels' `chan-<x>/` prefixes — on `put_page`, `delete_page`,
|
||||
`restore_page`, `add_tag`, `remove_tag`, `add_link`/`remove_link`,
|
||||
`add_timeline_entry`, `revert_version` and `put_raw_data`, plus the
|
||||
`POST /ingest` webhook route. Not by convention.
|
||||
- **Every op that is not a plain read is denied unless allow-listed.** Ops
|
||||
that write by a key other than a slug — `extract_entities` and
|
||||
`extract_facts` (which mutate `people/*` and `companies/*`), `forget_fact`
|
||||
(targets a fact by numeric id, across sources), `ontology_propose`, and the
|
||||
`sources_admin` pair `sources_add`/`sources_remove` — cannot be fenced by
|
||||
slug, so a bound client gets `permission_denied` at dispatch. The gate keys
|
||||
on "not a pure read", not on a list of scope strings, so a write op added
|
||||
later (or one carrying a bespoke scope) is denied until it is explicitly
|
||||
fenced and added to `CLIENT_FENCED_WRITE_OPS` (`src/core/operations.ts`).
|
||||
`think` is allow-listed because remote callers cannot persist from it;
|
||||
`submit_agent` because it enforces this same column itself.
|
||||
- **Indirect write paths are gated too, not just the ops.** `put_page`'s
|
||||
facts backstop would otherwise extract entities from the page body and
|
||||
write fact rows (and a `## Facts` fence on git-backed sources) onto
|
||||
`people/*` pages the caller never named — the same capability
|
||||
`extract_facts` is denied for, reached through an in-prefix write. It is
|
||||
skipped for bound clients. `POST /ingest` is refused outright: its handler
|
||||
bypasses the op layer *and* discards the source grant for untrusted
|
||||
payloads, so it would write into the `default` source.
|
||||
### Known limitations — read these before you rely on the fence
|
||||
|
||||
The write fence is a **write** boundary within a source. It is not a privacy
|
||||
boundary, and it does not make every side effect prefix-clean. As of
|
||||
v0.42.73.2:
|
||||
|
||||
- **The fence follows a delegated write.** When a client with `agent` scope
|
||||
hands work to a subagent via `submit_agent`, that subagent runs under its own
|
||||
slug confinement rather than the parent's OAuth binding. Both confinements are
|
||||
enforced, including on the path where deduplication redirects a write onto an
|
||||
existing page: the redirected target is checked against whichever confinement
|
||||
the calling context actually carries, so delegation does not widen what a
|
||||
client can write.
|
||||
|
||||
- **`add_link`/`remove_link` fence the `from` endpoint only.** A bound client
|
||||
can create an edge pointing AT a page it cannot write; the edge's `context`
|
||||
text surfaces in that page's backlinks and contributes to its search
|
||||
ranking. Fencing `to` would break legitimate cross-referencing into
|
||||
`org-wiki`, so this is deliberate — treat inbound-edge context as untrusted
|
||||
content, the same way you treat page bodies.
|
||||
- **Reads are source-granular, never prefix-granular.** Everyone entitled to
|
||||
a source can read every prefix in it. If a scope needs genuine read
|
||||
privacy, give it its own source.
|
||||
- **`put_page` can create one reverse graph edge outside the fence.** If a
|
||||
page body cites a code location (`src/x.ts:42`) and a code page for it
|
||||
exists *in the same source*, doc↔impl reconciliation adds an edge
|
||||
originating from that code page. It affects graph/backlink ranking, not
|
||||
page content. Unreachable in the layout above (the `agents` source is
|
||||
path-less and holds no code pages); it applies only if you point employee
|
||||
writes at a code-synced source.
|
||||
- **A few read ops are still brain-wide** and ignore the federated grant:
|
||||
`get_recent_salience`, `find_anomalies`, `find_contradictions`, and
|
||||
`sources_list`/`sources_status` (which expose source ids, paths and URLs).
|
||||
A read-scoped client can learn facts derived from sources it was not
|
||||
granted. Pre-existing, not introduced by the fence; if that matters for
|
||||
your deployment, withhold those tools at the harness layer for now.
|
||||
- **Reads touch `last_retrieved_at`** on the pages they return, including
|
||||
pages in read-only sources. Freshness/usage signals are therefore
|
||||
writable-by-reading; nothing else about the page is.
|
||||
- **`POST /ingest` writes land in the `default` source** regardless of the
|
||||
calling client's `source_id`, because the handler discards the source for
|
||||
untrusted payloads. Bound clients are refused the route outright for this
|
||||
reason; if you point a webhook integration at it, scope that brain's
|
||||
`default` source deliberately.
|
||||
- **Tradeoff to state out loud:** read isolation is per-source, so within the
|
||||
shared `agents` source every employee can *read* every prefix (including
|
||||
other employees' `emp-*/`). That matches qm's transparent-by-default,
|
||||
everything-audited posture. If you need hard read privacy for personal
|
||||
memory, give those employees their own write source instead of a prefix
|
||||
(one `sources add emp-<slug>` + `--source emp-<slug>` per client) and keep
|
||||
channel prefixes in `agents` via a second, channels-only client — at the
|
||||
cost of two credentials in that sandbox.
|
||||
|
||||
## Host setup (the machine running qm's core, or any box its sandboxes can reach)
|
||||
|
||||
```bash
|
||||
# 1. Engine: Postgres/Supabase. PGLite is single-process and cannot serve
|
||||
# many concurrent sandboxes.
|
||||
gbrain init --supabase --embedding-model voyage:voyage-4-large
|
||||
|
||||
# 2. Modes + gates (publish_* default OFF and fail as silent 403s):
|
||||
gbrain config set search.mode balanced
|
||||
gbrain config set mcp.publish_skills true
|
||||
gbrain config set mcp.publish_advisor true
|
||||
|
||||
# 3. Read-only org sources + first sync:
|
||||
gbrain sources add org-wiki --path ~/brains/org-wiki
|
||||
gbrain sync --all # cron this
|
||||
|
||||
# 4. Serve over HTTP MCP (OAuth 2.1):
|
||||
gbrain serve --http --bind 0.0.0.0 --port 3131 \
|
||||
--public-url https://brain.acme-example.com
|
||||
```
|
||||
|
||||
Never hand sandboxes `DATABASE_URL` — direct DB access bypasses OAuth, source
|
||||
federation, and the write fence entirely.
|
||||
|
||||
## Provision scopes from a roster
|
||||
|
||||
[`qm-harness-snippets/provision-scopes.sh`](qm-harness-snippets/provision-scopes.sh)
|
||||
converges the brain to a roster file
|
||||
([`roster.example.tsv`](qm-harness-snippets/roster.example.tsv)):
|
||||
|
||||
```bash
|
||||
bash provision-scopes.sh roster.tsv --read-sources org-wiki
|
||||
```
|
||||
|
||||
- Creates the path-less `agents` source (agent-written memory needs no git
|
||||
clone; if the host has `sync.repo_path` configured, pages also write
|
||||
through to `.sources/agents/` for git-backed durability).
|
||||
- Registers `qm-emp-<slug>` clients: `--scopes "read write"`,
|
||||
`--source agents`, `--federated-read agents,org-wiki`,
|
||||
`--bound-slug-prefixes emp-<slug>/,chan-<a>/,...`, per-day budget.
|
||||
- **Idempotent:** re-run after every roster edit; existing clients are
|
||||
`rescope-client`ed in place (channel joins/leaves update the write fence
|
||||
without rotating secrets).
|
||||
- New client secrets land once in `<roster>.new-credentials.tsv` — deliver
|
||||
each row to its scope (qm keychain / one-time secret drop), then delete
|
||||
the file.
|
||||
|
||||
## qm deployment directory
|
||||
|
||||
In the org's qm deployment repo (the directory `qm init` produced):
|
||||
|
||||
1. **Tool:** copy [`qm-harness-snippets/tool.json`](qm-harness-snippets/tool.json)
|
||||
to `sandbox/tools/gbrain/tool.json` and drop the compiled `gbrain` binary
|
||||
beside it (`bun build --compile --outfile gbrain src/cli.ts`, built for
|
||||
the sandbox image's OS/arch). `auth.credentialPaths` marks
|
||||
`~/.gbrain/config.json` as the scope's resident credential file;
|
||||
`auth.check` wires `gbrain whoami` into qm's connector status (read-scope;
|
||||
see the note below on why `remote doctor` cannot be used here).
|
||||
2. **Skill:** copy [`qm-harness-snippets/SKILL.md`](qm-harness-snippets/SKILL.md)
|
||||
to `sandbox/skills/gbrain/SKILL.md` (edit slug conventions to taste).
|
||||
3. Ship it: `qm sandbox build && qm sandbox publish && qm up`.
|
||||
|
||||
Per scope, one-time (agent- or operator-run, credentials from the handoff):
|
||||
|
||||
```bash
|
||||
gbrain init --mcp-only \
|
||||
--issuer-url https://brain.acme-example.com \
|
||||
--mcp-url https://brain.acme-example.com/mcp \
|
||||
--oauth-client-id gbrain_cl_... --oauth-client-secret gbrain_cs_...
|
||||
gbrain whoami # must succeed
|
||||
```
|
||||
|
||||
Use `--oauth-client-secret`, not `GBRAIN_REMOTE_CLIENT_SECRET`: an env-sourced
|
||||
secret is deliberately not written to `~/.gbrain/config.json`
|
||||
(`src/commands/init.ts`), so with the env var alone every later command fails
|
||||
once it leaves scope — and qm's `sandbox.secretEnv` is org-wide, so there is no
|
||||
per-scope env to keep it in. With the flag, the credential lands in the config
|
||||
file on the scope's durable disk and this runs once per scope, ever.
|
||||
|
||||
`gbrain remote doctor` is **not** the health check here: `run_doctor` is an
|
||||
`admin`-scope op and these clients are `read write` on purpose. `gbrain whoami`
|
||||
is read-scope and reports the client's identity, source, and grants.
|
||||
|
||||
## Verify isolation before rollout
|
||||
|
||||
From two differently-scoped sandboxes (or two thin-client configs):
|
||||
|
||||
```bash
|
||||
# alice-example (bound to emp-alice-example/, chan-eng/):
|
||||
gbrain put emp-alice-example/notes/test --content "mine" # OK
|
||||
gbrain put chan-eng/notes/test --content "shared" # OK
|
||||
gbrain put emp-bob-example/notes/test --content "not mine" # permission_denied
|
||||
gbrain put chan-product/notes/test --content "not my channel" # permission_denied
|
||||
gbrain search "test" # sees agents + org-wiki only
|
||||
```
|
||||
|
||||
## Cost + operations
|
||||
|
||||
- `search.mode balanced` (12K token budget, relational retrieval on) is the
|
||||
right default for a startup fleet; see `docs/guides/search-modes.md` for
|
||||
the cost matrix before changing it.
|
||||
- Budgets: `--budget-usd-per-day` is recorded on the client but only enforced
|
||||
on the `submit_agent` path (`src/core/minions/budget-meter.ts`), which these
|
||||
`read write` clients cannot reach — so it does **not** cap spend from
|
||||
ordinary `search`/`put_page` traffic. Treat runaway-agent containment as an
|
||||
open item: watch the admin SPA (`/admin`) and `gbrain search stats`, and cap
|
||||
at the model/harness layer.
|
||||
- Backfills on a live brain: `gbrain embed --stale --pace` (see Pace Mode in
|
||||
CLAUDE.md / `docs/operations/spend-controls.md`).
|
||||
|
||||
## Deliberately deferred
|
||||
|
||||
- **qm `MemoryService` decorator** (mirror notebook captures into gbrain,
|
||||
fan `recall` out and merge, `volunteer_context` push): needs a qm code
|
||||
change; today's integration is agent-initiated via the CLI + skill.
|
||||
- **MCP-native attach:** qm pins `strictMcpConfig` with only its in-process
|
||||
server, so gbrain's MCP-discovered brain-resident skillpacks don't reach
|
||||
qm agents; the sandbox skill above covers it.
|
||||
- **Read-side prefix fencing** (hard privacy for `emp-*/` inside a shared
|
||||
source) — tracked upstream; the roster layout is forward-compatible with
|
||||
it.
|
||||
@@ -2,8 +2,9 @@
|
||||
|
||||
GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it
|
||||
accessible from other devices and AI clients, run `gbrain serve --http`
|
||||
(built-in HTTP transport with bearer auth, Postgres-only ... see
|
||||
[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options.
|
||||
(built-in HTTP transport with OAuth 2.1 + bearer auth, works on both PGLite
|
||||
and Postgres brains — see [DEPLOY.md](DEPLOY.md)) behind a public tunnel.
|
||||
Here are your tunnel options.
|
||||
|
||||
## ngrok (recommended)
|
||||
|
||||
@@ -58,10 +59,11 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits.
|
||||
| Works when laptop is off | No | No | Yes |
|
||||
| Cold start | None | None | None |
|
||||
| Timeout limits | None | None | None |
|
||||
| All 30 operations | Yes | Yes | Yes |
|
||||
| Full remote operation surface (100+ ops, minus `localOnly`) | Yes | Yes | Yes |
|
||||
| Setup time | 5 min | 10 min | 15 min |
|
||||
|
||||
**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth
|
||||
against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap,
|
||||
per-request audit log. Postgres-only by design (PGLite is local-only). See
|
||||
[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables.
|
||||
**Note:** `gbrain serve --http` is the built-in HTTP transport. OAuth 2.1 plus
|
||||
bearer auth against the `access_tokens` table, default-deny CORS, two-bucket rate
|
||||
limit, body cap, per-request audit log. Works on both PGLite and Postgres brains.
|
||||
See [DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and
|
||||
tunables.
|
||||
|
||||
+28
-34
@@ -1,35 +1,39 @@
|
||||
# Connect GBrain to ChatGPT
|
||||
|
||||
**Status (v0.26.0):** Unblocked. GBrain's `gbrain serve --http` ships OAuth 2.1
|
||||
with PKCE, which is the ChatGPT MCP connector's hard requirement. Before v1.0,
|
||||
this was a P0 TODO — the only major AI client that could not connect.
|
||||
ChatGPT's MCP connector requires OAuth 2.1 with PKCE — it does not support
|
||||
bearer-token MCP servers. GBrain's `gbrain serve --http` speaks exactly that,
|
||||
so ChatGPT connects natively.
|
||||
|
||||
ChatGPT does not support bearer-token MCP servers. You must use the OAuth 2.1
|
||||
HTTP server.
|
||||
This page covers only the ChatGPT-specific parts. The full server setup —
|
||||
starting `gbrain serve --http`, the admin bootstrap token, the `/admin`
|
||||
dashboard, tunnels, and `--bind` / `--public-url` — lives in
|
||||
[DEPLOY.md](DEPLOY.md). Do steps 1 (start the server) and 3 (expose it)
|
||||
from there, then come back for the ChatGPT client.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Start the HTTP server
|
||||
### 1. Start and expose the server (DEPLOY.md steps 1 + 3)
|
||||
|
||||
```bash
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
Save the admin bootstrap token printed on stderr. Open
|
||||
`http://localhost:3131/admin` and paste it to access the dashboard.
|
||||
Follow [DEPLOY.md — OAuth 2.1 Setup](DEPLOY.md#oauth-21-setup) to start
|
||||
`gbrain serve --http`, save the admin bootstrap token, and expose the server
|
||||
at a public HTTPS URL (e.g. `https://your-brain.ngrok.app`). ChatGPT's
|
||||
connector auto-discovers the spec-compliant endpoint at
|
||||
`/.well-known/oauth-authorization-server`.
|
||||
|
||||
### 2. Register a ChatGPT client
|
||||
|
||||
ChatGPT uses the authorization code flow with PKCE (browser-based OAuth).
|
||||
Register from the `/admin` dashboard:
|
||||
The ChatGPT-specific delta: ChatGPT uses the **authorization code flow with
|
||||
PKCE** (browser-based OAuth), so the client needs the `authorization_code`
|
||||
grant type and a redirect URI. Register from the `/admin` dashboard:
|
||||
|
||||
1. Click **Register client**.
|
||||
2. Name: `chatgpt`.
|
||||
3. Grant type: `authorization_code`.
|
||||
4. Scopes: `read`, `write` (leave `admin` unchecked for ChatGPT).
|
||||
5. Redirect URI: ChatGPT's OAuth redirect (copy it from the ChatGPT
|
||||
connector setup screen — something like
|
||||
`https://chat.openai.com/connector_platform_oauth_redirect`).
|
||||
5. Redirect URI: ChatGPT's OAuth redirect — **always copy the exact value
|
||||
from the ChatGPT connector setup screen** (it looks like
|
||||
`https://chatgpt.com/connector_platform_oauth_redirect`, but the domain
|
||||
has changed before; trust the setup screen, not this doc).
|
||||
6. Hit **Register**. The credential-reveal modal shows the `client_id` once
|
||||
with Copy and Download JSON buttons. There is no client secret for
|
||||
PKCE-based public clients.
|
||||
@@ -41,22 +45,11 @@ await oauthProvider.registerClientManual(
|
||||
'chatgpt',
|
||||
['authorization_code'],
|
||||
'read write',
|
||||
['https://chat.openai.com/connector_platform_oauth_redirect'],
|
||||
['<ChatGPT redirect URI from the connector setup screen>'],
|
||||
);
|
||||
```
|
||||
|
||||
### 3. Expose the server publicly
|
||||
|
||||
```bash
|
||||
brew install ngrok
|
||||
ngrok http 3131 --url your-brain.ngrok.app
|
||||
```
|
||||
|
||||
Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. ChatGPT's
|
||||
connector auto-discovers the spec-compliant endpoint at
|
||||
`/.well-known/oauth-authorization-server`.
|
||||
|
||||
### 4. Add the connector in ChatGPT
|
||||
### 3. Add the connector in ChatGPT
|
||||
|
||||
1. Open ChatGPT > Settings > Connectors.
|
||||
2. Click **Add connector**.
|
||||
@@ -71,10 +64,11 @@ calls show up in the admin dashboard's live SSE feed in real time.
|
||||
## Scopes
|
||||
|
||||
ChatGPT clients can request any combination of `read`, `write`, `admin`. The
|
||||
scopes granted at consent time are enforced on every tool call. Four
|
||||
operations are `localOnly` and rejected over HTTP regardless of scope:
|
||||
`sync_brain`, `file_upload`, `file_list`, `file_url`. The HTTP server fails
|
||||
closed for any attempt to reach local filesystem surface area.
|
||||
scopes granted at consent time are enforced on every tool call. Operations
|
||||
flagged `localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain`
|
||||
and the `file_*` ops among them) are rejected over HTTP regardless of scope.
|
||||
The HTTP server fails closed for any attempt to reach local filesystem
|
||||
surface area.
|
||||
|
||||
Recommended ChatGPT scope: `read write`. Leave `admin` for your local CLI
|
||||
and the admin dashboard.
|
||||
|
||||
+16
-3
@@ -4,16 +4,26 @@
|
||||
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
|
||||
> end to end, plus the brain-first protocol that makes it worth it. This page is
|
||||
> the connection reference.
|
||||
>
|
||||
> Want the **full agent** — identity, per-turn context, schedules, and a private
|
||||
> repo as its durable body — not just a memory? That's `gbrain bootstrap`:
|
||||
> see the paste block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
|
||||
|
||||
## Option 1: Local (recommended, zero server needed)
|
||||
|
||||
```bash
|
||||
claude mcp add gbrain -- gbrain serve
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs
|
||||
```
|
||||
|
||||
That's it. Claude Code spawns `gbrain serve` as a stdio subprocess. No server, no
|
||||
tunnel, no token needed. Works with both PGLite and Supabase engines.
|
||||
|
||||
`--surface verbs` exposes the five-verb memory protocol (`recall`, `remember`,
|
||||
`entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)),
|
||||
the surface built for agents and quickstarts. Drop the flag for the full
|
||||
operation catalog (`get_page`, `put_page`, `search`, graph ops, …) — `full` is
|
||||
the default and what existing installs already run.
|
||||
|
||||
## Option 2: Remote, one command (fastest from a bearer token)
|
||||
|
||||
If GBrain is running somewhere as an HTTP server (`gbrain serve --http`, see the
|
||||
@@ -79,9 +89,12 @@ You should see results from your GBrain knowledge base.
|
||||
> **`list_skills` returns nothing?** Skill discovery is gated by `mcp.publish_skills`
|
||||
> on the host. New brains from `gbrain init` default it ON; brains upgraded from an
|
||||
> older release stay OFF until you opt in. Enable it on the host with
|
||||
> `gbrain config set mcp.publish_skills true`. The core tools (search, query,
|
||||
> get_page, put_page, think, find_experts) work regardless. Note: `capture` is a
|
||||
> `gbrain config set mcp.publish_skills true`. Skill discovery and the core tools
|
||||
> named here (search, query, get_page, put_page, think, find_experts) are
|
||||
> full-surface — on `--surface verbs` the agent sees only the five memory verbs,
|
||||
> and `list_skills` isn't on the surface at all. Note: `capture` is a
|
||||
> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`.
|
||||
> Why brains differ on the default: [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
|
||||
|
||||
## Remove
|
||||
|
||||
|
||||
@@ -27,6 +27,19 @@ Desktop bridges local MCP servers into Cowork via its SDK layer.
|
||||
This means: if `gbrain serve` is running and configured in Claude Desktop,
|
||||
you don't need a separate server for Cowork.
|
||||
|
||||
## Verify
|
||||
|
||||
In a Cowork session, try:
|
||||
|
||||
```
|
||||
Call get_brain_identity, then search my brain for [any topic]
|
||||
```
|
||||
|
||||
You should get pages from your brain back. If `list_skills` returns nothing,
|
||||
skill publishing is off on the host — enable it with
|
||||
`gbrain config set mcp.publish_skills true` (see
|
||||
[CLAUDE_CODE.md](CLAUDE_CODE.md) for the full gotcha).
|
||||
|
||||
## Which to use?
|
||||
|
||||
- **Remote server:** works even when your laptop is closed, available to all org members
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Connect GBrain to Claude Desktop
|
||||
|
||||
This page covers connecting Claude Desktop to a **remote** brain. For a brain
|
||||
on the same machine as Claude Desktop, a local stdio entry in
|
||||
`claude_desktop_config.json` with `"command": "gbrain", "args": ["serve"]`
|
||||
works too — but only against a full local install, never a thin-client one.
|
||||
|
||||
**Important:** Claude Desktop does NOT connect to remote MCP servers via
|
||||
`claude_desktop_config.json`. That file only works for local stdio servers.
|
||||
Remote HTTP servers must be added through the GUI.
|
||||
|
||||
+15
-9
@@ -4,10 +4,14 @@
|
||||
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
|
||||
> end to end, plus the brain-first protocol that makes it worth it. This page is
|
||||
> the connection reference.
|
||||
>
|
||||
> Want the **full agent** — identity, memory, schedules, and a private repo as its
|
||||
> durable body — not just a connection? That's `gbrain bootstrap`: see the paste
|
||||
> block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
|
||||
|
||||
Codex CLI (`@openai/codex`, v0.130+) supports remote streamable-HTTP MCP servers
|
||||
with a bearer token read from an environment variable. The token lives in your
|
||||
shell env, not in Codex's config file.
|
||||
Recent versions of the Codex CLI (`@openai/codex`) support remote
|
||||
streamable-HTTP MCP servers with a bearer token read from an environment
|
||||
variable. The token lives in your shell env, not in Codex's config file.
|
||||
|
||||
## Fastest path: `gbrain connect`
|
||||
|
||||
@@ -51,11 +55,11 @@ Call get_brain_identity, then search my brain for [topic].
|
||||
`get_brain_identity` confirms whose brain you're connected to; `list_skills` shows
|
||||
everything it can do.
|
||||
|
||||
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host (default
|
||||
> ON for `gbrain init` brains, OFF for brains upgraded from older releases). Enable
|
||||
> it on the host: `gbrain config set mcp.publish_skills true`. The core tools
|
||||
> (search, query, get_page, put_page, think, find_experts) work regardless.
|
||||
> `capture` is CLI-only, not an MCP tool — write over MCP with `put_page`.
|
||||
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host — enable
|
||||
> it with `gbrain config set mcp.publish_skills true`. The core tools (search,
|
||||
> query, get_page, put_page, think, find_experts) work regardless; `capture` is
|
||||
> CLI-only, so write over MCP with `put_page`. Why brains differ on the default:
|
||||
> [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
|
||||
|
||||
## Remove
|
||||
|
||||
@@ -68,4 +72,6 @@ codex mcp remove gbrain
|
||||
- The token is a long-lived, full-access secret. Keep `GBRAIN_REMOTE_TOKEN` out of
|
||||
version control and prefer a scoped token if your host supports one.
|
||||
- Local stdio also works if you run the brain on the same machine:
|
||||
`codex mcp add gbrain -- gbrain serve`.
|
||||
`codex mcp add gbrain -- gbrain serve --surface verbs` — the five-verb memory
|
||||
protocol ([MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)); drop the flag
|
||||
for the full operation catalog.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user