Compare commits

..
Author SHA1 Message Date
22022182f8 fix(cli): honor search --json output (#2042)
`gbrain search <q> --json` silently printed human text: the search/query
ops declare no json param, parseOpArgs dropped the undeclared trailing
flag, and formatResult only emitted human text or --explain output.

Treat --json as a CLI-local boolean formatter flag in parseOpArgs (never
added to the operation contract; MCP validateParams ignores the extra
key on the routed path) and have formatResult's search/query case emit
the raw result array as parseable JSON when set.

Takeover of #2531, rebased onto master (keeps bigintToStringReplacer at
the local-path formatResult call site).

Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:20:25 -07:00
1512 changed files with 8045 additions and 156350 deletions
-29
View File
@@ -1,29 +0,0 @@
# Line-ending policy.
#
# Shell scripts MUST be checked out with LF endings on every platform.
# Git for Windows installs with `core.autocrlf=true` by default, which
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
# then chokes on the trailing CR:
#
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
#
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
# and `bun run test:e2e` for Windows contributors, since all four dispatch
# through bash. `eol=lf` pins the checkout regardless of the user's
# core.autocrlf setting.
*.sh text eol=lf
# Markdown gets the same pin, for a different failure mode: the frontmatter
# parsers anchor on LF. Under a CRLF checkout the opening fence becomes
# "---\r\n", which an LF-only /^---\n/ (or a startsWith("---\n")) does not
# match, so a well-formed document silently parses as having no frontmatter.
# There is no error -- the field just comes back empty. That has surfaced as
# blank skill descriptions, a fixer inserting its banner above the
# frontmatter instead of below it, resolver trigger extraction dropping
# entries, and a generated-doc freshness check reporting every line as
# drifted. The parsers stay CR-tolerant on their own merits (gbrain reads
# Markdown it does not own), but pinning this repo's own .md checkout to LF
# removes the whole class for anyone working here.
*.md text eol=lf
+1 -1
View File
@@ -28,5 +28,5 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: rhysd/actionlint@393031adb9afb225ee52ae2ccd7a5af5525e03e8 # v1.7.11
+6 -9
View File
@@ -45,7 +45,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -61,10 +61,7 @@ jobs:
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
# --timeout also raises bun's 5s default hook budget (beforeAll/afterAll
# do NOT inherit a test's third-arg timeout; verified on bun 1.3.x).
# Every runner script in scripts/ passes it; bare invocations must too.
run: bun test --timeout=60000 test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
@@ -85,13 +82,13 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -119,7 +116,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -158,7 +155,7 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test --timeout=60000 test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+2 -249
View File
@@ -1,7 +1,7 @@
name: Heavy Tests
# Heavy ops-shape tests under tests/heavy/. Cost minutes per run; NOT part
# of default PR CI. Three triggers:
# of default PR CI. Two triggers:
# - Nightly schedule (catches regressions within 24h of merge to master).
# - On-demand opt-in via PR label `heavy-tests` (slow loop kept off by default).
# - Manual workflow_dispatch for triage.
@@ -55,7 +55,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -66,20 +66,6 @@ jobs:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun run test:heavy
# Agent-bootstrap offline Docker e2e [A7] — placeholder registration.
# The networkless cold-machine container test (interview → render →
# verify with a fake gh) lands with the bootstrap doors task at
# tests/docker/bootstrap-e2e.sh. Gated on file existence so this stays
# a visible no-op until then. It runs at heavy cadence (nightly +
# `heavy-tests` label) by design — never in the PR shard matrix.
- name: Bootstrap offline Docker e2e (placeholder)
run: |
if [ -f tests/docker/bootstrap-e2e.sh ]; then
bash tests/docker/bootstrap-e2e.sh
else
echo "SKIP: tests/docker/bootstrap-e2e.sh not present yet — placeholder until the bootstrap doors task lands."
fi
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
# run. Upload those + the rss workload JSON on failure for triage
# without re-running locally.
@@ -101,236 +87,3 @@ jobs:
path: heavy-artifacts/
retention-days: 14
if-no-files-found: ignore
# Real-agent door e2e: drives the ACTUAL `claude` + `codex` + `hermes`
# binaries (no PATH shims) against a real gbrain over MCP. These pay real API
# cost and need the binaries installed + authed, which a stock GitHub runner
# does NOT have — so the tests self-SKIP (describe.skipIf on binary/auth) and
# the job is a clean no-op here. It exists so a self-hosted /
# manually-provisioned runner WITH authed claude/codex/hermes (and
# ANTHROPIC/OPENAI creds) actually exercises the real binaries. Heavy cadence
# only (nightly + `real-agent-e2e` label + dispatch); NEVER the PR shard
# matrix.
real-agent-e2e:
name: Real-agent door e2e (skips without authed binaries)
if: |
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
runs-on: ubuntu-latest
timeout-minutes: 20
env:
# Open the hermes opt-in door here so binary/auth absence — not the
# opt-in var — is what skips (same posture as the claude/codex doors).
GBRAIN_REAL_HERMES_E2E: '1'
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
# Reference the door tests; run only the ones present (a door may land
# in a sibling PR). Missing binary/auth → the file self-skips, so a
# stock runner reports a green no-op rather than failing.
- name: Run real-agent door tests
run: |
files=()
for f in \
test/e2e/bootstrap-real-claude.serial.test.ts \
test/e2e/bootstrap-real-codex.serial.test.ts \
test/e2e/install-real-hermes.serial.test.ts; do
[ -f "$f" ] && files+=("$f")
done
if [ "${#files[@]}" -eq 0 ]; then
echo "SKIP: no real-agent door test files present yet."
exit 0
fi
echo "Running: ${files[*]}"
# --timeout: real-agent turns are slow (live claude/codex); the door
# tests self-skip without authed binaries so this is a no-op elsewhere.
bun test --timeout=600000 "${files[@]}"
# Hermes door e2e: unlike real-agent-e2e above (best-effort, self-skipping),
# this job PROVISIONS the real hermes binary itself — pinned installer digest,
# non-interactive auth + model pin — and then requires the door tests to
# actually execute. It pays real API cost, so it stays label-gated at heavy
# cadence (nightly + `real-agent-e2e`/`heavy-tests` label + dispatch); NEVER
# the PR shard matrix. Loud-fail throughout: a runner that cannot install or
# auth hermes fails this job rather than skipping.
hermes-door:
name: Hermes door e2e (real binary, loud-fail)
if: |
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
runs-on: ubuntu-latest
# Four serial door tests at 600s each plus the installer budget cannot
# fit the sibling job's 20 minutes.
timeout-minutes: 40
env:
# Pin values documented in docs/mcp/HERMES-CLI-PIN.md — update them
# together, deliberately, after reviewing upstream changes. The digest
# pins the INSTALLER SCRIPT; the tag + commit pin the PAYLOAD it clones
# (without them, the installer pulls upstream main into the runner that
# later holds secrets). The commit is v2026.8.3's dereferenced SHA —
# immutable even if the tag moves.
HERMES_VERSION: "0.20.0"
HERMES_GIT_TAG: "v2026.8.3"
HERMES_GIT_COMMIT: "3c27eb6234bf91b8ceee9e9071591b31e9b148cb"
HERMES_INSTALL_SHA256: "c118ff31618dc70339049ce71061b8f1351a1c70d9c2a236ed50d8a2550c550d"
GBRAIN_REAL_HERMES_E2E: '1'
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
# `runner.temp` is not an allowed context in job-level env, so the
# evidence dir is derived here and exported for every later step (the
# door test writes into it; the failure-path upload reads it — exporting
# early keeps that upload working even when an install/precondition step
# is what failed).
- name: Prepare evidence dir
run: |
echo "GBRAIN_E2E_EVIDENCE_DIR=$RUNNER_TEMP/hermes-door-evidence" >> "$GITHUB_ENV"
mkdir -p "$RUNNER_TEMP/hermes-door-evidence"
# NO secrets in this step's env: the installer is downloaded from the
# network and executed, so it must never see credentials. The digest pin
# turns an upstream installer change into a loud failure instead of
# unreviewed code running in CI. Bound + retry the install run itself
# (same posture as the e2e tier2 OpenClaw install): `timeout` kills a
# hung attempt, 3 attempts ride out transient flakes, step cap backstops.
- name: Install hermes (pinned installer digest)
timeout-minutes: 15
run: |
curl -fsSL --retry 3 -o hermes-install.sh https://hermes-agent.nousresearch.com/install.sh
if ! echo "$HERMES_INSTALL_SHA256 hermes-install.sh" | sha256sum -c -; then
echo "::error::hermes installer digest drift — re-pin deliberately: update HERMES_INSTALL_SHA256 + HERMES_VERSION in this workflow and docs/mcp/HERMES-CLI-PIN.md after reviewing upstream changes" >&2
exit 1
fi
for attempt in 1 2 3; do
if timeout 600 bash hermes-install.sh --skip-setup --non-interactive --branch "$HERMES_GIT_TAG" --commit "$HERMES_GIT_COMMIT"; then
# The branch/commit flags above are ASSERTED here, not trusted:
# a shell installer that silently ignores unknown flags would
# clone upstream main into a runner that later holds secrets.
# Verify the actual checkout before anything else runs it.
actual_commit=$(git -C "$HOME/.hermes/hermes-agent" rev-parse HEAD 2>/dev/null || echo "no-git-checkout")
if [ "$actual_commit" != "$HERMES_GIT_COMMIT" ]; then
echo "::error::hermes payload drift — installed checkout is $actual_commit, pinned $HERMES_GIT_COMMIT. Either the installer ignored its branch/commit flags or the layout moved from ~/.hermes/hermes-agent; re-pin deliberately (HERMES_GIT_TAG/HERMES_GIT_COMMIT + docs/mcp/HERMES-CLI-PIN.md) after reviewing upstream." >&2
exit 1
fi
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
exit 0
fi
echo "::warning::hermes install attempt $attempt failed or timed out; retrying in 10s" >&2
sleep 10
done
echo "::error::hermes install failed after 3 attempts" >&2
exit 1
- name: Preconditions (binary, secret, version pin)
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
if ! command -v hermes >/dev/null 2>&1; then
echo "::error::hermes did not resolve on PATH after install" >&2
exit 1
fi
if [ -z "$ANTHROPIC_API_KEY" ]; then
echo "::error::ANTHROPIC_API_KEY secret is empty — fork PRs get no secrets from GitHub, and this labeled job cannot run without them" >&2
exit 1
fi
version_output=$(hermes --version)
echo "$version_output"
# Observed shape: `Hermes Agent v0.20.0 (2026.8.3)`.
if ! printf '%s' "$version_output" | grep -qF "v$HERMES_VERSION"; then
echo "::error::hermes version drift — expected v$HERMES_VERSION in: $version_output" >&2
exit 1
fi
- name: Configure hermes (auth + model pin)
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
mkdir -p ~/.hermes
printf 'ANTHROPIC_API_KEY=%s\n' "$ANTHROPIC_API_KEY" > ~/.hermes/.env
chmod 600 ~/.hermes/.env
# `hermes model` is interactive-only; `config set` is the observed
# non-interactive model pin.
hermes config set model.default anthropic/claude-haiku-4.5
# Global health check — informational only, never a gate here.
hermes doctor || true
- name: Run hermes door tests
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
# Redirect to a file, then tail — never pipe bun through tail (the
# pipe eats the real exit code and truncates failure details).
# `|| EXIT=$?` keeps the default `-e` shell from bailing before the
# tail runs.
EXIT=0
bun test --timeout=600000 test/e2e/install-real-hermes.serial.test.ts > door.txt 2>&1 || EXIT=$?
tail -40 door.txt
if [ "$EXIT" -ne 0 ]; then
# Preserve the FULL bun output for the failure artifact — bun
# prints failure details before the summary, so the 40-line tail
# above can drop exactly the lines a paid-CI triage needs.
cp door.txt "$GBRAIN_E2E_EVIDENCE_DIR/" 2>/dev/null || true
exit "$EXIT"
fi
# This job provisions the binary + auth above, so the door must
# actually EXECUTE: a summary with zero passing tests means the
# suite ran nothing or self-skipped everything — never let that
# read as green.
pass_count=$(grep -Eo '[0-9]+ pass' door.txt | tail -1 | grep -Eo '^[0-9]+' || true)
if [ -z "$pass_count" ] || [ "$pass_count" -eq 0 ]; then
echo "::error::hermes door summary shows no passing tests (nothing ran or everything skipped) — refusing to go green while testing nothing" >&2
exit 1
fi
# The door test copies its evidence into GBRAIN_E2E_EVIDENCE_DIR; the
# workflow only uploads it. The test already excludes credential files —
# the scrub below is defensive belt-and-suspenders before upload. Both
# steps also require the evidence-dir env (a failure before the prepare
# step leaves it unset, and there is nothing to upload then anyway).
- name: Scrub credentials from evidence (defensive)
if: failure() && env.GBRAIN_E2E_EVIDENCE_DIR != ''
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
# Three layers, because the evidence dir carries files WRITTEN BY
# THE THIRD-PARTY HERMES BINARY (logs/sessions) and upload-artifact
# follows symlinks:
# 1. filename scrub (env files),
# 2. symlink delete (an agent-dropped link could dereference to a
# real credential file at upload time),
# 3. content scrub (any file that embeds the key — auth-error dumps
# are most likely exactly on the failure path that uploads).
find "$GBRAIN_E2E_EVIDENCE_DIR" -type f \( -name '.env' -o -name '*.env' \) -exec rm -f {} + 2>/dev/null || true
find "$GBRAIN_E2E_EVIDENCE_DIR" -type l -delete 2>/dev/null || true
if [ -n "$ANTHROPIC_API_KEY" ]; then
grep -rlF "$ANTHROPIC_API_KEY" "$GBRAIN_E2E_EVIDENCE_DIR" 2>/dev/null | while IFS= read -r f; do
echo "::warning::removing evidence file containing the API key: ${f#"$GBRAIN_E2E_EVIDENCE_DIR"/}" >&2
rm -f "$f"
done
fi
- name: Upload hermes door evidence
if: failure() && env.GBRAIN_E2E_EVIDENCE_DIR != ''
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: hermes-door-evidence
path: ${{ env.GBRAIN_E2E_EVIDENCE_DIR }}
retention-days: 14
if-no-files-found: ignore
# Hosted ubuntu-latest runners are ephemeral, but this must not depend
# on that: if the job ever moves to a self-hosted runner (the sibling
# real-agent-e2e job is designed for one), a key left in ~/.hermes/.env
# would persist for every later workload on that runner.
- name: Remove hermes credentials (unconditional)
if: always()
run: rm -f ~/.hermes/.env
+8 -200
View File
@@ -1,75 +1,14 @@
name: Release
# Publishes a GitHub release for every VERSION bump that lands on master:
# tag + title `v<VERSION>`, notes from that version's CHANGELOG.md entry,
# compiled binaries attached (#3521).
#
# Why every bump: `gbrain check-update` resolves the latest version from the
# VERSION file on master, but binary self-update
# (src/core/binary-self-update.ts) downloads assets from `releases/latest`.
# If releases lag VERSION, binary installs are told an upgrade exists that
# self-update cannot apply. Keeping releases/latest == VERSION closes that gap.
#
# Idempotent: the `version` job skips build+release when a release for
# v<VERSION> already exists WITH all expected assets. A half-published release
# (tag exists / assets incomplete) is repaired on the next run — softprops
# updates the existing release in place. Historical 3-segment tags are never
# touched; a new 4-segment VERSION always mints a new tag.
#
# The asset names are a contract with expectedAssetName() in
# src/core/binary-self-update.ts, pinned by test/release-workflow.test.ts.
on:
push:
branches: [master]
paths: [VERSION]
workflow_dispatch: {} # manual first run / backfill of the current VERSION
tags: ['v*']
permissions:
contents: read
concurrency:
group: release
cancel-in-progress: false
contents: write
jobs:
version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
template_tree_hash: ${{ steps.v.outputs.template_tree_hash }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: v
name: Read VERSION and check for an existing complete release
env:
GH_TOKEN: ${{ github.token }}
run: |
version="$(tr -d '[:space:]' < VERSION)"
echo "version=$version" >> "$GITHUB_OUTPUT"
# [C3] Record the vendored template tree's git hash alongside the
# asset completeness check. The publish-template job embeds it in
# the template repo's commit message, so template-repo HEAD can be
# audited back to the exact reviewed tree in THIS repo. Empty until
# templates/bootstrap/template-repo/ is vendored.
tree_hash="$(git rev-parse "HEAD:templates/bootstrap/template-repo" 2>/dev/null || true)"
echo "template_tree_hash=$tree_hash" >> "$GITHUB_OUTPUT"
# Complete = release exists AND carries every asset the self-updater
# can request. A partial release must NOT short-circuit, so a re-run
# can repair it.
assets="$(gh release view "v$version" --repo "$GITHUB_REPOSITORY" \
--json assets --jq '[.assets[].name] | sort | join(",")' 2>/dev/null || true)"
if [ "$assets" = "gbrain-darwin-arm64,gbrain-linux-x64" ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
echo "Release v$version already published with all assets — nothing to do."
else
echo "exists=false" >> "$GITHUB_OUTPUT"
fi
build:
needs: version
if: needs.version.outputs.exists == 'false'
strategy:
matrix:
include:
@@ -85,24 +24,14 @@ jobs:
id-token: write # for attest-build-provenance (Sigstore OIDC)
attestations: write # for attest-build-provenance
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
# No test re-run here: the Test workflow already gated this exact SHA at
# merge (10 shards + E2E). Re-running the whole suite serially on the
# release runner is a flakier duplicate gate — it blocked the first
# release on ambient-env tests (run 30698650484). The build job's gate
# is the artifact itself: compile, then smoke-test the binary.
- run: bun test
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Smoke-test the compiled binary
run: |
chmod +x bin/${{ matrix.artifact }}
out="$(./bin/${{ matrix.artifact }} --version)"
echo "binary reports: $out"
v="$(tr -d '[:space:]' < VERSION)"
case "$out" in *"$v"*) echo "version matches VERSION file" ;; *) echo "binary version '$out' does not contain '$v'" >&2; exit 1 ;; esac
- name: Attest build provenance
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
@@ -113,137 +42,16 @@ jobs:
path: bin/${{ matrix.artifact }}
release:
needs: [version, build]
if: needs.version.outputs.exists == 'false'
needs: build
runs-on: ubuntu-latest
permissions:
contents: write # create the tag + release (scoped to this job only)
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
path: artifacts
- name: Extract CHANGELOG entry for release notes
# env-bound, not inlined into the script: VERSION comes from master so
# it isn't attacker-reachable today, but a `${{ }}` inside `run:` is
# shell injection by construction if that ever changes.
env:
RELEASE_VERSION: ${{ needs.version.outputs.version }}
run: |
v="$RELEASE_VERSION"
if ! bash scripts/changelog-entry.sh "$v" > /tmp/release-notes.md || ! [ -s /tmp/release-notes.md ]; then
echo "See [CHANGELOG.md](https://github.com/${GITHUB_REPOSITORY}/blob/master/CHANGELOG.md) for v$v." > /tmp/release-notes.md
fi
- name: Create release
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2
with:
tag_name: v${{ needs.version.outputs.version }}
name: v${{ needs.version.outputs.version }}
target_commitish: ${{ github.sha }}
body_path: /tmp/release-notes.md
fail_on_unmatched_files: true
files: |
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
artifacts/gbrain-linux-x64/gbrain-linux-x64
# [C1 = D6-A] `latest-stable` is the single sanctioned distribution ref:
# the README paste block, the BOOTSTRAP_FOR_AGENTS.md fetch URL, and
# `bun install -g github:garrytan/gbrain#latest-stable` all reference it
# permanently, so paste blocks copied into the wild never rot and there
# is no 404 window between VERSION landing and assets publishing. It is
# a maintainer-controlled tag advanced ONLY here — the FINAL step of the
# release job, after binaries + provenance attestation have fully
# published — so a half-built release never moves it. The force-move
# (`+`) is intentional: latest-stable tracks the newest verified release;
# per-release history lives in the immutable v<VERSION> tags.
# scripts/check-bootstrap-tag.sh keeps the entry docs pinned to this ref.
#
# If this step ALONE fails, re-advance by hand (a full re-run would skip:
# the release already exists with all assets):
# git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
- name: Advance latest-stable to this release commit
run: git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"
# [G7/S3#4] Publishes the rendered agent-workspace template repo (the
# GitHub "Use this template" door) from CI ONLY — no human pushes it by
# hand, so what adopters clone is exactly what this repo reviewed. Guarded
# three ways:
# 1. The release above fully published (needs: release + the exists gate).
# 2. The vendored tree templates/bootstrap/template-repo/ exists — the
# generator/doors work may not have landed yet; skip, never fail.
# 3. The TEMPLATE_REPO_PAT secret is configured. Secrets are not readable
# in job-level `if:` expressions, so the secret is bound to env (the
# same env-indirection pattern as the release-notes step) and checked
# by the gate step's shell.
#
# TEMPLATE_REPO_PAT scope: a fine-grained PAT with `contents: write` on the
# template repository ONLY — no other repositories, no other permissions.
# Documented in docs/RELEASING.md.
publish-template:
needs: [version, release]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
permissions:
contents: read
env:
TEMPLATE_REPO_PAT: ${{ secrets.TEMPLATE_REPO_PAT }}
# owner/name of the template repo; override via repository variable.
TEMPLATE_REPO: ${{ vars.TEMPLATE_REPO || 'garrytan/gbrain-agent-template' }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: gate
name: Gate on PAT + vendored template tree
run: |
if [ -z "$TEMPLATE_REPO_PAT" ]; then
echo "publish=false" >> "$GITHUB_OUTPUT"
echo "SKIP: TEMPLATE_REPO_PAT secret not configured — template-repo publishing disabled."
elif [ ! -d templates/bootstrap/template-repo ]; then
echo "publish=false" >> "$GITHUB_OUTPUT"
echo "SKIP: templates/bootstrap/template-repo/ not vendored yet — nothing to publish."
else
echo "publish=true" >> "$GITHUB_OUTPUT"
fi
- if: steps.gate.outputs.publish == 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- if: steps.gate.outputs.publish == 'true'
run: bun install
- if: steps.gate.outputs.publish == 'true'
name: Generate template tree and byte-diff against the vendored copy
run: |
bun run scripts/generate-template-repo.ts --out /tmp/template-tree
# [A1] Publish gate: fresh generator output must equal the vendored
# tree byte-for-byte. A mismatch means the vendored tree is stale —
# regenerate + commit it (scripts/check-bootstrap-templates.sh runs
# this same diff offline in `bun run verify`).
diff -r /tmp/template-tree templates/bootstrap/template-repo
- if: steps.gate.outputs.publish == 'true'
name: Force-push the template repo
env:
RELEASE_VERSION: ${{ needs.version.outputs.version }}
TEMPLATE_TREE_HASH: ${{ needs.version.outputs.template_tree_hash }}
run: |
set -euo pipefail
cd /tmp/template-tree
git init -q -b main
git config user.name "gbrain-release-bot"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add -A
# History-less by design: each release force-publishes one commit
# whose message binds (gbrain version, vendored tree hash) [C3].
git commit -q -m "gbrain v${RELEASE_VERSION} template (tree ${TEMPLATE_TREE_HASH:-unvendored})"
# Out-of-band credential: a one-shot GIT_ASKPASS script reads the
# PAT from env at prompt time, so the token never rides argv (where
# `ps`, runner traces, and error messages echoing the remote URL
# could surface it).
ASKPASS="$(mktemp)"
# shellcheck disable=SC2016 # $1/$TEMPLATE_REPO_PAT are literal on
# purpose — they must expand when /bin/sh runs the askpass script at
# git's credential prompt, not when this outer shell writes the file.
printf '%s\n' '#!/bin/sh' \
'case "$1" in Username*) printf "x-access-token\n" ;; *) printf "%s\n" "$TEMPLATE_REPO_PAT" ;; esac' \
> "$ASKPASS"
chmod +x "$ASKPASS"
GIT_ASKPASS="$ASKPASS" GIT_TERMINAL_PROMPT=0 \
git push --force "https://github.com/${TEMPLATE_REPO}.git" HEAD:main
rm -f "$ASKPASS"
generate_release_notes: true
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
container:
image: semgrep/semgrep:1.170.0@sha256:c98f8829eea377274ee4b10656458b078b88232469b2ff913f091c2317347c9d
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
# Non-blocking initially (continue-on-error): the first runs establish a
# baseline without failing unrelated PRs. Graduation path: once the
# baseline findings are triaged (fixed or `# nosemgrep`'d), remove
+15 -99
View File
@@ -43,7 +43,7 @@ jobs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Compute content hash
id: compute
run: |
@@ -84,39 +84,12 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
fetch-depth: 0
# Use the open-source gitleaks CLI, not gitleaks-action@v2: the v2 action
# now enforces a paid GITLEAKS_LICENSE (fails the job with "missing
# gitleaks license" for accounts it can't validate). The CLI is free, uses
# the committed .gitleaks.toml allowlist, and scans the same commit range.
- name: Install gitleaks (pinned + checksum-verified)
run: |
set -euo pipefail
VER=8.30.1
BASE="gitleaks_${VER}_linux_x64.tar.gz"
URL="https://github.com/gitleaks/gitleaks/releases/download/v${VER}"
curl -fsSL -o "/tmp/${BASE}" "${URL}/${BASE}"
curl -fsSL -o /tmp/gitleaks_checksums.txt "${URL}/gitleaks_${VER}_checksums.txt"
( cd /tmp && grep " ${BASE}\$" gitleaks_checksums.txt | sha256sum -c - )
tar -xzf "/tmp/${BASE}" -C /tmp gitleaks
install /tmp/gitleaks /usr/local/bin/gitleaks
gitleaks version
- name: Scan for secrets (gitleaks CLI, .gitleaks.toml)
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "pull_request" ]; then
RANGE="${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}"
else
BEFORE="${{ github.event.before }}"
case "$BEFORE" in
""|0000000000000000000000000000000000000000) RANGE="${{ github.sha }}~1..${{ github.sha }}" ;;
*) RANGE="${BEFORE}..${{ github.sha }}" ;;
esac
fi
echo "Scanning commit range: $RANGE"
gitleaks detect --redact --no-banner --log-opts "$RANGE"
- uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
verify:
# Pre-test gates: privacy/jsonb/source-id/etc + typecheck + admin-build.
@@ -130,7 +103,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -140,11 +113,6 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
# default regardless of per-test third-arg timeouts. Runs directly
# (not via verify's CHECKS array) to avoid a package.json edit.
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
@@ -156,7 +124,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -181,7 +149,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -192,40 +160,6 @@ jobs:
- run: bun install
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
brainbench:
# BrainBench memory-conformance gate (Cathedral 2). Hermetic: in-memory
# PGLite, zero API keys, ~15s for the full 141-fixture × 3-harness run.
# Governance (decision 4): compares HEAD's run against MAIN's committed
# baseline via `git show origin/master:evals/brainbench/baselines/main.json`
# — a PR cannot rewrite the thing it's compared against. Exit 1 blocks the
# merge until the regression is fixed or blessed (justification in the
# updated baseline / fixture diff in corpus-bless mode).
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 10 # ~15s hermetic run; matches the per-job-timeout hardening (#2254)
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
fetch-depth: 0 # the gate needs origin/master's baseline
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bash scripts/ci-brainbench-gate.sh
env:
BRAINBENCH_OUT: ${{ runner.temp }}/brainbench-result.json
- name: BrainBench delta → step summary
if: always()
run: |
if [ -f "${{ runner.temp }}/brainbench-result.json" ]; then
bun scripts/render-brainbench-delta.ts "${{ runner.temp }}/brainbench-result.json" >> "$GITHUB_STEP_SUMMARY"
fi
slow-entity-resolve-perf:
# Dedicated runner for the entity-resolve perf test (~159s, single perf
# describe with one test that builds 5000+ pages and asserts the NEW
@@ -238,7 +172,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -248,19 +182,6 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
# (20K-page corpus + ratio guard) shares this runner — same perf-job
# shape, runs in parallel with the matrix.
- run: bun test test/entity-card-perf.slow.test.ts --timeout=300000
# Protocol self-certification: init a scratch brain and run the
# conformance kit against gbrain's own stdio server. --synthesize is
# safe here: no LLM key in CI, so it asserts the clean `unavailable`
# protocol error instead of spending tokens.
- name: MEMORY_VERBS conformance (self-certify, stdio)
run: |
export GBRAIN_HOME="$RUNNER_TEMP/gbrain-conformance"
bun run src/cli.ts init --pglite --no-embedding --non-interactive
bun run src/cli.ts protocol conformance --synthesize
test:
# Pure matrix shard — no verify, no serial. Each shard runs its slice
@@ -285,17 +206,13 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
# 22, not 15: under parallel PR load the PGLite WASM cold-starts stretch a
# shard past 15 min while every test is still passing — the timeout then
# cancels the job and the test-status gate reads it as a failure. 13 runs
# died this way on 2026-07-21/22 alone.
timeout-minutes: 22
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -316,7 +233,7 @@ jobs:
# entry written when tests hadn't actually passed).
# ──────────────────────────────────────────────────────────────────────
cache-write:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: success() && needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
@@ -339,7 +256,7 @@ jobs:
# Branch protection (when configured) gates on this single job name.
# ──────────────────────────────────────────────────────────────────────
test-status:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
@@ -352,16 +269,15 @@ jobs:
SERIAL="${{ needs.serial-tests.result }}"
SLOW_EVAL="${{ needs.slow-eval-longmemeval.result }}"
SLOW_PERF="${{ needs.slow-entity-resolve-perf.result }}"
BRAINBENCH="${{ needs.brainbench.result }}"
TEST="${{ needs.test.result }}"
echo "cache-check.hit=$HIT"
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF brainbench=$BRAINBENCH test=$TEST"
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF test=$TEST"
if [ "$HIT" = "true" ]; then
echo "✓ cache HIT for hash ${{ needs.cache-check.outputs.hash }} — CI green"
exit 0
fi
# Cache miss: every gated job must have succeeded.
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$BRAINBENCH" "$TEST"; do
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$TEST"; do
if [ "$r" != "success" ]; then
echo "✗ gated job did not succeed (got $r) — CI fail"
exit 1
+2 -10
View File
@@ -1,7 +1,4 @@
# No trailing slash: a bare `node_modules/` pattern matches directories only,
# so a *symlink* named node_modules slips past it and can be committed
# (that's how the /tmp-pointing symlink in faf5cdba got in). Match any type.
node_modules
node_modules/
bin/
.DS_Store
*.log
@@ -18,7 +15,7 @@ supabase/.temp/
# self-contained binaries (the bun --compile path embeds it via
# `import path from 'admin/dist/index.html' with { type: 'file' }`).
# Build via: cd admin && bun install && bun run build.
admin/node_modules
admin/node_modules/
.idea
eval/reports/
eval/data/world-v1/world.html
@@ -38,11 +35,6 @@ export/
# .context/test-shards/. Workspace-local by design — never committed.
.context/
# Local agent instruction overrides (CLAUDE.local.md / AGENTS.local.md) — personal,
# per-clone, loaded after the committed CLAUDE.md/AGENTS.md. Never committed.
CLAUDE.local.md
AGENTS.local.md
# Tier 3 PGLite snapshot fixture (built on demand by build:pglite-snapshot)
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
+13 -19
View File
@@ -4,16 +4,8 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
start here.
> **Becoming someone's persistent personal agent** (identity + memory + private repo)?
> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap`
> flow — instead of the plain install below, then come back here for the operating
> protocol. Connecting to an EXISTING remote brain from a laptop agent?
> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP
> table in [`README.md`](./README.md)).
## Install (5 min)
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
1. Install gbrain via Bun (the canonical path):
```bash
curl -fsSL https://bun.sh/install | bash
@@ -34,8 +26,8 @@ start here.
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
for existing users (search modes were added in v0.32.3).
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step
flow (API keys, identity, cron, verification).
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
(API keys, identity, cron, verification).
## Read this order
@@ -77,10 +69,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
and `gbrain eval replay --against base.ndjson`. For public benchmark
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
<dataset.jsonl>` runs against an isolated in-memory PGLite
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
per question — your `~/.gbrain` is never opened. Full guide:
[`docs/eval-bench.md`](./docs/eval-bench.md).
- **Drive the brain to a target health score:** the one-command
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
loop. `gbrain doctor --remediation-plan --json` previews what would be
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
walks a dependency-ordered plan (sync before extract, embed after
@@ -89,20 +81,22 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
keys hit a `max_reachable_score` ceiling and bail with what's missing.
Three phase handlers (synthesize / patterns / consolidate) are
PROTECTED — only trusted local callers can submit them; MCP cannot.
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md).
- **Track a founder/company over time:** when an entity has
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
and the CHANGELOG entry for v0.36.4.0.
- **Track a founder/company over time (v0.35.7):** when an entity has
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
`unit: USD`, `period: monthly` columns), run
`gbrain eval trajectory <entity-slug>` for the chronological history
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
for a four-signal JSON rollup (claim_accuracy / consistency /
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
same data — read scope, visibility-filtered for remote callers.
`gbrain think` uses this substrate automatically on temporal /
same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:**
`gbrain think` now uses this substrate automatically on temporal /
knowledge_update intent (default ON; flip `think.trajectory_enabled=false`
to opt out). Non-metric event rows (`meeting`, `job_change`,
`location_change`) ride through the same pipeline via `facts.event_type`;
pass `kind: 'event'` or `'all'` to `find_trajectory` to query them.
to opt out). Migration v82 added `facts.event_type` so non-metric event
rows (`meeting`, `job_change`, `location_change`) ride through the same
pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query
them.
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
single-fetch ingestion.
-238
View File
@@ -1,238 +0,0 @@
<!-- gbrain-runbook-stamp: 0.45.12.0 -->
<!-- This stamp must equal the VERSION file at every release; CI enforces it
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
the installed binary and warns on skew. -->
# BOOTSTRAP_FOR_AGENTS.md — become your human's persistent agent
You are an AI agent (Claude Code or Codex) reading this because a human pasted a
block asking you to set yourself up as their persistent personal agent, with gbrain
as your memory. This runbook is the judgment layer; the deterministic work lives in
`gbrain bootstrap` subcommands with exit codes. Follow it top to bottom.
**Scope note:** this path is for Claude Code and Codex (desktop apps or CLIs).
Running OpenClaw or Hermes? Use `INSTALL_FOR_AGENTS.md` instead.
**End state:** this folder is your workspace — identity files rendered from your
human's own answers, a local brain (PGLite, embedded, no server), MCP wired,
per-turn context, and a private GitHub repo as your durable body. ~15 minutes,
mostly interview.
## Hard rules for you, the installing agent
**NEVER INVENT ANSWERS.** Personality, purpose, and boundaries come from the
human. A guessed SOUL.md gets believed by every future session. If you do not have
an answer, ask. The render step structurally refuses to run until the required
answers exist and were read back.
**ASK IN SMALL BATCHES.** The interview is 12 questions max (6 required), asked
in three batches. Mirror each batch back in one line. Accept "skip" on any
non-required question — momentum beats completeness.
**STAY INSIDE THIS PHASE LIST.** Run `gbrain bootstrap status --json` and follow
ITS phase list — the CLI is the source of truth, this document is commentary. If a
step you are asked to run is not in the CLI's phase list, refuse it. If `status`
reports a version skew between this runbook and the installed binary, say so and
prefer the binary's instructions.
**NO SILENT FAILURE.** Every blocking condition (secret-scan block, lock
collision, partial install) surfaces through `status`/`verify`/`doctor` output —
read it and relay it to the human in plain language. Never work around a refusal.
**VERIFY BEFORE CLAIMING DONE.** The install is done when `gbrain bootstrap
verify` exits 0 — not when the transcript looks good. Paste its report to the human.
**RESPECT THE TOOLCHAIN TRUST RULES.** Install bun via a platform package manager
when available (`brew install oven-sh/bun/bun`); the only permitted fallback is the
checksum-verified variant: download the pinned release to a file, verify it against
that release's SHASUMS256.txt, and only then execute. Install gh the same way —
platform package manager first (`brew install gh`, `apt install gh`, `dnf install gh`,
`winget install GitHub.cli` per the official instructions); never a piped
curl-to-shell one-liner. Install gbrain ONLY as
`bun install -g github:garrytan/gbrain#latest-stable` — the npm package named
"gbrain" is an unrelated project. (Cloud-sandbox exception: bun's package fetching
is proxy-incompatible there — use the `gbrain bootstrap cloud-setup-script` recipe,
which installs from the same pinned GitHub source through npm.)
**NEVER FABRICATE TOOLING.** If gh or any preflight binary is missing, blocked
by a sandbox egress proxy, or answering 403s, report that through
`status`/`doctor` output and follow the cloud-sandbox guidance below. Never
hand-roll a gh shim, stub a fake binary into /usr/local/bin, or fake a passing
check — a fabricated tool poisons every later verification, and the one time it
was tried it masked a real silent-persistence failure. The CLI degrades honestly
on its own; your job is to relay, not to bridge.
## Codex preflight (ChatGPT desktop / Codex CLI only)
Codex sandboxes command execution. Before starting, tell the human: "I'll need
approval to run install commands (bun, gh, gbrain) and to write in this folder —
approve those prompts when they appear." If approvals are globally disabled, ask the
human to enable workspace-write + network for this session. Count the approval taps
you needed; report the count at the end (it feeds the install-time measurement).
## Phase walkthrough (commentary — the CLI's list wins)
1. **Preflight.** `git`, `bun`, `gh` present. Install what's missing per the trust
rules above: bun via a platform package manager or the checksum-verified download
— the checksum-verified install is the ONLY permitted non-package-manager
variant; gh via the platform package manager. (On a clean Mac, `git` may trigger
the Xcode tools dialog — that download does not count against the 15 minutes,
tell the human to let it run.)
`gh auth status` — if logged out, the human's ONE manual step:
`gh auth login -h github.com -p https -w` (you run it; they click Authorize).
Then `gbrain bootstrap status` — it is idempotent and resume-aware; after any
partial failure, re-run it and continue where it points.
2. **Engine.** `gbrain init --pglite` (2 seconds, no server). Search mode is
auto-selected silently (conservative when keyless, tokenmax with an
expansion key) and printed with an `[AGENT]` cost matrix — surface that
matrix to the human and confirm before running high-volume queries (see
INSTALL_FOR_AGENTS.md Step 3.5); they can change it any time with
`gbrain search modes`. The one thing to raise here is the OPTIONAL provider
key — with no key you run keyless: keyword search plus memory you author
yourself through the write tools; everything works, one key upgrades search to
semantic and enables automatic fact extraction. Never pressure for a key. If the
human provides one, pass it to the CLI prompt — it goes to the 0600 config file,
never into the interview answers, never into chat logs you keep.
3. **Interview.** `gbrain bootstrap interview --init`, then ask the questions from
the bank (the CLI prints them) in three batches, recording each answer verbatim
with `--set KEY "value"`. Push once on vague answers to the required questions.
Claude Code only: with the final batch, also ask the ONE operational consent —
MCP scope. It is not one of the 12 interview questions; consents ride alongside
the bank. The choice: project (recommended — any other repo you open cannot
read your brain) vs user (your agent everywhere, but any repo you open can
reach it — read and write — and two open sessions contend for the database).
Record it with
`gbrain bootstrap interview --set MCP_SCOPE <project|user>` BEFORE the
read-back, so the confirmation covers it. On Codex, skip this question
entirely — the wiring step states the Codex reality instead.
After the last batch: read ALL answers back in one compact block, ask "Is this
the thing you want in the room?", and only then run
`gbrain bootstrap interview --confirm <hash>` with the hash `--status` printed
for the read-back set. The gate fails if you confirm a set the human never saw.
4. **Render.** `gbrain bootstrap render` — identity files appear. Show the human
SOUL.md. Existing files are never overwritten (re-runs are safe; `--force`
backs up first).
5. **Skills.** `gbrain skillpack scaffold --all` — the CLI scaffolds the skill
set. Nothing to judge here; relay the output.
6. **Wire the harness + register the brain source.** `gbrain bootstrap hooks
--harness <detected>` creates `<workspace>/brain` and prints the exact
`gbrain sources add <source_id> --path <brain> --force` command for THIS
workspace — run it verbatim (don't guess a different id; a guessed id
only surfaces as an FK error at `verify` time, by which point a wrong
guess also blocks the correct id with an `overlapping_path` error). It
also:
- Claude Code: installs per-turn hooks ON by default — do NOT ask; loading the
brain every turn is the whole point of installing gbrain for your agent. Tell
the human it is on and how to turn it off (`GBRAIN_HOOKS=0`, or re-run with
`--no-hooks`, or `gbrain bootstrap uninstall`). MCP scope is NOT asked here —
`hooks` consumes the MCP_SCOPE answer recorded during the interview.
- Codex: registers MCP (`codex mcp add`) and relies on the AGENTS.md protocol —
say plainly that Codex gets pull-based context, not per-turn push.
Do NOT offer an MCP scope choice: `codex mcp add` has no scope flag, so
the registration is always user-global. State it as fact — any repo opened
on this machine can reach the brain (read and write) through its MCP
tools; the off-ramps are `codex mcp remove gbrain` (registration only) or
`gbrain bootstrap uninstall` (full teardown).
7. **Private repo.** `gbrain bootstrap repo` — creates a PRIVATE GitHub repo from
the workspace, verifies the privacy bit through the API, pushes. If the human
started from a repo they created themselves (create-repo-first: an EMPTY private
repo under their own account, cloned and opened here), this ADOPTS that repo
instead of creating one — verifies it is private and pushes the workspace. A
non-empty repo, or one owned by an org, is refused with a clear message (make an
empty personal repo, or run `gbrain bootstrap attach` for an existing agent
clone). Asks the background-persistence consent (a git post-commit auto-push
plus a 30-minute pull job for multi-machine freshness; declining still persists
via the per-turn and session-end pushes). If the human has no GitHub or declines:
local-only mode with an honest warning; `bootstrap repo` can run any time later.
Note: the per-turn/session push stays deferred until this phase records the
verified repo, so nothing is ever pushed to an unverified-privacy origin.
8. **Verify.** `gbrain bootstrap verify` — the whole contract: brain round-trip
through the real write path, graph floor, token sweep, secret scan, repo
privacy, hooks smoke, capability report (keyless or keyed). Exit 0 or it is not
done. Paste the report. Then relay the first-run tour it prints (three prompts
the human should try, starting with restarting the session) AND the hand-off
block below it — the ownership line and the cold-start offer are the two
things the human must actually understand, not fine print.
## Machine two
If this workspace was cloned from an existing agent repo (agent.json says
initialized), run `gbrain bootstrap attach` instead of the interview/render/repo
phases — it wires this machine (source, hooks, MCP) and verifies. If agent.json
says it is an uninitialized template, proceed with the normal flow from phase 1.
## Cloud sandboxes (claude.ai/code and similar proxied environments)
**How you know:** `gbrain bootstrap status --json` reports
`execution_environment: "cloud-sandbox"` (the CLI detects the documented
signals — the CLAUDE_CODE_REMOTE env var, the proxy-injected token
placeholder). Trust the CLI's detection over your own guesses.
**Expected degradations — these are facts to relay, not bugs to bridge:**
- **No crontab, no surviving background processes.** The VM is reclaimed after
inactivity. The scheduled pull is skipped honestly; the per-turn (Stop hook)
and session-end pushes carry persistence. Decline nothing, fabricate nothing.
- **GitHub GraphQL is always blocked** by the egress proxy, and **REST reaches
only repos attached to the session** — a repo created mid-session is NOT
attached, so `gbrain bootstrap repo` refuses fast in cloud with the flow
that works. Privacy verification falls back to pure git protocol on its own.
- **`git push` works only against the session's working branch.** A user PAT
does not bypass any of this.
- **Only repo-committed files carry into the next session.** `~/.gbrain`,
`~/.claude`, and the gitignored `.claude/settings.local.json` evaporate.
Hooks therefore live in the COMMITTED `.claude/settings.json` (the CLI
writes PATH-resolved, fail-open commands there in cloud); hook config is
snapshotted at session start, so hooks written mid-session activate on the
NEXT session — say so instead of debugging it.
**The correct cloud flow:**
1. The human creates the private repo from a normal machine (or github.com)
and opens the cloud session ON that repo.
2. The environment's setup script installs the gbrain binary — print it with
`gbrain bootstrap cloud-setup-script` and have the human paste it into the
environment config (npm-based; bun's fetching is proxy-incompatible there).
3. Inside the session: `gbrain bootstrap attach`, then
`gbrain bootstrap hooks --harness claude-code` (writes the committed
carrier), commit + push, and tell the human the hooks go live next session.
## Failure modes, and what they actually mean
| Symptom | Real cause | Fix |
|---|---|---|
| `interview --status` exits nonzero forever | A required answer is genuinely missing | Ask the human. Do not default it. |
| Render refuses with unresolved tokens | Interview incomplete or a template edit broke a token | Finish the interview; `status` names the tokens. |
| `verify` fails the magic-moment check | The fact never landed (keyless: the Facts fence was not written) | Re-run the write step it names; check `gbrain doctor`. |
| Secret-scan block on push | A credential-shaped string in a tracked file | Fix or allowlist deliberately (`.gbrain-scan-allow`); never force. |
| "bootstrap already running (pid N)" | A concurrent bootstrap holds the lock | Wait or investigate that pid; the lock self-clears when stale. |
| Brain tools fail with a lock error | Another live session's serve owns the database | Close the other session; sequential use is the v1 contract. |
| Hook reports "brain context unavailable" | serve not running or degraded | `gbrain doctor` names it; hooks fail open by design. |
| gh answers 403 "not enabled for this session" | Cloud proxy scoping — the repo is not attached to the session | Expected in cloud; the visibility ladder falls back to git protocol. NEVER shim gh. |
| "crontab: command not found" / cron skipped | Containers and cloud sandboxes ship without a scheduler | Expected; event-driven pushes cover it — the skip message says exactly this. |
| A turn shows "workspace push is FAILING" | The background push is refusing (visibility, secret-scan, or network reasons) | Run `gbrain doctor`; the banner repeats every 30 min until fixed. |
## Hand off
Two things the human must UNDERSTAND before you finish — say them plainly, in
this order, and confirm they landed:
1. **They own the brain.** Every memory you keep is a markdown file in THEIR
private GitHub repo — name the URL. Owning it means: they can read it any
time, take it to a second machine (`gbrain bootstrap attach`), or delete the
repo and the brain is gone. If they went local-only, say that instead, with
`gbrain bootstrap repo` as the any-time upgrade.
2. **The first skill to run is cold-start.** An empty brain is a database; a
filled one is a memory — and every flagship skill (book-mirror, briefings,
meeting prep) only becomes magical once the brain holds their real life.
OFFER to run the cold-start skill now: it imports Gmail, calendar, and
contacts through ClawVisor (clawvisor.com — an OAuth vault; you never hold
raw tokens), or offline archives (Google Takeout, a notes folder) if they
prefer no third-party gateway. Every phase is consent-gated and
independently valuable — they can stop after any one. If they say "later",
that is a complete install; they can say "fill my brain" any time.
Then the routine facts: the capability mode (keyless vs keyed), and the three
commands they will actually reuse (`gbrain doctor`, `gbrain bootstrap verify`,
`gbrain sources push`). Then delete nothing — this runbook was fetched, not
installed.
+6 -1311
View File
File diff suppressed because it is too large Load Diff
+8 -28
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md` — and the seven frozen MEMORY_VERBS `recall`/`remember`/`entity`/`synthesize`/`forget`/`context_pack`/`delta` — the last two are v0.45.7 ambient-recall boundary verbs (budget-packed pack + "what changed since"), all seven stamp `protocol_version: 1`, servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md` + `docs/guides/ambient-recall.md`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -67,19 +67,6 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -119,14 +106,12 @@ detail on demand.)
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| memory verbs / MCP tool surface (`--surface`) / conformance | `docs/protocol/MEMORY_VERBS_v1.md` + the `verbs*`/`surface.ts`/`protocol.ts` entries in `KEY_FILES.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
| running or writing tests | `docs/TESTING.md` |
| bulk-command progress wiring | `docs/progress-events.md` |
| eval methodology / metrics | `docs/eval/` |
| brains vs sources / topology | `docs/architecture/brains-and-sources.md`, `topologies.md` |
| skill routing | `skills/RESOLVER.md` |
| agent bootstrap (paste-in install, hooks, `gbrain bootstrap`, sweep, keyless) | `docs/guides/bootstrap.md` + `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + the KEY_FILES bootstrap cluster |
| shipping a release / CHANGELOG / PR conventions | `docs/RELEASING.md` (ship IRON RULES stay inline below) |
The per-file index (`## Key files`), the thin-client routing seam, and the testing
@@ -192,10 +177,9 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
expensive one.
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix
has three verbatim homes: this section, the `gbrain init` picker copy
(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step
3.5** — update all three when refreshing.
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
picker copy in `gbrain init` carries the same matrix verbatim** — update
both when refreshing.
**Per-query math vs real-world spend.** The matrix above is what an
isolated benchmark would measure. Real agent loops with disciplined
@@ -275,9 +259,8 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills
(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md`
(`AGENTS.md` is also accepted as of v0.19):
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
briefing, migrate, setup, publish.
@@ -481,7 +464,7 @@ ms, max waiters) for `--json`; a one-line summary prints to stderr.
## Version locations (single source of truth: `VERSION` file)
Every release advances the version in **six files at once**. Keep these in
Every release advances the version in **five files at once**. Keep these in
sync. `/ship` enforces this via Step 12's idempotency check (VERSION vs
package.json drift), but the canonical list lives here so future runs and
the auto-update agent know where to look.
@@ -497,7 +480,7 @@ four numeric segments are required first. Historical 3-segment versions
(`0.31.3`, `0.22.1`) remain valid in `git log` and migration filenames
(`skills/migrations/v0.21.0.md`); do NOT rewrite them. Going forward only.
**Required (every release must update all six):**
**Required (every release must update all five):**
| File | What lives there | Format |
|---|---|---|
@@ -506,9 +489,6 @@ four numeric segments are required first. Historical 3-segment versions
| `CHANGELOG.md` | Top entry header `## [0.31.4.1] - YYYY-MM-DD` plus the "To take advantage of v0.31.4.1" block. | Standard Keep-a-Changelog header. |
| `TODOS.md` | Any TODO entries that mention "follow-up from vX.Y.Z.W" use the version of the release that filed them. Update only when filing NEW follow-up TODOs. | Inline `vX.Y.Z.W` references in TODO bodies. |
| `CLAUDE.md` | The Key Files section's per-file annotations carry `vX.Y.Z.W (#NNN)` tags noting which release introduced a behavior. Update whenever a wave's annotations get folded in. | Inline `vX.Y.Z.W (#NNN, contributed by @user)` references. |
| `openclaw.plugin.json` | OpenClaw plugin manifest (v0.45.6.0, #4033). Hand-maintained; `test/openclaw-plugin-manifest.test.ts` fails the suite if it drifts from `package.json`. Merges from master auto-resolve it to master's version — re-bump it with the trio. | `"version": "0.45.12.0"` |
| `BOOTSTRAP_FOR_AGENTS.md` | Runbook stamp on line 1. `scripts/check-bootstrap-tag.sh` (in `bun run verify` + CI) fails when it drifts from `VERSION`; refresh it in the same commit as the bump. | `<!-- gbrain-runbook-stamp: X.Y.Z.W -->` |
| `templates/bootstrap/template-repo/` | Vendored template tree with an embedded version stamp. Auto-derived, but NOT by `bun install`: run `bun run scripts/generate-template-repo.ts --out templates/bootstrap/template-repo` after the bump; `scripts/check-bootstrap-templates.sh` fails CI on drift. | `<!-- gbrain-template-stamp: X.Y.Z.W -->` in generated files. |
**Auto-derived (no manual edit; refreshed by their own commands):**
+78 -106
View File
@@ -11,34 +11,6 @@ bun test
Requires Bun 1.0+.
### Windows
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
the shell scripts under `scripts/` must be checked out with Unix line endings.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
correct with no extra steps.
`.gitattributes` pins `*.md text eol=lf` for the same reason. The frontmatter
readers anchor on a `---` fence followed by a Unix line ending, so a CRLF
checkout makes a well-formed document parse as having no frontmatter. That
failure is silent: no error, the field just comes back empty.
If you cloned before either pin existed, your working copy still has the old
Windows line endings. Bash will fail with `$'\r': command not found`, and
frontmatter will read as absent. Refresh it once, from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
git ls-files --eol -- '*.md' | grep -c w/crlf # 0 means Markdown is clean
```
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
directly. Keep that prefix when you add a new shell-script check.
## Project structure
```
@@ -48,9 +20,7 @@ src/
core/
operations.ts Contract-first operation definitions (the foundation)
engine.ts BrainEngine interface
engine-factory.ts Engine factory (dynamic import of the configured engine)
postgres-engine.ts Postgres + pgvector implementation
pglite-engine.ts PGLite (embedded Postgres via WASM) implementation
postgres-engine.ts Postgres implementation
db.ts Connection management + schema loader
import-file.ts Import pipeline (chunk + embed + tags)
types.ts TypeScript types
@@ -61,16 +31,12 @@ src/
supabase-admin.ts Supabase admin API
file-resolver.ts MIME detection + content hashing
migrate.ts Migration helpers
bootstrap/ Agent-bootstrap flow (interview, hooks, repo, verify)
yaml-lite.ts Lightweight YAML parser
chunkers/ 3-tier chunking (recursive, semantic, llm)
search/ Hybrid search (vector, keyword, hybrid, expansion, dedup)
embedding.ts Embedding service (provider-routed; ZeroEntropy default)
embedding.ts OpenAI embedding service
mcp/
server.ts MCP stdio server (generated from operations)
http-transport.ts HTTP MCP transport (OAuth, body caps)
dispatch.ts Op dispatch + scope enforcement + param redaction
rate-limit.ts Rate limiting
schema.sql Postgres DDL
skills/ Fat markdown skills for AI agents
test/ Unit tests (bun test, no DB required)
@@ -83,21 +49,15 @@ test/e2e/ E2E tests (requires DATABASE_URL, real Postgres+pgvect
docs/ Architecture docs
```
Per-file invariants live in `docs/architecture/KEY_FILES.md` — read a file's entry
before editing it.
## Running tests
The canonical reference for test tiers, isolation rules, timing, and the E2E
lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version:
```bash
# Inner edit loop (~85s on a Mac dev box)
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
bun run test # parallel 8-shard fan-out + serial post-pass
bun test test/markdown.test.ts # specific unit test
# Pre-push gate (19+ parallel checks + typecheck)
bun run verify
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck
# Pre-merge sanity (everything CI runs)
bun run test:full # verify + parallel unit + slow + smart e2e
@@ -115,64 +75,94 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
DATABASE_URL=postgresql://... bun run test:e2e
```
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
Use `bun run verify` before pushing. The guard chain catches: banned fork-name
leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
(`scripts/check-progress-to-stdout.sh`), test-isolation rule violations
(`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel
loop" below), silent fallback to recursive chunking in the compiled binary
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
(`scripts/check-admin-build.sh`), resolver drift on bundled skills
(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full
historical sweep including the trailing-newline and exports-count checks.
(`scripts/check-admin-build.sh`), and resolver drift on bundled skills
(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any
warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift
before merge). `bun run check:all` runs the full historical sweep including the
trailing-newline and exports-count checks.
### Writing tests that survive the parallel loop
`bun run test` shards 1000+ unit-test files across up to 4 worker processes,
capping total concurrency (shards × intra-shard files) to available memory and
re-running OOM-killed or externally-killed files serially before calling them
failures (see `docs/TESTING.md` for the rescue-pass details and knobs). Files
in the same shard share a process, so process-global state leaks between them.
Four lint rules (`scripts/check-test-isolation.sh`, R1R4) enforce isolation:
no direct `process.env` mutation (use `withEnv()` from
`test/helpers/with-env.ts`), no `mock.module(...)` outside `*.serial.test.ts`,
and every `new PGLiteEngine(` goes inside the canonical `beforeAll` block with
a paired `afterAll(disconnect)`.
`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the
same shard share a process, so process-global state leaks between them. Four
lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
**The full rules, the canonical PGLite block, the `withEnv` pattern, and the
`*.serial.test.ts` quarantine policy live in
[`docs/TESTING.md`](docs/TESTING.md#test-isolation-lint-and-helpers)
— read that before writing a new test file.** Files that predate the rules are
listed in `scripts/check-test-isolation.allowlist`; the allow-list MUST shrink
over time — never add new entries.
| Rule | What it bans | Fix |
|---|---|---|
| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` |
| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` |
| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) |
| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` |
### Local CI gate (recommended before pushing)
Canonical PGLite block (R3 + R4 compliant — paste this verbatim):
```ts
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => { await engine.disconnect(); });
beforeEach(async () => { await resetPgliteState(engine); });
```
Env-touching tests:
```ts
import { withEnv } from './helpers/with-env.ts';
test('reads OPENAI_API_KEY', async () => {
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
expect(loadConfig().openai_key).toBe('sk-test');
});
});
```
`withEnv` saves and restores keys via try/finally including when the callback
throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is
process-global). Files using `withEnv` stay outside the future
`test.concurrent()` codemod's eligibility filter.
When to quarantine instead of fix: rename to `*.serial.test.ts` if the file
uses `mock.module(...)`, is genuinely env-coupled (module-load env readers +
ESM caching defeat dynamic-import-after-env tricks), or intentionally shares
state across `it()` boundaries. Quarantine count cap: 10 (informational).
Files that violated these rules at the v0.26.7 baseline are listed in
`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over
time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep
+ codemod) remove entries as files get fixed.
### Local CI gate (recommended before pushing, v0.23.1+)
```bash
bun run ci:local # full gate: gitleaks + guards/typecheck + 4-shard parallel unit + E2E
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
bun run ci:local:diff # gate with diff-aware E2E selector
bun run ci:select-e2e # print which E2E files the selector would run
```
`ci:local` spins up four pgvector services plus a transaction-mode PgBouncer via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite
sharded 4 ways in parallel, then tears down. Named volumes keep the install warm
across runs. Requires Docker (Docker Desktop, OrbStack, or Colima) and `gitleaks`
on host (`brew install gitleaks`). Override the postgres host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
Fail-closed selector: an unmapped `src/` change runs ALL E2E files. Hand-tune
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
narrower mappings via `scripts/e2e-test-map.ts`.
### PR-side security checks
Besides the test gate, PRs may trigger three security workflows: Semgrep CE
SAST (every PR — **advisory/non-blocking** while the baseline is tuned, so a
Semgrep finding won't fail your PR), OSV-Scanner (only when `package.json` or
`bun.lock` change), and actionlint (only when `.github/workflows/**` change).
See `SECURITY.md` → "Automated security scanning" for details.
## Building
```bash
@@ -191,15 +181,6 @@ automatically appears in the CLI, MCP server, and tools-json:
For CLI-only commands (init, upgrade, import, export, files, embed, doctor, sync):
1. Create `src/commands/mycommand.ts`
2. Add the case to `src/cli.ts`
3. Regenerate the flag registry: `bun run build:flag-registry`. The CLI rejects
unknown flags before dispatch; each CLI-only command's legal flag set is
derived from its source into `src/core/cli-flag-registry.generated.ts`.
`test/cli-flag-validation.test.ts` pins registry freshness, drift, and
consumption evidence (a safety flag like `--dry-run` may only be advertised
if the command's code actually reads it), so a stale registry fails the
build. At runtime a missing registry entry fails open — a forgotten regen
never bricks a command. Rerun the regen whenever you add or remove a flag
on an existing command, too.
Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
@@ -208,7 +189,7 @@ Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
See `docs/ENGINES.md` for the full guide. In short:
1. Create `src/core/myengine-engine.ts` implementing `BrainEngine`
2. Add to the engine factory in `src/core/engine-factory.ts`
2. Add to engine factory in `src/core/engine.ts`
3. Run the test suite against your engine
4. Document in `docs/`
@@ -293,7 +274,7 @@ NDJSON wire format is documented in
[`docs/eval-capture.md`](./docs/eval-capture.md).
For public benchmark coverage on top of replay, `gbrain eval longmemeval
<dataset.jsonl>` runs LongMemEval against gbrain's hybrid
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
retrieval. One in-memory PGLite per question, runtime-enumerated
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
published `evaluate_qa.py`. Use it alongside replay when changes affect
@@ -302,18 +283,9 @@ regressions on YOUR queries, LongMemEval catches them on a public set the
benchmark community already cites. See the "Public benchmarks: LongMemEval"
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
## Shipping
Releases go through the `/ship` skill, never hand-rolled. The full release +
contributor process (CHANGELOG voice, version-locations sync, PR conventions,
community-PR-wave workflow) lives in [`docs/RELEASING.md`](docs/RELEASING.md).
Community PRs are batched into release waves rather than merged one-by-one;
contributor attribution stays attached via `Co-Authored-By:` trailers and every
accepted contribution is credited in `CHANGELOG.md`.
## Welcome PRs
- Additional engine implementations (see [`docs/ENGINES.md`](docs/ENGINES.md))
- SQLite engine implementation
- Docker Compose for self-hosted Postgres
- Additional migration sources
- New enrichment API integrations
+9 -51
View File
@@ -3,11 +3,6 @@
Read this entire file, then follow the steps. Ask the user for API keys when needed.
Target: ~30 minutes to a fully working brain.
> **Scope note:** this guide installs the BRAIN (for agent platforms like OpenClaw
> and Hermes, or a standalone brain). If you are Claude Code or Codex and the human
> asked you to become their persistent personal agent — identity + memory + private
> repo — follow `BOOTSTRAP_FOR_AGENTS.md` instead.
## Step 0: If you are not Claude Code
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
@@ -21,16 +16,6 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` (optionally pinned as
> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a
> git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
@@ -90,14 +75,12 @@ NOT inside ~/gbrain.
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
tier is Haiku-class or no expansion-capable API key — Anthropic, OpenAI, or
Google — is configured). The init output included the cost matrix below preceded
by `[AGENT]` markers. You must NOT silently accept the default. Stop and ask the
operator.
tier is Haiku-class or no OpenAI key is configured). The init output included the
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
default. Stop and ask the operator.
**Present this matrix verbatim:**
<!-- Cost matrix: three verbatim homes — CLAUDE.md "Search Mode", src/commands/init-mode-picker.ts, and this block. Sync all three when refreshing. -->
```
Per-query cost @ 10K queries/mo (typical single-user volume):
@@ -208,25 +191,15 @@ scaffold the bundled skills into it:
```bash
cd /path/to/agent/workspace
gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
```
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
diff against gbrain's bundle when you want upstream improvements. (The legacy
`gbrain skillpack install` managed-block model was removed in v0.33 — run
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
**If you are Hermes:** register gbrain as your MCP server:
```bash
printf 'Y\n' | hermes mcp add gbrain --env GBRAIN_HOME=$HOME --connect-timeout 60 --command $(which gbrain) --args serve
```
Keep `--args` last (everything after it becomes server argv) and verify with
`hermes mcp test gbrain` — the add exits 0 even on failure. Full reference:
[docs/mcp/HERMES.md](docs/mcp/HERMES.md).
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
bundled copy at `~/gbrain/skills/RESOLVER.md` when running from the cloned repo). It's
the skill dispatcher — tells you which skill to read for any task. Save this to your
@@ -262,13 +235,7 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), o
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
— or `gbrain sync --watch` for a continuous loop. Safe on keyless brains:
a bare `gbrain embed --stale` exits 0 with a stderr note when embeddings
are disabled, so the chain doesn't break.
- **Health gate** (daily): `gbrain autopilot --status` — exit 0 fresh (or
nothing installed), 1 needs attention (stale heartbeat, never ran, or
paused), 2 the daemon took itself out of rotation. Filesystem-only, so it
works during DB outages.
— or `gbrain sync --watch` for a continuous loop.
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
@@ -287,17 +254,8 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
## Step 9: Verify
Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4
(live sync actually works) is the most important.
Once verification passes and the brain has content, run the activation probe:
```bash
gbrain onboard --check --json
```
See "The onboard surface" below for what the recommendations mean and the
consent gates around unattended remediation.
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
actually works) is the most important.
## Upgrade
@@ -340,7 +298,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
`compiled_truth` from source markdown.
## The onboard surface
## v0.42.0+ onboard surface (NEW)
`gbrain onboard` is the activation surface gbrain did not have before.
Once your brain has any content, run `gbrain onboard --check --json` to
+81 -116
View File
@@ -2,7 +2,7 @@
**Search gives you raw pages. GBrain gives you the answer.** It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box. Run a full autonomous agent on top of it, or just wire it into Claude Code or Codex as a supercharged retrieval layer in one command; either way your coding agent stops being amnesiac about everything that isn't code.
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **155,795 pages, 24,589 people, 5,340 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
**And now it works as a company brain too.** Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the [company-brain](https://www.ycombinator.com/rfs#company-brain) shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. **[Tutorial: set up GBrain as your company brain →](docs/tutorials/company-brain.md)**
@@ -11,11 +11,11 @@ Lots of personal-knowledge systems give you keyword matching and grep in a box.
- **A synthesis layer that gives you the actual answer.** Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
- **A self-wiring knowledge graph.** Every page write extracts entity refs and creates typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, **+31.4 points P@5** over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
The point of building a 150K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.
> **~15 minutes to a working personal agent** on the recommended Codex / Claude Code path (mostly a short interview); ~30 minutes for the always-on OpenClaw / Hermes setup. Database ready in 2 seconds either way (PGLite, no server).
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
@@ -65,60 +65,14 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
GBrain is designed to be installed and operated by an AI agent. **New to GBrain? Start with Codex** — it runs on the ChatGPT subscription you already have, takes ~15 minutes, and deploys nothing. Already living in Claude Code? Its path is identical. Want GBrain running the way it was designed to run — always on, enriching your brain around the clock? That's OpenClaw or Hermes, at real server + API cost. Each path below is complete on its own. (Wiring it up by hand instead? Jump to [CLI standalone](#cli-standalone-no-agent) or the [MCP table](#connect-gbrain-to-your-ai-client-mcp).)
### Have your agent install it (recommended)
### For Codex — the recommended first step
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
Turn Codex into your persistent personal agent. Works in the **ChatGPT desktop app** (open Codex on a folder) and in the **Codex CLI** (`codex` in a terminal) — same install, same result. Open Codex in a **new, empty folder** (not an existing code project) — that folder becomes your agent's own **private GitHub repo**, which bootstrap creates and privacy-verifies for you. Then paste:
```
Read and follow every step of:
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
as your memory. Interview me before writing any identity file — never invent
answers. Ask before anything destructive. You are not done until
`gbrain bootstrap verify` exits 0.
```
Codex will ask for command approvals during the install — approving them is the sandbox working as intended. What you get, in about 15 minutes: a short interview (6 required questions) → your agent's identity (SOUL.md, USER.md, MEMORY.md) rendered from your own answers, never invented → a local PGLite brain (2 seconds, no server, no Docker) → MCP wired so every session can search and write memory → a **private** GitHub repo, created and privacy-verified, as your agent's durable body. Works with **zero API keys** — keyword search plus memory your agent writes itself; one optional key upgrades capabilities (OpenAI: semantic search + automatic fact extraction; Voyage: semantic search; Anthropic: fact extraction). Codex reads brain context through its tools each turn (pull-based). The click moment: tell it one small thing to remember, restart Codex, then ask for it back — the answer comes from the brain, not from this chat's context (which the restart cleared). That cross-session round-trip is the whole product; "what's my name / my top jobs?" is answered from your identity files, which is nice but not the same trick.
Two things worth understanding once it's running: **you own the brain** — every memory is a markdown file in that private repo (read it, clone it to a second machine, delete it and the brain is gone) — and **the first skill to run is `cold-start`**: say "fill my brain" and your agent imports your Gmail, calendar, and contacts (via [ClawVisor](https://clawvisor.com), an OAuth vault so the agent never holds raw tokens) or offline archives like Google Takeout, one consented step at a time. An empty brain is a database; a filled one is a memory.
> **Prefer to make the repo yourself?** Create a new **empty** private repo **under your own GitHub account** (no README/.gitignore/license), clone it, open the clone in Codex, and paste the same block — bootstrap detects your empty repo and adopts it instead of creating one. The repo must be empty and personal-account-owned; org-owned repos are refused (create one under your account, or let bootstrap make it).
### For Claude Code — turn it into your persistent personal agent
Works in the **desktop app** and in the **CLI** (`claude` in a terminal) — identical harness, identical result. Open Claude Code in a **new, empty folder** (not an existing code project) — that folder becomes your agent's own **private GitHub repo**, created and privacy-verified for you. Then paste the same block:
```
Read and follow every step of:
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
as your memory. Interview me before writing any identity file — never invent
answers. Ask before anything destructive. You are not done until
`gbrain bootstrap verify` exits 0.
```
Everything from the Codex path applies — interview, identity from your own answers, local brain, private repo, keyless mode — plus Claude Code gets **per-turn context hooks** (on by default, with an opt-out): your brain loads automatically into every prompt, and your work persists to your private repo on a per-turn cadence (debounced ~5 min locally, every turn in a cloud sandbox — this covers the `/exit` case the harness never fires a session-end hook on), with a notice on your next turn if a push ever fails. This works in a **Claude Code cloud session** too, not just on your laptop: verification falls back to pure git protocol when the sandbox blocks the GitHub API, and `gbrain bootstrap cloud-setup-script` prints the environment setup recipe. The click moment: tell it one small thing to remember, restart the session, then ask for it back — a fresh session has no chat context, so the answer can only come from the brain. That cross-session round-trip is the whole product ("what's my name?" is answered from your identity files — nice, but not the same trick). Same two follow-ups as the Codex path: you own the brain (markdown in your private repo), and `cold-start` is the first skill to run — "fill my brain" imports your email, calendar, and contacts (ClawVisor) or offline archives, one consented step at a time. Full contract, security posture, cloud sandboxes, and uninstall: [docs/guides/bootstrap.md](docs/guides/bootstrap.md).
> **Prefer to make the repo yourself?** Create a new **empty** private repo **under your own GitHub account** (no README/.gitignore/license), clone it, open the clone in Claude Code (CLI or the desktop app's open-a-repo flow), and paste the same block — bootstrap adopts your empty repo instead of creating one. The repo must be empty and personal-account-owned; org-owned repos are refused.
### For OpenClaw or Hermes — GBrain as intended, always on
This is GBrain used the way it was designed to be used: a server-hosted agent with 24/7 crons, continuous ingestion, and the overnight dream cycle that enriches your brain while you sleep — your agent works whether your laptop is open or not. It's also the highest-cost path: a deployed server (8GB+ RAM) plus raw API token usage that scales with how hard your agent runs, well beyond a chat subscription. Start here if you want the full experience from day one; start with Codex above if you want to feel it first. If you don't have a platform running yet, both deploy in one click:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
@@ -127,21 +81,21 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
### Lighter ways in
### Quick start: Claude Code or Codex
**Just want a memory for your coding agent — no identity, no repo.** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the seven-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget`, plus `context_pack` + `delta` since v0.45.7 — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full tool wall; drop the flag for every operation:
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel:
```bash
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve --surface verbs # or: codex mcp add gbrain -- gbrain serve --surface verbs
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve # or: codex mcp add gbrain -- gbrain serve
```
If `claude` is not found, install Claude Code first — or use the per-harness blocks in the [protocol doc](docs/protocol/MEMORY_VERBS_v1.md). Heads-up: memories agents save default to brain-wide visibility (every connected agent can recall them); pass `visibility: "private"` for local-only facts.
**Already have a brain on a remote host** (OpenClaw, Hermes, or any `gbrain serve --http`)? Point your laptop agents at it with one command each — `--install` wires it up and smoke-tests the token before handoff:
```bash
@@ -149,9 +103,18 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --install
gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install # Codex
```
**Brain-only install into another coding agent** (Cursor, Claude Cowork, or anything that can fetch a URL and run shell commands) — paste the OpenClaw/Hermes block above (`INSTALL_FOR_AGENTS.md`); it installs the brain, skills, and dream cycle without the personal-agent identity layer. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — both paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — the memory-only paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
### Install the full autonomous setup into your existing agent
Want the whole thing — local brain, 43 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
```
Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
### CLI standalone (no agent)
@@ -167,13 +130,11 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
### Connect GBrain to your AI client (MCP)
GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side) — or exactly the seven memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
- **[Cursor / Windsurf / any stdio MCP client](docs/mcp/CLAUDE_CODE.md)** — same shape, add `{"command": "gbrain", "args": ["serve"]}` to your MCP config.
- **[Hermes](docs/mcp/HERMES.md)** — `printf 'Y\n' | hermes mcp add gbrain --env GBRAIN_HOME=$HOME --connect-timeout 60 --command $(which gbrain) --args serve`. Keep `--args` last, and verify with `hermes mcp test gbrain` (the add exits 0 even on failure).
- **[OpenClaw](docs/mcp/OPENCLAW.md)** — the ClawHub bundle plugin registers gbrain automatically (`openclaw.plugin.json` ships in this repo), or add `{"command": "gbrain", "args": ["serve"]}` to `~/.openclaw/config.json`'s `mcpServers`.
- **[Claude Desktop (Cowork)](docs/mcp/CLAUDE_DESKTOP.md)** — Settings → Integrations → add the URL of your HTTP server. Remote only; the local `claude_desktop_config.json` does not work for remote servers.
- **[Claude Cowork (team plan)](docs/mcp/CLAUDE_COWORK.md)** — org Owner adds the connector under Organization Settings → Connectors.
- **[Perplexity Computer](docs/mcp/PERPLEXITY.md)** — `gbrain connect https://your-host/mcp --agent perplexity --oauth --register` mints a least-privilege OAuth client and prints the Issuer/Client ID/Secret to paste into Settings → Connectors (OAuth is the right path for a cloud connector; a bearer token also works for local use). Pro subscription required.
@@ -205,7 +166,7 @@ gbrain think "who's working on AI agents at portfolio companies?"
**`gbrain think`** runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 150K-page brain is worth the effort.
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.
`gbrain agent run "..."` exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.
@@ -245,7 +206,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -291,7 +252,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
## Capabilities
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
@@ -315,9 +276,9 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. `gbrain eval brainbench` runs the cross-harness memory conformance suite: know-to-ask, push precision/recall, write-back fidelity, and cross-session continuity, scored per harness seam (your OpenClaw's production pipeline plus Claude Code and Codex injection contracts) against a committed 141-fixture synthetic corpus — hermetic by default (in-memory PGLite, no keys, seconds), and CI gates every PR against master's committed baseline. Methodology in [`docs/eval/BRAINBENCH.md`](docs/eval/BRAINBENCH.md); search-mode methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
**Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
@@ -329,14 +290,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
## Architecture
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source.
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
**Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md).
@@ -346,13 +307,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
## Troubleshooting
**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
**Hourly cron sync keeps timing out on a federated brain?** Switch your
cron to a per-source loop with shell `timeout(1)` doing the OS-level kill
and gbrain self-terminating gracefully half-a-minute earlier:
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
two flags + a recommended pattern. Switch your cron to a per-source loop
with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating
gracefully half-a-minute earlier:
```bash
gbrain sync --break-lock --all --max-age 1800
@@ -365,17 +327,19 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status
`partial` and `last_commit` UNCHANGED — the next run re-walks the same
diff and `content_hash` short-circuits already-imported files. The
`--max-age 1800` first command self-heals any wedged-but-alive locks
left by a hung previous run, keyed on the lock's last refresh time
(NOT when it was acquired) so healthy long-running holders are safe by
construction. Scope note: the extract + embed phases still run to
completion once started; `--timeout` interrupts the import walk only.
left by a hung previous run, using the v98 `last_refreshed_at` semantic
(NOT `acquired_at`) so healthy long-running holders are safe by
construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md)
for the honest scope notes (extract + embed phases run to completion;
30-min rollout window for `--max-age` post-migration v98; full-sync
triggers deferred to v0.42+).
**Dream cycle silently losing wiki links on Supabase?** The engine
self-retries every bulk batch write (`addLinksBatch` /
`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips,
with a 12s worst-case wait that covers the full 5-10s circuit-breaker
recovery window. `gbrain doctor` surfaces incidents via the
`batch_retry_health` check (reads the last 24h of
**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes
the bug class structurally. The engine now self-retries every bulk batch
write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on
Supavisor pooler blips, with a 12s worst-case wait that covers the full
5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents
via the new `batch_retry_health` check (reads the last 24h of
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually
slow pooler:
@@ -393,33 +357,34 @@ retry wrap is engine-level, but PGLite has no pooler so retries never
fire in practice.
**Dream cycle losing ~150 link rows per run with `'No database
connection: connect() has not been called'` errors in the log?** The
retry layer self-heals on a nulled-out database singleton: a
`reconnect` callback on `withRetry` rebuilds the connection between
attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()`
connection: connect() has not been called'` errors in the log?** v0.41.27.0
makes the retry layer self-heal on a nulled-out database singleton. A
new `reconnect` callback on `withRetry` rebuilds the connection between
attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()`
so engine-level batch writes survive a mid-cycle disconnect by something
else in the same process. `gbrain capture` also no longer trails a
`'No database connection'` stderr line from a background facts:absorb
worker firing after CLI exit — op dispatch awaits
else in the same process. Same release: `gbrain capture` no longer trails
a `'No database connection'` stderr line from a background facts:absorb
worker firing after CLI exit — the op-dispatch finally block awaits
`getFactsQueue().drainPending({timeout: 1000})` before
`engine.disconnect()`. To find which code path is still calling
disconnect mid-process, run `gbrain doctor --json | jq '.checks[] |
select(.id=="batch_retry_health")'`; the check surfaces the
24h disconnect-call count and the most-recent caller frame from the
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit.
select(.id=="batch_retry_health")'`; the extended check now surfaces
24h disconnect-call count and the most-recent caller frame from a new
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.)
**`gbrain brainstorm` returning `judge_failed: true` with 0 scored
ideas?** Two historical bugs caused it, both fixed: the judge
hard-coded a 4K-token output cap (any run past ~40 ideas truncated
mid-JSON and the parser threw), and slash-form model ids
(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because
pricing lookups only matched the colon form. Both shapes work now. No
config change, no schema migration — `gbrain upgrade` is the whole fix.
ideas?** v0.41.21.0 closes the two bugs that caused it. The judge
hard-coded a 4K-token output cap; for any run past ~40 ideas the call
truncated mid-JSON and the parser threw. Same release closes a slash-
form pricing miss: `gbrain brainstorm --judge-model
anthropic/claude-sonnet-4-6 --max-cost 5` failed with
`BudgetExhausted reason=no_pricing` because every pricing site only
matched the colon form. Both shapes work now. No config change, no
schema migration — `gbrain upgrade` is the whole fix.
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
tags?** Upgrade — tag reconciliation is add-only now. Re-import and
`reindex --markdown` ADD current frontmatter tags and never delete,
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
`reindex --markdown` now ADD current frontmatter tags and never delete,
so enrichment tags written to the DB (auto-tag, dream synthesize,
signal-detector) survive a re-chunk. The reindex DB-only fallback also
reconstructs the full markdown (frontmatter + body + timeline) before
@@ -427,10 +392,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter,
title, and timeline instead of getting overwritten with empty
frontmatter. Trade-off: removing a tag from a page's frontmatter no
longer removes it from the DB on the next sync (frontmatter-tag removal
needs a provenance column, deferred).
needs a provenance column, deferred). (Closes #1621.)
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
Three tools. First, name the stalling file:
v0.41.37.0 ships three things. First, name the stalling file:
```bash
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
@@ -445,28 +410,28 @@ the sync with the pack disabled and re-run extraction later:
gbrain sync --no-schema-pack --no-pull --no-embed --yes
```
`gbrain schema lint` warns on the classic nested-quantifier ReDoS
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
PGLite is single-writer and a live MCP server contends for the write
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
for the full triage.
for the full triage. (Closes #1569.)
**`gbrain init --migrate-only` / a schema migration fails on Windows
with `getaddrinfo ENOTFOUND`?** Upgrade schema bring-up now runs its
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
phases in-process instead of spawning a child `gbrain init
--migrate-only` per phase. The spawned child died on
Windows + bun + Supabase pooler with a DNS-resolution failure even
though the parent connected fine; running in-process removes the spawn
entirely. The grandfather migration that used to hang 70+ minutes on an
80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on
the page PK, soft-delete-filtered, source-safe) and completes in seconds.
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
completes in ~1-2 seconds. (Closes #1605, #1581.)
## Docs
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall)
- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows`
- [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory
+23 -54
View File
@@ -8,44 +8,17 @@ on GitHub.
Do not open a public issue for security vulnerabilities.
## Automated security scanning
CI runs three automated security checks alongside secret scanning (Gitleaks):
- **Dependency vulnerabilities** — OSV-Scanner
(`.github/workflows/osv-scanner.yml`) runs weekly and on any PR that touches
`package.json` or `bun.lock`.
- **Static analysis (SAST)** — Semgrep CE (`.github/workflows/semgrep.yml`)
runs on every PR and weekly. It is currently **advisory (non-blocking)**
while the finding baseline is tuned; the graduation path to a blocking check
is documented in the workflow file.
- **Release binary provenance** — release builds
(`.github/workflows/release.yml`) attest each compiled binary with
[GitHub artifact attestations](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations).
Verify a downloaded release binary with:
```bash
gh attestation verify ./gbrain-darwin-arm64 -R garrytan/gbrain
gh attestation verify ./gbrain-linux-x64 -R garrytan/gbrain
```
All security workflows use SHA-pinned actions and least-privilege permissions,
enforced structurally by actionlint on every workflow change.
## Remote MCP Security
### Keep dynamic client registration disabled unless explicitly needed
### ⚠️ Do NOT use open OAuth client registration for remote MCP
GBrain disables Dynamic Client Registration (DCR) by default. Keep that
default for internet-reachable deployments and pre-register trusted clients
with operator-approved scopes and source access. Enabling DCR lets network
callers create OAuth client records, so use it only when the deployment's
trust model requires self-service registration and browser approval remains
part of the authorization flow.
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
support, **never allow unauthenticated client registration**. An attacker
who discovers your server URL can:
Do not enable `--enable-dcr-insecure` on an untrusted network. That option is
reserved for deployments that intentionally allow self-registered
machine-to-machine clients without browser approval.
1. Register a new OAuth client via `POST /register`
2. Use `client_credentials` grant to obtain a bearer token
3. Access all brain data via the MCP tools
### Recommended: `gbrain serve --http`
@@ -107,10 +80,12 @@ Auth methods (`--token-endpoint-auth-method`):
- `none` — public PKCE-only client (no secret minted; ChatGPT custom
connector, Claude Code, Cursor)
The same validator applies to CLI, admin, and DCR registration paths, so
unknown authentication methods are rejected consistently. Browser-based
clients can be configured entirely through the supported CLI flags; operators
do not need to edit OAuth database rows by hand.
The validator rejects unknown methods at the registration boundary, and
the same gate applies to the admin endpoint `POST /admin/api/register-client`
and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
`redirect_uris = []` and `token_endpoint_auth_method = NULL`, forcing
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
without `--enable-dcr`. That footgun is gone.
### DCR consent default (v0.42.55+)
@@ -160,18 +135,6 @@ the PGLite schema. Local agents continue to use stdio (`gbrain serve`).
Running `--http` against a PGLite-backed install fails fast with a clear
error message at startup.
### Docker network isolation (self-hosted Postgres)
OAuth and source scoping enforce isolation on the `serve --http` path only.
Raw Postgres reachability bypasses both: a container that shares Docker's
default `bridge` network with the brain's Postgres can open a direct DB
session without any token and read every source. Put the brain's Postgres on
a user-defined Docker network with nothing untrusted on it, publish its port
loopback-only (if at all), and never put `DATABASE_URL` or a Postgres
password in untrusted agent containers — those should reach the brain
exclusively via OAuth against `serve --http`. Full operator checklist:
[docs/mcp/DEPLOY.md — Co-located Docker workloads](docs/mcp/DEPLOY.md#co-located-docker-workloads-self-hosted-postgres).
### CORS
Default-deny: no `Access-Control-Allow-Origin` header is sent unless an
@@ -187,10 +150,16 @@ When the request `Origin` matches the allowlist, the server echoes it
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
CORS header is sent and the browser blocks the request.
The same allowlist gates the complete MCP and OAuth HTTP surface. Actual
requests and browser preflight requests use one allowlist-gated policy, so
unlisted origins receive no cross-origin authorization. A startup stderr
warning fires when `--bind 0.0.0.0` is set without
**v0.41.3:** the same allowlist now gates every OAuth endpoint (`/mcp`,
`/token`, `/authorize`, `/register`, `/revoke`). Pre-v0.41.3 these used
default-wide-open `cors()` middleware, leaking
`Access-Control-Allow-Origin: *` on every response — any web origin could
complete a token exchange from a logged-in operator's browser. The CORS
preflight handler in the legacy bearer transport was also asymmetric
(actual-request path correctly default-deny, but OPTIONS preflight leaked
`Access-Control-Allow-Methods` + `Access-Control-Allow-Headers` to every
Origin); both are now consolidated through a single allowlist-gated path.
A startup stderr WARN fires when `--bind 0.0.0.0` is set without
`GBRAIN_HTTP_CORS_ORIGIN`, surfacing the default-deny posture before the
first request.
+42 -779
View File
@@ -1,318 +1,18 @@
# TODOS
## Onboarding DX follow-ups (filed v0.45.9.0)
- [ ] **Retire the `config set embedding_model` dead-end across ALL surfaces.** v0.45.9.0 fixed the keyless-init notice to point at `gbrain init --force --pglite --embedding-model <id>`, but `src/core/embed-preflight.ts` (lines ~73/83/90/115) and `src/core/embedding-dim-check.ts:78` still advertise `gbrain config set embedding_model <...>`, which `src/commands/config.ts:142` hard-refuses as a schema-sizing no-op. Same dead-end class, different surfaces. Sweep them to the re-init recipe. Priority: P2.
- [ ] **`gbrain init --supabase` migrate-model dead-end doc.** The Postgres branch of config.ts points at `docs/embedding-migrations.md`; confirm that doc exists and describes a working switch, or write it. Priority: P3.
- [ ] **DX harness binary cache keyed on nothing.** `scripts/dx-explore.ts` reuses `.context/dx-runs/bin/gbrain` unless `--rebuild` is passed, so a second run after code changes can produce transcripts from a stale binary. Key the cache by a source hash (or rebuild when any `src/` file is newer). Dev instrument only. Priority: P3.
- [ ] **`verify` has no MCP-registration check.** v0.45.9.0 made `bootstrap status` report the wire phase `partial` when only hooks landed (host CLI missing), but `bootstrap verify` still exits 0 in that state. Add an MCP-registration probe to verify so the "done when verify exits 0" contract also covers MCP. Priority: P2.
- [ ] **`hasExpansionKey` misses config-plane keys + init-before-key sequencing.** The mode picker reads `process.env` only; a key routed to the 0600 config by the interview (which runs AFTER init) never influences the auto-selected search mode, and the picker never re-fires. Resolve keys through the capability/gateway fold and consider re-running the recommendation when a key is first configured. Priority: P3.
- [ ] **`findEnvKeyTypos` KEY_SHAPE misses no-underscore typos.** `OPENAI_APIKEY` (no `_` before `KEY`) escapes the near-miss net, so that typo class now completes keyless silently instead of failing loud. Widen the regex. Priority: P3.
- [ ] **`init-nudge` stale "4 checks" comment + 6-probe accounting.** The header still says "4 onboard checks" but six probes now run; the partial-checks message counts the page-count probe. Cosmetic. Priority: P3.
- [ ] **FIRST LIGHT (the real first-magical-moment feature).** The v0.45.9.0 tour rewrite is the ship-now slice; the full seed-phase → compendium → scout design is PR-A (seed phase + Mirror + baton) / PR-B (compendium + scout) with one-way-door decisions (new bootstrap phase, consent key, `skills/first-light/`, a one-time Gate-3 narration exemption). Priority: P2.
## Ambient recall follow-ups (filed v0.45.7.0, issue #1)
Deferred from the ambient-recall wave (`context_pack` + `delta` frozen verbs +
boundary runtime; CEO+ENG cleared, plan at
`~/.claude/plans/system-instruction-you-are-working-vectorized-gem.md`). Each was
explicitly scoped OUT with a one-line rationale — none is a bug, all are additive.
- [ ] **Autonomous transcript watchers (D3=B).** The shipped event contract covers session boundaries (start, compaction, heartbeat) but relies on the harness emitting a lifecycle event. A per-harness transcript watcher would drive ambient recall for harnesses that can't emit — but watchers are fragile and compaction is often invisible on disk. Add per harness that proves it can't emit a boundary event. Priority: P3.
- [ ] **Materialized `thread_state` table.** `delta`'s thread-change arm derives open-thread deltas from facts/timeline `updated_at` scans. If a perf gate ever forces it, materialize a `thread_state` table instead of deriving. Not needed until the derive-path SLO is threatened. Priority: P3.
- [ ] **Codex native boundary hooks.** Codex has no hooks upstream (`CODEX_HAS_HOOKS=false`), so its ambient path is pull-only (AGENTS.md gate tells it to call `context_pack`/`delta` at boundaries). When Codex ships a hook mechanism, register the boundary events the way the Claude Code lane does; the IPC `context_pack` kind + `--harness codex` attribution channel are already reserved for it. Priority: P3.
## Brain-currency harness-e2e follow-ups (filed with the PR-A wave)
- [ ] **P1 — Extend engine-identity convergence to the other long-lived planes.**
The autopilot daemon now detects a post-migration engine flip
(`autopilotEngineIdentity` per-tick compare → clean exit for supervisor
relaunch), but `gbrain serve` (MCP) and a standalone `gbrain jobs work`
worker hold their engine handle indefinitely and keep writing into the
abandoned source engine after a flip — the same silent-divergence class,
still open on those planes (adversarial-review catch). Fix shape: the same
boot-identity compare in their main loops.
- [ ] **P2 — DB-visible pause for cross-host workers.** The pause marker now
fences local job pickup (pre-claim check + post-claim release-back in
`src/core/minions/worker.ts`), but the marker is a local file: a worker on
ANOTHER host or container pointed at the same Postgres brain never sees it
and keeps claiming jobs during a migration copy (its in-flight work IS
visible to the drain via `minion_jobs`/lock rows; new claims are the gap).
Fix shape: a row in a control table (or a pause flag in `gbrain_cycle_locks`)
that the claim query itself honors — atomic with claiming, visible
cluster-wide.
- [ ] **P2 — Route file→symlink typechanges to delete.** `buildSyncManifest`
maps git status `T` to modified, but import-file deliberately SKIPS symlinks
(the exfil guard), so replacing an indexed file with a symlink leaves the
old content indexed forever with no delete. Fix shape: when the post-change
path is a symlink, emit a delete instead of a modify.
- [ ] **P3 — Surface daemon-internal degradation in status.** A daemon stuck
in the reconnect-retry loop (crash-classified errors) keeps heartbeating,
so `--status` reads fresh while zero work happens. Fix shape: a breadcrumb
file with consecutive-failure count that showStatus reads.
- [ ] **P3 — Extract a shared `seedBrain` test helper.** The keyless-PGLite +
tmp-HOME + shimmed-PATH setup is duplicated between
`test/autopilot-launchd-lifecycle.serial.test.ts` and
`test/agent-scheduler-contract.serial.test.ts` (review-army maintainability
finding). A third harness-e2e file (the PR-B tier) should force the
extraction into `test/helpers/`; don't extract before then — two instances
is a coincidence, three is a pattern.
- [ ] **P3 — Name the quiesce protocol's magic numbers.** `migrate-engine.ts`
and `autopilot.ts` share three constants by value, not by name: the 600s
heartbeat-freshness window, the 35s default grace, and the daemon's paused
fast-poll interval. Hoist into `src/core/autopilot-paths.ts` (the shared
leaf) as named exports so the two planes can't drift.
- [ ] **P3 — Migration manifest rows don't carry content_hash.** A resume
trusts `(source_id, slug)` membership in `completed_slugs`; a page edited
BETWEEN the failed run and the resume is skipped with its stale copy left on
the target (review-army data-migration finding; pre-existing design, not a
regression). Fix shape: stamp `content_hash` per completed entry and re-copy
on mismatch during resume.
- [ ] **P2 — Keyless `gbrain dream` contract test.** The documented nightly cron
(INSTALL_FOR_AGENTS.md Step 7) runs `gbrain dream` unconditionally, and the cycle's
embed phase hits the same `EmbeddingDisabledError` class that broke the documented
sync-and-embed chain on keyless brains (fixed in `runEmbed` for the `--stale`
spelling; `test/agent-scheduler-contract.serial.test.ts` pins it). Nobody has verified that a
full keyless dream exits 0 — if any phase surfaces the disabled-embeddings error as a
phase failure, the documented nightly cron is broken identically for every
`init --no-embedding` install. **Where to start:** `src/core/cycle.ts` embed phase +
`src/commands/dream.ts` exit-code handling; test shape mirrors
`test/agent-scheduler-contract.serial.test.ts` (keyless PGLite brain, real CLI spawn, exit-code
assertion). Surfaced by the harness-e2e outside-voice review.
## BrainBench follow-ups (filed v0.44.0.0, Cathedral 2)
Deferred from the BrainBench wave (eng-reviewed; plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-bright-firefly.md`).
- [ ] **`--live` agent-in-the-loop know-to-ask.** Replay fixtures with a real model deciding whether to issue retrieval calls; grade the agent, not just the deterministic reflex. Pre-registered in `docs/eval/BRAINBENCH.md` (the v1 metric grades the injection decision, which IS the shipped mechanism). Needs: seeded N-repeat methodology for model stochasticity + budget rails. Priority: P2.
- [ ] **Intrusion-budget gating calibration.** `avg_injected_tokens` is reported, non-gating (decision 18) — a wrong threshold is worse than none. After a few weeks of scoreboard data across PRs, pick calibrated per-seam thresholds and promote it to a gated metric. Priority: P2.
- [ ] **Flip contract adapters to production — claude-code half now unblocked.** `adapters/claude-code.ts` exports the UserPromptSubmit hook wire types; the real hook (`gbrain hook user-prompt`, shipped with the bootstrap lane and extended with cross-turn dedupe + the channel feedback loop in the cathedral-3 convergence) swaps the in-process transport for an exec of the hook script and flips `seam: 'contract'``'production'` with continuous bench numbers. Note the production hook also exercises transcript-based dedupe, which the memoryless contract row deliberately doesn't. Same for codex fragments when that integration lands. Priority: P1 (the claude-code integration has landed; this is now standalone-actionable).
- [ ] **Cathedral 1 conformance-kit fixture import.** The memory-verbs conformance scenarios convert to BrainBench fixtures via the published `evals/brainbench/schema/fixture.schema.json` once `garrytan/cathedral-1` merges ("conformance tests double as BrainBench seed fixtures", decision log 2026-06-12). Free corpus growth from already-reviewed scenarios. Blocked by: cathedral-1 on master. Priority: P2.
- [ ] **Live-embeddings fidelity mode (`--embeddings`).** Hermetic CI grades the keyword/alias arms only (disclosed); an opt-in mode seeding real embeddings would grade write-back/continuity retrieval through the vector path. Same budget rails as `--llm`. Priority: P3.
- [ ] **Community fixture intake + competitor adapters.** The TD1 remainder after the generated corpus absorbed in-PR growth: an `external-authors/`-style intake path for contributed fixtures (validator + privacy guard already gate them) and adapters for non-gbrain memory systems against the published schemas, enabling true head-to-head rows in the gbrain-evals scorecard. Priority: P3.
- [ ] **JSON-Schema ↔ validator parity test.** `evals/brainbench/schema/*.schema.json` is the foreign-runner contract but only existence/parse is pinned — the schemas could drift from `fixtures.ts`'s validator silently. Needs a JSON-schema validator dep (ajv) or a hand-rolled subset checker; deferred to avoid a new dependency mid-wave. Priority: P3.
- [ ] **`eval compare` markdown rendering for `mode: 'n/a'` rows.** EvalRunRecord v3 records brainbench under 'n/a'; the markdown renderer iterates SEARCH_MODES only, so those rows surface in `--json` output but not the mode table (documented in the v3 docstring). Add an un-grouped section to `renderMarkdown`. Priority: P3.
- [ ] **Periodic re-baselining (the ratchet doesn't auto-tighten).** Improvements aren't banked into master's baseline until a PR updates it, so a regression back to a stale baseline level passes. Documented as an accepted residual in `docs/eval/BRAINBENCH.md`; the fix is an operator habit or a scheduled job that re-runs `--update-baseline` after metric-improving merges. Priority: P3.
- [ ] **Hermetic-ize the 7 env-sensitive LLM-availability tests.** `test/think-gateway-adapter.test.ts`, `test/conversation-parser/llm-base.test.ts`/`llm-fallback.test.ts`, `test/doctor-ze-checks.test.ts` assert behavior "when ANTHROPIC_API_KEY is unset" by reading the live process env — they fail on any dev shell that exports provider keys (verified failing on clean master in such a shell; green in keyless CI). Stub/save-restore the env per test so local runs match CI. Priority: P2.
## #2416 follow-ups (query-steering wave)
- [ ] **P2 — MCP-envelope `hint` field for concept-shaped `search` calls.**
**What:** surface the concept→query nudge to remote/MCP agent callers, not
just the CLI. **Why:** MCP agents are the primary misrouting class the
#2416 issue describes; the shipped CLI stderr nudge covers the caller class
*least* at risk. **Context:** the `search` op returns a bare
`SearchResult[]` (`src/core/operations.ts` — both return sites), so a hint
needs an envelope change that ripples into `formatResult`, MCP
serialization, and array-shape tests — deliberately kept out of the atomic
#2416 commit. The pure classifier already exists
(`looksConceptShaped`/`conceptNudge` in `src/core/search/query-intent.ts`);
only the transport is missing. Consider a sibling metadata channel (like
`_meta.metric_glossary`) rather than changing the array shape.
**Depends on:** agreeing an envelope pattern that doesn't break existing
MCP consumers.
## MEMORY_VERBS v1 follow-ups (filed v0.43.0.0 — Cathedral 1)
Deferred from the Cathedral 1 ship (CEO review, EXPANSION mode). Both are
additive to the frozen v1 contract — neither breaks it. See plan + GSTACK
REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-agile-iverson.md`
and the scope record at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-06-12-memory-verbs-protocol.md`.
- [ ] **P3 — external-implementation certification PROGRAM.** The conformance
TOOLING shipped (`gbrain protocol conformance --target <endpoint>`); the
PROGRAM around it (badges, a registry of conformant implementations, listed
third-party servers) waits for a second implementation to exist. **Why:** the
protocol-not-product thesis only pays off once someone else implements
MEMORY_VERBS; until then a certification program certifies an empty set.
**Where:** new — would build on `src/commands/protocol.ts` conformance output.
- [ ] **P3 — persistent open-threads model for the entity card.** v1 derives
`entity.open_threads` from active commitment-kind facts + recent timeline
entries (best-effort, possibly empty). A richer model (a real threads table:
conversation id, opened/closed state, last activity) would make open-threads
authoritative. **Why:** the card's open-threads field is the weakest signal
in v1; a first-class threads store would make it load-bearing. **Where:**
`src/core/verbs/entity-card.ts` open-threads assembly + a new schema table
(additive — the card field already exists, so this is a quality upgrade, not
a contract change).
- [ ] **P2 — `recall` filter composition vs the spec (found by the v0.43.0.0
cross-model doc review).** The handler dispatch is first-match
(`supersessions` > `entity` > `session_id` > `since`), so `since` is
silently ignored when `entity`/`session_id` is supplied, and `limit` has no
server-side cap. Either compose the filters (additive — the spec's "filters
the FACTS arm" wording already reads that way) or spell the precedence out
in `docs/protocol/MEMORY_VERBS_v1.md`. **Where:** the `recall` handler in
`src/core/operations.ts`.
- [ ] **P3 — widen `synthesize`'s `unavailable` mapping.** Only the
missing-key gateway warning maps to the `unavailable` error today; other
no-usable-model failures can surface as `internal` (contract-legal but less
actionable) or, worst case, a stubbed success. Audit the gateway failure
modes and map every model-unusable path to `unavailable` with a fix
suggestion. **Where:** the `synthesize` handler in `src/core/verbs.ts`.
## Fix-wave 1 follow-ups (upgrade-wedge + trust-seam wave, 2026-08)
Deferred from the un-wedge-v121 hotfix wave (eng review + codex outside voice
CLEARED; every item an explicit review decision). Waves 26 of the sequence are
planned separately (provider-compat rescue is next; its original 2026-07-24
DeepSeek-deprecation deadline has now PASSED — re-verify each cluster against
master before starting, several fixes landed independently).
- [ ] **P2 — Shared strict `parseFlags` helper as the #2185 end-state (eng
review 2B).** This wave ships the generated known-flags registry +
pre-dispatch validator (parser and registry can drift only until the
freshness guard fires). The structural end-state migrates commands onto one
shared strict parser so parser == registry by construction; mechanical but
touches 60+ command files — its own PR. Where: `src/commands/*.ts`,
`src/cli.ts`, `scripts/generate-flag-registry.ts` (retires).
- [ ] **P2 — `whoknows` CLI routing (surfaced by the #2035-class sweep).**
`handleCliOnly`'s `whoknows` case (the dedicated CLI renderer with
thin-client routing) is dead code — the command resolves via the
`find_experts` op alias, and adding it to CLI_ONLY trips the alias-collision
guard. Decide the intended surface alongside PR #2509 (whoknows --explain
per-result factor breakdown) and delete whichever lane loses. Where:
`src/cli.ts`, `src/commands/whoknows.ts`, PR #2509.
- [ ] **P3 — #2544 second half: per-put_page `getAllSlugs` full scan.** The
getChunks egress half shipped in this wave (explicit non-vector column
list). The remaining Postgres-egress cost is put_page's per-call
`getAllSlugs` table scan — needs a targeted existence probe or cached slug
set. Where: `src/core/operations.ts` put_page path, both engines.
- [ ] **P3 — #1558 admin-UI register form.** The `/admin/api/register-client`
API now accepts `source` + `federatedRead` (this wave, PR #2016 absorbed);
the admin SPA form fields + `/admin/api/sources` picker are the UI layer.
Where: `src/commands/serve-http.ts` admin SPA blob.
- [ ] **P3 — jsonb-integrity surfaces: batch + share (ship-review follow-up).**
doctor's jsonbIntegrityCheck runs 2 queries per target (16 round-trips) and
duplicates the TARGETS table with repair-jsonb (already drifted once on the
jsonPayloadOnly predicate before being mirrored by hand). Batch the counts
into one UNION ALL query and extract a shared targets constant
(src/core/jsonb-integrity-targets.ts) consumed by both. Where:
`src/commands/doctor.ts` jsonbIntegrityCheck, `src/commands/repair-jsonb.ts`.
- [ ] **P3 — register-client HTTP-level e2e (ship-review follow-up).** The
source/federatedRead lane is covered by unit normalizers + a structural
route pin; a DATABASE_URL-gated serve-http e2e (register with bindings →
assert stored client via /admin/api/agents; invalid source → 400
invalid_source) closes the wire-level gap. Where:
`test/e2e/serve-http-oauth.test.ts`.
- [ ] **P3 — get_chunks `__all__` sentinel narrows to 'default' (red-team,
Wave 3 territory).** `sourceScopeOpts` returns `{}` for a trusted local
`--source __all__` caller (documented "spans the brain"), but both engines'
getChunks map empty scope to the 'default' floor — the one read op where
`{}` is reinterpreted. Fold into the Wave 3 source-federation cluster's
`__all__` work (an explicit unscoped signal in the engine signature, or
handler-side expansion for trusted callers). Where: `src/core/operations.ts`
get_chunks, both engines' getChunks.
- [ ] **P3 — #2536 wedged-migration diagnostics.** The v121 wedge aborted
initSchema BEFORE runMigrations, so the wedged-migration diagnostics row was
never written — operators got a bare SQL error with no remediation hint.
Write the diagnostics row (or a stderr remediation block) from the blob-replay
catch path too. Where: `src/core/migrate.ts`, `src/commands/apply-migrations.ts`.
## WAL-repair wave follow-ups (#223/#1670/#2575)
- [ ] **P2 — gate auto-repair on an unclean-shutdown marker (adversarial F7).** The classifier
deliberately over-matches (`RuntimeError`/`unreachable``wasm-abort`). If an unclean
shutdown leaves a REPLAYABLE WAL tail (normal crash recovery would restore those committed
txns) and the reopen then fails on a transient WASM error (OOM), auto-repair fires, layout
validation can't tell torn from replayable, and resetWal discards the tail while the notice
says "data preserved." Bounded today (backup always taken + restore + honest failure + repeated
attempts capped), but a false-positive-with-successful-retry silently drops committed data.
Fix direction (probe-verified): PGLite removes `postmaster.pid` on clean close, so gate AUTO
repair (not the manual command) on `postmaster.pid` presence — a clean dir that aborts is not
torn-WAL. Requires making the serial regression test stamp a `postmaster.pid` before corrupting
(it currently clean-disconnects then corrupts, which the red-team flagged as unfaithful anyway).
Needs a recall/precision call before landing.
- [ ] **P3 — live non-gbrain PGLite consumer not caught by the postmaster.pid liveness guard
(adversarial F8).** PGLite writes a sentinel `postmaster.pid` of `-42`; the liveness refusal in
`validateWalRepairTarget` requires `pid > 0`, so it protects native Postgres dirs but not a
non-gbrain pglite app that has the dir open (such an app writes no `.gbrain-lock`). Deliberate
misuse of `pglite-repair --path <foreign pglite dir>` required. Option: refuse when
postmaster.pid holds pid ≤ 0 with a very recent mtime, or document the boundary.
- [ ] **P3 — mixed-version torn-lock read (adversarial F10 residual).** The heartbeat + initial
lock writes are atomic (tmp+rename) now, but an OLD gbrain binary writing heartbeats IN PLACE
while a NEW binary poll-reads can still catch a torn read → corrupt-lock verdict → a live
holder's lock reaped → two writers (the #2348 class, version-skew-triggered). The reap marker
quarantines repair, not the concurrent open. Cheap hardening: double-read the lock file (~50ms
apart) before declaring it corrupt.
- [ ] **P2 — graceful PGLite close on SIGTERM for the remaining long-running paths.**
The torn-WAL genesis this wave repairs is an unclean shutdown: `src/core/process-cleanup.ts`
releases locks on SIGTERM but never closes the PGlite handle, so `serve` / `jobs work` /
`sync` killed mid-write (macOS-upgrade reboot, `systemctl stop`) leave the WAL torn.
Autopilot already ships the pattern (d2fd1f29, #3178/#1872: `registerCleanup('autopilot-engine-close', ...)`
— abort in-flight work → ≤2s bounded wait inside the 3s cleanup deadline →
`engine.disconnect()`, double-call safe; rationale comment at autopilot.ts:438-452).
Extend that exact pattern to the remaining long-running PGLite paths (register in
connect()/command scope; dedupe so autopilot doesn't double-close), pinned by a serial
lifecycle test. Interacts with #2084 exitCode containment + #1337 close ordering — read
those comments in pglite-engine.ts first. Auto-repair makes recurrence self-healing
meanwhile, so this is prevention, not recovery.
- [ ] **P3 — pglite upgrade blocker tracker.** Two couplings make a "routine" pglite bump a
breaking change: (a) pglite ≥0.5 removes the `@electric-sql/pglite/vector` export that
`pglite-engine.ts` imports (verified against npm); (b) the pg_resetwal port
(`src/core/pglite-resetwal.ts`) is coupled to the PG17 pg_control layout
(`PG_CONTROL_VERSION` 1700 — guarded at runtime by `WalResetUnsupportedError`, so a
mismatched bump makes the repair tool refuse every dir rather than corrupt, but it still
means the repair feature silently dies). Any future pglite upgrade wave must revisit BOTH
together and re-derive the ControlFileData offset table for the new PG major.
## serve --http takes-holders + agent-voice hardening follow-ups (filed v0.42.74.0)
Deferred from the #2529/#2477 security-fix wave (plan-eng-review + codex outside
voice CLEARED). None block the wave.
- [ ] **P2 — Per-OAuth-client `takes_holders` storage (#2529 follow-up).** Legacy
bearer tokens honor `access_tokens.permissions.takes_holders` through
`verifyAccessToken`; OAuth clients have no equivalent column on `oauth_clients`,
so OAuth-minted tokens fail closed to `['world']`. Needs a schema migration
(`oauth_clients.takes_holders` JSONB or TEXT[]) + a `register-client` flag +
the `verifyAccessToken` JOIN projection. Include surfacing the EFFECTIVE
takes-holder scope in `whoami` output as part of this follow-up, so operators
can self-diagnose the legacy-vs-OAuth semantic split instead of reading docs.
Where: `src/schema.sql`, `src/core/migrate.ts`, `src/core/oauth-provider.ts`,
`src/commands/auth.ts`, `src/core/operations.ts` (whoami).
- [ ] **P3 — agent-voice Host-header allowlist (DNS-rebinding hardening).** The
#2477 fix ships default-deny CORS + an Origin gate on `/session`/`/tool`, but
the gate derives self-origin from the `Host` header, so a DNS-rebound page
(attacker origin whose host resolves to the operator's loopback) still passes.
Validate `Host` against `localhost`/`127.0.0.1`/operator-configured hosts and
403 otherwise; slots beside `originAllowed()` in the router. Issue #2477
explicitly deferred this. Where: `recipes/agent-voice/code/server.mjs`.
- [ ] **P3 — Debounce `last_used_at` in the oauth-provider legacy path.** The
legacy branch of `verifyAccessToken` fires an unconditional
`UPDATE access_tokens SET last_used_at = now()` on EVERY request, while the
legacy HTTP transport debounces the same write to once per 60s via a WHERE
clause (`src/mcp/http-transport.ts` validateToken). Apply the same pattern —
one fewer write per request on the `serve --http` hot path.
Where: `src/core/oauth-provider.ts`.
## v0.42.67.0 follow-ups (Windows build tooling)
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
`bash` prefix on the 33 `package.json` check commands). Both items are newly
observable: before that release these checks never executed on Windows at all,
so nothing about their runtime was measurable.
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
per-file shell loops, which is far slower under Windows process creation. Either raise the
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
copy, or skipping the check with a clear message when symlink creation is unavailable.
## community fix-wave follow-ups (filed v0.42.60.0)
- [x] **P2cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
- [ ] **P1take-writes source scoping fails open when source resolution errors (#2684 residual).**
`resolveTakesSourceId` (src/commands/takes.ts) swallows resolution errors and returns
`undefined`, which falls back to the unscoped slug-only page lookup — so an invalid
`GBRAIN_SOURCE` (or a broken dotfile chain) silently restores the pre-#2698 cross-source
write behavior on multi-source brains. Decide fail-closed semantics: error out when a
source was explicitly requested but doesn't resolve; keep the unscoped fallback only for
brains with no source configuration at all. Add a regression test for the invalid-source
path. Found by cross-model adversarial review during the v0.42.60.0 release ship.
- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent`
before `models.tier.subagent`). Implemented: `checkSubagentCapability` now resolves
`models.subagent` before tier/default fallbacks and has regression coverage.
before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered.
## v0.42.59.0 follow-ups (five-fix rollup #2735#2739)
@@ -369,20 +69,17 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
native providers (anthropic/openai/google) do. To make expansion work on
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
chat-capable recipes AND add a `generateObject``generateText` capability fallback for
backends without strict structured outputs. Feature-shaped; overlaps the general
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
The general OpenAI-compat proxy story.
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
@@ -572,42 +269,6 @@ Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at
filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates
through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan.
## Harness hook lane follow-ups (filed from the cathedral-3 convergence)
Filed when the cathedral-3 branch converged its push-adapter work into the
#3975 hook lane (feedback loop + cross-turn dedupe for `gbrain hook
user-prompt`). Context: the hook lane now logs channel-attributed volunteer
events at the IPC delivery point and dedupes via the transcript's
`hook_additional_context` attachments.
- [ ] **P3 — PostToolUse / mid-turn push adapter, evaluated against per-channel stats.**
The user-prompt hook fires at prompt time only; entities that first appear mid-turn
in tool output (a file opened, a person named in a search result) get no pointer
until the NEXT prompt. Harnesses expose a PostToolUse hook, but it fires dozens of
times per turn (one `gbrain hook` process spawn each). Now that the feedback loop
exists, the per-channel `--stats` precision + volume data is exactly the evidence
needed to decide. **Trigger:** claude-code channel stats showing healthy precision
plus user reports of "it only noticed on my next message". **Start:**
`src/commands/hook.ts` (the event already has a dispatch slot pattern),
`src/core/bootstrap/hooks.ts` registration writers.
- [ ] **P3 — engine-uniform IPC listener (Postgres serves).** serve's resolve/turn_context
socket is PGLite-gated (`src/mcp/server.ts`: `cfg?.engine === 'pglite'`), so on a
Postgres brain `gbrain hook user-prompt` short-circuits (`no_pglite_path`) and the
hook lane is PGLite-only. Extending the listener needs (a) a canonical per-connection
socket path for brains with no data dir (e.g. `~/.gbrain/run/resolve-<hash12(database_url)>.sock`,
0700 dir) and (b) a secret-file home for `turn_context` auth (same hash-keyed run dir).
The cathedral-3 branch prototyped (a) as `resolveSocketPathForConfig` (see branch
history at commit 2350294c) before the convergence dropped it pending the secret
design. **Trigger:** a Postgres-brain user asking why hooks stay silent. **Start:**
`src/core/context/resolve-ipc.ts` socket-path helpers + `src/mcp/server.ts` listener gate
+ `src/commands/hook.ts:no_pglite_path` branch.
- [ ] **P3 — thin-client remote push route.** Thin-client installs (remote_mcp) have no
local engine and no serve socket — every push channel is dead there and only the
hook's typed heartbeat reason says why. The natural route is `volunteer_context`
over the remote MCP transport (`callRemoteTool`), rate-limited per prompt.
**Trigger:** thin-client adoption of bootstrap. **Start:** `src/commands/hook.ts`
user-prompt branch + `src/cli.ts` remote-tool plumbing.
## gbrain#2095 push-based context follow-ups (v0.43+)
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
@@ -621,22 +282,20 @@ are the bar). Plan + GSTACK REVIEW REPORT at
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
**Blocked by:** a real consumer (revisit when one exists).
- [ ] **P3 — policy skill (recipe) for push-context.** The doctor-check half of
this item shipped with the harness hook lane: `volunteer_channels`
(`src/commands/doctor.ts:checkVolunteerChannels`) reads the events table
per-channel on both the local and remote doctor. Remaining scope: if
`volunteer-context --stats` adoption shows agents not discovering the
surface, ship a `push-context` recipe (mirror `recipes/retrieval-reflex/`).
**Where:** `recipes/`.
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
needed doctor visibility because silent failure was invisible; volunteer is
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
agents not discovering the surface, ship a `push-context` recipe (mirror
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
**Where:** `recipes/`, `src/commands/doctor.ts`.
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
If MCP callers accumulate parsing bugs, add a structured array param beside it.
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver
(`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE
ANY('%/...')`) predates #2095 but now runs per turn on four channel surfaces
(reflex window, volunteer_context, watch, and the harness-hook `turn_context`
lane) federated across sources. Neither
ANY('%/...')`) predates #2095 but now runs per turn on three channels
(reflex window, volunteer_context, watch) federated across sources. Neither
the leading-wildcard suffix arm nor `lower(title)` is index-served. If
per-turn latency telemetry on large brains comes back hot: add
`(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or
@@ -2628,25 +2287,10 @@ at plan time and got carved out:
via `buildPerSourceBindings`. Document workaround: register
source-scoped OAuth clients.
- [x] **v0.41+: T20 — extends-chain merging in registry.ts.** DONE (#1749).
`resolvePack` now merges parent → child (child-wins) for the six
ingest/query-shaping fields (`page_types`, `link_types`,
`frontmatter_links`, `enrichable_types`, `filing_rules`, `takes_kinds`)
plus `borrow_from` materialization, in `src/core/schema-pack/merge.ts`.
The cascade was transparent (consumers already read `resolved.manifest`),
not per-consumer. `phases`/`calibration_domains` deliberately excluded —
see the P3 follow-up below.
- [ ] **P3: explicit opt-in to inherit `phases` / `calibration_domains`.**
T20 excludes these two from the child-wins merge because they gate real
cycle execution (`cycle.ts` `packDeclaresPhase`) and the manifest
contract says each pack declares its own participation explicitly —
auto-inheriting would silently make a child run cycle phases it never
requested. Multi-level lens packs (`gbrain-everything`) therefore still
re-declare them by hand. If that redeclaration becomes painful, add an
explicit manifest flag (e.g. `inherit_phases: true`) so a pack author
opts in consciously. Depends on: T20 (landed). Start in
`src/core/schema-pack/merge.ts` (`mergeInheritedManifest`).
- [ ] **v0.41+: T20 — extends-chain merging in registry.ts.**
`registry.ts:167` documents the gap. Implementing full child-wins
merge cascades through every consumer of `manifest.page_types`. ~1
day CC.
- [ ] **v0.41+: T21 — comment-preserving YAML emitter.**
v0.40.7.0 emitter does NOT preserve comments. Authors who care
@@ -2708,9 +2352,8 @@ contributor traps.
- [ ] **P2: Document `FREE_LOCAL_RERANK_PROVIDERS` invariant.** `src/core/budget/budget-tracker.ts:lookupPricing`
returns `{input:0, output:0}` for any model id under the `llama-server-reranker:`
provider on the rerank kind. The contract relies on all callers going through
`gateway.rerank()`'s own model-list check (rerank-specific; it validates the
model exists before pricing fires — note this was never `assertTouchpoint`,
which checks provider touchpoints only). Theoretical bypass: a future caller that
`gateway.rerank()`'s `assertTouchpoint`-with-extended-models check (which validates
the model exists before pricing fires). Theoretical bypass: a future caller that
reserves directly against BudgetTracker with `kind: 'rerank'` and an arbitrary
`llama-server-reranker:<anything>` model id gets free pricing. Fix: code comment
documenting the invariant, OR move the freeness check to gateway.rerank() where
@@ -2906,7 +2549,7 @@ contributor traps.
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
- [ ] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
@@ -3840,99 +3483,28 @@ After the sweep, both should be fixable and renameable back to plain `*.test.ts`
## claw-test E2E (v0.22.16 follow-ups)
### ~~Hermes runner — `src/core/claw-test/runners/hermes.ts`~~ DONE (hermes-harness wave)
Shipped: `HermesRunner` (`hermes -z <brief>`, `$HERMES_BIN` > `which hermes`,
`HERMES_HOME` env-allowlist delta) + the full hermes install door
(`test/e2e/install-real-hermes.serial.test.ts`, opt-in-gated) + the label-gated
`hermes-door` CI job in heavy-tests.yml. The cross-agent
`gbrain friction diff --base openclaw --compare hermes` payoff shipped in the
same wave (below). Observed-CLI pins live in `docs/mcp/HERMES-CLI-PIN.md` and
`docs/mcp/HERMES.md`.
### Hermes runner — `src/core/claw-test/runners/hermes.ts`
**Priority:** P2
**What:** Add a Hermes implementation of the `AgentRunner` interface. v1 ships only OpenClaw; v1.1 lands hermes once we have real friction reports from openclaw to validate the contract against.
**Why:** Cross-agent diff (`gbrain friction diff --base openclaw --compare hermes`) is the highest-leverage next signal. Friction unique to one agent vs common-to-both separates "agent contract bug" from "gbrain bug" automatically.
**Effort:** S (CC ~30m). Depends on: v1 openclaw runner producing real friction reports first.
---
### Friction analytics suite — `trend` / `migration-stub` (diff SHIPPED)
### Friction analytics suite — `diff` / `trend` / `migration-stub`
**Priority:** P2
**What:** Two remaining `gbrain friction` subcommands deferred from v1
(`diff` shipped in the hermes-harness wave — see `src/commands/friction.ts`):
**What:** Three new `gbrain friction` subcommands deferred from v1:
- `gbrain friction diff --base <run-or-agent> --compare <run-or-agent>` (cross-agent comparison; ~80 LOC)
- `gbrain friction trend [--since <version-or-date>] [--phase <name>]` (time-series across runs; ~60 LOC)
- `gbrain friction migration-stub [--threshold N]` (clusters friction by phase + tokens, emits `skills/migrations/v[N+1].md` stub; ~150 LOC)
**Why:** Turns point-in-time reports into a slope. Pairs with the v1.1 public scoreboard.
**Effort:** M (CC ~1.5h total).
---
### Promote hermes-door soft probes to hard assertions + build the REAL cron test
**Priority:** P2
**What:** Two follow-ups now that the hermes CLI surface is pinned (v0.20.0,
`docs/mcp/HERMES-CLI-PIN.md`): (1) promote the door's logged-evidence probes
(`hermes mcp list` output shape; session-artifact tool-call traces under
`<home>/.hermes/`) to hard assertions once a couple of CI runs confirm their
stability across hermes releases; (2) build the real cron pairing test — the
surface is fully non-interactive (`hermes cron create [--name N] [--no-agent]
[--script PATH] <schedule> [prompt]` + `hermes cron tick` runs due jobs once
and exits) — create a job that runs `gbrain sync --json`, tick, and assert the
sync actually executed against the run's brain. (A self-skipping probe was
deliberately CUT in review: a test that cannot fail is not coverage.)
**Why:** INSTALL_FOR_AGENTS.md's recurring-jobs step has zero coverage; the
evidence sweep is the promotion signal the door already logs.
**Effort:** S-M (CC ~45m). Depends on: first labeled hermes-door CI runs.
---
### Wire the orphaned `voice-agent-install` ScenarioKind
**Priority:** P2
**What:** `test/fixtures/claw-test-scenarios/voice-agent-install/` carries the
richest install-assertion template in the repo (60-line expected.json:
filesystem manifest, `.gbrain-source.json` sha256s, resolver rows, PII
blocklist, health probe, tiered soft-fail) but `scenario.json` declares
`kind: "voice-agent-install"`, which `ScenarioKind` rejects — the fixture
cannot load. Extend `ScenarioKind` + `loadScenario` + a `postInstallHook`
implementation so the scenario runs.
**Why:** Integrations-recipe install coverage (the `gbrain integrations
install` path) has a fully-designed scenario sitting dead.
**Effort:** M (CC ~1h). Integrations-lane work, deliberately kept out of the
hermes-harness wave.
---
### Cold-install container test — fill the `tests/docker/bootstrap-e2e.sh` placeholder
**Priority:** P3
**What:** heavy-tests.yml carries a gated no-op step for
`tests/docker/bootstrap-e2e.sh` (networkless cold-machine container install of
gbrain itself: global install, PATH discovery, migrations). The file doesn't
exist. Write it.
**Why:** The agent-platform door tests (claude/codex/hermes) all deliberately
run gbrain from the dev tree / compiled binary — none of them proves gbrain's
own cold install. That gap was re-flagged in the hermes-harness wave's outside
review and scoped OUT of that wave on purpose.
**Effort:** M (CC ~1-2h, docker).
---
### BrainBench hermes adapter
**Priority:** P3
**What:** ~50-100 lines in `src/eval/brainbench/adapters/hermes.ts` + an
`ALL_HARNESSES` entry + baseline cells in `evals/brainbench/baselines/main.json`.
**Why:** Cross-harness memory-conformance coverage for the third platform.
Eval seam (memory conformance), NOT install — kept out of the install wave on
purpose; needs baseline-governance care per the BrainBench gate rules.
**Effort:** S-M (CC ~1h + baseline runs).
**Effort:** M (CC ~2h total).
---
@@ -3952,7 +3524,7 @@ purpose; needs baseline-governance care per the BrainBench gate rules.
### Real v0.18 SQL dump for upgrade scenario
**Priority:** P2
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. Both scripted and live runs now FAIL LOUDLY on the missing dump (a silent skip used to init a current database and false-green the "upgrade"), so the shipped scenario is unrunnable until the dump lands. Generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. The harness gracefully no-ops the seed phase when absent, so the scenario currently behaves like fresh-install. v1.1: generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
**Why:** Without a real seed, the scenario doesn't actually exercise the migration chain forward-walk. That's the whole point of the upgrade scenario — proves issue #239/#243/#266/#357 class regressions stay fixed.
@@ -5205,312 +4777,3 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
+ the models-doctor tests.
**Depends on:** nothing.
## Agent-bootstrap wave follow-ups (filed at build time)
- [ ] **P2 — repoPhaseComplete is single-workspace (one global receipt).** The
no-daemon push gate binds to the one `receipt.repo_url`, so with two bootstrap
workspaces sharing a gbrain home, workspace B's `bootstrap repo` overwrites the
receipt and permanently leaves A's per-turn/session-end pushes at
`push_deferred_repo_pending`. Fails CLOSED (defers, never mis-pushes) and
matches the v1 single-workspace contract, but the per-turn push made it more
visible. Fix = per-root repo binding (a receipt map or a per-root marker).
Surfaced by both v0.45.9.0 adversarial reviewers.
- [ ] **P2 — visibility ladder subprocess/body bounds.** `runWithTimeout`
(`src/core/repo-visibility.ts`) races the `gh`/`git` probe against a timer but
doesn't kill the raced child, and the anon-probe `res.text()` buffers the whole
(operator-configured-origin) body before slicing. Bounded in practice by the
detached push child's lifetime, but a proper fix kills the raced process and
caps the body read. Filed from the v0.45.9.0 Codex adversarial pass.
- [ ] **P3 — `config set` for the file-plane hook-lane keys is engine-bound.**
`runConfig` dispatches through the engine path, so `gbrain config set
push.allow_unverified_remote true` can fail while a live PGLite serve holds the
writer lock — the documented recovery command, unavailable exactly when needed.
The env-var form (`GBRAIN_ALLOW_UNVERIFIED_REMOTE=1`) is the cloud path and needs
no engine, so this is convenience-only; fix = route these two keys through the
no-engine CLI dispatch. Filed from the v0.45.9.0 Codex adversarial pass.
- [ ] **P3 — plugin-based hook distribution for Claude Code.** Ship gbrain's
hooks as a Claude Code plugin (`hooks/hooks.json` + `.claude-plugin/plugin.json`
manifest, installed via the plugin marketplace flow) instead of two settings
files. Plugins merge hooks first-class across scopes and update centrally —
it would REPLACE both current carriers (repo-committed `.claude/settings.json`
for cloud installs + gitignored `settings.local.json` for local), so it must
migrate, not join; a third simultaneous carrier would double-fire events.
Cons: needs marketplace repo hosting; enterprise `allowManagedHooksOnly`
policies can block plugin hooks entirely. Start at
`src/core/bootstrap/hooks.ts` (both writers + the dedupe rule live there).
Filed from the cloud-DX eng review (v0.46.x wave).
- [ ] **P3 — watch Claude Code Channels as the push path for
volunteer_context/signals.** Channels (research preview) push external events
into a LIVE session — the native version of gbrain's push-context lane
(`docs/guides/push-context.md`). Not actionable today: delivery requires an
always-on session plus an Anthropic-allowlisted channel plugin. Revisit when
channel-plugin distribution opens; the win is replacing per-turn pull with
event push for signals/reflex windows. Filed from the cloud-DX eng review.
- [ ] **P1 — enforce op scope/localOnly on the stdio MCP dispatch when no auth
context is present, and consider a narrower default surface for pull-mode
harness registrations.** HTTP dispatch enforces `scope`/`localOnly` before
handlers run; the stdio surface should reach parity so a registration that is
user-global by host design (no per-project scoping available) does not expose
more authority than the session needs. Surfaced by the v0.45.x ship
adversarial pass (cross-model); pre-existing behavior, not introduced by the
Codex scope-consent fix — that fix's prose now states the read+write reality
honestly. Needs its own design pass (interaction with `--surface` pinning,
MEMORY_VERBS, and the trust-boundary invariant in CLAUDE.md).
- [ ] **P2 — consent-key answers vs the A8 confirm gate.** Decide whether
`consent: true` bank keys should be exempt from `setAnswer`'s confirmation
invalidation (`src/core/bootstrap/interview.ts:308-309` `[A8]` deletes
`state.confirmed` on ANY set) so operational consents can be recorded at their
designed phase-contextual moment post-confirm without regressing
`bootstrap status` to "answers complete but not confirmed" (status.ts
interview detector). Deferred from the Codex MCP-scope fix (eng review option
3B chose prose realignment instead: the runbook now records `MCP_SCOPE` in
phase 3, pre-confirm, so the confirm hash covers it). An exemption touches a
tamper-tripwire — a post-confirm flip of `PERSIST_CRON` (background-push
consent) would no longer invalidate anything — so it needs its own
adversarial review before landing. Also cover the healing half: pre-fix
installs that recorded `MCP_SCOPE` at the old wire-phase moment have a
permanently-invalidated confirm, and `bootstrap status` can't distinguish a
consent-key invalidation from a tampered answer set — a status detail for
that case would stop resumed installs being steered into a redundant
re-confirm loop (ship-review data-migration finding). Context: eng review +
codex consult of the Codex scope fix, 2026-08-11.
- [ ] **P2 — bootstrap first-push secret scan reads the working tree, not the
index blobs; fail-open on binary/large files.** `secretScanOrThrow` /
`scanFiles` (src/core/bootstrap/repo.ts + src/core/secret-scan.ts) read
working-tree bytes and silently skip unreadable, binary, and >25 MiB files, so
a git clean filter could commit a secret whose working-tree copy scans clean,
and a secret in a binary/large file is never seen. Pre-existing across ALL
bootstrap pushes (create + adopt), not specific to create-repo-first. Fix:
scan the staged index blobs (`git show :file` / `git cat-file`) fail-closed,
or reuse the hardened scanner path from `workspacePush`. Filed from the
v0.45.2.0 /ship Codex adversarial pass (P0 there; scoped to P2 here as a
shared-scanner hardening that needs its own tests, deliberately out of the
create-repo-first change).
- [x] **P2 — compiled `gbrain` binary can now `serve` a PGLite brain.** FIXED:
`src/core/pglite-embedded-assets.ts` embeds PGLite's runtime payload
(`pglite.wasm`, `initdb.wasm`, `pglite.data`, `vector.tar.gz`,
`pg_trgm.tar.gz`) with Bun's `import ... with { type: 'file' }` and hands them
to PGLite via `PGliteOptions` (`pgliteWasmModule` / `initdbWasmModule` /
`fsBundle` + custom `vector`/`pg_trgm` extensions whose `setup()` returns a
`bundlePath` pointing at the embedded tarball, materialized to a temp file
because `createReadStream` — unlike `readFileSync` — can't read `/$bunfs`
paths). `src/core/pglite-engine.ts` spreads `getEmbeddedPgliteOptions()` at
both `PGlite.create()` sites, so the Bun-vfs #1340 ENOENT no longer fires for
a correctly-built binary. Guarded by `scripts/check-pglite-embedded.sh`
(compiles a smoketest and asserts a real PGLite query round-trips), wired into
`bun run verify` + `check:all` + `check:pglite-embedded`. The real-agent e2e
harness (`test/helpers/agent-harness.ts`) now resolves to the fast compiled
MCP server; the `bun run` fallback stays as a safety net. Upstream Bun issue
(still open since Nov 2024) is now moot for gbrain.
- [ ] **P1 — `--background --follow` spawns a nonexistent subcommand.**
`src/core/cli-options.ts` (~:391) spawns `gbrain jobs follow <id>` after a
background submit, but jobs.ts has no `follow` subcommand (`jobs watch
--follow` exists and takes no id; `jobs submit --follow` is inline-only).
The spawned child hits the unknown-subcommand path, so `--background
--follow` submits fine but the live stream never attaches. Fix: implement
`jobs follow <id>` (poll get_job + stream progress) or retarget the spawn.
Found during the v0.45.0.0 doc audit; KEY_FILES documents current behavior.
- [ ] **P1 — bootstrap status is not workspace-scoped.** status.ts reads the
newest GLOBAL verify snapshot + global push-status, so a green verify in
workspace A can report workspace B as verified/healthy. Key the verify
snapshots + push-status by workspace path (or store them under the
workspace's own state dir) so status reflects the workspace it runs in.
Found by the v0.45.0.0 adversarial pass (Codex P2, raised to P1 — multi-
workspace is the graduation/multi-device story).
- [ ] **P2 — mkdir-lock ABA steal race (folds into the shared-primitive TODO).**
Both acquirePushLock (workspace-push.ts) and acquireBootstrapLock (lock.ts)
steal a dead+stale lock as rmSync-then-mkdir with no re-verify between, so
two simultaneous stealers can both believe they hold it (same known class as
pglite-lock; git index.lock bounds the push damage). When extracting the
shared mkdir-lock primitive, harden the steal: re-read + token-compare the
stale owner immediately before rmSync, or use an atomic rename-based steal.
Cross-model finding (Claude + Codex) v0.45.0.0 adversarial.
- [ ] **P2 — uninstall TOCTOU on a live PGLite DB.** uninstall probes the serve
lock then recursively deletes later; a serve starting in between wins the
race. Gated today by the lock probe + refuse-on-live-serve, so this is the
residual known lock-class window — close it when the shared lock primitive
lands (hold the lock across the probe→delete span). Codex P1 v0.45.0.0.
- [ ] **P2 — sweep fairness: recency-only selection starves old pages.**
sweep.ts pass 2 repeatedly takes the newest batchLimit pages with no
extraction-watermark filter, so frequently-updated pages monopolize every
sweep and older pages never get link/timeline extraction. Add an
extracted-watermark (or round-robin) so the backlog drains. Codex P2.
- [ ] **P2 — untracked deny-glob exclusion is invisible under detached push.**
A first-time deny-glob file (e.g. a fresh .env) is dropped from staging and
reported only via the logger callback + excludedUntracked — but session-end/
session-start auto-push runs via spawnDetachedPush with stdio:'ignore', so
the warning is discarded. Fail-safe (never leaves the machine) but the user
gets a false "synced" impression. Surface excluded deny-matches via the
heartbeat/push-status file so doctor + status can report them. Claude
adversarial (minor) v0.45.0.0.
- [ ] **P2 — shared receipt upsert helper.** InstallReceipt construct/merge
logic is quadruplicated (bootstrap.ts writeRenderReceipt +
appendReceiptRegistration, attach.ts, repo.ts recordRepoInReceipt) with
slightly different defaults; guardReceiptOverwrite is wired at each site
individually. One `upsertReceipt(home, ws, patch)` in format.ts.
- [ ] **P2 — promote the atomic tmp+rename write idiom to one core helper.**
~7 hand-rolled copies across format.ts/interview.ts/lock.ts/render.ts/
hook.ts/workspace-push.ts/verify.ts; hooks.ts already has a private
atomicWriteJson to promote (plus a text variant).
- [ ] **P2 — extract a shared mkdir-lock primitive.** bootstrap/lock.ts and
workspace-push.ts both implement atomic-mkdir + PID/age/token steal rules
citing the same pglite-lock learnings (#2058/#2348); three parallel
implementations counting pglite-lock. One primitive with injectable stale
window + throw-vs-result adapters.
- [ ] **P2 — move the hook heartbeat read surface into core.** core/bootstrap/
status.ts and core/bootstrap/verify.ts dynamic-import readHeartbeatTail
from commands/hook.ts (core→commands inversion). A core/hooks-telemetry.ts
owning the read/write surface removes the reach-ins.
- [ ] **P2 — consolidate the two GitHub remote parsers + privacy probes.**
workspace-push.ts parseGithubOwnerRepo (accepts ssh:// + trailing /) vs
repo.ts parseGithubRemote (scp-form only), and verifyRemotePrivacy (gh repo
view) vs verifyRepoPrivate (gh api): grammars/failure classes can drift.
One exported parser + one privacy-probe core returning the verdict union.
- [ ] **P2 — connector-ingest capability probe (with v1.1 connector ingest).**
The plan's build-order-2 probe (email/calendar connector availability wired
into install output) deferred with the feature it gates; the capability
report today covers embeddings + extraction only.
- [ ] **P2 — G15 retention warnings in doctor.** MEMORY.md size cap + corpus
retention + orphaned stop-hook buffers are enforced/pruned but doctor never
warns when they accumulate; add the three checks to the bootstrap group.
- [ ] **P2 — NER-based graph floor in verify.** The graph-floor check asserts
wikilink-edge/backlink link-table rows; a true entity-extraction floor
(≥1 NER entity + one edge-only query op) needs the extraction path or a
keyed provider, so it rides with a keyed-mode verify extension.
- [ ] **P2 — `--isolated` flag productization.** GBRAIN_HOME threading through
MCP registration env, hooks, and uninstall guards ships in v1; the
documented `bootstrap --isolated` one-flag wrapper (set env + assert
database_path containment + gitignore check, per D2=C) is not yet a flag.
- [ ] **P2 — sweep N+1 page fetch.** The links/timeline pass getPage-per-slug
loop is bounded (batchLimit 20) but serializes round-trips on the live
serve connection; add a getPagesBySlugs batch read to both engines (engine
parity + bootstrap-probe obligations) and use it.
- [ ] **P2 — `gbrain quota` meter command.** Productize the release-gate quota
measurement (script+doc ship with the bootstrap wave) into a command that reports
a session/day's token burn against the harness subscription allowance. Blocked on
a proven per-harness token-count method — an inaccurate meter erodes the trust it
exists to build (CEO review D3.4; not promoted by a passing quota gate, by
decision). Start: the spike's measurement scripts under docs/designs/.
- [ ] **P2 — Networked Docker paste-flow e2e.** The offline container e2e (interview
→ render → verify with fake gh) ships with the wave; the full networked flow (bun
install, gh auth, repo create, MCP registration) stays manual. Unblocks after 10
consecutive green offline runs in heavy-tests (CEO review D3.3b). Start:
tests/docker/ harness + the fake-gh recording shim.
- [ ] **P2 — Real-agent door e2e needs a provisioned runner.** The real-binary door
tests (`test/e2e/bootstrap-real-{claude,codex}.serial.test.ts`) drive the ACTUAL
`claude`/`codex` binaries against a real gbrain and pay API cost. They self-SKIP
(`describe.skipIf` on binary/auth) everywhere else, so the `real-agent-e2e` job in
`.github/workflows/heavy-tests.yml` is a green no-op on stock GitHub runners. To
actually EXERCISE them we need a self-hosted / manually-provisioned runner with
authed `claude` + `codex` and provider creds exported
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`). Until then they
run locally on an operator machine only. Start: stand up a labeled runner with the
binaries pre-authed, or a scheduled self-hosted lane.
- [ ] **P1 — unit-shard exit hang: bun test process leaks a ref'd handle and never
exits after all tests pass.** Probabilistic (scales with file count/duration),
PRE-EXISTING: reproduces on origin/master with the same 274-file combination
(`bun test $(cat list) --max-concurrency=2` — shard-2 composition from the
agent-bootstrap wave's reshuffled split; solo halves/eighths pass, full list
hangs on both branches; a repeat of the same 68-file quarter passed, so it is a
race, not a single file). Forensics: the "[process-cleanup] db-lock:test:r1:
PGlite is closed" tail line is printed BY the SIGTERM cleanup at kill time, not
pre-hang; db-lock refresh + process-watchdog intervals are already unref'd —
the leak is elsewhere (candidate class: a ref'd handle in a PGLite/worker
teardown race). run-unit-parallel.sh now classifies the exact signature
(killed at cap + all assigned files started + zero fail markers) as a loud
warn-pass so it can't red-X unrelated work; fixing the leak removes the
classifier's reason to exist. Start: reproduce with `--inspect` /
Bun.unsafe process handle dumps on the full shard-2 list.
- [ ] **P2 — WAL-repair unit tests flake under parallel shard load.**
`test/pglite-repair.test.ts` (`repairPgliteWal rename-based backup`,
`inspectPgliteDataDir verdicts`) failed once in a full 4-shard run and pass
44/44 standalone — load-sensitive, from the v0.42.75.0 WAL auto-repair wave
(pre-existing on master; observed during the agent-bootstrap gate runs).
Start: run the file under `--max-concurrency=4` alongside PGLite-heavy
neighbors to reproduce; suspect tmp-dir or timing assumptions.
## Giftable-import wave follow-ups (filed at build time)
- [ ] **P1 — Wire citation edge types into relational retrieval.** `relational-intent.ts`
recognizes a hardcoded edge-type set that excludes `overrules`/`distinguishes`/
`relies-on` (the types citation-graph-ingest creates). Until they're walked by
natural-language relational recall, the skill's value is explicit `graph-query`
only. Add the types + an eval fixture proving a relational question traverses a
citation edge. Files: `src/core/search/relational-intent.ts`,
`src/core/search/relational-recall.ts`.
- [ ] **P2 — Native operation-boundary confirm for destructive ops.** data-loss-gate
is routing prose; destructive paths (bulk forget, `delete_page` sweeps, source
removal, mounts remove) can bypass it via CLI/MCP/jobs. Add a native confirm
(TTY prompt / `--yes` flag / MCP scope) at the operation boundary.
- [ ] **P2 — `gbrain ingest feed`: native feed adapter.** blog-ingest ships the
agent-procedure layer; the durable path is a deterministic RSS/Atom adapter
(discovery, pagination, canonical-URL dedup, 429 backoff) behind one command.
- [ ] **P2 — Native AI-chat export importer.** conversation-archive converts
ChatGPT/Claude/Perplexity exports via agent procedure; a native importer
(export JSON → conversations/ pages) makes it deterministic. Pairs with the
existing conversation-parser surface.
- [ ] **P2 — Entity-guard as a native op.** phonetic-name-guard's own changelog
proves prose-only failed: ASR-variant entity collisions need a native check
(registry + alias table consulted at put/import time). The wave shipped the
registry-first discipline in brain-ingest-gate; this hardens it.
- [ ] **P2 — Premiere-repo program ① distribution:** list gbrain on skills.sh +
Claude Code plugin marketplace + agentskills.io conformance; README cross-
harness matrix (CI-verified). First fast-follow PR after this wave.
- [ ] **P2 — Premiere-repo program ② receipts:** public BrainBench receipts page
pairing accuracy with token cost per query, regenerated per release; "trust
layer" framing (data-loss-gate + brain-ingest-gate + correction-pipeline).
- [ ] **P3 — Premiere-repo program ③ protocol moat:** Anthropic memory-tool
(`memory_20250818`) adapter backed by recall/remember; publish MEMORY_VERBS_v1
as an open spec with BrainBench as its conformance suite. Own cathedral.
- [ ] **P3 — Premiere-repo program ④ badges:** per-skill conformance badges
(security-scan + eval-receipt + provenance hash) surfaced in manifest/README;
generalize the functional-area-resolver A/B harness into `evals/skills/`.
- [ ] **P3 — RESOLVER two-layer compression as its own PR.** Deferred out of the
wave at eng review: requires arrow-form dispatcher entries, the A/B run at
>=95% (per the functional-area-resolver contract), resolver.test.ts updates,
and fixture backfill for fixture-less skills. RESOLVER.md is now past the 12KB
gate, so the skill's precondition is satisfied.
- [ ] **P3 — extract-atoms quality-gate prompt patch.** Fold the donor pack's
truism filter / statistic-punchline test / entity-page routing test / named-
attribution rule into `src/core/cycle/extract-atoms.ts`'s EXTRACT_PROMPT,
eval-gated (the native prompt's only bar today is "not a generic platitude").
- [ ] **P3 — cross-modal eval `--corpus` hub-and-spoke mode + judge-leniency
normalization.** Follow relative .md links from a hub page so multi-page brain
artifacts aren't falsely penalized; normalize per-judge leniency in
`src/core/cross-modal-eval/aggregate.ts` (mean+floor only today).
- [ ] **P3 — Advisor collectors: freshness-monitor + context-audit token drift.**
Two new collectors: per-source staleness SLA (the donor freshness-monitor
kernel) and a deterministic loaded-context token-drift check feeding the
context-audit skill.
- [ ] **P3 — idea-miner import (deferred at CEO review, fit 6).** Daily brain-
grounded "what could I build" mining feeding skill-creator; below the wave's
fit bar but a strong self-improvement story.
- [ ] **P3 — public-repo-guard revisit.** Only egress leak-gate candidate; its
upstream scan script fails open (`SCAN_EXIT` captured after `|| true`). Fix
upstream first; template-ize the patterns file; mind the gstack cso boundary.
- [ ] **P3 — `search --fm` + schema-pack fragment** from the social-json-store
audit disposition: frontmatter-ID/JSONB query kernel as a native search flag
+ a schema-pack fragment, not a skill.
- [ ] **P3 — back-catalog-check kernel.** Optional pre-publish own-corpus
consistency pass folding into fact-check (per-claim own-record search);
`find_contradictions` + idea-lineage cover the rest today.
## Skill self-knowledge — semantic skill search (deferred subsystem, from the migration-harness build)
- [ ] **P2 — Make built-in skills semantically searchable in the brain.** Today skills
are markdown the harness routes to via triggers + a host catalog (`list_skills`/
`get_skill`); `gbrain search "how do I verify claims"` can't surface `fact-check`.
Making skills first-class searchable content needs a real design pass (tenancy +
source-isolation: skill pages must not pollute user-source query results; embedding
storage + backfill; search-steering to include/exclude the skill catalog; engine
parity). Deliberately NOT built in the currency/preconditions wave — it is a
subsystem that deserves its own eng + CEO review, not a rider. The currency work
(`skillpack status`/`sync`, doctor `skill_currency`) already keeps the brain's skill
set current on upgrade; this item is purely about semantic retrieval of skills.
+1 -1
View File
@@ -1 +1 @@
0.45.12.0
0.42.64.0
+3 -4
View File
@@ -19,8 +19,7 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"nanoid": "^3.3.17",
"postcss": "^8.5.23",
"postcss": "^8.5.10",
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
@@ -225,7 +224,7 @@
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nanoid": ["nanoid@3.3.18", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w=="],
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
@@ -233,7 +232,7 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.25", "", { "dependencies": { "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw=="],
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -7,7 +7,7 @@
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<script type="module" crossorigin src="/admin/assets/index-CviJXT-1.js"></script>
<script type="module" crossorigin src="/admin/assets/index-CoGEje3-.js"></script>
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
</head>
<body>
+1 -2
View File
@@ -20,7 +20,6 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.23",
"nanoid": "^3.3.17"
"postcss": "^8.5.10"
}
}
+2 -12
View File
@@ -39,21 +39,11 @@ export const api = {
stats: () => apiFetch('/admin/api/stats'),
health: () => apiFetch('/admin/api/health-indicators'),
agents: () => apiFetch('/admin/api/agents'),
sources: () => apiFetch('/admin/api/sources'),
requests: (page = 1, qs = '') => apiFetch(`/admin/api/requests?page=${page}${qs}`),
apiKeys: () => apiFetch('/admin/api/api-keys'),
createApiKey(keyName: string) {
return apiFetch('/admin/api/api-keys', { method: 'POST', body: JSON.stringify({ name: keyName }) });
},
revokeApiKey(keyName: string) {
return apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name: keyName }) });
},
createApiKey: (name: string) => apiFetch('/admin/api/api-keys', { method: 'POST', body: JSON.stringify({ name }) }),
revokeApiKey: (name: string) => apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name }) }),
updateClientTtl: (clientId: string, tokenTtl: number | null) => apiFetch('/admin/api/update-client-ttl', { method: 'POST', body: JSON.stringify({ clientId, tokenTtl }) }),
rescopeClient: (clientId: string, sourceId: string, federatedRead: string[]) =>
apiFetch('/admin/api/rescope-client', {
method: 'POST',
body: JSON.stringify({ clientId, sourceId, federatedRead }),
}),
revokeClient: (clientId: string) => apiFetch('/admin/api/revoke-client', { method: 'POST', body: JSON.stringify({ clientId }) }),
// v0.36.1.0 (T15 / E6) — calibration endpoints.
calibrationProfile: (holder?: string) =>
+4 -169
View File
@@ -18,8 +18,6 @@ interface Agent {
client_name?: string; // compat
grant_types: string[];
scope: string;
source_id: string | null;
federated_read: string[];
created_at: string;
last_used_at: string | null;
total_requests: number;
@@ -28,12 +26,6 @@ interface Agent {
status: 'active' | 'revoked';
}
interface Source {
id: string;
name: string;
federated: boolean;
}
interface ApiKey {
id: string;
name: string;
@@ -44,7 +36,6 @@ interface ApiKey {
export function AgentsPage() {
const [agents, setAgents] = useState<Agent[]>([]);
const [sources, setSources] = useState<Source[]>([]);
const [hideRevoked, setHideRevoked] = useState(true);
const [showRegister, setShowRegister] = useState(false);
const [showCredentials, setShowCredentials] = useState<{ clientId: string; clientSecret: string; name: string } | null>(null);
@@ -52,10 +43,7 @@ export function AgentsPage() {
const [showApiKeyToken, setShowApiKeyToken] = useState<{ name: string; token: string } | null>(null);
const [selectedAgent, setSelectedAgent] = useState<Agent | null>(null);
useEffect(() => {
loadAgents();
api.sources().then(setSources).catch(() => {});
}, []);
useEffect(() => { loadAgents(); }, []);
const loadAgents = () => { api.agents().then(setAgents).catch(() => {}); };
@@ -100,7 +88,6 @@ export function AgentsPage() {
<th>Name</th>
<th>Type</th>
<th>Scopes</th>
<th>Sources</th>
<th>Status</th>
<th>Requests</th>
<th>Last Used</th>
@@ -121,11 +108,6 @@ export function AgentsPage() {
<span key={s} className={`badge badge-${s}`} style={{ marginRight: 4 }}>{s}</span>
))}
</td>
<td style={{ color: 'var(--text-secondary)', fontSize: 12 }}>
{a.auth_type === 'oauth'
? `${a.source_id || 'none'} · ${(a.federated_read || []).length} readable`
: 'Unscoped'}
</td>
<td>
<span className={`badge ${a.status === 'active' ? 'badge-success' : 'badge-danger'}`}>{a.status}</span>
</td>
@@ -162,21 +144,7 @@ export function AgentsPage() {
)}
{selectedAgent && (
<AgentDrawer
key={selectedAgent.id}
agent={selectedAgent}
sources={sources}
onClose={() => setSelectedAgent(null)}
onRevoked={loadAgents}
onRescoped={({ sourceId, federatedRead }) => {
setSelectedAgent(current => current ? {
...current,
source_id: sourceId,
federated_read: federatedRead,
} : current);
loadAgents();
}}
/>
<AgentDrawer agent={selectedAgent} onClose={() => setSelectedAgent(null)} onRevoked={loadAgents} />
)}
{showApiKeyCreate && (
@@ -413,127 +381,7 @@ function CredentialsModal({ credentials, onClose }: {
);
}
function SourceAccessEditor({ clientId, agent, sources, onRescoped }: {
clientId: string;
agent: Agent;
sources: Source[];
onRescoped: (scope: { sourceId: string; federatedRead: string[] }) => void;
}) {
const [writeSource, setWriteSource] = useState(agent.source_id || 'default');
const [readSources, setReadSources] = useState<string[]>(agent.federated_read || []);
const [saving, setSaving] = useState(false);
const [error, setError] = useState('');
const [saved, setSaved] = useState(false);
const readableSet = new Set(readSources);
const activeSourceIds = new Set(sources.map(source => source.id));
const unavailableReadSources = readSources.filter(sourceId => !activeSourceIds.has(sourceId));
const primaryUnavailable = !activeSourceIds.has(writeSource);
const save = async () => {
if (readSources.length === 0) {
setError('Select at least one readable source.');
return;
}
setSaving(true);
setError('');
setSaved(false);
try {
const result = await api.rescopeClient(clientId, writeSource, readSources) as {
sourceId: string;
federatedRead: string[];
};
setWriteSource(result.sourceId);
setReadSources(result.federatedRead);
setSaved(true);
onRescoped(result);
} catch (e) {
setError(e instanceof Error ? e.message : 'Failed to save source access');
} finally {
setSaving(false);
}
};
return (
<>
<div className="section-title">Source Access</div>
<div style={{ color: 'var(--text-secondary)', fontSize: 12, lineHeight: 1.5, marginBottom: 12 }}>
The primary source is the write destination. Read access is an explicit allowlist and does not widen automatically.
</div>
<div style={{ marginBottom: 14 }}>
<label htmlFor="agent-write-source">Primary / write source</label>
<select
id="agent-write-source"
value={writeSource}
onChange={e => { setWriteSource(e.target.value); setSaved(false); }}
style={{ width: '100%', background: 'var(--bg-secondary)', color: 'var(--text-primary)', border: '1px solid var(--border)', borderRadius: 6, padding: '6px 10px', fontSize: 14 }}
>
{primaryUnavailable && (
<option value={writeSource} disabled>{writeSource} · unavailable</option>
)}
{sources.map(source => (
<option key={source.id} value={source.id}>{source.name} ({source.id})</option>
))}
</select>
</div>
<fieldset style={{ border: 0, padding: 0, margin: '0 0 14px' }}>
<legend>Readable sources</legend>
<div className="checkbox-group" style={{ marginTop: 6 }}>
{sources.map(source => (
<label key={source.id} className="checkbox-label">
<input
type="checkbox"
checked={readableSet.has(source.id)}
onChange={e => {
setSaved(false);
setReadSources(current => e.target.checked
? [...current, source.id]
: current.filter(id => id !== source.id));
}}
/>
{source.name} ({source.id}){source.federated ? ' · federated' : ' · private'}
</label>
))}
{unavailableReadSources.map(sourceId => (
<label key={sourceId} className="checkbox-label" style={{ color: 'var(--warning)' }}>
<input
type="checkbox"
checked
onChange={() => {
setSaved(false);
setReadSources(current => current.filter(id => id !== sourceId));
}}
/>
{sourceId} · unavailable (clear to remove grant)
</label>
))}
</div>
</fieldset>
{(primaryUnavailable || unavailableReadSources.length > 0) && (
<div style={{ color: 'var(--warning)', fontSize: 13, marginBottom: 10 }}>
This client references unavailable or archived sources. Choose an active primary source and clear unavailable read grants before saving.
</div>
)}
{error && <div style={{ color: 'var(--error)', fontSize: 13, marginBottom: 10 }}>{error}</div>}
{saved && <div style={{ color: 'var(--success)', fontSize: 13, marginBottom: 10 }}>Source access saved.</div>}
<button
type="button"
className="btn btn-primary"
disabled={saving || readSources.length === 0 || sources.length === 0 || primaryUnavailable || unavailableReadSources.length > 0}
onClick={save}
>
{saving ? 'Saving...' : 'Save Source Access'}
</button>
</>
);
}
function AgentDrawer({ agent, sources, onClose, onRevoked, onRescoped }: {
agent: Agent;
sources: Source[];
onClose: () => void;
onRevoked: () => void;
onRescoped: (scope: { sourceId: string; federatedRead: string[] }) => void;
}) {
function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: () => void; onRevoked: () => void }) {
const [tab, setTab] = useState<'claude-code' | 'chatgpt' | 'claude-cowork' | 'perplexity' | 'cursor' | 'json'>('claude-code');
const copy = (text: string) => navigator.clipboard.writeText(text);
const serverUrl = window.location.origin;
@@ -705,15 +553,6 @@ function AgentDrawer({ agent, sources, onClose, onRevoked, onRescoped }: {
<span>{agent.token_ttl ? (agent.token_ttl >= 31536000 ? 'No expiry' : agent.token_ttl >= 86400 ? `${Math.floor(agent.token_ttl / 86400)}d` : agent.token_ttl >= 3600 ? `${Math.floor(agent.token_ttl / 3600)}h` : `${agent.token_ttl}s`) : '1h (default)'}</span>
</div>
{isOAuth && (
<SourceAccessEditor
clientId={cid}
agent={agent}
sources={sources}
onRescoped={onRescoped}
/>
)}
{/*
Config Export visible for both auth_type=oauth AND auth_type=api_key.
Claude Code + Cursor + JSON tabs render real snippets regardless
@@ -740,11 +579,7 @@ function AgentDrawer({ agent, sources, onClose, onRevoked, onRescoped }: {
{(() => {
const oauthOnlyTabs = new Set(['chatgpt', 'claude-cowork', 'perplexity']);
if (!isOAuth && oauthOnlyTabs.has(tab)) {
const clientName = tab === 'chatgpt'
? 'ChatGPT'
: tab === 'claude-cowork'
? 'Claude.ai'
: 'Perplexity';
const clientName = { chatgpt: 'ChatGPT', 'claude-cowork': 'Claude.ai', perplexity: 'Perplexity' }[tab] || tab;
return (
<div style={{
background: 'rgba(255, 200, 100, 0.08)',
+12 -17
View File
@@ -26,7 +26,7 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.15.1",
"js-yaml": "^3.15.0",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
@@ -51,15 +51,14 @@
"@electric-sql/pglite",
],
"overrides": {
"@hono/node-server": "^2.0.5",
"body-parser": "^2.3.0",
"fast-uri": "^3.1.5",
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.34",
"ip-address": "^10.3.1",
"js-yaml": "^3.15.1",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"js-yaml": "^3.15.0",
"qs": "^6.15.2",
},
"packages": {
@@ -163,7 +162,7 @@
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
"@hono/node-server": ["@hono/node-server@2.1.0", "", { "peerDependencies": { "hono": "^4" } }, "sha512-XovyyCCnBzW+zKu+z/zq8hwNs4KOR5rEMAOxo2f40Q5xoOI37IMm6MIg2COOUtUApo0i6850MTBKH2u4QLGIqg=="],
"@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="],
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
@@ -327,7 +326,7 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"body-parser": ["body-parser@2.3.0", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^2.0.0", "debug": "^4.4.3", "http-errors": "^2.0.1", "iconv-lite": "^0.7.2", "on-finished": "^2.4.1", "qs": "^6.15.2", "raw-body": "^3.0.2", "type-is": "^2.1.0" } }, "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw=="],
"body-parser": ["body-parser@2.2.2", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^1.0.5", "debug": "^4.4.3", "http-errors": "^2.0.0", "iconv-lite": "^0.7.0", "on-finished": "^2.4.1", "qs": "^6.14.1", "raw-body": "^3.0.1", "type-is": "^2.0.1" } }, "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA=="],
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
@@ -401,7 +400,7 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
"fast-uri": ["fast-uri@3.1.3", "", {}, "sha512-i70LwGWUduXqzicKXWshooq+sWL1K3WUU5rKZNG/0i3a1OSoX3HqhH5WbWwTmqWfor4urUakGPiRQcleRZTwOg=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
@@ -437,7 +436,7 @@
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.13.0", "", {}, "sha512-jhunvfHWxd7J5EFfSgH4xsYJzSe/lfqbUCxiyyeaQasUsXeEHXtzVid+7EOGByc5JnFa23SSFL3Y2RV/z1T+eQ=="],
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -447,7 +446,7 @@
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"ip-address": ["ip-address@10.4.0", "", {}, "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ=="],
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
@@ -461,7 +460,7 @@
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
"js-yaml": ["js-yaml@3.15.1", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag=="],
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
@@ -615,10 +614,6 @@
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"body-parser/content-type": ["content-type@2.0.0", "", {}, "sha512-j/O/d7GcZCyNl7/hwZAb606rzqkyvaDctLmckbxLzHvFBzTJHuGEdodATcP3yIRoDrLHkIATJuvzbFlp/ki2cQ=="],
"body-parser/type-is": ["type-is@2.1.0", "", { "dependencies": { "content-type": "^2.0.0", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA=="],
"es-set-tostringtag/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"eventsource/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
+1 -6
View File
@@ -18,9 +18,4 @@ timeout = 60_000
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
# test/helpers/audit-dir-preload.ts for the full rationale.
#
# Same treatment for the sync failure ledger: broken-fixture import/sync tests
# were appending rows into the operator's real ~/.gbrain/sync-failures.jsonl,
# which `gbrain doctor` reads and warns on. See
# test/helpers/sync-failures-preload.ts.
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts", "./test/helpers/sync-failures-preload.ts"]
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
+82 -123
View File
@@ -4,7 +4,7 @@
Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else.
Two engines ship today: `PGLiteEngine` embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
## Why this matters
@@ -12,7 +12,7 @@ Different users have different constraints:
| User | Needs | Best engine |
|------|-------|-------------|
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) |
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) |
| Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase |
| Open source hacker | Single file, no server, git-friendly | PGLiteEngine |
| Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted |
@@ -23,30 +23,72 @@ The engine interface means we don't have to choose. PGLite is the zero-friction
## The interface
**The single source of truth is `export interface BrainEngine` in
`src/core/engine.ts`.** It is large (100+ methods) and grows with every
feature wave — do NOT work from any snapshot of it, including an old copy of
this doc. Read the interface itself, and let
`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you
whether both engines agree.
```typescript
// src/core/engine.ts
The method families, to orient you before opening the file:
export interface BrainEngine {
// Lifecycle
connect(config: EngineConfig): Promise<void>;
disconnect(): Promise<void>;
initSchema(): Promise<void>;
transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T>;
- **Lifecycle + identity**`connect` / `disconnect` / `reconnect`,
`initSchema`, `transaction`, `withReservedConnection`, and the `kind`
discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch.
- **Pages CRUD**`getPage`, `putPage`, `deletePage`, `listPages`, slug
resolution.
- **Search**`searchKeyword`, `searchVector`, chunk-level variants, takes
search (keyword + vector), and `relationalFanout` (the typed-edge recall
arm).
- **Chunks + embeddings** — upsert/get, embedding-bearing variants.
- **Graph** — links (single + batch writers), backlinks, `traverseGraph`,
`traversePaths`.
- **Tags, timeline (single + batch), raw data, versions.**
- **Takes / facts / eval / salience** — the epistemological layer and the
instruments over it.
- **Stats, health, ingest log, config, migrations.**
// Pages CRUD
getPage(slug: string): Promise<Page | null>;
putPage(slug: string, page: PageInput): Promise<Page>;
deletePage(slug: string): Promise<void>;
listPages(filters: PageFilters): Promise<Page[]>;
// Search
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
// Chunks
upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void>;
getChunks(slug: string): Promise<Chunk[]>;
// Links
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
removeLink(from: string, to: string): Promise<void>;
getLinks(slug: string): Promise<Link[]>;
getBacklinks(slug: string): Promise<Link[]>;
traverseGraph(slug: string, depth?: number): Promise<GraphNode[]>;
// Tags
addTag(slug: string, tag: string): Promise<void>;
removeTag(slug: string, tag: string): Promise<void>;
getTags(slug: string): Promise<string[]>;
// Timeline
addTimelineEntry(slug: string, entry: TimelineInput): Promise<void>;
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
// Raw data
putRawData(slug: string, source: string, data: object): Promise<void>;
getRawData(slug: string, source?: string): Promise<RawData[]>;
// Versions
createVersion(slug: string): Promise<PageVersion>;
getVersions(slug: string): Promise<PageVersion[]>;
revertToVersion(slug: string, versionId: number): Promise<void>;
// Stats + health
getStats(): Promise<BrainStats>;
getHealth(): Promise<BrainHealth>;
// Ingest log
logIngest(entry: IngestLogInput): Promise<void>;
getIngestLog(opts?: IngestLogOpts): Promise<IngestLogEntry[]>;
// Config
getConfig(key: string): Promise<string | null>;
setConfig(key: string, value: string): Promise<void>;
// Migration + advanced (added v0.7)
runMigration(sql: string): Promise<void>;
getChunksWithEmbeddings(slug: string): Promise<ChunkWithEmbedding[]>;
}
```
### Key design choices
@@ -89,7 +131,7 @@ The method families, to orient you before opening the file:
RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific.
## PostgresEngine
## PostgresEngine (v0, ships)
**Dependencies:** `postgres` (porsager/postgres), `pgvector`
@@ -102,7 +144,9 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
- JSONB for frontmatter with GIN index
- Connection pooling via Supabase Supavisor (port 6543)
**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same.
**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in.
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
@@ -149,17 +193,17 @@ run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine
## PGLiteEngine (v0.7, ships)
**Dependencies:** `@electric-sql/pglite`
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep.
**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented.
**PGLite-specific details:**
- Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes)
- Parameterized queries throughout (shared utilities in `src/core/utils.ts`)
- `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set
- Data stored at `~/.gbrain/brain.pglite` (configurable)
- Data stored at `~/.gbrain/brain.db` (configurable)
- pgvector HNSW index for cosine similarity vector search (same as Postgres)
- tsvector + ts_rank for full-text search (same as Postgres)
- pg_trgm for fuzzy slug resolution (same as Postgres)
@@ -177,85 +221,6 @@ live in `test/postgres-engine-rls-scope.test.ts`.
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
The migration and the autopilot daemon do not race: `migrate --to` claims a
cooperative pause marker before touching the target. The marker doubles as a
migration mutex — a second concurrent migrate refuses to run, and a marker
that cannot be written refuses the migration outright. Background job workers
stop picking up new work while it is parked, and the migration waits for
in-flight sync/embed/cycle work and running jobs to actually drain (watching
the DB lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS` — default 300;
`0` skips the wait). Cleanup registers the moment the claim lands, so the
marker is released on failure and on catchable signals; a marker orphaned by
an uncleanly killed run is adopted by a later migrate only after a
pid-liveness check (a live migrate's marker is never stolen), and the daemon
clears an orphan whose owning process died on its next poll. `gbrain
autopilot --status` reports `paused` (exit 1) while the marker is parked and
prints the marker path; on a host with no daemon running to self-heal,
remove an orphan by hand only after confirming the pid it names is dead.
After a clean flip the daemon detects the engine change on its next
tick and relaunches onto the new engine, and the migration warns if an
exported connection-string env var would override the new config.
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
**Symptom:** every PGLite-touching command dies at startup with
`PGLite failed to initialize its WASM runtime … Aborted(). Build with
-sASSERTIONS for more info.` — commonly first seen right after a macOS
upgrade.
**Real root cause:** corrupt WAL/checkpoint state in the data dir after an
unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the
write-ahead log; every subsequent open fails WAL replay inside WASM and
Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM
incompatibility — the same signature reproduces across macOS versions and on
Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun
version bump changes it.
**Recovery ladder** (top rung first):
1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort,
backs up `pg_wal/` + `pg_control` into a sibling
`<dataDir>.wal-repair-backup-<ts>/` dir, resets the WAL in place
(pg_resetwal semantics — data files preserved; transactions not
checkpointed before the corruption may be lost), and retries once. On
success it prints a loud stderr notice naming the backup and recommending
`gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired
data-dir lock (never after reaping another process's lock), skips for a
cooldown window after a failed attempt
(`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one
backup per corruption episode (newest 3 episodes retained), and restores
the original files if the retry still fails. Kill-switch:
`GBRAIN_PGLITE_WAL_REPAIR=off`.
2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir
(read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset
deliberately. Refuses when another gbrain process holds the brain (a live
`gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`.
3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default
from your config) wipes and re-creates the brain from your brain repo, or
manually: back up `~/.gbrain`, move `brain.pglite` aside,
`gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`.
Required for *catalog* corruption (58P01 / pgvector load failure) — WAL
repair cannot fix that class.
4. **Switch engines.** `gbrain init --supabase`, or native Postgres +
pgvector (recipe below, contributed by @roysaurav):
```bash
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# ~/.gbrain/config.json: { "engine": "postgres",
# "database_url": "postgresql://localhost:5432/gbrain" }
gbrain apply-migrations --yes && gbrain doctor
```
`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails
to connect: it diagnoses the dir from disk, names the repair command, reports
retained repair backups, and escalates when repairs keep recurring (that
means the unclean-shutdown genesis is still active — see the ladder's rung 4).
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
@@ -294,22 +259,16 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make.
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
2. Add to engine factory in `src/core/engine-factory.ts`:
```typescript
export async function createEngine(config: EngineConfig): Promise<BrainEngine> {
switch (config.engine || 'postgres') {
case 'pglite': {
const { PGLiteEngine } = await import('./pglite-engine.ts');
return new PGLiteEngine();
}
case 'myengine': {
const { MyEngine } = await import('./my-engine.ts');
return new MyEngine();
}
// ...
export function createEngine(type: string): BrainEngine {
switch (type) {
case 'pglite': return new PGLiteEngine();
case 'postgres': return new PostgresEngine();
case 'myengine': return new MyEngine();
default: throw new Error(`Unknown engine: ${type}`);
}
}
```
The factory uses dynamic imports so an engine's dependencies (e.g. the
PGLite WASM blob) are only loaded when that engine is selected.
The factory uses dynamic imports so engines are only loaded when selected.
3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }`
4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor.
5. Document in this file + add a design doc in `docs/`
@@ -340,7 +299,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu
| JSONB queries | GIN index | GIN index | Identical |
| Concurrent access | Connection pooling | Single process | PGLite limitation |
| Hosting | Supabase, self-hosted, Docker | Local file | |
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical |
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 |
## Future engine ideas
-8
View File
@@ -6,14 +6,6 @@ A system prompt for any AI agent that wants to build and maintain a personal kno
Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest.
> **Relationship to schema packs:** this document is the prose, paste-in
> version of the schema pattern. gbrain also ships a machine-enforced
> counterpart — schema packs (`gbrain schema`, typed pages, extraction,
> aliases, lint) — documented in `docs/architecture/schema-packs.md` and
> `docs/schema-author-tutorial.md`. The prose schema here and the active
> schema pack should describe the same brain; when you evolve one, evolve
> the other.
---
## What this is
+2 -4
View File
@@ -1,10 +1,10 @@
<!-- skillpack-version: 0.7.0 -->
<!-- source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md -->
# GBrain Skillpack: Reference Architecture for AI Agents
This is a reference architecture for how a production AI agent uses gbrain as its
knowledge backbone. Based on patterns from a real deployment with 14,700+ brain
files, the 50+ bundled skills (`skills/manifest.json`), and 20+ cron jobs running
continuously.
files, 40+ skills, and 20+ cron jobs running continuously.
**The memex vision, realized.** Vannevar Bush imagined a device where an individual
stores everything, mechanized so it may be consulted with exceeding speed. GBrain is
@@ -25,7 +25,6 @@ The foundational read-write loop and data model.
| [Entity Detection](guides/entity-detection.md) | Run it on every message. Capture original thinking + entity mentions |
| [The Originals Folder](guides/originals-folder.md) | Capturing WHAT YOU THINK, not just what you found |
| [Brain-First Lookup](guides/brain-first-lookup.md) | Check the brain before calling any external API |
| [Push-Based Context](guides/push-context.md) | volunteer_context: the brain volunteers relevant pages instead of waiting to be asked |
| [Compiled Truth + Timeline](guides/compiled-truth.md) | Above the line: current synthesis. Below: append-only evidence |
| [Source Attribution](guides/source-attribution.md) | Every fact needs a citation. Format and hierarchy |
@@ -100,7 +99,6 @@ Keeping it running and up to date.
| Guide | What It Covers |
|-------|---------------|
| [Agent Bootstrap](guides/bootstrap.md) | The paste-in install: `gbrain bootstrap`, hooks, `bootstrap verify`, keyless mode |
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
+14 -28
View File
@@ -1,13 +1,5 @@
# GBrain Installation Verification Runbook
> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install
> contract (round-trip, graph floor, and more) automatically and exits non-zero
> on failure — it is the modern first thing to run after any install. See
> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the
> **manual, deep-verification** companion: use it when `bootstrap verify` fails
> and you need to isolate which layer broke, or when you want to understand
> what "healthy" looks like check by check.
Run these checks after install to confirm every part of GBrain is working.
Each check includes the command, expected output, and what to do if it fails.
@@ -28,8 +20,7 @@ gbrain doctor --json
**Expected:** All checks return `"ok"`:
- `connection`: connected, N pages
- `pgvector`: extension installed
- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains
skip this check; the embedded engine has no remote surface)
- `rls`: enabled on all tables
- `schema_version`: current
- `embeddings`: coverage percentage
@@ -42,12 +33,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table.
**Check:** Ask the agent: "What is the brain-agent loop?"
**Expected:** The agent describes the read-write cycle documented in
[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities,
read brain, respond with context, write brain, sync.
**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes
the read-write cycle: detect entities, read brain, respond with context, write
brain, sync.
**If it fails:** The agent hasn't loaded the skillpack. Have it read
`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links.
**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the
install paste (read `docs/GBRAIN_SKILLPACK.md`).
---
@@ -62,8 +53,8 @@ gbrain check-update --json
**Expected:** Returns JSON with `current_version`, `latest_version`,
`update_available` (boolean). The cron `gbrain-update-check` is registered.
**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md)
for how to register the update-check cron.
**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md
Section 17.
---
@@ -97,9 +88,8 @@ find /data/brain -name '*.md' \
Some difference is normal (files added since last sync), but if page count is
less than half the file count, sync is silently skipping pages.
**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an
unreachable direct connection on an IPv4-only host. (PGLite brains have no
network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543)
**If page count is way too low:** The #1 cause is an unreachable direct
connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543)
for reads, but routes migrations, DDL, and sync transactions to a derived direct
connection (`db.<ref>.supabase.co:5432`), which is IPv6-only.
- On an IPv4-only host, reads work but sync transactions fail and silently skip
@@ -132,7 +122,7 @@ This is the real test. Edit a brain page, push, wait, search.
1. Edit a page in the brain repo (e.g., correct a fact on a person's page):
```bash
# Example: fix a line in alice-example's page
# Example: fix a line in Gustaf's page
cd /data/brain
# Make a small edit to any .md file
git add -A && git commit -m "test: verify live sync" && git push
@@ -263,23 +253,19 @@ gbrain repair-jsonb
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
**Bonus check** — the doctor's dedicated JSONB scan agrees:
**Bonus check** — frontmatter-keyed queries actually resolve:
```bash
gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*'
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
```
**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store
objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`.
If this returns rows on a brain with person pages, the JSONB path is healthy.
---
## Quick Verification (all checks in one pass)
```bash
# 0. The one-command contract check (exits non-zero on failure)
gbrain bootstrap verify
# 1. Schema
gbrain doctor --json
+40 -40
View File
@@ -1,24 +1,15 @@
# Install
**Recommended door: the agent bootstrap.** Open your agent (Codex, Claude Code,
or any harness) in the folder that will become its home and paste the block
from the [README's install section](../README.md) — the agent fetches
`BOOTSTRAP_FOR_AGENTS.md` from the `latest-stable` tag, installs the CLI,
initializes a local PGLite brain, wires MCP, and isn't done until
`gbrain bootstrap verify` exits 0. Full contract, security posture, and
uninstall: [docs/guides/bootstrap.md](guides/bootstrap.md).
Three install paths. Pick one. Mix later if needed.
The paths below are the manual equivalents and deep-dive detail. Pick one.
Mix later if needed.
## 1. Run with an agent platform (recommended)
## 1. Run with an agent platform
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/NousResearch/hermes-agent)?
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
```bash
bun install -g github:garrytan/gbrain#latest-stable
bun install -g github:garrytan/gbrain
gbrain init --pglite # 2 seconds; no server
gbrain skillpack scaffold --all # scaffolds every bundled skill (skills/manifest.json) into your agent workspace
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
gbrain doctor # green checks all the way down
```
@@ -33,7 +24,7 @@ To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts
No agent platform, just shell + MCP-aware editor.
```bash
bun install -g github:garrytan/gbrain#latest-stable
bun install -g github:garrytan/gbrain
gbrain init --pglite
```
@@ -48,11 +39,10 @@ gbrain migrate --to pglite # Postgres → PGLite (rare)
For shared / large / multi-machine deployments (a team or company brain with multiple users hitting one server over HTTP MCP with OAuth scoping per user), follow the dedicated walkthrough: **[Tutorial: set up GBrain as your company brain](tutorials/company-brain.md)**.
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
```bash
gbrain config set zeroentropy_api_key sk-...
gbrain config set openrouter_api_key sk-or-...
gbrain config set anthropic_api_key sk-ant-...
```
@@ -67,17 +57,16 @@ gbrain autopilot --install # background daemon for nightly enrichment
**Wire this same local brain into your coding agent** — zero server, zero token:
```bash
claude mcp add gbrain -- gbrain serve --surface verbs # Claude Code
codex mcp add gbrain -- gbrain serve --surface verbs # Codex
claude mcp add gbrain -- gbrain serve # Claude Code
codex mcp add gbrain -- gbrain serve # Codex
```
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. `--surface verbs` gives the agent the seven-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget`, `context_pack`, `delta` — [MEMORY_VERBS v1](protocol/MEMORY_VERBS_v1.md)) instead of the full tool catalog; drop the flag (default `full`) for every operation. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
## 3. MCP server (any MCP client)
```bash
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --surface verbs # stdio MCP, just the 7 memory verbs (quickstart)
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
```
@@ -98,8 +87,6 @@ Per-client setup guides live in [`docs/mcp/`](mcp/):
- [`docs/mcp/CODEX.md`](mcp/CODEX.md)
- [`docs/mcp/CLAUDE_DESKTOP.md`](mcp/CLAUDE_DESKTOP.md)
- [`docs/mcp/CHATGPT.md`](mcp/CHATGPT.md)
- [`docs/mcp/HERMES.md`](mcp/HERMES.md)
- [`docs/mcp/OPENCLAW.md`](mcp/OPENCLAW.md)
- [`docs/mcp/PERPLEXITY.md`](mcp/PERPLEXITY.md)
- [`docs/mcp/DEPLOY.md`](mcp/DEPLOY.md) — production deploy patterns
@@ -118,31 +105,44 @@ Useful for: team mounts, brain-as-a-service deployments, dev machines without di
## Verifying the install
```bash
gbrain bootstrap verify # the whole install contract; exits non-zero on failure
gbrain doctor --json # full health check
gbrain models # which AI models are configured for what
gbrain models doctor # 1-token probe per configured model
```
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). For the manual check-by-check runbook, see [docs/GBRAIN_VERIFY.md](GBRAIN_VERIFY.md).
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
## Troubleshooting
### PGLite crashes at startup (`RuntimeError: Aborted()`)
### PGLite crashes on macOS 26.x (Tahoe)
This crash (typically first seen after a macOS upgrade) is **not** a
macOS/WASM incompatibility — an unclean shutdown tore the data dir's
write-ahead log, and every subsequent open fails WAL replay. The short
version of the recovery ladder:
PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL:
1. **Auto-repair (default):** run any gbrain command — gbrain detects the
abort, resets the WAL in place (data preserved, backup kept), and
continues. Then run `gbrain doctor`.
2. **Manual repair:** `gbrain pglite-repair --dry-run`, then
`gbrain pglite-repair --yes`.
3. **Rebuild:** `gbrain reinit-pglite`.
4. **Switch engines:** Supabase or native Homebrew Postgres + pgvector.
```bash
# Install PostgreSQL + pgvector
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
The full ladder — safety bounds, kill-switches, when WAL repair can't help,
and the Homebrew Postgres recipe — lives in
[docs/ENGINES.md](ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted).
# Build pgvector from source (required for vector search)
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# Point gbrain at your local Postgres
cat > ~/.gbrain/config.json << 'EOF'
{
"engine": "postgres",
"database_url": "postgresql://localhost:5432/gbrain",
"schema_pack": "gbrain-base-v2"
}
EOF
# Run migrations and verify
gbrain apply-migrations --yes
gbrain doctor
```
All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend.
> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again.
+11 -90
View File
@@ -26,21 +26,19 @@ Two equivalent paths:
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
- Follow the "E2E test DB lifecycle" steps in
[docs/TESTING.md](TESTING.md) to spin up the test DB, run
`bun run test:e2e`, then tear it down.
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
run `bun run test:e2e`, then tear it down.
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
**Always run typecheck before pushing.** Neither `bun test` (the bun runner)
nor `bun run test` gates on types — `bun run test` is just
`bash scripts/run-unit-parallel.sh` (the sharded unit runner; no typecheck,
no shell pre-checks — see the test-tier table in [docs/TESTING.md](TESTING.md)).
**Always run typecheck before pushing.** `bun test` (the bun runner)
skips TypeScript type checking — it only enforces runtime behavior.
Three ways to actually gate on types:
1. `bun run verify` — runs the shell guard checks (privacy, jsonb, source-id,
progress-to-stdout, …) plus `bun run typecheck` in parallel
(`scripts/run-verify-parallel.sh`). Use this mid-branch.
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
2. `bun run typecheck``tsc --noEmit` standalone. Fast (~5s on this repo).
3. `bun run ci:local` — the full local CI gate from Path A.
@@ -292,8 +290,7 @@ matter" with BrainBench-style before/after table, "what this means" closer, then
Create a migration file at `skills/migrations/v[version].md` when a release
includes changes that existing users need to act on. The auto-update agent
reads these files post-upgrade (see `docs/guides/upgrades-auto-update.md`)
and executes them.
reads these files post-upgrade (Section 17, Step 4) and executes them.
**You need a migration file when:**
- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
@@ -346,8 +343,8 @@ canonical.
## Schema state tracking
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
adopted, declined, or added custom. The auto-update agent
(`docs/guides/upgrades-auto-update.md`) reads this during upgrades to suggest new schema additions without re-suggesting
adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
reads this during upgrades to suggest new schema additions without re-suggesting
things the user already declined. The setup skill writes the initial state during
Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
@@ -365,82 +362,6 @@ done
If any SHA differs from what's in the workflow files, update the pin and version comment.
## GitHub releases (binary assets + self-update) — #3521
`.github/workflows/release.yml` publishes a GitHub release automatically for
**every VERSION bump that lands on master** (trigger: push to master touching
`VERSION`, plus `workflow_dispatch` for a manual first run or repair). No
manual tag push is part of the ship flow — the workflow reads `VERSION` (the
single source of truth), mints tag `v<VERSION>` at the pushed commit, titles
the release the same, uses that version's `CHANGELOG.md` entry as the notes
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
missing), and attaches the compiled binaries.
### The `latest-stable` tag
The **final step of the release job** force-advances the `latest-stable` tag to
the release commit (`git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"`).
`latest-stable` is the single sanctioned distribution ref: the README paste
block, the `BOOTSTRAP_FOR_AGENTS.md` fetch URL, and
`bun install -g github:garrytan/gbrain#latest-stable` all reference it
permanently, so paste blocks copied into the wild never rot and there is no 404
window between VERSION landing and assets publishing.
`scripts/check-bootstrap-tag.sh` keeps the entry docs pinned to this ref.
Because it moves ONLY after binaries + provenance attestation have fully
published, a half-built release never advances it. If the tag-advance step
alone fails, re-advance by hand (a full workflow re-run would skip — the
release already exists with all assets):
```bash
git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
```
### The `publish-template` job
After the release job, a `publish-template` job force-pushes the rendered
agent-workspace template repo (the GitHub "Use this template" door,
`vars.TEMPLATE_REPO`, default `garrytan/gbrain-agent-template`) from CI only —
no human pushes it by hand, so what adopters clone is exactly what this repo
reviewed. It is guarded three ways: the release above fully published; the
vendored tree `templates/bootstrap/template-repo/` exists (skip, never fail,
if not); and the `TEMPLATE_REPO_PAT` secret is configured (skip if not).
Before pushing, it regenerates the template tree
(`bun run scripts/generate-template-repo.ts`) and byte-diffs it against the
vendored copy — a mismatch fails the job; regenerate + commit the vendored
tree (`scripts/check-bootstrap-templates.sh` runs the same diff offline in
`bun run verify`).
**`TEMPLATE_REPO_PAT` scope:** a fine-grained PAT with `contents: write` on
the template repository ONLY — no other repositories, no other permissions.
Configure it as a repo secret; when absent, template publishing is disabled
and the job skips cleanly.
Why every bump, not selective: `gbrain check-update` resolves the latest
version from `VERSION` on master, while binary self-update
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
Any release that lags `VERSION` tells binary installs an upgrade exists that
self-update cannot apply. `releases/latest` must track `VERSION`.
Invariants:
- **Asset names are a contract.** The build matrix's `artifact:` names must
equal what `expectedAssetName()` in `src/core/binary-self-update.ts`
returns (`gbrain-darwin-arm64`, `gbrain-linux-x64` today). Adding a
platform means updating BOTH plus the version job's completeness check;
`test/release-workflow.test.ts` pins all of it.
- **Idempotent + self-repairing.** The version job skips when a release for
`v<VERSION>` already exists with all expected assets; a partial release
(tag but no release, or missing assets) is completed on re-run. Racing
master pushes queue via the `release` concurrency group — a skipped
intermediate version is fine, latest is what matters.
- **Historical tags are never rewritten.** Old 3-segment versions keep their
history; every new 4-segment `VERSION` mints a fresh tag.
- **Permissions stay scoped.** `contents: write` lives on the release job
only; everything else runs read-only.
- **Never advance `latest-stable` on a partial release.** The tag moves only
as the final release-job step, after every asset has published. Manual
re-advances must point at a fully published `v<VERSION>` release.
## PR descriptions cover the whole branch
+8 -60
View File
@@ -11,46 +11,20 @@ Seven test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### Shell dispatch and Windows
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
`scripts/*.ts` entries run under bun and take no prefix.
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. It pins `*.md` the
same way, because the frontmatter readers anchor on a `---` fence followed by a
Unix line ending and a CRLF checkout makes a document parse as having no
frontmatter, silently. Working copies cloned
before those pins need a one-time `git rm --cached -r . -q && git reset --hard` to
pick them up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
there even though they pass on Linux and macOS.
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. CI is the ground truth for "did everything pass."
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
### Failure-first logging
@@ -61,35 +35,19 @@ When `bun run test` finds any failure, the wrapper:
3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
If a shard hits the per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap (default 3000s — sized so the heaviest count-balanced shard finishes under 4-way contention; `GBRAIN_TEST_SHARD_KILL_AFTER` sets the grace after TERM before KILL, default 30s), the wrapper classifies the kill one of two ways:
- **EXIT-HANG → warn-pass.** If the shard's log had been silent for ≥300s at kill time AND shows zero `(fail)` markers, the shard finished all its work, leaked a handle, and never exited (a pre-existing, master-reproducible PGLite-adjacent leak — see TODOS.md "unit-shard exit hang"). The wrapper prints a `⚠️ shard N/M: EXIT-HANG ... Treating as pass-with-warning` banner, writes `EXIT-HANG (idle Ns, 0 fails) ... warn-pass` to the summary, and does NOT fail the run. Its pass counts are undercounted (bun never printed its final summary). Bun's per-test `--timeout` turns a genuinely hung TEST into a printed `(fail)` — new output — so this classification cannot mask a hung test; the residual maskable case is a file-level import hang in the very last file, which the banner keeps visible.
- **WEDGED → hard failure.** Anything else (failures present, or the log was still growing) writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log with the last 50 lines of the shard log, marks the run failed, and proceeds with other shards' results.
Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT a test failure — don't burn time bisecting it; a `WEDGED` line is.
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
### File taxonomy
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
### Skills-manifest freshness guard
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
### Test-isolation lint and helpers
**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules.
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
| Rule | What it bans | Fix |
@@ -175,9 +133,6 @@ Unit tests and what they cover:
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
- `test/watch-command.test.ts``gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
- `test/watch-sigint.serial.test.ts``gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
- `test/autopilot-launchd-lifecycle.serial.test.ts` — autopilot lifecycle behavior, not generated-string assertions: the full install → self-disable → status → reinstall → uninstall arc with `launchctl` replaced by an argv recorder and the generated wrapper executed by a REAL bash against a genuinely deleted repo (every platform), plus a darwin-only fail-SKIP describe against the real launchd under a per-run unique label (`GBRAIN_AUTOPILOT_LABEL`) so it can never collide with — or tear down — a real install on the host. Serial: spawns subprocesses and pins HOME/GBRAIN_HOME for the whole file.
- `test/autopilot-fanout.test.ts` — Autopilot fan-out and #4046 policy regression: targeted idempotency keys reopen per dispatch interval while stable doctor/remediate keys remain unchanged; the 60-minute full-cycle floor wins with a remaining small plan, and an all-fresh restart check advances the process-local clock without masking failed stale-source submissions.
- `test/agent-scheduler-contract.serial.test.ts` — the documented external agent-scheduler shell chain (`gbrain sync --repo X && gbrain embed --stale`, live-sync.md / INSTALL_FOR_AGENTS.md Step 7) driven end-to-end through a real `/bin/sh` against a keyless PGLite brain: the `&&` short-circuit IS the contract (argv arrays can't exercise it), the keyless bare stale embed exits 0, and the pull-failure case that must break the chain does. Anti-vacuity: the fixture commits a real page and every read-back asserts pages >= 1. Serial: real spawned CLI + tmpdir HOME.
- `test/cli-format-volunteer.test.ts``formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
- `test/config.test.ts` — config redaction.
- `test/files.test.ts` — MIME/hash.
@@ -211,8 +166,6 @@ Unit tests and what they cover:
- `test/dedup.test.ts` — source-aware dedup, compiled truth guarantee, layer interactions.
- `test/intent.test.ts` — query intent classification: entity/temporal/event/general.
- `test/eval.test.ts` — retrieval metrics: `precisionAtK`, `recallAtK`, `mrr`, `ndcgAtK`, `parseQrels`.
- `test/brainbench-fixtures.test.ts` / `test/brainbench-generator.test.ts` / `test/brainbench-metrics.test.ts` / `test/brainbench-continuity.test.ts` / `test/brainbench-writeback.test.ts` / `test/brainbench-adapters.test.ts` / `test/brainbench-scoreboard.test.ts` — the BrainBench memory-conformance unit suites (`src/eval/brainbench/`): fixture loader/validator + the sealed-gold seal (a `gold` key inside a fixture must reject) and committed-corpus integrity; generator determinism (the committed corpus is exactly what `gen.ts` produces, holdout discipline, category counts); metric formulas over hand-built turn rows (zero should-retrieve turns, empty injections, acceptable-vs-gold asymmetry, micro-averaging); cross-harness continuity (writer's decision persists through the production write-back pipeline, reader recalls on the SAME brain); write-back grading the PRODUCTION conversation→facts pipeline via the injected gold extractor; adapter seam contracts over hermetic PGLite (budget caps, suppression modes); scoreboard + gate governance (baseline determinism, count-aware gating, corpus-bless modes, justification flow, isolation gates-at-zero).
- `test/eval-brainbench-e2e.test.ts` — BrainBench CLI end-to-end via subprocess against a small tmp corpus: the literal exit codes (0 pass / 1 regression / 2 error-or-inconclusive — the CI product), `--out` artifact validity incl. `_meta.metric_glossary`, byte-deterministic `--update-baseline`, anti-vacuous-pass, and the `eval run-all` in-process wiring.
- `test/check-resolvable.test.ts` — resolver reachability, MECE overlap, gap detection, proximity-based DRY detection, `extractDelegationTargets` coverage.
- `test/dry-fix.test.ts` — auto-fix: three shape-aware expander pure-function tests; five guards (working-tree-dirty, no-git-backup, inside-code-fence, already-delegated within 40 lines, ambiguous-multi-match, block-is-callout).
- `test/doctor-fix.test.ts``gbrain doctor --fix` CLI integration: dry-run preview, apply path, JSON output shape.
@@ -236,10 +189,8 @@ Unit tests and what they cover:
- `test/orphans.test.ts` — orphans command: detection, pseudo filtering, text/json/count outputs, MCP op.
- `test/postgres-engine.test.ts``statement_timeout` scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against a reintroduced bare `SET statement_timeout`.
- `test/sync.test.ts` — sync logic + regression guard asserting top-level `engine.transaction` is not called.
- `test/sync-pull-failed-anchor.serial.test.ts`#3068 regression: a failed internal `git pull` (local-path origin vs `protocol.file.allow=never`) with zero imports returns `partial`/`pull_failed` (not `up_to_date`), freezes `last_commit` + `last_sync_at`, recovers after a manual pull; fall-through import of local commits preserved. Serial: pins `GBRAIN_HOME` to a temp dir for the whole file.
- `test/sync-concurrency.test.ts``autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping; `shouldRunParallel()` explicit-bypasses-floor contract; `parseWorkers()` validation rejecting `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars.
- `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract.
- `test/sync-all-missing-path.test.ts``sync --all --missing-path <fail|skip>` pure helpers: `parseMissingPathMode` (default fail, explicit values, loud rejection of bad/dangling values, never swallows a following flag) and `partitionMissingPathSources` (classification driven only by the injected pathExists predicate — no fs; null `local_path` passes through runnable; order preserved).
- `test/sync-failures.test.ts``classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` `AcknowledgeResult` shape + backfill on legacy entries.
- `test/doctor.test.ts` — doctor command; assertions that `jsonb_integrity` scans the four JSONB write sites and `markdown_body_completeness` is present.
- `test/utils.test.ts` — shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics.
@@ -282,9 +233,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972#2015#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
- `test/e2e/volunteer-context-postgres.test.ts``volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
- `test/e2e/openclaw-reference-compat.test.ts``check-resolvable` + skillpack install-model against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
- `test/e2e/workspace-generic-compat.test.ts` — always-on (PGLite, no binary): pins the INSTALL_FOR_AGENTS.md "any repo with a workspace" contract against `test/fixtures/generic-agents-workspace/` (Hermes is the motivating consumer): `cwd_walk_up` detection, the `GBRAIN_SKILLS_DIR` override, `check-resolvable` on a root AGENTS.md, and scaffold additivity + refuse-overwrite. The real Hermes-behavior proof is the door suite below.
- `test/e2e/install-real-hermes.serial.test.ts` — the hermes "door": real `hermes` binary + real `hermes mcp add` handshake (full-catalog tool discovery; the count tracks the op catalog, so the test asserts discovery happened, not a number) + a paid `hermes -z` recall turn against a seeded brain. Triple-gated: `GBRAIN_REAL_HERMES_E2E=1` (explicit opt-in — run-e2e.sh scrubs GBRAIN_*, so it can never fire under `bun run test:e2e`) + resolvable binary + non-empty ANTHROPIC key (anthropic-pinned on purpose: a second provider key flips hermes provider-auto into a mis-routed 401). Hermetic HOME + HERMES_HOME with a tripwire on the operator's real config; evidence copies to `GBRAIN_E2E_EVIDENCE_DIR` for CI upload. Venue: heavy-tests.yml (`real-agent-e2e` + `hermes-door` jobs).
- `test/e2e/openclaw-reference-compat.test.ts``check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts``test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
- `test/e2e/engine-parity.test.ts` — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector` (Postgres ranks pages then picks best chunk while PGLite returns chunks directly, so the source-boost behavior needs parity coverage). Skips without `DATABASE_URL`.
@@ -299,7 +248,6 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/think-source-isolation-pglite.test.ts` — PGLite in-memory suite pinning the `think` gather stage's source scope: seeds three sources with cross-source links and embedded takes, then asserts `runGather` under a federated `sourceIds` grant (and under a scalar `sourceId`) keeps every stream — hybrid retrieval, takes keyword + vector (`searchTakes`/`searchTakesVector`), and the `traversePaths` graph walk — inside the grant while still reaching authorized neighboring sources. No `DATABASE_URL` needed.
- `test/e2e/skill-brain-first.test.ts` — doctor reports `skill_brain_first` check with structured issues; `--fix --dry-run` previews insertion without writing; `--fix` applies the canonical Convention callout idempotently; `brain_first: exempt` frontmatter resolves the warn; `brain_first_typo` surfaces a paste-ready hint; audit JSONL records `detected` / `resolved` / `fixed` transitions; stable brain emits 0 audit lines/run.
- Tier 2 (`test/e2e/skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI.
- `test/e2e/claw-test.test.ts` also covers live mode token-free via shim agents (`OPENCLAW_BIN=<sh script>`): the success-oracle break path (a do-nothing agent now FAILS), the E0 child-friction merge surviving tempdir cleanup, and the upgrade staging + schema-version-probe regression.
- If `.env.testing` doesn't exist in this directory, check sibling worktrees: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
- **Run E2E tests without asking permission.** When you want to verify behavior, there's a relevant E2E test, or you're shipping anything covered by an E2E suite — spin up the test DB, run the tests, tear down. Don't ask, don't propose it, don't defer. The lifecycle is short (~2-30s startup, sub-minute tests, instant teardown) and the gate value is high. Skipping with "DATABASE_URL unset" is silent regression, not caution.
-8
View File
@@ -1,13 +1,5 @@
# Upgrading Downstream Agents
> **Currency note:** this file is an append-only historical log and stopped
> receiving new sections after v0.36.5.0. **The canonical, maintained upgrade
> channel is `skills/migrations/v*.md`** (the agent-executed migration files
> that `gbrain upgrade` / `gbrain post-upgrade` route through), plus
> `CHANGELOG.md` for what each release changed. Use this file only to catch a
> long-diverged fork up through the versions it covers; for anything after
> v0.36.5.0, walk the migration files and CHANGELOG instead.
GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments,
agent forks of any kind) often **copy** these skill files into their own workspace and
diverge over time — adding agent-specific phases, removing irrelevant ones, tightening
-18
View File
@@ -1,23 +1,5 @@
# ZeroEntropy — zembed-1 + zerank-2
> **Hosted API shutdown: 2026-09-04.** ZeroEntropy announced (2026-07-24)
> that its hosted endpoints — `/models/embed` and `/models/rerank` — shut
> down on that date. A brain still embedding through the hosted API loses
> semantic retrieval entirely on that date: query embedding uses the same
> endpoint, so **existing vectors become unqueryable**, not just new
> content. Two fixes, either works:
>
> 1. **Self-host the same model** — zembed-1 weights are Apache-2.0. Serve
> them via `llama-server` or Ollama and point the config at the local
> endpoint. Keeps every existing vector; no re-embed at all.
> 2. **Migrate to another provider** — `gbrain migrate embeddings --to
> <provider:model> --dim <N> --dry-run` (resumable; see
> [the migration guide](../guides/embedding-migration.md)). `gbrain
> doctor` (check `provider_sunset`) prints this command with your
> brain's actual `--dim` filled in.
>
> The hosted setup below remains accurate until the shutdown date.
[ZeroEntropy](https://zeroentropy.dev) ships two specialized small models
for retrieval pipelines:
File diff suppressed because one or more lines are too long
+22 -62
View File
@@ -4,14 +4,14 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
## The four strategies in concert
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at acme-example?" → pages mentioning "alice-example + retrieval" even when the user never typed "acme".
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
## Why each one alone fails
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in alice-example's portfolio" returns essays about portfolios, not company pages.
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
@@ -21,7 +21,7 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
## The benchmark
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus. (This is the retrieval-ranking benchmark; the in-repo `gbrain eval brainbench` suite — [`docs/eval/BRAINBENCH.md`](../eval/BRAINBENCH.md) — gates the memory behaviors *above* retrieval: unprompted context push, write-back fidelity, cross-session continuity.)
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus.
| Strategy | P@5 | R@5 | Notes |
|---|---|---|---|
@@ -36,8 +36,8 @@ BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/ga
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
- Standard markdown links: `[Alice Example](wiki/people/alice-example)`
- Obsidian wikilinks: `[[wiki/people/alice-example|Alice Example]]`
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
- Typed-link blockquotes: `> **Convention:** see [path](path).`
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') JOIN pages ON CONFLICT DO NOTHING RETURNING 1` (free-text-safe; the prior `unnest(${arr}::text[])` form crashed on calendar/Zoom context per gbrain#1861). The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
@@ -46,7 +46,7 @@ Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`,
## ZeroEntropy as reranker: 60% top-1 reshuffle
ZeroEntropy's `zerank-2` is the default reranker (on for the `balanced` and `tokenmax` mode bundles, off for `conservative`). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
@@ -62,9 +62,8 @@ The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `Sea
## Named-thing retrieval (per-page pool + title + alias + evidence)
A brain organized around *chosen names* (project codenames, place nicknames —
say a project named "Helios" whose page is also known as "the Sun Room") needs
more than embedding proximity. Four layers, added after the incident in
A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than
embedding proximity. Four layers, added after the incident in
[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md):
- **Per-page max-pool**`searchVector` (both engines) collapses chunk-grain
@@ -80,7 +79,7 @@ more than embedding proximity. Four layers, added after the incident in
`page_aliases` table (separate from the `slug_aliases` wikilink redirect) and
consulted at query time: a full normalized-query match injects/boosts the
canonical page (`applyAliasHop`). The only layer that bridges true synonyms
with zero surface overlap ("the Sun Room" → the Helios page). Backfill
with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill
existing pages with `gbrain reindex --aliases`.
- **Evidence contract** — every result carries `evidence`
(`alias_hit | exact_title_match | high_vector_match | keyword_exact |
@@ -88,27 +87,14 @@ more than embedding proximity. Four layers, added after the incident in
deciding "is this page already here, safe to NOT write a duplicate?" keys off
`create_safety`, not a raw blended score.
**Extraction quarantine lane (issue #160):** pages carrying the unverified
auto-extracted markers (frontmatter `provenance: auto-extracted` +
`status: unverified`, see `src/core/extraction-review.ts`) rank as ordinary
content — they are skipped by the compiled-truth fusion boost and by the
`people/`/`companies/` namespace source-boost, and every search result from
such a page carries `unverified: true` so agents can label the provenance.
Promote or reject them via `gbrain extraction-pending` / `gbrain
extraction-review`.
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
title + alias, expansion off); `query` is the full-control variant. Route
concept / landscape / "all-of-X" questions to `query` — expansion recovers
synonym-phrased matches `search` can miss, and a populated `search` result set
is not proof of coverage (both are top-K; exhaustive enumeration belongs to
`list_pages`). NamedThingBench
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
specific miss with `gbrain search diagnose "<q>" --target <slug>`.
## Intent-aware query rewriting
`src/core/search/query-intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
@@ -121,68 +107,42 @@ The classifier is deterministic (no LLM call). Wrong classification degrades gra
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale. The `query` op is the exception: it defaults `expand: true` per call (pass `expand: false` to opt out) — expansion-by-default is what makes it the concept/landscape verb.
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
## Putting it together
The full pipeline for a `query` op:
```
intent classify (query-intent.ts — deterministic, no LLM)
intent classify
expansion (if enabled — tokenmax only by default)
expansion (if enabled)
hybrid recall + fusion:
├── vector (HNSW on chunk embeddings, per-page max-pool)
hybrid search:
├── vector (HNSW on chunk embeddings)
├── keyword (BM25 via tsvector)
├── title-phrase arm
├── relational (typed-edge recall arm — relational queries only)
├── relational (v0.42.34.0: typed-edge recall arm — relational queries only)
├── source-aware re-rank (CASE in SQL)
└── RRF fusion → cosine re-score → post-fusion boosts
(backlink / salience / recency / graph signals / exact-match)
└── RRF fusion → top 30
graph augment (optional two-pass structural expansion — walkDepth > 0)
graph augment (typed-edge traversal from any seed)
deduplication (4-layer: per-page cap, Jaccard, type diversity)
reranker (zerank-2 cross-encoder, top 30 → reordered)
reranker (zerank-2 cross-encoder — balanced/tokenmax; fail-open)
token-budget enforcement (per mode bundle)
alias hop (exact alias match injects/boosts the canonical page)
evidence stamp → adaptive return (opt-in) → autocut (reranked modes)
limit slice → token-budget enforcement (per mode bundle)
deduplication (same slug, different chunks → keep best)
results
```
The stage order is pinned by `hybridSearch` in `src/core/search/hybrid.ts`:
dedup runs BEFORE the reranker (so the reranker sees a diverse candidate pool,
capped by its own `topNIn`), the alias hop runs AFTER the reranker (so a query
that is a page's declared name reliably surfaces that page regardless of how
the reranker scored body chunks), and the token budget is enforced last, on
the final slice.
### Autocut: score-discontinuity result-sizing
Default-on for `balanced` and `tokenmax` (off for `conservative`, which has no
reranker and therefore no trustworthy cliff signal). `applyAutocut`
(`src/core/search/autocut.ts`) cuts the ranked set at the largest
cross-encoder rerank-score cliff, before the limit slice, first page only.
Never-empty failsafe (`minKeep`), no-op when fewer than 2 results carry a
finite rerank score (covers the fail-open reranker path), and alias-hop exact
matches are preserved through the cut. Knobs: per-call `SearchOpts.autocut`
`search.autocut` / `search.autocut_jump` config → mode bundle.
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
## How to verify on your own brain
+9 -16
View File
@@ -19,18 +19,18 @@ need to understand both of them, or queries misroute silently.
A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase.
Each brain has:
- Its own `pages` table, `chunks` table, `embeddings`, etc.
- Its own OAuth surface if served over HTTP MCP.
- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2).
- Its own separate lifecycle, backup, access control.
Brains are enumerated by:
- **host** — your default brain, configured in `~/.gbrain/config.json`.
- **mounts** — additional brains registered in `~/.gbrain/mounts.json` via
`gbrain mounts add <id>`.
`gbrain mounts add <id>` (v0.19+).
Routing: `--brain <id>`, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or
longest-path match against registered mount paths. Falls back to `host`.
### Sources (the repo axis)
### Sources (the repo axis, v0.18.0+)
A **source** is a named content repo *inside* one brain. Every `pages` row
carries a `source_id`. Slugs are unique per source, not globally.
@@ -142,7 +142,7 @@ Use this topology when:
You're senior enough to sit across multiple teams. You maintain your personal
brain (with N sources inside) AND mount several work team brains. Each team
brain is itself a multi-source brain — organized
brain is itself a multi-source brain in the v0.18.0 sense — organized
internally however the team owner chose.
```
@@ -181,7 +181,7 @@ Use this topology when:
- You need latent-space federation (agent decides when to query across
brains), not SQL federation.
Cross-brain queries are **not deterministic**. The agent sees the
Cross-brain queries are **not deterministic** in v0.19. The agent sees the
brain list and re-queries as needed. That's the feature — it keeps debugging
sane and access control clean.
@@ -202,13 +202,6 @@ WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)?
Both axes follow the same layered pattern on purpose. If you know one, you
know the other.
One addition on the source axis for remote (MCP/OAuth) callers: a client
registered with federated reads carries `ctx.auth.allowedSources` — an
ARRAY of readable sources that takes precedence over the scalar
`ctx.sourceId` on every read path (`sourceScopeOpts(ctx)` in the
operations layer). Local CLI callers never set it; the scalar chain above
is the whole story for them.
---
## For agents reading this
@@ -243,7 +236,7 @@ is the whole story for them.
## Further reading
- [`topologies.md`](./topologies.md) — where the DB lives (operator recipes
for each deployment shape).
- `skills/conventions/brain-routing.md` — the agent-facing decision table.
- `CHANGELOG.md` — release history for the `sources` and `mounts` primitives.
- v0.18.0 CHANGELOG — introduced `sources` primitive.
- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`.
- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain
publisher, not just the subscriber.
@@ -10,16 +10,14 @@
> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80
> during successive master merges — v0.37.0.0's autonomous-remediation wave
> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79.
> - **Follow-up minor — NEVER IMPLEMENTED.** The falsifiability + category
> extraction at `propose_takes`, SQL-side grade gate, per-category
> calibration scorecards, and pg_trgm-based proposal dedup described in the
> sections below remain UNSHIPPED design. Do not read §§14 as current
> behavior; only the `unresolvable` hotfix above landed.
> - **Follow-up minor** (forthcoming): falsifiability + category extraction at
> `propose_takes`, SQL-side grade gate, per-category calibration scorecards,
> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation
> against the v0.36.1.0 fixtures.
>
> Preserved here per the hotfix plan's PR #1191 close protocol so the
> production context (falsifiability rate + category breakdown observed on a
> large real brain) doesn't get lost in the CHANGELOG → release-notes
> condensation.
> production context (96K-page brain, 6.8% falsifiability rate, category
> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation.
## Problem
@@ -1,146 +0,0 @@
# Conversation parser patterns
The conversation parser turns exported chat and meeting transcripts into a
common message stream without requiring an LLM call for known formats. This
document describes the built-in pattern contract and the checks required when
adding or changing a format.
## Data flow
`parseConversation` uses this sequence:
1. Resolve the page date and timezone context.
2. Score every enabled built-in and user pattern against the first ten
non-blank lines.
3. Re-score the full body when the head score is inconclusive, or when a broad
pattern explicitly requires full-body scoring.
4. Reject the winner when its acceptance score is below the false-positive
floor.
5. Apply the winning pattern to every line and attach continuation lines to the
preceding message.
6. Optionally run LLM polish or fallback when those features are enabled.
Pattern order is only a tie-breaker. A new regex must be structurally distinct
from neighboring formats; moving it earlier in the registry is not a valid
non-shadowing strategy.
## Built-in pattern contract
Every `PatternEntry` in `builtins.ts` declares:
- A stable, kebab-case `id`.
- A hand-vetted line regex and explicit capture-group indexes.
- Where the date comes from and how the time is represented.
- A timezone policy.
- Whether the format supports multi-line message bodies.
- Positive and negative samples that run during module initialization.
- A documentation pointer describing the source format.
The registry refuses to load when a positive sample stops matching, a negative
sample starts matching, or a capture map becomes invalid. This catches local
regex mistakes before extraction can silently produce empty conversations.
### Date and timezone rules
Formats with an inline date should capture it from each message. Time-only
formats use an explicit caller fallback first, then the page frontmatter date,
then the page effective date. If none is available, the parser uses
`1970-01-01` so the missing date remains visible instead of inventing a current
date.
Time-only formats normally use `utc_assumed_with_warn`. The parser constructs a
UTC timestamp and returns a timezone warning when the page does not provide a
timezone. A new pattern should not imply local-time precision that the source
format does not contain.
### Multi-line messages
An anchor regex identifies the first line of a message. Subsequent non-anchor
lines are appended to that message until another anchor appears. Set
`multi_line: true` when continuation content is part of the documented format,
such as Markdown bullets, blockquotes, or an exported message body on the next
line.
Tests for a multi-line format should assert the complete message text, including
newlines. A message-count assertion alone will not detect lost bullets or a
continuation attached to the wrong speaker.
### Scoring and false positives
The score compares matched anchors with the pattern's relevant candidate lines.
The first pass uses the head of the page for speed. Low-confidence pages are
re-scored across the full body before the parser accepts a winner.
Multi-line formats may opt into `score_continuations_as_body` when their anchor
grammar is distinctive. Candidate-only scoring activates only after two anchors
match, or when the first non-blank line is an anchor. This evidence threshold
lets a single long message keep its continuation body without turning one stray
anchor in a prose page into a conversation. Candidate anchor lines that fail the
full regex still lower the score. Other patterns continue to use all non-blank
lines in their density score.
Use `score_full_body: true` for a broad grammar that also occurs in ordinary
prose. For example, `**Label:** text` can be either a transcript line or a bold
label in meeting notes. Narrow formats with a timestamp and a distinctive
separator generally do not need this override.
`quick_reject` is a performance hint, not an acceptance rule. It should cheaply
exclude obviously unrelated lines while admitting every string accepted by the
main regex.
## Normalized Slack Markdown
The `bold-time-dash` pattern parses message anchors shaped like:
```text
**Alice Example** 09:15 — first message
- supporting detail
**Bob Example** 09:18 — second message
```
Its grammar is:
```text
**speaker** H:MM <dash> text
```
where:
- `H:MM` is a valid 24-hour time from `0:00` through `23:59`.
- `<dash>` may be an em dash (`—`), en dash (``), or ASCII hyphen (`-`).
- The date comes from the resolved page date context.
- Continuation lines belong to the preceding message.
- The captured clock value is emitted with `Z`. Timezone metadata suppresses
the missing-timezone warning but is not currently used for IANA conversion.
The required time and dash distinguish it from all existing bold-speaker
formats:
- `**Speaker** (09:15): text` uses `bold-paren-time`.
- `**Speaker** (9:15 AM): text` uses `bold-paren-time-12h`.
- `**Speaker:** text` uses `bold-name-no-time`.
- `**Speaker** (2026-04-09 9:15 AM): text` uses `imessage-slack`.
Keeping these examples in both `test_negative` and parser regression tests makes
the non-shadowing contract executable.
## Adding a built-in format
1. Collect multiple anonymized examples, including separator and timestamp
variants that occur in the same export family.
2. Choose the narrowest grammar that represents the format. Constrain numeric
fields such as hours and minutes when possible.
3. Add at least two positive module-load samples and negative samples for every
neighboring pattern that could plausibly overlap.
4. Add parser tests that verify speakers, timestamps, text, continuation
handling, and non-shadowing behavior.
5. Add a dedicated JSONL fixture and include the same cases in
`test/fixtures/conversation-formats/all.jsonl`.
6. Run the focused parser tests and the fixture evaluator.
7. Run the repository verification and full test suites before submission.
8. Update `docs/architecture/KEY_FILES.md` when the registry count or supported
format inventory changes.
Use generic fixture identities such as `Alice Example`, `Bob Example`, and
`Summary Bot`. Never copy real transcript names or private content into source,
tests, documentation, commits, or pull-request descriptions.
@@ -68,13 +68,11 @@ existing.
## Migration shape
```ts
// src/core/migrate.ts — append after the CURRENT last entry in the
// MIGRATIONS array (take the next unused version number at implementation
// time; the numbers below are placeholders, not a reserved slot)
// src/core/migrate.ts — append after the v80 entry
const migrations = [
// ...existing entries...
// ...existing v1-v80...
{
version: NEXT_VERSION, // next unused number in the MIGRATIONS array
version: 81,
name: 'frontmatter_scan_state',
sql: `
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
@@ -196,7 +194,7 @@ stale data as authoritative.
```
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
Design lives at docs/architecture/frontmatter-scan-incremental.md.
New schema migration + sync-side UPSERT + incremental scan command
Schema migration v81 + sync-side UPSERT + incremental scan command
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
table + writes first; flip the reader one release later.
```
+90 -18
View File
@@ -1,33 +1,105 @@
# GBrain Infrastructure Layer (orientation pointer)
# GBrain Infrastructure Layer
The shared foundation that all skills, recipes, and integrations build on.
This page is a router — the detailed, current-state references live in the
docs below (this file once carried its own copies of the pipeline and schema;
those rotted, so each concept now has exactly one home).
## Where things live
## Data Pipeline
| Topic | Home |
|---|---|
| Ingest pipeline (file resolution → frontmatter parse → content-hash idempotency → chunking → embedding → atomic write) | per-file entries in [`KEY_FILES.md`](./KEY_FILES.md): `src/core/import-file.ts`, `src/core/sync.ts`, `src/core/markdown.ts`, `src/core/embedding.ts`, `src/core/chunkers/*` |
| Chunking strategies (recursive / semantic / LLM-guided) | `src/core/chunkers/{recursive,semantic,llm}.ts` entries in [`KEY_FILES.md`](./KEY_FILES.md) |
| Search pipeline (hybrid RRF, graph, reranker, autocut, dedup, budgets) | [`RETRIEVAL.md`](./RETRIEVAL.md) |
| Search modes + cost knobs | `docs/guides/search-modes.md` + the CLAUDE.md Search Mode table |
| Per-file index of `src/` (what each file does + its invariants) | [`KEY_FILES.md`](./KEY_FILES.md) |
| Schema DDL | the `MIGRATIONS` array in `src/core/migrate.ts` (source of truth) + `src/schema.sql`; per-table classification in [`system-of-record.md`](./system-of-record.md) |
| Engines (PGLite vs Postgres, parity rules) | `docs/ENGINES.md` + the engine entries in [`KEY_FILES.md`](./KEY_FILES.md) |
| Operations contract (CLI + MCP generated from one source) | `src/core/operations.ts` (100+ operations; run `gbrain --tools-json` for the live list) |
| Brains vs sources (which database vs which repo inside it) | [`brains-and-sources.md`](./brains-and-sources.md) |
```
INPUT (markdown files, git repo)
FILE RESOLUTION (local → .redirect → .supabase → error)
MARKDOWN PARSER (gray-matter frontmatter + body)
→ compiled_truth + timeline separation
CONTENT HASH (SHA-256 idempotency check — skip if unchanged)
CHUNKING (3 strategies, configurable)
├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy
├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing
└── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates
EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions)
→ batch 100, exponential backoff, non-fatal if fails
DATABASE TRANSACTION (atomic: page + chunks + tags + version)
SEARCH (hybrid, available immediately)
```
## Search Architecture
GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search:
```
User Query
EXPANSION (optional: Claude Haiku generates 2 alternative phrasings)
├── VECTOR SEARCH (pgvector HNSW, cosine distance)
│ → 2x limit results per query variant
└── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank)
→ 2x limit results
RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly)
4-LAYER DEDUP
├── Best 3 chunks per page (source dedup)
├── Jaccard similarity > 0.85 (text dedup)
├── No type exceeds 60% (diversity)
└── Max 2 chunks per page (page cap)
TOP N RESULTS (default 20)
```
## Key Components
| File | Purpose |
|------|---------|
| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) |
| `src/core/postgres-engine.ts` | Postgres + pgvector implementation |
| `src/core/import-file.ts` | importFromFile + importFromContent pipeline |
| `src/core/sync.ts` | Git-based incremental change detection |
| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing |
| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff |
| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) |
| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection |
| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking |
| `src/core/search/hybrid.ts` | RRF merge of vector + keyword |
| `src/core/search/dedup.ts` | 4-layer result deduplication |
| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku |
| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) |
| `src/core/operations.ts` | Contract-first operation definitions (31 ops) |
| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) |
## Schema Overview
10 tables in Postgres:
- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB)
- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline)
- **links** — typed edges (knows, works_at, invested_in, founded, etc.)
- **tags** — many-to-many page tagging
- **timeline_entries** — structured events (date, source, summary, detail)
- **page_versions** — snapshot history for diff/revert
- **raw_data** — sidecar JSON from external APIs (preserves provenance)
- **files** — binary attachments in storage backend
- **ingest_log** — audit trail of import operations
- **config** — brain-level settings (version, embedding model, chunk strategy)
Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C).
Vector search uses HNSW index with cosine distance on content_chunks.embedding.
## The Thin Harness Principle
GBrain is the deterministic layer. Skills and recipes are the latent-space layer.
GBrain is the deterministic layer. Skills and recipes are the latent space layer.
See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full
architecture philosophy.
- **GBrain CLI** = thin harness (same input → same output)
- **Skills** (the bundled set routed by `skills/RESOLVER.md`) = fat skills
- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills
- **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure
The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work.
+21 -26
View File
@@ -1,4 +1,4 @@
# Lens packs
# Lens packs (v0.41.2.0)
Four bundled schema packs that turn the gbrain dream cycle into a multi-lens
brain. Activate one with `gbrain config set schema_pack <name>` and the cycle
@@ -7,7 +7,7 @@ picks up the pack's declared phases on the next `gbrain dream` run.
## The four packs
```
gbrain-base
gbrain-base (shipped v0.38)
│ extends
┌──────────────┼──────────────────────┐
@@ -60,33 +60,28 @@ conviction so high-stakes misses cost more).
### gbrain-engineer
Bridge-only pack. Declares `learning` page type + reuses base `code`.
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource
(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
each JSONL line as a `learning` page when this pack is active. Three
calibration domains: `architecture_calls` (scalar_brier),
`effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier).
Speculative ADR/postmortem/refactor_thesis/tech_debt types are
deferred — they'll ship when a real user authors the first one.
Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred
to v0.42+ — they'll ship when a real user authors the first one (D8).
### gbrain-everything
Meta-pack stacking creator + investor + engineer via the
Meta-pack stacking creator + investor + engineer via the v0.38
`extends` + `borrow_from` chain. Single-active-pack constraint
preserved — this IS the active pack; the registry walks extends +
borrow to materialize the merged view.
**Merge contract.** The full `extends` + `borrow_from` merge rules live in
[`schema-packs.md` § Merge contract](./schema-packs.md#merge-contract-extends--borrow_from).
The one rule that matters here: `phases` and `calibration_domains` are
**NOT** inherited (they gate cycle execution, so each pack must declare
its own participation explicitly) — which is why `gbrain-everything`
re-declares all its phases and all 7 `calibration_domains`.
Activate via `gbrain config set schema_pack gbrain-everything` and
calibration_profile produces all 7 domain scorecards in one JSONB.
## Calibration profile domains
## Calibration profile widening (T10)
Each declared
Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a
`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared
domain produces a `{n, brier, accuracy, aggregator, page_types,
extras}` entry. Four aggregator algorithms (closed enum):
@@ -105,20 +100,20 @@ Domain names are OPEN (third-party packs can declare new domain labels
without a gbrain release). Aggregator algorithms are CLOSED (safe SQL
stays in code, validated at pack-load).
## take_domain_assignments table
## take_domain_assignments table (T1)
JOIN table (migration v94):
New JOIN table (migration v94):
`take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT,
source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id,
domain))`. Multi-domain assignment honest — a take about "fund-a's
investment in acme-example" can land in BOTH `deal_success` AND
domain))`. Multi-domain assignment honest — a take about "Sequoia's
investment in Anthropic" can land in BOTH `deal_success` AND
`market_call` rather than being force-bucketed.
## What this enables for the user
- **Atoms + concepts ship in the binary.** Your OpenClaw's parallel
atom-pipeline-coordinator + atom-backfill-coordinator + concept-
synthesis crons can retire. One `gbrain dream` cron
synthesis crons can retire (T12 follow-up). One `gbrain dream` cron
covers everything.
- **gstack learnings reach gbrain.** Engineer-pack-active brains
surface every gstack-logged learning as a queryable page within
@@ -127,22 +122,22 @@ investment in acme-example" can land in BOTH `deal_success` AND
often you're wrong on deals AND market calls AND architecture
AND effort estimates in one `gbrain calibration --json` call.
- **Lossless OpenClaw migration.** The `markdown-greenfield`
importer (mode='migration') re-ingests existing OpenClaw
importer (T7, mode='migration') re-ingests existing OpenClaw
pages with permanent slug-keyed idempotency + per-row JSONL audit
+ the `imported_from` marker so extract_atoms + synthesize_concepts
don't re-extract already-atomized material.
## Known gaps / deferred follow-ups
## v0.41.2.1 follow-ups (filed in plan)
- Per-page-type `frontmatter_validators` on PageTypeSchema so the
atom_type enum (currently hardcoded in extract_atoms.ts) reads from
the active pack manifest at runtime.
the active pack manifest at runtime per D11.
- 3-check quality gate (truism / punchline / entity-page reject) as
a multi-pass extract_atoms refinement.
- Embedding-similarity dedup in synthesize_concepts (currently
exact-string concept ref match only).
- Voice gate integration for concept narratives.
- Voice gate integration for T1 Canon narratives.
- op_checkpoint resumability for cross-cycle continuation in both
phases.
- Parity-baseline eval gates against a pre-existing downstream
atom/concept corpus on a sample subset.
- Parity-baseline eval gates against your OpenClaw's existing 13K atoms
+ 11K concepts on a 500-page sample subset.
+13 -14
View File
@@ -56,8 +56,7 @@ that tuple lights up the `pack_upgrade_available` onboard check.
│ gbrain onboard --check --explain shows per-cluster narrative │
│ User reviews; if OK, runs: │
│ gbrain jobs submit unify-types --allow-protected \ │
│ --params '{"target_pack":"gbrain-base-v2","apply":true}'
│ (omit "apply":true for a dry-run; that is the default) │
│ --params '{"target_pack":"gbrain-base-v2"}'
│ (Autopilot never auto-fires this; manual_only) │
└──────────────────────────┬─────────────────────────────────────┘
@@ -127,10 +126,10 @@ candidate ≠ the active pack name, loads the manifest via
migration_from.version)`. Returns matching packs sorted by version
descending.
Successor detection covers bundled packs only. Future work: enumerate
user-installed packs at `~/.gbrain/schema-packs/*/pack.yaml` (deferred
because the filesystem-scan cost needs the cache invalidation strategy
from `registry.ts`).
v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed
packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the
filesystem-scan cost needs the cache invalidation strategy from
`registry.ts`).
## The manual_only apply policy
@@ -173,8 +172,8 @@ migration_from:
version: "1.x"
page_types:
# Inherit gbrain-base-v2's 15 types here (or declare `extends:
# gbrain-base-v2` and let the merge contract in schema-packs.md merge them)
# Inherit gbrain-base-v2's 15 types here (or use extends to merge
# automatically once v0.43+ extends-chain composition lands)
- { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true }
- { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true }
# ... all 13 other v2 canonicals ...
@@ -222,17 +221,17 @@ Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl`
identities (before + after), per-phase counts (would_apply + applied),
warnings, completion timestamp. Privacy: page slugs are NOT logged in
bulk (only the per-rule sample_slugs[≤10]); for forensic debugging
a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired.
add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
## What's NOT yet supported
- Subprocess sandbox for the publish-gate
- Subprocess sandbox for the publish-gate (v0.43+ TODO)
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (a proposed
`gbrain schema detect-mappings`)
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
## Reference
@@ -242,6 +241,6 @@ a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired.
- Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable`
- Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS`
- Handler: `src/core/schema-pack/unify-types-handler.ts`
- Migration: the `slug_aliases` entry in `src/core/migrate.ts`'s `MIGRATIONS` array
- Migration: `src/core/migrate.ts:105` (slug_aliases table)
- Type taxonomy doc: `docs/architecture/type-taxonomy.md`
- Skill: `skills/schema-unify/SKILL.md`
+48 -81
View File
@@ -7,25 +7,20 @@ paths, and which link verbs connect what to what. The schema pack is the
querying, or routing experts. It is the single source of truth for
"what's in your brain."
This doc is the user-facing reference; for implementation details see
`docs/designs/V038_SCHEMA_PACKS.md` (the original design) and the engine
layer in `src/core/schema-pack/`.
The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the
user-facing reference; for implementation details see
`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in
`src/core/schema-pack/`.
## What ships in the box
Seven bundled packs (`src/core/schema-pack/base/`):
Two bundled packs:
- **`gbrain-base-v2`** — the 15-type canonical taxonomy. Fresh installs
(`gbrain init`) activate this by default. See
[`type-taxonomy.md`](./type-taxonomy.md) for the full type list and the
upgrade path from `gbrain-base`.
- **`gbrain-base`** — the original hardcoded behavior, byte-for-byte
(person, company, deal, meeting, project, place, concept, writing,
analysis, guide, hardware, architecture, etc. — the original
`ALL_PAGE_TYPES` list). Still the resolution-chain fallback (tier 7)
for brains with no pack configured anywhere, so pre-existing brains see
zero behavior change until they opt in to something newer.
- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior
byte-for-byte. Existing brains see zero behavior change after upgrade.
Covers: person, company, deal, meeting, project, place, concept, writing,
analysis, guide, hardware, architecture, etc. (the original
`ALL_PAGE_TYPES` list).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional
directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal,
@@ -37,17 +32,12 @@ Seven bundled packs (`src/core/schema-pack/base/`):
gbrain schema use gbrain-recommended
```
- **`gbrain-creator`**, **`gbrain-investor`**, **`gbrain-engineer`**,
**`gbrain-everything`** — the lens packs, which add cycle phases and
calibration domains on top of the base taxonomy. See
[`lens-packs.md`](./lens-packs.md).
Plus user-installed packs at `~/.gbrain/schema-packs/<name>/pack.yaml`
that you author with `gbrain schema init` or `gbrain schema fork`.
## CLI surface
Inspection verbs:
Five inspection verbs (shipped in v0.38):
```bash
gbrain schema active # show resolved pack + which tier set it
@@ -57,7 +47,7 @@ gbrain schema validate # validate a manifest's shape
gbrain schema use <pack> # activate a pack (writes ~/.gbrain/config.json)
```
Authoring + discovery verbs:
Eight authoring + discovery verbs (shipped in v0.39):
```bash
gbrain schema detect # propose types matching brain shape
@@ -72,12 +62,12 @@ gbrain schema graph # ASCII type listing (experimental)
gbrain schema lint # flag duplicates + missing prefixes
gbrain schema explain <type> # plain-English type description (experimental)
gbrain schema downgrade --to <p> # restore previous pack (recovery)
gbrain schema usage --since 30d # per-verb invocation counts (telemetry)
gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry)
```
The verbs marked `experimental` are demand-gated: usage is tracked via the
schema-events audit (`gbrain schema usage`), which informs whether
rarely-used verbs get deprecated.
The verbs marked `experimental` are demand-gated per D14: their usage is
tracked via T15's schema-events audit, and v0.40+ retro decides whether
to deprecate any that stay <5% usage.
## Resolution chain (7 tiers)
@@ -88,10 +78,10 @@ this chain top-down. First match wins.
|------|--------|-------|
| 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. |
| 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. |
| 3 | Per-source DB config key `schema_pack:source:<id>` | |
| 3 | Per-source DB config key `schema_pack:source:<id>` | New in v0.38. |
| 4 | Brain-wide DB config key `schema_pack` | |
| 5 | `gbrain.yml schema:` section | Repo-checked. |
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` (and `gbrain init`, which sets `gbrain-base-v2`) writes. |
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. |
| 7 | Default: `gbrain-base` | Always present. |
## How the agent uses the active pack
@@ -107,18 +97,18 @@ Every read + write path consults the active pack at runtime:
- **`extract_facts`** runs only on `extractable: true` types.
- **`enrichment-service`** routes person/company enrichment based on the
pack's primitive declarations.
- **Search hybrid cache** (`knobsHash`) folds in pack name + version.
A cache row written under pack A is unreachable when pack
- **Search hybrid cache** (`knobsHash`) folds in pack name + version
(v0.39 T21). A cache row written under pack A is unreachable when pack
B is active. Cross-pack contamination is structurally impossible.
## The magical moment
## The magical moment (T2-T4 + T10)
Persona A (Notion refugee) installs gbrain, imports her exports, and the
brain looks unfamiliar — the default `gbrain-base` pack expects
`people/`, `companies/`, etc., but her files live under `Projects/`,
`Reading/`, `Daily Notes/`. The friction signal fires in two places:
1. **Import warn:** the end of `gbrain import` prints
1. **Import warn (T7):** the end of `gbrain import` prints
`[schema] X of Y pages (Z%) have no type matching the active schema
pack. Run gbrain schema detect to propose a pack matching your
content shape.`
@@ -134,7 +124,7 @@ gbrain schema review-candidates # human gate on promotion
gbrain schema review-candidates --apply Projects/ # accept
```
The agent (via the EIIRP skill, `skills/eiirp/SKILL.md`) automates phases 1-3 of this for any
The agent (via the new EIIRP skill) automates phases 1-3 of this for any
significant work session. The brain's schema becomes a living artifact
the agent maintains, not a hardcoded ceremony the user authors.
@@ -155,7 +145,7 @@ api_version: gbrain-schema-pack-v1
name: my-pack
version: 0.0.1
gbrain_min_version: 0.39.0
extends: gbrain-base # inherits base's TYPES (see Merge contract below); add overrides
extends: gbrain-base # inherits everything from base; add overrides below
description: |
My personal pack.
@@ -180,39 +170,11 @@ enrichable_types: []
filing_rules: []
```
## Merge contract (`extends` + `borrow_from`)
This section is the single home for the merge rules (other docs link here).
`resolvePack` composes a pack against its `extends` chain (and any
`borrow_from` targets) into the `resolved.manifest` every consumer reads.
The rules:
- **Six fields inherit, child-wins:** `page_types`, `link_types`,
`frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`.
A child value with the same key (type name, link name, etc.) overrides the
parent's; keys the child doesn't declare come through from the parent.
- **`page_types` ordering:** overrides of a base type keep the base's declared
position (base's `inferType` prefix priority is authoritative); a genuinely
new type — from the child, a `borrow_from`, or a middle pack in the chain —
is prepended nearest-first, so a more-derived type's `path_prefix` wins
regardless of how deep the chain is.
- **`takes_kinds` is UNION, not replace** — it carries a Zod default, so an
omitted field is indistinguishable from an explicit one. A child can ADD
kinds but **cannot narrow** `takes_kinds` below base parent. If you need a
smaller set, don't `extends` a pack that declares the larger one.
- **`phases` and `calibration_domains` are NOT inherited** (child-only). They
gate real cycle execution, so each pack must declare its own participation
explicitly — inheriting them would silently make a child run phases it never
requested. This is why `gbrain-everything` re-declares all its phases and
calibration domains by hand. See `lens-packs.md` for the worked example.
- **`borrow_from` is selective + non-transitive + fail-closed:** it pulls only
the named `types`/`link_types` from the target's OWN declarations (omitting a
category borrows none of it); a missing target throws `UnknownPackError`.
## Recovery + revert
A pack activation is config, not code, so reverting code alone doesn't
undo it. `gbrain schema downgrade` restores the active-pack config field:
The single-PR cathedral is hard to revert atomically. Per codex finding
#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore
the active-pack config field:
```bash
gbrain schema downgrade --to gbrain-base
@@ -224,19 +186,19 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h`hard-deletes
soft-deleted pages that no longer have a matching type in the active
3. (Optional) `gbrain pages purge-deleted --older-than 0h`drops
v0.39-typed pages that no longer have a matching type in the active
pack.
The cache + eval rows that pack-aware code wrote are isolated by the
`knobsHash` pack-folding — they become unreachable under the
`knobsHash` pack-folding (T21) — they become unreachable under the
restored pack so no eviction is needed.
## Distribution
`.gbrain-schema` tarballs ride the same distribution pipeline as
`.gbrain-skillpack` tarballs. The discriminator is `api_version` in the
manifest:
`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as
`.gbrain-skillpack` tarballs (T14 artifact abstraction). The
discriminator is `api_version` in the manifest:
- `gbrain-schema-pack-v1` → schemapack
- `gbrain-skillpack-v1` → skillpack
@@ -247,17 +209,22 @@ respectively.
Publication to the public registries (`garrytan/gbrain-schema-registry`,
`garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR
workflow as skillpack publishing.
workflow as v0.37 skillpack publishing.
## Known limits / deferred work
## What's deferred to v0.40+
- **Per-source pack federation across mounts.** A query crossing multiple
sources rejects with `permission_denied` when those sources have
divergent active packs (`src/core/schema-pack/op-trust-gate.ts`). A true
per-source closure via the existing `buildSourceClosureCte` engine
surface remains future work.
- **Pack version upgrades** (e.g. `gbrain-base``gbrain-base-v2`) are
handled by the successor-detection + unify-types mechanism — see
[`pack-upgrade-mechanism.md`](./pack-upgrade-mechanism.md).
sources currently rejects with `permission_denied` when those sources
have divergent active packs (T19 + codex finding #2). The v0.40+ work
computes a true per-source closure via the existing
`buildSourceClosureCte` engine surface.
- **`extends` chain semver compatibility checks** between pack versions.
- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack
can declare "I work best with these primitives present in your pack."
- **Live schema migration helpers** — when you add a type, auto-suggest
backfill of existing pages.
- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the
full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+
retro reads T23 usage telemetry to decide which to deprecate.
The live deferred list is in `TODOS.md`.
See `TODOS.md` v0.40+ section for the full deferred list.
@@ -52,13 +52,3 @@ gbrain sync --no-schema-pack --no-pull --no-embed --yes
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
The manual diagnosis above has an automated cousin: the progress-aware stall
watchdog. If the import drain makes no forward progress for
`GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900; keyed on file-import
progress, not the lock heartbeat), the run aborts with
`reason: 'stall_timeout'` and releases the per-source lock so the next
`gbrain sync` resumes from the checkpoint. It fires BETWEEN files — a hang
inside one file's import runs until the wall-clock hard deadline. `0`
disables it. The full sync-resumability knob table lives in CLAUDE.md
("Sync resumability + lock tuning").
+13 -11
View File
@@ -19,13 +19,11 @@ entire DB from scratch.
This means:
- **Disaster recovery is a short, boring sequence.** If your DB volume
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
don't need a backup. You wipe the derived tables (on PGLite,
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
your brain repo with `gbrain sync`, and `gbrain extract all`
regenerates the derived state. See "Disaster recovery" below for the
exact commands.
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
@@ -88,7 +86,6 @@ the repo. The architectural rule still holds — these aren't
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
| `op_checkpoint_paths` | Sync-resume checkpoint. Append-only progress banking; a completed sync makes it irrelevant. |
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
A new derived table that holds user-knowledge MUST land FS-first.
@@ -149,9 +146,11 @@ The promise the rule makes:
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild — delete the derived tables (pages + content_chunks
# survive the CASCADE-safe design), then re-derive from the repo.
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
@@ -190,6 +189,9 @@ reconciler / migration layer without the explicit allow-list comment.
## Related
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
and round 2 finds)
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
- `scripts/check-system-of-record.sh` — the CI gate that enforces
+45 -40
View File
@@ -3,63 +3,68 @@
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
only; release history lives in `CHANGELOG.md` + git.
`gbrain init --mcp-only` sets up a thin-client install: no local brain content,
just an OAuth client pointing at a remote `gbrain serve --http`. Every operation
surface routes through the remote brain — a thin-client install never opens the
empty local PGLite, so a populated remote brain can't silently return
"No results." Local-only commands refuse with a pinpoint hint instead of
falling through.
`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local
brain content, just an OAuth client pointing at a remote `gbrain serve --http`.
v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25
silently fell through to `connectEngine()` and opened the empty local PGLite,
returning "No results." against a populated remote brain. v0.31.1 fixes the
silent-empty-results bug class for every operation surface.
Key files (per-file detail lives in each file's `KEY_FILES.md` entry; this doc
carries the routing-seam picture):
Key files:
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (no
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no
parallel `src/core/thin-client/` module; routing is a ~80-line conditional
in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine`
so thin-client installs never open the empty PGLite. localOnly ops on
thin-client refuse via `refuseThinClient` (with pinpoint hint table
`THIN_CLIENT_REFUSE_HINTS`, which covers the full DB-bound command surface —
sync, embed, extract, migrate, enrich, dream, jobs, sources, pages, files,
eval, code-*, and more). Banner via `printIdentityBannerBestEffort`
`THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort`
before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`,
non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason`
for canned, actionable error messages. Renderer parity: the local-engine
for canned, actionable error messages. ENG-2 renderer parity: local-engine
path runs `JSON.parse(JSON.stringify(result))` so renderers see the same
shape on both paths (kills the Date/bigint/Buffer drift class).
- `src/core/mcp-client.ts``callRemoteTool(config, toolName, args, opts)`,
the transport under the routing seam. All transport errors normalize to
`RemoteMcpError` via the `toRemoteMcpError` funnel, with a stable
`RemoteMcpErrorReason` union the dispatcher's `never` switch keys off.
Full symbol-level detail: the `src/core/mcp-client.ts` entry in
[`KEY_FILES.md`](./KEY_FILES.md).
- `src/core/cli-options.ts``parseGlobalFlags` supports `--timeout=Ns`
(accepts `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command
default (30s for most ops, 180s for `think`). `parseTimeout(s)` exported
helper.
- `src/core/doctor-remote.ts``gbrain remote doctor` includes the
`oauth_client_scopes_probe` check. Probes the read tier via
`get_brain_identity` and the admin tier via `get_health`; reports per-tier
shape on both paths (kills Date/bigint/Buffer drift class).
- `src/core/mcp-client.ts``callRemoteTool(config, toolName, args, opts)`.
Hardened in v0.31.1 (CDX-4): all transport errors normalized to
`RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions
{timeoutMs, signal}`; `buildAbortController` composes external signal with
timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind`
('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code`
field carrying server-supplied error codes (e.g. `missing_scope`).
`extractToolErrorCode` parses JSON envelopes first, falls back to substring
detection for legacy server messages. `unpackToolResult<T>(res)` unchanged
(parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape.
- `src/core/cli-options.ts``parseGlobalFlags` adds `--timeout=Ns` (accepts
`30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s
for most ops, 180s for `think`). `parseTimeout(s)` exported helper.
- `src/core/doctor-remote.ts``gbrain remote doctor` adds the
`oauth_client_scopes_probe` check (CDX-5). Probes the read tier via
`get_brain_identity` and admin tier via `get_health`; reports per-tier
status with pinpoint remediation when admin is missing. `buildScopeCheck`
+ `ScopeProbeResult` exported for test access. Skippable via
`GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC
initialize level only (MCP SDK Client hangs on shape mismatch).
- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries.
- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller).
- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality``'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error.
- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).
- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex).
- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed.
- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on).
- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command).
- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows.
- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)``date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query.
- `src/core/operations.ts``get_brain_identity` op (read scope, no params,
banner-only): cheap counter packet `{version, engine, page_count,
chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses
`engine.getStats()`; the banner's 60s client-side TTL bounds frequency to
≤1/60s per CLI process.
`engine.getStats()`; banner's 60s client-side TTL bounds frequency to
≤1/60s per CLI process (well below the Fly.io health-check cadence that
motivated the original `getStats` cost warning).
- `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command
thin-client routing branches. These commands bypass the operation-layer
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
to op params. `think` is a special case: the server's `think` op is
read-scoped for OAuth/MCP and intentionally disables `--save`/`--take` for
remote callers (the `safeSave`/`safeTake` trust-boundary gate in the `think`
handler in `operations.ts`); thin-client `think` warns loudly when those
flags are set.
Cross-modal search files (image query, SSRF-guarded image loading, spend
tracking, multimodal reindex) are indexed per-file in
[`KEY_FILES.md`](./KEY_FILES.md) and described behaviorally in
[`RETRIEVAL.md`](./RETRIEVAL.md) — they are not part of the thin-client
routing seam.
to op params. `think` is a special case: the server's `think` op
intentionally disables `--save`/`--take` for remote callers
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
loudly when those flags are set.
+5 -10
View File
@@ -108,13 +108,11 @@ instead of a local DB connection:
}
```
The CLI dispatch guard refuses every DB-bound command (`sync`, `embed`,
`extract`, `migrate`, `serve`, `enrich`, `jobs`, `sources`, `pages`,
`files`, `eval`, and the rest of the local-only surface — the full hint
table is `THIN_CLIENT_REFUSE_HINTS` in `src/cli.ts`) on a thin-client
install with a clear error pointing at the remote host. `gbrain doctor`
runs a dedicated thin-client check set (OAuth discovery, token round-trip,
MCP smoke). See [`thin-client.md`](./thin-client.md) for the routing seam.
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
`integrity`, `serve`) on a thin-client install with a clear error pointing
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
(OAuth discovery, token round-trip, MCP smoke).
### Setup
@@ -396,9 +394,6 @@ simultaneously — that's by design.
## See also
- `docs/guides/bootstrap.md``gbrain bootstrap`, the paved-road paste-in
install for Topology 1 with a desktop coding agent (interview, hooks,
MCP registration, verify).
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
vs sources axes).
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
+11 -14
View File
@@ -1,7 +1,7 @@
# Type Taxonomy (gbrain-base-v2)
# Type Taxonomy (v0.41.22: gbrain-base-v2)
> The 14-canonical-type DRY/MECE taxonomy. Predecessor
> `gbrain-base` (24 types) stays bundled for back-compat; fresh installs
> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor
> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs
> default to `gbrain-base-v2`.
## Why
@@ -76,10 +76,9 @@ gbrain onboard --check --explain # per-cluster narrative dry-run
gbrain jobs submit unify-types \ # PROTECTED + manual_only
--allow-protected \
--params '{"target_pack":"gbrain-base-v2","apply":true}'
# omit "apply":true → dry-run (default)
--params '{"target_pack":"gbrain-base-v2"}'
Handler runs 8 phases:
Handler runs 4 phases:
┌─────────────────────────────────────┐
│ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL)
├─────────────────────────────────────┤
@@ -109,8 +108,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
@@ -128,8 +127,7 @@ For brains with substantial custom types that deserve their own canonical
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
--params '{"target_pack":"my-pack","apply":true}'` (omit `"apply":true`
for a dry-run preview — that is the default)
--params '{"target_pack":"my-pack"}'`
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
version: "1.x"}` to register itself as a successor — future agents
@@ -165,10 +163,8 @@ explicitly disambiguated this as canonical, so it should outrank fuzzy
matches that hit aliases by accident."
`SearchResult.alias_resolved_boost` is stamped on touched results for
`--explain` formatter visibility. The stage participates in the search
cache key (`KNOBS_HASH_VERSION` in `src/core/search/mode.ts` is the
single source of truth for the current cache-key version), so cache rows
written before the stage existed are unreachable.
`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to
invalidate pre-v0.42 cache rows that don't reflect the new stage.
## Reference
@@ -178,3 +174,4 @@ written before the stage existed are unreachable.
- Migration handler: `src/core/schema-pack/unify-types-handler.ts`
- Onboard checks: `src/core/onboard/checks.ts`
- Skill: `skills/schema-unify/SKILL.md`
- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md`
+19 -21
View File
@@ -1,4 +1,4 @@
# gbrain eval suspected-contradictions
# gbrain eval suspected-contradictions (v0.32.6)
The contradiction probe samples retrieval results, asks an LLM judge whether
any pair contradicts on a factual claim relevant to the user's query, and
@@ -142,27 +142,25 @@ pay near-zero on re-runs (until you bump PROMPT_VERSION).
gate makes accidental private-data commits hard, but the operator MUST
inspect every redaction before commit.
## Temporal axis
The judge distinguishes real contradictions from legitimate change-over-time.
The verdict enum has six members (`no_contradiction | contradiction |
temporal_supersession | temporal_regression | temporal_evolution |
negation_artifact`), and `pages.effective_date` is threaded into the judge
prompt so the probe doesn't cry wolf on facts that simply changed.
The trajectory substrate builds on the same signal:
`gbrain eval trajectory <entity>` shows the chronological typed-claim
history with regressions flagged inline; `gbrain founder scorecard
<entity>` rolls up four signals (accuracy, consistency, growth
direction, red flags) into a stable JSON contract. MCP op
`find_trajectory` (read scope, visibility-filtered for remote callers)
exposes the same data to agents. The probe's `temporal_supersession`
verdict and the consolidate phase's `valid_until` writeback both
preserve the `auto-supersession.ts` "NEVER auto-applies" invariant
— the probe only emits paste-ready commands; only `consolidate`
writes `valid_until` (a grep guard pins this).
## See also
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
+ trend-tracking workflow.
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
six-member verdict enum (`no_contradiction | contradiction |
temporal_supersession | temporal_regression | temporal_evolution |
negation_artifact`) and threaded `pages.effective_date` into the judge
prompt so the probe stops crying wolf on legitimate change-over-time.
v0.35.7 lands the trajectory substrate the probe pointed at:
`gbrain eval trajectory <entity>` shows the chronological typed-claim
history with regressions flagged inline; `gbrain founder scorecard
<entity>` rolls up four signals (accuracy, consistency, growth
direction, red flags) into a stable JSON contract. MCP op
`find_trajectory` (read scope, visibility-filtered for remote callers)
exposes the same data to agents. The probe's `temporal_supersession`
verdict and the consolidate phase's `valid_until` writeback both
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
— the probe still emits paste-ready commands, only `consolidate`
writes `valid_until` (R1+R8 grep guard pins this).
-130
View File
@@ -1,130 +0,0 @@
# Agent Bootstrap — Product Design (normative for scope & sequencing)
**Status:** APPROVED (product review, 3 adversarial rounds; CEO review; eng review —
0 unresolved decisions). Implementation source of truth:
[AGENT_BOOTSTRAP_PLAN.md](AGENT_BOOTSTRAP_PLAN.md).
## Problem
Agent platforms (OpenClaw, Hermes) deliver the full personal-agent experience —
scheduled work, skill files, SOUL.md identity, persistent memory — but setting one up
means deploying a server, wiring API keys, and paying raw token prices. Meanwhile
nearly everyone already runs Claude Code or Codex, both subsidized by subscriptions,
both capable of executing an install protocol themselves.
**The feature:** a paste-in install that adds memory + skills + identity +
session-triggered schedules + knowledge persistence to a private GitHub repo,
mimicking the agent-platform experience on the desktop apps people already have.
"Just use your local harness as your agent."
## Target surfaces & order
| Surface | In v1? | Order | Per-turn context seam | Persistence (write) seam |
|---|---|---|---|---|
| Codex in ChatGPT desktop | YES | 1st | spike exit question #2 | spike exit question #1 (blocking); fallback: mandated MCP writes + end-of-session sweep |
| Codex CLI | YES | with 1st | AGENTS.md pull protocol + `volunteer_context` | mandated MCP write ops |
| Claude Code desktop | YES | 2nd | hooks: SessionStart / UserPromptSubmit via serve IPC | SessionEnd hook: transcript ingest + scan-gated push |
| Claude Code CLI | YES | with 2nd | same hooks | same hooks |
The wedge is NOT platform feature-parity on day one. It is: paste → interview → an
agent that knows who it is and who you are → recalls what you told it in the next
session → visibly compounds during week one.
## The repo format (the contract)
A private GitHub repo is the product artifact — the agent's portable body:
- **Manifest:** `agent.json` (`format_version: 1`, provisional; `initialized`
sentinel distinguishes a template clone from a bootstrapped workspace).
- **Identity:** SOUL.md, USER.md, MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md,
ACCESS_POLICY.md, GITHUB.md — rendered ONLY from interview answers, never invented.
- **Content:** `brain/` (the gbrain source), `memory/` (daily notes), `skills/`,
`state/` (committed: interview.json, portable mcp.json; local-only files gitignored).
- **Compatibility promise:** hosted gbrain mounts `format_version: 1` repos natively;
compatibility is a test against this spec.
## Premises (all settled)
1. **Free desktop tier of the hosted ladder** — the desktop ceiling (laptop asleep =
agent asleep; data outgrows the disk) is the graduation mechanic to hosted gbrain.
2. **One agent-body format, two doors.** Portability up the ladder is moat #1.
3. **Awake-when-you-are is the honest desktop contract.** Session-triggered schedules
(jobs run at turn/session boundaries while the harness is open); true 24/7 crons
are hosted-tier, stated in-product.
4. **Day-one-empty-brain is the #1 churn risk.** Magic moment with zero corpus
(interview → next-session recall) + fast ramp (file import; connector ingest v1.1).
5. **Wire-level truth before build:** a clean-machine spike gates door 1 (write-seam
pass bar: 0 durable-write failures in 20 sessions over ≥3 days, else extend to 50).
6. **The graph is moat #2:** v1 exercises entity extraction, backlinks, and
graph-aware recall; verify enforces a graph floor.
7. **Keyless mode is first-class:** the harness agent IS the subsidized LLM — with
zero API keys, memory is agent-authored through write ops, search is keyword-only,
and the magic moment still passes. One optional key unlocks embeddings +
auto-extraction.
## Build order (one cathedral PR; size trip-wire: split at build order 2 if PR is
open >10 days from first code commit)
0. **Spike + quota gate** (manual, gates door-1 ship; per-harness quota measured; a
p90 day must fit ≤10% of weekly subscription quota or schedule scope is cut).
1. **Shared body + engine machinery:** `gbrain bootstrap` family, templates, format
spec, secret-scan-gated persistence, verify, uninstall (v1 via CEO-review
expansion; receipt-keyed scope per the PLAN's CX2-12).
2. **Codex door ships first** (runbook variant + approvals preflight + capability
probe; CLI path not spike-gated).
3. **Claude Code door:** hooks, IPC turn_context, transcript ingestion, greeting
digest, schedule mechanism.
4. **Graduation seam, desktop half:** format spec + documented upgrade path (advisor
nudge ships with the hosted mount in v1.1).
## Out of scope for v1
| Deferred | Lands | Why |
|---|---|---|
| Connector-driven ingest (email/calendar) | v1.1 (keyed to probe) | unverified host capability |
| Hosted mount + "outgrowing this laptop" nudge | v1.1 together | never point at a destination that can't accept the repo |
| `serve --attach` (simultaneous multi-harness) | fast-follow | v1 documents one-live-serve politely |
| Codex `notify` transcript sweeper | fast-follow | mandated MCP writes cover v1 |
| Windows (named-pipe IPC) | deferred | v1 = macOS + Linux |
| Networked Docker paste-flow e2e | fast-follow | offline container e2e covers 80% at 20% of the flake |
| `gbrain quota` meter command | TODOS | measurement ships as script+doc; productize when per-harness token counting is proven |
| True 24/7 crons on desktop | never | hosted-tier by design |
## Success criteria
- **TTFM ≤15 min** paste→verified install, excluding first-run toolchain downloads
(published separately); every human action counted.
- **Magic moment, deterministically:** verify asserts an interview fact is retrievable
through the agent's own MCP path; fresh-session end-to-end is a scripted human
confirmation. Must pass keyless.
- **Pilot tests:** a non-developer pilot completes the door-1 install unaided and
still uses it in week two; a developer completes the door-2 README install ≤10 min
with a week-two recall check.
- **Graph floor:** ≥1 entity extracted, ≥1 backlink resolved, one edge-only query
answered — via the real MCP write path.
- **Ladder proof:** a desktop-grown repo validates against `format_version: 1`.
- **Honesty checks:** quota number published; desktop contract stated in-product;
door 1 demotes to documented-beta on its trigger rather than shipping flaky.
## Distribution
Paste block + tag-pinned runbook (`BOOTSTRAP_FOR_AGENTS.md`, fetched at the
`latest-stable` ref — advanced by the release job only after assets publish, so
published copies never rot); optional GitHub template repo (generated at release from
the same renderer); binary via `bun install -g github:garrytan/gbrain#latest-stable`
(never npm). The paste block lives in the README's `## Install` section, as
per-harness subsections ordered "For Codex — the recommended first step" → "For
Claude Code" → "For OpenClaw or Hermes" (the 2026-08-09 ordering decision, recorded
in the PLAN's artifact table). `INSTALL_FOR_AGENTS.md` remains the paste path for
agent platforms and lives inside the OpenClaw/Hermes subsection.
## Threat model (v1 summary)
Tag-pinned fetch + version-stamp skew check + runbook phase allowlist; secret-scan
gates every commit AND corpus write (loud block, per-finding override); repo privacy
verified via API after create; hooks in gitignored local settings with a kill switch;
interview answers rendered as fenced data (never instructions) with escaping and
caps; uninstall keyed to a machine-local receipt, never deletes a brain it didn't
create; provider-policy drift acknowledged as residual risk — posture: measured
sustainable load, no absent-user background burn, portable body as the exit plan.
Full posture: `docs/guides/bootstrap.md`.
-820
View File
@@ -1,820 +0,0 @@
# Agent Bootstrap — Implementation Plan (normative)
The engineering source of truth for `gbrain bootstrap` — the paste-in install that
turns Claude Code / Codex desktop apps into a persistent personal agent (identity +
memory + skills + session-triggered schedules + private-repo persistence + local
PGLite brain). Product scope/sequencing source of truth:
[AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md). Where they disagree, the
design doc wins on scope; this doc wins on implementation.
Reviews absorbed: 3-designer panel + adversarial critique; product design review (3
rounds); CEO review (SELECTIVE EXPANSION, ~55 findings); outside voice x2 (37
findings, 35 accepted); eng review (32 findings). All accepted fixes are inlined
below with their finding IDs. 0 unresolved decisions.
---
## As-shipped deltas (read this first — where the code moved after the plan froze)
This plan is layered: later absorption sections (the post-design-review deltas, the
CX2 series) override earlier prose, and THIS section overrides everything below it.
The shipped implementation matches the plan except for these deltas:
1. **Verify runs LAST, not before host registration.** [CX2-5]'s determinism goal
survived, but the shipped phase order (single TS source:
`src/core/bootstrap/status.ts` `PHASES`) is
preflight → engine → interview → render → skills → wire → repo → **verify**,
and verify runs in-process on the caller-held engine, calling
`runMaintenanceSweep` directly — no transient serve. It works pre-registration
AND as the weekly re-run (`src/core/bootstrap/verify.ts`).
2. **Uninstall scope: [CX2-12] wins over the CEO-expansion bullet.** `~/.gbrain` is
NEVER deleted wholesale — only receipt-enumerated bootstrap-created state
(`src/core/bootstrap/uninstall.ts`).
3. **Module naming/layout:** `private-repo.ts` shipped as `repo.ts`; additional
shipped modules the artifact table doesn't list: `attach.ts, assets.ts,
format.ts, host-specs.ts, hooks.ts, lock.ts, status.ts, template-repo.ts,
uninstall.ts`.
4. **Templates layout:** all bootstrap templates live under `templates/bootstrap/`
(not at `templates/` root).
5. **Test filenames:** `test/hook-command.serial.test.ts` and
`test/e2e/bootstrap-*.serial.test.ts` — the `.serial` variants the plan's own
[A7] mandated; the artifact table predates that.
6. **README ordering:** the D5 placement was superseded by the 2026-08-09 user
decision — per-harness `## Install` sections ordered Codex → Claude Code →
OpenClaw/Hermes, with `INSTALL_FOR_AGENTS.md` living inside the OpenClaw/Hermes
section (annotated in the artifact table; the D5 prose at the bottom is stale).
## Post-design-review deltas (2026-08-07, /office-hours APPROVED — these override below)
Product: **"GBrain for Codex" + "GBrain for Claude Code"** (names contingent on
trademark review) — the free desktop tier of the hosted-gbrain ladder (the hosted service is the
graduation path; the desktop ceiling is the mechanic, not a bug). Target surfaces are
the DESKTOP APPS; **Codex door ships first** (named pilot user is a non-developer
ChatGPT-app user). CLIs come along via shared machinery.
**Build order (replaces the PR1/FF phasing below where they conflict):**
0. **Spike (gates door-1 ship; starts immediately, before build):** clean-machine
reproduction of Garry's working Codex-in-ChatGPT + gbrain prototype. Exit questions
with pass bars: (#1 blocking) reliable per-turn/per-session WRITE trigger — 0
durable-write failures in 20 sessions across ≥3 days on a non-founder machine, else
extend to 50; causes logged. (#2) per-turn READ seam — context block present at turn
start or degraded pull-mode documented. Also: folder access, MCP registration path,
approval-tap count, connector availability (yes/no/degraded). PLUS in parallel: the
**quota release gate** measured against Garry's production usage — pass: p90 day
≤10% of weekly Max/Pro quota; failure = scope change to "session-agent, no schedules
by default" BEFORE schedule build effort. Codex CLI is NOT spike-gated.
1. **Shared body + engine machinery (PR1):** everything in this plan's PR1 (bootstrap
family, gbrain hook + IPC turn_context, templates, question bank, secret scan,
sources push, skillpack scaffold wiring, doctor checks) PLUS: `agent.json` repo
manifest with `format_version: 1` (the ladder contract — the hosted gbrain service mounts this
format; compatibility is a test, not a hope); wire EXISTING import + entity
extraction + backlinks + graph-aware query into install/verify (funds design
Premises 4/6 — new verify check: ≥1 entity extracted, ≥1 backlink resolved, one
edge-only query answered); byte floors scaled to answered-question count (they catch
skipped interviews, NOT invention — never pressure padding); commit this plan to
docs/designs/AGENT_BOOTSTRAP_PLAN.md.
2. **Codex door (ships first):** runbook variant for Codex desktop approvals/sandbox
(seam per spike; fallback = mandated MCP writes via AGENTS.md gates + end-of-session
sweep, shipped only with measured reliability) + Codex CLI path (AGENTS.md pull
protocol; session-start gate includes "run due jobs"). Connector-ingest capability
probe wired into install output; connector ingest itself is v1.1.
3. **Claude Code door:** hooks per this plan (SessionStart/UserPromptSubmit/SessionEnd
via serve IPC), MCP project-scope default, transcript ingestion. Includes the v1
schedule mechanism: hook layer checks HEARTBEAT.md due-job list at session start +
turn boundaries. If the spike finds no turn-boundary trigger on ChatGPT desktop,
schedules are Claude-Code-only in v1 and per-surface copy says so.
4. **Graduation seam, desktop half only:** format spec + documented upgrade path +
honest-contract copy. The "outgrowing this laptop" advisor nudge moves to v1.1
WITH the hosted mount (never point at a destination that can't accept the repo).
**New v1 requirements from the design review (fold into implementation):**
- Binary install is tag-pinned: `bun install -g github:garrytan/gbrain#<release-tag>`,
same stamp as the runbook, so `bootstrap status` skew check is meaningful.
- Non-terminal-buyer error channel: every blocking condition (scan block, lock
collision, partial install, verify fail) surfaces through the agent's reply channel
(agent-readable status/verify/doctor output the runbook + AGENTS.md instruct the
agent to relay; hook failures write a status file surfaced at session start).
- Both doors, one machine: shared brain, one live serve; installer detects + reuses
existing registration; simultaneous-session collision fails politely, doctor names
it, agent relays it. serve --attach lifts the limit later.
- state/ boundary: interview.json + mcp.json COMMITTED (machine-2 re-render must not
hard-fail); *.local/caches/heartbeat files gitignored; DB never committed.
- OS scope v1: macOS + Linux (unix-socket IPC). Windows deferred (named-pipe = spike
question if prioritized).
- Success criteria additions: TTFM ≤15 min EXCLUDING first-run toolchain downloads
(published separately); door-2 developer test (README paste → ≤10 min install,
week-two recall check); graph floor verify check; magic moment = deterministic
retrieval assertion + scripted human confirmation (never fake-automated).
- Provider-policy drift acknowledged as unmitigable external risk; posture = quota
gate + no absent-user background burn + portable body as exit plan.
- README: bootstrap paste block is the "full agent" option INSIDE the existing "Quick
start: Claude Code or Codex" section; the 2-line memory-only quick start STAYS,
labeled "just memory, no agent"; INSTALL_FOR_AGENTS.md remains the platform headline.
- Consolidated v1 non-goals (from design doc, amended by CEO review): connector ingest
(v1.1), hosted mount + advisor nudge (v1.1), serve --attach, notify sweeper, Windows,
networked-paste Docker e2e (fast-follows); 24/7 crons never on desktop.
**CEO-review accepted expansions (2026-08-07, SELECTIVE EXPANSION — now v1 scope):**
- **GitHub template repo** ("Use this template") as a second distribution artifact for
the ChatGPT door (resolves design OQ2 = yes). Same rendered file set, published;
kept in sync with templates/ by extending scripts/check-bootstrap-templates.sh to
diff the template repo content. Build order 2.
- **`gbrain bootstrap uninstall`** in v1 (was fast-follow): removes MCP registration +
hooks + bootstrap-created state (confirm-gated), leaves the repo ("the body remains
yours"). [Scope superseded by CX2-12 + as-shipped delta 2: `~/.gbrain` is never
deleted wholesale — only receipt-enumerated bootstrap-created state.]
- **Docker cold-machine e2e (offline parts) in CI** in v1: networkless read-only
container running interview → render → verify with fake gh (codex-as-agent
tests/docker shape). The full networked paste flow stays a fast-follow (flake).
- **Hot-memory greeting digest in SessionStart**: top facts via IPC turn_context when
serve is up; file-plane MEMORY.md digest fallback when not (session start often
precedes serve spawn); fail-open. Claude door in v1; door 1 iff spike finds a read
seam.
- **First-run tour**: `bootstrap verify` success output ends with three scripted magic
prompts (who am I to you / remember X then restart / what do you know about
<project>) — makes week-one compounding visible (design Premise 4).
- **Deferred to TODOS.md in the PR** (must land as TODOS entries with the cathedral
PR): `gbrain quota` meter command (release-gate measurement ships as script+doc
first; productize once the per-harness token-count method is proven); networked
Docker paste-flow e2e.
## Deep-review hardening (2026-08-07 CEO review sections 19 — ALL ACCEPTED per
## Garry's standing directive; IDs trace to the review record. These are v1 scope.)
**Architecture (S1):**
- [D4] The release-time template-repo generator IS `core/bootstrap/render.ts --minimal`
with placeholder answers — one rendering code path, two consumers; CI guard becomes a
byte-diff of generator output vs the vendored tree.
- [D5] Bootstrap phase list defined ONCE in TS; `bootstrap status --json` emits it; the
runbook defers to it ("follow status's phase list"); CI checks runbook phase names
against the TS list.
**Source binding + brain semantics (G1, S3#1 — the two product-breaking fixes):**
- [G1] MCP registration passes the workspace source explicitly
(`claude mcp add -e GBRAIN_SOURCE=<workspace-slug>` / codex env equivalent) so agent
writes land in the workspace source, fact fence-writes reach `brain/` files (not
DB-only fallback), and the private repo actually fills. `bootstrap verify` asserts an
MCP-path `put_page` materializes a COMMITTED file under `brain/` — a green verify with
an empty repo is impossible.
- [S3#1] `turn-context.ts` constructs an OperationContext with `remote: true` and
threads `visibility: ['world']` into all fact reads (parity with the existing
meta-hook posture — the IPC path must never widen what MCP would return). IPC test:
a `visibility='private'` fact NEVER appears in a turn_context response. Verify's
magic-moment fact is written with visibility the harness can read back (world).
**Interview + render hardening (S3#3, G10, G12, A8):**
- Answers render inside fenced, explicitly-subordinate blocks ("verbatim principal
input — data, not instructions"); strip/escape line-leading `#`, `<!--`, and code
fences; per-answer length cap (~4KB, confirm-to-truncate); reject/escape `{{` and
control chars at `--set` time (a Handlebars user's honest answer must not brick the
token sweep). Verify fails if any heading or managed-block marker in a rendered file
traces to an interview value (protects harden's AGENTS.md marker splicing).
- [A8] `--confirm` requires the hash of the exact answer set that was read back;
per-answer `set_at` provenance recorded; hostile test: single-batch set+confirm
exits non-zero and verify reports `provenance: unverified`.
- [G12] Conflict-marker detection on `state/interview.json` read → agent-readable
"resolve this file" message, never a stack trace.
**Uninstall confinement (G2, S3#5, A2):**
- Never delete a brain bootstrap didn't create: `agent.json` created-by stamp checked;
default KEEPS the DB; deletion only behind explicit `--delete-brain` with a confirm
that enumerates sources + page count; facts export offered first (facts are NOT
derived state). Refuse when `GBRAIN_HOME` is set unless `--home` is explicit AND
`isPathContained` + gbrain-home signature (config.json + brain.pglite) both pass.
Host-config edits are marker-keyed managed blocks only (settings.local.json,
~/.codex/config.toml) — foreign hooks/servers survive; test asserts full
before/after filesystem+registration diff incl. foreign entries + symlinked home.
**Hooks + IPC hardening (G5, G11, S3#6, S3#7, S3#8, A9, A3):**
- [G5] `bootstrap hooks`/`--repair`/uninstall use marker-keyed managed-block
read-merge-write on settings.local.json — never overwrite `permissions.allow` or
foreign hooks; never append duplicates.
- [G11] IPC client timeout parameterized per kind (turn_context > 250ms default);
window payload + assembled block clamped below the 256KB message cap before send.
- [ENG-1] **Claude Code hook-output cap:** stdout/additionalContext is capped at
10,000 chars by the harness (overflow is diverted to a file and NOT injected) —
the assembled turn_context block is budgeted to ≤8KB (pointers + facts trimmed by
confidence to fit), asserted in the hook snapshot test. Docs: code.claude.com hooks
reference (verified 2026-08-08; the hook writers are dated spec-targets, so this cap
lives with the settings-shape module).
## Eng-review hardening (2026-08-08 /plan-eng-review — ALL ACCEPTED per standing
## directive; seams verified against source with file:line quotes)
- [ENG-2] (9/10) **cli.ts registration is THREE touchpoints, not one:** `CLI_ONLY` set
(cli.ts:58), the engine-free if-chain inside `handleCliOnly` (add before the
`connectEngine` terminator at cli.ts:1840), and `CLI_ONLY_SELF_HELP` (cli.ts:61 —
omit it and the subcommands' `--help` is dead code, the documented init.ts:117 trap).
`bootstrap`/`hook` must NOT enter `THIN_CLIENT_REFUSED_COMMANDS`. Membership test per
the #2035 precedent (test/cli-bigint-normalize.test.ts:46 shape).
- [ENG-3] (9/10) **IPC widening = handler restructure, not a field add.** Correct path
is `src/core/context/resolve-ipc.ts` (constants at :26-28). `ResolveHandler` is a
single-function type (:42) and the server handler dispatches unconditionally
(:127-128) — turn_context needs a discriminated-union request + handler map
(restructuring the closure at src/mcp/server.ts:90-108), NAMED response types
(today's responses are inline literals), and per-kind MAX_MSG_BYTES/CLIENT_TIMEOUT.
Absent `kind` defaults to 'resolve'; old-serve grace confirmed (malformed → client
IPC_UNAVAILABLE fail-soft, resolve-ipc.ts:80-84).
- [ENG-4] (8/10) **Postgres-hook silent degrade on unmigrated brains:** volunteer.ts
(:247-249) and retrieval-reflex.ts (:180-183) swallow missing-table errors
(pre-v110/v117), so a direct-engine hook returns empty, not an error. Fix: hook
heartbeat records `degraded_reason: 'schema_pre_vNNN'`; doctor pairs
hook-in-use + unmigrated-brain into a named warning.
- [ENG-5] (8/10) **Sweep layer ownership decided:** startup sweep attaches in
src/mcp/server.ts immediately after `server.connect` (:77) in the same best-effort
try/catch shape as the resolve-IPC block (:85-112), with cleanup added to shutdown
(:122); the idle sweep lives in src/commands/serve.ts reusing the `armIdle` pattern
(:437-451) through the injectable `deps.setInterval` seam (:281,297), every timer
`unref()`d (the :424/:444 convention) so the sweep can never hold the process open.
- [ENG-6] (9/10) **Compiled-binary asset rule:** `bin/gbrain` ships via
`bun build --compile`; `dirname(dirname(__dirname))` template resolution
(init.ts:1514 pattern) breaks in the binary. Templates + questions.json + runbook
stamps are STATICALLY IMPORTED (bundled) — same mechanism as the existing
`skills/_brain-filing-rules.json` static import in brain-repo-durability.ts. A
compiled-binary e2e asserts `bootstrap render` works with NO repo checkout present.
- [ENG-7] (7/10) **Host-format module precedents named:** registration strings follow
connect.ts `AGENT_SPECS` + argv builders (:61-67, :262-267); file-writing hook/
settings writers follow the integrations.ts / frontmatter-install-hook.ts idiom
(backup + marker + restore). The dated-spec-target scaffold is imported from
codex-as-agent (greenfield here — docs/plans/ has n=1 file); it becomes
`src/core/bootstrap/host-specs.ts` with TARGETS entries carrying id/status/
verifiedAt/references.
- [ENG-8] (9/10) **Facts visibility knob = ONE resolver helper.** The 'private'
default is duplicated at backstop.ts:185, :352, operations.ts:4468, :5812 — and the
:4468 ternary coerces any non-'world' to 'private', so a config default needs an
explicit caller-unset check. Implement `resolveDefaultVisibility(engine)` (reads
`facts.default_visibility` via the getConfig precedent, extract.ts:44) feeding
ctx.visibility at ALL FOUR sites; no schema change (CHECK already permits 'world',
migrate.ts:2319). Documented as security-relevant: it widens what remote/MCP
callers read via meta-hook.ts:66 — the intended effect, stated as such.
- [ENG-9] (8/10) **Secret-scan module reuses residents:** seed exclusion list from
`.gitleaks.toml` allowlist paths (test/, skills/, .claude/skills/) so the scanner
doesn't fire on fixtures CI already ignores; findings render through
`redactSecretsInText` (shell-redact.ts:36) for consistent `<REDACTED:name>` output.
- [ENG-10] (7/10) **Renderer must not eat intentional literals:** skillpack scaffolds
carry a literal `{{output-from-skill}}` token (init-scaffold.ts:269) that must
survive to disk — the renderer is never pointed at skillpack scaffold paths, pinned
by a negative test.
- [ENG-11] (7/10) **turn_context reuses the existing hot-memory cache** (30s TTL keyed
on source+session, meta-hook.ts) instead of a fresh facts query per turn — the
per-turn cost profile is then identical to what every MCP tool call already pays.
- [ENG-tests] New tests from this review: CLI_ONLY membership (bootstrap, hook);
compiled-binary render e2e (no repo checkout); Postgres-hook degraded_reason;
resolveDefaultVisibility across all 4 call sites (unset/world/private × config);
IPC handler-map dispatch (kind absent/resolve/turn_context/unknown); renderer
negative test on scaffold literals; 8KB block-budget snapshot assert.
- [ENG-tests-2] Coverage-trace gaps (Section 3): **attach-mode e2e** (clone a
bootstrap-created fixture repo on "machine 2" → attach → hooks repair → verify);
**sweep tests** (bounded batch per idle tick, spend-gate off ⇒ no LLM calls,
unref/shutdown never held open, corpus file marked processed exactly once);
**keyless-mode e2e** (zero API keys: install → agent-authored fact via ops → BM25
recall → magic-moment passes → capability report says keyless);
**decline-everything e2e** (no gh, no keys, hooks declined: install completes
local-only with honest warnings, verify exits 0-with-warnings, nothing silently
broken).
## Eng outside-voice absorption (Codex round 2, 2026-08-08 — 17/17 ACCEPTED,
## all with file:line evidence; CX2 ids)
- [CX2-1 P0] **Template-vs-attach discriminator:** agent.json presence cannot
discriminate a template clone from a machine-2 clone. The template ships
`agent.json` with `initialized: false`; `bootstrap render` flips it true atomically
AND writes a machine-local install receipt (`~/.gbrain/bootstrap/receipt.json`);
`attach` requires `initialized: true`. [pairs with CX2-12]
- [CX2-2 P0] **One-live-serve wording clarified:** each door spawns its OWN stdio
serve via its MCP registration (process-bound transport, server.ts:76); "reuse
registration" means config, never process. v1 contract restated: one live serve at
a time per brain; sequential across doors works; simultaneous fails politely.
- [CX2-3 P0] **Durability must be parent-repo-aware:** write-through targets
`repo/brain` but hardening asserts `.git` in that exact dir (sources-harden.ts:83,
brain-repo-durability.ts:621) — would fail on the workspace layout. Fix: durability
resolves the repo root via `git rev-parse --show-toplevel` (the sync.ts:1002
precedent); commit/push operate on the parent repo; integration test on the
workspace fixture.
- [CX2-4 P0] **Keyless facts ingestion made deterministic:** put_page only queues
extraction (operations.ts:1337); facts-fence reconciliation lives in the cycle
extractor (extract-facts.ts:337). The serve sweep INCLUDES the zero-LLM facts-fence
reconciliation pass, so agent-authored `## Facts` fences populate the facts table
with no API key. Keyless e2e asserts it.
- [CX2-5 P0] **Graph-floor verify made deterministic:** verify cannot command the
host's serve (stdio owned by the desktop app). Fix: the sweep gets a trusted
local-only CLI entry (`gbrain sweep --once`, CLI_ONLY, never over MCP), and
`bootstrap verify` runs BEFORE host registration on its own transient serve/engine:
write via op → `sweep --once` → edge query. No timing nondeterminism.
[Sequencing superseded by as-shipped delta 1: verify shipped as the LAST phase,
in-process on the caller-held engine; the determinism goal is unchanged.]
- [CX2-6 P1] **Cross-platform lock replaces flock dependence:** flock(1) absent ⇒
locking silently disabled (brain-repo-durability.ts:137) — macOS is the v1 target.
One cross-platform lock (atomic mkdir/lockfile with PID+age+token semantics) spans
scan → stage → commit → pull → push, coordinated with the post-commit hook.
[As-shipped delta: a TOCTOU fix reordered the scanned phase to stage FIRST, then
secret-scan the STAGED index blobs (`git cat-file`), so scanned bytes == committed
bytes; unscannable staged blobs fail closed (`blocked_unscannable`). Lock span
otherwise unchanged.]
- [CX2-7 P1] **Push ordering pinned:** commit FIRST, then divergenceSafePull, then
push (the existing durability ordering, brain-repo-durability.ts:200 —
divergenceSafePull returns skipped_dirty on a dirty tree, git-remote.ts:489); test:
dirty local + advanced remote.
- [CX2-8 P1] **GBRAIN_HOME dual semantics normalized:** config appends `.gbrain`
(config.ts:1210) while durability uses the value directly
(brain-repo-durability.ts:95) — the S3#10 `ensureGbrainHome()` choke point is also
the single semantic resolver; the --isolated e2e asserts the credential store's
ACTUAL location is gitignored, not just the expected path.
- [CX2-9 P1] **Provider-key resolution re-specified honestly:** env legitimately
overrides file config (config.ts:568) — the CX-P1.4 claim "never from shell env" is
wrong. Real contract: interview-provided keys go to the 0600 config file so
GUI-spawned serves (which lack shell env) find them; normal env>file precedence
stands; test = GUI-launch simulation with empty env + file key.
- [CX2-10 P1] **IPC authorization, not just authentication:** turn_context binds
server-side to the registered GBRAIN_SOURCE and rejects caller-supplied cross-source
requests (existing handler accepts caller sourceId, server.ts:89); cross-source
rejection test.
- [CX2-11 P1] **Hot-memory cache session key made real:** meta-hook reads an ad-hoc
`source_session` that dispatch never sets (meta-hook.ts:49, dispatch.ts:208) — all
callers collapse to the null-session cache key today. Session identity becomes
typed OperationContext state set from MCP `_meta.session_id`; two-session isolation
test.
- [CX2-12 P1] **Uninstall ownership = machine-local receipt** (never the repo-carried
agent.json, which template/attach clones inherit); uninstall stops/refuses a live
serve before touching state; `~/.gbrain` global config/sources/clones are NEVER
deleted wholesale — only receipt-enumerated bootstrap-created state.
- [CX2-13 P1] **Committed-state hygiene:** the optional interview API key bypasses
answers/hashes/provenance/logs entirely → written only to the 0600 config sink
(config.ts:1138); committed `state/mcp.json` is the PORTABLE snippet (no absolute
paths, no machine GBRAIN_HOME) — machine-specific wiring lives in local state and
is regenerated by attach/`hooks --repair`.
- [CX2-14 P1] **Generator determinism:** template-repo renders use canonical
placeholder provenance (frozen timestamps), exclude runtime state; guard test =
two independent renders are byte-identical.
- [CX2-15 P1] **Two scan policies, not one:** the `.gitleaks.toml` allowlist is a CI
fixture policy for a PUBLIC repo — importing it into the personal-repo runtime
scanner creates blind spots (skills/ trees). Runtime scanner ships its own minimal
allowlist + per-finding override; ENG-9 amended.
- [CX2-16 P1] **Bootstrap lock done properly:** atomic acquisition + PID liveness +
age guard + ownership token (the pid-reuse learning applied); kill→immediate-rerun
recovery test; export `LiveServeLockError` (currently unexported, pglite-lock.ts:28)
so the planned class assertion can import it.
- [CX2-17 P2] **Format-aware host-config writers:** JSON has no comment-marker
boundary — settings.local.json gets a structural JSON merger (gbrain entries keyed
by a `_gbrain` marker property, semantic dup detection surviving reordering);
config.toml gets a TOML-aware writer; both atomic write+backup; the
frontmatter-install-hook replace/backup idiom applies only to whole-file targets.
G5 amended.
- [S3#6] Socket binds in a 0700 dir with mode set BEFORE exposure; turn_context
requires a shared secret from a 0600 file in the data dir; heartbeat counts
turn_context serves so doctor can flag unexplained callers.
- [S3#7] Heartbeat JSONL schema pinned to counters + durations + error codes — NO
prompt/fact/slug text; dir 0700; line cap; CI test greps fixture for keys outside
the allowlist.
- [S3#8] `transcript_path` confined: `isPathContained(path, ~/.claude/projects)`,
`.jsonl` + valid envelope on line 1, lstat-reject symlinks, byte cap.
- [A9] IPC request carries `protocol: 2`; hook treats a response lacking the protocol
echo as "stale serve" and degrades LOUDLY (heartbeat entry + doctor warn); live test
against a v1-shaped server.
- [A3] Session-start digest: explicit list of digest-eligible MEMORY.md sections
(respecting the template's own security-boundary note); three-case test (socket up /
socket absent / malformed MEMORY.md).
**Persistence + sync (G4, G6, G8, G9, G13, G14/A5, G15, S3#2, S3#10):**
- [G4] `hook session-start` checks for unpushed commits / dirty tree from crashed
sessions, pushes, and names it in the greeting digest (SessionEnd is not the only
persistence path anymore).
- [G6] Verify + every push gate run `git ls-files` against a deny-glob list
(`*.pglite`, `.env*`, keys) — a truncated or pre-existing .gitignore can't leak.
- [G8] `bootstrap repo` creates a dedicated repo, OR adopts a pre-existing `origin`
when the authed gh user owns it, no `repo_url` is recorded yet, and it is empty (or
already carries our history) — the create-repo-first path; a foreign-content or
org-owned origin is refused and pointed at attach. "couldn't verify visibility" is
refuse-and-name-the-reason, never fail-open; idempotency keys off the remote URL,
not the name probe.
- [G9] Workspace lockfile (pid+timestamp) makes concurrent `bootstrap` runs impossible;
second run exits "bootstrap already running (pid N)".
- [G13] Fixed verify probe slug; sweep any prior probe before writing; excluded from
retrieval; delete failure reported as a verify warning.
- [G14/A5] The ENTIRE `sources push` (add+commit+push) runs under the existing
durability flock; single-flight test: N concurrent pushes → 1 winner, N-1 clean
"skipped, push in flight" exits, zero leftover locks.
- [G15] Retention policy everywhere: MEMORY.md size cap in the template contract with
rotation into `memory/reference/`; corpus pruned via
`dream.synthesize.corpus_retention_days` (default 30); orphaned stop-hook buffers
GC'd; doctor warns on all three.
- [S3#2] Secret scan runs at corpus-WRITE time (redact matched span + log redaction);
bootstrap adds one consent line naming the extraction/embedding provider that will
see session text.
- [S3#10] One `ensureGbrainHome()` choke point creates ~/.gbrain 0700 (all callers);
brain-push.log 0600 + rotated; `--isolated` e2e asserts `git check-ignore
.gbrain/git-credentials` passes and push never stages it.
**Transcripts (G3, A6):**
- Parser registered as a dated spec-target (same pattern as host-format writers);
heartbeat logs parsed-turns/bytes; `bytes>0 && turns==0` raises a LOUD status-file
failure ("the agent stopped learning" is never silent); scrubbed
`claude-code.jsonl` fixture (tool_use/tool_result/thinking/image/sidechain/summary/
compact-boundary shapes) added to `test/fixtures/conversation-formats/` and wired
into `check:conversation-parser` + fixture-privacy guard; session-id-keyed corpus
filenames prevent double-ingest on resume.
**Template repo (G7, S3#4, A1, C3):**
- Published ONLY from a CI release job (branch-protected, no direct pushes) behind the
placeholder assertion + secret scan + privacy scripts run against the RENDERED
artifact; fine-grained PAT secret scoped `contents:write` to the template repo only,
documented in docs/RELEASING.md; version-job completeness check verifies template
repo HEAD tree hash == vendored tree. Vendored rendered tree lives at
`templates/bootstrap/template-repo/` — the CI guard diffs generator↔vendored OFFLINE
(network comparison happens only in the release workflow). Template's first runbook
step is `bootstrap status`, which hard-fails on a public `origin` before any
identity file lands (closes the no-privacy-gate-on-this-door hole).
**Toolchain trust (S3#9):**
- Prefer platform package managers for bun/gh; curl fallback downloads to a file,
verifies against the pinned bun release's SHASUMS256.txt, then executes; the
runbook's phase allowlist names the verified form as the only permitted variant.
**Observability (B1B5):**
- [B1] Append-only `~/.gbrain/bootstrap/install.jsonl` ({ts, phase, outcome,
duration_ms, binary_version, harness, workspace}) written by every subcommand;
`status` renders the tail.
- [B2] Every verify run persisted to `~/.gbrain/bootstrap/verify-<ts>.json` (keep 5);
doctor reports last verify timestamp/status/deltas.
- [B3] Hook heartbeat records outcome + reason on EVERY invocation; trailing-20
failure rate over threshold → one visible line inside the injected context block
("brain context unavailable for the last N turns — run `gbrain doctor`").
- [B4] `~/.gbrain/bootstrap/push-status.json`; doctor fails when last successful push
>48h old with a dirty tree; surfaced in the SessionStart digest.
- [B5] `bootstrap status --json` emits a support blob (workspace, binary version,
harness, engine, last verify, last push, hook failure rate); AGENTS.md instructs
the agent to relay it verbatim on any "something's broken" report.
**Tests (A1A9 not already covered above):**
- [A4] First-run tour prompts pinned in the questions.json↔template bijection guard +
verify success-output snapshot test.
- [A7] Flake pinning: hook deadline asserted against an injected slow-IPC stub (real
latency kept as non-gating benchmark); kill-mid-phase via deterministic
`GBRAIN_BOOTSTRAP_ABORT_AFTER=<phase>` injection; lock-contention asserts on
LiveServeLockError class; all subprocess tests named `*.serial.test.ts` with
explicit --timeout; offline Docker e2e runs from heavy-tests.yml (nightly + label),
not the PR shard matrix.
**Release mechanics (C1 RESOLVED + C2, C4, C5, C6, C8):**
- [C1 = D6-A, decided by Garry] **Distribution ref: single `latest-stable` ref.** The
release job force-updates `latest-stable` to the just-verified release commit as its
FINAL step (after binary assets publish + provenance attestation). README paste
block, runbook URL, and `bun install -g github:garrytan/gbrain#latest-stable` all
reference it permanently — no per-release tag pins, no 404 window, no per-ship
README edits. The fetched runbook embeds the concrete VERSION it was cut from;
`bootstrap status` compares that stamp to the installed binary (skew check intact).
New guard `scripts/check-bootstrap-tag.sh`: README/runbook reference ONLY the
sanctioned ref AND the runbook's embedded stamp equals VERSION. Modeled on
test/release-workflow.test.ts.
- [C2] README.md + BOOTSTRAP_FOR_AGENTS.md re-admitted to `scripts/ci-cache-hash.sh`
ALLOW_PATTERNS (a README-only paste-block change must never skip CI green).
- [C4/C5] `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + a SCRUBBED
`docs/designs/AGENT_BOOTSTRAP_DESIGN.md` (banned names → capability-class phrasing,
founder quotes/pilot identifiers/pricing-funnel strategy removed) land in the SAME
commit; normativity paragraphs point in-repo; `bash scripts/check-privacy.sh` run
against the staged index before that commit.
- [C6] CHANGELOG states capabilities functionally ("installs hooks that run on each
prompt; installs an opt-in background push job") linking to docs/guides/bootstrap.md
which owns the full security/consent posture together with the rendered
ACCESS_POLICY.md; zero plan IDs / review-round references.
- [C7] Public product names ("GBrain for Codex"/"GBrain for Claude Code") are a
pre-merge checklist item owned by Garry (trademark review); all committed copy uses
the neutral `gbrain bootstrap` verb until sign-off.
- [C8] Zero-migration release confirmed (facts/context_volunteer_events already
exist); ALL new bootstrap telemetry stays on the filesystem — the moment it moves
into a table it becomes migration v126 + bootstrap-coverage + engine-parity
obligations.
- [G16] User-scope consent question names the project-scope tradeoff explicitly;
CLAUDE.md renders a one-line "this agent lives in <dir>" note.
## Outside-voice absorption (Codex, 2026-08-07 — dispositions under Garry's
## auto-accept directive; CX ids trace to the codex output)
**ACCEPTED — product-critical (the two the whole review chain missed):**
- [CX-P0.5 **Keyless mode is a first-class design requirement.**] The named pilot user
has ChatGPT Pro but NO API keys — and gbrain's embeddings + extract_facts (Haiku)
are API-metered. Bootstrap MUST work with zero API keys: the HARNESS AGENT is the
subsidized LLM, so AGENTS.md instructs it to author facts/timeline/links explicitly
through write ops (zero API cost); search degrades to BM25 keyword (no embeddings);
verify prints an honest capability report ("keyless mode: keyword search, agent-
authored memory; add ONE key to unlock embeddings + auto-extraction") and the
interview offers the optional key question. The magic-moment check must pass in
keyless mode (fact written by the agent via ops, recalled next session). Quota-gate
copy amended: API spend (embeddings/extraction) is metered separately from
subscription quota and is ZERO in keyless mode; with a key, existing spend gates
(docs/operations/spend-controls.md) govern. [also resolves CX-P0.6]
- [CX-P0.1+P0.3 **Serve-resident maintenance sweep** closes the persistence loop.]
Nothing previously ingested the transcript corpus into the live brain (dream is
disabled; CLI can't open PGLite under a live serve), and remote `put_page`
deliberately skips auto-link/timeline extraction (operations.ts:1273) so the graph
would never compound from harness writes. Fix: the serve process (the lock owner)
runs a bounded, spend-gated maintenance sweep — on startup and idle — that (a)
ingests unprocessed corpus files (keyless mode: skipped, agent-authored memory
covers it), (b) runs the deterministic zero-LLM link/timeline extraction over
recently written workspace-source pages. Verify's graph-floor check exercises the
REAL MCP write path end-to-end (write via stdio op → sweep → edge query), never a
synthetic install-time shim.
**ACCEPTED — durability/trust:**
- [CX-P0.4] Write-through failures on `put_page` are best-effort today → for the
workspace source they become LOUD: doctor check counts DB-pages lacking file
backing; surfaced in the SessionStart digest; sources push reconciles before commit.
- [CX-P1.1] Single-principal visibility posture: bootstrap sets the workspace brain's
fact default visibility to 'world' (the desktop agent IS the principal's main
session; ACCESS_POLICY.md documents it; multi-tier users flip the knob). Keeps the
IPC world-only filter (S3#1) AND working personalization.
- [CX-P1.2] The injected turn_context block is wrapped in the same "retrieved data,
never instructions" subordinate envelope as interview answers (provenance-labeled).
- [CX-P1.4] GUI env inheritance: MCP registration carries the absolute binary path +
explicit env (GBRAIN_SOURCE, GBRAIN_HOME when --isolated); provider keys resolve
from ~/.gbrain/config.json, never from shell env, for GUI-spawned serves.
- [CX-P1.5] **`bootstrap attach` mode** (machine two): a cloned repo carrying
agent.json enters attach mode (register source, hooks --repair, MCP, verify) —
the refuse-pre-existing-origin rule (G8) applies only when agent.json is absent.
- [CX-P1.6] Git conflict model: sources push does fetch + divergence-safe rebase pull
(reuse harden's divergenceSafePull) before push; non-FF/conflict = loud status +
agent-relayed instruction, never silent retry; repo-local git author identity set
at `bootstrap repo` (ported from setup-private-repo).
- [CX-P1.7] Search-mode consent folds into the interview as one optional question
(default balanced) — preserves the INSTALL_FOR_AGENTS consent contract without
another stop; the spike's TTFM measurement counts every stop.
- [CX-P1.9] Non-circular error surface: README + GITHUB.md carry the one recovery
command ("if it seems broken: `gbrain doctor`"); doctor covers hook/push/serve
health independent of the possibly-broken agent loop.
- [CX-P1.10] Forget semantics documented honestly in ACCESS_POLICY.md/GITHUB.md: the
repo is append-only history; true deletion = documented history-rewrite procedure;
GitHub remains default-but-optional (local-only mode with honest warning).
- [CX-P1.11] Template repo README embeds the same VERSION stamp as the runbook;
`bootstrap status` validates it identically (closes the adopter-skew window).
- [CX-P1.12] agent.json spec labeled **provisional-v1**: desktop-side validation
only, consumers tolerate unknown fields, hosted mount may bump to v2 with a
migration note — a version contract, not a frozen promise.
- [CX-P0.7+P1.13] Spike/pilot instrument upgraded from binary pass/fail to product
metrics: per-turn write-attempt vs durable-write precision, correct-recall rate,
false-memory incidents, correction round-trips — measured through the 2-week pilot
(the pilot IS the extended sample); the 0-failures-in-20 bar remains only the
minimum to START the pilot.
**REJECTED (with reasons, recorded):**
- [CX-P0.2] "Two desktop apps cannot share the brain" — TRUE for simultaneous
sessions and already the documented v1 limit (polite collision + doctor + attach
proxy fast-follow); sequential use works. No change beyond what's accepted.
- [CX-P1.8] "Networked paste flow untested" — known; manual clean-machine acceptance
in v1 + networked Docker e2e deliberately deferred (D3.3b). Stands.
**CROSS-MODEL TENSION (recorded, not re-litigated):** Codex's "fundamental
simplification" (Codex-only, 3-question interview, no hooks/template/GitHub, 2-week
manual pilot before building) is a REDUCTION-mode argument against the settled
cathedral decision (D1-A) and accepted Approach B scope. Disposition: rejected as
sequencing (Garry chose the cathedral 30 minutes prior, with the two-wave case
recorded for revisit + a size trip-wire), but its measurement substance was absorbed
via CX-P0.7/P1.13, and the spike + pilot ARE the "prove the loop" instrument, run
before the doors ship rather than instead of building them.
## Design (synthesized)
Synthesis of three independent designs (DX lens, runtime-parity lens, architecture lens) +
an adversarial critique that verified the load-bearing claims against both repos. Where the
designs disagreed, the critique's evidence-checked winner is taken.
### The experience (end state)
One paste block (README + tweet) → the agent fetches `BOOTSTRAP_FOR_AGENTS.md` (raw GitHub
URL **pinned to a release tag**, same mechanism as INSTALL_FOR_AGENTS.md) and drives:
preflight → interview (chat, 12Q/6-required, hard gate) → render identity files → skills →
MCP + hooks wiring → sources add/sync/embed → private GitHub repo (created, privacy-verified,
pushed) → `gbrain bootstrap verify` (exit-code contract) → completion manifest.
Human actions: paste, answer interview in chat, `gh auth login` if needed, ~2 consent
questions. Target ≤15 min. Definition of done includes the magic moment: after restart, the
agent recalls a fact the human said during the interview.
Second session: SessionStart hook injects identity digest + "since last time"; every prompt
gets Live Context + ≤3 volunteered brain pointers + hot memory injected via hook; the agent
writes facts back same-turn through MCP ops; SessionEnd ingests the transcript into the dream
corpus and fires a scan-gated commit+push. "Your local harness IS your agent."
### Decisions (settled by panel + critique)
- **D1 Topology: single private repo.** Workspace root = identity files (SOUL.md, USER.md,
MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md, GITHUB.md, ACCESS_POLICY.md) + `skills/` +
`memory/` (daily notes + README) + `state/` + **`brain/`** (people/ companies/ meetings/
concepts/ daily/). **Only `brain/` is registered as the gbrain source** (critique: indexing
the whole repo puts the contract files into the retrieval corpus — noise + self-referential
injection surface). `.gbrain-source` dotfile at root routes CLI calls. PGLite DB is NEVER
in the repo (`.gitignore` renders first: `*.pglite`, `.env*`, `state/*.local`, corpus paths,
keys/PEM). Two-repo model stays documented as the graduation path in GITHUB.md/docs.
- **D2 Workspace = the cwd the user pasted in.** Guard: if cwd is an existing code project
(tracked files/remote), ask ONE relocation question. DB default: global `~/.gbrain` host
brain (workspace is a source); `--isolated` escape hatch per resolved D2.
- **D3 Delivery: hybrid.** New `gbrain bootstrap` command family (deterministic, idempotent,
exit-coded) + fetched runbook (judgment: interview wording, read-backs, consent). Bootstrap
is **CLI-only like init/connect — NOT an operation**; zero new `ctx.remote` surface (a
remote-triggerable `gh repo create` + filesystem render is forbidden by the trust model).
Subcommands: `status` (resume entrypoint; also verifies runbook version stamp vs binary —
supply-chain skew check), `interview --init|--set K V|--skip K|--status|--confirm|--show`,
`render [--force] [--only F] [--minimal]`, `repo`, `hooks --harness claude-code|codex
[--repair]`, `verify [--json]`.
- **D4 Interview: one question bank, two entry points.** Port codex-as-agent's
`questions.json` (scrubbed) to `templates/bootstrap/questions.json`: 12 asked / 6 required
(AGENT_NAME, PRINCIPAL_NAME, AGENT_PURPOSE, AGENT_TOP_JOBS, PRINCIPAL_CONTEXT,
VOICE_REGISTER — the wince question and vibe samples from soul-audit Phase 2 fold in as
optional keys). Answers → `<ws>/state/interview.json` (committed; identity source of truth,
same sensitivity as the rendered USER.md; makes re-render-on-new-machine work).
`--status` exits non-zero until required present; `render` additionally refuses until
`--confirm` (set only after the read-back "Is this the thing you want in the room?").
Hard rules verbatim in runbook: NEVER INVENT ANSWERS, accept "skip", quote literal words.
`skills/soul-audit/SKILL.md` becomes the re-run/deepen surface over the same bank
(`interview --set` + `render --only SOUL.md`) — one bank, CI-guarded against drift.
- **D5 Per-turn context (critique-decided; the PGLite lock is the constraint).**
`gbrain serve` holds the PGLite single-writer lock for its lifetime (#2348: live holders
are never stolen) — so hooks must NEVER open the engine on PGLite. New engine-free
`gbrain hook <event>` command (no-engine dispatch branch in cli.ts) talks ONLY to serve's
existing resolve-IPC unix socket (`src/core/context/resolve-ipc.ts`), widened with
back-compat `kind: 'resolve' | 'turn_context'`. `turn_context` request carries
{window, priorContextText, sourceId}; serve assembles server-side (new
`src/core/context/turn-context.ts`): reflex pointers + volunteerContext pages (≤3) +
hot-memory facts (the same content `_meta.brain_hot_memory` carries — this ROUTES AROUND
the _meta invisibility rather than fixing the harness). Postgres fallback: hook opens the
engine directly (multi-connection safe) since the IPC socket only exists on PGLite.
- Claude Code hooks (written to **`.claude/settings.local.json`** — gitignored; committed
hooks with absolute paths are a portability trap + persistence-of-execution surface;
`bootstrap hooks --repair` re-renders on a new machine):
- SessionStart → `gbrain hook session-start`: zero-DB file reads (date/tz, MEMORY.md
open-commitments, ops/tasks.md if present) + best-effort socket warm probe. ≤1.5s.
- UserPromptSubmit → `gbrain hook user-prompt`: stdin hook JSON → tail last 4 turns from
transcript_path → socket turn_context → stdout additionalContext block.
**800ms hard self-deadline**, fail-open (exit 0, empty stdout), heartbeat JSONL at
`~/.gbrain/integrations/hooks/` for doctor.
- Stop → `gbrain hook stop`: append turn to live per-session buffer (pure file append).
- SessionEnd → `gbrain hook session-end`: parse full transcript .jsonl → corpus .txt in
`dream.synthesize.session_corpus_dir` (default `~/.gbrain/transcripts/corpus/`, 0700,
never in the repo) + fire scan-gated commit+push (D6). Closes the transcript gap in PR1.
- Absolute binary path everywhere; `GBRAIN_HOOKS=0` kill switch.
- Codex (no hooks): honest pull model. Rendered AGENTS.md carries adapted per-message
gates 07 (entity lookup = "call recall/volunteer_context with recent window";
receipts; WRITE IT DOWN same turn via extract_facts/put_page) +
`codex mcp add gbrain -- gbrain serve`. FF2: `notify` hook in ~/.codex/config.toml as
transcript sweeper (validate the event semantics first). Runbook/verify state the
degradation plainly: Claude Code = push-on-hook, Codex = pull-on-protocol.
- **Write-path rule rendered into AGENTS.md** (critique hole 2): on PGLite, durable
knowledge is written through MCP ops (put_page/extract_facts/add_timeline_entry), never
by editing brain/ files directly — file edits are invisible to retrieval until a sync
can run, and sync can't run while serve holds the lock. MEMORY.md/memory/ file edits are
fine (file-plane, loaded by path not retrieval).
- **D6 Private repo + persistence.** `gbrain bootstrap repo` = TS port of
setup-private-repo.mjs: gh-auth exit-2 gate (only human step: `gh auth login`), slugified
`<agent-name>-workspace` collision probe, `gh repo create --private --source . --push`,
**privacy verified via `gh api ... --jq .private` (hard fail)**. Sync: extract secret scan
to `src/core/secret-scan.ts` (sk-/gh[pousr]_/github_pat_/xox[baprs]-/PEM; blocks commit);
new `gbrain sources push [<id>|--path]` = scan-gated add+commit+push, refuses public
remotes, pushes even on clean tree. `hardenBrainRepo` gains the scan as a step (its
existing post-commit hook + cron machinery is reused, NOT a parallel sync system).
Cadence per resolved D3: 15-min cron installed after an explicit consent question;
SessionEnd-hook push always on as the no-daemon backstop/fallback. GITHUB.md persistence
contract rendered.
- **D7 Rendered files.** AGENTS.md (adapted gates + hard gates: WRITE IT DOWN, NO SILENT
FAILURE, VERIFY BEFORE CLAIMING DONE, RED LINES, PRIVATE REPO PERSISTENCE + brain-first
protocol from docs/tutorials/connect-coding-agent.md + brain filing contract rendered from
skills/_brain-filing-rules.md), CLAUDE.md (thin: @AGENTS.md @SOUL.md @USER.md @MEMORY.md +
hooks note), SOUL.md (codex-as-agent section skeleton: Identity/Mission/Worldview/The
Standard/Honesty/Voice+wince/Good vs bad output/High agency/Never — IDENTITY.md merged in),
USER.md ("their literal words are ground truth"), **MEMORY.md (new template**: hot state,
corrections format `- YYYY-MM-DD — rule (Bug: ...)`, open commitments, security-boundary
note), ACCESS_POLICY.md, HEARTBEAT.md (quiet hours, verify-time-first, silence contract,
jobs disabled), GITHUB.md, memory/README.md, brain/ dirs + READMEs, .gitignore,
state/interview.json, state/mcp.json (portable snippet). Skills via existing
`gbrain skillpack scaffold --all` (resolved D4) + inert-skill report (skillpack check
wired into verify AND into the completion manifest's next-steps block with the exact API
keys to add). ALL templates REWRITTEN generic (privacy IRON RULE —
never copy codex-as-agent prose verbatim; CI placeholder assertion). TOOLS.md deferred
(essential lines fold into AGENTS.md). `installDefaultTemplates` (init.ts:1513) finally
gets its caller via `render --minimal`.
- **D8 Verify (union).** (1) doctor green; (2) DB round-trip: put_page → get → query-with-
score → DELETE probe page; (3) MCP registered (`claude|codex mcp list`) + probeBrainIdentity
smoke; (4) {{TOKEN}} sweep hard-fail; (5) byte floors (SOUL.md ≥3000B, USER.md ≥1000B);
(6) secret scan clean; (7) repo private via API; (8) **hooks smoke UNDER LIVE SERVE**
(spawn real serve, pipe fixture UserPromptSubmit stdin, assert non-empty block + <800ms +
never acquires the lock — bootstrap-time-only smoke is a false green); (9) one manual
sources push succeeded; (10) inert-skill report; (11) transcript parser dry-run on fixture.
Prints ranked completion manifest. Re-runnable weekly as the workspace rot self-check.
- **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
daemon). OPT-IN: 15-min harden cron. Autopilot NOT default on PGLite (verified: its
sync/embed children would contend with every live serve for the single-writer lock, and
nothing handles LiveServeLockError politely today) — recommended on Postgres; any future
scheduled job must treat lock-held as skip-silently-and-log. LLM crons (briefing, dream via
`claude -p`/`codex exec`) ship rendered-but-disabled with the enable-one-at-a-time ritual.
- **D10 Phasing.**
- **PR1 (complete usable experience):** BOOTSTRAP_FOR_AGENTS.md + README paste block;
`gbrain bootstrap` family; `gbrain hook` + IPC v2 turn_context; new templates + question
bank; secret-scan + `sources push` + harden integration; SessionEnd transcript ingest
(Claude Code .jsonl parser); skillpack scaffold wiring; soul-audit re-run update; doctor
checks (hooks heartbeat, dual-serve report, sync check); docs; unit + e2e (incl.
hook-under-serve + lock-contention pins); CI guards.
- **FF2:** Codex notify sweeper + ~/.codex/sessions parser. **FF3:** `gbrain serve
--attach` stdio proxy (two harnesses fully concurrent on one PGLite brain). **FF4:**
cron fleet + cron-doctor port + heartbeat activation. **FF5:** Docker fresh-machine e2e
in CI, _meta surfacing experiments, per-turn-context BrainBench eval, upgrade re-render
nudge via runPostUpgrade (`bootstrap render --diff`).
### Security/trust invariants (critique holes, addressed)
1. Paste block pinned to release tag; runbook version-stamped; `bootstrap status` compares
stamp vs binary and warns; runbook instructs refusing steps outside its phase list.
2. Write-through-ops rule in AGENTS.md (above).
3. MCP scope: project scope default, user-scope opt-in (resolved D1); threat named in
ACCESS_POLICY.md either way.
4. Codex sandbox reality: runbook carries a Codex-specific preflight (approval mode /
workspace-write + network consent) — the paste block warns the human they'll be asked.
5. Hooks in settings.local.json (gitignored) + --repair.
6. Upgrade story: FF5 re-render nudge; render never clobbers (backup on --force).
7. Transcript corpus + answers privacy: corpus 0700 outside repo; interview.json committed
(same sensitivity as rendered USER.md, which is committed); responsible-disclosure
phrasing in CHANGELOG (functional, no attack-surface enumeration).
8. Process invariants: CLAUDE.md dispatcher row edit → `bun run build:llms` same commit;
KEY_FILES.md entries current-state prose; version-first PR title; ship via /ship;
/document-release after.
### New/changed artifacts (paths)
| Path | New/changed |
|---|---|
| `BOOTSTRAP_FOR_AGENTS.md` | NEW root runbook (fetched by paste block) |
| `README.md` | dedicated "For Codex" / "For Claude Code" paste-block sections, ordered Codex → Claude Code → OpenClaw/Hermes at equal weight (user decision 2026-08-09, supersedes D5's ordering; both platform paths preserved) |
| `src/commands/bootstrap.ts` (+ `src/commands/bootstrap/*.ts`) | NEW dispatcher + subcommands |
| `src/commands/hook.ts` | NEW `gbrain hook session-start|user-prompt|stop|session-end` |
| `src/cli.ts` | CHANGED: `bootstrap`+`hook` in no-engine dispatch branch |
| `src/core/bootstrap/{interview,render,private-repo,verify}.ts` | NEW (TS ports) |
| `src/core/secret-scan.ts` | NEW (shared: sources push, harden, verify) |
| `src/commands/sources.ts` + `src/core/brain-repo-durability.ts` | CHANGED: `sources push`, scan-gated hook/cron |
| `src/core/context/resolve-ipc.ts` + `src/mcp/server.ts` | CHANGED: IPC v2 `turn_context` (back-compat) |
| `src/core/context/turn-context.ts` | NEW server-side block assembly |
| `src/core/transcripts/claude-code-jsonl.ts` | NEW parser + corpus writer |
| `templates/{MEMORY,AGENTS,CLAUDE,GITHUB,memory-README}.md.template` + `gitignore.template` + enriched SOUL/USER/HEARTBEAT/ACCESS_POLICY | NEW/CHANGED (generic, scrubbed) |
| `templates/bootstrap/questions.json` | NEW shared question bank |
| `skills/soul-audit/SKILL.md` | CHANGED: re-run surface over the bank |
| `src/commands/doctor.ts` | CHANGED: hooks heartbeat, dual-serve, sync checks |
| `docs/guides/bootstrap.md` + docs/mcp/ + connect-coding-agent.md cross-links | NEW/CHANGED |
| `scripts/check-bootstrap-templates.sh` | NEW CI guard (token↔bank bijection + placeholder-only assertion) |
| `docs/architecture/KEY_FILES.md`, `CLAUDE.md` (+build:llms) | CHANGED |
| `test/bootstrap-*.test.ts`, `test/hook-command.test.ts`, `test/secret-scan.test.ts`, `test/e2e/bootstrap-lifecycle.test.ts`, IPC back-compat tests | NEW |
### Port map (codex-as-agent → gbrain)
interview.mjs → core/bootstrap/interview.ts · render-templates.mjs → core/bootstrap/render.ts
· setup-private-repo.mjs → core/bootstrap/private-repo.ts · verify-install.mjs (+
install-gbrain.mjs round-trip) → core/bootstrap/verify.ts · git-sync.mjs → secret-scan.ts +
sources push · questions.json → templates/bootstrap/questions.json (scrubbed) · AGENTS.md
gates 07 / SOUL/USER/MEMORY/HEARTBEAT/GITHUB templates → REFERENCE structure, REWRITE
content (privacy rule) · install-skills.mjs → REFERENCE (skillpack scaffold exists) ·
cron fleet/cron-doctor → FF4 · codex-plugin-spec one-file-owns-format pattern → governs the
settings.local.json + config.toml writers (single module owns each host format).
### Verification (how we know it works end-to-end)
- Unit: interview gate exit codes; render token hard-fail/no-clobber/backup; secret-scan
fixture corpus (positives + benign lookalikes); hook stdin→JSON contract; IPC v1↔v2
back-compat both directions; questions.json↔template token bijection.
- E2E: full lifecycle in temp dir (sandboxed GBRAIN_HOME, PATH-shimmed fake gh/claude/codex
recording invocations): render → repo → verify exit 0; idempotency (second run no-op);
kill-mid-phase → `status` resumes. **hook-under-serve** and **lock-contention** pins
(permanent). Engine parity for turn-context on both engines.
- Manual acceptance at ship: fresh macOS account, real paste, both harnesses, timed
(≤15 min, ≤3 human actions).
### Resolved user decisions (Garry, 2026-08-07)
- **D1 = A. MCP scope: project scope default, user-scope opt-in** (consent question during
bootstrap; threat named in ACCESS_POLICY.md).
- **D2 = C. DB: global `~/.gbrain` default + documented `--isolated` escape hatch.**
`gbrain bootstrap --isolated` threads `GBRAIN_HOME=<workspace>/.gbrain` through init, MCP
registration env (`claude mcp add -e` / codex config env), and the hook commands in
settings.local.json. Port install-gbrain.mjs's guards: GBRAIN_HOME does NOT isolate
`sync.repo_path` (set it explicitly), strip ambient GBRAIN_DATABASE_URL/DATABASE_URL/
GBRAIN_BRAIN_ID, assert database_path is inside the workspace. `.gbrain/` already
gitignored by the rendered template.
- **D3 = B with consent. 15-min scan-gated commit+push cron is the default posture, but the
runbook ASKS PERMISSION before installing it** ("Enable background persistence? Installs a
15-min launchd/cron job that commits + pushes this workspace, secret-scan-gated"). The
SessionEnd-hook push stays as the always-available, no-daemon backstop and is the fallback
when the cron is declined — persistence never silently disappears. Autopilot posture
unchanged: off on PGLite, recommended on Postgres.
- **D4 = B. ALL bundled skills scaffold in** (`skillpack scaffold --all`). The onboarding
ENDS with a next-steps block in the completion manifest: the inert-skill report (which
installed skills are dormant for lack of which API key), the exact keys to add and where,
and pointers to soul-audit (deepen identity this week) + cold-start (fill the brain with
your data). The routing-table-size concern is mitigated by frontmatter-trigger routing
(authoritative since v0.36) and noted for a future curated-profile fast-follow if dispatch
accuracy suffers in practice.
- **D5 = Codex/Claude-Code-scoped placement.** [Superseded by the 2026-08-09 user
decision — see as-shipped delta 6 and the artifact table's README row.] This is NOT the new headline install —
most users still use GBrain with OpenClaw/Hermes, so `INSTALL_FOR_AGENTS.md` remains the
primary paste path at the top of the README. The bootstrap paste block becomes the
flagship "For Codex" / "For Claude Code" README sections, ahead of the OpenClaw/Hermes path at equal weight (and
docs/tutorials/connect-coding-agent.md cross-links it). Command name stays
`gbrain bootstrap`; paste block is pinned to a release tag (supply-chain integrity).
BOOTSTRAP_FOR_AGENTS.md opens with a scope note: "For Claude Code / Codex. Running
OpenClaw or Hermes? Use INSTALL_FOR_AGENTS.md instead."
-63
View File
@@ -1,63 +0,0 @@
# Agent Bootstrap — Spike Instrument (build order 0)
Manual validation that gates door-1 (Codex/ChatGPT desktop) ship. Run on a machine
the maintainer does not own, with fresh accounts. Owner: the maintainer. Timebox:
~1 week wall-clock. Outcomes feed the design doc's gate
([AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md)).
## Exit questions and pass bars
**#1 (blocking) — the write seam.** What reliably persists memory per turn/session
on the ChatGPT-desktop surface?
- Protocol: run 20 sessions across ≥3 days of ordinary use. Each session must
produce at least one durable write (a page/fact retrievable in the NEXT session).
- Pass: 0 durable-write failures in 20 sessions; else extend to 50 and log every
failure's cause (crash, sleep, approval friction, format drift, model forgot).
- Below the bar → door 1 demotes to documented beta (Codex CLI unaffected).
**#2 — the read seam.** Is injected/pulled context demonstrably present at turn
start? Pass: context block present (or the degraded pull-mode documented as
door-1's v1 behavior). Pass expands the greeting digest to door 1.
**#3 — capability surface.** Record, with screenshots: local folder access
(yes/no/how), MCP registration path (config file? `codex mcp add`? UI?), approval
taps for each toolchain step (count them), connector availability for
email/calendar (yes/no/degraded).
**#4 — quota.** Per harness the doors run on (Claude Code: Max plan; Codex: the
ChatGPT plan): log each day's usage-meter readings during the pilot.
- Load model: ordinary sessions + hooks + one session-triggered schedule.
- Pass: a p90 day consumes ≤10% of the weekly allowance (per-door; one harness
failing cuts schedule scope for that door only).
- Measurement: the harness's own usage UI (screenshot at day start/end) + a tally
of sessions/turns from the transcript dir. No telemetry — this is a manual
instrument by design.
**#5 — TTFM baseline.** One full paste-to-verified install, timed. Count every
human action (paste / auth click / interview answers / consents / approval taps).
Toolchain download time recorded separately (excluded from the 15-minute target).
## Pilot metrics (continue for 2 weeks after the spike)
Per week, from session review (screen recordings + self-report — no telemetry):
correct-write rate (things worth remembering that got written), correct-recall
rate (recalls that were right), false-memory incidents (recalled things that were
wrong), correction round-trips (corrections that stuck as standing rules). The
0-failures-in-20 bar is the minimum to START the pilot, not the proof — the pilot
is the sample.
## Log template (one row per session)
| # | date | door | duration | writes attempted | writes durable | recalls right/wrong | approvals | notes |
|---|---|---|---|---|---|---|---|---|
## Deliverable
A filled copy of this doc committed as `AGENT_BOOTSTRAP_SPIKE_RESULTS.md`
(scrubbed: no real names beyond the maintainer, no account identifiers), plus the
gate decision recorded in the design doc: door-1 ships full / ships as documented
beta / schedule scope cut per quota.
**Gate status:** not yet run — no `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` is committed,
so no gate decision is recorded and door 1 has not been promoted past the
documented-beta bar by this instrument. Update this line when the results land.
-406
View File
@@ -1,406 +0,0 @@
# Brain currency — fix the incident, then build the ladder
Generated by /plan-ceo-review on 2026-08-10
Rev 3, after two adversarial spec-review rounds (6/10 → 7/10) and an independent outside voice.
Branch: garrytan/gbrain-commit-indexing | Mode: SELECTIVE EXPANSION
Repo: garrytan/gbrain
**Citation convention:** repo-relative paths. `src/core/sync.ts` (540 lines) and
`src/commands/sync.ts` (5804 lines) are different files; both are cited.
## Origin
An investigation into "how does gbrain pick up new commits from GitHub" found it never
talks to GitHub. It diffs `git diff last_commit..HEAD` against a **local checkout**
(`src/core/sync-delta.ts:113`). Getting remote commits into that checkout is a separate,
opt-in concern.
It then found worse: on the founder's machine `gbrain autopilot` was installed, died
2026-05-31, and stayed dead **71 days** while three surfaces reported healthy.
**1. `autopilot --status` is an artifact-presence check.**
`src/commands/autopilot.ts:1775-1786` — plist `existsSync` on darwin, crontab grep
elsewhere. Never asks whether the job is loaded, the process alive, the baked `--repo`
present, or the log fresh. Always exits 0.
**2. `doctor`'s `sync_freshness` computes the 71-day number and throws it away.**
`src/core/source-health.ts:182-194`:
```ts
const wallClockSeconds = Math.floor((nowMs - lastSyncMs) / 1000); // ← the 71 days
if (wallClockSeconds < 0) return wallClockSeconds;
if (contentMs !== null && Number.isFinite(contentMs)) {
return contentMs <= lastSyncMs ? 0 : wallClockSeconds; // ← discarded
}
```
When the clone is unreachable, `src/commands/doctor.ts:4306-4344` routes the verdict here.
The function measures *drain completeness*, not *staleness*. "We caught up when we last
looked" and "we have not looked in 71 days" both return 0.
**3. `gbrain status` inherits it.** `src/commands/sync.ts:5440-5453``'fresh'` beside a
71-day-old date, exit 0. (`gbrain sources status` does report the real lag in its LAG
column, but has no warn line for it and no exit contract.)
**Root cause of the death:** `src/commands/migrate-engine.ts` (22,733 bytes) contains
**zero** autopilot references. The Supabase-to-PGLite migration rewrote
`~/.gbrain/config.json` while a daemon built on the old config kept running and died on
`config.database_url`.
## The key insight the reviews converged on
The content comparison in #2 is not a bug someone forgot. `src/commands/doctor.ts:4288-4305`
documents why it exists:
> a container restart wipes `local_path` ... **and since a no-op sync doesn't advance
> `last_sync_at`**, every QUIET source read as stale/FAIL after a restart (score-sinking
> alert storm; observed live: 16-source brain, 12 clones gone after a config-update
> restart, doctor 70→30).
**The premise in bold was invalidated after that code was written.** v0.42.52.0 added a
heartbeat at `src/commands/sync.ts:2287-2298`:
```ts
// bump last_sync_at as a heartbeat on every successful 0-changes sync...
if (opts.sourceId) {
await engine.executeRaw(`UPDATE sources SET last_sync_at = now() WHERE id = $1`, [opts.sourceId]);
}
```
A no-op sync **does** advance `last_sync_at` now. So a quiet source that is being checked
has a recent `last_sync_at` and survives a wall-clock ceiling; the 71-day case has an old
one because **no sync ran at all**. The two cases are now distinguishable, and the
fallback's justification has expired.
That is the whole incident: a wall-clock ceiling on the discard branch, in one pure
function that `doctor`, `gbrain status`, and `sources status` all call. It fixes all three
by construction, with no new table, no new command, and no migration.
It also means **the heartbeat this plan originally proposed to build already ships.** A
separate `live_ticks` table would be a fourth status surface on a fifth data source,
curing "three surfaces disagreed" by adding one more that can disagree.
## Base branch
The whole wave (PR-A, PR-B, PR-C) is based on
`garrytan/codex-as-agent-default-install`, not `master`. That branch carries the
bootstrap surface (`src/core/bootstrap/{host-specs,hooks}.ts`, `detectHarness()`) that
PR-B's harness tier needs, so **PR-B is not blocked** — an earlier revision of this doc
assumed it was.
That branch moves frequently; re-fetch before comparing anything against it. A stale
remote-tracking ref is an easy way to reach a confidently wrong conclusion here.
## Sequencing (decided)
Three PRs. Nothing is cut; the order changed.
### PR-A — close the incident (ships first)
1. **Wall-clock ceiling** in `lagFromContentMs` (`src/core/source-health.ts:189`): return
`wallClockSeconds` once it exceeds an absolute bound regardless of the content
comparison. Bound is a named env knob per repo convention
(`GBRAIN_STALENESS_CEILING_HOURS`, default 72, matching the existing
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`).
2. **Regression test** (acceptance criterion 1 below).
3. **E3**`src/commands/migrate-engine.ts` reconciles the running daemon.
4. **Wrapper self-disable**`src/commands/autopilot.ts:1314-1359`. Adapted from
`src/core/brain-repo-durability.ts:509-512`, NOT copied: two corrections the
engineering review established.
- Predicate is `[ ! -d "$repo" ]`, not `[ ! -d "$repo/.git" ]`. `--repo` may be a
subdirectory of the checkout (sync resolves the root itself by walking up), and
`.git` is a FILE in worktrees and submodules — either shape would self-disable a
healthy install.
- `exit 0` is correct for the durability wrapper because launchd fires it on
`StartInterval` (one shot). Autopilot runs under `KeepAlive=true` +
`ThrottleInterval=60` and systemd `Restart=always`, where exiting disables nothing
and instead produces a silent respawn-every-60s loop. The wrapper must
`launchctl bootout` / `systemctl --user disable --now` itself on those targets and
drop a marker that `--status` surfaces.
5. **Reconnect classifier**`src/commands/autopilot.ts:58-78`; a JS `TypeError` must not
substring-match as a config verdict.
6. **`autopilot --status` reads the heartbeat** instead of `existsSync`, and exits nonzero
when stale.
7. **E8 hygiene** — test-run pollution of `~/.gbrain/sync-failures.jsonl`;
`buildSyncManifest` (`src/core/sync.ts:105-140`) dropping git **`T`** (typechange).
Narrowed: `src/core/sync-delta.ts:130` passes `-M` only, so `C` is unreachable without
`--find-copies` and `U` needs a conflicted worktree. `C`/`U` handled defensively.
**Not in PR-A:** the lockfile-leak fix. Removing the leaked `~/.gbrain/autopilot.lock`
deletes the signal that distinguishes *crashed* from *never installed*
(`src/commands/status.ts:595-598`) before its replacement exists. It lands in PR-C
alongside `live status`.
### PR-B — the `harness` tier alone
The harness tier is the only tier the modal gbrain user can actually run (PGLite default,
desktop harness, behind NAT), so it ships alone and early rather than buried inside the
cathedral. Reuses `src/core/bootstrap/{host-specs,hooks}.ts`, which the base branch
already provides.
### PR-C — the ladder
`live` command family, `live.mode` bundle, shape detection, `cron`/`daemon`/`webhook`
tiers, advisor collector, `init` offer, watch tier, shared `os-scheduler.ts`, and the
lockfile-leak fix. **`live_ticks` is re-examined here against the shipped
`last_sync_at` heartbeat rather than assumed** — the burden is on the new table to justify
itself.
## The constraint being satisfied (quoted so it can be checked)
`docs/designs/AGENT_BOOTSTRAP_PLAN.md` on `origin/garrytan/codex-as-agent-default-install`,
decision **D9**:
> **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
> daemon). OPT-IN: 15-min harden cron. **Autopilot NOT default on PGLite** (verified: its
> sync/embed children would contend with every live serve for the single-writer lock, and
> nothing handles `LiveServeLockError` politely today) — **recommended on Postgres**; any
> future scheduled job must treat lock-held as skip-silently-and-log.
*Reconciliation:* D9 says "15-min harden cron"; the shipped default is **1800s / 30 min**
(`src/core/brain-repo-durability.ts:76`, `:659`). D9's figure is stale. This plan uses 30.
This plan's decisions are labelled **L1..L14** to avoid collision with that document.
## PR-C design (carried forward, not yet committed to a diff)
### Tiers — five active plus `off`
| tier | mechanism | expected cadence | engine gate |
|---|---|---|---|
| `off` | nothing | n/a — `live status` exits **0** | — |
| `harness` | agent hook / session boundary | event-driven, **age-exempt** | any (incl. Windows, containers) |
| `webhook` | HMAC push from GitHub | event-driven, **age-exempt**; paired keepalive `cron` supplies the age signal | any + reachable `serve --http` |
| `cron` | OS scheduler | declared `expected_cadence_seconds` | any; **PGLite floor 1800s + lock-aware skip** |
| `daemon` | resident autopilot, `runCycle` | 300s | **Postgres only** (D9) |
| `watch` | daemon + chokidar | **floor 300s for freshness purposes**, not the ~1s event latency | **Postgres only** (D9) |
Event-driven tiers are exempt from age-based failure; a webhook repo with no pushes for
three days is healthy, not failed. `watch`'s freshness cadence is decoupled from its event
latency so a GC pause is not a FAIL.
`off` is a first-class bundle member with `enabled: false`, copied from
`src/core/pace-mode.ts:65-71`.
### L1 — Shape detection predicate
| Signal | Source | Meaning |
|---|---|---|
| engine | `config.engine` | `postgres` required for `daemon`/`watch` |
| interactive desktop harness | `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT`, `CODEX_HOME`, `CODEX_SANDBOX`, `CODEX_CI` (**env only**) | any present → cap at `harness` |
| long-lived host | `detectInstallTarget()` ∈ {`macos`, `linux-systemd`, `ephemeral-container`+injection point} | a reboot-surviving scheduler exists |
| server posture | `serve --http` configured, or `minion_mode != 'off'` | corroborating, never sufficient alone |
`macos` is in the long-lived row deliberately: `detectInstallTarget()` returns `'macos'`
unconditionally on darwin (`src/commands/autopilot.ts:1277`), and darwin is the platform
of the origin incident. Omitting it would make the incident host permanently
shape-ineligible.
**No filesystem probes for harness identity.** The `~/.claude/hooks/...` class of probe
(`src/commands/autopilot.ts:1304`) is what false-positives today. Env vars only.
Any inconclusive read falls to `harness`, never `daemon`.
### L2 — `live status` exit codes
| Condition | Exit |
|---|---|
| fresh, or `live.mode == off` | 0 |
| PGLite lock held by a live `serve` (`blocked_by_serve`) | 0 |
| tier enabled + heartbeat missing or stale | 1 |
| drifted install, or DB **connect failure** | 2 |
`live.mode == off` exiting 0 is load-bearing: otherwise every fresh install exits nonzero,
which is the `cycle_freshness` #2540 lesson (never-configured must not turn the surface
red). And lock-held is **not** an outage: `src/core/pglite-engine.ts:444` acquires the file
lock on every `connect()` and throws if it fails, so on the default engine with a resident
`serve`, treating that as exit 2 would make FAIL the steady state.
### L3 — `skipped_locked` semantics
A tick that cannot acquire the PGLite lock **does not satisfy freshness and does not
degrade it**. It is neutral: logged, not recorded as work-done, and not counted toward
staleness for a grace window of 3 consecutive skips, after which the surface reports
`blocked_by_serve` with the remediation inline. Treating it as work-done rebuilds the
71-day false-green; treating it as failure makes the default engine permanently red.
### L4 — Scheduler ownership
Ownership lives in a sidecar `~/.gbrain/live-ownership.json`, **not** in an entry comment.
On darwin both harden and autopilot install launchd **plists** (files, not comment-bearing
crontab lines), so the `# gbrain:autopilot v0.11.0` marker convention does not generalize.
The sidecar covers all install targets uniformly.
Three enumerated cases:
1. **Harden cron exists + pull opted in** → rewrite through `os-scheduler.ts`,
`ownership=live-adopted`.
2. **Harden cron exists + pull declined** → leave it entirely alone; install a separately
labelled `live` entry. **This is the default and lands first**, so PR-C's `live on`
never meets an existing harden cron without a rule.
3. **Neither exists** → install a `live` entry, `ownership=live`.
`live off` removes only entries `live` created and reverts adopted ones to harden.
Pre-existing `gbrain autopilot` installs are **migrated, not orphaned**: first `live
status` after upgrade reports `tier: daemon (legacy autopilot)` and offers one-time
adoption.
### L5 — Op scopes
| Op | scope | localOnly | remote |
|---|---|---|---|
| `live_status` | `read` | no | allowed; omits `local_path`, scheduler artifact paths, and log tail |
| `live_tick` | `write` | **yes** | reject |
| `live_on` / `live_off` | `admin` | **yes** | reject |
| `live_self_heal` | `admin` | **yes** | reject |
Self-heal walks a **DB-supplied** `local_path` and then writes a scheduler entry.
`src/commands/doctor.ts` already gates its git short-circuit on `localOnly === true`
(*"a remote-callable code path must NOT walk DB-supplied `local_path` values with
subprocess calls"*). Self-heal honors that and additionally requires a realpath match
against the anchor via `isAnchorOwnedSyncPath` (`src/commands/sync.ts:1296`).
**Bootstrap paradox, acknowledged:** if the broken thing is the scheduler entry, a
scheduled self-heal never runs. Non-scheduled triggers are the `harness` tier (PR-B) and
an explicit `gbrain live doctor`. PR-C ships self-heal with both, not with a scheduled
trigger alone.
### L6 — Revert
A code revert leaves plists, crontab lines, systemd units, and (E1) a GitHub webhook
installed and unowned. Therefore:
- **Revert requires `gbrain live off` first** on any enabled host. Stated in the PR body.
- The generated wrapper self-disables on a **marker file** written by `live on` and removed
by `live off`. Not a `gbrain live --help` probe: that adds a process spawn per tick and
assumes an exit code the CLI does not guarantee.
- The migration, if `live_ticks` survives PR-C's re-examination, is additive and uses the
**next free version at implementation time** (125 is the current max; two waves may land
first).
### L7 — E5 must not use `nag-state.ts`
`src/core/skillpack/nag-state.ts` is skillpack-scoped (schema `gbrain-skillpack-nag-v1`,
entries keyed on `pack_version`, `DEFAULT_NAG_CEILING = 3`, suppressed thereafter). Wiring
a dead-sync alarm through it means a genuinely broken brain goes silent after three
notices, which is a suppression mechanism for the exact failure mode whose defining
property was 71 days of silence.
E5 instead uses a **rate limit, not a ceiling**: at most once per session, never
suppressed permanently, escalating in terseness rather than disappearing.
### L8 — E1 webhook dependencies (previously unpriced)
Creating a GitHub webhook programmatically needs an `admin:repo_hook` token. No
acquisition, storage, scope, or rotation story existed. Therefore E1 ships in **manual
mode only**: `live on --tier webhook` generates the secret, resolves and prints the payload
URL, and the user pastes it into GitHub, matching what `gbrain sources webhook set`
(`src/commands/sources.ts:909-916`) already does. No token, no remote hook creation, no
`live off` remote deletion problem.
The "verified test ping" must originate **from GitHub**, not locally. A local ping proves
nothing through NAT and would be an artifact-presence check, the precise anti-pattern in
the Origin section.
### L9 — `live_ticks` retention
If the table survives PR-C, the sweep runs **inside `live tick`** (bounded best-effort
DELETE on a TTL), not only in the cycle's `purge` phase. `purge` is a `runCycle` phase
(`src/core/cycle.ts:1434`), and `runCycle` runs only on `daemon`/`watch` — the `cron`,
`webhook`, and `harness` tiers would accumulate forever.
## Scope decisions (all accepted; PR assignment added)
| # | Item | PR | Note |
|---|---|---|---|
| L10 | Approach C: full ladder | A/B/C | user chose the cathedral; resequenced, not cut |
| L11 | Tier default keys on deployment shape, not vendor | C | Hermes has zero detectable signal |
| E1 | Webhook tier, **manual mode** (L8) | C | |
| E2 | Self-heal with `.tmp`+rename+`.bak` rollback | C | bootstrap paradox handled per L5 |
| E3 | `migrate-engine` reconciles the daemon | **A** | the literal root cause |
| E4 | Pull cron adoption per L4, separate opt-in per L12 | C | |
| E5 | Agent-facing staleness, rate-limited not nag-ceilinged (L7) | C | |
| E6 | Windows hard error naming `--tier harness` | C | `detectInstallTarget()` has no win32 branch |
| E7 | `live_ticks`**re-examined, not assumed** | C | the shipped `last_sync_at` heartbeat may suffice |
| E8 | Hygiene, narrowed to git `T` | **A** | |
### L12 — E4's pull cron is an autonomy question
`docs/guides/upgrades-auto-update.md:41-43` states *"`auto` is deliberately NOT a default
anywhere — it's an explicit autonomy grant, because applying code from GitHub unattended
is, by design, remote code execution."* This plan does **not** flip `self_upgrade.mode`.
E4 schedules `git pull` every 30 minutes. That is content, not code, and durability keeps
gbrain's hooks local and untracked so a pulled commit cannot rewrite executable hook code.
But it is still unattended network fetch into a directory gbrain runs tooling against.
Therefore the pull cron is a **separate opt-in from the tier**, proposed and explained by
`live on`, never silently bundled.
### L13 — The directive's internal tension, stated
"OpenClaw and Hermes default to always-up-to-date" sits against L1's "shape detection
recommends, never installs" and D9's "almost nothing on by default." These are reconciled
by scope: shape detection sets the **recommended tier** and pre-selects it in the `init`
consent prompt, so a shape-matching host is one keystroke from always-on rather than
silently converted. Whether that consent is required on **upgrade** as well as fresh
install is **open decision F1** below.
## L14 — Acceptance criteria
1. **Three-surface honesty.** A source whose `local_path` is deleted, whose `last_sync_at`
is 71 days old, whose `newest_content_at` is **non-NULL**, and whose `chunker_version`
**matches** must report stale/fail from `doctor` and `gbrain status`, and must surface
the lag in `sources status`. Both fixture preconditions are required: a NULL
`newest_content_at` already falls through to wall-clock
(`src/commands/doctor.ts:4335-4342`) and a chunker mismatch already disables the
fallback (`:4318`), so a naive fixture passes against unfixed code.
*`sources status` is held to output, not exit code — it has no exit contract today and
adding one is an undeclared breaking change to a read-only dashboard.*
2. **Quiet-source non-regression.** A source with a recent `last_sync_at`, an unreachable
clone, and no new content must still report **OK**. This is the 16-source / doctor
70→30 incident; the ceiling must not re-light it.
3. **Install honesty** (PR-C). `live on --tier cron` verifies the job loaded and exits
nonzero if not; deleting the repo makes `live status` exit nonzero and name the path;
`live off` leaves nothing.
4. **Concurrency** (PR-C, **Postgres only**). Two tiers ticking produce one import and one
neutral skip record. On PGLite the second process cannot open the DB at all, so the
defined outcome is a log line and no row.
5. **Watch tier** (PR-C). E2E expects **queued-job-failure**, not synchronous rejection —
`ingest_capture` enqueues and returns.
6. **Engine parity** (PR-C, if `live_ticks` survives). DDL identical in both engines,
pinned by `test/e2e/engine-parity.test.ts`; bootstrap probe-set entry pinned by
`test/schema-bootstrap-coverage.test.ts`.
## Open decisions (unanswered — do not silently default)
- **F1.** Does shape-detected always-on apply on **upgrade** as well as fresh install?
Codebase precedent (`src/commands/upgrade.ts:513-516`, `mcp.publish_skills`) is
new-installs-only with a one-time prompt for existing. Gates PR-C only.
- **F2.** Command noun and config key: `gbrain live` + `live.mode` (requires renaming the
existing `liveSyncStatus` helper at `src/core/db-lock.ts:749` to `syncInProgress`, two
call sites) vs `gbrain sync live` + `sync.live.mode`. Gates PR-C only.
## Deferred to TODOS.md
- Full Windows `schtasks` tier — no test machine; `harness` covers it
- Per-tier cost meter for `daemon` / `watch`
- Cross-OS scheduler probing as a `live status` diagnostic (TODO-V19-D stays open; the
heartbeat makes it optional rather than load-bearing)
- Centralize the three freshness call sites onto one `freshnessVerdict()` helper
(existing filed P3, now partially satisfied by PR-A's single-function fix)
## Dream state delta
PR-A leaves brain currency *honest*. PR-B leaves it *workable for the modal user*. PR-C
leaves it *a product feature*. Remaining gap to the 12-month ideal: currency is still
something the user turns on, not something simply true of a configured brain. F1 is the
decision that closes or preserves that gap.
## Reviewer concerns (unresolved after 3 iterations)
- **Scope, from both reviewers:** PR-C remains large (command family, mode bundle, shape
detector, three tiers, advisor collector, init prompt, webhook, watch tier, scheduler
extraction). The PR-A/B/C split answers the sequencing objection but not the size of C
itself. Revisit at PR-C planning with the incident already fixed.
- **`live_ticks` necessity** is explicitly unresolved and assigned to PR-C rather than
decided here.
+4 -5
View File
@@ -154,11 +154,10 @@ these are the densest source of real bugs in the whole backlog.
`aliases:`), first-H1-title, and basename fallback resolution (path-equality-only gives
~5.5% edge recall on real vaults). Master shipped global-basename (#1388); the alias/
title fallbacks are the still-novel part.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **RESOLVED via #2576.**
The extractor no longer gates on the frozen `DIR_PATTERN` whitelist: any dir-shaped
path produces a candidate and the persist paths' page-existence checks decide, so
pack-declared directories (`person/`, `writing/`, `wiki/*`, `ops/`) link without a
prefix registry.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **OPEN, high.** The
link extractor's `DIR_PATTERN` is a frozen 16-prefix const that ignores pack-declared
`path_prefixes`, so default-pack installs silently lose wikilinks to `person/`,
`writing/`, `wiki/*`. Resolve prefixes from the active pack.
- **DB-source extraction** (#1539, @afshaker) — **OPEN, high.** The cycle's extract phase
only walks the filesystem, so DB-resident pages (imported transcripts, remote-DB brains)
never get links/timeline and `brain_score` is capped. Thread `source:'db'`.
+1 -1
View File
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain skillpack scaffold voice-agent`
`gbrain install voice-agent`
That's it.
+29 -31
View File
@@ -8,19 +8,11 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
that lives on top of that format.
If you're touching **memory behavior** rather than retrieval ranking — the
Retrieval Reflex push path, conversation→facts write-back, cross-session
continuity, source isolation — the gate for that layer is **BrainBench**
(`gbrain eval brainbench`): see [`eval/BRAINBENCH.md`](./eval/BRAINBENCH.md).
The two stack: this doc's capture→baseline→replay loop gates query-level
result sets; BrainBench gates the memory behaviors above them, with its own
committed baseline (`evals/brainbench/baselines/main.json`) compared against
MAIN's copy in CI so a PR can't self-approve a regression.
## v0.41 update — the LOOP is now real
## The eval gate loop
`gbrain bench publish` + `gbrain eval gate` stitch captured eval rows into
a pass/fail gate. Two gates:
Before v0.41, you could capture eval rows and replay them but nothing
stitched them into a gate. `gbrain bench publish` + `gbrain eval gate`
close the loop. Two gates:
- **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline
you captured against your current brain. Catches: "did my refactor break
@@ -47,7 +39,7 @@ gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/person
gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson
```
### Privacy posture
### Privacy posture (D9)
**Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real
user captures stay local in `~/.gbrain/baselines/`. The boundary is
@@ -139,9 +131,14 @@ gbrain query "anything" >/dev/null
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
```
The full on/off resolution order (config beats env var, both directions) is
documented once in [`eval-capture.md`](./eval-capture.md) — that file is the
capture contract.
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
```json
{"eval": {"capture": true}} // force on
{"eval": {"capture": false}} // force off
```
Explicit config beats the env var both directions.
## The 4-command loop
@@ -208,7 +205,7 @@ retrieval, and which queries did it move most?"
For a third evaluation axis — public benchmark, ground-truth labels, full
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
<dataset.jsonl>` runs the LongMemEval benchmark against gbrain's
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
imported, the question asked, the hypothesis emitted as JSONL — exactly the
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
@@ -340,7 +337,7 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
## Public benchmarks: LongMemEval
## Public benchmarks: LongMemEval (v0.28.8)
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
benchmark directly against gbrain's hybrid retrieval. Different evaluation
@@ -401,7 +398,7 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
LLM latency.
## Measuring brain consistency over time
## Measuring brain consistency over time (v0.32.6)
`gbrain eval suspected-contradictions` is a complementary measurement
instrument: it samples retrieval results for unmarked semantic
@@ -438,19 +435,20 @@ commands per high-severity finding.
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
decision criteria gated on Wilson CI lower-bound.
## Eval infrastructure: by-type breakdowns, the hermetic gate, batch scoring
## v0.40.1.0 Track D — Eval infrastructure
Three further eval surfaces, and the dev loop that uses them.
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
covers the dev loop that uses them and the gates they enforce.
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
LongMemEval computes per-question-type recall internally, and surfaces it in
machine-readable form:
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
surfaces it in machine-readable form. Two additive changes:
1. Every per-question JSONL row includes a `question: string` field so the
1. Every per-question JSONL row now includes a `question: string` field so the
`gbrain eval cross-modal --batch` consumer (below) can read it without
joining back against the source dataset.
2. The `--by-type` flag emits a final aggregate line keyed by `question_type`:
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
```json
{"schema_version": 1, "kind": "by_type_summary",
@@ -482,11 +480,11 @@ echo "exit=$?" # 1 if any type fell below 0.80
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
The structural fix for "PRs touching `src/core/search/` silently regress
retrieval." A "replay against captured eval_candidates" design can't work in
CI (CI has no captured production queries), so the gate is hermetic; see the
`contributor-mode CI capture` TODO in `TODOS.md` for the deferred
real-query version.
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
silently regress retrieval." Replaces the original "replay against captured
eval_candidates" design (which Codex caught as non-functional in CI — see
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
real-query version).
How it works:
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
@@ -501,7 +499,7 @@ How it works:
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
#### Refreshing the qrels fixture (the `Why:` discipline)
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
When CI fails because a legitimate ranking change moved expected slugs, the
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
+9 -9
View File
@@ -28,10 +28,10 @@ receipt file from disk and re-renders it. The other modes need the brain.
|---|---|---|
| `--limit N` | 100 | Random sample of N takes from the brain. |
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud rather than silently blowing the budget. |
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
| `--json` | off | Emit the full receipt to stdout. |
## Receipt JSON shape (`schema_version: 1`)
@@ -50,7 +50,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
},
"prompt_sha8": "abcd1234",
"models_sha8": "abcd1234",
"models": ["openai:gpt-5.2", "anthropic:claude-opus-4-7", "google:gemini-2.0-flash"],
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
"cycles_run": 3,
"successes_per_cycle": [3, 3, 2],
"verdict": "pass",
@@ -73,8 +73,8 @@ receipt file from disk and re-renders it. The other modes need the brain.
- `schema_version` — locks the contract. Adding optional fields is additive
and compatible. Renaming, removing, or changing semantics bumps the version.
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch.
When the rubric definition changes, both fields update,
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
(codex review #3). When the rubric definition changes, both fields update,
and trend mode groups runs accordingly so a stricter rubric doesn't
silently look like a quality drop.
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
@@ -83,7 +83,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
models in `--models` doesn't change the sha (sort is stable).
- `successes_per_cycle` — count of contributing models per cycle. A model
contributes when (a) its JSON parsed AND (b) every declared rubric dim
has a finite score (a missing dim drops the whole contribution).
has a finite score (codex review #5 missing-dim drops the contribution).
- `verdict``pass` if every dim mean >= 7 AND every dim min across
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
2/3 models contributed complete scores.
@@ -93,11 +93,11 @@ receipt file from disk and re-renders it. The other modes need the brain.
## Receipt persistence
Receipts persist to **`eval_takes_quality_runs`** (the DB is authoritative)
AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
as a best-effort artifact. The DB row carries the full receipt JSON in the
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
can still reconstruct via `loadReceiptFromDb`.
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
-162
View File
@@ -1,162 +0,0 @@
# BrainBench — cross-harness memory conformance methodology
BrainBench generalizes gbrain's internal eval surface into a reproducible,
cross-harness benchmark for agent memory. It operationalizes the four failure
modes of the agent-memory thesis: **know-to-ask** (nobody has a push path),
**push precision/recall** (the intrusion budget must be enforced),
**write-back fidelity** (memory write is even less solved than read), and
**cross-session continuity** (continuity that survives the harness hop).
Every subsequent memory PR must move — or hold, with a recorded justification —
a BrainBench number to merge.
Operator quickstart, corpus layout, and fixture-authoring rules live in
[`evals/brainbench/README.md`](../../evals/brainbench/README.md). This document
is the methodology: what the numbers mean, what they deliberately do not mean,
and how the gate governs change.
## Seam disclosure (read this before comparing rows)
Every scoreboard row carries a `seam` column:
| Harness | Seam | What the row actually measures |
|---|---|---|
| `openclaw` | **production** | The shipped OpenClaw context-engine pipeline, byte-for-byte (`extractCandidates``resolveEntitiesToPointers`, 3-pointer budget, prior-context suppression, markdown pointer block). |
| `claude-code` | **contract** | gbrain's memory primitives driven through the UserPromptSubmit hook wire contract (`{prompt, session_id, cwd}` in → `{hookSpecificOutput.additionalContext}` out, exported from `src/eval/brainbench/adapters/claude-code.ts`). 2-pointer budget; NO conversation memory — this row deliberately models the memoryless wire contract (suppression off), so the re-injection cost is visible as `false_fire_rate`; the shipped `gbrain hook user-prompt` layers transcript-based cross-turn dedupe on top of this same contract. |
| `codex` | **contract** | The fragments model: a static entity-index preamble (computed once, slugs not counted as injections) + at most ONE per-turn fragment. Measures how much push quality degrades when injection is mostly static. |
**Contract rows do NOT measure third-party harness behavior.** They measure
gbrain's primitives under each harness's injection-shape constraints. The rows
are comparable because fixtures, brain, and gold are identical — only the seam
contract varies. The real Claude Code integration has landed (`gbrain hook
user-prompt`, registered by `gbrain bootstrap`); flipping this adapter to exec
the real hook and report `production` numbers is a filed follow-up (TODOS.md —
"Flip contract adapters to production"). Same for codex fragments when that
integration lands. Also not graded, by design: the production orchestrator's
config gate, integration heartbeat, and 1500 ms timeout wrapper.
All three adapters drive ONE shared pipeline (`adapters/shared.ts`) with
declarative configs — comparability is structural, not disciplined.
## Metrics (formulas)
All micro-averaged per (harness × suite) cell; registered in
`src/core/eval/metric-glossary.ts` (plain-English in
[`METRIC_GLOSSARY.md`](METRIC_GLOSSARY.md)); JSON output carries one
`_meta.metric_glossary` block.
- `know_to_ask_failure_rate` = |should-retrieve turns where injected ∩ (gold acceptable) = ∅| / |should-retrieve turns|. Lower better.
- `false_fire_rate` = |stay-silent turns with any injection| / |stay-silent turns|. Lower better. Anti-gaming companion: "always inject" cannot win both.
- `push_precision` = Σ|injected ∩ (gold acceptable)| / Σ|injected| over turns with injection. `acceptable_slugs` count for precision, not recall.
- `push_recall` = Σ|injected ∩ gold| / Σ|gold| over should-retrieve turns. Pointer budgets cap this by design.
- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
- `continuity_rate` = |decision probes recalled by the reader| / |probes|, per READER harness. The writer fixture's decisions persist through the production write-back pipeline — which is harness-INDEPENDENT in v1 — so each pair preps once and every harness replays the read-only reader against the same persisted state (an ordered writer×reader sweep would rebuild byte-identical brains for identical scores). A probe succeeds via pointer injection or stored-fact keyword lookup. The per-writer axis activates when harness-specific write paths land.
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant. Granularity disclosure: detection is slug-keyed, so it catches injection of slugs seeded ONLY in a foreign source; a same-slug cross-source CONTENT leak would require the engine's source-scoped SQL itself to fail, which the engine-layer source-isolation fuzz (gbrain-evals Cat 22) covers directly.
- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
- `extraction_recall` / `extraction_precision``--llm` runs only: the real extractor's output vs gold keyword probes.
### What know-to-ask deliberately means in v1
It grades the **deterministic injection decision** — the Reflex pipeline that
ships at the seam. The agent never "knows to ask"; the reflex pushes. An
agent-LLM-in-the-loop replay (did the *model* issue a retrieval call when the
reflex stayed silent?) is **pre-registered as the `--live` extension**:
fixture-compatible, seeded, N-repeat methodology — and unimplemented. No LLM
grading is faked in v1.
### Difficulty is stratified on purpose
Several know-to-ask variants exercise documented v1 reflex limits (lowercase
mentions, surname-only references — `src/core/context/entity-salience.ts`).
Gold records what SHOULD happen; the committed baseline records what the
current system does (`know_to_ask_failure_rate` ≈ 0.15 at v1). The gap is the
measured roadmap, not a bug in the bench.
## Pre-registered expectations (v1, recorded before the first published run)
1. The production seam (openclaw) leads `push_recall` strictly: 3-pointer > 2-pointer > 1-fragment budgets. *(Observed at landing: 0.81 / 0.65 / 0.45.)*
2. The no-suppression contract (claude-code) is the only seam with `false_fire_rate` > 0. *(Observed: 0.020.03.)*
3. `write_back_fidelity` = 1.0 and `provenance_accuracy` = 1.0 in deterministic mode — the production pipeline must not lose or mis-attribute gold facts it was handed. Anything below 1.0 is a pipeline bug, not benchmark noise.
4. `source_isolation_violations` = 0 everywhere.
5. `push_precision` = 1.0 at v1 (exact-match resolution arms cannot inject an irrelevant page on this corpus); expected to dip below 1.0 when fuzzy/semantic resolution lands — that dip is the precision/recall trade made visible.
## Determinism & statistical posture
The harness is deterministic end-to-end: regex extraction + SQL resolution
(zero LLM, zero embeddings — facts seed with NULL embeddings; keyword/alias
arms carry retrieval), seeded PRNG corpus, one in-memory PGLite reset between
fixtures. Two runs produce identical metrics, so N-repeat error bars are
meaningless here (stddev = 0 by construction, the gbrain-evals "deterministic
adapters" convention) and the gate can be exact: **any flipped gold item is a
real behavior change.** Bootstrap/CI discipline applies to the future `--live`
and `--llm` published runs, which are model-stochastic.
## Gate governance (decision 4 — why a PR can't self-approve)
CI (`.github/workflows/test.yml` `brainbench` job, local parity
`scripts/ci-brainbench-gate.sh`) fetches the baseline **from main**
(`git show origin/master:evals/brainbench/baselines/main.json`) and compares
HEAD's fresh run against it:
- **Same `fixtures_hash`** → count-aware gate: any newly-failed gold item, any
adverse gated-metric move, or any isolation violation fails (exit 1).
- **Different hash** (the PR changed fixtures) → **corpus-bless mode**: the
PR's committed baseline must EXACTLY match HEAD's actual run (the file
cannot lie; exit 2 until `--update-baseline` is re-run), and any adverse
move vs main's baseline requires a `justification` string in the committed
baseline — visible in the PR diff, judged by the reviewer.
- `--allow-regression "reason"` is the local one-off escape hatch; the reason
is recorded in the run output. It is not available to CI.
The committed baseline is diff-stable by construction (metrics rounded to 4
decimals, keys sorted, receipts excluded; the run CONFIG — holdout/llm/
harness/suite sets — is bound into it, and comparisons across mismatched
configs are inconclusive). Same-hash hardening: any committed-baseline edit
without a fixture change must byte-match the actual run (receipts-backed), a
regressing receipts-backed update still needs a `justification`, gold_total
may not move at all under an unchanged corpus, and the CI script refuses a
working-tree baseline deletion. Holdout fixtures (~15%) are excluded from the
gate and scored only in published runs (`--include-holdout`).
Accepted residuals (review-enforced, by design): a `justification` string is
judged by the human reviewer, not parsed; count-preserving corpus dilution
(replacing hard fixtures with easy ones at equal gold_total) is visible only
in the fixture diff; and the ratchet does not auto-tighten — improvements
aren't banked into main's baseline until a PR updates it (a regression back
to the stale baseline level passes; periodic re-baselining is the operator's
job, filed as a TODO).
## Gold methodology
Gold derives from the corpus generator (the same PRNG step that authors a turn
authors its annotation, so gold-vs-text drift is structurally impossible for
generated fixtures), plus hand-authored spike fixtures that froze the schema.
A 10% double-label validation pass (independent agent review of fixture text vs
gold, blind to the generator's intent) is run at corpus-change time; its
receipt is recorded in the corpus `_ledger.json` and any disagreement is a
fixture bug to fix, not a tolerance to average over.
## Interop
- **Foreign runners (gbrain-evals):** the subprocess contract is
`gbrain eval brainbench --fixtures DIR --gold DIR --json --out FILE`;
schemas in `evals/brainbench/schema/`. The sibling gbrain-evals repo wires
this as `eval/runner/brainbench-memory.ts` with a published scorecard.
- **Memory-verbs conformance kit (Cathedral 1):** conformance scenarios
convert to BrainBench fixtures via the published fixture schema
(`schema_version` 1) once that wave lands — the conversion path is the
schema itself; no bespoke importer is required.
- **Naming note:** "BrainBench" historically also names the in-house
retrieval corpus in the sibling gbrain-evals repo (the 145-query relational
suite, Cat taxonomy) and `test/cathedral-ii-brainbench.test.ts` (v0.20.0
code-graph recall pins). This suite — the cross-harness memory conformance
bench — is the generalization the name now primarily refers to; the older
references stand unchanged.
## Extends docs/eval-bench.md
The capture → baseline → replay loop in [`eval-bench.md`](../eval-bench.md)
gates *retrieval result sets* at the query level. BrainBench gates the
*memory behaviors* above them. The two share the receipts discipline and the
.gbrain-evals run ledger (`EvalRunRecord` v3; brainbench records once per
sweep under `mode: 'n/a'`).
-90
View File
@@ -168,96 +168,6 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
## BrainBench — Cross-Harness Memory Conformance
### Know-to-ask failure rate (BrainBench)
**Key:** `know_to_ask_failure_rate`
**Plain English:** Of the conversation turns where memory SHOULD have surfaced something unprompted, the fraction where nothing relevant was injected. This is the thesis failure mode every agent harness shares: the agent can't ask for what it doesn't know it forgot — the memory layer has to volunteer it.
**Range:** 0..1, LOWER is better. 0.15 means memory stayed silent on 15% of the turns where it had the answer.
### False-fire rate (BrainBench)
**Key:** `false_fire_rate`
**Plain English:** Of the turns where memory should have stayed SILENT, the fraction where it injected anyway. The anti-gaming companion to the know-to-ask rate — "always inject" would ace one and bomb the other. Silence beats noise.
**Range:** 0..1, LOWER is better.
### Push precision (BrainBench)
**Key:** `push_precision`
**Plain English:** Of everything the memory layer volunteered into context, what fraction was actually relevant to the turn? Micro-averaged over injected pointers, so a 3-pointer turn weighs three times a 1-pointer turn — the way a token budget experiences it.
**Range:** 0..1, higher is better.
### Push recall (BrainBench)
**Key:** `push_recall`
**Plain English:** Of everything that SHOULD have been volunteered (the gold pointers), what fraction actually was? Pointer budgets cap this by design: a seam that may inject only 1 fragment cannot reach full recall on a 3-entity turn — that constraint is what the per-harness rows measure.
**Range:** 0..1, higher is better.
### Write-back fidelity (BrainBench)
**Key:** `write_back_fidelity`
**Plain English:** Of the facts stated in a conversation, what fraction survived the PRODUCTION conversation→memory pipeline (segmentation, insertion, dedup) and are findable afterward with the right entity attached? Measures the write path users actually run, not a test-only insert.
**Range:** 0..1, higher is better.
### Provenance accuracy (BrainBench)
**Key:** `provenance_accuracy`
**Plain English:** Of the facts that survived write-back, what fraction carry correct provenance — the right source tag, session id, and origin page? A fact you can't trace is a fact you can't trust, audit, or expire.
**Range:** 0..1, higher is better.
### Cross-session continuity rate (BrainBench)
**Key:** `continuity_rate`
**Plain English:** A decision is recorded in one session and persisted through the production write path; a different harness asks about it later on the same brain. What fraction of those decision probes were recalled — by pointer injection or stored-fact lookup? This is the continuity-that-survives-the-harness-hop moat, measured.
**Range:** 0..1, higher is better. Scored per reader harness (the v1 write path is harness-independent, disclosed in docs/eval/BRAINBENCH.md).
### Source-isolation violations (BrainBench)
**Key:** `source_isolation_violations`
**Plain English:** Count of injected pointers that belong to a source other than the active one. Cross-source leakage is gbrain's must-never-violate invariant (a missed source filter is a data leak), so this gates at ZERO — any baseline, any run.
**Range:** 0..n, count. MUST be 0; any value above 0 fails the gate.
### Average injected tokens per turn (BrainBench)
**Key:** `avg_injected_tokens`
**Plain English:** Estimated tokens of volunteered context per replayed turn (chars/4 heuristic). The intrusion-budget diagnostic: two seams with equal precision can differ 3x in how much context they spend to get it. Reported, not gated, until calibration data exists.
**Range:** 0..n tokens, judgment call — lower is cheaper, but starving the agent has its own cost. Non-gating.
### Extraction recall (BrainBench --llm)
**Key:** `extraction_recall`
**Plain English:** With the real LLM extractor running (instead of the deterministic gold extractor), what fraction of the gold facts did it actually extract and persist? Only scored in --llm runs — the hermetic CI gate never calls a model.
**Range:** 0..1, higher is better. Absent in deterministic runs.
### Extraction precision (BrainBench --llm)
**Key:** `extraction_precision`
**Plain English:** Of everything the real LLM extractor persisted, what fraction matches a gold fact? Low precision means the extractor invents or over-extracts — junk memory that pollutes future recall.
**Range:** 0..1, higher is better. Absent in deterministic runs.
---
## Coverage
+9 -10
View File
@@ -1,6 +1,6 @@
# Search Mode Evaluation Methodology
_How gbrain measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible — pinned datasets, recorded seeds, and the exact run commands below._
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
## 1. What this measures and what it doesn't
@@ -21,15 +21,14 @@ If you want to know how a mode behaves on YOUR brain, run `gbrain search stats -
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
- **BrainBench v1**`n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
No private brain content is used in any reported result. The NDJSON run records under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
## 3. Sample selection
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode bundle is the only independent variable. A mode Δ therefore measures the joint effect of every knob the bundles differ on — today that's `tokenBudget`, `expansion`, `relationalRetrieval` (the typed-edge fourth recall arm, ON for balanced/tokenmax, OFF for conservative), and `searchLimit`; the canonical diff is `MODE_BUNDLES` in `src/core/search/mode.ts`.
- **Cache comparability across upgrades.** The query cache keys on a versioned knobs hash (`KNOBS_HASH_VERSION` in `mode.ts`) that folds in the active knob set + embedding column/provider, so one mode's cached results can't be served to another mode's queries — and a version bump makes prior rows unreachable (one-time miss spike). Cross-run comparisons that straddle a knobs-hash bump see a cold cache on the first re-run.
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from a run's `--output` dumps.
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
## 4. Run procedure
@@ -47,14 +46,14 @@ gbrain eval run-all \
--limit 500 \
--budget-usd-retrieval 5 \
--budget-usd-answer 20 \
--output docs/eval/results/<version>/
--output docs/eval/results/v0.32.3/
# Render the comparison.
gbrain eval compare --md > docs/eval/results/<version>/README.md
gbrain eval compare --json > docs/eval/results/<version>/comparison.json
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
```
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. When a release publishes eval numbers, the `--output` dumps under `docs/eval/results/<version>/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. **No dumps are committed in the repo right now** — reproduce by running the commands above; determinism (§3) means your re-run matches the reported orderings.
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
## 5. Threats to validity
@@ -69,7 +68,7 @@ Honest list. We name what would let a critic dismiss the numbers.
## 6. Per-question raw outputs
Every reported metric is reproducible from the NDJSON dumps a run writes to its `--output` directory (`docs/eval/results/<version>/` when a release publishes numbers; none are committed right now — see §4). The commit SHA in the methodology footer pins the code version.
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
+37 -45
View File
@@ -12,13 +12,13 @@ surfaces**, and which one you pick depends on the operation.
│ gbrain process │
│ │
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
openclaw, fork) ────┼──▶ MCP ops surface │ │ local-only │ │
│ │ (HTTP + OAuth) │ │ commands │ │
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
│ │ (HTTP + OAuth) │ │ admin ops │ │
│ │ │ │ │ │
│ │ search, query, │ │ sync, embed, │ │
│ │ put_page, │ │ extract, │ │
│ │ get_page, │ │ dream, │ │
│ │ find_experts, │ │ enrich, ... │ │
│ │ put_page, │ │ dream, doctor,│ │
│ │ get_page, │ │ autopilot, │ │
│ │ find_experts, │ │ init, secrets │ │
│ │ ... │ │ │ │
│ └──────────────────┘ └────────────────┘ │
│ ▲ ▲ │
@@ -26,7 +26,7 @@ surfaces**, and which one you pick depends on the operation.
│ │ │ │
│ thin-client OAuth shell-job `inherit:`
│ (preferred for (only path for │
│ MCP-equivalent ops) local-only work)
│ MCP-equivalent ops) localOnly ops)
└─────────────────────────────────────────────┘
```
@@ -45,7 +45,7 @@ the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
The host runs gbrain as a long-lived HTTP server:
```bash
gbrain serve --http --port 3131
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
```
The agent registers as an OAuth client (one-time):
@@ -75,27 +75,22 @@ commands through the configured remote MCP. The agent can call
agent to a specific source within a federated brain.
- One audit surface (`mcp_request_log`) covers every op call uniformly.
## Surface 2 — local-only work via shell-job `inherit:`
## Surface 2 — localOnly admin ops via shell-job `inherit:`
Two mechanisms keep local-only work off the remote surface, and they operate
at different layers:
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
list (as of v0.36.5.0) includes:
- **Op layer:** operations flagged `localOnly: true` in
`src/core/operations.ts` are filtered out of the HTTP MCP surface entirely
— a remote caller never sees them.
- **CLI layer:** on a thin-client install (remote MCP configured, no local
engine), commands that require a local engine or the local filesystem are
refused at dispatch with a pinpoint hint naming the closest alternative.
The authoritative set is `THIN_CLIENT_REFUSED_COMMANDS` in `src/cli.ts`
read it there rather than trusting any list copied into a doc; it covers
`sync`, `embed`, `extract`, `dream`, `enrich`, `serve`, `config`, and a
couple dozen more.
- `sync` (filesystem walks need local FS access)
- `embed` (orchestrates the embed pipeline)
- `extract` (walks markdown files)
- `dream` (synthesis cycle)
- `doctor` (filesystem hygiene checks)
- `autopilot` (background daemon orchestration)
- `init` (creates `~/.gbrain/`)
- `secrets` (config management)
Notable non-members: `doctor` is NOT refused on a thin client — it reroutes
to an outbound-HTTP probe set (`src/core/doctor-remote.ts`); `bootstrap` and
`hook` are engine-free and work on any install shape.
For refused commands, the agent cannot route through HTTP MCP. The path is to run
For these, the agent cannot route through HTTP MCP. The only path is to run
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
DLQ / audit trail all come for free.
@@ -119,11 +114,12 @@ full validation rules and error catalog.
### Why this is preferred over writing secrets into `env:` per-job
- Passing `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` per job would
land the URL plaintext in `minion_jobs.data` and the shell-audit JSONL —
visible to anyone with brain-DB read access (or a brain dump, or a shared
brain via mounts). Pre-enqueue validation rejects it; the error message
names `inherit: ["database_url"]` as the replacement.
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
validation. The error message names `inherit: ["database_url"]` as the
replacement.
### Worker setup (one-time, per host)
@@ -150,11 +146,11 @@ proxy for worker env.
| `get_page` / `list_pages` | HTTP MCP | Same. |
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
| `sync` / `embed` / `extract` | Shell job + `inherit:` | Thin-client refused; needs local engine + FS. |
| `dream` | Shell job + `inherit:` | Thin-client refused; synthesis runs on the host. |
| `doctor` | Run directly (any install) | Not refused: thin clients get the remote probe set. |
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
| `init` / `config` | One-time host setup | Operator action, not agent action. |
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
## Recommended patterns
@@ -163,26 +159,22 @@ proxy for worker env.
If a brain DB ever traverses a trust boundary, secrets stay out.
- **Free-form names.** `inherit:` accepts any snake_case config-key on your
worker — `database_url`, `anthropic_api_key`, `openai_api_key`,
`openrouter_api_key`, `voyage_api_key`, `groq_api_key`,
`zeroentropy_api_key`, or any custom
`voyage_api_key`, `groq_api_key`, `zeroentropy_api_key`, or any custom
field you stuff into `~/.gbrain/config.json`. The agent picks what it
needs.
- **`env:` still works** for non-secret values, or for cases where you
WANT the value in the row (e.g. an opaque correlation token your audit
flow needs to read back later). The validator doesn't second-guess you.
- **Never try to route a refused command through a thin client.** The CLI
refuses it at dispatch with a hint. Use shell-job + `inherit:` (for
secrets) or `env:` (for non-secrets) on the host instead.
- **Push-based context.** Beyond request/response ops, MCP clients can
receive volunteered context via the `volunteer_context` op — see
[push-context.md](./push-context.md).
- **Never try to route a `localOnly` op through thin-client MCP.** It will
fail with `localOnly op refused in thin-client mode`. Use shell-job +
`inherit:` (for secrets) or `env:` (for non-secrets).
## Migration: from `env:`-passed secrets
## Migration: from pre-v0.36.5.0
If your agent submits shell jobs that pass secrets via `env:`:
```jsonc
// Rejected at submit: the URL would persist in minion_jobs.data plaintext.
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
@@ -193,7 +185,7 @@ If your agent submits shell jobs that pass secrets via `env:`:
Switch to (recommended):
```jsonc
// Name in row, value resolved at child-spawn from worker config.
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
{
"cmd": "gbrain sync --skip-failed",
"cwd": "/data/gbrain",
-89
View File
@@ -1,89 +0,0 @@
# Ambient recall — placing retrieval at session boundaries
Long-lived agent harnesses (your OpenClaw, Hermes, Codex, Claude Code) get the
most value from the brain not on every message, but at the moments where a fresh
question rarely fires on its own: **session start, right after compaction, and
on heartbeats.** This guide is the Pareto frontier of where to place each verb.
The bottleneck for a long-lived agent is not retrieval quality — the corpus
answers well when asked. It is **placement**: the misses come from moments when
no question fires. Two frozen verbs close that gap with 2-3 deterministic calls
per session instead of per-message overhead.
## The frontier — which verb goes where
| Moment | Call | Why | Cost |
|---|---|---|---|
| Any entity-bearing message | `entity(name)` | Zero-LLM, p99 < 100ms. Safe to run synchronously almost anywhere. | negligible |
| **Session start** | `context_pack(entities, budget_tokens)` | Warm the thread's 1-3 standing entities before the first message. | zero-LLM, sub-second |
| **After compaction** | `context_pack(entities, budget_tokens)` | Rehydrate the verbatim detail the summary dropped. | zero-LLM, sub-second |
| **Heartbeat / periodic wake** | `delta(session_id, budget_tokens)` | "What changed since my last wake" in O(changes), deduped. | zero-LLM, sub-second |
| Explicit memory question | `recall(query \| entity, budget_tokens)` | The budget-packed read for "what do we know that we SAVED about X". | sub-second (+1 embedding if `query`) |
| Answer needs cross-page reasoning | `synthesize(question)` | LLM-backed. **Never** on a hot or ambient path. | seconds-to-minutes, $$ |
Observed shape: per-message retrieval beyond `entity` cards adds latency faster
than insight; session-start packs and post-compaction rehydration are nearly
pure win. See the per-verb latency table in
[`docs/protocol/MEMORY_VERBS_v1.md`](../protocol/MEMORY_VERBS_v1.md#latency-classes-per-verb).
## Two integration surfaces
- **Pull (works everywhere, including Codex + Postgres/Supabase):** the harness
calls `context_pack` / `delta` over MCP (they are on `--surface verbs`) or the
CLI (`gbrain context-pack`, `gbrain delta`) at the boundary and injects the
returned `text` (or renders the structured arms). This is the portable path —
no hooks required. It is the primary path for Codex (which has no hooks) and
for Postgres brains (which have no local IPC socket).
- **Push (PGLite + Claude Code):** the bundled hook framework fires
automatically at `SessionStart` (injects a warm pack — including the
post-compaction re-entry, `source=compact`) and `PreCompact` (banks the
window's standing entities for that rehydration pack). Heartbeat deltas are
the PULL path — there is deliberately no push heartbeat; call `delta` per
the HEARTBEAT cadence table.
## Visibility — world-only by default
A pack is injected into an agent context window that may be logged or synced to a
cloud model, so **every arm is world-visibility by default.** To pull private
facts in, pass `include_private` — and it is honored ONLY for trusted-local
callers (`remote === false`, i.e. the CLI/hook path). A remote MCP caller never
widens, even if it asks (fail-closed). When it does widen, all arms widen
together, so a pack is never a mix of private facts beside world-stripped
synopses.
## Budgets
Every pack/delta call takes `budget_tokens`. The server packs highest-priority
arms first (cards → facts for packs; pages → facts for deltas) and reports
`budget_used` + `dropped_count`; the injectable `text` field is rendered from
the packed sets, so it honors the same budget the structured arrays report. It
never trims client-side — you always know what was left out (`dropped_count`,
and `has_more` on deltas). Pick a budget to fit the boundary: a session-start
pack can afford more than a heartbeat delta.
## Heartbeat cursor + dedup
Pass a stable `session_id` to `delta` and the brain keeps a per-session cursor:
the first wake establishes it, each wake advances it. Dedup is **cursor-based**
— a delivered page reappears only if it changes again after delivery (and then
it should). Delivery is **at-least-once**: pages arrive oldest-first, and when
a budget or the fetch limit drops some, the response sets `has_more: true` and
the cursor advances only to the newest *delivered* page, so the tail surfaces
on the next wake — nothing is silently lost. With no `session_id` you can still
pass an explicit `since` for a stateless delta. The cursor is namespaced per
caller (`(source_id, client_id, session_id)`; authenticated remotes use their
client id, auth-less remotes share a `remote` namespace, and `local` is
reserved for the trusted CLI/hook lane), so a remote harness can never read or
advance the local lane's cursor. Idle session cursors are garbage-collected
after **7 days** — a wake on an expired session re-establishes the cursor at
now and returns an empty delta, so a harness returning from a long sleep
should run one stateless `since`-based catch-up first.
## Example — a cold session start (pull)
```bash
gbrain context-pack --entities "acme-example,alice-example" --budget-tokens 4000
```
Returns entity cards + open threads + hot facts, budget-packed, world-only. Inject
the `text` field into the model's context before the first user message.
-247
View File
@@ -1,247 +0,0 @@
# GBrain Bootstrap — your harness as your agent
`gbrain bootstrap` turns a Claude Code or Codex session into a persistent personal
agent: identity files rendered from your own answers, a local PGLite brain,
per-turn context, session-triggered schedules, and a private GitHub repo as the
agent's durable, portable body. This guide is the full contract — what gets
installed, what runs when, what it can and cannot do, and how to undo all of it.
Normative design docs: [AGENT_BOOTSTRAP_DESIGN.md](../designs/AGENT_BOOTSTRAP_DESIGN.md)
(scope) and [AGENT_BOOTSTRAP_PLAN.md](../designs/AGENT_BOOTSTRAP_PLAN.md)
(implementation). The paste block lives in the README; the runbook your agent
follows is `BOOTSTRAP_FOR_AGENTS.md` at the repo root, fetched at the
`latest-stable` ref.
## What gets installed, exactly
| Piece | Where | Runs when |
|---|---|---|
| Identity files (SOUL/USER/MEMORY/AGENTS/CLAUDE/HEARTBEAT/ACCESS_POLICY/GITHUB) | your workspace folder | loaded at session start |
| `agent.json` manifest + `brain/`, `memory/`, `skills/`, `state/` | workspace | — |
| Local brain (PGLite) | `~/.gbrain/` (never in the repo) | while a session's MCP serve is open |
| MCP registration (`gbrain serve`) | Claude Code: project scope by default; Codex: user-global (no scope flag) | spawned by your harness per session |
| Hooks (Claude Code, ON by default) | local installs: `.claude/settings.local.json` (gitignored); cloud sandboxes: the COMMITTED `.claude/settings.json` (PATH-resolved, fail-open commands) | each prompt; fail-open; `--no-hooks` opts out at install, `GBRAIN_HOOKS=0` disables at runtime |
| Per-turn persistence | Stop hook → debounced, detached scan-gated push (per workspace; 5 min default, every turn in cloud sandboxes) | after each assistant turn; `GBRAIN_STOP_PUSH=0` disables; `GBRAIN_STOP_PUSH_DEBOUNCE_MIN` / config `hooks.stop_push_debounce_min` tune it |
| Session persistence | SessionEnd hook → scan-gated commit+push | at session end (note: the harness never fires SessionEnd on `/exit` — the per-turn push is what covers that) |
| Push-failure visibility | next turn's context + a user-visible notice; re-announces every 30 min while failing | whenever a background push fails |
| Optional background job (consent-gated) | git post-commit auto-push + launchd/cron 30-min pull (pull job skipped honestly on hosts without a scheduler) | while logged in |
| Private GitHub repo | your account, created by `bootstrap repo` (or an empty repo you made yourself, adopted) | privacy verified via API |
| Machine receipt | `~/.gbrain/bootstrap/receipt.json` | uninstall is keyed to it |
**What does NOT run:** anything while the harness is closed. Session-triggered
schedules fire at turn/session boundaries only. True 24/7 operation is what a
hosted brain provides — this is the honest desktop contract.
## Cloud sandboxes (claude.ai/code and similar)
Cloud sessions run in a reclaimed-after-inactivity VM behind a
credential-injecting egress proxy. `gbrain bootstrap status --json` reports
`execution_environment: "cloud-sandbox"` there, and the install adapts:
- **Hooks live in the committed `.claude/settings.json`** with PATH-resolved,
fail-open commands (no machine paths). The gitignored local settings file
never survives into the next session's fresh clone, and hook config is
snapshotted at session start — so hooks written mid-session go live on the
NEXT session. Commit and push the file.
- **The per-turn push runs every turn** (debounce 0) — a reclaimed VM's tail
loss is permanent, so each turn banks to the private repo.
- **Repo-privacy verification falls back to pure git protocol** when the proxy
blocks the GitHub API (GraphQL is always pinned there; REST reaches only
session-attached repos). Confirmed-public origins still always refuse.
- **Repo creation is refused in cloud** with the flow that works: create the
private repo from a normal machine or github.com, open the cloud session ON
that repo, run `gbrain bootstrap attach`.
- **The gbrain binary installs via the environment setup script** — print it
with `gbrain bootstrap cloud-setup-script` and paste it into the environment
config (npm-based; bun's package fetching is proxy-incompatible there).
- **No scheduler exists** — the consent-gated pull job is skipped with an
honest message; event-driven pushes cover persistence.
Escape hatch for self-hosted git you trust (every use warns loudly):
the CLI flag on `sources push`, `GBRAIN_ALLOW_UNVERIFIED_REMOTE=1`, or
`gbrain config set push.allow_unverified_remote true` (file-plane — the only
form that reaches detached hook children inside a sandbox).
## Bring your own repo (create-repo-first)
By default bootstrap creates the private GitHub repo for you. If you prefer to own
that step — pick the name/org-under-your-account, or just work the familiar way —
create a new **empty** private repo **under your own GitHub account** (no
README/.gitignore/license), clone it, open the clone in your harness, and run the
bootstrap block. `gbrain bootstrap repo` detects the empty repo you created and
**adopts** it: it verifies the repo is private, sets a repo-local git identity, and
pushes your workspace. Two constraints, both enforced with a clear message rather
than a silent failure:
- **Empty.** A repo that already has commits (a README, a license, an existing
project) is refused — create it empty, or run `gbrain bootstrap attach` if it is
an existing agent workspace. (A repo already carrying *this* workspace's history,
e.g. from an interrupted run, is recognized as yours and resumed.)
- **Personal account.** The repo must be owned by your authenticated GitHub user.
Org-owned repos are refused today; create one under your own account, or let
bootstrap make it.
Until the repo phase verifies the repo, the per-turn/session-end push stays
deferred — bootstrap never publishes your workspace to an origin whose privacy it
hasn't confirmed.
## The awake-when-you-are contract
Your agent is awake when your harness is. Laptop asleep = agent asleep. What this
buys you: no daemon fleet, no background token burn while you're away, and a load
profile that fits inside a subscription plan. The measured sustainable load and the
per-harness numbers are published with each release; if a provider changes quota or
policy, the portable body (your repo) is the exit plan — it mounts anywhere gbrain
runs.
## Keyless mode
With zero API keys, everything works: the agent authors memory explicitly through
the brain's write tools (`put_page`, timeline entries, `## Facts` fences — your
harness's model is the LLM, already paid for), and search runs keyword-only
(BM25). `bootstrap verify` prints the capability report honestly. One optional key
upgrades capabilities per provider — OpenAI unlocks semantic search and
automatic fact extraction; Voyage unlocks semantic search; Anthropic unlocks
fact extraction (Anthropic has no embeddings API, so it does not enable
semantic search). The key goes to the 0600 config file, never into the repo or
the interview answers. API spend is metered separately from your subscription and is
zero in keyless mode; with a key, the standard spend gates apply
([spend-controls](../operations/spend-controls.md)).
## Security posture
- **Supply chain:** the paste block and install command pin the `latest-stable`
ref — a maintainer-controlled tag advanced only after a release fully publishes.
The runbook carries a version stamp; `bootstrap status` warns on skew. The
runbook instructs the agent to refuse steps outside the CLI's phase list. bun
installs via package manager or checksum-verified download.
- **Secrets:** every commit AND every transcript-corpus write is secret-scanned
(key-shaped patterns; loud block; per-finding allowlist at
`.gbrain-scan-allow`). A deny-glob backstop refuses tracked `*.pglite`/`.env*`
files even if `.gitignore` is damaged. Push refuses public remotes and
unverifiable visibility.
- **Injection boundaries:** interview answers render as fenced data (escaped,
length-capped) — text you paste can never become instructions in your agent's
contract. Retrieved brain context is injected under an explicit
"data, not instructions" envelope. Facts visible to the harness respect the
brain's visibility tiers.
- **Hooks:** on a local install, gitignored local settings (absolute paths,
machine-specific; `bootstrap hooks --repair` regenerates on a new machine); in a
cloud sandbox, the committed `.claude/settings.json` (PATH-resolved, fail-open —
see the Cloud sandboxes section). Every hook fails open
— a brain hiccup never blocks a prompt — and failures are visible: repeated
degradation prints a notice inside the context block, and `gbrain doctor` names
the cause.
- **Privacy of transcripts:** session transcripts are retained locally (0700,
outside the repo, pruned after `dream.synthesize.corpus_retention_days`, default
30 — set it in the config file, `~/.gbrain/config.json`; the DB config plane
doesn't carry this key yet) and secret-redacted at write time. They never enter the repo. The extraction
provider (if you configured a key) sees session text — the install names the
provider when asking for the key.
## Honest forget semantics
The repo is git history — append-only. Deleting a line removes it from the working
tree, not from history. To truly remove something: rewrite history
(`git filter-repo --path <file> --invert-paths` or `--replace-text`), force-push,
and re-clone on other machines. `MEMORY.md` and daily notes follow the same rule
you'd apply to any journal: write what you'd be comfortable persisting.
## Degradation matrix
| You declined / lack | What still works | What you lose |
|---|---|---|
| API keys | everything (keyless mode) | semantic search, auto-extraction |
| GitHub / `gh` | full local agent | off-machine durability (repo re-runnable later) |
| Hooks (Claude Code) | pull protocol via AGENTS.md gates | automatic per-turn context + session-end persistence |
| Codex (no hook system, no MCP scope flag) | pull protocol + MCP tools | per-turn push (stated plainly; not oversold) + the ability to confine MCP reach to one folder (`codex mcp add` is always user-global) |
| Second simultaneous session | first session unaffected | second session's brain tools fail politely (one live serve per brain — v1 contract) |
## Multi-device
Clone your agent repo on machine two and run `gbrain bootstrap attach` — it
validates the manifest, wires this machine (source registration, hooks repair,
MCP), and verifies. The brain database is derived state, rebuilt from `brain/` +
re-ingestion; hot facts extracted only on machine one arrive via the repo's pages
and fences. Simultaneous editing from two machines is ordinary git conflict
territory — `sources push` pulls divergence-safely (commit first, rebase pull,
loud on conflicts).
## Uninstall
`gbrain bootstrap uninstall` removes exactly what this machine's install receipt
records: hook wiring, MCP registrations (surgically — foreign servers and hooks
survive), and bootstrap-created state. Your repo is never touched — the body
remains yours. The brain database is KEPT by default; `--delete-brain` is offered
only when bootstrap created the brain, offers a facts export first, and enumerates
what it is about to remove. It refuses to run while a session's serve is live.
## If something seems broken
One command: `gbrain doctor`. It covers hook health, push staleness, serve/lock
collisions, schema state, and prints fixes. `gbrain bootstrap status --json` emits
a support blob (versions, harness, last verify/push, hook failure rate) your agent
can relay verbatim when you report a problem.
## Real-agent e2e
Most bootstrap tests drive the dispatcher with PATH-shimmed `claude`/`codex`
recorders — fast, hermetic, no API cost. Two additional "door" tests drive the
ACTUAL binaries end to end so we catch real-world drift (a `codex mcp add` flag
that changed shape, a harness that stopped calling our MCP server):
- `test/e2e/bootstrap-real-claude.serial.test.ts` — real `claude -p` over MCP.
- `test/e2e/bootstrap-real-codex.serial.test.ts` — real `codex exec`. It runs the
keyless-`init` → interview → render → `gbrain bootstrap hooks --harness codex`
path (executing the real `codex mcp add` into a hermetic `~/.codex/config.toml`),
asserts the rendered `AGENTS.md` carries the Gate-3 brain-first pull protocol
(Codex has no hook system, so the pull protocol is its per-turn seam), then
spends one live `codex exec` turn to prove real codex → gbrain MCP → brain →
a seeded, brain-only fact (falling back to a shell `gbrain query` if headless
stdio-MCP is unavailable).
These pay real API cost and take 30s2min per turn, so they are NOT in the PR
shard. Everything is hermetic (temp `HOME` / `CODEX_HOME` / `CLAUDE_CONFIG_DIR` /
`GBRAIN_HOME` per test — the operator's real `~/.claude`, `~/.gbrain`, `~/.codex`
are never touched; auth is copied read-only). Each file self-SKIPS via
`describe.skipIf` when its binary or auth is absent, so on a machine without the
tool it is a clean no-op that never fails. CI wires them into the `real-agent-e2e`
job in `.github/workflows/heavy-tests.yml` (nightly + the `real-agent-e2e` /
`heavy-tests` label); on a stock runner they self-skip. To actually exercise the
binaries you need a runner with authed `claude`/`codex` and the provider creds
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`) exported.
Run locally (where both are installed + authed):
```bash
bun test test/e2e/bootstrap-real-codex.serial.test.ts
```
## DX exploration harness (developer instrument, not a test)
The door tests prove the install WORKS; they say nothing about how it FEELS.
`test/helpers/tty-harness.ts` spawns any CLI (gbrain, `claude`, `codex`) under a
real pseudo-terminal (Bun's `terminal:` spawn option) and records every output
burst with a millisecond timestamp, so unnecessary pauses become a measurable
artifact (`computeStalls``stalls.md`) instead of a vibe. Same hermetic env as
`agent-harness.ts`; pure helpers are unit-tested in `test/tty-harness.test.ts`
(zero subprocesses, PTY smokes self-skip where `terminal:` is unavailable).
`scripts/dx-explore.ts` drives it to capture the fresh-user funnel as timestamped
transcripts under `.context/dx-runs/` (gitignored — nothing asserts, no CI):
```bash
bun run scripts/dx-explore.ts help # comprehension surfaces (no keys)
bun run scripts/dx-explore.ts init [--keyless] # interactive init, naive-user autopilot
bun run scripts/dx-explore.ts claude-install # REAL claude running the paste-in bootstrap
bun run scripts/dx-explore.ts codex-install # REAL codex, same
bun run scripts/dx-explore.ts drive -- gbrain init # manual: steer a live TUI via a file channel
```
`drive` mode is how an agent in a Conductor workspace explores a live TUI across
separate tool calls: `cat <dir>/session/screen.txt` to watch, append
`{"line":"..."}` / `{"key":"Down"}` / `{"stop":true}` to `<dir>/session/input.jsonl`
to steer. Each run writes `meta.json`, `visible.txt`, `frames.jsonl`, and
`stalls.md`. `--keyless` strips provider keys so the true no-key first-touch path
is exercised (a Conductor session's ambient `ANTHROPIC_API_KEY` would otherwise
leak in). Install scenarios pay real API cost — launch them as background tasks.
+5 -8
View File
@@ -27,7 +27,6 @@ READ: check brain FIRST (before responding)
│ → gbrain search "{entity name}"
│ → gbrain get {slug} (if you know it)
│ → gbrain query "what do we know about {topic}"
│ → full protocol: brain-first-lookup.md
RESPOND with brain context (every answer is better with context)
@@ -101,14 +100,12 @@ on_message(text):
Write immediately after the conversation, while the context is fresh.
3. **Sync after every write batch.** Without sync, the brain search index is
stale. The next query won't find what you just wrote. On installs set up
via `gbrain bootstrap`, per-turn context injection and session-end
persistence hooks automate parts of this loop — see
[bootstrap.md](bootstrap.md) and [push-context.md](push-context.md).
stale. The next query won't find what you just wrote.
4. **External APIs are fallback, not primary.** `gbrain search` before any
web or enrichment API. The full brain-before-external protocol (and why)
lives in [brain-first-lookup.md](brain-first-lookup.md).
4. **External APIs are fallback, not primary.** `gbrain search` before
Brave Search. `gbrain get` before Crustdata. The brain has relationship
history, your own assessments, meeting transcripts, cross-references.
No external API can provide that.
## How to Verify It Works
+9 -12
View File
@@ -51,25 +51,22 @@ The brain has context no external API can provide:
- Timeline (what changed recently, what's trending)
A LinkedIn scrape gives you their job title. The brain gives you: "co-founded
widget-co, you had coffee with her 3 times, last discussed the payments
infrastructure thesis, she's interested in your take on AI agents."
Brex, you had coffee with him 3 times, last discussed the payments infrastructure
thesis, he's interested in your take on AI agents."
## Tricky Spots
1. **Try keyword first, then hybrid.** Keyword search works without embeddings
(day one — and it's ALL you get in keyless mode, see
[bootstrap.md](bootstrap.md)). Hybrid search needs embeddings but finds
semantic matches. Try both in sequence.
(day one). Hybrid search needs embeddings but finds semantic matches. Try
both in sequence.
2. **Fuzzy slug matching is opt-in.** Pass `--fuzzy` (the `fuzzy: true` param
on `get_page`) and a near-miss slug resolves to the unique candidate, or
returns an `ambiguous_slug` error listing the candidates. WITHOUT the flag
a miss just throws `page_not_found` (with a hint to retry with
`fuzzy: true`). Use it for name variants ("Alice" → "alice-example").
2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact
slug doesn't exist, it suggests alternatives. Use this for name variants
("Pedro" → "pedro-franceschi").
3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?"
should check the brain first. The brain might have it, and a keyword
lookup is fast enough to be effectively free.
should check the brain first. The brain might have it, and the lookup adds
no latency (< 100ms for keyword search).
4. **Load compiled truth + recent timeline.** The compiled truth gives you the
state of play in 30 seconds. The timeline gives you what changed recently.
+4 -5
View File
@@ -17,8 +17,8 @@ on new_information(info):
# This is world knowledge -- facts about entities external to the agent
gbrain put <slug> --content "..."
# Examples:
# "alice-example is CEO of widget-co" -> gbrain (person page)
# "widget-co raised Series D at $12B" -> gbrain (company page)
# "Pedro is CEO of Brex" -> gbrain (person page)
# "Brex raised Series D at $12B" -> gbrain (company page)
# "Tuesday's meeting covered Q2" -> gbrain (meeting page)
# "The meatsuit maintenance tax" -> gbrain (originals page)
@@ -57,16 +57,15 @@ on user_asks(question):
## Tricky Spots
1. **Don't store people in agent memory.** "alice-example prefers email over Slack" feels like a preference, but it's a fact about Alice -- it goes in GBrain on her page. Agent memory is for the agent's own operational state, not facts about people in the world.
1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world.
2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration.
3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory.
4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything.
On installs set up via `gbrain bootstrap`, "agent memory" has a concrete file form: MEMORY.md and the other identity files in the agent repo (see [bootstrap.md](bootstrap.md)). The routing rule is unchanged -- those files hold operational state and identity, not world knowledge.
5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session.
## How to Verify
1. Ask the agent "Who is alice-example?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state.
3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers.
4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory).
+7 -17
View File
@@ -44,12 +44,12 @@ Sharp technical leader. Under-appreciated internally. Watch for signs of burnout
Ascending. Likely CTO track if the migration succeeds.
## Relationship
Met through alice-example. Had coffee 3x. Last: discussed API architecture thesis.
Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis.
## Contact
sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
<!-- timeline -->
---
## Timeline
@@ -58,7 +58,7 @@ sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
[Source: Meeting notes, 2026-04-07 2:00 PM PT]
- **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops.
[Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT]
- **2026-03-15** | First meeting. Intro from alice-example. Strong technical background.
- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background.
[Source: User, direct conversation, 2026-03-15 3:00 PM PT]
```
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain timeline-add {slug} {
gbrain add_timeline_entry {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
@@ -113,19 +113,9 @@ support that claim.
truth chunks with higher relevance than timeline chunks. This means the freshest
synthesis surfaces first in search results.
4. **The timeline sentinel matters — and a bare `---` is NOT one.** GBrain
splits compiled_truth from timeline at the first recognized sentinel, in
order of precedence:
1. `<!-- timeline -->` — preferred; unambiguous, and what GBrain itself
emits when it writes a page.
2. `--- timeline ---` — decorated separator.
3. `---` ONLY when the next non-empty line is `## Timeline` or
`## History` (backward-compat for older gbrain-written files).
A plain `---` line anywhere else is a markdown horizontal rule, not a
separator. Author new pages with `<!-- timeline -->` (as in the example
above); everything above it is compiled truth, everything below is
timeline.
4. **The --- separator matters.** GBrain uses the first standalone `---` after
frontmatter to split compiled_truth from timeline. Everything above is compiled
truth, everything below is timeline.
5. **Don't skip the Assessment section.** The assessment is the value. "Strong
technical leader" is something no API can provide. It's YOUR read on this
+10 -16
View File
@@ -8,12 +8,6 @@ Without this: media links are bookmarks that decay -- you remember watching a vi
## Implementation
gbrain's own media surfaces complement this pattern: the bundled
`media-ingest` skill (`skills/media-ingest/`) ships the ingestion workflow,
and `gbrain files` handles binary/file upload for attachments that should
live alongside pages. For meeting recordings specifically, see
[meeting-ingestion.md](meeting-ingestion.md).
```
on user_shares_media(url_or_file):
@@ -52,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -86,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -115,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -125,7 +119,7 @@ on user_shares_media(url_or_file):
## Tricky Spots
1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it.
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "The speaker made a specific claim about compute scaling that contradicts what another researcher said in their NeurIPS talk -- see media/youtube/a-researcher-neurips-2025" is useful. The analysis connects the new media to the existing brain.
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain.
3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page.
4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry.
5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power.
@@ -133,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
+16 -43
View File
@@ -26,27 +26,6 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
| Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) |
| Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below |
### Prefer gbrain's native schedulers where they fit
System cron is the lowest common denominator, but gbrain ships its own
scheduling surfaces — reach for these first:
- **`gbrain dream`** — the shipped nightly maintenance cycle (lint,
backlinks, extract, sync, embed, synthesize). Schedule THIS instead of
hand-rolling the dream cycle below.
- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry,
backoff, and an audit trail. See the `minion-orchestrator` skill.
- **`gbrain autopilot`** — the long-lived background daemon that runs cycles
on its own cadence.
- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to
manage its harness's scheduler.
- **Bootstrap session-triggered schedules**`gbrain bootstrap` installs
HEARTBEAT.md-driven schedules that fire on session activity; see
[bootstrap.md](bootstrap.md).
For scheduling `sync` + `embed --stale` specifically, the home doc is
[live-sync.md](live-sync.md).
## Implementation: Setting Up Cron Jobs
```bash
@@ -65,24 +44,24 @@ For scheduling `sync` + `embed --stale` specifically, the home doc is
# Brain health — weekly Mondays at 6 AM
0 6 * * 1 gbrain doctor --json >> /tmp/gbrain-health.log 2>&1 && gbrain embed --stale
# Autopilot health gate — daily at 7 AM. The exit code is the signal:
# 0 fresh (or nothing installed), 1 needs attention (stale heartbeat,
# never ran, or paused), 2 the daemon took itself out of rotation.
# Status is filesystem-only, so it works even during a DB outage.
0 7 * * * gbrain autopilot --status >> /tmp/gbrain-autopilot-health.log 2>&1 || your-notify "gbrain autopilot needs attention"
# Dream cycle — nightly at 2 AM
0 2 * * * /path/to/dream-cycle.sh
```
### Quiet Hours Gate (MANDATORY)
Every cron job that sends notifications MUST check quiet hours first. The
gate is a small script YOU create (it doesn't ship with gbrain) and call at
the top of every notification-sending cron script; held output goes to a
holding directory that the morning briefing drains. See
[Quiet Hours](quiet-hours.md) for the gate script and the full pattern —
don't copy a snippet from here, that page is the single home.
Every cron job that sends notifications MUST check quiet hours first.
See [Quiet Hours](quiet-hours.md) for the full pattern.
```bash
# In every cron script:
if ! bash scripts/quiet-hours-gate.sh; then
mkdir -p /tmp/cron-held
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
exit 0
fi
# Not quiet hours — send normally
```
### Travel-Aware Timezone Handling
@@ -109,12 +88,6 @@ morning briefing. Zero config change needed.
The most important cron job. Runs while you sleep.
**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle
(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule
it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is
covered. The pseudocode that follows is the harness-side variant for agents
that also do LLM-driven entity sweeps and memory consolidation on top.
### What It Does
```
@@ -177,12 +150,12 @@ echo "Dream cycle starting at $(date)"
# Phase 1: Entity sweep (spawn sub-agent)
# Read today's conversation logs, extract entities, update brain
# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize)
gbrain dream
# Phase 3: Surface anything the cycle flagged
# Phase 2: Citation hygiene
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
# Phase 3: Embed any stale content
gbrain embed --stale
echo "Dream cycle complete at $(date)"
```
+1 -4
View File
@@ -143,7 +143,4 @@ the same pass.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The
[email-to-brain recipe](../../recipes/email-to-brain.md) implements this
collector pattern end-to-end; [cron-schedule.md](cron-schedule.md) covers
scheduling it.*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+1 -4
View File
@@ -53,8 +53,6 @@ Write extracted content to `brain/diligence/{company}/{doc-name}.md`:
**Step 5: Save Raw Files.**
Copy original PDFs/files to `brain/diligence/{company}/.raw/`
Preserve originals for reference. The diarized version is for search.
This is safe by design: `.raw/` directories are excluded from sync, so the
originals never enter the search index — only your extracted markdown does.
**Step 6: Create or Update index.md.**
Every diligence directory needs an `index.md`:
@@ -105,8 +103,7 @@ cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} in
**Step 9: Publish (if asked).**
When the user wants a shareable brief, create a password-protected
published version via the `publish` skill (`skills/publish/`). Strip
internal notes and raw assessment language.
published version. Strip internal notes and raw assessment language.
### Quality Bar
-175
View File
@@ -1,175 +0,0 @@
# Embedding migration — moving a brain to another embedding provider
`gbrain migrate embeddings` re-embeds an entire brain onto a different
embedding provider/model, safely and resumably. It is the forward path off a
sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
2026-09-04 and is the shipped default for brains that never picked a model) —
but it is provider-agnostic: any configured `provider:model` works as a
target.
Also reachable as `gbrain retrieval-upgrade` — the alias that `gbrain doctor`
repair hints and the README point at.
## Quick start
```bash
# Preview the work + cost. Changes nothing.
gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run
# Run it (interactive confirm shows chunk count + $ estimate first).
gbrain migrate embeddings --to openai:text-embedding-3-small
# Non-interactive (cron / scripts): --yes is required, else exit 2.
gbrain migrate embeddings --to voyage:voyage-3-large --yes
```
`--dim <N>` overrides the target width; it defaults to the provider recipe's
declared width and is required for recipes that don't declare one (litellm,
llama-server, and other bring-your-own-model providers).
**Pick `--dim` = your brain's current column width when the target supports
it.** A different width triggers the destructive schema transition (column +
index rebuild across all three dim-pinned tables); the same width skips it
entirely. `gbrain doctor` (check `provider_sunset`, for providers with an
announced shutdown) prints the paste-ready command with your actual width
already filled in — it reads the real `vector(N)` column, not the config
value, which can drift.
## How affected brains find out (provider sunsets)
Two surfaces flag a brain whose embedding model (or reranker) is on a
provider with an announced hosted-API shutdown, such as ZeroEntropy
(2026-09-04):
- **`gbrain doctor`** — the `provider_sunset` check warns on every run until
the brain is off the provider. After the shutdown date it escalates to
`fail` only when embedded vectors actually exist on the dead provider
(retrieval is genuinely down); a zero-vector brain whose config merely
resolves to the dead default stays `warn`, so doctor-as-CI-gate setups
don't start exiting 1 on the date. The reranker side resolves through the
same plane search actually reranks with (the mode bundle +
`search.reranker.*` overrides). The message carries the paste-ready
migration command with the brain's actual `--dim`. Accepted the risk?
`gbrain config set doctor.suppress_provider_sunset true` silences it.
- **`gbrain upgrade`** — a one-shot banner (gated by
`ze_sunset_notice_shown`) with the same two fixes.
Both state the full consequence: after the shutdown, **existing vectors
become unqueryable** — query embedding uses the same endpoint as ingestion —
not just new content.
## What it does, in order
1. **Plan.** Counts every chunk not already in the target embedding space —
including chunks on pages with **no recorded embedding signature**
(pages embedded before the v108 provenance stamp). Prices the re-embed
from the pricing table; unknown providers print "estimate unavailable"
instead of a fabricated number.
2. **Consent gate.** Prints the plan; requires an interactive `y` or `--yes`.
Non-TTY without `--yes` refuses with exit 2 (mirrors the `reindex-code`
gate in [spend-controls](../operations/spend-controls.md)). Unlike the pure
cost gates there, `spend.posture=tokenmax` does **not** bypass this one:
posture waives the spend *ceiling*, and this gate also guards a
destructive schema rebuild. Under `tokenmax` the dollar figure is marked
informational and the confirmation is still asked. `--yes` is the single
scripted bypass.
3. **Live probe.** One tiny embed against the TARGET provider before any
mutation — validates the API key, model id, and dimension support in a
single call. A bad key fails here, with nothing changed.
4. **Env-override gate.** Refuses when `GBRAIN_EMBEDDING_MODEL` /
`GBRAIN_EMBEDDING_DIMENSIONS` would silently defeat the switch at
runtime (the same guard `ze-switch` uses). `--ignore-env-override` for
people running deliberate experiments.
5. **Apply.** When the target width differs from the actual column width,
runs the same atomic schema transition `ze-switch` uses, in one
transaction. It rebuilds **all three dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
`facts.embedding` — at the new width, preserving each column's type
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
three leaves it silently broken: a narrow `query_cache.embedding` makes
every cache write and read fail *by design* (the cache swallows errors so
it can never break search) for a permanent 0% hit rate, and a narrow
`facts.embedding` fails every per-fact embed write. The image/multimodal
columns ARE deliberately untouched — they use separate models whose
dimensions are independent of the text embedding model.
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
(file plane for the runtime gateway, DB plane for doctor), invalidates
every chunk still in the old space — **including NULL-signature pages**
and purges the semantic query cache so stale cached results can't be
served across the swap.
6. **Re-embed.** The standard embed pipeline (`embed --stale --catch-up`)
with per-source single-flight locks, rate-limit backoff, stderr progress,
and optional DB-contention pacing (`--pace[=mode]`).
## What the rebuild deletes
The dimension change **deletes every stored embedding vector** in the brain —
they are in the old model's space and unusable. They are not recoverable:
going back to the previous provider means paying for a second full re-embed.
`content_chunks` vectors are rebuilt by the re-embed pass, the query cache
refills on the next query, and fact embeddings are rewritten on their next
write (or a `gbrain extract` pass).
## Resume after a kill
The NULL-embedding column is the checkpoint. If the run is killed (or some
pages fail to embed), re-run the **same command**: chunks already embedded on
the target are never re-embedded, the schema/config steps no-op, and the run
continues where it stopped. An in-flight marker (`embedding_migration.state`
in DB config) records the target; it is cleared only when the backlog drains
to zero.
One caveat after a HARD kill (SIGKILL, crash, power loss — not Ctrl-C): the
run's per-source single-flight embed lock is left behind, and an immediate
re-run skips the re-embed and reports the migration as paused. The command
says so explicitly (`lock_skipped` in `--json`); the lock expires on its own
after at most 60 minutes, then the same re-run resumes normally.
A page whose chunks straddle two stale batches is embedded correctly but not
stamped by the embed loop (which only stamps all-or-nothing per batch), so the
migration runs one reconcile pass after the drain that stamps every
fully-embedded page. Without it a large brain would report "incomplete" and the
re-run would pay again for those pages. `--batch-size N` tunes the batch
(default 2000).
`--no-embed` applies schema + config + invalidation and stops, so you can run
the (potentially long) re-embed later or in the background:
```bash
gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed
gbrain embed --stale --catch-up --include-null-signature --background
```
## During the migration
While the re-embed runs, semantic search returns degraded (lexical-arm-only)
results for not-yet-re-embedded content. Pick a quiet window for large
brains, or use `--pace` to keep the DB responsive.
## Pages without an embedding signature (#3391)
Pages embedded before provenance stamping have `embedding_signature IS NULL`
and are grandfathered by the routine stale sweep (so an upgrade never
surprise-re-embeds a whole corpus). After a provider swap that grandfather
clause would silently leave those pages in the OLD embedding space — mixed
vector spaces in one index, degrading retrieval with nothing in the logs.
- `gbrain migrate embeddings` always includes them.
- Plain `gbrain embed --stale` warns when a model swap leaves NULL-signature
pages behind, and `gbrain embed --stale --include-null-signature` re-embeds
them.
## Reranker
Migrating embeddings does not touch the reranker. If
`search.reranker.model` points at the outgoing provider, the plan prints a
warning; disable it (`gbrain config set search.reranker.enabled false`) or
point it at another provider.
## Self-hosting instead of migrating
If the outgoing model's weights are available (zembed-1's are Apache-2.0),
serving them locally via `llama-server` / `ollama` / a LiteLLM proxy
preserves your existing vectors — no re-embed at all. Point
`embedding_model` at the local recipe and keep the same dimensions. The
migration command is for when you'd rather move to a hosted provider.
+13 -19
View File
@@ -8,11 +8,6 @@ Without this: brain pages are thin shells with only what the user manually typed
## Implementation
gbrain ships both halves of this: `gbrain enrich` is the batch enrichment
primitive (finds thin pages and enriches at scale), and the `enrich` skill
(`skills/enrich/`) is the agent-driven page-at-a-time workflow. The pipeline
below is the pattern they implement — use it to customize or extend.
```
on enrich(entity, trigger):
# trigger: meeting mention, email thread, social interaction, user request
@@ -54,29 +49,28 @@ on enrich(entity, trigger):
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain call put_raw_data \
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections: use the person-page structure from compiled-truth.md
# (Executive Summary, State, What They Believe, ... Timeline) -- that doc is
# the single home for the section taxonomy. Enrichment can add texture
# sections on top (What Motivates Them, Hobby Horses, Open Threads).
# People page sections (not a LinkedIn profile -- a living portrait):
# Executive Summary, State, What They Believe, What They're Building,
# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline
# Facts are table stakes. TEXTURE is the value.
# Extract texture, not just facts:
@@ -100,10 +94,10 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Compiled Truth](compiled-truth.md) for the person-page section taxonomy, [Spend Controls](../operations/spend-controls.md) for gbrain's own embedding/LLM spend gates.*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+9 -21
View File
@@ -7,10 +7,10 @@ Every inbound message gets scanned for original thinking AND entity mentions so
## What the User Gets
Without this: the agent answers questions but forgets everything. You mention
Alice in a meeting, and next week the agent doesn't know who Alice is.
Pedro in a meeting, and next week the agent doesn't know who Pedro is.
With this: every person, company, and idea mentioned in conversation gets a
brain page. Next time Alice comes up, the agent already has context. The
brain page. Next time Pedro comes up, the agent already has context. The
brain compounds.
## Implementation
@@ -18,11 +18,6 @@ brain compounds.
Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it
to finish before responding. It runs in parallel.
This pattern is harness-side by design, but gbrain ships help on both ends:
the `signal-detector` skill (`skills/signal-detector/`) is the bundled
version of this detection loop, and `gbrain extract` runs gbrain's own
extraction machinery (entities, facts) over already-synced content.
```
on_every_message(message_text, source_context):
@@ -114,7 +109,6 @@ is_notable(entity):
| Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions |
| Hot takes with reasoning | Routine operational messages |
| Metaphors that reveal new angles | Requests without embedded insight |
| Emotional/psychological insights about self or others | |
### Filing Rules
@@ -122,17 +116,11 @@ is_notable(entity):
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| Person mentioned | `brain/people/{slug}.md` |
| Company mentioned | `brain/companies/{slug}.md` |
| Media referenced | `brain/media/{type}/{slug}.md` |
| Article ABOUT the user | `brain/media/writings/{slug}.md` |
This table is the single home for the capture/filing taxonomy. Other guides
([idea-capture](idea-capture.md) especially) link here rather than carrying
their own copy.
### The Iron Law of Back-Linking
@@ -140,21 +128,21 @@ Every entity mention MUST create a back-link FROM the entity page TO the
source. This is not optional.
```
// When message mentions "Alice" and creates a meeting page:
// When message mentions "Pedro" and creates a meeting page:
// 1. Update the meeting page (normal)
brain/meetings/2026-04-10-board-sync.md:
- Alice presented Q1 numbers
- Pedro presented Q1 numbers
// 2. ALSO update Alice's page (back-link)
brain/people/alice-example.md:
// 2. ALSO update Pedro's page (back-link)
brain/people/pedro-franceschi.md:
## Timeline
- **2026-04-10** | Presented Q1 numbers at board sync
[Source: User, board meeting, 2026-04-10]
```
Without back-links, you can't traverse the graph. "Show me everything related
to Alice" only works if Alice's page links back to every mention.
to Pedro" only works if Pedro's page links back to every mention.
## Tricky Spots
@@ -175,7 +163,7 @@ to Alice" only works if Alice's page links back to every mention.
5. **Dedup before creating.** Always `gbrain search` before creating a page.
Variant spellings, nicknames, and company abbreviations cause duplicates.
"Alice Example" and "Alice" might be the same person.
"Pedro Franceschi" and "Pedro" might be the same person.
## How to Verify
@@ -194,7 +182,7 @@ to Alice" only works if Alice's page links back to every mention.
4. **Send a boring message.** Say "ok sounds good." Verify: nothing was
created. The detector should report "No signals detected."
5. **Check for duplicates.** Mention "Alice" then later "Alice Example."
5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi."
Verify: one page, not two.
---
+9 -14
View File
@@ -8,11 +8,6 @@ Without this: the agent triages email mechanically ("you have 12 unread"), preps
## Implementation
Before hand-rolling these: gbrain bundles the morning-briefing half of this
pattern as the `briefing` skill (`skills/briefing/`) and the task-prep half
as `daily-task-prep` (`skills/daily-task-prep/`). Use the workflows below to
extend or customize what those skills already ship.
```
# WORKFLOW 1: Email Triage
on email_batch(emails):
@@ -58,28 +53,28 @@ on upcoming_meeting(meeting):
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
"relevant_deals": gbrain get_links <attendee_slug>,
}
else:
briefing[attendee] = "No brain page -- consider enriching"
# Surface: shared history, what to follow up on, what to watch for
# "Last time you discussed the Series B timeline. alice-example was
# concerned about burn rate. Here's the latest from her company page."
# "Last time you discussed the Series B timeline. Pedro was concerned
# about burn rate. Here's the latest from his company page."
# WORKFLOW 3: Post-Inbox Brain Updates
on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain timeline-add <sender_slug> {date} \
"Email re: {subject}. Key info: {extracted_signal}" \
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain timeline-add <entity_slug> {date} \
"{what_was_said_about_them}" \
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
@@ -98,9 +93,9 @@ on schedule_request(meeting):
1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical.
2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise.
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with alice-example -- last time you discussed the Series B, she was concerned about burn rate."
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate."
4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip.
5. **Scheduling nudges require timeline data.** "You haven't met with charlie-example in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
## How to Verify
+39 -13
View File
@@ -61,13 +61,14 @@ capture_idea(message_text, source_context):
### The Authorship Test
Who authored the idea determines where it files: user-generated ideas,
syntheses, and ghostwritten work go to `brain/originals/`; borrowed world
concepts to `brain/concepts/`; product ideas to `brain/ideas/`; articles
ABOUT the user to `brain/media/writings/`. The full filing-rules table (and
the what-counts-as-original-thinking criteria) lives in
[entity-detection.md](entity-detection.md) — the single home for the capture
taxonomy.
| Signal | Destination |
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
| Article ABOUT user | `brain/media/writings/` |
### Capture Standards
@@ -77,6 +78,21 @@ taxonomy.
"tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase.
The vivid version is the real version.
**What counts as worth capturing:**
- Original observations about how the world works
- Novel connections between disparate things
- Frameworks and mental models
- Pattern recognition moments ("I keep seeing X in every Y")
- Hot takes with reasoning behind them
- Metaphors that reveal new angles
- Emotional/psychological insights about self or others
**What does NOT count:**
- Routine operational messages ("ok", "do it")
- Pure questions without embedded observations
- Echoing back something the agent said
- Acknowledgments and reactions
### The Depth Test
**Could someone unfamiliar with the user read this page and understand not
@@ -121,9 +137,21 @@ Every original MUST link to:
### Notability Filtering
Before creating any entity page, check notability — the full create/skip
criteria live in [entity-detection.md](entity-detection.md#notability-filtering).
The decision rule: if notable AND no page exists, create a FULL page with web
Before creating any entity page, check notability:
**Create a page for:**
- People you know or discuss with specificity
- Companies you're evaluating, working with, or investing in
- Media you mention with personal reaction
- Anyone you've explicitly engaged with
**Don't create pages for:**
- Generic references or passing examples
- Low-engagement accounts who mentioned you once
- Pure metaphors ("like the Roman Empire...")
- One-off encounters with no follow-up
**Decision:** If notable AND no page exists, create a full page with web
search enrichment. No stubs. If you make a page, make it good.
## Tricky Spots
@@ -159,6 +187,4 @@ search enrichment. No stubs. If you make a page, make it good.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The bundled
`idea-ingest` skill (`skills/idea-ingest/`) ships this workflow. See also:
[Entity Detection](entity-detection.md).*
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+11 -54
View File
@@ -21,17 +21,14 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
### The Primitives
@@ -43,18 +40,9 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
- `gbrain sync --repo <path>` -- one-shot incremental sync. Detects changes via
`git diff`, imports only what changed. For small changesets (<= 100 files),
embeddings are generated inline during import — unless the inline cost gate
intervenes: when the estimated embedding spend crosses the configured floor
in a non-interactive session (cron, `--json`), sync auto-defers embeds to a
capped `embed-backfill` job instead of spending silently. Either way the
chunks get embedded; a deferred run just finishes asynchronously. See
[spend controls](../operations/spend-controls.md).
embeddings are generated inline during import.
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
On a keyless brain (installed with `--no-embedding`), a bare stale embed
refuses cleanly — exit 0 with a stderr note — so this chain is safe to
schedule on keyless installs; keyword search keeps working. Explicit embed
requests (a slug, `--slugs`, `--all`) still exit 1 on a keyless brain.
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
(configurable with `--interval N`). Embeds inline for small changesets. Exits
after 5 consecutive failures, so run under a process manager or pair with a
@@ -106,27 +94,15 @@ Triggers sync on push events for instant sync (<5s).
### What Gets Synced
Sync only indexes "syncable" markdown files. These are excluded by design:
- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees
(`node_modules/`, `dist/`, `build/`, `venv/`)
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md`
- Hidden paths (`.git/`, `.raw/`, etc.)
- The `ops/` directory
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`
Everything else is ordinary synced content — including `ops/` (the bundled
daily-task-manager skill files its canonical page under `ops/tasks`).
### Sync is Idempotent — and Resumable
### Sync is Idempotent
Concurrent runs are safe. Two syncs on the same commit no-op because content
hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
Long syncs also survive being killed: progress checkpoints into the database
as files drain, so a killed or aborted run resumes from where it stopped, and
the sync bookmark only advances on true completion. A progress-aware stall
watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts
a run that stops making forward progress and releases the per-source lock so
the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence
and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env
vars — incident-time escape hatches, not everyday knobs.
## Tricky Spots
1. **Always chain sync + embed.** Running `gbrain sync` without
@@ -155,17 +131,7 @@ vars — incident-time escape hatches, not everyday knobs.
history rewrite still hard-blocks even with `--skip-failed`. Run
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
5. **Staleness can't read "fresh" forever.** A source whose content stopped
moving (or whose local clone vanished) used to report fresh indefinitely
off the stored content timestamp. Content-relative staleness now ramps
toward stale once wall-clock time since the last sync passes a ceiling
(default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune — it tracks
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` unless set). The ramp is gradual, so
the warn tier still fires before the fail tier. `gbrain status` source
rows carry `hours_since_last_sync` (raw wall-clock truth) alongside the
threshold-relative `staleness_hours` that drives the fresh/stale class.
6. **Import checkpoints name the import target, not the caller's CWD.**
5. **Import checkpoints name the import target, not the caller's CWD.**
Interrupted `gbrain import <dir>` runs may leave
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
checkpoint `dir` is the absolute, resolved import target captured when
@@ -192,15 +158,6 @@ vars — incident-time escape hatches, not everyday knobs.
`gbrain embed --stale` isn't running after sync, leaving chunks invisible
to vector search.
4. **Gate on the daemon's heartbeat.** If the built-in daemon runs your sync
(`gbrain autopilot --install`), wire your scheduler's health check to
`gbrain autopilot --status`. The exit code is the signal: 0 fresh (or
nothing installed), 1 needs attention (stale heartbeat, never ran, or
paused by a migration), 2 the daemon took itself out of rotation.
`--json` emits the full report, including `heartbeat_age_seconds`. Status
reads only the filesystem — no database connection — so it keeps working
during the exact outages it exists to diagnose.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+9 -20
View File
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain timeline-add <person_slug> {date} \
"Met in '{meeting.title}' on {date}. Key points: ..." \
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain timeline-add <company_slug> {date} \
"Discussed in '{meeting.title}': {what_was_said}" \
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -58,34 +58,23 @@ on new_meeting_transcript(meeting):
# Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings
# Source: Circleback (https://circleback.ai) or any service with
# speaker diarization + API/webhook access
# Automation: the built-in `extract-timeline-from-meetings` Minion job
# automates step 3 (entity timeline propagation) for already-ingested
# meeting pages: gbrain jobs submit extract-timeline-from-meetings --follow
```
## Tricky Spots
1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth.
2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation.
3. **Mentioned people are not just attendees.** If the meeting discussed "Alice's team at widget-co," then Alice's page AND widget-co's page need updates -- even though Alice wasn't in the room.
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Alice pushed back on the burn rate, Charlie didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room.
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always.
6. **`--source` on `timeline-add` is the citation text, not source routing.** Because the op declares its own `source` parameter, the CLI binds `--source` to it. To write into a different registered source, use the `.gbrain-source` dotfile or `GBRAIN_SOURCE` env for routing instead.
## How to Verify
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
## Related
- `skills/meeting-ingestion/SKILL.md` — the bundled, agent-executable skill
for this workflow (the canonical step-by-step home; this guide is the
pattern overview).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+22 -29
View File
@@ -40,10 +40,6 @@ gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
# Optional: cap worker memory in MB (--max-rss). Without the flag the RSS
# watchdog is still on, at a RAM-relative auto-sized cap.
gbrain jobs supervisor --concurrency 4 --max-rss 4096
```
**Exit codes:**
@@ -54,11 +50,9 @@ gbrain jobs supervisor --concurrency 4 --max-rss 4096
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
| 4 | Queue-scoped DB lock lost mid-run (`LOCK_LOST` — exited rather than risk a split-brain) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=4 as "restart me — the DB lock refresh failed"; exit=1 should page
a human.
exit=1 should page a human.
### Lowering scheduling priority (`--nice`)
@@ -239,7 +233,7 @@ use a dedicated queue name like `nightly-enrich` above.
## Upgrading from an older deployment
### From `minion-watchdog.sh`
### From `minion-watchdog.sh` (pre-v0.20)
Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
@@ -276,10 +270,10 @@ Regardless of which deployment path you're upgrading from:
in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** pass `--allow-shell-jobs` to the
supervisor (or keep `GBRAIN_ALLOW_SHELL_JOBS=1` in
`/etc/gbrain.env`). Submitters don't need the flag; only the worker
does.
3. **If you run shell jobs:** from v0.14 onward, pass
`--allow-shell-jobs` to the supervisor (or keep
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
need the flag; only the worker does.
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
should show no unexplained growth in `dead` between pre- and
@@ -289,30 +283,29 @@ Regardless of which deployment path you're upgrading from:
### Supabase connection drops
If Supabase drops the worker's Postgres connection (maintenance,
connection limits, network blip), this now self-heals under the
supervisor: the worker's DB-liveness probe self-exits (`db_dead`) on a
dead pool and the supervisor respawns it with a fresh pool, and the
supervisor also restarts a worker that stops making progress while
claimable work waits. The escalation commands and thresholds live in the
[queue operations runbook](queue-operations-runbook.md) — that's the
canonical home for wedge recovery.
The worker uses a single Postgres connection. If Supabase drops it
(maintenance, connection limits, network blip), lock renewal fails
silently. The stall detector then dead-letters the job after
`max_stalled` misses.
What can still bite: a *brief* blip during a long-running job can make
lock renewal miss, and the stall detector dead-letters the job after
`max_stalled` misses (schema column default 5; lock duration and stall
check interval are both 30 s).
**Current defaults that make this worse:**
**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`,
- `lockDuration: 30000` (30 s) — too short for long jobs during
connection blips.
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags.
These write onto the job row at submit time — which is what
`handleStalled()` reads — so per-job tuning is the real knob.
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today.
### DO NOT pass `maxStalledCount` to `MinionWorker`
It's a no-op. The stall detector reads the row's `max_stalled` column
(set at submit time), not the worker opt in `src/core/minions/worker.ts`.
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children
+6 -12
View File
@@ -1,9 +1,5 @@
# Minions fix — repairing a half-migrated install
> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're
> on any recent release, the canonical fix below (`gbrain apply-migrations
> --yes`) is all you need; the stopgap sections exist for archaeology.
**tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially
set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs
still on `agentTurn`), run:
@@ -38,16 +34,17 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`.
gbrain doctor
```
If the install is half-migrated, you'll see the `minions_migration` check
fail:
If the install is half-migrated, you'll see:
```
[FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes
```
(Missing `~/.gbrain/preferences.json` on a fresh install is a valid
pre-`apply-migrations` state — doctor deliberately does NOT fail on that
alone; the partial-migration record is the canonical half-migration signal.)
or
```
[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes
```
For a machine-readable report (cron-friendly):
@@ -121,9 +118,6 @@ cat ~/.gbrain/preferences.json
cat ~/.gbrain/migrations/completed.jsonl
# 3. Autopilot is supervising a Minions worker child
# (v0.46+: the exit code is the verdict — 0 fresh, 1 needs attention,
# 2 self-disabled — so a nonzero exit here IS the finding, not a
# broken verify step. Under `set -e`, append `|| true` to keep going.)
gbrain autopilot --status
ps aux | grep 'jobs work'
+7 -7
View File
@@ -129,10 +129,11 @@ JSONL records the same. Pre-enqueue validation rejects the submission if the
worker can't resolve the requested key, with a paste-ready
`gbrain config set database_url <value>` hint.
**Why not just write the URL into `env:` directly?** You *can*:
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
wrote things like:
```jsonc
// ❌ Works, but plants the secret in the job row. Prefer inherit:.
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
{
"cmd": "gbrain stats",
"cwd": "/data/gbrain",
@@ -140,21 +141,20 @@ worker can't resolve the requested key, with a paste-ready
}
```
This plants plaintext secrets in `minion_jobs.data` (DB row) and in the
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
or a shared brain via the mounts feature) sees the URL. The validator
doesn't forbid the pattern — it trusts the agent — but **prefer
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
forbid that pattern — the validator trusts the agent — but **prefer
`inherit:`** for any secret you want kept out of the row. Names land in the
row; values resolve at child-spawn from the worker's config.
**Scope:** `inherit:` is **free-form**. Pass any snake_case
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
config-key name and the worker resolves the value from `loadConfig()` at
child-spawn time:
- `inherit: ["database_url"]` → child env `GBRAIN_DATABASE_URL`
- `inherit: ["anthropic_api_key"]` → child env `ANTHROPIC_API_KEY`
- `inherit: ["openai_api_key"]` → child env `OPENAI_API_KEY`
- `inherit: ["openrouter_api_key"]` → child env `OPENROUTER_API_KEY`
- `inherit: ["voyage_api_key"]` → child env `VOYAGE_API_KEY`
- `inherit: ["groq_api_key", "zeroentropy_api_key"]` → both injected
- Or any arbitrary config-key your worker has (`my_custom_field`
-11
View File
@@ -53,23 +53,12 @@ gbrain reindex-search-vector --dry-run # preview: language + row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The stamp survives later schema work: `initSchema()` — including the replay
behind `gbrain init --migrate-only` on every upgrade — applies the schema
template under the configured language, so it re-creates the trigger
functions as they already are instead of reverting them to `english`.
The command recreates both trigger functions under the new language and
backfills every existing `pages` and `content_chunks` row in batches,
streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
No cache purge is needed. The resolved language is part of the query-cache
key, so rows written under the previous language are unreachable after the
switch — searches read the retokenized index immediately instead of being
served pre-switch results for up to `search.cache.ttl_seconds`. Switching
back reaches the original rows rather than rebuilding them.
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
+22 -37
View File
@@ -5,11 +5,6 @@ is a `source`: a logical brain-within-the-brain with its own slug
namespace, its own sync state, and its own federation policy. The rest
of this guide walks the three canonical scenarios.
(Sources are the *within-one-database* axis. If you want to connect a
whole separate database — a team-published brain with its own access
policy — that's the *brain* axis: `gbrain mounts add`. See
`docs/architecture/brains-and-sources.md` for the two-axis topology.)
## The three scenarios
### 1. Unified knowledge recall (wiki + gstack)
@@ -71,14 +66,14 @@ gbrain search "tech layoffs" --source yc-media,garrys-list
### 3. Mixed (wiki federated + sessions isolated)
Your main wiki is federated with a few trusted sources. Your session
transcripts (`gbrain transcripts` ingests them) land in a separate
isolated source so they don't dominate every search result.
transcripts (coming in v0.18) land in a separate isolated source so
they don't dominate every search result.
```bash
# Federated sources
gbrain sources add gstack --path ~/.gstack --federated
# Isolated source for session transcripts
# Isolated source (future v0.18 — sessions use this shape today for ingest)
gbrain sources add sessions --path ~/.claude/sessions --no-federated
```
@@ -96,7 +91,7 @@ first):
6. The seeded `default` source.
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
the `gstack` source. Outside any registered directory with no env/dotfile
set, it writes to the default.
@@ -109,16 +104,14 @@ Every source row stores `config.federated: boolean` in its JSONB config.
| `true` | Source participates in unqualified `gbrain search "X"` results. |
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
The seeded `default` source is `federated=true` so single-source brains
behave as you'd expect — every page appears in search.
The seeded `default` source is `federated=true` so pre-v0.17 brains
behave exactly as before — every page appears in search.
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
## Commands
The most-used subcommands (run `gbrain sources --help` for the full,
always-current reference — it also covers `status`, `current`,
`set-cr-mode`, and the `push`/`pull` durability surface):
Full subcommand reference:
```
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
@@ -126,17 +119,9 @@ gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--
--path must be a git repo (or a subdirectory of one) — see
"The git requirement for --path sources" below. --force
skips that check to register before git-init exists.
gbrain sources add <id> --url <git-url> [--pat-file <p>] [--clone-dir <path>] [--no-harden]
Clone + register a remote repo in one step; auto-hardens
for durability when a PAT is provided (see "Durability" below).
gbrain sources list [--json] List all sources with page counts + federation state.
gbrain sources archive <id> Soft-delete: hide from search, keep data for a TTL
grace window. Prefer this over `remove`.
gbrain sources restore <id> Un-archive. `gbrain sources archived` lists expiries;
`gbrain sources purge` permanently deletes expired archives.
gbrain sources remove <id> [--confirm-destructive] [--dry-run]
Permanently cascade-delete a source (pages, chunks,
timeline). Shows an impact preview first.
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
Cascade-delete a source (pages, chunks, timeline).
gbrain sources rename <id> <new-name>
Change display name only; id is immutable.
gbrain sources default <id> Set the brain-level default.
@@ -203,10 +188,10 @@ citations keep working.
```bash
# Pass --source explicitly
gbrain put topics/ai ... --source wiki
gbrain put-page topics/ai ... --source wiki
# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put plans/multi-repo ...
cd ~/.gstack && gbrain put-page plans/multi-repo ...
# → source auto-resolves to gstack
```
@@ -268,8 +253,8 @@ reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
## Upgrading an existing brain
`gbrain upgrade` runs the needed schema migrations automatically. Your
existing pages all live under `source_id='default'`. Behavior is
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
existing pages all move under `source_id='default'`. Behavior is
unchanged until you add a second source.
To add one:
@@ -281,13 +266,13 @@ cd ~/.gstack && gbrain sources attach gstack && gbrain sync
Two commands. The existing default source is untouched.
## Related features that build on sources
## Not in v0.18.0
- **Session transcript ingest**`gbrain transcripts` (server-private:
raw chat exports stay on the host machine).
- **Per-source retention**`gbrain sources archive` / `archived` /
`purge` (soft-delete with a TTL grace window).
- **One-shot remote bootstrap**`gbrain sources add <id> --url <git-url>`
(clone + register + auto-harden).
- **Access control across brains** — the *brain* axis (`gbrain mounts`);
see `docs/architecture/brains-and-sources.md`.
- Session transcript ingest (`.jsonl`, raised size cap, session
PageType) — v0.18.
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
- ACL enforcement via caller-identity — v0.17.1.
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
release after the core plumbing stabilizes.
All of these build on the `sources` primitive shipped here.
+18 -21
View File
@@ -20,8 +20,8 @@ on every_inbound_message(message):
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain timeline-add <entity_slug> {date} \
"{what_was_said}" \
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
@@ -43,14 +43,13 @@ on information_needed(topic):
# An agent that reaches for the web before checking its own brain
# is wasting money and giving worse answers.
# DISCIPLINE 3: Sync After Every Repo Write (MANDATORY)
on brain_repo_files_changed():
# DISCIPLINE 3: Sync After Every Write (MANDATORY)
on brain_write_complete():
gbrain sync
# `gbrain put` indexes immediately -- pages written through the CLI/MCP
# are searchable the moment the command returns. No sync needed there.
# But files written DIRECTLY to the brain repo (an editor, a script,
# another agent committing markdown) are invisible to search until
# `gbrain sync` imports them. If anything touched repo files, sync.
# Without this, search results are stale.
# The page you just wrote won't appear in gbrain search or gbrain query
# until sync runs. Skipping this means the next lookup misses the
# most recent data.
# DISCIPLINE 4: Daily Heartbeat Check
on daily_schedule("09:00"):
@@ -63,17 +62,15 @@ on daily_schedule("09:00"):
on nightly_schedule("02:00"):
# The dream cycle is the most important discipline.
# The brain COMPOUNDS overnight.
# gbrain ships this as a first-class command: `gbrain dream`.
# The pseudocode below is the shape of the work it does.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list
pages = gbrain list_pages
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
existing_links = gbrain get_links <page.slug>
for mention in mentions:
if mention not in existing_links:
gbrain link <page.slug> <mention_slug> # fix broken graph
gbrain add_link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
@@ -83,7 +80,7 @@ on nightly_schedule("02:00"):
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain timeline <page.slug>
timeline = gbrain get_timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
@@ -106,18 +103,18 @@ on nightly_schedule("02:00"):
## Tricky Spots
1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots.
2. **Skipping Discipline 3 (sync after repo writes) means stale search results.** A file lands in the brain repo, then you search for it -- and get nothing back. The file exists but isn't imported. Always sync after repo-file writes. (`gbrain put` is exempt: it indexes on write.)
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Alice yesterday about the board seat" in passing -- that's a timeline entry on Alice's page, a potential update to her State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Alice Example" returns a LinkedIn profile. The brain returns the full picture including private context.
2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes.
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context.
5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss.
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a markdown file directly into the brain repo (not via `gbrain put`), run `gbrain sync`, then `gbrain search` for it. Confirm it appears in results (verifies the sync discipline). A `gbrain put` page should appear in search immediately, with no sync.
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+3 -3
View File
@@ -47,8 +47,8 @@ on user_message(message):
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
+10 -14
View File
@@ -1,4 +1,4 @@
# Plugin authors guide — subagent definitions
# Plugin authors guide (v0.15)
`gbrain` discovers subagent definitions from outside this repo via
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
@@ -7,11 +7,6 @@ subagents alongside it, drop a plugin directory on that env path.
This guide is for plugin authors. The CLI user doesn't need to read it.
> **Two plugin systems.** This doc covers *subagent definitions*
> (markdown prompts the `subagent` job handler runs). Custom *job
> handlers* (code the Minion worker executes) are a separate system —
> see [plugin-handlers.md](plugin-handlers.md).
## Minimum viable plugin
```
@@ -70,7 +65,7 @@ You control where your plugin lives on disk; `gbrain` doesn't guess.
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
with a warning naming both sources.
**Trust policy.** Plugins ship subagent definitions ONLY:
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
- You **cannot** declare new tools.
- You **cannot** extend the brain tool allow-list.
@@ -81,8 +76,8 @@ with a warning naming both sources.
your plugin gives you a loud startup error, not a silent "tool never
fires" at 3am.
Plugin-declared tools would require a new `plugin_version` contract;
nothing under `gbrain-plugin-v1` opens that up.
v0.16+ may open up plugin-declared tools with a separate contract. Don't
expect it.
## `gbrain.plugin.json`
@@ -90,9 +85,9 @@ nothing under `gbrain-plugin-v1` opens that up.
|------------------|--------|----------|--------------------------------------------------------------------|
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"`. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Informational; appears in load/collision warnings. |
| `description` | string | no | Shown in future `gbrain plugin list`. |
## Subagent definition files
@@ -108,7 +103,8 @@ Recognized frontmatter fields:
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
Unknown frontmatter fields are preserved but ignored by the handler.
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
may consume more of them.
## Caveats that will bite you
@@ -119,8 +115,8 @@ Unknown frontmatter fields are preserved but ignored by the handler.
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
worker runs on a different host than the `gbrain agent logs` caller,
the CLI won't see heartbeats from that worker. Assume worker + CLI
share a filesystem.
the CLI won't see heartbeats from that worker. v0.16 will unify this;
for now assume worker + CLI share a filesystem.
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
invocation. Tools that gate on `remote=true` (file_upload's strict
+7 -26
View File
@@ -1,12 +1,7 @@
# Plugin handlers — registering host-specific Minion handlers
GBrain's Minion worker ships with a full set of built-in handlers,
registered by `registerBuiltinHandlers` in `src/commands/jobs.ts`
that registry is the source of truth. Examples: `sync`, `embed`,
`lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`, `shell`,
`subagent`, `orphans`, `integrity`, plus dream-cycle phases and other
maintenance jobs. Submitting an unknown job name with
`gbrain jobs submit <name> --follow` prints the full registered list.
GBrain's Minion worker ships with seven built-in handlers: `sync`,
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
These cover every background operation the gbrain CLI itself performs.
Host platforms (OpenClaw deployments, future hosts) register their own
@@ -15,11 +10,6 @@ handlers via a plugin bootstrap that imports
code, loaded by the worker, with the same trust model as any other
code in the host's repo.
> **Two plugin systems.** This doc covers *job handlers* (code the Minion
> worker runs). Custom *subagent definitions* (markdown prompts loaded via
> `GBRAIN_PLUGIN_PATH`) are a separate system — see
> [plugin-authors.md](plugin-authors.md).
## Why code, not data
An earlier design draft shipped `~/.claude/gbrain-handlers.json` where
@@ -74,23 +64,14 @@ auto-loads on startup (configurable via a host-provided entry point).
## Handler contract
Every handler receives a `MinionJobContext` (canonical definition:
`src/core/minions/types.ts`). The load-bearing fields:
Every handler receives a `MinionJobContext`:
```ts
interface MinionJobContext {
id: number; // job id
name: string; // job type
data: Record<string, unknown>; // job params (whatever the cron submit passed)
attempts_made: number;
signal: AbortSignal; // fires on timeout, cancel, pause, or lock loss
shutdownSignal: AbortSignal; // fires only on worker SIGTERM/SIGINT
deadlineAtMs: number | null; // wall-clock deadline from timeout_at, if set
updateProgress(progress: unknown): Promise<void>;
updateTokens(tokens: TokenUpdate): Promise<void>;
log(message: string | TranscriptEntry): Promise<void>;
isActive(): Promise<boolean>; // is the job lock still held?
readInbox(): Promise<InboxMessage[]>; // unread messages sent to this job
data: Record<string, unknown>; // job params (whatever the cron submit passed)
job: MinionJob; // full job row (id, queue, attempts, etc.)
signal: AbortSignal; // set to aborted when the worker is shutting down
inbox: MinionInbox; // read messages sent to this job while it runs
}
```
+1 -38
View File
@@ -5,14 +5,13 @@ contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
The push channels share one zero-LLM core (`src/core/context/volunteer.ts`):
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
| `claude-code` / `codex` | `gbrain hook user-prompt` (registered by `gbrain bootstrap`) | per-prompt injection inside a harness; see "Harness hooks" below |
## How it decides
@@ -54,42 +53,6 @@ through a running serve's resolve socket rather than taking the lock. Routing
watch through that same socket is a filed follow-up (TODOS.md). Postgres
brains are unaffected.
## Harness hooks (the prompt-time channel)
`gbrain bootstrap` registers `gbrain hook user-prompt` as a Claude Code
`UserPromptSubmit` hook: every prompt is assembled into a per-turn context
block (reflex pointers + volunteered pages + hot facts) through a running
serve's IPC socket and injected as `additionalContext`. Two properties make
this channel production-grade rather than spammy-and-invisible:
- **Cross-turn dedupe.** The hook reads its OWN previous injections back out
of the session transcript (Claude Code records them as structured
`hook_additional_context` attachments; only gbrain-marked blocks count) and
passes them as prior context — so a page is volunteered once per session,
not once per mention. The dedupe horizon is bounded (the recent transcript
window, byte-capped), so a marathon session can eventually re-volunteer its
oldest injections. The extraction is structural, never substring matching
over raw turn text, so a short slug appearing in a tool payload can't
over-suppress.
- **The feedback loop.** The serve logs each DELIVERED block's volunteered
pages and pointers to `context_volunteer_events` under the hook's channel
(`claude-code` by default; a codex hook registration passes
`--harness codex`). `gbrain volunteer-context --stats` then shows
per-harness precision, and `gbrain doctor`'s `volunteer_channels` check
shows which channels actually fire, with guidance for the two quiet cases:
"hook installed but never registered (restart the session)" and "registered
but quiet". Logging happens at the delivery point only — a block abandoned
before the serve responded is never counted — and because a delivered
response still isn't proof of injection (the hook can trim or drop it
client-side), the doctor check reconciles the counts against the hook's own
heartbeat and cautions when they diverge.
The hook lane rides the PGLite serve's IPC socket: on a Postgres brain or a
thin-client install the hook stays quiet by design (pull-mode retrieval covers
those; extending the lane is a filed follow-up in TODOS.md).
Kill switch: `GBRAIN_HOOKS=0`. Install/uninstall: `docs/guides/bootstrap.md`.
## Config
| Key | Default | What it does |
+13 -19
View File
@@ -1,8 +1,8 @@
# Queue operations runbook
"My queue looks wedged — what do I run?" The commands below are in the order
you probably want them. Born from a production incident where the queue held
for 90+ minutes before the operator noticed.
you probably want them. Shipped with v0.19.1 after a production incident
where the queue held for 90+ minutes before the operator noticed.
## First signal: jobs aren't running
@@ -23,18 +23,16 @@ container health), but its DB connection died (common behind a transaction
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
pile up with **0 active**. Liveness checks all pass; nothing crashes.
This self-heals — you usually won't have to do anything:
As of v0.42.22.0 this self-heals — you usually won't have to do anything:
- **The worker exits on its own dead pool.** Under a supervisor, the worker's
DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the
supervisor respawns it with a fresh pool.
- **The supervisor restarts a worker that stops making progress.** If a queue
has claimable work, **0 live-lock active jobs**, and no completions for 15
minutes across 3 consecutive health checks while the child is alive, the
supervisor restarts it (covers stuck handlers too, not just dead pools).
These thresholds are built in — there are no CLI flags to tune them. A
restart-loop breaker caps wedge restarts at 3 per 30-minute window, then
switches to a one-shot `wedge_restart_loop` alert in the audit log.
minutes while the child is alive, the supervisor restarts it (covers stuck
handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
`--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables).
The signal is loud now — check either:
@@ -47,12 +45,8 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
stale completions). Manual fix if you ever need it:
```bash
# Restart the supervisor with a fresh pool. `start` alone runs in the
# FOREGROUND (blocks); use --detach to get your shell back.
gbrain jobs supervisor stop && gbrain jobs supervisor start --detach --json
# Re-queue any jobs that were dead-lettered during the wedge.
gbrain jobs retry <id>
gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool
gbrain jobs retry <id> # dead-lettered jobs
```
## Triage commands
@@ -107,9 +101,9 @@ claiming. Start one:
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
```
## Related
## Follow-ups tracked for v0.20+
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
exit codes, and per-platform deployment (systemd / Fly / Render).
- [Minions shell jobs](minions-shell-jobs.md) — the `shell` job type's
security model and error table.
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
subcheck both need this).
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
+1 -21
View File
@@ -113,22 +113,6 @@ fi
send_notification "$OUTPUT"
```
### GBrain-native hooks
Two places gbrain already understands quiet hours natively — use these
before rolling your own gate for the same job:
- **Self-upgrade**`auto` mode only applies upgrades during quiet hours,
configured via `gbrain config set self_upgrade.quiet_hours
'{"start":23,"end":8,"tz":"US/Pacific"}'`. See
[upgrades-auto-update.md](upgrades-auto-update.md).
- **Cron prompts** — schedule-driven notification jobs should carry the
gate described in this doc; [cron-schedule.md](cron-schedule.md) covers
the scheduling side.
The shell pattern below is for everything else: your own cron jobs,
collectors, and notification paths that gbrain doesn't gate for you.
### Configurable Hours
Some users want different quiet hours. Store the config:
@@ -156,11 +140,7 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring)
skill reads and clears the held directory. Orphaned held files mean the
pickup integration is broken.
3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a
held message must not be lost across a restart, use a durable held
directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`.
4. **Timezone auto-detection is fragile.** Calendar-based timezone detection
3. **Timezone auto-detection is fragile.** Calendar-based timezone detection
relies on the user having airline/hotel events with location data. If the
user books travel without calendar entries, the system won't detect the
move. Fall back to activity-hour analysis (responding at 3 AM PT = probably
+6 -18
View File
@@ -9,9 +9,8 @@ Separate agent behavior (replaceable) from world knowledge (permanent) into two
Without this: agent config and world knowledge are mixed together. Switch agents
and you lose your knowledge. Switch knowledge tools and you lose your agent setup.
With this: your brain (thousands of files of people, companies, meetings,
ideas) survives any agent swap. Your agent config survives any knowledge
tool swap.
With this: your brain (14,700+ files of people, companies, meetings, ideas)
survives any agent swap. Your agent config survives any knowledge tool swap.
## Implementation
@@ -120,15 +119,7 @@ without losing your agent setup.
notes). The agent repo contains operational config. Different access controls.
**GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep
the search index current. The agent repo is not indexed by default.
**Multi-source nuance.** With multi-source brains
([multi-source-brains.md](multi-source-brains.md)), "the brain repo" means
"each registered source." You CAN deliberately register a non-brain repo
(e.g. `~/.gstack`) as its own isolated or federated source — that's a
conscious registration with its own slug namespace, not a violation of the
boundary. The rule below is about *unregistered, accidental* indexing of
agent config.
the search index current. The agent repo is never indexed by GBrain.
## Tricky Spots
@@ -142,12 +133,9 @@ agent config.
belongs in the brain. Agent configs, skills, cron jobs, and operational
state are replaceable. People, companies, ideas, and meetings are not.
3. **Don't casually index the agent repo.** Running `gbrain sync` against
the agent repo pollutes search results with operational config instead
of world knowledge. (Registering it deliberately as an isolated source
is different — see the multi-source nuance above. Pin the working
directory to the right source with a `.gbrain-source` dotfile via
`gbrain sources attach <id>`.)
3. **Don't index the agent repo.** GBrain indexes the brain repo only.
Running `gbrain sync` against the agent repo pollutes search results
with operational config instead of world knowledge.
## How to Verify
+20 -43
View File
@@ -34,35 +34,35 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
## Auto-RLS: the event trigger and the one-time backfill
## v0.26.7 — auto-RLS event trigger and one-time backfill
gbrain ships two mechanisms (schema migration v35) that close the gap where a
table could exist in your `public` schema without RLS for any amount of time
at all.
Starting in v0.26.7 (migration v35), gbrain ships two changes that close the
gap where a table could exist in your `public` schema without RLS for any
amount of time at all.
**1. The event trigger.** A Postgres DDL event trigger named
`auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY`
on every newly created `public.*` table. It covers `CREATE TABLE`,
`CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres
reports as a table-creation command. Tables created by gbrain itself, by
any other app sharing the same Supabase project, or by a human running raw
SQL all get RLS enabled the moment they exist.
your other apps sharing the same Supabase project (Baku, Hermes, anything),
or by a human running raw SQL all get RLS enabled the moment they exist.
Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly
ignored — Supabase manages those, and we should not touch them.
**2. The one-time backfill.** The first upgrade that applies migration v35
**2. The one-time backfill.** When you upgrade to v0.26.7, the migration
walks every existing `public.*` base table whose RLS is off and whose comment
doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS
on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op
on every brain.
### Read this before upgrading a pre-auto-RLS brain
### Breaking change: read this before upgrading
If you have public tables that are intentionally RLS-off and you want them
to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before**
the upgrade that applies migration v35. The backfill flips RLS on for any
public table that doesn't carry the exact comment contract documented below.
There is no `--dry-run` flag on the migration.
running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public
table that doesn't carry the exact comment contract documented below. There
is no `--dry-run` flag on the migration.
The minimum cost of getting this wrong is one round-trip: the operator runs
the SQL to enable RLS on a table that should have been exempt, then
@@ -71,7 +71,7 @@ prevent a re-flip on a later doctor run. No data is lost.
### Cross-app implications
If a non-gbrain app (a side project, a script you wrote, anything) creates
If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates
tables in the same Supabase project, the trigger will enable RLS on those
tables too. Two ways to handle that:
@@ -90,40 +90,17 @@ ship a policy.
### What if the trigger gets dropped?
`gbrain doctor` includes an `rls_event_trigger` check that verifies the
`gbrain doctor` includes a new `rls_event_trigger` check that verifies the
trigger is installed and enabled. If you drop it manually for any reason
(debugging, migration testing, anything), doctor warns and points you here.
(debugging, migration testing, anything), doctor warns and gives you the
recovery command:
Recreate it by re-running the trigger DDL from migration v35 — idempotent
(`CREATE OR REPLACE` + `DROP EVENT TRIGGER IF EXISTS`), safe to paste into
psql as a BYPASSRLS role (e.g. `postgres`):
```sql
CREATE OR REPLACE FUNCTION auto_enable_rls()
RETURNS event_trigger AS $$
DECLARE
obj record;
BEGIN
FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands()
WHERE object_type = 'table'
AND schema_name = 'public'
LOOP
EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', obj.object_identity);
END LOOP;
END;
$$ LANGUAGE plpgsql;
DROP EVENT TRIGGER IF EXISTS auto_rls_on_create_table;
CREATE EVENT TRIGGER auto_rls_on_create_table
ON ddl_command_end
WHEN TAG IN ('CREATE TABLE', 'CREATE TABLE AS', 'SELECT INTO')
EXECUTE FUNCTION auto_enable_rls();
```
gbrain apply-migrations --force-retry 35
```
(This is the same DDL migration v35 runs — the canonical copy lives in the
`MIGRATIONS` array in `src/core/migrate.ts`. There's no CLI shortcut:
`gbrain apply-migrations --force-retry` targets the vX.Y.Z orchestrator
registry, not numeric schema migrations like v35.)
Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS`
and recreates cleanly.
### Why no FORCE ROW LEVEL SECURITY?
@@ -170,7 +147,7 @@ Rules:
```sql
ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY;
COMMENT ON TABLE public.expenses_ramp IS
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=you, 2026-04-22';
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22';
```
After that, `gbrain doctor` reports:
+11 -44
View File
@@ -127,9 +127,9 @@ you can use as a reference shape. The skillpack story for distributing
your own resolvers across machines is covered in
[skillpacks as scaffolding](skillpacks-as-scaffolding.md).
## The compact list format
## The compact list format (v0.41.7.0)
GBrain's resolver parser originally required markdown tables:
GBrain's resolver parser used to require markdown tables:
```markdown
| Trigger | Skill |
@@ -146,14 +146,14 @@ format that scales better:
- **flight-tracker**: track my flight | flight status | when does my flight land
```
When `gbrain doctor` only spoke the table dialect, a 306-skill
compact-format resolver reported every skill as unreachable: **238 FAIL
errors on every doctor run**. The parser was silently treating the compact
dialect as zero skills.
Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a
306-skill compact-format resolver, the doctor reported every skill as
unreachable: **238 FAIL errors on every doctor run**. The parser was
silently treating the compact dialect as zero skills.
Today the parser supports both. The same `parseResolverEntries`
function reads table rows and list rows in the same file, with the
multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
v0.41.7.0 ships dual-format support. The same `parseResolverEntries`
function reads both table rows and list rows in the same file, with the
v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
`../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor`
and the 238 FAILs collapse to 0.
@@ -267,7 +267,8 @@ I initially converted my resolver from a clean list format to a table
format because the validator only spoke tables. That was wrong. When a
tool fails against valid data, the right move is to fix the tool, not
reshape the data. The list format was correct, compact, readable, easy
to maintain. The parser needed to support both shapes — and now it does.
to maintain. The parser needed to support both shapes. v0.41.7.0 is
that fix.
The same principle applies everywhere in agent systems. Your SKILL.md is
the source of truth. Your AGENTS.md is the source of truth. Your resolver
@@ -294,40 +295,6 @@ architecture that gets you from 10 to 50. That's normal. Systems that
scale change shape. The important thing is that each tier preserves full
capability. You're organizing, not deleting.
## Plugin bundling is a curation decision
Not every skill in `skills/` reaches downstream installs. The plugin
manifest (`openclaw.plugin.json`) is the bundled set; everything else is a
recorded exclusion in `skills/plugin-exclusions.json`, each with a reason.
The two are test-pinned in both directions: every manifest skill is either
bundled or a recorded exclusion, and no skill is both. Adding a skill to
the tree does NOT ship it — bundling is an explicit decision, and an
unbundled skill never reaches a downstream install. When you write a new
skill, decide (and record) which side of that line it lives on.
`bun run gate:skills` (`scripts/skills-commit-gate.sh`) is the per-commit gate
for any change under `skills/`. It runs the conformance + resolver +
plugin-manifest tests, `check-resolvable --strict`, the `skills.lock.json`
regen + freshness check, and `check-skill-refs` in seconds — run it before
committing a skills change so the membership/closure and `plugin.version`
assertions fail locally instead of in CI.
## When a skill misroutes
Treat a misroute like a failing test, because it becomes one. First
reproduce it as a fixture in the skill's `routing-eval.jsonl` — the utterance
that misrouted, with the expected skill (or `null`). Rewrite the misrouted
utterance onto placeholder entities (`alice-example`, `acme-example`) before
committing the fixture — same rule as skill-autobench; a routing fixture is a
public artifact and must not carry a real contact or company name. Only then
fix the cause:
usually a trigger in the skill's frontmatter or its row in
`skills/RESOLVER.md`. Regenerate the lock (`bun run
scripts/generate-skills-manifest.ts`) and the llms bundles (`bun run
build:llms`), verify with `gbrain check-resolvable --strict`, and ship it as
a MICRO release. Downstream installs heal on their next upgrade — the fix
travels with the skillpack, not with a support thread.
## Related
- [Skill development cycle](skill-development.md) — the 5-step loop for
+28 -128
View File
@@ -1,152 +1,54 @@
# Search Modes
Two decisions shape every gbrain lookup, and this guide covers both:
## Goal
Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result.
1. **Which mode bundle** your brain runs — `conservative` / `balanced` /
`tokenmax`, the named cost-knob presets that control cache, token budget,
query expansion, and result count. This is the config-level decision you
make once (at `gbrain init` or via `gbrain config set search.mode`).
2. **Which lookup verb** to use per call — `gbrain search` (keyword),
`gbrain query` (hybrid), or `gbrain get` (direct). This is the
per-lookup decision an agent makes on every question.
## What the User Gets
Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls.
## The three mode bundles
A search mode is a named preset that sets every search-cost knob at once.
The bundles are frozen in `src/core/search/mode.ts` (`MODE_BUNDLES`):
| Knob | `conservative` | `balanced` | `tokenmax` |
|-------------------------------|----------------|------------|----------------|
| `cache.enabled` | true | true | true |
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
| `intentWeighting` | true | true | true |
| `tokenBudget` | **4000** | **12000** | **off** |
| `expansion` (LLM multi-query) | false | false | **true** |
| `relationalRetrieval` | false | **true** | **true** |
| `searchLimit` default | 10 | 25 | 50 |
- **`conservative`** — smallest payloads. Pairs naturally with a cheap
downstream model (Haiku-class) or a high query volume.
- **`balanced`** — the default and the fallback when no mode is set.
- **`tokenmax`** — no token budget, LLM query expansion on, 50 results.
Pairs with an expensive downstream model you want fully fed.
Two of the knobs deserve a sentence:
- **`expansion`** rewrites your query into multiple variants via a cheap
LLM call per search (adds roughly $1.50 per 1K queries) — better recall,
small extra cost.
- **`relationalRetrieval`** adds a graph-walk recall arm for relational
questions ("who invested in X", "what connects A and B"); it's a pure
no-op for non-relational queries. The `query` op's `relational` flag
forces it on/off per call.
### Setting and resolving the mode
```bash
gbrain config set search.mode tokenmax
```
Per-knob resolution (highest first):
per-call SearchOpts → per-key config override (search.cache.enabled, …) →
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
Mode resolution lives in bare `hybridSearch`, not just the cached wrapper,
so eval replays test the same mode-affected behavior as the production
`query` op. The query cache folds the active knobs into its key
(`knobs_hash`), so switching modes never serves you a stale result set
from a different configuration.
### Cost intuition
gbrain's own cost is rounding error; what the mode really controls is how
many tokens your *downstream agent* pays to read per query. The
corner-to-corner spread is ~25x once you pair mode with downstream model.
Rough anchors at 10K queries/month, full payload, no cache savings:
| Mode \ Downstream | Haiku-class (\$1/M in) | Sonnet-class (\$3/M in) | Opus-class (\$5/M in) |
|---|---|---|---|
| conservative (~4K tok) | **\$40/mo** | \$120/mo | \$200/mo |
| balanced (~10K tok) | \$100/mo | \$300/mo | \$500/mo |
| tokenmax (~20K tok) | \$200/mo | \$600/mo | **\$1,000/mo** |
Scales linearly with volume. Cache hits cut all numbers ~50%; disciplined
prompt caching in the agent loop cuts further. Mismatched pairings waste
capacity in both directions — a tokenmax payload overwhelms a cheap model,
a conservative payload starves an expensive one. The full methodology and
realistic-scale walkthrough live in
[`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
### CLI surfaces
```bash
gbrain search modes # what is running, with per-knob attribution
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
gbrain search tune [--apply] # data-driven recommendations
gbrain search diagnose "<query>" --target <slug>
# trace where a page surfaces (or fails to)
# across the keyword/vector/alias/hybrid layers
```
The mode picker runs inside `gbrain init` (non-TTY auto-selects `balanced`).
## Choosing a lookup verb (search vs query vs get)
Independent of which bundle is active, every individual lookup should use
the cheapest verb that answers the question.
## Implementation
```
on user_asks_about(topic):
# Decision tree: pick the right lookup verb
# Decision tree: pick the right search mode
if know_exact_slug(topic):
# Direct get -- instant, no search overhead
# MODE 3: Direct get -- instant, no search overhead
result = gbrain get <slug>
# e.g., "Tell me about Alice" -> gbrain get alice-example
# e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi
# Returns the FULL page -- compiled truth + timeline
elif topic.is_exact_name or topic.is_keyword:
# MODE 1: Cheap-hybrid search -- vector + keyword + RRF, NO LLM
# expansion. Embeds the query when embeddings are configured; the
# keyword arm still works day-one without them (keyword-only is
# also available via the search.mcp_keyword_only opt-out).
# MODE 1: Keyword search -- fast, no embeddings needed, day-one ready
results = gbrain search "{name_or_keyword}"
# e.g., "Find anything about Series A" -> gbrain search "Series A"
# Returns CHUNKS, not full pages
# IMPORTANT: search returns chunks
# IMPORTANT: keyword search returns chunks
# If the chunk confirms relevance, THEN load the full page:
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
elif topic.is_semantic_question or topic.is_concept_or_landscape:
# MODE 2: Full hybrid -- adds multi-query LLM expansion on top of
# vector + keyword + RRF. Owns concept / landscape / "all-of-X"
# questions: expansion recovers synonym- and outcome-phrased
# matches a single embedding misses. Costs one LLM expansion call
# per query -- worth it for these question shapes.
elif topic.is_semantic_question:
# MODE 2: Hybrid search -- semantic + keyword, needs embeddings
results = gbrain query "{natural language question}"
# e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts"
# e.g., "all the companies doing offshore wind" -> gbrain query "..."
# Returns ranked chunks via vector + keyword + expansion + RRF
# Returns ranked chunks via vector + keyword + RRF
# Same rule: chunks first, then get full page if needed
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
# Quick reference:
# | Mode | Command | Needs Embeddings | Speed | Best For |
# |-------------|----------------------|------------------|---------|-------------------------------------------|
# | Cheap-hybrid| gbrain search "term" | Uses if present | Fastest | Known names, exact tokens |
# | Full hybrid | gbrain query "..." | Yes | Fast | Concept / landscape / "all-of-X", synonyms |
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
# | Mode | Command | Needs Embeddings | Speed | Best For |
# |---------|----------------------|------------------|---------|---------------------------------|
# | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches |
# | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match |
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
# Progression over time:
# Day 1: search (keyword arm works without embeddings)
# After first embed: vector arm + full hybrid (query) unlocked
# Day 1: keyword search (works without embeddings)
# After first embed: hybrid search unlocked
# Once you know slugs: direct get for speed
# Precedence for conflicting information within a page:
@@ -156,23 +58,21 @@ on user_asks_about(topic):
# 4. External sources (web search, APIs)
```
### Tricky Spots
## Tricky Spots
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
2. **Search works without embeddings.** On day one before any embedding run, `gbrain search` still works (the keyword arm carries it; the vector arm joins once embeddings exist). Don't tell the user "search isn't available yet" -- search is always available.
3. **Don't use full hybrid for known names.** `gbrain query "Alice Example"` wastes an LLM expansion call. Use `gbrain search "Alice Example"` or better yet `gbrain get alice-example` if you know the slug.
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Alice" -- get the full page.
5. **Full hybrid needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
6. **A populated `gbrain search` result set is not proof you found everything.** Search runs without query expansion, so synonym- and outcome-phrased matches can be missed even when it returns plenty of hits. For "find every / all / the landscape of" questions, use `gbrain query`; for literal exhaustive enumeration ("list every page of type X"), use `list_pages` pagination. A nonzero count is not a completeness signal.
2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available.
3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug.
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page.
5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
### How to Verify
## How to Verify
1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references.
1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references.
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline.
3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline.
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
6. Run `gbrain search modes` -- confirm the active mode bundle and any per-key overrides are what you expect.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+2 -5
View File
@@ -40,9 +40,7 @@ Show the user the results. Get feedback.
- Revise the process based on what you learned.
**Step 4: Codify into a Skill.**
Write the SKILL.md (`gbrain skillify scaffold <name>` scaffolds the tree for
you; `gbrain skillopt` optimizes an existing skill against a benchmark).
Either:
Write the SKILL.md. Either:
- **New skill** -- genuinely new capability
- **Add to existing skill** -- variation of something that exists (parameterize it)
@@ -65,8 +63,7 @@ Skills should be **Mutually Exclusive, Collectively Exhaustive**:
- Each signal source has exactly ONE owner skill
- Two skills creating the same brain page = MECE violation
**Example ownership (no overlap — illustrative; your skill roster will
differ):**
**Example ownership (no overlap):**
| Signal Source | Owner Skill | Creates |
|--------------|-------------|---------|
+4 -4
View File
@@ -65,14 +65,14 @@ For each step:
more than 0.05.
6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe.
After each epoch with no improvement: D6 slow-update fires. Today it emits
the audit event only; the full meta-edit proposal is a tracked follow-up.
After each epoch with no improvement: D6 slow-update fires one meta-edit
proposal (this lives in v0.42 follow-up; v1 emits the audit event).
## Flags
| Flag | Default | Purpose |
|---|---|---|
| `--benchmark <path>` | `skills/<name>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
| `--benchmark <path>` | `skills/<n>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
| `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) |
| `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) |
| `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl |
@@ -136,7 +136,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
- **No benchmark.** Optimizing against guesses is worse than not optimizing.
- **Write-flavored skills.** Skills whose job is to `put_page` heavily can't
use the read-only sandbox; mocked-write capture is a tracked follow-up.
use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up.
- **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful
validation needs ≥20 tasks total per the paper.
+15 -26
View File
@@ -1,11 +1,12 @@
# Skillpacks as scaffolding, not amber
`gbrain skillpack` is a scaffold + reference library, not a package
manager. This guide explains the model and the workflow.
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
scaffold + reference library. This guide explains the model and the
workflow.
## Why it works this way
## Why we changed it
An earlier design (the "amber" model):
Pre-v0.33 (the "amber" model):
- `gbrain skillpack install <name>` copied bundled skills into your
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
@@ -25,16 +26,7 @@ repo. You scaffold once, you own them, you fork and edit freely. When
gbrain ships a new version, you ask "what changed?" — the agent reads
the diff and decides what (if anything) to integrate.
## The core workflow commands
The five commands below are the scaffold-and-own workflow. The full
`gbrain skillpack` surface is larger (`list`, `diff`, `check`, `search`,
`info`, `registry`, `doctor`, `init`, `pack`, `endorse`, …) — run
`gbrain skillpack --help` for the always-current list. One worth calling
out here: **`gbrain skillpack init-brain-pack <name>`** scaffolds a
*brain-resident* pack inside a brain/source repo (`brain_resident: true`
plus a machine-parseable README) that connecting harnesses discover on
`gbrain sources add`.
## The five commands
### `gbrain skillpack scaffold <name> [--workspace PATH]`
@@ -85,15 +77,15 @@ gbrain skillpack reference book-mirror
`reference <name> --apply-clean-hunks` is the auto-apply path. It
parses the diff between gbrain's bundle and your local copy, applies
every hunk whose pre-change context matches uniquely. **Two-way merge
limitation**: without scaffold-time base tracking (intentionally out of
scope), this cannot distinguish "gbrain changed X"
limitation**: without scaffold-time base tracking (intentionally
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
from "you changed X." Applied hunks align everything to gbrain. Use
`--dry-run` first to preview, or run plain `reference` to inspect the
diff before letting auto-apply touch anything.
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
One-shot conversion for workspaces still on the legacy managed-block
One-shot conversion for workspaces on the pre-v0.33 managed-block
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
markers and the manifest receipt comment from your resolver file.
@@ -139,9 +131,7 @@ into gbrain so other clients can scaffold it. Default behavior:
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted. Harvest must preserve
the top-level OpenClaw-native plugin fields (`id`, `configSchema`, `contracts`)
because OpenClaw validates those before it can install the package.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
@@ -165,7 +155,7 @@ Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
frontmatter, and match the user's intent against every skill's
`triggers:` array. When a match scores high enough, invoke that skill.
This replaces the legacy model where `gbrain skillpack install` wrote
This replaces the v0.32 model where `gbrain skillpack install` wrote
table rows into your `RESOLVER.md`. Rows are gone (or, for users
migrating from the old model, preserved transitionally by
`migrate-fence` until they run `scrub-legacy-fence-rows`).
@@ -181,7 +171,7 @@ If you're a downstream agent author updating to this model:
## Removing a scaffolded skill
There's no `gbrain skillpack uninstall` command. The files
There's no `gbrain skillpack uninstall` command in v0.33. The files
in your `skills/<slug>/` are first-class members of your repo —
delete them like any other code:
@@ -203,17 +193,16 @@ You own the files. There's no manifest to update, no fence to rebuild.
## When to use which command (quick decision tree)
- **New host repo, want a gbrain skill**`scaffold`
- **Shipping a pack from inside a brain/source repo**`init-brain-pack`
- **gbrain shipped a new version, want to see what's changed**
`reference` (read-only) or `reference --apply-clean-hunks` (auto)
- **Upgrading from the legacy managed-block model**`migrate-fence` (one-shot)
- **Upgrading from v0.32 or earlier**`migrate-fence` (one-shot)
- **Cleanup after `migrate-fence`**`scrub-legacy-fence-rows`
- **Lift your fork's skill back into gbrain**`harvest` + the
`skillpack-harvest` editorial skill
## What about `install` and `uninstall`?
Both are removed. Running either prints an error pointing at the
replacement command. No deprecated alias — this is a clean break.
Both are removed in v0.33. Running either prints an error pointing at
the replacement command. No deprecated alias — this is a clean break.
If you have existing scripts referencing the old names, update them
once and move on.

Some files were not shown because too many files have changed in this diff Show More