mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
121
Commits
v0.42.72.1
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7fdcd8bd2e | ||
|
|
6411150071 | ||
|
|
ac402f55f8 | ||
|
|
cf3527a40f | ||
|
|
e2b2819e63 | ||
|
|
2ae5d60b98 | ||
|
|
bd4c976a85 | ||
|
|
9b9bd8b241 | ||
|
|
fd9bb12b42 | ||
|
|
5bd2c51053 | ||
|
|
189bf856ee | ||
|
|
735dec83b7 | ||
|
|
fd0e371d5b | ||
|
|
136fc109c1 | ||
|
|
1243a7d3bb | ||
|
|
44eea64084 | ||
|
|
45bd04ff9f | ||
|
|
8a626999f0 | ||
|
|
033029f25d | ||
|
|
a729ca8a8c | ||
|
|
5087507de0 | ||
|
|
ed6e4e3219 | ||
|
|
f8b0ececcb | ||
|
|
ca260baaaa | ||
|
|
0c485415a5 | ||
|
|
0cfedd026d | ||
|
|
b92cc967df | ||
|
|
3eccd4ccd6 | ||
|
|
154814b095 | ||
|
|
9b720b04af | ||
|
|
e8785c1ab1 | ||
|
|
638dd0d247 | ||
|
|
697016f69d | ||
|
|
758a2d4293 | ||
|
|
52389dbe5b | ||
|
|
fb141969f5 | ||
|
|
fa5ec8399f | ||
|
|
3ce296e315 | ||
|
|
1ec6a6e842 | ||
|
|
068f586128 | ||
|
|
4dc77c3979 | ||
|
|
3e4bc112d8 | ||
|
|
a4422f96ef | ||
|
|
447f81956d | ||
|
|
dcad42534e | ||
|
|
ce156eb8ed | ||
|
|
3f595083fe | ||
|
|
d8e3772810 | ||
|
|
810d1c5540 | ||
|
|
2dc33fb865 | ||
|
|
dc6e61b07f | ||
|
|
30c81b709c | ||
|
|
0a1890bbf8 | ||
|
|
a849d833eb | ||
|
|
23c7b0eb16 | ||
|
|
94ec7e31e0 | ||
|
|
3fa0a5acb5 | ||
|
|
8ecd52022e | ||
|
|
e795324ec5 | ||
|
|
f7d63c7159 | ||
|
|
0a34ced5d7 | ||
|
|
2dbaebbe16 | ||
|
|
b0b9af042f | ||
|
|
8db07e4a79 | ||
|
|
f7d4f19124 | ||
|
|
b966d2682c | ||
|
|
636628fdb2 | ||
|
|
cb07cfda8d | ||
|
|
9b6db85d39 | ||
|
|
8d5bdfe125 | ||
|
|
15ecc65b24 | ||
|
|
3f22f51e5d | ||
|
|
99dd1a083c | ||
|
|
6fae2c10ff | ||
|
|
a996e42856 | ||
|
|
c2cc8b0207 | ||
|
|
d35c9c9e44 | ||
|
|
fc310db3ea | ||
|
|
52306ed438 | ||
|
|
a2521d0a32 | ||
|
|
75fae742d5 | ||
|
|
a13ecb1f0a | ||
|
|
e2ff128e35 | ||
|
|
69472c24e7 | ||
|
|
ea17f5911c | ||
|
|
d050969da6 | ||
|
|
7c65bcee9b | ||
|
|
9afbf4a09a | ||
|
|
e11e407752 | ||
|
|
c5952b8714 | ||
|
|
130d321d23 | ||
|
|
485c4773ca | ||
|
|
f8878be54e | ||
|
|
0df56f1efa | ||
|
|
01d9639204 | ||
|
|
f15480b9d0 | ||
|
|
0b47afbf40 | ||
|
|
9f81e1d8b2 | ||
|
|
86555e39bb | ||
|
|
3257758492 | ||
|
|
a948dfd6e2 | ||
|
|
15b9863d13 | ||
|
|
aecb33e795 | ||
|
|
e5dee4fb78 | ||
|
|
3c26f2eaf2 | ||
|
|
615c33e5b5 | ||
|
|
40d1d4cabc | ||
|
|
d610a845a8 | ||
|
|
f87488ff36 | ||
|
|
ddb39df23a | ||
|
|
88731d8cf3 | ||
|
|
4a00c31b12 | ||
|
|
94662eb1e1 | ||
|
|
2f65ed8da6 | ||
|
|
ca1eed3e95 | ||
|
|
82fe0216ff | ||
|
|
fa313f2969 | ||
|
|
2e554500e2 | ||
|
|
630786dc65 | ||
|
|
d92f0dc86f | ||
|
|
a68379050e |
@@ -4,19 +4,6 @@ about: Something isn't working
|
||||
labels: bug
|
||||
---
|
||||
|
||||
**Why are you opening this? (human-written, required)**
|
||||
|
||||
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
|
||||
doing, what happened, why does it matter to you? Rough is fine.
|
||||
Issues/PRs without this are closed unreviewed. -->
|
||||
|
||||
|
||||
**Screenshot of gbrain in use (required)**
|
||||
|
||||
<!-- Your terminal / agent session / logs showing the real situation.
|
||||
Redact private names, keys, and brain contents first. -->
|
||||
|
||||
|
||||
**What happened?**
|
||||
|
||||
|
||||
|
||||
@@ -4,19 +4,6 @@ about: Suggest an improvement
|
||||
labels: enhancement
|
||||
---
|
||||
|
||||
**Why are you opening this? (human-written, required)**
|
||||
|
||||
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
|
||||
doing, what happened, why does it matter to you? Rough is fine.
|
||||
Issues/PRs without this are closed unreviewed. -->
|
||||
|
||||
|
||||
**Screenshot of gbrain in use (required)**
|
||||
|
||||
<!-- Your terminal / agent session / logs showing the real situation.
|
||||
Redact private names, keys, and brain contents first. -->
|
||||
|
||||
|
||||
**What problem does this solve?**
|
||||
|
||||
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
**Why are you opening this? (human-written, required)**
|
||||
|
||||
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
|
||||
doing, what went wrong or what you needed, why it matters to you.
|
||||
Rough grammar is fine. PRs without this are closed unreviewed. -->
|
||||
|
||||
|
||||
**Screenshot of gbrain in use (required)**
|
||||
|
||||
<!-- Your terminal / agent session / logs showing the real need this fixes.
|
||||
Redact private names, keys, and brain contents first. -->
|
||||
|
||||
|
||||
**What changed**
|
||||
|
||||
|
||||
**How it was tested**
|
||||
@@ -1,7 +1,7 @@
|
||||
name: Heavy Tests
|
||||
|
||||
# Heavy ops-shape tests under tests/heavy/. Cost minutes per run; NOT part
|
||||
# of default PR CI. Two triggers:
|
||||
# of default PR CI. Three triggers:
|
||||
# - Nightly schedule (catches regressions within 24h of merge to master).
|
||||
# - On-demand opt-in via PR label `heavy-tests` (slow loop kept off by default).
|
||||
# - Manual workflow_dispatch for triage.
|
||||
@@ -66,6 +66,20 @@ jobs:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
run: bun run test:heavy
|
||||
|
||||
# Agent-bootstrap offline Docker e2e [A7] — placeholder registration.
|
||||
# The networkless cold-machine container test (interview → render →
|
||||
# verify with a fake gh) lands with the bootstrap doors task at
|
||||
# tests/docker/bootstrap-e2e.sh. Gated on file existence so this stays
|
||||
# a visible no-op until then. It runs at heavy cadence (nightly +
|
||||
# `heavy-tests` label) by design — never in the PR shard matrix.
|
||||
- name: Bootstrap offline Docker e2e (placeholder)
|
||||
run: |
|
||||
if [ -f tests/docker/bootstrap-e2e.sh ]; then
|
||||
bash tests/docker/bootstrap-e2e.sh
|
||||
else
|
||||
echo "SKIP: tests/docker/bootstrap-e2e.sh not present yet — placeholder until the bootstrap doors task lands."
|
||||
fi
|
||||
|
||||
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
|
||||
# run. Upload those + the rss workload JSON on failure for triage
|
||||
# without re-running locally.
|
||||
@@ -87,3 +101,236 @@ jobs:
|
||||
path: heavy-artifacts/
|
||||
retention-days: 14
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Real-agent door e2e: drives the ACTUAL `claude` + `codex` + `hermes`
|
||||
# binaries (no PATH shims) against a real gbrain over MCP. These pay real API
|
||||
# cost and need the binaries installed + authed, which a stock GitHub runner
|
||||
# does NOT have — so the tests self-SKIP (describe.skipIf on binary/auth) and
|
||||
# the job is a clean no-op here. It exists so a self-hosted /
|
||||
# manually-provisioned runner WITH authed claude/codex/hermes (and
|
||||
# ANTHROPIC/OPENAI creds) actually exercises the real binaries. Heavy cadence
|
||||
# only (nightly + `real-agent-e2e` label + dispatch); NEVER the PR shard
|
||||
# matrix.
|
||||
real-agent-e2e:
|
||||
name: Real-agent door e2e (skips without authed binaries)
|
||||
if: |
|
||||
github.event_name != 'pull_request' ||
|
||||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
|
||||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
# Open the hermes opt-in door here so binary/auth absence — not the
|
||||
# opt-in var — is what skips (same posture as the claude/codex doors).
|
||||
GBRAIN_REAL_HERMES_E2E: '1'
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
|
||||
# Reference the door tests; run only the ones present (a door may land
|
||||
# in a sibling PR). Missing binary/auth → the file self-skips, so a
|
||||
# stock runner reports a green no-op rather than failing.
|
||||
- name: Run real-agent door tests
|
||||
run: |
|
||||
files=()
|
||||
for f in \
|
||||
test/e2e/bootstrap-real-claude.serial.test.ts \
|
||||
test/e2e/bootstrap-real-codex.serial.test.ts \
|
||||
test/e2e/install-real-hermes.serial.test.ts; do
|
||||
[ -f "$f" ] && files+=("$f")
|
||||
done
|
||||
if [ "${#files[@]}" -eq 0 ]; then
|
||||
echo "SKIP: no real-agent door test files present yet."
|
||||
exit 0
|
||||
fi
|
||||
echo "Running: ${files[*]}"
|
||||
# --timeout: real-agent turns are slow (live claude/codex); the door
|
||||
# tests self-skip without authed binaries so this is a no-op elsewhere.
|
||||
bun test --timeout=600000 "${files[@]}"
|
||||
|
||||
# Hermes door e2e: unlike real-agent-e2e above (best-effort, self-skipping),
|
||||
# this job PROVISIONS the real hermes binary itself — pinned installer digest,
|
||||
# non-interactive auth + model pin — and then requires the door tests to
|
||||
# actually execute. It pays real API cost, so it stays label-gated at heavy
|
||||
# cadence (nightly + `real-agent-e2e`/`heavy-tests` label + dispatch); NEVER
|
||||
# the PR shard matrix. Loud-fail throughout: a runner that cannot install or
|
||||
# auth hermes fails this job rather than skipping.
|
||||
hermes-door:
|
||||
name: Hermes door e2e (real binary, loud-fail)
|
||||
if: |
|
||||
github.event_name != 'pull_request' ||
|
||||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
|
||||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
|
||||
runs-on: ubuntu-latest
|
||||
# Four serial door tests at 600s each plus the installer budget cannot
|
||||
# fit the sibling job's 20 minutes.
|
||||
timeout-minutes: 40
|
||||
env:
|
||||
# Pin values documented in docs/mcp/HERMES-CLI-PIN.md — update them
|
||||
# together, deliberately, after reviewing upstream changes. The digest
|
||||
# pins the INSTALLER SCRIPT; the tag + commit pin the PAYLOAD it clones
|
||||
# (without them, the installer pulls upstream main into the runner that
|
||||
# later holds secrets). The commit is v2026.8.3's dereferenced SHA —
|
||||
# immutable even if the tag moves.
|
||||
HERMES_VERSION: "0.20.0"
|
||||
HERMES_GIT_TAG: "v2026.8.3"
|
||||
HERMES_GIT_COMMIT: "3c27eb6234bf91b8ceee9e9071591b31e9b148cb"
|
||||
HERMES_INSTALL_SHA256: "c118ff31618dc70339049ce71061b8f1351a1c70d9c2a236ed50d8a2550c550d"
|
||||
GBRAIN_REAL_HERMES_E2E: '1'
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
|
||||
# `runner.temp` is not an allowed context in job-level env, so the
|
||||
# evidence dir is derived here and exported for every later step (the
|
||||
# door test writes into it; the failure-path upload reads it — exporting
|
||||
# early keeps that upload working even when an install/precondition step
|
||||
# is what failed).
|
||||
- name: Prepare evidence dir
|
||||
run: |
|
||||
echo "GBRAIN_E2E_EVIDENCE_DIR=$RUNNER_TEMP/hermes-door-evidence" >> "$GITHUB_ENV"
|
||||
mkdir -p "$RUNNER_TEMP/hermes-door-evidence"
|
||||
|
||||
# NO secrets in this step's env: the installer is downloaded from the
|
||||
# network and executed, so it must never see credentials. The digest pin
|
||||
# turns an upstream installer change into a loud failure instead of
|
||||
# unreviewed code running in CI. Bound + retry the install run itself
|
||||
# (same posture as the e2e tier2 OpenClaw install): `timeout` kills a
|
||||
# hung attempt, 3 attempts ride out transient flakes, step cap backstops.
|
||||
- name: Install hermes (pinned installer digest)
|
||||
timeout-minutes: 15
|
||||
run: |
|
||||
curl -fsSL --retry 3 -o hermes-install.sh https://hermes-agent.nousresearch.com/install.sh
|
||||
if ! echo "$HERMES_INSTALL_SHA256 hermes-install.sh" | sha256sum -c -; then
|
||||
echo "::error::hermes installer digest drift — re-pin deliberately: update HERMES_INSTALL_SHA256 + HERMES_VERSION in this workflow and docs/mcp/HERMES-CLI-PIN.md after reviewing upstream changes" >&2
|
||||
exit 1
|
||||
fi
|
||||
for attempt in 1 2 3; do
|
||||
if timeout 600 bash hermes-install.sh --skip-setup --non-interactive --branch "$HERMES_GIT_TAG" --commit "$HERMES_GIT_COMMIT"; then
|
||||
# The branch/commit flags above are ASSERTED here, not trusted:
|
||||
# a shell installer that silently ignores unknown flags would
|
||||
# clone upstream main into a runner that later holds secrets.
|
||||
# Verify the actual checkout before anything else runs it.
|
||||
actual_commit=$(git -C "$HOME/.hermes/hermes-agent" rev-parse HEAD 2>/dev/null || echo "no-git-checkout")
|
||||
if [ "$actual_commit" != "$HERMES_GIT_COMMIT" ]; then
|
||||
echo "::error::hermes payload drift — installed checkout is $actual_commit, pinned $HERMES_GIT_COMMIT. Either the installer ignored its branch/commit flags or the layout moved from ~/.hermes/hermes-agent; re-pin deliberately (HERMES_GIT_TAG/HERMES_GIT_COMMIT + docs/mcp/HERMES-CLI-PIN.md) after reviewing upstream." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
exit 0
|
||||
fi
|
||||
echo "::warning::hermes install attempt $attempt failed or timed out; retrying in 10s" >&2
|
||||
sleep 10
|
||||
done
|
||||
echo "::error::hermes install failed after 3 attempts" >&2
|
||||
exit 1
|
||||
|
||||
- name: Preconditions (binary, secret, version pin)
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
run: |
|
||||
if ! command -v hermes >/dev/null 2>&1; then
|
||||
echo "::error::hermes did not resolve on PATH after install" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ -z "$ANTHROPIC_API_KEY" ]; then
|
||||
echo "::error::ANTHROPIC_API_KEY secret is empty — fork PRs get no secrets from GitHub, and this labeled job cannot run without them" >&2
|
||||
exit 1
|
||||
fi
|
||||
version_output=$(hermes --version)
|
||||
echo "$version_output"
|
||||
# Observed shape: `Hermes Agent v0.20.0 (2026.8.3)`.
|
||||
if ! printf '%s' "$version_output" | grep -qF "v$HERMES_VERSION"; then
|
||||
echo "::error::hermes version drift — expected v$HERMES_VERSION in: $version_output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Configure hermes (auth + model pin)
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
run: |
|
||||
mkdir -p ~/.hermes
|
||||
printf 'ANTHROPIC_API_KEY=%s\n' "$ANTHROPIC_API_KEY" > ~/.hermes/.env
|
||||
chmod 600 ~/.hermes/.env
|
||||
# `hermes model` is interactive-only; `config set` is the observed
|
||||
# non-interactive model pin.
|
||||
hermes config set model.default anthropic/claude-haiku-4.5
|
||||
# Global health check — informational only, never a gate here.
|
||||
hermes doctor || true
|
||||
|
||||
- name: Run hermes door tests
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
run: |
|
||||
# Redirect to a file, then tail — never pipe bun through tail (the
|
||||
# pipe eats the real exit code and truncates failure details).
|
||||
# `|| EXIT=$?` keeps the default `-e` shell from bailing before the
|
||||
# tail runs.
|
||||
EXIT=0
|
||||
bun test --timeout=600000 test/e2e/install-real-hermes.serial.test.ts > door.txt 2>&1 || EXIT=$?
|
||||
tail -40 door.txt
|
||||
if [ "$EXIT" -ne 0 ]; then
|
||||
# Preserve the FULL bun output for the failure artifact — bun
|
||||
# prints failure details before the summary, so the 40-line tail
|
||||
# above can drop exactly the lines a paid-CI triage needs.
|
||||
cp door.txt "$GBRAIN_E2E_EVIDENCE_DIR/" 2>/dev/null || true
|
||||
exit "$EXIT"
|
||||
fi
|
||||
# This job provisions the binary + auth above, so the door must
|
||||
# actually EXECUTE: a summary with zero passing tests means the
|
||||
# suite ran nothing or self-skipped everything — never let that
|
||||
# read as green.
|
||||
pass_count=$(grep -Eo '[0-9]+ pass' door.txt | tail -1 | grep -Eo '^[0-9]+' || true)
|
||||
if [ -z "$pass_count" ] || [ "$pass_count" -eq 0 ]; then
|
||||
echo "::error::hermes door summary shows no passing tests (nothing ran or everything skipped) — refusing to go green while testing nothing" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The door test copies its evidence into GBRAIN_E2E_EVIDENCE_DIR; the
|
||||
# workflow only uploads it. The test already excludes credential files —
|
||||
# the scrub below is defensive belt-and-suspenders before upload. Both
|
||||
# steps also require the evidence-dir env (a failure before the prepare
|
||||
# step leaves it unset, and there is nothing to upload then anyway).
|
||||
- name: Scrub credentials from evidence (defensive)
|
||||
if: failure() && env.GBRAIN_E2E_EVIDENCE_DIR != ''
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
run: |
|
||||
# Three layers, because the evidence dir carries files WRITTEN BY
|
||||
# THE THIRD-PARTY HERMES BINARY (logs/sessions) and upload-artifact
|
||||
# follows symlinks:
|
||||
# 1. filename scrub (env files),
|
||||
# 2. symlink delete (an agent-dropped link could dereference to a
|
||||
# real credential file at upload time),
|
||||
# 3. content scrub (any file that embeds the key — auth-error dumps
|
||||
# are most likely exactly on the failure path that uploads).
|
||||
find "$GBRAIN_E2E_EVIDENCE_DIR" -type f \( -name '.env' -o -name '*.env' \) -exec rm -f {} + 2>/dev/null || true
|
||||
find "$GBRAIN_E2E_EVIDENCE_DIR" -type l -delete 2>/dev/null || true
|
||||
if [ -n "$ANTHROPIC_API_KEY" ]; then
|
||||
grep -rlF "$ANTHROPIC_API_KEY" "$GBRAIN_E2E_EVIDENCE_DIR" 2>/dev/null | while IFS= read -r f; do
|
||||
echo "::warning::removing evidence file containing the API key: ${f#"$GBRAIN_E2E_EVIDENCE_DIR"/}" >&2
|
||||
rm -f "$f"
|
||||
done
|
||||
fi
|
||||
- name: Upload hermes door evidence
|
||||
if: failure() && env.GBRAIN_E2E_EVIDENCE_DIR != ''
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
||||
with:
|
||||
name: hermes-door-evidence
|
||||
path: ${{ env.GBRAIN_E2E_EVIDENCE_DIR }}
|
||||
retention-days: 14
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Hosted ubuntu-latest runners are ephemeral, but this must not depend
|
||||
# on that: if the job ever moves to a self-hosted runner (the sibling
|
||||
# real-agent-e2e job is designed for one), a key left in ~/.hermes/.env
|
||||
# would persist for every later workload on that runner.
|
||||
- name: Remove hermes credentials (unconditional)
|
||||
if: always()
|
||||
run: rm -f ~/.hermes/.env
|
||||
|
||||
@@ -38,6 +38,7 @@ jobs:
|
||||
outputs:
|
||||
version: ${{ steps.v.outputs.version }}
|
||||
exists: ${{ steps.v.outputs.exists }}
|
||||
template_tree_hash: ${{ steps.v.outputs.template_tree_hash }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- id: v
|
||||
@@ -47,6 +48,13 @@ jobs:
|
||||
run: |
|
||||
version="$(tr -d '[:space:]' < VERSION)"
|
||||
echo "version=$version" >> "$GITHUB_OUTPUT"
|
||||
# [C3] Record the vendored template tree's git hash alongside the
|
||||
# asset completeness check. The publish-template job embeds it in
|
||||
# the template repo's commit message, so template-repo HEAD can be
|
||||
# audited back to the exact reviewed tree in THIS repo. Empty until
|
||||
# templates/bootstrap/template-repo/ is vendored.
|
||||
tree_hash="$(git rev-parse "HEAD:templates/bootstrap/template-repo" 2>/dev/null || true)"
|
||||
echo "template_tree_hash=$tree_hash" >> "$GITHUB_OUTPUT"
|
||||
# Complete = release exists AND carries every asset the self-updater
|
||||
# can request. A partial release must NOT short-circuit, so a re-run
|
||||
# can repair it.
|
||||
@@ -137,3 +145,105 @@ jobs:
|
||||
files: |
|
||||
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
|
||||
artifacts/gbrain-linux-x64/gbrain-linux-x64
|
||||
# [C1 = D6-A] `latest-stable` is the single sanctioned distribution ref:
|
||||
# the README paste block, the BOOTSTRAP_FOR_AGENTS.md fetch URL, and
|
||||
# `bun install -g github:garrytan/gbrain#latest-stable` all reference it
|
||||
# permanently, so paste blocks copied into the wild never rot and there
|
||||
# is no 404 window between VERSION landing and assets publishing. It is
|
||||
# a maintainer-controlled tag advanced ONLY here — the FINAL step of the
|
||||
# release job, after binaries + provenance attestation have fully
|
||||
# published — so a half-built release never moves it. The force-move
|
||||
# (`+`) is intentional: latest-stable tracks the newest verified release;
|
||||
# per-release history lives in the immutable v<VERSION> tags.
|
||||
# scripts/check-bootstrap-tag.sh keeps the entry docs pinned to this ref.
|
||||
#
|
||||
# If this step ALONE fails, re-advance by hand (a full re-run would skip:
|
||||
# the release already exists with all assets):
|
||||
# git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
|
||||
- name: Advance latest-stable to this release commit
|
||||
run: git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"
|
||||
|
||||
# [G7/S3#4] Publishes the rendered agent-workspace template repo (the
|
||||
# GitHub "Use this template" door) from CI ONLY — no human pushes it by
|
||||
# hand, so what adopters clone is exactly what this repo reviewed. Guarded
|
||||
# three ways:
|
||||
# 1. The release above fully published (needs: release + the exists gate).
|
||||
# 2. The vendored tree templates/bootstrap/template-repo/ exists — the
|
||||
# generator/doors work may not have landed yet; skip, never fail.
|
||||
# 3. The TEMPLATE_REPO_PAT secret is configured. Secrets are not readable
|
||||
# in job-level `if:` expressions, so the secret is bound to env (the
|
||||
# same env-indirection pattern as the release-notes step) and checked
|
||||
# by the gate step's shell.
|
||||
#
|
||||
# TEMPLATE_REPO_PAT scope: a fine-grained PAT with `contents: write` on the
|
||||
# template repository ONLY — no other repositories, no other permissions.
|
||||
# Documented in docs/RELEASING.md.
|
||||
publish-template:
|
||||
needs: [version, release]
|
||||
if: needs.version.outputs.exists == 'false'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
env:
|
||||
TEMPLATE_REPO_PAT: ${{ secrets.TEMPLATE_REPO_PAT }}
|
||||
# owner/name of the template repo; override via repository variable.
|
||||
TEMPLATE_REPO: ${{ vars.TEMPLATE_REPO || 'garrytan/gbrain-agent-template' }}
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- id: gate
|
||||
name: Gate on PAT + vendored template tree
|
||||
run: |
|
||||
if [ -z "$TEMPLATE_REPO_PAT" ]; then
|
||||
echo "publish=false" >> "$GITHUB_OUTPUT"
|
||||
echo "SKIP: TEMPLATE_REPO_PAT secret not configured — template-repo publishing disabled."
|
||||
elif [ ! -d templates/bootstrap/template-repo ]; then
|
||||
echo "publish=false" >> "$GITHUB_OUTPUT"
|
||||
echo "SKIP: templates/bootstrap/template-repo/ not vendored yet — nothing to publish."
|
||||
else
|
||||
echo "publish=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
run: bun install
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
name: Generate template tree and byte-diff against the vendored copy
|
||||
run: |
|
||||
bun run scripts/generate-template-repo.ts --out /tmp/template-tree
|
||||
# [A1] Publish gate: fresh generator output must equal the vendored
|
||||
# tree byte-for-byte. A mismatch means the vendored tree is stale —
|
||||
# regenerate + commit it (scripts/check-bootstrap-templates.sh runs
|
||||
# this same diff offline in `bun run verify`).
|
||||
diff -r /tmp/template-tree templates/bootstrap/template-repo
|
||||
- if: steps.gate.outputs.publish == 'true'
|
||||
name: Force-push the template repo
|
||||
env:
|
||||
RELEASE_VERSION: ${{ needs.version.outputs.version }}
|
||||
TEMPLATE_TREE_HASH: ${{ needs.version.outputs.template_tree_hash }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
cd /tmp/template-tree
|
||||
git init -q -b main
|
||||
git config user.name "gbrain-release-bot"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add -A
|
||||
# History-less by design: each release force-publishes one commit
|
||||
# whose message binds (gbrain version, vendored tree hash) [C3].
|
||||
git commit -q -m "gbrain v${RELEASE_VERSION} template (tree ${TEMPLATE_TREE_HASH:-unvendored})"
|
||||
# Out-of-band credential: a one-shot GIT_ASKPASS script reads the
|
||||
# PAT from env at prompt time, so the token never rides argv (where
|
||||
# `ps`, runner traces, and error messages echoing the remote URL
|
||||
# could surface it).
|
||||
ASKPASS="$(mktemp)"
|
||||
# shellcheck disable=SC2016 # $1/$TEMPLATE_REPO_PAT are literal on
|
||||
# purpose — they must expand when /bin/sh runs the askpass script at
|
||||
# git's credential prompt, not when this outer shell writes the file.
|
||||
printf '%s\n' '#!/bin/sh' \
|
||||
'case "$1" in Username*) printf "x-access-token\n" ;; *) printf "%s\n" "$TEMPLATE_REPO_PAT" ;; esac' \
|
||||
> "$ASKPASS"
|
||||
chmod +x "$ASKPASS"
|
||||
GIT_ASKPASS="$ASKPASS" GIT_TERMINAL_PROMPT=0 \
|
||||
git push --force "https://github.com/${TEMPLATE_REPO}.git" HEAD:main
|
||||
rm -f "$ASKPASS"
|
||||
|
||||
@@ -87,9 +87,36 @@ jobs:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
# Use the open-source gitleaks CLI, not gitleaks-action@v2: the v2 action
|
||||
# now enforces a paid GITLEAKS_LICENSE (fails the job with "missing
|
||||
# gitleaks license" for accounts it can't validate). The CLI is free, uses
|
||||
# the committed .gitleaks.toml allowlist, and scans the same commit range.
|
||||
- name: Install gitleaks (pinned + checksum-verified)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
VER=8.30.1
|
||||
BASE="gitleaks_${VER}_linux_x64.tar.gz"
|
||||
URL="https://github.com/gitleaks/gitleaks/releases/download/v${VER}"
|
||||
curl -fsSL -o "/tmp/${BASE}" "${URL}/${BASE}"
|
||||
curl -fsSL -o /tmp/gitleaks_checksums.txt "${URL}/gitleaks_${VER}_checksums.txt"
|
||||
( cd /tmp && grep " ${BASE}\$" gitleaks_checksums.txt | sha256sum -c - )
|
||||
tar -xzf "/tmp/${BASE}" -C /tmp gitleaks
|
||||
install /tmp/gitleaks /usr/local/bin/gitleaks
|
||||
gitleaks version
|
||||
- name: Scan for secrets (gitleaks CLI, .gitleaks.toml)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "${{ github.event_name }}" = "pull_request" ]; then
|
||||
RANGE="${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}"
|
||||
else
|
||||
BEFORE="${{ github.event.before }}"
|
||||
case "$BEFORE" in
|
||||
""|0000000000000000000000000000000000000000) RANGE="${{ github.sha }}~1..${{ github.sha }}" ;;
|
||||
*) RANGE="${BEFORE}..${{ github.sha }}" ;;
|
||||
esac
|
||||
fi
|
||||
echo "Scanning commit range: $RANGE"
|
||||
gitleaks detect --redact --no-banner --log-opts "$RANGE"
|
||||
|
||||
verify:
|
||||
# Pre-test gates: privacy/jsonb/source-id/etc + typecheck + admin-build.
|
||||
@@ -165,6 +192,40 @@ jobs:
|
||||
- run: bun install
|
||||
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
|
||||
|
||||
brainbench:
|
||||
# BrainBench memory-conformance gate (Cathedral 2). Hermetic: in-memory
|
||||
# PGLite, zero API keys, ~15s for the full 141-fixture × 3-harness run.
|
||||
# Governance (decision 4): compares HEAD's run against MAIN's committed
|
||||
# baseline via `git show origin/master:evals/brainbench/baselines/main.json`
|
||||
# — a PR cannot rewrite the thing it's compared against. Exit 1 blocks the
|
||||
# merge until the regression is fixed or blessed (justification in the
|
||||
# updated baseline / fixture diff in corpus-bless mode).
|
||||
needs: cache-check
|
||||
if: needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10 # ~15s hermetic run; matches the per-job-timeout hardening (#2254)
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
with:
|
||||
fetch-depth: 0 # the gate needs origin/master's baseline
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bash scripts/ci-brainbench-gate.sh
|
||||
env:
|
||||
BRAINBENCH_OUT: ${{ runner.temp }}/brainbench-result.json
|
||||
- name: BrainBench delta → step summary
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f "${{ runner.temp }}/brainbench-result.json" ]; then
|
||||
bun scripts/render-brainbench-delta.ts "${{ runner.temp }}/brainbench-result.json" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
slow-entity-resolve-perf:
|
||||
# Dedicated runner for the entity-resolve perf test (~159s, single perf
|
||||
# describe with one test that builds 5000+ pages and asserts the NEW
|
||||
@@ -187,6 +248,19 @@ jobs:
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
|
||||
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
|
||||
# (20K-page corpus + ratio guard) shares this runner — same perf-job
|
||||
# shape, runs in parallel with the matrix.
|
||||
- run: bun test test/entity-card-perf.slow.test.ts --timeout=300000
|
||||
# Protocol self-certification: init a scratch brain and run the
|
||||
# conformance kit against gbrain's own stdio server. --synthesize is
|
||||
# safe here: no LLM key in CI, so it asserts the clean `unavailable`
|
||||
# protocol error instead of spending tokens.
|
||||
- name: MEMORY_VERBS conformance (self-certify, stdio)
|
||||
run: |
|
||||
export GBRAIN_HOME="$RUNNER_TEMP/gbrain-conformance"
|
||||
bun run src/cli.ts init --pglite --no-embedding --non-interactive
|
||||
bun run src/cli.ts protocol conformance --synthesize
|
||||
|
||||
test:
|
||||
# Pure matrix shard — no verify, no serial. Each shard runs its slice
|
||||
@@ -242,7 +316,7 @@ jobs:
|
||||
# entry written when tests hadn't actually passed).
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
cache-write:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
|
||||
if: success() && needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
@@ -265,7 +339,7 @@ jobs:
|
||||
# Branch protection (when configured) gates on this single job name.
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
test-status:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, brainbench, test]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
@@ -278,15 +352,16 @@ jobs:
|
||||
SERIAL="${{ needs.serial-tests.result }}"
|
||||
SLOW_EVAL="${{ needs.slow-eval-longmemeval.result }}"
|
||||
SLOW_PERF="${{ needs.slow-entity-resolve-perf.result }}"
|
||||
BRAINBENCH="${{ needs.brainbench.result }}"
|
||||
TEST="${{ needs.test.result }}"
|
||||
echo "cache-check.hit=$HIT"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF test=$TEST"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF brainbench=$BRAINBENCH test=$TEST"
|
||||
if [ "$HIT" = "true" ]; then
|
||||
echo "✓ cache HIT for hash ${{ needs.cache-check.outputs.hash }} — CI green"
|
||||
exit 0
|
||||
fi
|
||||
# Cache miss: every gated job must have succeeded.
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$TEST"; do
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$BRAINBENCH" "$TEST"; do
|
||||
if [ "$r" != "success" ]; then
|
||||
echo "✗ gated job did not succeed (got $r) — CI fail"
|
||||
exit 1
|
||||
|
||||
@@ -4,8 +4,16 @@ This is your install + operating protocol. Claude Code reads `./CLAUDE.md` autom
|
||||
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
|
||||
start here.
|
||||
|
||||
> **Becoming someone's persistent personal agent** (identity + memory + private repo)?
|
||||
> Follow [`BOOTSTRAP_FOR_AGENTS.md`](./BOOTSTRAP_FOR_AGENTS.md) — the `gbrain bootstrap`
|
||||
> flow — instead of the plain install below, then come back here for the operating
|
||||
> protocol. Connecting to an EXISTING remote brain from a laptop agent?
|
||||
> `gbrain connect https://your-host/mcp --token gbrain_xxx --install` (see the MCP
|
||||
> table in [`README.md`](./README.md)).
|
||||
|
||||
## Install (5 min)
|
||||
|
||||
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
|
||||
1. Install gbrain via Bun (the canonical path):
|
||||
```bash
|
||||
curl -fsSL https://bun.sh/install | bash
|
||||
@@ -26,8 +34,8 @@ start here.
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full step-by-step
|
||||
flow (API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
@@ -69,10 +77,10 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
|
||||
and `gbrain eval replay --against base.ndjson`. For public benchmark
|
||||
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
|
||||
<dataset.jsonl>` runs against an isolated in-memory PGLite
|
||||
per question — your `~/.gbrain` is never opened. Full guide:
|
||||
[`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
|
||||
- **Drive the brain to a target health score:** the one-command
|
||||
loop. `gbrain doctor --remediation-plan --json` previews what would be
|
||||
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
|
||||
walks a dependency-ordered plan (sync before extract, embed after
|
||||
@@ -81,22 +89,20 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
keys hit a `max_reachable_score` ceiling and bail with what's missing.
|
||||
Three phase handlers (synthesize / patterns / consolidate) are
|
||||
PROTECTED — only trusted local callers can submit them; MCP cannot.
|
||||
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
|
||||
and the CHANGELOG entry for v0.36.4.0.
|
||||
- **Track a founder/company over time (v0.35.7):** when an entity has
|
||||
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md).
|
||||
- **Track a founder/company over time:** when an entity has
|
||||
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
|
||||
`unit: USD`, `period: monthly` columns), run
|
||||
`gbrain eval trajectory <entity-slug>` for the chronological history
|
||||
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
|
||||
for a four-signal JSON rollup (claim_accuracy / consistency /
|
||||
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
|
||||
same data — read scope, visibility-filtered for remote callers. **v0.40.2.0:**
|
||||
`gbrain think` now uses this substrate automatically on temporal /
|
||||
same data — read scope, visibility-filtered for remote callers.
|
||||
`gbrain think` uses this substrate automatically on temporal /
|
||||
knowledge_update intent (default ON; flip `think.trajectory_enabled=false`
|
||||
to opt out). Migration v82 added `facts.event_type` so non-metric event
|
||||
rows (`meeting`, `job_change`, `location_change`) ride through the same
|
||||
pipeline; pass `kind: 'event'` or `'all'` to `find_trajectory` to query
|
||||
them.
|
||||
to opt out). Non-metric event rows (`meeting`, `job_change`,
|
||||
`location_change`) ride through the same pipeline via `facts.event_type`;
|
||||
pass `kind: 'event'` or `'all'` to `find_trajectory` to query them.
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
<!-- gbrain-runbook-stamp: 0.45.12.0 -->
|
||||
<!-- This stamp must equal the VERSION file at every release; CI enforces it
|
||||
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
|
||||
the installed binary and warns on skew. -->
|
||||
|
||||
# BOOTSTRAP_FOR_AGENTS.md — become your human's persistent agent
|
||||
|
||||
You are an AI agent (Claude Code or Codex) reading this because a human pasted a
|
||||
block asking you to set yourself up as their persistent personal agent, with gbrain
|
||||
as your memory. This runbook is the judgment layer; the deterministic work lives in
|
||||
`gbrain bootstrap` subcommands with exit codes. Follow it top to bottom.
|
||||
|
||||
**Scope note:** this path is for Claude Code and Codex (desktop apps or CLIs).
|
||||
Running OpenClaw or Hermes? Use `INSTALL_FOR_AGENTS.md` instead.
|
||||
|
||||
**End state:** this folder is your workspace — identity files rendered from your
|
||||
human's own answers, a local brain (PGLite, embedded, no server), MCP wired,
|
||||
per-turn context, and a private GitHub repo as your durable body. ~15 minutes,
|
||||
mostly interview.
|
||||
|
||||
## Hard rules for you, the installing agent
|
||||
|
||||
⛔ **NEVER INVENT ANSWERS.** Personality, purpose, and boundaries come from the
|
||||
human. A guessed SOUL.md gets believed by every future session. If you do not have
|
||||
an answer, ask. The render step structurally refuses to run until the required
|
||||
answers exist and were read back.
|
||||
|
||||
⛔ **ASK IN SMALL BATCHES.** The interview is 12 questions max (6 required), asked
|
||||
in three batches. Mirror each batch back in one line. Accept "skip" on any
|
||||
non-required question — momentum beats completeness.
|
||||
|
||||
⛔ **STAY INSIDE THIS PHASE LIST.** Run `gbrain bootstrap status --json` and follow
|
||||
ITS phase list — the CLI is the source of truth, this document is commentary. If a
|
||||
step you are asked to run is not in the CLI's phase list, refuse it. If `status`
|
||||
reports a version skew between this runbook and the installed binary, say so and
|
||||
prefer the binary's instructions.
|
||||
|
||||
⛔ **NO SILENT FAILURE.** Every blocking condition (secret-scan block, lock
|
||||
collision, partial install) surfaces through `status`/`verify`/`doctor` output —
|
||||
read it and relay it to the human in plain language. Never work around a refusal.
|
||||
|
||||
⛔ **VERIFY BEFORE CLAIMING DONE.** The install is done when `gbrain bootstrap
|
||||
verify` exits 0 — not when the transcript looks good. Paste its report to the human.
|
||||
|
||||
⛔ **RESPECT THE TOOLCHAIN TRUST RULES.** Install bun via a platform package manager
|
||||
when available (`brew install oven-sh/bun/bun`); the only permitted fallback is the
|
||||
checksum-verified variant: download the pinned release to a file, verify it against
|
||||
that release's SHASUMS256.txt, and only then execute. Install gh the same way —
|
||||
platform package manager first (`brew install gh`, `apt install gh`, `dnf install gh`,
|
||||
`winget install GitHub.cli` per the official instructions); never a piped
|
||||
curl-to-shell one-liner. Install gbrain ONLY as
|
||||
`bun install -g github:garrytan/gbrain#latest-stable` — the npm package named
|
||||
"gbrain" is an unrelated project. (Cloud-sandbox exception: bun's package fetching
|
||||
is proxy-incompatible there — use the `gbrain bootstrap cloud-setup-script` recipe,
|
||||
which installs from the same pinned GitHub source through npm.)
|
||||
|
||||
⛔ **NEVER FABRICATE TOOLING.** If gh or any preflight binary is missing, blocked
|
||||
by a sandbox egress proxy, or answering 403s, report that through
|
||||
`status`/`doctor` output and follow the cloud-sandbox guidance below. Never
|
||||
hand-roll a gh shim, stub a fake binary into /usr/local/bin, or fake a passing
|
||||
check — a fabricated tool poisons every later verification, and the one time it
|
||||
was tried it masked a real silent-persistence failure. The CLI degrades honestly
|
||||
on its own; your job is to relay, not to bridge.
|
||||
|
||||
## Codex preflight (ChatGPT desktop / Codex CLI only)
|
||||
|
||||
Codex sandboxes command execution. Before starting, tell the human: "I'll need
|
||||
approval to run install commands (bun, gh, gbrain) and to write in this folder —
|
||||
approve those prompts when they appear." If approvals are globally disabled, ask the
|
||||
human to enable workspace-write + network for this session. Count the approval taps
|
||||
you needed; report the count at the end (it feeds the install-time measurement).
|
||||
|
||||
## Phase walkthrough (commentary — the CLI's list wins)
|
||||
|
||||
1. **Preflight.** `git`, `bun`, `gh` present. Install what's missing per the trust
|
||||
rules above: bun via a platform package manager or the checksum-verified download
|
||||
— the checksum-verified install is the ONLY permitted non-package-manager
|
||||
variant; gh via the platform package manager. (On a clean Mac, `git` may trigger
|
||||
the Xcode tools dialog — that download does not count against the 15 minutes,
|
||||
tell the human to let it run.)
|
||||
`gh auth status` — if logged out, the human's ONE manual step:
|
||||
`gh auth login -h github.com -p https -w` (you run it; they click Authorize).
|
||||
Then `gbrain bootstrap status` — it is idempotent and resume-aware; after any
|
||||
partial failure, re-run it and continue where it points.
|
||||
2. **Engine.** `gbrain init --pglite` (2 seconds, no server). Search mode is
|
||||
auto-selected silently (conservative when keyless, tokenmax with an
|
||||
expansion key) and printed with an `[AGENT]` cost matrix — surface that
|
||||
matrix to the human and confirm before running high-volume queries (see
|
||||
INSTALL_FOR_AGENTS.md Step 3.5); they can change it any time with
|
||||
`gbrain search modes`. The one thing to raise here is the OPTIONAL provider
|
||||
key — with no key you run keyless: keyword search plus memory you author
|
||||
yourself through the write tools; everything works, one key upgrades search to
|
||||
semantic and enables automatic fact extraction. Never pressure for a key. If the
|
||||
human provides one, pass it to the CLI prompt — it goes to the 0600 config file,
|
||||
never into the interview answers, never into chat logs you keep.
|
||||
3. **Interview.** `gbrain bootstrap interview --init`, then ask the questions from
|
||||
the bank (the CLI prints them) in three batches, recording each answer verbatim
|
||||
with `--set KEY "value"`. Push once on vague answers to the required questions.
|
||||
Claude Code only: with the final batch, also ask the ONE operational consent —
|
||||
MCP scope. It is not one of the 12 interview questions; consents ride alongside
|
||||
the bank. The choice: project (recommended — any other repo you open cannot
|
||||
read your brain) vs user (your agent everywhere, but any repo you open can
|
||||
reach it — read and write — and two open sessions contend for the database).
|
||||
Record it with
|
||||
`gbrain bootstrap interview --set MCP_SCOPE <project|user>` BEFORE the
|
||||
read-back, so the confirmation covers it. On Codex, skip this question
|
||||
entirely — the wiring step states the Codex reality instead.
|
||||
After the last batch: read ALL answers back in one compact block, ask "Is this
|
||||
the thing you want in the room?", and only then run
|
||||
`gbrain bootstrap interview --confirm <hash>` with the hash `--status` printed
|
||||
for the read-back set. The gate fails if you confirm a set the human never saw.
|
||||
4. **Render.** `gbrain bootstrap render` — identity files appear. Show the human
|
||||
SOUL.md. Existing files are never overwritten (re-runs are safe; `--force`
|
||||
backs up first).
|
||||
5. **Skills.** `gbrain skillpack scaffold --all` — the CLI scaffolds the skill
|
||||
set. Nothing to judge here; relay the output.
|
||||
6. **Wire the harness + register the brain source.** `gbrain bootstrap hooks
|
||||
--harness <detected>` creates `<workspace>/brain` and prints the exact
|
||||
`gbrain sources add <source_id> --path <brain> --force` command for THIS
|
||||
workspace — run it verbatim (don't guess a different id; a guessed id
|
||||
only surfaces as an FK error at `verify` time, by which point a wrong
|
||||
guess also blocks the correct id with an `overlapping_path` error). It
|
||||
also:
|
||||
- Claude Code: installs per-turn hooks ON by default — do NOT ask; loading the
|
||||
brain every turn is the whole point of installing gbrain for your agent. Tell
|
||||
the human it is on and how to turn it off (`GBRAIN_HOOKS=0`, or re-run with
|
||||
`--no-hooks`, or `gbrain bootstrap uninstall`). MCP scope is NOT asked here —
|
||||
`hooks` consumes the MCP_SCOPE answer recorded during the interview.
|
||||
- Codex: registers MCP (`codex mcp add`) and relies on the AGENTS.md protocol —
|
||||
say plainly that Codex gets pull-based context, not per-turn push.
|
||||
Do NOT offer an MCP scope choice: `codex mcp add` has no scope flag, so
|
||||
the registration is always user-global. State it as fact — any repo opened
|
||||
on this machine can reach the brain (read and write) through its MCP
|
||||
tools; the off-ramps are `codex mcp remove gbrain` (registration only) or
|
||||
`gbrain bootstrap uninstall` (full teardown).
|
||||
7. **Private repo.** `gbrain bootstrap repo` — creates a PRIVATE GitHub repo from
|
||||
the workspace, verifies the privacy bit through the API, pushes. If the human
|
||||
started from a repo they created themselves (create-repo-first: an EMPTY private
|
||||
repo under their own account, cloned and opened here), this ADOPTS that repo
|
||||
instead of creating one — verifies it is private and pushes the workspace. A
|
||||
non-empty repo, or one owned by an org, is refused with a clear message (make an
|
||||
empty personal repo, or run `gbrain bootstrap attach` for an existing agent
|
||||
clone). Asks the background-persistence consent (a git post-commit auto-push
|
||||
plus a 30-minute pull job for multi-machine freshness; declining still persists
|
||||
via the per-turn and session-end pushes). If the human has no GitHub or declines:
|
||||
local-only mode with an honest warning; `bootstrap repo` can run any time later.
|
||||
Note: the per-turn/session push stays deferred until this phase records the
|
||||
verified repo, so nothing is ever pushed to an unverified-privacy origin.
|
||||
8. **Verify.** `gbrain bootstrap verify` — the whole contract: brain round-trip
|
||||
through the real write path, graph floor, token sweep, secret scan, repo
|
||||
privacy, hooks smoke, capability report (keyless or keyed). Exit 0 or it is not
|
||||
done. Paste the report. Then relay the first-run tour it prints (three prompts
|
||||
the human should try, starting with restarting the session) AND the hand-off
|
||||
block below it — the ownership line and the cold-start offer are the two
|
||||
things the human must actually understand, not fine print.
|
||||
|
||||
## Machine two
|
||||
|
||||
If this workspace was cloned from an existing agent repo (agent.json says
|
||||
initialized), run `gbrain bootstrap attach` instead of the interview/render/repo
|
||||
phases — it wires this machine (source, hooks, MCP) and verifies. If agent.json
|
||||
says it is an uninitialized template, proceed with the normal flow from phase 1.
|
||||
|
||||
## Cloud sandboxes (claude.ai/code and similar proxied environments)
|
||||
|
||||
**How you know:** `gbrain bootstrap status --json` reports
|
||||
`execution_environment: "cloud-sandbox"` (the CLI detects the documented
|
||||
signals — the CLAUDE_CODE_REMOTE env var, the proxy-injected token
|
||||
placeholder). Trust the CLI's detection over your own guesses.
|
||||
|
||||
**Expected degradations — these are facts to relay, not bugs to bridge:**
|
||||
|
||||
- **No crontab, no surviving background processes.** The VM is reclaimed after
|
||||
inactivity. The scheduled pull is skipped honestly; the per-turn (Stop hook)
|
||||
and session-end pushes carry persistence. Decline nothing, fabricate nothing.
|
||||
- **GitHub GraphQL is always blocked** by the egress proxy, and **REST reaches
|
||||
only repos attached to the session** — a repo created mid-session is NOT
|
||||
attached, so `gbrain bootstrap repo` refuses fast in cloud with the flow
|
||||
that works. Privacy verification falls back to pure git protocol on its own.
|
||||
- **`git push` works only against the session's working branch.** A user PAT
|
||||
does not bypass any of this.
|
||||
- **Only repo-committed files carry into the next session.** `~/.gbrain`,
|
||||
`~/.claude`, and the gitignored `.claude/settings.local.json` evaporate.
|
||||
Hooks therefore live in the COMMITTED `.claude/settings.json` (the CLI
|
||||
writes PATH-resolved, fail-open commands there in cloud); hook config is
|
||||
snapshotted at session start, so hooks written mid-session activate on the
|
||||
NEXT session — say so instead of debugging it.
|
||||
|
||||
**The correct cloud flow:**
|
||||
|
||||
1. The human creates the private repo from a normal machine (or github.com)
|
||||
and opens the cloud session ON that repo.
|
||||
2. The environment's setup script installs the gbrain binary — print it with
|
||||
`gbrain bootstrap cloud-setup-script` and have the human paste it into the
|
||||
environment config (npm-based; bun's fetching is proxy-incompatible there).
|
||||
3. Inside the session: `gbrain bootstrap attach`, then
|
||||
`gbrain bootstrap hooks --harness claude-code` (writes the committed
|
||||
carrier), commit + push, and tell the human the hooks go live next session.
|
||||
|
||||
## Failure modes, and what they actually mean
|
||||
|
||||
| Symptom | Real cause | Fix |
|
||||
|---|---|---|
|
||||
| `interview --status` exits nonzero forever | A required answer is genuinely missing | Ask the human. Do not default it. |
|
||||
| Render refuses with unresolved tokens | Interview incomplete or a template edit broke a token | Finish the interview; `status` names the tokens. |
|
||||
| `verify` fails the magic-moment check | The fact never landed (keyless: the Facts fence was not written) | Re-run the write step it names; check `gbrain doctor`. |
|
||||
| Secret-scan block on push | A credential-shaped string in a tracked file | Fix or allowlist deliberately (`.gbrain-scan-allow`); never force. |
|
||||
| "bootstrap already running (pid N)" | A concurrent bootstrap holds the lock | Wait or investigate that pid; the lock self-clears when stale. |
|
||||
| Brain tools fail with a lock error | Another live session's serve owns the database | Close the other session; sequential use is the v1 contract. |
|
||||
| Hook reports "brain context unavailable" | serve not running or degraded | `gbrain doctor` names it; hooks fail open by design. |
|
||||
| gh answers 403 "not enabled for this session" | Cloud proxy scoping — the repo is not attached to the session | Expected in cloud; the visibility ladder falls back to git protocol. NEVER shim gh. |
|
||||
| "crontab: command not found" / cron skipped | Containers and cloud sandboxes ship without a scheduler | Expected; event-driven pushes cover it — the skip message says exactly this. |
|
||||
| A turn shows "workspace push is FAILING" | The background push is refusing (visibility, secret-scan, or network reasons) | Run `gbrain doctor`; the banner repeats every 30 min until fixed. |
|
||||
|
||||
## Hand off
|
||||
|
||||
Two things the human must UNDERSTAND before you finish — say them plainly, in
|
||||
this order, and confirm they landed:
|
||||
|
||||
1. **They own the brain.** Every memory you keep is a markdown file in THEIR
|
||||
private GitHub repo — name the URL. Owning it means: they can read it any
|
||||
time, take it to a second machine (`gbrain bootstrap attach`), or delete the
|
||||
repo and the brain is gone. If they went local-only, say that instead, with
|
||||
`gbrain bootstrap repo` as the any-time upgrade.
|
||||
2. **The first skill to run is cold-start.** An empty brain is a database; a
|
||||
filled one is a memory — and every flagship skill (book-mirror, briefings,
|
||||
meeting prep) only becomes magical once the brain holds their real life.
|
||||
OFFER to run the cold-start skill now: it imports Gmail, calendar, and
|
||||
contacts through ClawVisor (clawvisor.com — an OAuth vault; you never hold
|
||||
raw tokens), or offline archives (Google Takeout, a notes folder) if they
|
||||
prefer no third-party gateway. Every phase is consent-gated and
|
||||
independently valuable — they can stop after any one. If they say "later",
|
||||
that is a complete install; they can say "fill my brain" any time.
|
||||
|
||||
Then the routine facts: the capability mode (keyless vs keyed), and the three
|
||||
commands they will actually reuse (`gbrain doctor`, `gbrain bootstrap verify`,
|
||||
`gbrain sources push`). Then delete nothing — this runbook was fetched, not
|
||||
installed.
|
||||
+863
-6
@@ -2,6 +2,869 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.45.12.0] - 2026-08-13
|
||||
|
||||
**Hermes joins the tested-install club: a real-binary harness now proves gbrain works inside Hermes, and `gbrain friction diff` tells you whether an install problem is the agent's or ours.**
|
||||
|
||||
GBrain has long said "works with OpenClaw and Hermes." For OpenClaw that claim was
|
||||
backed by tests; for Hermes it was backed by hope. This release closes that gap with a
|
||||
full end-to-end harness: a Hermes runner for the claw-test friction lab, a real-binary
|
||||
"door" test that registers gbrain into an actual Hermes install over MCP and asks it to
|
||||
recall a seeded fact, and a CI job that installs a pinned Hermes release and runs the
|
||||
door on demand. Every Hermes CLI behavior the harness relies on was pinned by observing
|
||||
a real install — the flag-order traps, the interactive prompts, the exit-code quirks —
|
||||
and those observations ship as documentation so your own Hermes setup benefits too.
|
||||
|
||||
The live claw-test lane also got honest: it now stages the scenario workspace before the
|
||||
agent starts and verifies real outcomes after it finishes, so an agent that does nothing
|
||||
and exits cleanly finally FAILS the run instead of passing it. And with two runners in
|
||||
the registry, the new `gbrain friction diff --base openclaw --compare hermes` turns
|
||||
friction reports into a comparison instrument: pain unique to one agent is that agent's
|
||||
contract problem; pain common to both is ours.
|
||||
|
||||
## To take advantage of v0.45.12.0
|
||||
|
||||
`gbrain upgrade` is enough — no schema migration.
|
||||
|
||||
1. **Running Hermes?** Wire gbrain in with one command (full guide at
|
||||
`docs/mcp/HERMES.md`, including the non-obvious flag-order and prompt gotchas):
|
||||
```bash
|
||||
printf 'Y\n' | hermes mcp add gbrain --env GBRAIN_HOME=$HOME --connect-timeout 60 --command $(which gbrain) --args serve
|
||||
hermes mcp test gbrain
|
||||
```
|
||||
2. **Want the friction lab on your own agent?**
|
||||
```bash
|
||||
gbrain claw-test --live --agent hermes # or --agent openclaw
|
||||
gbrain friction diff --base openclaw --compare hermes
|
||||
```
|
||||
3. **If anything looks wrong,** file an issue at https://github.com/garrytan/gbrain/issues
|
||||
with `gbrain doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Added**
|
||||
- Hermes runner for the claw-test harness: `gbrain claw-test --live --agent hermes` drives a real Hermes install headlessly (`$HERMES_BIN` override supported; `--list-agents` shows availability for both runners).
|
||||
- `gbrain friction diff --base <run-or-agent> --compare <run-or-agent>`: cross-agent friction comparison with unique-to-each and changed sections, count deltas, and a compatibility banner that warns when runs cover different scenarios or versions. `--json` for machines.
|
||||
- Real-binary Hermes install door e2e (`test/e2e/install-real-hermes.serial.test.ts`): registers this checkout's gbrain into a hermetic Hermes home via a live MCP handshake (110 tools discovered), verifies both the CLI and direct-config registration surfaces, and proves recall of a seeded fact in a paid one-shot turn. Triple-gated so it can never burn tokens by accident.
|
||||
- Label-gated `hermes-door` CI job: installs a digest-and-tag-pinned Hermes release, refuses to go green if nothing actually ran, and uploads scrubbed evidence on failure.
|
||||
- Per-client MCP docs for Hermes (`docs/mcp/HERMES.md`) and OpenClaw (`docs/mcp/OPENCLAW.md`), plus a dev-facing pin of every observed Hermes CLI behavior (`docs/mcp/HERMES-CLI-PIN.md`) and an install snippet in `INSTALL_FOR_AGENTS.md`.
|
||||
- Generic agent-workspace compatibility test pinning the documented "any repo with a workspace" install flow (detection, scaffold additivity, resolver health).
|
||||
|
||||
**Changed**
|
||||
- claw-test live mode now stages the scenario before the agent runs (fresh-install: brain pages + routing file + init; upgrade: seed replay) and verifies outcomes after it exits — doctor health, a scenario-declared query returning results, expected files existing, and for upgrades a non-mutating schema-version probe that a do-nothing agent cannot satisfy.
|
||||
- The brief handed to live agents now matches the current CLI exactly (extract argument shape, doctor status vocabulary), and bare `gbrain` inside a live run resolves to the harness's own binary via a per-run PATH shim.
|
||||
- Every claw-test run opens and closes with a machine-readable marker carrying the agent name and scenario, so friction analytics can resolve runs by agent; scripted runs are now labeled `scripted` instead of borrowing an agent's name.
|
||||
- Scenario oracle configuration is validated on load — misdeclared oracles fail loudly instead of silently not being enforced.
|
||||
|
||||
**Fixed**
|
||||
- claw-test now works out of the box when gbrain runs from source (`bun run src/cli.ts`): child invocations resolve to a real gbrain launcher instead of the bun runtime itself, which previously made the default harness unusable outside compiled builds.
|
||||
- Upgrade-scenario runs in BOTH modes fail loudly when the scenario ships no seed dump, instead of quietly initializing a current database and reporting an "upgrade" that never exercised a migration.
|
||||
- Every harness child process now runs under a wall-clock timeout, and live-agent timeouts kill the agent's whole process tree — a hung child no longer wedges a run (or a CI job) forever.
|
||||
- Agent-side friction entries now survive the run's tempdir cleanup: they merge into your friction log before the workspace is deleted, so `friction render` and `friction diff` finally see both halves of a live run.
|
||||
- `claw-test --list-agents` no longer races CLI teardown; output is complete and ordered.
|
||||
- Live runs keep the agent's gbrain children pointed at the run's own hermetic brain even when the surrounding shell exports a database-pointing environment variable — the harness's verification and the agent's work can no longer land in two different places.
|
||||
- The test real-name guard now correctly distinguishes the public Hermes platform (documented and tested) from private deployment names (still banned).
|
||||
## [0.45.11.0] - 2026-08-12
|
||||
|
||||
**The install now ends by telling you the two things that matter: you own the brain, and here's the first skill to run.** A working install used to finish on a health report and three tour prompts — technically complete, but a new user walked away without the two facts that make gbrain worth trusting and worth using. Now `gbrain bootstrap verify` ends with a hand-off: **what you own** (every memory is a markdown file in YOUR private GitHub repo — read it, take it to a second machine, delete it and the brain is gone; or the local-only variant with the one command that gives it a durable home) and **what to do next** (run the `cold-start` skill — say "fill my brain" and your agent imports your Gmail, calendar, and contacts through ClawVisor, an OAuth vault so the agent never holds raw tokens, or offline archives like Google Takeout, one consented phase at a time).
|
||||
|
||||
The structural fix underneath: `cold-start` — the skill designed exactly for "I just installed this, now what?" — was excluded from the downstream skill bundle, so the paste-in install audience it was written for could never scaffold it. It's now bundled, it's the #1 recommended skill (ahead of the book-mirror flagship, because every flagship skill only becomes magical once the brain holds your real life), and a new drift guard fails CI if any recommended skill ever becomes unscaffoldable again.
|
||||
|
||||
To take advantage of v0.45.11.0: existing installs can run `gbrain skillpack scaffold cold-start` and say "fill my brain"; fresh installs get the full hand-off automatically.
|
||||
|
||||
### Added
|
||||
- **The verify hand-off block.** On PASS, `gbrain bootstrap verify` prints (and returns in `--json` as `handoff`) the ownership statement — with the actual repo URL, or the local-only variant pointing at `gbrain bootstrap repo` — followed by the cold-start next action. The runbook's Hand off section now instructs the installing agent to make both land ("say them plainly, confirm they landed") and to OFFER running cold-start on the spot.
|
||||
- **`cold-start` ships in the downstream bundle** (61 skills) and leads the recommended set, so the post-install advisory, `gbrain advisor`, and `gbrain skillpack scaffold --all` all surface it. Its prior bundle exclusion ("host onboarding flow") predated the personal-agent bootstrap and was reversed deliberately.
|
||||
- **Recommended-set drift guard**: every recommended slug must be scaffoldable from the plugin bundle — recommended-but-unscaffoldable is a dead-end call-to-action and now fails the suite.
|
||||
|
||||
### Changed
|
||||
- README's Codex and Claude Code paths spell out the same two follow-ups after the click moment: ownership (markdown in a repo you own) and cold-start as the first skill, with ClawVisor named as the credential path and offline archives as the no-gateway alternative.
|
||||
|
||||
**Also in this release — the first-five-minutes DX wave** (re-versioned from an unpublished 0.45.9.0 after the release queue moved):
|
||||
|
||||
**The first five minutes stop making you think.** We built a real-terminal harness that drives the actual install the way a new user does — every picker, prompt, silence window, and line of copy — and then fixed what it surfaced. Keyless `gbrain init` used to dead-end at an error before it created anything; now it just works, keyless, and says so. A fresh brain used to scroll ~240 lines of internal migration names; now it prints one line. The success screen used to bury the one thing to do next under eight competing calls to action; now the copy-paste memory demo is the last, obvious thing on screen. And the "here's the magic" moment in the README now points at the trick that only a brain can do — tell it something, restart, ask for it back — instead of a question your identity files answer for free.
|
||||
|
||||
Under the hood: the upgrade nudge now compares the version you're actually running (a stale or foreign cache can't tell you to upgrade to something you already have), a broken settings file makes the installer stop and tell you rather than quietly replace it, and `gbrain init --supabase` fails loudly in a script instead of pretending it worked. Every fix landed with a test, and a two-model adversarial review pass (Claude + Codex) caught a cluster of follow-on issues in the fixes themselves — a keyless upgrade command that pointed at a rejected path, a compiled-binary detection that broke for renamed binaries — which are fixed here too.
|
||||
|
||||
To take advantage of v0.45.11.0: nothing to do — `gbrain self-upgrade` (or your next `gbrain` invocation's upgrade nudge) brings you current, and the improvements are all in the install/first-run path a new brain hits automatically.
|
||||
|
||||
### Added
|
||||
- **A real-PTY DX exploration harness** (`test/helpers/tty-harness.ts` + `scripts/dx-explore.ts`). It spawns any CLI — gbrain, `claude`, `codex` — under a true pseudo-terminal, timestamps every output burst, and turns silence windows into a measurable stall report, so "the user stared at a frozen screen for nine seconds" is an artifact, not a hunch. A `drive` mode lets an agent steer a live TUI across separate tool calls. Developer instrument only; transcripts are gitignored and nothing in the shipped product depends on it.
|
||||
|
||||
### Changed
|
||||
- **Keyless is now the default when you have no embedding key**, on both the interactive and scripted paths: `gbrain init` completes with a loud, honest "keyless mode — keyword search plus memory your agent writes; everything works" notice instead of exiting with an error. A near-miss key typo still fails loudly (so a fat-fingered `OPENAPI_API_KEY` isn't silently buried). Multiple keys auto-pick the canonical default rather than refusing.
|
||||
- **Fresh-brain init prints one schema-setup line** instead of the full migration replay; upgrades keep the per-migration detail where it has diagnostic value (`GBRAIN_MIGRATE_VERBOSE=1` restores it).
|
||||
- **The init success screen leads with one action** — the three-command memory demo, last on screen — with import/scale-up/health collapsed into a single terse footer and the recommended-skills advisory reduced to a human-voiced pointer.
|
||||
- **The provider picker offers "continue keyless" explicitly** and probe-gates a local Ollama daemon (a running daemon that hasn't pulled the model is annotated, not silently selected); a bare Enter never picks a broken local provider.
|
||||
- **The upgrade nudge tells the truth about your binary**: it compares the running version to the latest and prints the running version, so a stale or foreign-written cache can't nag about an upgrade you already have. The raw machine marker stays off an interactive human's screen (override with `GBRAIN_FORCE_UPGRADE_MARKER=1` for PTY-based agent harnesses that parse it).
|
||||
- **Copy honesty pass**: provider capabilities are attributed per provider (OpenAI unlocks semantic search + fact extraction; Voyage semantic search; Anthropic fact extraction — it has no embeddings API); the install-time estimate reads ~15 minutes for the personal-agent path (~30 for the always-on setup); the first-run tour says to restart first and frames the genuine cross-session round-trip.
|
||||
|
||||
### Fixed
|
||||
- **A parse-broken `.claude/settings.local.json` aborts the hooks write** with a fix-and-re-run message instead of being replaced — your permissions and allowlist are never silently dropped.
|
||||
- **`gbrain init --supabase` in a non-interactive shell fails loudly** (exit 1, names the `--url` escape hatch) instead of the old silent exit-0 that wrote no config.
|
||||
- **`gbrain bootstrap hooks` with a missing harness CLI** now still installs per-turn hooks and reports the phase as partial (so a resuming agent re-runs it once the CLI is on PATH) instead of leaving a false "wire complete".
|
||||
- **`gbrain bootstrap interview --set/--skip` after a confirmation** warns that it voided the read-back instead of failing silently later at render.
|
||||
- Review-pass self-fixes: the keyless upgrade hint now names the re-init command that actually works (not the schema-sizing field `config set` rejects); compiled-binary detection for the detached update refresh no longer breaks for a renamed/official-named binary; the DX harness scrubs copied credentials even on interrupt and reaps the child's whole process tree.
|
||||
## [0.45.10.0] - 2026-08-13
|
||||
|
||||
**21 more community and maintainer bug fixes. Search answers get more complete, sync gets safer, and doctor learns to warn you before a provider dies.**
|
||||
|
||||
This wave continues the v0.45.8.0 cleanup: no new product surface, just fixes. The
|
||||
standouts: pages created by the idea-extraction cycle were invisible to search (they
|
||||
were written without search chunks) and now show up like everything else, with a repair
|
||||
path for existing brains. Query caching now keys on your detail setting, so a compact
|
||||
answer is never served to a full-detail request. And doctor now warns you loudly if your
|
||||
brain is pinned to an embedding provider that has announced a shutdown, weeks before it
|
||||
happens instead of after.
|
||||
|
||||
Also riding: the rerank budget fix that landed directly this week. Contributed by @javieraldape.
|
||||
|
||||
## To take advantage of v0.45.10.0
|
||||
|
||||
`gbrain upgrade` is enough. No schema migration.
|
||||
|
||||
1. **Upgrade and check:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain doctor
|
||||
```
|
||||
2. **If doctor now warns about your embedding provider,** that is the new sunset check
|
||||
doing its job. It names the provider, the date, and the migration command.
|
||||
3. **Heal previously-invisible atom pages:**
|
||||
```bash
|
||||
gbrain embed --stale
|
||||
```
|
||||
4. **Things to watch:** the query cache key version moved, so the first re-ask of a
|
||||
cached question is a one-time cache miss. If anything else looks wrong, file an issue
|
||||
with `gbrain doctor` output: https://github.com/garrytan/gbrain/issues
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Search and recall**
|
||||
- Atom pages produced by the extraction cycle are chunked and embedded like every other page, so they appear in search results. Contributed by @awilhite.
|
||||
- `embed --stale` detects and heals pages that have content but no chunks. Contributed by @Masashi-Ono0611.
|
||||
- The query cache folds the detail knob into its key, so compact and full-detail answers never cross. Contributed by @time-attack.
|
||||
- Rerank budget failures are bucketed under their real cause instead of "unknown". Contributed by @javieraldape.
|
||||
|
||||
**Sync, import, and write-through**
|
||||
- Deferred link extraction above the size gate is consumed instead of dropped. Contributed by @time-attack.
|
||||
- Import error summaries name the failing table and constraint. Contributed by @bo-developing.
|
||||
- Write-through honors the page's recorded source path instead of recomputing it. Contributed by @JonMcCutchen.
|
||||
- The managed filing-rules block renders each repo's own taxonomy, not the bundled default. Contributed by @dovstern.
|
||||
- Timeline extraction no longer splits on bare hyphens inside link labels. Contributed by @time-attack.
|
||||
- Export scopes tag and raw-data sidecar reads to the page's source. Contributed by @alexey-metaengage.
|
||||
- Cross-source link targets survive an engine migration. Contributed by @RerankerGuo.
|
||||
|
||||
**Doctor and diagnostics**
|
||||
- A damaged PGLite store is reported as store damage, with runtime problems kept separate, and the verdict requires positive evidence. Contributed by @time-attack.
|
||||
- New check: brains pinned to an embedding provider with an announced shutdown get a loud warning with the migration path. Contributed by @time-attack.
|
||||
- Source listing distinguishes unset federation from explicit false. Contributed by @dovstern.
|
||||
- `put_page` reports push state honestly instead of implying success. Contributed by @dovstern.
|
||||
- Flow-style skill triggers parse correctly in skill health checks. Contributed by @RerankerGuo.
|
||||
- Sync-failure records auto-skipped as chronic stay visible to doctor until a human resolves them. Contributed by @RerankerGuo.
|
||||
|
||||
**Autopilot and agents**
|
||||
- The drain worker no longer self-deadlocks at concurrency=1, and its DB reconnect logic is shared with queue operations. Contributed by @time-attack.
|
||||
- Stale-lock reaping ignores foreign PIDs it did not create. Contributed by @javieraldape.
|
||||
- Agent jobs resolve their brain source at submit time, not execution time. Contributed by @Masashi-Ono0611.
|
||||
|
||||
**OAuth**
|
||||
- Dynamic client registration accepts `token_ttl_seconds`, clamped to admin policy, and an unset TTL cap now derives from `--token-ttl` instead of a permissive default. Contributed by @time-attack.
|
||||
|
||||
**Models**
|
||||
- The claude-cli recipe lists the Claude 5 family ids the CLI already serves, with pins. Contributed by @clement0909472.
|
||||
|
||||
**For contributors**
|
||||
- The CLI flag registry, one wave rider test, and the bootstrap version stamps were refreshed as part of assembly.
|
||||
|
||||
## [0.45.9.0] - 2026-08-12
|
||||
|
||||
**Your agent's memory keeps saving itself — even in a cloud sandbox, even on `/exit`, and it tells you the moment it can't.** The paste-in personal-agent install now works first-class in Claude Code's cloud environment, not just on a laptop. The persistence lane got three fixes that matter whether you're local or in the cloud: the workspace push now verifies repo privacy through a portable ladder that keeps working when the sandbox blocks the GitHub API, it runs after every turn (not only at session end, which the harness never fires on `/exit`), and a failed push surfaces on your next turn instead of failing in silence. Setup adapts to where it runs — no more scheduled-job errors on hosts without a scheduler, and no half-created repos in an environment that can't push them.
|
||||
|
||||
To take advantage of v0.45.9.0: upgrade and re-run `gbrain bootstrap verify` on each machine — it re-attests the install and now reports the execution environment and any push-health or hygiene issue with the exact one-line fix. Existing installs pick up the per-turn push and the new verification automatically on the binary update; no re-render needed. If you run in a cloud sandbox, `gbrain bootstrap cloud-setup-script` prints the environment setup recipe, and `gbrain bootstrap status --json` now tells you which environment you're in.
|
||||
|
||||
### Added
|
||||
- **Execution-environment detection** — `local`, `cloud-sandbox`, or `ephemeral-container`. Bootstrap, the doctor, and the runbook branch on it so each environment gets honest behavior and honest messages. `gbrain bootstrap status --json` and `gbrain bootstrap verify` both report it.
|
||||
- **Per-turn workspace persistence.** A debounced, detached push runs after each assistant turn (default every 5 minutes locally, every turn in a reclaimed-VM cloud sandbox), closing the gap where a session that ends on `/exit` — which never fires the session-end hook — could strand committed work. Off-ramp: `GBRAIN_STOP_PUSH=0`; cadence: `GBRAIN_STOP_PUSH_DEBOUNCE_MIN` or `gbrain config set hooks.stop_push_debounce_min <n>`.
|
||||
- **Same-session push-failure notice.** When a background push is refused or fails, the next turn surfaces it both to the agent and to you directly (not buried where only the model sees it), re-announced at most every 30 minutes until it clears. `gbrain doctor` and `gbrain bootstrap status` name the failing workspace and the fix.
|
||||
- **`gbrain bootstrap cloud-setup-script`** — prints the ready-to-paste cloud environment setup script that installs the gbrain binary into the environment's cached filesystem so it survives across sessions.
|
||||
- **`bootstrap_durability_job` doctor check** — presence *and* liveness of the optional background-persistence job, so a job that exists on disk but no longer runs is reported instead of certified healthy.
|
||||
|
||||
### Changed
|
||||
- **Repo-privacy verification is now a portable ladder** (`src/core/repo-visibility.ts`), replacing three separate probes with one: it checks via the GitHub REST API first, then falls back to pure git protocol so verification keeps working where a sandbox proxy blocks the API. It fails closed in both directions — an origin that can't be proven private is refused, and a proven-public origin is always refused. Fresh private verdicts are cached briefly to keep the per-turn push cheap. Escape hatch for self-hosted git you trust (each use warns): `--allow-unverified-remote`, `GBRAIN_ALLOW_UNVERIFIED_REMOTE=1`, or `gbrain config set push.allow_unverified_remote true`; the escape hatch only relaxes an *unverifiable* verdict, never a proven-public one.
|
||||
- **Cloud sandboxes get a committed hook carrier.** Because a cloud session starts from a fresh clone and never sees the machine-local settings file, cloud installs write hooks into the repo-committed `.claude/settings.json` with a PATH-resolved, fail-open command; local installs keep the gitignored settings file, and the writers guarantee one event never fires from both.
|
||||
- **Background-persistence copy tells the truth.** The optional job is a git post-commit auto-push plus a 30-minute freshness pull; the interview, docs, and templates now describe exactly that. On a host without a scheduler the pull is skipped with an honest note rather than a failed-install warning.
|
||||
- The installing-agent runbook gains a hard rule against fabricating tooling (no hand-rolled `gh` shims), a cloud-sandbox section, and the honest degradation matrix for a proxied environment.
|
||||
|
||||
### Fixed
|
||||
- `gbrain bootstrap uninstall` now tears down the background-persistence wiring it installed (scheduled job, the untracked auto-push hook, credential wiring) instead of leaving it behind; the committed helper and agent-rules stay, since those are your repo's content.
|
||||
- Machine-specific harness wiring (`.mcp.json`, hook-settings backups) is gitignored so it can't be committed into the private brain repo; `gbrain bootstrap verify` warns and gives the one-line fix for installs that already committed it.
|
||||
- Repo creation is refused inside a cloud sandbox with the flow that actually works (create the repo elsewhere, open the session on it, `gbrain bootstrap attach`) instead of leaving a half-created, unpushable repo.
|
||||
- Push-status is tracked per workspace, so with more than one brain workspace on a machine, one workspace's success can no longer mask another's failed pushes.
|
||||
- Hardening pass (both an in-house and a cross-model adversarial review): the privacy ladder never treats an ambiguous authentication challenge as proof a repo is private, the per-turn retry can't turn into an every-turn network storm, remote-supplied text is sanitized before it reaches any agent- or user-visible surface, and stale state from a deleted workspace no longer re-fires notices forever.
|
||||
|
||||
## [0.45.8.0] - 2026-08-12
|
||||
|
||||
**25 community bug fixes in one wave. Your MCP server, sync, and doctor all get more careful.**
|
||||
|
||||
This release is all fixes, no new surface. 24 community contributors sent small, tested
|
||||
bug fixes over the past weeks. Each one was reviewed, tested in isolation against a real
|
||||
checkout, checked by an adversarial second reviewer, security reviewed, and then tested
|
||||
again as one combined branch. The themes: the MCP server now handles edge-case inputs
|
||||
the way an agent expects, sync and import stop losing or misplacing data in rare
|
||||
situations, and doctor stops crying wolf on healthy setups.
|
||||
|
||||
If you connect an agent to gbrain over MCP, or you sync a brain repo with unusual file
|
||||
names, non-English content, or multiple sources, this release removes a set of paper
|
||||
cuts you may have already hit.
|
||||
|
||||
## To take advantage of v0.45.8.0
|
||||
|
||||
`gbrain upgrade` is enough. These are behavior fixes with no schema migration.
|
||||
|
||||
1. **Upgrade and verify:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain doctor
|
||||
```
|
||||
2. **If doctor output changed for you,** that is likely the point: several checks
|
||||
(supervisor, PGLite store health, base-URL hints) now report accurately where they
|
||||
previously false-alarmed.
|
||||
3. **If anything looks wrong,** file an issue at https://github.com/garrytan/gbrain/issues
|
||||
with `gbrain doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**MCP server correctness**
|
||||
- `sources_add` over a remote transport now rejects a caller-supplied path outright instead of silently ignoring it. Contributed by @gregario.
|
||||
- Stdio serve advertises the tools the caller can actually use. Contributed by @gregario.
|
||||
- All stdout logging routes to stderr under stdio MCP, keeping the protocol stream clean. Contributed by @BenSheridanEdwards.
|
||||
- Null and empty-string optional params are treated as absent at dispatch. Contributed by @SeanGearin.
|
||||
- File ops (`file_list`, `file_upload`) use the connected engine instead of the global DB singleton, so they work on every configured engine. Contributed by @dpaluy.
|
||||
- Stdio serve honors the `.gbrain-source` dotfile. Contributed by @javieraldape.
|
||||
|
||||
**Sync and import data safety**
|
||||
- Global sync anchors only move for the brain repo they describe, so a second repo can no longer skip another repo's pending imports. Contributed by @smdesai27.
|
||||
- Sync never writes a baseline commit over an already-populated repo. Contributed by @NidTamil.
|
||||
- Git C-style-quoted paths (quotes, backslashes, unicode escapes) unquote correctly in the sync manifest. Contributed by @SergeyShol.
|
||||
- Malformed YAML frontmatter is rejected with a clear error instead of importing garbage. Contributed by @javieraldape.
|
||||
- Paths that fail once but succeed on a later run clear their failure record. Contributed by @bo-developing.
|
||||
- Autopilot resolves the gbrain CLI on Windows via PATH enumeration instead of assuming a POSIX shell. Contributed by @veltri-23.
|
||||
- Ctrl-C cleanly terminates bulk commands using the shared progress reporter. Contributed by @javieraldape.
|
||||
|
||||
**Engines and search**
|
||||
- PGLite batches code-edge inserts below the bind-parameter limit, fixing silent data loss on large code graphs. Contributed by @kyle944.
|
||||
- The configured FTS language survives schema replay, so non-English brains no longer revert to English tokenization on re-init. Contributed by @paul-0320.
|
||||
- Hyphenated Qwen3-Embedding model ids resolve their dimensions correctly. Contributed by @mikez93.
|
||||
|
||||
**Doctor and diagnostics**
|
||||
- Doctor surfaces abandoned PGLite stores left behind after an engine migration. Contributed by @Masashi-Ono0611.
|
||||
- The base-URL hint uses a real models-probe classifier instead of guessing /v1. Contributed by @brettdavies.
|
||||
|
||||
**Models and cycle**
|
||||
- Sonnet 5, Fable 5, and Opus 4.8 are in the synthesize context map and brainstorm output caps. Contributed by @p3ob7o.
|
||||
- Truncated or degenerate significance verdicts are no longer cached permanently. Contributed by @Masashi-Ono0611.
|
||||
- The `models.subagent` config path goes through the same capability checks as every other model path. Contributed by @Masashi-Ono0611.
|
||||
- `takes add` resolves the target page before writing markdown. Contributed by @ghizi.
|
||||
- A shipped filing rule that bound a personal folder name to a sensitive category is gone. Contributed by @Masashi-Ono0611.
|
||||
- BrainBench eval defaults resolve from the package root, so evals run from any working directory. Contributed by @philip-rossoneri.
|
||||
- The OpenClaw plugin-loader E2E inspects the real runtime. Contributed by @arisgysel-design.
|
||||
|
||||
**For contributors**
|
||||
- The committed CLI flag registry, the cycle-sync test mocks, and two test fixtures were updated to match the combined branch.
|
||||
|
||||
## [0.45.7.0] - 2026-08-12
|
||||
|
||||
**Ambient recall: your brain shows up at the moments that matter, not just when you ask.** Long-lived agents lose the thread at session boundaries — a fresh start with no warm context, a compaction that drops verbatim detail nothing rehydrates, a heartbeat that re-derives state from scratch. This release adds two new memory verbs that assemble a budget-packed, zero-LLM bundle of exactly what a boundary needs, and wires them into the agent's lifecycle hooks so a warm pack lands automatically at session start and after compaction. It's opt-in, fail-open, and reaches every host: Claude Code gets it pushed through hooks; Codex and any MCP host pull the same two verbs at their own boundaries. Whether your brain is embedded (PGLite) or managed (Postgres), the ambient value is the same.
|
||||
|
||||
### Added
|
||||
- **`context_pack` — a deterministic, budget-packed boundary bundle.** `gbrain context-pack --entities a,b,c --budget-tokens 4000` returns entity cards, open threads, and top facts for a set of standing entities, trimmed to the token budget (cards first, then facts) with no model call in the path — sub-second on a large brain. Response reports `budget_used` and `dropped_count`. World-visible by default; private facts are included only for a local trusted caller that passes `--include-private`, and never over a remote connection.
|
||||
- **`delta` — cheap "what changed since".** `gbrain delta --since <ISO8601>` returns only the pages, facts, and thread changes newer than a timestamp — the right shape for a heartbeat that wants to maintain warm state in proportion to what changed, not re-read everything. Pass a stable `--session-id` and each call advances a per-session cursor so the next wake sees only what's new, with at-least-once delivery when a change tail spills past the budget.
|
||||
- **Boundary runtime for Claude Code.** Session start injects a warm context pack; a pre-compaction hook banks the window's standing entities so the session that resumes after a compaction rehydrates what the summary lost. Every boundary hook fails open and honors `GBRAIN_HOOKS=0`.
|
||||
- **Ambient-recall guide + published latency classes.** New `docs/guides/ambient-recall.md` maps where each verb belongs — `entity` per message, `context_pack`/`delta` at boundaries, `synthesize` never in the ambient path — with per-harness recipes. The memory-verbs protocol doc now carries a latency table for all seven verbs.
|
||||
|
||||
### Changed
|
||||
- The frozen memory-verb set grows from five to seven — `context_pack` and `delta` join `recall`/`remember`/`entity`/`synthesize`/`forget`. The wire protocol is unchanged: all seven verbs stamp `protocol_version: 1`, so existing harnesses keep working untouched and simply gain two tools.
|
||||
|
||||
### Fixed
|
||||
- `gbrain delta --session-id <id>` no longer hangs after printing its response — the CLI now exits cleanly on first wake (a background cleanup task raced process teardown). This is the exact command the heartbeat template tells agents to run.
|
||||
- On a Postgres brain whose config carries a leftover local database path, the pre-compaction hook now degrades cleanly instead of probing a local socket that has no server behind it — matching the session-start hook's behavior.
|
||||
- The `--surface verbs` startup banner now reports the actual verb count instead of a hardcoded five.
|
||||
|
||||
### Hardening
|
||||
- The boundary behavior is now pinned end to end, not just in units: a real spawned serve answers the compact→session-start warm-pack round trip over its real socket; a real stdio MCP session on `--surface verbs` advertises and serves exactly the seven verbs fail-closed; the new verbs are exercised over real HTTP with per-token session-cursor isolation; keyset pagination and the session-cursor table are parity-pinned on real Postgres; migration shape, sub-second latency gates, CLI invocations, and a live-Codex boundary-call check round it out (~55 new tests).
|
||||
|
||||
To take advantage of v0.45.7.0: upgrade with `bun install -g github:garrytan/gbrain#latest-stable`. A schema migration runs automatically on first use — a new per-session cursor table, additive, no existing data touched. Codex and other MCP hosts see the two new verbs immediately; Claude Code installs pick up the boundary hooks on the next `gbrain bootstrap`. Read `docs/guides/ambient-recall.md` for where each verb belongs and how to wire your heartbeat to `delta`.
|
||||
## [0.45.6.0] - 2026-08-12
|
||||
|
||||
**Seventeen new production skills, distilled from a 324-skill audit of a mature personal-agent deployment.** The built-in pack grows from ~52 to 69 skills and picks up the trust disciplines a memory product lives or dies by: corrections that fix the source instead of papering over it, a confirmation gate before anything irreversible, claim verification before anything ships, an ingest gate that stops duplicate and misfiled pages at the door, and a sanitization procedure for turning a personal brain into a team brain. Every import was adversarially reviewed, privacy-scrubbed onto generic placeholders, pinned to its upstream source, and shipped with routing fixtures.
|
||||
|
||||
### Added
|
||||
- **Trust layer:** `correction-pipeline` (root-cause every user correction across a 7-class error taxonomy and fix the contaminated source), `data-loss-gate` (recoverability checklist + explicit-yes confirmation before bulk deletes, forget sweeps, source/mount removal, or history rewrites), `fact-check` (extract-and-verify every claim pre-publication, with producer-never-verifies re-derivation for data-derived claims), `brain-ingest-gate` (no raw copies, registry-first named-entity resolution, read-the-top-hit dedup).
|
||||
- **Team brains:** `company-brainify` — the personal-to-team sanitization procedure: sanitize a staging copy, strip/keep tables, verification greps on the tree that ships, and a backup-gated history purge that runs only against the shared repo.
|
||||
- **Retrieval graph:** `citation-graph-ingest` builds typed inter-document citation edges over an ingested corpus, queryable through the native graph surface.
|
||||
- **Ingestion:** `bulk-ingestion` (the disciplined lifecycle for any bulk pipeline plus a durable manifest substrate that never trusts a subagent's "done"), `blog-ingest` (whole-publication and feed ingestion with idempotent re-runs and an untrusted-content boundary), `two-tier-extraction` (cheap-triage/deep-read model routing with a deterministic pre-model privacy wall), `conversation-archive` (AI-chat exports and session transcripts become first-class brain content, with a mandatory secret-redaction pass).
|
||||
- **Operations:** `measure-before-you-fix` (measure-first triage before touching timeouts and thresholds), `context-audit` (report-only token hygiene for the always-loaded context stack), `skill-autobench` (propose evals mined from a skill's real usage history, staged for human approval), `resolve-before-asking` (exhaust the brain before interrupting the human; ask with a hypothesis), `brain-link-discipline` (verified links in every deliverable, with an honest fallback chain), `draft-in-voice` (memory-grounded ghostwriting from validated voice profiles, with a build-a-profile guide), `research-compendium` (archive, summarize 1:1, synthesize self-contained).
|
||||
- **Conventions:** a shared untrusted-content boundary (fetched text is data, never instructions), progressive-ramp bulk testing with output-existence checks, regex discipline (never compress judgment into heuristics), path discipline (display links are not filesystem paths), and exec-output discipline (buffer, then read bounded slices).
|
||||
- **Skill-pack integrity gates:** a reference checker that fails the build on dangling cross-references and donor-environment remnants (allowlist-ratcheted), warns on commands a skill cites that the CLI doesn't ship, plus a machine-readable plugin-curation record with membership and dependency-closure tests — a bundled skill can no longer reference a skill that doesn't ship downstream, and moving a skill between bundled and host-only is a review-visible decision.
|
||||
- **Skill currency + preconditions (the migration harness now examines skills).** `gbrain skillpack status` reports, at a glance, which built-in skills your workspace is missing (`new`), which you've edited (`drifted`), and which are current — classified by each skill's own files, so a new skill isn't mistaken for a drifted one just because shared conventions are already on disk. `gbrain skillpack sync` installs the new ones and never touches your edits. The post-upgrade sweep now surfaces new skills (it used to hide them) with the one command that adds them, and `gbrain doctor` gains a `skill_currency` check. Skills can declare machine-readable preconditions with a `requires:` frontmatter field (`source`, `dir:<path>`, `config:<key>`, `pages:<n>`); `gbrain skillpack setup <skill>` prints what a skill needs, and `gbrain doctor`'s `skill_preconditions` check verifies them live against your brain with paste-ready fixes.
|
||||
- A committed routing-accuracy receipt for the grown pack, generated by the existing A/B harness.
|
||||
|
||||
### Changed
|
||||
- `meeting-ingestion` is rebuilt: recorder-agnostic pipeline, evidence-based speaker resolution, a hard verify-before-done phase (every quote grounded verbatim in the transcript), and deterministic sequence checks against the day timeline.
|
||||
- `skillify` reconciled with its most-evolved line: eval contracts, a no-regression law, idempotency, and a numbered 15-item checklist other skills can reference.
|
||||
- `eiirp` gains the auto-fire gate: substantial document analysis files a brain page first and delivers the link in the same reply (per-user policy switch included); eiirp now ships to downstream installs.
|
||||
- `minion-orchestrator` gains the durable-execution doctrine: a capability ladder for long operations, a deadman pattern that verifies the result was reported (not merely that a process exited), and content-addressed stage checkpoints.
|
||||
- `concept-synthesis` gains the curation cull: keep/delete verdicts with substance gates, grounding labels, and reversible merges.
|
||||
- `reports` gains the link Actionability Gate ("a missing link is honest, an indirect link is a broken promise"); `briefing` pulls salience, anomalies, and recall before composing; `daily-task-manager` gains stable task IDs and fail-closed action routing; `book-mirror`, `idea-ingest`, `media-ingest`, `brain-ops`, `maintain`, and `data-research` pick up targeted upstream improvements.
|
||||
- New routing rows ship with disambiguation rules (publication vs single article vs media vs chat exports; identity content vs context hygiene; measurement-first triage vs debugging) and negative routing fixtures across the pack.
|
||||
|
||||
### Fixed
|
||||
- Imported skill registrations that captured a YAML block-scalar marker instead of the skill's description now carry real prose, with a test pinning description quality and plugin-list uniqueness.
|
||||
- The commit gate fails loudly when the skills lock file is regenerated but unstaged (comparing the staged blob, not just the path), and the pack's plugin skill list is sorted with duplicates rejected.
|
||||
- Skill frontmatter now states its true effects: a skill that commits and pushes is marked mutating, and inert precedence markers were removed.
|
||||
|
||||
To take advantage of v0.45.6.0: upgrade with `bun install -g github:garrytan/gbrain#latest-stable`, then run `gbrain skillpack reference --all` to sweep the new and upgraded skills into your agent repo (or `gbrain skillpack scaffold --all --workspace <your-agent-repo>` on a fresh install). Nothing to migrate — new skills route via their trigger phrases immediately, and `gbrain check-resolvable --strict --skills-dir skills/` verifies the pack end to end.
|
||||
## [0.45.5.0] - 2026-08-12
|
||||
|
||||
Brain currency, part one: a brain is only useful if it's CURRENT, and until now
|
||||
the machinery keeping it current could die without anyone noticing. This release
|
||||
makes autopilot's health honest end-to-end — status that reads the heartbeat,
|
||||
a daemon that takes itself out of rotation when its repo vanishes, migrations
|
||||
that pause it instead of racing it, and staleness reporting that can no longer
|
||||
say "fresh" forever.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A dead autopilot can no longer report healthy.** `gbrain autopilot --status`
|
||||
now reads the daemon's heartbeat instead of checking that install artifacts
|
||||
exist, and gains real exit codes for cron and CI gates: 0 fresh (or nothing
|
||||
installed), 1 needs attention (stale heartbeat, never ran, or paused), 2 the
|
||||
daemon took itself out of rotation. Status runs without touching the
|
||||
database, so it keeps working during the exact outages it exists to
|
||||
diagnose. Staleness tolerance scales with the tick interval and accounts
|
||||
for the adaptive scheduler's longer healthy-brain sleeps, and a garbage
|
||||
interval value can no longer silence the alarm.
|
||||
- **Content-relative staleness now has a wall-clock ceiling.** A source whose
|
||||
content stopped moving (or whose local clone vanished) previously reported
|
||||
fresh forever off the stored content timestamp. `sync_freshness`,
|
||||
`federation_health`, and `gbrain status` now ramp toward stale past a
|
||||
ceiling (default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune) — ramping,
|
||||
not stepping, so the warn tier still fires before the fail tier instead of
|
||||
both alarms tripping at once.
|
||||
- **The documented agent-scheduler chain works on keyless brains.**
|
||||
`gbrain sync --repo <path> && gbrain embed --stale` used to exit 1 on every
|
||||
brain installed without an embedding key, breaking the always-current cron
|
||||
for external agent schedulers. A bare stale embed now refuses cleanly
|
||||
(exit 0, stderr hint); explicit embed requests (a slug, a slugs list, the
|
||||
all flag) still exit 1.
|
||||
- **Engine migrations and the autopilot daemon no longer race.**
|
||||
`gbrain migrate --to <engine>` claims a cooperative pause marker before
|
||||
touching the target — the marker doubles as a migration mutex, so a second
|
||||
concurrent migrate refuses to run instead of corrupting the first one's
|
||||
resume state, and a marker it cannot write refuses the migration outright
|
||||
rather than running unfenced. Background job workers stop picking up new
|
||||
work while the marker is parked. It then waits for in-flight
|
||||
sync/embed/cycle work and running jobs to actually drain (watching the DB
|
||||
lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS`) instead of
|
||||
sleeping a blind grace period.
|
||||
The marker is released even when the migration fails or is killed: cleanup
|
||||
registers the moment the claim lands, adoption of a dead run's orphan is
|
||||
pid-liveness-checked (a live migrate's marker is never stolen), and the
|
||||
daemon itself clears an orphan whose owning process died. After a clean
|
||||
flip the daemon detects the engine change on its next tick and relaunches
|
||||
onto the new engine — previously it kept syncing into the abandoned source
|
||||
engine until its process happened to restart — and the migration warns if
|
||||
an exported connection-string env var would override the new config.
|
||||
- **Self-disable requires three consecutive misses.** A repo on an external
|
||||
or cloud-synced volume that is briefly absent at login no longer
|
||||
permanently takes the daemon out of rotation; one successful probe resets
|
||||
the strike counter.
|
||||
- **A cron'd status monitor no longer reads as an install.** Machines whose
|
||||
only crontab reference is the recommended health-gate line stop reporting
|
||||
"installed but never ran".
|
||||
- **Malformed connection URLs stop the daemon immediately** with a clear
|
||||
config verdict instead of spending the whole reconnect budget retrying a
|
||||
value only the operator can fix.
|
||||
- **Sync no longer silently drops git typechange and unmerged statuses.**
|
||||
Replacing an indexed file's content in a way git reports as `T` or `U`
|
||||
now imports as a modification instead of never reaching the index; a
|
||||
copy status imports its destination path.
|
||||
- **A wedged sync can no longer read as "in progress" forever.** A sync
|
||||
lock holder that keeps heartbeating past the staleness ceiling without
|
||||
finishing now fails `gbrain doctor`'s freshness check, naming the holder
|
||||
and the exact `gbrain sync --break-lock --source <id>` remedy.
|
||||
|
||||
### Added
|
||||
|
||||
- **Autopilot self-disable guard.** The generated wrapper now stops the daemon
|
||||
for real when its `--repo` path vanishes: it writes an explanatory marker,
|
||||
then boots the job out of the supervisor (`launchctl bootout` on macOS,
|
||||
`systemctl --user disable --now` on systemd) — a bare `exit 0` under
|
||||
KeepAlive/Restart=always is just a quieter respawn loop. `--status` explains
|
||||
why it stopped; a reinstall against a restored path clears the marker;
|
||||
`--uninstall` clears it too.
|
||||
- **`paused` status state.** A daemon parked by a migration (or by an orphaned
|
||||
pause marker) now reports `paused` with exit 1 and the marker path, instead
|
||||
of "running" off its still-fresh heartbeat.
|
||||
- **Harness e2e tier.** A real-launchd lifecycle test on macOS (install →
|
||||
load → self-disable → status, under a per-run unique label) plus a
|
||||
shimmed-supervisor lifecycle that runs on every platform, and an
|
||||
agent-scheduler contract test that drives the documented sync-and-embed
|
||||
shell chain end-to-end against a keyless brain — including the
|
||||
pull-failure case that must break the chain.
|
||||
- **Honest staleness numbers in `gbrain status`.** Source rows now carry
|
||||
`hours_since_last_sync` (raw wall-clock truth) alongside the
|
||||
threshold-relative `staleness_hours` that drives the fresh/stale/severe
|
||||
class, so the escalation ordering and the human-facing number stop being
|
||||
the same field.
|
||||
- **Shared numeric env resolver.** The doctor and staleness-threshold
|
||||
`GBRAIN_*` numeric env vars now resolve through one warn-once helper
|
||||
(`src/core/env-number.ts`), so a typo'd value falls back loudly exactly
|
||||
once instead of NaN-ing a threshold silently.
|
||||
|
||||
### To take advantage of v0.45.5.0
|
||||
|
||||
- `gbrain upgrade`, then wire your scheduler's health gate to
|
||||
`gbrain autopilot --status` — the exit code is now trustworthy.
|
||||
- If autopilot is installed, reinstall once (`gbrain autopilot --install
|
||||
--repo <path>`) so the generated wrapper picks up the self-disable guard.
|
||||
- Keyless installs: your sync-and-embed cron chain now exits 0; no action
|
||||
needed beyond upgrading.
|
||||
## [0.45.3.0] - 2026-08-12
|
||||
|
||||
**Codex installs stop asking a question Codex can't honor.** The bootstrap used to offer every install a choice of MCP scope — this folder only, or the whole machine — but Codex has no per-folder registrations, so picking "this folder" led to a confusing round-trip where the agent asked permission to keep what it had already done. Now each harness gets the honest version: Claude Code records your scope choice during the interview (where it actually sticks), and Codex simply tells you the truth — its registration reaches the whole machine, read and write — along with the exact commands to remove it (just the registration, or the whole install).
|
||||
|
||||
### Added
|
||||
- CI tripwires pin the harness-scoping language in the runbook and question bank, so the dead question can't quietly come back in a future edit.
|
||||
- `gbrain bootstrap status` hints now carry the scope rule on both the interview and wiring phases, so an agent resuming from a stale runbook still sees it.
|
||||
|
||||
### Changed
|
||||
- The workspace's ACCESS_POLICY.md now describes MCP reach per harness — the project/user tradeoff on Claude Code, the always-machine-wide reality on Codex — and states plainly that reach means read and write, not just queries.
|
||||
- The Claude Code scope consent is asked and recorded during the interview, before the answer read-back, so your confirmation covers it and the wiring step just consumes it.
|
||||
|
||||
### Fixed
|
||||
- A hand-damaged answer file no longer crashes `gbrain bootstrap hooks` — and no longer silently grants anything: an unreadable consent answer is treated as declined, with a note telling you how to re-record it.
|
||||
- A leftover "this folder only" answer on a Codex machine (for example, carried over from a paired Claude Code machine) now gets a clear explanation and a safe way to clear it, instead of a silent mismatch between what you chose and what got registered.
|
||||
|
||||
To take advantage of v0.45.3.0: upgrade with `bun install -g github:garrytan/gbrain#latest-stable`. New installs pick everything up automatically. Existing workspaces (either harness) keep their previously rendered policy files; refresh the two that changed with `gbrain bootstrap render --force --only ACCESS_POLICY.md --only CLAUDE.md` (originals are backed up first). If render refuses because the answers show as not confirmed — a pre-fix install that recorded the scope answer after the read-back — run `gbrain bootstrap interview --show`, then `--confirm <hash>`, and retry. Or leave the files as they are — the CLI's own notes and status hints carry the correction either way.
|
||||
|
||||
## [0.45.2.0] - 2026-08-11
|
||||
|
||||
**Make your agent's repo yourself, then let it move in.** If you'd rather own the GitHub repo up front, create a new empty private repo under your own account, clone it, open it in Claude Code or Codex, and paste the bootstrap block — bootstrap now detects your empty repo and adopts it instead of creating one, verifying it is private before anything is pushed. The default (open an empty folder and let bootstrap make the repo) is unchanged and now stated plainly in the docs. Either way, the folder you open becomes your agent's durable, private body.
|
||||
|
||||
### Added
|
||||
- **Create-repo-first bootstrap.** `gbrain bootstrap repo` adopts an empty, private, personally-owned GitHub repo you created, instead of only ever creating one. The README (Claude Code + Codex), the bootstrap runbook, and the bootstrap guide now lead with the repo and document both paths (open an empty folder, or bring your own empty repo).
|
||||
|
||||
### Changed
|
||||
- Bootstrap now reports how the repo was set up — created, adopted, or already pushed.
|
||||
|
||||
### Fixed
|
||||
- Pointing bootstrap at a repo that already has content no longer reports success without pushing your workspace. It stops with a clear message: make an empty repo, or run `gbrain bootstrap attach` for an existing agent clone.
|
||||
- Adopting a repo on a fresh machine no longer fails at the first commit — a repo-local git identity is set on the adopt path, not just the create path.
|
||||
- A failed first push no longer looks "done" on the next run: the repo is recorded only after the push succeeds, so a re-run resumes instead of skipping.
|
||||
- The pre-push secret scan now also covers an already-committed tree, and a failure to enumerate files stops the push instead of passing silently.
|
||||
- Automatic per-turn and session-end pushes wait until the repo phase has verified the repo is private, so nothing is published to an unverified remote.
|
||||
|
||||
To take advantage of v0.45.2.0: upgrade with `bun install -g github:garrytan/gbrain#latest-stable`. Nothing to migrate. To use the new path, create an empty private repo under your own account, clone it, open it in your agent, and run the bootstrap block — it adopts your repo. If anything about the repo or push looks off, `gbrain doctor` names it with the exact fix.
|
||||
## [0.45.1.0] - 2026-08-11
|
||||
|
||||
**Your per-prompt brain hooks are now measurable and non-repetitive.** v0.45.0.0's paste-in agent install gave every prompt a context injection; this release makes that channel behave like a product instead of a firehose. The hook remembers what it already told you — a page it injected earlier in the session isn't re-injected every time the name comes up — and every delivery now lands in the same precision feedback loop the other push channels use, so `gbrain volunteer-context --stats` and a new doctor check show exactly which harnesses are firing and how useful their pushes are.
|
||||
|
||||
### Added
|
||||
- **Cross-turn dedupe for the per-prompt hook.** `gbrain hook user-prompt` reads its own previous injections back out of the session transcript (recorded as structured attachments — verified against a live Claude Code session) and suppresses re-volunteering, so a page is pushed once per session, not once per mention. The dedupe input is deduplicated and byte-capped, only gbrain-marked blocks count (another tool's hook output can't silence your brain), and the extraction is structural — a slug appearing in some tool payload can't over-suppress.
|
||||
- **Per-harness feedback loop.** Delivered hook context now logs to the volunteered-pages feedback table under its harness channel (`claude-code` today; `--harness codex` reserved for a codex hook registration), counted at the delivery point only — a block the hook abandoned mid-deadline is never counted, and the hook records partial trims so drift is visible.
|
||||
- **`volunteer_channels` doctor check** on both the local and remote doctor: per-channel activity over the last 7 days, with guidance that distinguishes "hook installed but never registered (restart the session)" from "registered but quiet", engine-aware messaging, and a caution when the hook's own heartbeat shows deliveries mostly degrading.
|
||||
|
||||
### Changed
|
||||
- The turn-context IPC response now carries the post-budget volunteered pages, and the request carries an attribution channel — both additive; older serves and clients interoperate unchanged (an older serve simply doesn't log hook deliveries until restarted).
|
||||
- When a turn-context request exceeds the IPC message cap, the advisory dedupe payload is dropped before any conversation turn — context quality is never sacrificed to preserve a hint.
|
||||
|
||||
### Fixed
|
||||
- A remote doctor report requested with a source-scoped token no longer aggregates push-activity metadata across sources it isn't authorized for.
|
||||
- The IPC connection handler processes exactly one request per connection — trailing bytes can no longer double-process a request (which would have double-counted deliveries).
|
||||
- A transient database error during the doctor's channel check is no longer misreported as an old-schema brain.
|
||||
|
||||
## To take advantage of v0.45.1.0
|
||||
|
||||
No migration and no re-registration needed. **Restart your `gbrain serve`** (or
|
||||
just restart the harness session — it respawns the MCP serve) so the new
|
||||
delivery logging activates; hooks registered by `gbrain bootstrap` pick up the
|
||||
dedupe automatically on the next prompt. Then check the loop is live:
|
||||
|
||||
```bash
|
||||
gbrain volunteer-context --stats # per-channel precision, incl. claude-code
|
||||
gbrain doctor # look for the volunteer_channels check
|
||||
```
|
||||
|
||||
## [0.45.0.0] - 2026-08-10
|
||||
|
||||
**Your coding agent can now become your personal agent.** Paste one block into Codex or Claude Code and it sets itself up as a persistent agent with a memory that survives across sessions: it interviews you, writes its own identity files from your answers, spins up a local brain, and keeps a private GitHub repo as its durable body. Close the laptop and reopen it tomorrow, and it still knows who you are, who you're talking to, and what you told it last time. This is the OpenClaw/Hermes experience — identity, memory, schedules, persistence — running on the subscription you already pay for, with nothing to deploy.
|
||||
|
||||
**Start with Codex.** It runs on your ChatGPT subscription, takes about fifteen minutes, and deploys nothing. Claude Code is the same install. OpenClaw and Hermes are still the way to run GBrain exactly as designed — always on, enriching around the clock — at real server and API cost; Codex is the recommended first step for anyone new to GBrain.
|
||||
|
||||
### How to use it
|
||||
|
||||
Pick the folder that will become your agent's home and paste (Codex shown; the Claude Code block is identical):
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
The agent runs `gbrain bootstrap` — a new command family (`status`, `interview`, `render`, `repo`, `hooks`, `verify`, `attach`, `uninstall`) that drives the whole install. It works with **zero API keys**: your harness's model is the LLM, so the agent authors memory directly and search runs keyword-only; add one optional key to upgrade capabilities (OpenAI: semantic search + automatic fact extraction; Voyage: semantic search; Anthropic: fact extraction). Everything is consent-gated — hooks, background push, MCP scope — and nothing runs while your harness is closed (the honest desktop contract; true 24/7 is what a hosted brain adds).
|
||||
|
||||
### What you get
|
||||
|
||||
- **A memory that compounds.** On Claude Code, session hooks inject relevant brain context at the start of each prompt and ingest the transcript when the session ends; on Codex, the rendered `AGENTS.md` carries the same pull protocol. Facts you state get written back through the brain's own tools and resurface next session.
|
||||
- **A portable body.** Your workspace is a private GitHub repo (privacy verified via the API before anything is pushed, secret-scanned before every commit). Clone it on a second machine and run `gbrain bootstrap attach`. It mounts anywhere GBrain runs — the exit plan if a provider ever changes its terms.
|
||||
- **Multi-model install.** Works in the ChatGPT desktop app, the Codex CLI, Claude Code desktop, and the Claude Code CLI. Same body format, two doors.
|
||||
|
||||
### What to know after upgrading
|
||||
|
||||
Existing installs are untouched — this is a new, opt-in surface. If you want it, run the paste block in a fresh folder. `gbrain doctor` gained bootstrap health checks (hook heartbeat, push staleness, serve/lock collisions, runbook skew); if anything looks off after an install, `gbrain doctor` names it and prints the fix.
|
||||
|
||||
### What we caught and fixed before merging
|
||||
|
||||
This shipped through a full review pass (architecture, security, adversarial, cross-model). Highlights, in plain terms: the credential scanner now recognizes today's API-key formats and reads the exact bytes being committed rather than a stale snapshot, so a key can't slip through a timing gap; private-key material is redacted whole, not just its header; and the scanner fails safe — an unreadable or oversized file blocks the push instead of passing unscanned. The GitHub repo is confirmed to belong to you and to be private before any content leaves the machine.
|
||||
|
||||
## To take advantage of v0.45.0.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
|
||||
warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **To try the new agent-bootstrap surface,** paste the block above into Codex
|
||||
or Claude Code in a fresh folder, or read
|
||||
[`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) for the full contract.
|
||||
Existing brains need no action — bootstrap is an opt-in new surface, not a
|
||||
change to how your current install works.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`
|
||||
and `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
## [0.44.1.0] - 2026-08-11
|
||||
|
||||
**Any current model works now. gbrain stops rejecting model ids it hasn't heard of.**
|
||||
|
||||
New models ship every week. Until now, gbrain kept a built-in list of "known" models for each major provider (Anthropic, OpenAI, Google), and if you configured a model that list hadn't learned yet, gbrain refused to run it, even when the provider was already serving that model to everyone else. In practice that meant a brand-new model like `openai:gpt-5.6-sol` read as "not available" on an install whose binary shipped three weeks earlier, and the error message steered people toward older models instead. Worse, the workaround was inconsistent: the same model id worked when set as `models.default` but was rejected when set as `models.think`, for no reason a user could see.
|
||||
|
||||
That whole class of failure is gone. gbrain now checks only what a provider can do (Anthropic has no embedding models, Voyage has no chat models). Which model id you use is your call. If you name one that does not exist, the provider says so at call time, in its own words, and gbrain shows you that message instead of guessing.
|
||||
|
||||
How to use it:
|
||||
|
||||
```bash
|
||||
gbrain config set models.default openai:gpt-5.6-sol # any current model id
|
||||
gbrain config set models.think google:gemini-3.6-flash # per-task keys work identically now
|
||||
gbrain models doctor # pre-flight: probes your configured models live
|
||||
```
|
||||
|
||||
What changed in each case:
|
||||
|
||||
| You do | Before | Now |
|
||||
|---|---|---|
|
||||
| Configure a model newer than your gbrain binary | Rejected: "not listed... Known models: ..." | Runs |
|
||||
| Set that model via `models.think` / `models.dream.*` | Rejected even when `models.default` worked | Identical behavior on every key |
|
||||
| Typo a model id | Caught instantly, locally | Fails at the provider with the provider's own message; `gbrain models doctor` still catches it pre-flight without spending tokens |
|
||||
| Use a chat model from an embeddings-only provider | Rejected | Still rejected (that check is about the provider, not the model) |
|
||||
|
||||
Things to watch: a typo'd model id now costs one failed provider call instead of failing free and instantly. Run `gbrain models doctor` after changing model config if you want the old fail-fast feel. `gbrain think`'s fallback answer now carries the provider's actual error text, so a bad model id no longer reads as an API-key problem.
|
||||
|
||||
## To take advantage of v0.44.1.0
|
||||
|
||||
No migration, no schema change. `gbrain upgrade` is enough.
|
||||
|
||||
1. **Set any current model:**
|
||||
```bash
|
||||
gbrain config set models.default <provider>:<model>
|
||||
```
|
||||
2. **Verify:**
|
||||
```bash
|
||||
gbrain models doctor
|
||||
```
|
||||
3. **If a model you know is real still fails,** the error now comes from the provider; check the id spelling and your key, then file an issue at https://github.com/garrytan/gbrain/issues with the `gbrain models doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/ai/model-resolver.ts` — `assertTouchpoint(recipe, touchpoint, modelId)` checks provider touchpoint capability only; the native-recipe model allowlist throw is removed. Recipe `models:` arrays remain as data: `models[0]` default selection for `--model <provider>` shorthand and env-ready pickers, guard-test fixtures for the repo's own hardcoded defaults, and `gbrain providers list` display.
|
||||
- `src/core/ai/gateway.ts` — the extended-models registry (`_extendedModels`, `registerExtendedModel`, `registerConfigSelectedChatModel`, both registration loops, and the tier-resolution loop that fed them) is deleted. Every per-task model key (`models.think`, `models.dream.*`, `facts.extraction_model`, ...) now behaves exactly like `models.default`. `gateway.rerank()` keeps its own model-list check: each listed reranker id maps to a known request/response wire shape.
|
||||
- `src/core/think/index.ts` — the graceful "no LLM available" sentinel surfaces the thrown `AIConfigError`'s own message and fix (which key is missing, or what the provider rejected) instead of generic key advice.
|
||||
- `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — drops the now-dead chat-model registration call.
|
||||
- Tests — `test/gateway-tier-extended-models.test.ts` deleted (pinned the removed machinery); rejection tests across `test/ai/`, `test/think-*`, and `test/cycle/` now pin the pass-through contract; `unknown_model` still fires for providers lacking the touchpoint, so every probe reason stays reachable; new test pins the sentinel carrying the provider's error text.
|
||||
- Docs — `docs/architecture/KEY_FILES.md` entries for the resolver, gateway, and reindex handler updated to the new contract.
|
||||
|
||||
## [0.44.0.0] - 2026-06-12
|
||||
|
||||
**BrainBench: agent memory now has a scorecard.** `gbrain eval brainbench` is a public, reproducible, cross-harness conformance suite for the four ways agent memory fails — and from this release forward, every memory PR must hold or move its numbers against a committed baseline that CI compares against master's own copy.
|
||||
|
||||
Four suites, scored per harness seam:
|
||||
|
||||
- **know-to-ask** — does memory volunteer the right context unprompted, and stay silent when it should? (`know_to_ask_failure_rate` + the anti-gaming `false_fire_rate` — "always inject" can't win both)
|
||||
- **push precision / recall** — when context is volunteered, was it the right context, within each harness's injection budget?
|
||||
- **write-back fidelity** — do conversation facts survive the PRODUCTION conversation→memory pipeline with correct provenance? (graded through a new injectable-extractor seam on the conversation-facts pipeline, so CI executes the shipped segmentation/insertion/dedup code with zero LLM calls; `--llm` opts into the real extractor under a run-scoped budget)
|
||||
- **continuity** — a decision recorded in one session, recalled by a different harness on the same brain
|
||||
|
||||
Plus two cross-cutting measures: `source_isolation_violations` (gates at zero, every run — cross-source leakage is the data-leak invariant) and `avg_injected_tokens` (the intrusion budget, reported per harness).
|
||||
|
||||
Every scoreboard row carries a `seam` label: the `openclaw` row exercises the shipped context-engine pipeline byte-for-byte (`production`); the `claude-code` and `codex` rows grade the same gbrain primitives through those harnesses' injection-shape contracts (`contract`) — the exported wire types are the contracts future integrations implement, and the rows flip to `production` with continuous numbers when they land. At v0.44.0.0 the production seam reads: know-to-ask failure 0.15 (the documented v1 reflex limits, now measured), push recall 0.81 at precision 1.0, write-back fidelity 1.0, continuity 1.0, zero isolation violations.
|
||||
|
||||
### Added
|
||||
|
||||
- `gbrain eval brainbench` — hermetic by default (in-memory PGLite, no keys, no LLM, ~7s for the full corpus), with `--harness`, `--suite`, `--fixtures DIR --gold DIR` (the foreign-runner surface), `--json`, `--out` (the canonical CI artifact), `--compare`, `--update-baseline`, `--justification`, `--allow-regression`, `--include-holdout`, and `--llm --budget-usd`. Exit codes are the contract: 0 pass, 1 regression, 2 error/inconclusive.
|
||||
- A 141-fixture / 241-gold-turn benchmark corpus across 7 stratified categories, generated deterministically (seed 42) over a whole-cloth fictional universe with SEALED gold (a `gold` key inside a fixture is a validation error; adapters only ever see sanitized turns), a ~15% holdout split, and a blind double-label validation receipt (96.4% agreement) recorded in the corpus ledger. Rebuild byte-identically with `bun evals/brainbench/generator/gen.ts`.
|
||||
- Published interchange contract for foreign runners: JSON Schemas for fixtures, gold, results, and baselines under `evals/brainbench/schema/` — point the CLI at any conforming corpus and parse the schema'd receipt back.
|
||||
- A CI gate (`brainbench` job + `scripts/ci-brainbench-gate.sh`) hardened through four adversarial review rounds: it compares against MASTER's committed baseline (a PR cannot rewrite the thing it's graded by), any baseline edit without a fixture change must byte-match the actual run, corpus changes route through a bless mode where shrinking gold coverage or regressing a metric requires a written justification visible in the PR diff, baselines bind their run config, and the gate fails hard — never silently open — on broken refs or deleted baselines.
|
||||
- Eleven new metrics in the metric glossary (plain-English definitions in `docs/eval/METRIC_GLOSSARY.md`); every JSON response carries the standard `_meta.metric_glossary` block.
|
||||
- Methodology doc at `docs/eval/BRAINBENCH.md`: seam disclosure, formulas, pre-registered expectations, determinism posture, gate governance, gold methodology, and accepted residuals.
|
||||
|
||||
### Changed
|
||||
|
||||
- `gbrain eval run-all --suites brainbench` now actually runs the suite in-process (closing the long-standing orchestrator-stub follow-up) and records one result per sweep under `EvalRunRecord` schema v3 with `mode: 'n/a'` — benchmark records no longer fabricate a search mode.
|
||||
- The conversation-facts pipeline accepts an injectable per-segment extractor (default unchanged — the LLM path); this is the seam the write-back suite grades production code through.
|
||||
- The synthetic-corpus privacy guard now scans the BrainBench corpus dirs (fixtures AND gold) alongside the calibration corpus, and the fixture-authoring guide adds scenario-level privacy rules: scenarios are invented whole-cloth, never anonymized from real situations.
|
||||
- The CI aggregate (`test-status`) now checks the brainbench job's result explicitly — a failing memory gate fails the branch-protection check.
|
||||
|
||||
### Fixed
|
||||
|
||||
- CLI exit codes for the new command route through the shared write-fence + aliveness-grace exit seam, so PGLite's WASM exit-code stomping and Bun's exit-time stdout discard can't corrupt the CI contract.
|
||||
|
||||
To take advantage of v0.44.0.0: run `gbrain eval brainbench` — no setup, no keys, no brain required. If it ever reports something broken after an upgrade, `bun evals/brainbench/generator/gen.ts` rebuilds the corpus byte-identically and `gbrain eval brainbench --update-baseline` re-derives the baseline from an actual run; both are safe to re-run any time.
|
||||
## [0.43.0.0] - 2026-08-08
|
||||
|
||||
**Your agent now has five memory verbs it can actually reach.** Cathedral 1 freezes
|
||||
a stable, versioned memory protocol — `recall`, `remember`, `entity`, `synthesize`,
|
||||
`forget` — over the brain's operation catalog, the way Postgres speaks one wire
|
||||
protocol to every client. Point any MCP harness at it (`claude mcp add gbrain --
|
||||
gbrain serve --surface verbs`, or the Codex/OpenClaw equivalents) and the agent sees
|
||||
exactly five self-describing tools instead of a wall of internal ops. Every response
|
||||
carries what it is, why it matched, where it came from, and what it spent (the token
|
||||
budget on `recall`, latency on `entity`, the full cost block on `synthesize`) — and the
|
||||
contract never breaks: v1 field names and meanings are frozen, changes are
|
||||
additive-forever.
|
||||
|
||||
What you can do now that you couldn't before:
|
||||
|
||||
- **Remember a fact once, recall it in a fresh session — in any harness.** `remember`
|
||||
takes mandatory provenance (where the fact came from) and an optional expiry, dedupes
|
||||
against what's already known, and supersedes the old fact when it changes
|
||||
("X joined acme-example" → "X left acme-example" — the outdated fact expires, the
|
||||
history stays). `recall` retrieves saved facts and,
|
||||
with a query, budget-packed page snippets — the server enforces the token budget and
|
||||
tells you what it dropped instead of trusting the client to trim.
|
||||
- **Look up one person/company/project as a compact card in well under 100ms, zero LLM
|
||||
calls.** `entity` resolves a name to a privacy-safe card (who/what, aliases,
|
||||
last-touched, open threads, top typed edges) — or, on a miss, near-miss suggestions
|
||||
instead of a dead end.
|
||||
- **Reason across pages when you actually need it.** `synthesize` is the explicitly
|
||||
expensive verb (it says so in its own description), with a best-effort cost block so
|
||||
agents choose it deliberately. No API key configured? It says `unavailable` with a
|
||||
fix, never a fake answer.
|
||||
- **Certify any memory server against the contract.** `gbrain protocol --json` publishes
|
||||
the machine-readable spec; `gbrain protocol conformance [--target <endpoint>]` runs the
|
||||
frozen-contract test suite against gbrain's own server or any MCP endpoint — and
|
||||
provably fails servers that don't comply. `gbrain protocol stats` shows per-verb
|
||||
adoption and your real time-to-first-use, all from a local file that never leaves
|
||||
your machine.
|
||||
|
||||
`gbrain serve` keeps every operation by default (`--surface full`) — existing installs
|
||||
are unchanged. The new `--surface verbs` is the quickstart surface for agents. No schema
|
||||
migration; the verbs ride the existing facts, pages, and typed-graph tables. The full
|
||||
contract, per-harness install blocks, and the additive-forever versioning policy live in
|
||||
[docs/protocol/MEMORY_VERBS_v1.md](docs/protocol/MEMORY_VERBS_v1.md).
|
||||
|
||||
**This release also fixes which search verb your agent reaches for (#2416).** The
|
||||
`search` and `query` tool descriptions, the mandatory lookup chain, and the search
|
||||
guides had drifted from reality: they described `search` as keyword-only (it has been
|
||||
cheap hybrid — vector + keyword, no LLM expansion — for many releases) and told agents
|
||||
to try `search` first for everything, falling back to `query` only when results looked
|
||||
"thin". For concept questions ("all the X that do Y", "the ecosystem around Z") that
|
||||
fallback never fired, and synonym-phrased matches dropped silently.
|
||||
|
||||
- **Concept questions now route to `query` by default.** The tool descriptions and the
|
||||
lookup-chain convention are intent-driven: exact known tokens → `search` (cheaper, no
|
||||
expansion call); concept / synonym / exhaustive-set questions → `query` first. Verified
|
||||
with a live LLM routing eval: all concept phrasings route to `query`, and personal
|
||||
questions still route to the salience ops.
|
||||
- **"Got results" is no longer treated as "got everything."** The descriptions, docs, and
|
||||
conventions now say it plainly: a populated `search` result set is not proof of
|
||||
coverage, and `query` is still top-K — literal exhaustive enumeration belongs to
|
||||
`list_pages` pagination.
|
||||
- **The CLI nudges you when it can help.** A concept-shaped `gbrain search "..."` prints a
|
||||
one-line hint on stderr suggesting the equivalent `gbrain query` call (results stay
|
||||
clean on stdout; `--quiet` silences it; never auto-reroutes).
|
||||
- **`think` cost accounting reads consistently.** When no LLM call ran (stubbed or no
|
||||
API key), `usage` is now uniformly `null` in `--json` output rather than sometimes
|
||||
missing.
|
||||
|
||||
To take advantage of v0.43.0.0: re-run `gbrain serve` with `--surface verbs` to give
|
||||
your coding agent the five-verb memory protocol (or keep `--surface full` for the
|
||||
complete operation catalog — both speak the verbs). Run `gbrain protocol conformance`
|
||||
to self-certify, and `gbrain protocol stats` to watch adoption. Memories your agent
|
||||
saves are readable by every agent connected to the brain by default; pass
|
||||
`visibility: "private"` for local-only facts. If your agent instructions or skill
|
||||
files copy the old "search first, query if thin" rule, refresh them from
|
||||
`skills/conventions/brain-first.md` — the shipped skillpack carries the corrected
|
||||
routing.
|
||||
## [0.42.76.0] - 2026-08-08
|
||||
|
||||
**Mistyped or unsupported flags now fail loudly instead of being silently ignored — including the ones that were supposed to make a command safe.**
|
||||
|
||||
**Strict flag validation, CLI-wide.** Every gbrain command now rejects a flag it does not understand, with a clear error naming the flag and the command, before any work runs. Before, commands read their flags ad hoc and ignored the rest — so `gbrain post-upgrade --dry-run` accepted the flag, ignored it, and applied migrations for real. That class is gone: the legal flags for every command are derived from each command's own source into a generated registry, checked before dispatch, and a command may only advertise a safety flag like `--dry-run` if its code actually reads it. On commands routed through the operations contract, a trailing `--dry-run` is now a real rehearsal switch rather than a no-op. `--json` invocations get the same error as a structured payload, so scripts fail cleanly too.
|
||||
|
||||
**A word of warning (intentional breaking change):** cron jobs or scripts that pass stray, misspelled, or long-removed flags have been running on luck — the flag did nothing. Those invocations now exit with an error naming the flag. That is the point: fix the invocation once and it means what it says forever. Everything after `--` is passthrough and remains untouched.
|
||||
|
||||
**Upgrades can't wedge on forward-referenced columns anymore — as a class.** The v0.42.56.0-era startup wedge (a schema blob referencing a column that pre-existing brains didn't have yet) had two more latent instances waiting in the jobs table. Both are now probed and healed at startup, and the schema coverage guard was rewritten to cross-reference every column referenced by the embedded schema against the set of columns any migration has ever added — so a new forward reference cannot ship without its startup probe. A recovery test walks the exact journey an affected brain takes: failed upgrade, retry on the fixed binary, converge with no leftover state blocking the way.
|
||||
|
||||
**Remote agents get more, within the same fences.** The `think` operation is now available to remote MCP callers as a read-only synthesis — the local CLI can still persist results, while remote callers are forced read-only. Chunk reads now resolve through the same source-scope rules as page reads, so a federated grant that can open a page can also read that page's chunks, and a caller without the grant cannot reach chunks outside its own floor. Chunk payloads also stop carrying raw embedding vectors over the wire — noticeably smaller responses with no behavior change, since no consumer ever read them. Two internal call sites that forward caller identity now treat anything ambiguous as untrusted, matching the fail-closed rule the rest of the codebase already follows.
|
||||
|
||||
**Source-bound clients can be minted over HTTP.** The `/admin/api/register-client` endpoint now accepts `source` and `federatedRead` bindings, mirroring the CLI's `--source` / `--federated-read` flags — so an admin UI or provisioning proxy can create a client confined to a specific brain source without shelling out to the CLI. Omitting both preserves the historical default, and invalid source ids get a structured 400.
|
||||
|
||||
**`gbrain doctor` and `repair-jsonb` see further and misfire less.** The double-encoded-JSON scan now covers the subagent execution columns, and the damage test requires the stored text to actually parse as JSON before flagging it — a legitimate string value that merely starts with `[` or `{` (a log line, a code snippet) is no longer misclassified, and a repair pass can no longer corrupt it. One damaged table no longer aborts the scan of the rest.
|
||||
|
||||
### To take advantage of v0.42.76.0
|
||||
## [0.42.75.0] - 2026-08-08
|
||||
|
||||
**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.**
|
||||
|
||||
The dreaded `RuntimeError: Aborted()` at startup — the one that made zero-config brains unusable after a macOS upgrade and pushed people onto Homebrew Postgres — was never a macOS or WASM bug. An unclean shutdown (typically the upgrade reboot) tears the write-ahead log inside the data dir, and every open after that dies replaying it. gbrain now detects that failure on any command, backs up the WAL state to a sibling directory, resets it in place (the pg_resetwal recovery Postgres has shipped for decades, ported to run against PGLite data dirs), and reopens your brain — pages, embeddings, and history intact. Transactions that never reached a checkpoint may be lost; that is the standard trade for a database that would otherwise not open at all.
|
||||
|
||||
### Added
|
||||
- **Automatic WAL repair on startup.** A torn-WAL abort self-heals on the next gbrain command: backup → in-place reset → retry, with a loud notice naming the backup and recommending `gbrain doctor`. Disable with `GBRAIN_PGLITE_WAL_REPAIR=off`.
|
||||
- **`gbrain pglite-repair`** — the deliberate version: `--dry-run` gives a read-only diagnosis of the data dir; `--yes` runs the same in-place repair manually. Refuses to operate while any live process holds the brain, and never force-removes another process's lock.
|
||||
- **`gbrain doctor` diagnoses unopenable PGLite brains.** A new `pglite_data_dir` check reads the data dir from disk when connect fails, names the right recovery rung (repair vs rebuild), inventories repair backups, and escalates when repairs keep recurring — the signal that something is still killing gbrain mid-write.
|
||||
- **Recovery guardrails throughout:** repair runs only under a cleanly-acquired lock (never after taking over another process's lock, with a quarantine window when a lock's holder couldn't be verified); a live database — including a native Postgres one — is refused by a `postmaster.pid` liveness check; repeated attempts inside one corruption episode reuse one backup instead of stacking copies (newest three episodes retained); a cooldown stops repair loops from silently eating data on machines where crashes keep recurring; and every restore path reports honestly whether your original files are back in place or waiting in the backup.
|
||||
|
||||
### Changed
|
||||
- **`gbrain reinit-pglite` works bare.** The embedding model and dimensions now default from your config file, so the rebuild rung of the recovery ladder is one command mid-outage (explicit flags still win; environment overrides are deliberately ignored so a stale shell export can't change the rebuild target).
|
||||
- **Honest error messages.** The startup-abort hint now names the real cause (torn WAL after an unclean shutdown), states exactly what auto-repair did or why it stood down, and lays out the full ladder: repair → rebuild → engine switch. The docs that claimed PGLite is "incompatible with macOS 26.x" have been rewritten (README, INSTALL, ENGINES) — thanks @roysaurav for the original native-Postgres walkthrough, which remains the engine-switch rung.
|
||||
- Message-less WASM error objects no longer surface as `[object Object]`.
|
||||
|
||||
### Fixed
|
||||
- The classifier that routes startup failures now matches the abort message PGLite actually produces (it previously fell through to a generic hint), while catalog corruption keeps routing to rebuild — WAL repair is never suggested for damage it cannot fix.
|
||||
- Lock-file reads can no longer misclassify a healthy live holder as corrupt (writes are atomic now), a holder owned by another user is treated as alive, and an in-flight acquisition is no longer mistaken for a corrupt lock.
|
||||
|
||||
Credit where due: @yang1996202-cpu (#2575), @AndreLYL (#223), and @roysaurav (#1670) for reports and diagnosis, the #223 thread contributors whose recoveries proved the root cause, and @yestheboxer, whose rejected upstream recovery PR (electric-sql/pglite#994) this port builds on.
|
||||
|
||||
### To take advantage of v0.42.75.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to configure. If a cron job or script starts failing with `unknown flag`, that invocation was passing a flag that did nothing — remove or fix the flag and it will not regress silently again.
|
||||
|
||||
### For contributors
|
||||
|
||||
Community fixes absorbed with credit: @colinagent (#2598 think read-scope; the upgrade-rewind e2e pattern from #2623), @guim4dev (#2016 register-client source bindings), @vinsew (#597 repair-jsonb coverage extension), @javieraldape (#2494/#2531 output-correctness class — BigInt-safe local rendering and the search `--json` regression pin land here; parts of both PRs shipped earlier from master). Thank you — superseded PRs are being closed with notes.
|
||||
|
||||
The unit-test runner is now memory-safe on machines running multiple workspaces: shard concurrency adapts to actually-available memory, and a serial rescue lane re-runs files that died to OOM or external kills before calling them failures — a red suite now means real failures, not memory pressure. The flag registry regenerates via `bun run build:flag-registry` and is pinned by freshness, drift, and consumption-evidence guards.
|
||||
If your brain currently won't open, that's it — the next command repairs it. If you'd rather look first: `gbrain pglite-repair --dry-run`.
|
||||
|
||||
## [0.42.74.0] - 2026-08-07
|
||||
|
||||
**Two fixes for agents that reach a brain over the network: takes-holder visibility now works the way you set it, and the voice recipe is safe by default.**
|
||||
|
||||
Legacy bearer tokens served over `gbrain serve --http` now honor the takes-holder allow-list you set with `gbrain auth permissions <token> set-takes-holders`. Before, that setting was read on one serving path but silently ignored on the other, so a remote agent saw only world-held takes no matter what you granted — a token you widened to see brain-held takes saw none of them, and a token you narrowed still saw the public ones. Both directions now behave as configured, an empty grant means "no takes" (not "the default set"), and the two serving paths decode and apply the grant through one shared piece of code so they cannot drift apart again. Tokens with no grant continue to fall back to public-only, so nothing widens on upgrade.
|
||||
|
||||
The bundled voice-agent recipe (`recipes/agent-voice`) ships secure by default. Its reference server now refuses cross-origin browser requests unless you name the origins in `AGENT_VOICE_CORS_ORIGIN`, gates the endpoints that spend your OpenAI key or read your brain so a stray web page can't trigger them, and listens on loopback only until you set `HOST` to expose it. The voice page you run locally is unaffected. Because this recipe is copied into your own repo at install time, `gbrain integrations install agent-voice --refresh` picks up the hardened version.
|
||||
|
||||
### To take advantage of v0.42.74.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Then, if you serve a brain to remote agents, set each token's takes-holder scope with `gbrain auth permissions <token> set-takes-holders world,brain` (or your desired holders). Voice-recipe operators run `gbrain integrations install agent-voice --refresh --target <your-host-repo>`, then set `AGENT_VOICE_CORS_ORIGIN` if a browser on another origin needs access and `HOST=0.0.0.0` only if the server must listen beyond loopback.
|
||||
|
||||
### For contributors
|
||||
|
||||
Both issues were reported by external security researchers who supplied fixes. Ship-stage adversarial review hardened two more spots: the two serving paths now share one permissions-decode helper (not just the allow-list parser) so a malformed double-encoded row can't make them disagree, and the hot-memory cache key encodes the allow-list collision-free so the empty-vs-absent distinction holds for every holder value. Credit @Derek95king (takes-holder threading) and @sebastiondev (voice-recipe CORS).
|
||||
|
||||
## [0.42.73.2] - 2026-08-05
|
||||
|
||||
**A write that deduplication redirects onto an existing page is now checked against the write scope of whoever asked for it.** When the same content arrives under a new slug, gbrain recognises it and points the write at the page that already holds it. That redirected target is now tested against the caller's own scope — under whichever mechanism confines that caller. One of the two mechanisms was consulted at that point; both are now.
|
||||
|
||||
Nothing changes for local CLI use, or for clients that hold unrestricted write access — neither was ever scope-confined. A confined caller whose write dedups onto a page **inside** its own scope keeps working exactly as before; that redirect is a feature and it is preserved, with a regression test to keep it that way. A confined caller whose write dedups onto a page **outside** its scope now gets `permission_denied`, with the remedy in the message: drop the `id:` frontmatter field, or change the content, to write a new page under your own prefix. The denial does not name the page the write resolved to.
|
||||
|
||||
Recommended for any brain served over HTTP to scope-restricted clients.
|
||||
|
||||
### To take advantage of v0.42.73.2
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to configure. Existing clients keep their scopes unchanged, and no re-registration is needed.
|
||||
|
||||
### For contributors
|
||||
|
||||
Reported privately by an external security researcher, who supplied a fix and a regression test with it. The version that shipped composes the two existing scope-matching rules into a single predicate rather than restating either one, so the check at the door and the check after a redirect cannot drift apart; the audit the report prompted closed the same gap on one further caller path.
|
||||
|
||||
## [0.42.73.1] - 2026-08-05
|
||||
|
||||
**Removes the PR gate that v0.42.73.0 added, and reverts the v0.42.72.1 contribution-policy change it enforced.** The gate cannot function on this repository, and it caused a real incident before that was understood.
|
||||
|
||||
The gate needed two things this repository does not grant it: an `ANTHROPIC_API_KEY` Actions secret for its verdict, and read-write workflow permissions to post a comment or set a label. Without them it can only skip. Worse, on its first live runs a read-only token turned every API call into a 403, the code treated that as a crash, and the check went red on an outside contributor's pull request four times with no comment explaining why. That was fixed in v0.42.73.0, but a check that runs on every pull request and can never reach a verdict does not earn its place in the repository.
|
||||
|
||||
The v0.42.72.1 contribution policy is also withdrawn: the human-written intent paragraph and gbrain-in-use screenshot are no longer required on issues and pull requests. `CONTRIBUTING.md`, both issue templates, and the pull-request template return to their pre-2026-08-02 state, and issues and PRs are reviewed on their content by maintainers, as before.
|
||||
|
||||
The code is preserved in git history at v0.42.73.0 and can be restored if the repository ever grants those permissions. If it is restored, the mechanical half — the intent and screenshot check, the version-first title rule, the red flags — should render to the Actions job summary instead of a comment, because that needs no token permission and no API key.
|
||||
|
||||
### To take advantage of v0.42.73.1
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
Nothing to change. Everything else v0.42.73.0 shipped — the five contributed correctness fixes, `slug_filter`, and the four dependency pins that cleared six CVEs — is unaffected and stays.
|
||||
|
||||
## [0.42.73.0] - 2026-08-04
|
||||
|
||||
**Every incoming pull request now gets a verdict before anyone reads it — and five contributed fixes for silent wrong answers.**
|
||||
|
||||
**The PR gate.** Open a pull request against gbrain and an automated check now posts a single verdict comment within a minute: **merge-lane**, **close-lane**, or **needs-maintainer**, with its reasons and a checklist of what a human reviewer should verify for that specific diff. It also checks mechanically that the description carries the human-written intent paragraph and the screenshot of gbrain in use that `CONTRIBUTING.md` requires, and that the title leads with its version.
|
||||
|
||||
It is deliberately **advisory** — a triage signal and a reviewer checklist, not an authorization boundary. A green verdict is not permission to merge; a maintainer still decides. Pull-request code is never checked out or executed: the verdict comes from the description and the diff read through the API. Maintainer, bot, and draft pull requests are exempt from the intent-and-screenshot floor only (release automation cannot screenshot itself); they still receive the full verdict. Where the rubric can be argued with, the decision is taken away from it: a merge-lane recommendation is downgraded automatically when a diff adds a dependency, a new provider recipe, or new config keys, edits workflows, deletes a test, exceeds 40 files or 400 net source lines, or changes `src/` without touching a single test.
|
||||
|
||||
**Your import output parses again.** `gbrain import <dir> --json` printed five informational lines to stdout ahead of the JSON payload, so anything parsing that output read zero imports while its own bookkeeping recorded the files as ingested — and the next run skipped them permanently. Those lines now go to stderr under `--json`; human output is byte-for-byte unchanged.
|
||||
|
||||
**`sources harden --dry-run` no longer changes anything.** It reset the helper's executable bit before reaching the dry-run check, so a documented preview quietly mutated permissions.
|
||||
|
||||
**Telemetry records the model that actually ran.** Two nightly-cycle phases wrote a hardcoded or unrelated model name into their verdict cache, evidence signature, and spend metering while the gateway ran whatever chat model you configured. On any brain with a non-default model, the recorded history was fiction.
|
||||
|
||||
**`gbrain integrity` stops contradicting itself.** Dead-link findings were counted in the "Review queue" total but written to a different file, so `integrity review` disagreed with `integrity auto`'s own summary. They now get their own line.
|
||||
|
||||
**Retype rules can address API-ingested pages.** Mapping rules could only filter on a file path, which is empty for every page written through `put_page` — so no rule could target that whole class. A new `slug_filter` filters on the slug instead, and combines with the path filter when both are given.
|
||||
|
||||
Also: the `integrity` source comment no longer documents a `--dry-run` subcommand form that exits with an error.
|
||||
|
||||
### To take advantage of v0.42.73.0
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain import <dir> --json | jq . # now parses
|
||||
gbrain integrity auto # dead links reported separately
|
||||
```
|
||||
|
||||
Nothing to configure for the gate — it runs on pull requests to this repository. If you maintain a fork and want it, the workflow needs an `ANTHROPIC_API_KEY` secret; without one it skips loudly rather than blocking anyone.
|
||||
|
||||
### For contributors
|
||||
|
||||
The gate went through six rounds against two independent blind reviewers, each judging cold. The findings that changed the design most were not exploits but false positives: a code fence that swallowed the rest of a description, an explanation written as bullet points scoring zero words, a word floor stricter than the published policy, and a comment telling contributors to reopen a pull request that was never closed. Those four descriptions are now permanent regression fixtures — a gate that insults a first-time contributor is worse than no gate. Two properties are deliberate and documented rather than fixed: the mechanical floor is a floor (a determined author clears it in seconds), and a bare URL in a cited reason still autolinks.
|
||||
|
||||
Contributed by @YiconZiwei (#2655), @time-attack (#3764, #3759, #3726, #3751, #3739, and the gate groundwork in #3573/#3698).
|
||||
|
||||
## [0.42.72.1] - 2026-08-02
|
||||
|
||||
**Every issue and pull request now needs a human-written paragraph and a screenshot of gbrain actually being used.**
|
||||
@@ -16211,8 +17074,6 @@ If anything looks off, file at https://github.com/garrytan/gbrain/issues
|
||||
with `gbrain doctor` output.
|
||||
|
||||
|
||||
|
||||
|
||||
## [0.28.11] - 2026-05-07
|
||||
|
||||
**Mix providers: OpenAI for text, Voyage for images. One brain, two embedding pipelines.**
|
||||
@@ -18178,9 +19039,6 @@ React admin dashboard baked into the binary. Seven screens designed through Stev
|
||||
- `test/oauth.test.ts` ... 34 test cases covering provider: register, getClient, client_credentials exchange, auth_code flow with PKCE, refresh rotation, verifyAccessToken (OAuth + legacy fallback), revokeToken, sweepExpiredTokens, scope annotations on all 30 operations. Plus the post-/cso security-fix regressions: 10-concurrent auth code exchange (only 1 wins), 10-concurrent refresh rotation (only 1 wins), redirect_uri HTTPS-or-loopback gate, and pgArray comma-element round-trip (1 element in → 1 element out).
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## [0.25.1] - 2026-05-01
|
||||
|
||||
## **Your brain can now read books with you. Nine new skills land at once.**
|
||||
@@ -19430,7 +20288,6 @@ Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tun
|
||||
If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both.
|
||||
|
||||
|
||||
|
||||
## [0.22.6.1] - 2026-04-26
|
||||
|
||||
**Old brains can upgrade again.**
|
||||
|
||||
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines 100+ shared operations (including `volunteer_context` — push-based context, see `docs/guides/push-context.md` — and the seven frozen MEMORY_VERBS `recall`/`remember`/`entity`/`synthesize`/`forget`/`context_pack`/`delta` — the last two are v0.45.7 ambient-recall boundary verbs (budget-packed pack + "what changed since"), all seven stamp `protocol_version: 1`, servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md` + `docs/guides/ambient-recall.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -119,12 +119,14 @@ detail on demand.)
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
| memory verbs / MCP tool surface (`--surface`) / conformance | `docs/protocol/MEMORY_VERBS_v1.md` + the `verbs*`/`surface.ts`/`protocol.ts` entries in `KEY_FILES.md` |
|
||||
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
|
||||
| running or writing tests | `docs/TESTING.md` |
|
||||
| bulk-command progress wiring | `docs/progress-events.md` |
|
||||
| eval methodology / metrics | `docs/eval/` |
|
||||
| brains vs sources / topology | `docs/architecture/brains-and-sources.md`, `topologies.md` |
|
||||
| skill routing | `skills/RESOLVER.md` |
|
||||
| agent bootstrap (paste-in install, hooks, `gbrain bootstrap`, sweep, keyless) | `docs/guides/bootstrap.md` + `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + the KEY_FILES bootstrap cluster |
|
||||
| shipping a release / CHANGELOG / PR conventions | `docs/RELEASING.md` (ship IRON RULES stay inline below) |
|
||||
|
||||
The per-file index (`## Key files`), the thin-client routing seam, and the testing
|
||||
@@ -190,9 +192,10 @@ Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The matrix
|
||||
has three verbatim homes: this section, the `gbrain init` picker copy
|
||||
(`src/commands/init-mode-picker.ts`), and `INSTALL_FOR_AGENTS.md` Step
|
||||
3.5** — update all three when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
@@ -272,8 +275,9 @@ audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 50+ skills
|
||||
(the current list lives in `skills/manifest.json`) organized by `skills/RESOLVER.md`
|
||||
(`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
briefing, migrate, setup, publish.
|
||||
@@ -477,7 +481,7 @@ ms, max waiters) for `--json`; a one-line summary prints to stderr.
|
||||
|
||||
## Version locations (single source of truth: `VERSION` file)
|
||||
|
||||
Every release advances the version in **five files at once**. Keep these in
|
||||
Every release advances the version in **six files at once**. Keep these in
|
||||
sync. `/ship` enforces this via Step 12's idempotency check (VERSION vs
|
||||
package.json drift), but the canonical list lives here so future runs and
|
||||
the auto-update agent know where to look.
|
||||
@@ -493,7 +497,7 @@ four numeric segments are required first. Historical 3-segment versions
|
||||
(`0.31.3`, `0.22.1`) remain valid in `git log` and migration filenames
|
||||
(`skills/migrations/v0.21.0.md`); do NOT rewrite them. Going forward only.
|
||||
|
||||
**Required (every release must update all five):**
|
||||
**Required (every release must update all six):**
|
||||
|
||||
| File | What lives there | Format |
|
||||
|---|---|---|
|
||||
@@ -502,6 +506,9 @@ four numeric segments are required first. Historical 3-segment versions
|
||||
| `CHANGELOG.md` | Top entry header `## [0.31.4.1] - YYYY-MM-DD` plus the "To take advantage of v0.31.4.1" block. | Standard Keep-a-Changelog header. |
|
||||
| `TODOS.md` | Any TODO entries that mention "follow-up from vX.Y.Z.W" use the version of the release that filed them. Update only when filing NEW follow-up TODOs. | Inline `vX.Y.Z.W` references in TODO bodies. |
|
||||
| `CLAUDE.md` | The Key Files section's per-file annotations carry `vX.Y.Z.W (#NNN)` tags noting which release introduced a behavior. Update whenever a wave's annotations get folded in. | Inline `vX.Y.Z.W (#NNN, contributed by @user)` references. |
|
||||
| `openclaw.plugin.json` | OpenClaw plugin manifest (v0.45.6.0, #4033). Hand-maintained; `test/openclaw-plugin-manifest.test.ts` fails the suite if it drifts from `package.json`. Merges from master auto-resolve it to master's version — re-bump it with the trio. | `"version": "0.45.12.0"` |
|
||||
| `BOOTSTRAP_FOR_AGENTS.md` | Runbook stamp on line 1. `scripts/check-bootstrap-tag.sh` (in `bun run verify` + CI) fails when it drifts from `VERSION`; refresh it in the same commit as the bump. | `<!-- gbrain-runbook-stamp: X.Y.Z.W -->` |
|
||||
| `templates/bootstrap/template-repo/` | Vendored template tree with an embedded version stamp. Auto-derived, but NOT by `bun install`: run `bun run scripts/generate-template-repo.ts --out templates/bootstrap/template-repo` after the bump; `scripts/check-bootstrap-templates.sh` fails CI on drift. | `<!-- gbrain-template-stamp: X.Y.Z.W -->` in generated files. |
|
||||
|
||||
**Auto-derived (no manual edit; refreshed by their own commands):**
|
||||
|
||||
|
||||
+70
-100
@@ -1,27 +1,5 @@
|
||||
# Contributing to GBrain
|
||||
|
||||
## Human-authored intent (required, no exceptions)
|
||||
|
||||
Effective 2026-08-02, every issue and every pull request must include:
|
||||
|
||||
1. **A paragraph you wrote yourself**, explaining why you are opening this.
|
||||
What you were doing, what went wrong or what you needed, why it matters to
|
||||
you. AI-generated or AI-polished text is not accepted here — this one
|
||||
paragraph is the human part. Rough grammar is fine and preferred over
|
||||
polish.
|
||||
2. **A screenshot showing gbrain actually being used** in the situation you
|
||||
are describing — your terminal, your agent session, your logs. Proof the
|
||||
need is real, not hypothetical.
|
||||
|
||||
Issues or PRs without both are closed without review. You may reopen once
|
||||
they're added.
|
||||
|
||||
Scrub anything private before you attach a screenshot: real names, companies,
|
||||
API keys, brain contents. See the privacy rule in `CLAUDE.md`. A redacted
|
||||
screenshot is fine; a missing one is not.
|
||||
|
||||
AI assistance for the *code* is fine. The intent paragraph is not code.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
@@ -70,7 +48,9 @@ src/
|
||||
core/
|
||||
operations.ts Contract-first operation definitions (the foundation)
|
||||
engine.ts BrainEngine interface
|
||||
postgres-engine.ts Postgres implementation
|
||||
engine-factory.ts Engine factory (dynamic import of the configured engine)
|
||||
postgres-engine.ts Postgres + pgvector implementation
|
||||
pglite-engine.ts PGLite (embedded Postgres via WASM) implementation
|
||||
db.ts Connection management + schema loader
|
||||
import-file.ts Import pipeline (chunk + embed + tags)
|
||||
types.ts TypeScript types
|
||||
@@ -81,12 +61,16 @@ src/
|
||||
supabase-admin.ts Supabase admin API
|
||||
file-resolver.ts MIME detection + content hashing
|
||||
migrate.ts Migration helpers
|
||||
bootstrap/ Agent-bootstrap flow (interview, hooks, repo, verify)
|
||||
yaml-lite.ts Lightweight YAML parser
|
||||
chunkers/ 3-tier chunking (recursive, semantic, llm)
|
||||
search/ Hybrid search (vector, keyword, hybrid, expansion, dedup)
|
||||
embedding.ts OpenAI embedding service
|
||||
embedding.ts Embedding service (provider-routed; ZeroEntropy default)
|
||||
mcp/
|
||||
server.ts MCP stdio server (generated from operations)
|
||||
http-transport.ts HTTP MCP transport (OAuth, body caps)
|
||||
dispatch.ts Op dispatch + scope enforcement + param redaction
|
||||
rate-limit.ts Rate limiting
|
||||
schema.sql Postgres DDL
|
||||
skills/ Fat markdown skills for AI agents
|
||||
test/ Unit tests (bun test, no DB required)
|
||||
@@ -99,15 +83,21 @@ test/e2e/ E2E tests (requires DATABASE_URL, real Postgres+pgvect
|
||||
docs/ Architecture docs
|
||||
```
|
||||
|
||||
Per-file invariants live in `docs/architecture/KEY_FILES.md` — read a file's entry
|
||||
before editing it.
|
||||
|
||||
## Running tests
|
||||
|
||||
The canonical reference for test tiers, isolation rules, timing, and the E2E
|
||||
lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version:
|
||||
|
||||
```bash
|
||||
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
|
||||
bun run test # parallel 8-shard fan-out + serial post-pass
|
||||
# Inner edit loop (~85s on a Mac dev box)
|
||||
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
|
||||
bun test test/markdown.test.ts # specific unit test
|
||||
|
||||
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
|
||||
bun run verify # privacy + jsonb + progress + test-isolation + wasm + admin-build + resolver + typecheck
|
||||
# Pre-push gate (19+ parallel checks + typecheck)
|
||||
bun run verify
|
||||
|
||||
# Pre-merge sanity (everything CI runs)
|
||||
bun run test:full # verify + parallel unit + slow + smart e2e
|
||||
@@ -125,92 +115,54 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
|
||||
DATABASE_URL=postgresql://... bun run test:e2e
|
||||
```
|
||||
|
||||
Use `bun run verify` before pushing. The guard chain catches: banned fork-name
|
||||
leaks (`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
|
||||
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
|
||||
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
|
||||
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
|
||||
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
|
||||
(`scripts/check-progress-to-stdout.sh`), test-isolation rule violations
|
||||
(`scripts/check-test-isolation.sh` — see "Writing tests that survive the parallel
|
||||
loop" below), silent fallback to recursive chunking in the compiled binary
|
||||
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
|
||||
(`scripts/check-admin-build.sh`), and resolver drift on bundled skills
|
||||
(`bun run check:resolver` — strict-mode `check-resolvable` that exit-1s on any
|
||||
warning, added in v0.41.14.0 to catch SKILL.md frontmatter ↔ RESOLVER.md drift
|
||||
before merge). `bun run check:all` runs the full historical sweep including the
|
||||
trailing-newline and exports-count checks.
|
||||
(`scripts/check-admin-build.sh`), resolver drift on bundled skills
|
||||
(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full
|
||||
historical sweep including the trailing-newline and exports-count checks.
|
||||
|
||||
### Writing tests that survive the parallel loop
|
||||
|
||||
`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the
|
||||
same shard share a process, so process-global state leaks between them. Four
|
||||
lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
|
||||
`bun run test` shards 1000+ unit-test files across up to 4 worker processes,
|
||||
capping total concurrency (shards × intra-shard files) to available memory and
|
||||
re-running OOM-killed or externally-killed files serially before calling them
|
||||
failures (see `docs/TESTING.md` for the rescue-pass details and knobs). Files
|
||||
in the same shard share a process, so process-global state leaks between them.
|
||||
Four lint rules (`scripts/check-test-isolation.sh`, R1–R4) enforce isolation:
|
||||
no direct `process.env` mutation (use `withEnv()` from
|
||||
`test/helpers/with-env.ts`), no `mock.module(...)` outside `*.serial.test.ts`,
|
||||
and every `new PGLiteEngine(` goes inside the canonical `beforeAll` block with
|
||||
a paired `afterAll(disconnect)`.
|
||||
|
||||
| Rule | What it bans | Fix |
|
||||
|---|---|---|
|
||||
| **R1** | Direct `process.env.X = ...` mutation | Use `withEnv()` from `test/helpers/with-env.ts`, or rename to `*.serial.test.ts` |
|
||||
| **R2** | `mock.module(...)` anywhere in the file | Rename to `*.serial.test.ts` |
|
||||
| **R3** | `new PGLiteEngine(` outside ~50 lines after `beforeAll(` | Use the canonical PGLite block (see below) |
|
||||
| **R4** | `new PGLiteEngine(` without paired `afterAll(disconnect)` | Add the `afterAll(() => engine.disconnect())` |
|
||||
**The full rules, the canonical PGLite block, the `withEnv` pattern, and the
|
||||
`*.serial.test.ts` quarantine policy live in
|
||||
[`docs/TESTING.md`](docs/TESTING.md#test-isolation-lint-and-helpers)
|
||||
— read that before writing a new test file.** Files that predate the rules are
|
||||
listed in `scripts/check-test-isolation.allowlist`; the allow-list MUST shrink
|
||||
over time — never add new entries.
|
||||
|
||||
Canonical PGLite block (R3 + R4 compliant — paste this verbatim):
|
||||
|
||||
```ts
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
afterAll(async () => { await engine.disconnect(); });
|
||||
beforeEach(async () => { await resetPgliteState(engine); });
|
||||
```
|
||||
|
||||
Env-touching tests:
|
||||
|
||||
```ts
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
test('reads OPENAI_API_KEY', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
|
||||
expect(loadConfig().openai_key).toBe('sk-test');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
`withEnv` saves and restores keys via try/finally including when the callback
|
||||
throws. Cross-test safe; **NOT** intra-file concurrent-safe (`process.env` is
|
||||
process-global). Files using `withEnv` stay outside the future
|
||||
`test.concurrent()` codemod's eligibility filter.
|
||||
|
||||
When to quarantine instead of fix: rename to `*.serial.test.ts` if the file
|
||||
uses `mock.module(...)`, is genuinely env-coupled (module-load env readers +
|
||||
ESM caching defeat dynamic-import-after-env tricks), or intentionally shares
|
||||
state across `it()` boundaries. Quarantine count cap: 10 (informational).
|
||||
|
||||
Files that violated these rules at the v0.26.7 baseline are listed in
|
||||
`scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over
|
||||
time** ... never add new entries. v0.26.8 (env sweep) and v0.26.9 (PGLite sweep
|
||||
+ codemod) remove entries as files get fixed.
|
||||
|
||||
### Local CI gate (recommended before pushing, v0.23.1+)
|
||||
### Local CI gate (recommended before pushing)
|
||||
|
||||
```bash
|
||||
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
|
||||
bun run ci:local # full gate: gitleaks + guards/typecheck + 4-shard parallel unit + E2E
|
||||
bun run ci:local:diff # gate with diff-aware E2E selector
|
||||
bun run ci:select-e2e # print which E2E files the selector would run
|
||||
```
|
||||
|
||||
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
|
||||
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
|
||||
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
|
||||
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
|
||||
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
|
||||
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
`ci:local` spins up four pgvector services plus a transaction-mode PgBouncer via
|
||||
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite
|
||||
sharded 4 ways in parallel, then tears down. Named volumes keep the install warm
|
||||
across runs. Requires Docker (Docker Desktop, OrbStack, or Colima) and `gitleaks`
|
||||
on host (`brew install gitleaks`). Override the postgres host port with
|
||||
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
|
||||
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
|
||||
Fail-closed selector: an unmapped `src/` change runs ALL E2E files. Hand-tune
|
||||
narrower mappings via `scripts/e2e-test-map.ts`.
|
||||
|
||||
### PR-side security checks
|
||||
@@ -239,6 +191,15 @@ automatically appears in the CLI, MCP server, and tools-json:
|
||||
For CLI-only commands (init, upgrade, import, export, files, embed, doctor, sync):
|
||||
1. Create `src/commands/mycommand.ts`
|
||||
2. Add the case to `src/cli.ts`
|
||||
3. Regenerate the flag registry: `bun run build:flag-registry`. The CLI rejects
|
||||
unknown flags before dispatch; each CLI-only command's legal flag set is
|
||||
derived from its source into `src/core/cli-flag-registry.generated.ts`.
|
||||
`test/cli-flag-validation.test.ts` pins registry freshness, drift, and
|
||||
consumption evidence (a safety flag like `--dry-run` may only be advertised
|
||||
if the command's code actually reads it), so a stale registry fails the
|
||||
build. At runtime a missing registry entry fails open — a forgotten regen
|
||||
never bricks a command. Rerun the regen whenever you add or remove a flag
|
||||
on an existing command, too.
|
||||
|
||||
Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
|
||||
|
||||
@@ -247,7 +208,7 @@ Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
|
||||
See `docs/ENGINES.md` for the full guide. In short:
|
||||
|
||||
1. Create `src/core/myengine-engine.ts` implementing `BrainEngine`
|
||||
2. Add to engine factory in `src/core/engine.ts`
|
||||
2. Add to the engine factory in `src/core/engine-factory.ts`
|
||||
3. Run the test suite against your engine
|
||||
4. Document in `docs/`
|
||||
|
||||
@@ -332,7 +293,7 @@ NDJSON wire format is documented in
|
||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||
|
||||
For public benchmark coverage on top of replay, `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
|
||||
<dataset.jsonl>` runs LongMemEval against gbrain's hybrid
|
||||
retrieval. One in-memory PGLite per question, runtime-enumerated
|
||||
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
|
||||
published `evaluate_qa.py`. Use it alongside replay when changes affect
|
||||
@@ -341,9 +302,18 @@ regressions on YOUR queries, LongMemEval catches them on a public set the
|
||||
benchmark community already cites. See the "Public benchmarks: LongMemEval"
|
||||
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
|
||||
## Shipping
|
||||
|
||||
Releases go through the `/ship` skill, never hand-rolled. The full release +
|
||||
contributor process (CHANGELOG voice, version-locations sync, PR conventions,
|
||||
community-PR-wave workflow) lives in [`docs/RELEASING.md`](docs/RELEASING.md).
|
||||
Community PRs are batched into release waves rather than merged one-by-one;
|
||||
contributor attribution stays attached via `Co-Authored-By:` trailers and every
|
||||
accepted contribution is credited in `CHANGELOG.md`.
|
||||
|
||||
## Welcome PRs
|
||||
|
||||
- SQLite engine implementation
|
||||
- Additional engine implementations (see [`docs/ENGINES.md`](docs/ENGINES.md))
|
||||
- Docker Compose for self-hosted Postgres
|
||||
- Additional migration sources
|
||||
- New enrichment API integrations
|
||||
|
||||
+45
-10
@@ -3,6 +3,11 @@
|
||||
Read this entire file, then follow the steps. Ask the user for API keys when needed.
|
||||
Target: ~30 minutes to a fully working brain.
|
||||
|
||||
> **Scope note:** this guide installs the BRAIN (for agent platforms like OpenClaw
|
||||
> and Hermes, or a standalone brain). If you are Claude Code or Codex and the human
|
||||
> asked you to become their persistent personal agent — identity + memory + private
|
||||
> repo — follow `BOOTSTRAP_FOR_AGENTS.md` instead.
|
||||
|
||||
## Step 0: If you are not Claude Code
|
||||
|
||||
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
|
||||
@@ -16,10 +21,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
|
||||
|
||||
## Step 1: Install GBrain
|
||||
|
||||
<!-- npm-trap + #218 recovery: canonical copy lives in README.md ("Install" warning) — sync edits. -->
|
||||
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
|
||||
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
|
||||
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
|
||||
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
|
||||
> supported sources are `github:garrytan/gbrain` (optionally pinned as
|
||||
> `github:garrytan/gbrain#latest-stable`, the form the bootstrap flow mandates) and a
|
||||
> git clone, exactly as shown below.
|
||||
> If an unrelated npm install is already present, remove it first
|
||||
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
|
||||
|
||||
@@ -82,12 +90,14 @@ NOT inside ~/gbrain.
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
tier is Haiku-class or no expansion-capable API key — Anthropic, OpenAI, or
|
||||
Google — is configured). The init output included the cost matrix below preceded
|
||||
by `[AGENT]` markers. You must NOT silently accept the default. Stop and ask the
|
||||
operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
<!-- Cost matrix: three verbatim homes — CLAUDE.md "Search Mode", src/commands/init-mode-picker.ts, and this block. Sync all three when refreshing. -->
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
@@ -198,15 +208,25 @@ scaffold the bundled skills into it:
|
||||
|
||||
```bash
|
||||
cd /path/to/agent/workspace
|
||||
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
|
||||
gbrain skillpack scaffold --all # copy the 50+ bundled skills + RESOLVER.md
|
||||
```
|
||||
|
||||
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
|
||||
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
|
||||
diff against gbrain's bundle when you want upstream improvements. (The legacy
|
||||
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
|
||||
`gbrain skillpack install` managed-block model was removed in v0.33 — run
|
||||
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
|
||||
|
||||
**If you are Hermes:** register gbrain as your MCP server:
|
||||
|
||||
```bash
|
||||
printf 'Y\n' | hermes mcp add gbrain --env GBRAIN_HOME=$HOME --connect-timeout 60 --command $(which gbrain) --args serve
|
||||
```
|
||||
|
||||
Keep `--args` last (everything after it becomes server argv) and verify with
|
||||
`hermes mcp test gbrain` — the add exits 0 even on failure. Full reference:
|
||||
[docs/mcp/HERMES.md](docs/mcp/HERMES.md).
|
||||
|
||||
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
|
||||
bundled copy at `~/gbrain/skills/RESOLVER.md` when running from the cloned repo). It's
|
||||
the skill dispatcher — tells you which skill to read for any task. Save this to your
|
||||
@@ -242,7 +262,13 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), o
|
||||
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
|
||||
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
— or `gbrain sync --watch` for a continuous loop.
|
||||
— or `gbrain sync --watch` for a continuous loop. Safe on keyless brains:
|
||||
a bare `gbrain embed --stale` exits 0 with a stderr note when embeddings
|
||||
are disabled, so the chain doesn't break.
|
||||
- **Health gate** (daily): `gbrain autopilot --status` — exit 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused), 2 the daemon took itself out of rotation. Filesystem-only, so it
|
||||
works during DB outages.
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
|
||||
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
@@ -261,8 +287,17 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
|
||||
|
||||
## Step 9: Verify
|
||||
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
|
||||
actually works) is the most important.
|
||||
Read `docs/GBRAIN_VERIFY.md` and run every verification check in it. Check #4
|
||||
(live sync actually works) is the most important.
|
||||
|
||||
Once verification passes and the brain has content, run the activation probe:
|
||||
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
|
||||
See "The onboard surface" below for what the recommendations mean and the
|
||||
consent gates around unattended remediation.
|
||||
|
||||
## Upgrade
|
||||
|
||||
@@ -305,7 +340,7 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
|
||||
## v0.42.0+ onboard surface (NEW)
|
||||
## The onboard surface
|
||||
|
||||
`gbrain onboard` is the activation surface gbrain did not have before.
|
||||
Once your brain has any content, run `gbrain onboard --check --json` to
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Search gives you raw pages. GBrain gives you the answer.** It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box. Run a full autonomous agent on top of it, or just wire it into Claude Code or Codex as a supercharged retrieval layer in one command; either way your coding agent stops being amnesiac about everything that isn't code.
|
||||
|
||||
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
|
||||
I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: **155,795 pages, 24,589 people, 5,340 companies**, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.
|
||||
|
||||
**And now it works as a company brain too.** Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the [company-brain](https://www.ycombinator.com/rfs#company-brain) shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. **[Tutorial: set up GBrain as your company brain →](docs/tutorials/company-brain.md)**
|
||||
|
||||
@@ -11,11 +11,11 @@ Lots of personal-knowledge systems give you keyword matching and grep in a box.
|
||||
- **A synthesis layer that gives you the actual answer.** Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
|
||||
- **A self-wiring knowledge graph.** Every page write extracts entity refs and creates typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, **+31.4 points P@5** over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
|
||||
The point of building a 150K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.
|
||||
|
||||
It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
> **~15 minutes to a working personal agent** on the recommended Codex / Claude Code path (mostly a short interview); ~30 minutes for the always-on OpenClaw / Hermes setup. Database ready in 2 seconds either way (PGLite, no server).
|
||||
|
||||
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
|
||||
|
||||
@@ -75,11 +75,47 @@ This is the difference between a search engine and a brain. Search finds the pag
|
||||
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
|
||||
> shadowing npm install and prints the fix.
|
||||
|
||||
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
|
||||
GBrain is designed to be installed and operated by an AI agent. **New to GBrain? Start with Codex** — it runs on the ChatGPT subscription you already have, takes ~15 minutes, and deploys nothing. Already living in Claude Code? Its path is identical. Want GBrain running the way it was designed to run — always on, enriching your brain around the clock? That's OpenClaw or Hermes, at real server + API cost. Each path below is complete on its own. (Wiring it up by hand instead? Jump to [CLI standalone](#cli-standalone-no-agent) or the [MCP table](#connect-gbrain-to-your-ai-client-mcp).)
|
||||
|
||||
### Have your agent install it (recommended)
|
||||
### For Codex — the recommended first step
|
||||
|
||||
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
|
||||
Turn Codex into your persistent personal agent. Works in the **ChatGPT desktop app** (open Codex on a folder) and in the **Codex CLI** (`codex` in a terminal) — same install, same result. Open Codex in a **new, empty folder** (not an existing code project) — that folder becomes your agent's own **private GitHub repo**, which bootstrap creates and privacy-verifies for you. Then paste:
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
Codex will ask for command approvals during the install — approving them is the sandbox working as intended. What you get, in about 15 minutes: a short interview (6 required questions) → your agent's identity (SOUL.md, USER.md, MEMORY.md) rendered from your own answers, never invented → a local PGLite brain (2 seconds, no server, no Docker) → MCP wired so every session can search and write memory → a **private** GitHub repo, created and privacy-verified, as your agent's durable body. Works with **zero API keys** — keyword search plus memory your agent writes itself; one optional key upgrades capabilities (OpenAI: semantic search + automatic fact extraction; Voyage: semantic search; Anthropic: fact extraction). Codex reads brain context through its tools each turn (pull-based). The click moment: tell it one small thing to remember, restart Codex, then ask for it back — the answer comes from the brain, not from this chat's context (which the restart cleared). That cross-session round-trip is the whole product; "what's my name / my top jobs?" is answered from your identity files, which is nice but not the same trick.
|
||||
|
||||
Two things worth understanding once it's running: **you own the brain** — every memory is a markdown file in that private repo (read it, clone it to a second machine, delete it and the brain is gone) — and **the first skill to run is `cold-start`**: say "fill my brain" and your agent imports your Gmail, calendar, and contacts (via [ClawVisor](https://clawvisor.com), an OAuth vault so the agent never holds raw tokens) or offline archives like Google Takeout, one consented step at a time. An empty brain is a database; a filled one is a memory.
|
||||
|
||||
> **Prefer to make the repo yourself?** Create a new **empty** private repo **under your own GitHub account** (no README/.gitignore/license), clone it, open the clone in Codex, and paste the same block — bootstrap detects your empty repo and adopts it instead of creating one. The repo must be empty and personal-account-owned; org-owned repos are refused (create one under your account, or let bootstrap make it).
|
||||
|
||||
### For Claude Code — turn it into your persistent personal agent
|
||||
|
||||
Works in the **desktop app** and in the **CLI** (`claude` in a terminal) — identical harness, identical result. Open Claude Code in a **new, empty folder** (not an existing code project) — that folder becomes your agent's own **private GitHub repo**, created and privacy-verified for you. Then paste the same block:
|
||||
|
||||
```
|
||||
Read and follow every step of:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/latest-stable/BOOTSTRAP_FOR_AGENTS.md
|
||||
Goal: set yourself up as my persistent personal agent in this folder, with gbrain
|
||||
as your memory. Interview me before writing any identity file — never invent
|
||||
answers. Ask before anything destructive. You are not done until
|
||||
`gbrain bootstrap verify` exits 0.
|
||||
```
|
||||
|
||||
Everything from the Codex path applies — interview, identity from your own answers, local brain, private repo, keyless mode — plus Claude Code gets **per-turn context hooks** (on by default, with an opt-out): your brain loads automatically into every prompt, and your work persists to your private repo on a per-turn cadence (debounced ~5 min locally, every turn in a cloud sandbox — this covers the `/exit` case the harness never fires a session-end hook on), with a notice on your next turn if a push ever fails. This works in a **Claude Code cloud session** too, not just on your laptop: verification falls back to pure git protocol when the sandbox blocks the GitHub API, and `gbrain bootstrap cloud-setup-script` prints the environment setup recipe. The click moment: tell it one small thing to remember, restart the session, then ask for it back — a fresh session has no chat context, so the answer can only come from the brain. That cross-session round-trip is the whole product ("what's my name?" is answered from your identity files — nice, but not the same trick). Same two follow-ups as the Codex path: you own the brain (markdown in your private repo), and `cold-start` is the first skill to run — "fill my brain" imports your email, calendar, and contacts (ClawVisor) or offline archives, one consented step at a time. Full contract, security posture, cloud sandboxes, and uninstall: [docs/guides/bootstrap.md](docs/guides/bootstrap.md).
|
||||
|
||||
> **Prefer to make the repo yourself?** Create a new **empty** private repo **under your own GitHub account** (no README/.gitignore/license), clone it, open the clone in Claude Code (CLI or the desktop app's open-a-repo flow), and paste the same block — bootstrap adopts your empty repo instead of creating one. The repo must be empty and personal-account-owned; org-owned repos are refused.
|
||||
|
||||
### For OpenClaw or Hermes — GBrain as intended, always on
|
||||
|
||||
This is GBrain used the way it was designed to be used: a server-hosted agent with 24/7 crons, continuous ingestion, and the overnight dream cycle that enriches your brain while you sleep — your agent works whether your laptop is open or not. It's also the highest-cost path: a deployed server (8GB+ RAM) plus raw API token usage that scales with how hard your agent runs, well beyond a chat subscription. Start here if you want the full experience from day one; start with Codex above if you want to feel it first. If you don't have a platform running yet, both deploy in one click:
|
||||
|
||||
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
|
||||
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
|
||||
@@ -91,21 +127,21 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
|
||||
The agent installs GBrain, creates the brain, asks for your API keys, loads the 50+ bundled skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
|
||||
|
||||
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
|
||||
|
||||
### Quick start: Claude Code or Codex
|
||||
### Lighter ways in
|
||||
|
||||
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
|
||||
|
||||
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel:
|
||||
**Just want a memory for your coding agent — no identity, no repo.** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the seven-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget`, plus `context_pack` + `delta` since v0.45.7 — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full tool wall; drop the flag for every operation:
|
||||
|
||||
```bash
|
||||
gbrain init --pglite # 2-second local brain (no Docker)
|
||||
claude mcp add gbrain -- gbrain serve # or: codex mcp add gbrain -- gbrain serve
|
||||
gbrain init --pglite # 2-second local brain (no Docker)
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs # or: codex mcp add gbrain -- gbrain serve --surface verbs
|
||||
```
|
||||
|
||||
If `claude` is not found, install Claude Code first — or use the per-harness blocks in the [protocol doc](docs/protocol/MEMORY_VERBS_v1.md). Heads-up: memories agents save default to brain-wide visibility (every connected agent can recall them); pass `visibility: "private"` for local-only facts.
|
||||
|
||||
**Already have a brain on a remote host** (OpenClaw, Hermes, or any `gbrain serve --http`)? Point your laptop agents at it with one command each — `--install` wires it up and smoke-tests the token before handoff:
|
||||
|
||||
```bash
|
||||
@@ -113,18 +149,9 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --install
|
||||
gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install # Codex
|
||||
```
|
||||
|
||||
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — both paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
|
||||
**Brain-only install into another coding agent** (Cursor, Claude Cowork, or anything that can fetch a URL and run shell commands) — paste the OpenClaw/Hermes block above (`INSTALL_FOR_AGENTS.md`); it installs the brain, skills, and dream cycle without the personal-agent identity layer. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
|
||||
|
||||
### Install the full autonomous setup into your existing agent
|
||||
|
||||
Want the whole thing — local brain, 43 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
|
||||
|
||||
```
|
||||
Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.
|
||||
**[→ Full walkthrough: give your coding agent a memory](docs/tutorials/connect-coding-agent.md)** — the memory-only paths end to end, plus the brain-first protocol you paste into `CLAUDE.md` / `AGENTS.md` and the four habits that make it actually change how you work.
|
||||
|
||||
### CLI standalone (no agent)
|
||||
|
||||
@@ -140,11 +167,13 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
|
||||
|
||||
### Connect GBrain to your AI client (MCP)
|
||||
|
||||
GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:
|
||||
GBrain exposes nearly all of its 100+ operations as MCP tools (stdio and HTTP; a handful of local-only ops stay CLI-side) — or exactly the seven memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
|
||||
|
||||
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
|
||||
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
|
||||
- **[Cursor / Windsurf / any stdio MCP client](docs/mcp/CLAUDE_CODE.md)** — same shape, add `{"command": "gbrain", "args": ["serve"]}` to your MCP config.
|
||||
- **[Hermes](docs/mcp/HERMES.md)** — `printf 'Y\n' | hermes mcp add gbrain --env GBRAIN_HOME=$HOME --connect-timeout 60 --command $(which gbrain) --args serve`. Keep `--args` last, and verify with `hermes mcp test gbrain` (the add exits 0 even on failure).
|
||||
- **[OpenClaw](docs/mcp/OPENCLAW.md)** — the ClawHub bundle plugin registers gbrain automatically (`openclaw.plugin.json` ships in this repo), or add `{"command": "gbrain", "args": ["serve"]}` to `~/.openclaw/config.json`'s `mcpServers`.
|
||||
- **[Claude Desktop (Cowork)](docs/mcp/CLAUDE_DESKTOP.md)** — Settings → Integrations → add the URL of your HTTP server. Remote only; the local `claude_desktop_config.json` does not work for remote servers.
|
||||
- **[Claude Cowork (team plan)](docs/mcp/CLAUDE_COWORK.md)** — org Owner adds the connector under Organization Settings → Connectors.
|
||||
- **[Perplexity Computer](docs/mcp/PERPLEXITY.md)** — `gbrain connect https://your-host/mcp --agent perplexity --oauth --register` mints a least-privilege OAuth client and prints the Issuer/Client ID/Secret to paste into Settings → Connectors (OAuth is the right path for a cloud connector; a bearer token also works for local use). Pro subscription required.
|
||||
@@ -176,7 +205,7 @@ gbrain think "who's working on AI agents at portfolio companies?"
|
||||
|
||||
**`gbrain think`** runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.
|
||||
|
||||
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.
|
||||
**Why it compounds.** Pair the brain layer with `find_trajectory` and you get answers like *"how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here"*: well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 150K-page brain is worth the effort.
|
||||
|
||||
`gbrain agent run "..."` exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.
|
||||
|
||||
@@ -262,7 +291,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
## Capabilities
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). The install picker default-applies `tokenmax` (it recommends `conservative` for Haiku-class subagent tiers or keyless setups); a brain with `search.mode` unset resolves to `balanced` at query time. The ZeroEntropy reranker is on in `balanced` and `tokenmax`, off in `conservative`. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
@@ -286,9 +315,9 @@ gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
**50+ curated skills** (the current list lives in [`skills/manifest.json`](skills/manifest.json)). Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. `gbrain eval brainbench` runs the cross-harness memory conformance suite: know-to-ask, push precision/recall, write-back fidelity, and cross-session continuity, scored per harness seam (your OpenClaw's production pipeline plus Claude Code and Codex injection contracts) against a committed 141-fixture synthetic corpus — hermetic by default (in-memory PGLite, no keys, seconds), and CI gates every PR against master's committed baseline. Methodology in [`docs/eval/BRAINBENCH.md`](docs/eval/BRAINBENCH.md); search-mode methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
|
||||
|
||||
@@ -300,14 +329,14 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
|
||||
|
||||
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
|
||||
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
|
||||
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
|
||||
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
|
||||
- **Embedding providers**: a dozen providers covered — OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
|
||||
- **Rerankers**: ZeroEntropy `zerank-2` hosted (the default; on in `balanced` and `tokenmax` modes) plus the `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
|
||||
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
|
||||
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
|
||||
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines the 140+ methods both engines implement; CLI and MCP server are generated from one source.
|
||||
|
||||
**Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md).
|
||||
|
||||
@@ -317,14 +346,13 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
|
||||
**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
|
||||
|
||||
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
|
||||
|
||||
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
|
||||
two flags + a recommended pattern. Switch your cron to a per-source loop
|
||||
with shell `timeout(1)` doing the OS-level kill and gbrain self-terminating
|
||||
gracefully half-a-minute earlier:
|
||||
**Hourly cron sync keeps timing out on a federated brain?** Switch your
|
||||
cron to a per-source loop with shell `timeout(1)` doing the OS-level kill
|
||||
and gbrain self-terminating gracefully half-a-minute earlier:
|
||||
|
||||
```bash
|
||||
gbrain sync --break-lock --all --max-age 1800
|
||||
@@ -337,19 +365,17 @@ When `--timeout` fires mid-import, `gbrain sync` exits 0 with status
|
||||
`partial` and `last_commit` UNCHANGED — the next run re-walks the same
|
||||
diff and `content_hash` short-circuits already-imported files. The
|
||||
`--max-age 1800` first command self-heals any wedged-but-alive locks
|
||||
left by a hung previous run, using the v98 `last_refreshed_at` semantic
|
||||
(NOT `acquired_at`) so healthy long-running holders are safe by
|
||||
construction. See the v0.41.13.0 entry in [`CHANGELOG.md`](CHANGELOG.md)
|
||||
for the honest scope notes (extract + embed phases run to completion;
|
||||
30-min rollout window for `--max-age` post-migration v98; full-sync
|
||||
triggers deferred to v0.42+).
|
||||
left by a hung previous run, keyed on the lock's last refresh time
|
||||
(NOT when it was acquired) so healthy long-running holders are safe by
|
||||
construction. Scope note: the extract + embed phases still run to
|
||||
completion once started; `--timeout` interrupts the import walk only.
|
||||
|
||||
**Dream cycle silently losing wiki links on Supabase?** v0.41.19.0 fixes
|
||||
the bug class structurally. The engine now self-retries every bulk batch
|
||||
write (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) on
|
||||
Supavisor pooler blips, with a 12s worst-case wait that covers the full
|
||||
5-10s circuit-breaker recovery window. `gbrain doctor` surfaces incidents
|
||||
via the new `batch_retry_health` check (reads the last 24h of
|
||||
**Dream cycle silently losing wiki links on Supabase?** The engine
|
||||
self-retries every bulk batch write (`addLinksBatch` /
|
||||
`addTimelineEntriesBatch` / `upsertChunks`) on Supavisor pooler blips,
|
||||
with a 12s worst-case wait that covers the full 5-10s circuit-breaker
|
||||
recovery window. `gbrain doctor` surfaces incidents via the
|
||||
`batch_retry_health` check (reads the last 24h of
|
||||
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl`). To tune for an unusually
|
||||
slow pooler:
|
||||
|
||||
@@ -367,34 +393,33 @@ retry wrap is engine-level, but PGLite has no pooler so retries never
|
||||
fire in practice.
|
||||
|
||||
**Dream cycle losing ~150 link rows per run with `'No database
|
||||
connection: connect() has not been called'` errors in the log?** v0.41.27.0
|
||||
makes the retry layer self-heal on a nulled-out database singleton. A
|
||||
new `reconnect` callback on `withRetry` rebuilds the connection between
|
||||
attempts; `PostgresEngine.batchRetry` injects `() => this.reconnect()`
|
||||
connection: connect() has not been called'` errors in the log?** The
|
||||
retry layer self-heals on a nulled-out database singleton: a
|
||||
`reconnect` callback on `withRetry` rebuilds the connection between
|
||||
attempts, and `PostgresEngine.batchRetry` injects `() => this.reconnect()`
|
||||
so engine-level batch writes survive a mid-cycle disconnect by something
|
||||
else in the same process. Same release: `gbrain capture` no longer trails
|
||||
a `'No database connection'` stderr line from a background facts:absorb
|
||||
worker firing after CLI exit — the op-dispatch finally block awaits
|
||||
else in the same process. `gbrain capture` also no longer trails a
|
||||
`'No database connection'` stderr line from a background facts:absorb
|
||||
worker firing after CLI exit — op dispatch awaits
|
||||
`getFactsQueue().drainPending({timeout: 1000})` before
|
||||
`engine.disconnect()`. To find which code path is still calling
|
||||
disconnect mid-process, run `gbrain doctor --json | jq '.checks[] |
|
||||
select(.id=="batch_retry_health")'`; the extended check now surfaces
|
||||
24h disconnect-call count and the most-recent caller frame from a new
|
||||
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit. (Closes #1570.)
|
||||
select(.id=="batch_retry_health")'`; the check surfaces the
|
||||
24h disconnect-call count and the most-recent caller frame from the
|
||||
`~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl` audit.
|
||||
|
||||
**`gbrain brainstorm` returning `judge_failed: true` with 0 scored
|
||||
ideas?** v0.41.21.0 closes the two bugs that caused it. The judge
|
||||
hard-coded a 4K-token output cap; for any run past ~40 ideas the call
|
||||
truncated mid-JSON and the parser threw. Same release closes a slash-
|
||||
form pricing miss: `gbrain brainstorm --judge-model
|
||||
anthropic/claude-sonnet-4-6 --max-cost 5` failed with
|
||||
`BudgetExhausted reason=no_pricing` because every pricing site only
|
||||
matched the colon form. Both shapes work now. No config change, no
|
||||
schema migration — `gbrain upgrade` is the whole fix.
|
||||
ideas?** Two historical bugs caused it, both fixed: the judge
|
||||
hard-coded a 4K-token output cap (any run past ~40 ideas truncated
|
||||
mid-JSON and the parser threw), and slash-form model ids
|
||||
(`gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
|
||||
--max-cost 5`) failed with `BudgetExhausted reason=no_pricing` because
|
||||
pricing lookups only matched the colon form. Both shapes work now. No
|
||||
config change, no schema migration — `gbrain upgrade` is the whole fix.
|
||||
|
||||
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
|
||||
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
|
||||
`reindex --markdown` now ADD current frontmatter tags and never delete,
|
||||
tags?** Upgrade — tag reconciliation is add-only now. Re-import and
|
||||
`reindex --markdown` ADD current frontmatter tags and never delete,
|
||||
so enrichment tags written to the DB (auto-tag, dream synthesize,
|
||||
signal-detector) survive a re-chunk. The reindex DB-only fallback also
|
||||
reconstructs the full markdown (frontmatter + body + timeline) before
|
||||
@@ -402,10 +427,10 @@ re-chunking, so a page with no on-disk source keeps its frontmatter,
|
||||
title, and timeline instead of getting overwritten with empty
|
||||
frontmatter. Trade-off: removing a tag from a page's frontmatter no
|
||||
longer removes it from the DB on the next sync (frontmatter-tag removal
|
||||
needs a provenance column, deferred). (Closes #1621.)
|
||||
needs a provenance column, deferred).
|
||||
|
||||
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
|
||||
v0.41.37.0 ships three things. First, name the stalling file:
|
||||
Three tools. First, name the stalling file:
|
||||
|
||||
```bash
|
||||
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
|
||||
@@ -420,28 +445,28 @@ the sync with the pack disabled and re-run extraction later:
|
||||
gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
|
||||
`gbrain schema lint` warns on the classic nested-quantifier ReDoS
|
||||
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
|
||||
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
|
||||
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
|
||||
PGLite is single-writer and a live MCP server contends for the write
|
||||
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
|
||||
for the full triage. (Closes #1569.)
|
||||
for the full triage.
|
||||
|
||||
**`gbrain init --migrate-only` / a schema migration fails on Windows
|
||||
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
|
||||
with `getaddrinfo ENOTFOUND`?** Upgrade — schema bring-up now runs its
|
||||
phases in-process instead of spawning a child `gbrain init
|
||||
--migrate-only` per phase. The spawned child died on
|
||||
Windows + bun + Supabase pooler with a DNS-resolution failure even
|
||||
though the parent connected fine; running in-process removes the spawn
|
||||
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
|
||||
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
|
||||
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
|
||||
completes in ~1-2 seconds. (Closes #1605, #1581.)
|
||||
entirely. The grandfather migration that used to hang 70+ minutes on an
|
||||
80K-page PGLite brain also runs as a chunked bulk SQL pass now (keyed on
|
||||
the page PK, soft-delete-filtered, source-safe) and completes in seconds.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
|
||||
- [`docs/guides/bootstrap.md`](docs/guides/bootstrap.md) — the persistent-personal-agent bootstrap contract (interview, identity files, hooks, private repo, security posture, uninstall)
|
||||
- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
|
||||
- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows`
|
||||
- [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory
|
||||
|
||||
@@ -1,5 +1,292 @@
|
||||
# TODOS
|
||||
|
||||
## Onboarding DX follow-ups (filed v0.45.9.0)
|
||||
|
||||
- [ ] **Retire the `config set embedding_model` dead-end across ALL surfaces.** v0.45.9.0 fixed the keyless-init notice to point at `gbrain init --force --pglite --embedding-model <id>`, but `src/core/embed-preflight.ts` (lines ~73/83/90/115) and `src/core/embedding-dim-check.ts:78` still advertise `gbrain config set embedding_model <...>`, which `src/commands/config.ts:142` hard-refuses as a schema-sizing no-op. Same dead-end class, different surfaces. Sweep them to the re-init recipe. Priority: P2.
|
||||
- [ ] **`gbrain init --supabase` migrate-model dead-end doc.** The Postgres branch of config.ts points at `docs/embedding-migrations.md`; confirm that doc exists and describes a working switch, or write it. Priority: P3.
|
||||
- [ ] **DX harness binary cache keyed on nothing.** `scripts/dx-explore.ts` reuses `.context/dx-runs/bin/gbrain` unless `--rebuild` is passed, so a second run after code changes can produce transcripts from a stale binary. Key the cache by a source hash (or rebuild when any `src/` file is newer). Dev instrument only. Priority: P3.
|
||||
- [ ] **`verify` has no MCP-registration check.** v0.45.9.0 made `bootstrap status` report the wire phase `partial` when only hooks landed (host CLI missing), but `bootstrap verify` still exits 0 in that state. Add an MCP-registration probe to verify so the "done when verify exits 0" contract also covers MCP. Priority: P2.
|
||||
- [ ] **`hasExpansionKey` misses config-plane keys + init-before-key sequencing.** The mode picker reads `process.env` only; a key routed to the 0600 config by the interview (which runs AFTER init) never influences the auto-selected search mode, and the picker never re-fires. Resolve keys through the capability/gateway fold and consider re-running the recommendation when a key is first configured. Priority: P3.
|
||||
- [ ] **`findEnvKeyTypos` KEY_SHAPE misses no-underscore typos.** `OPENAI_APIKEY` (no `_` before `KEY`) escapes the near-miss net, so that typo class now completes keyless silently instead of failing loud. Widen the regex. Priority: P3.
|
||||
- [ ] **`init-nudge` stale "4 checks" comment + 6-probe accounting.** The header still says "4 onboard checks" but six probes now run; the partial-checks message counts the page-count probe. Cosmetic. Priority: P3.
|
||||
- [ ] **FIRST LIGHT (the real first-magical-moment feature).** The v0.45.9.0 tour rewrite is the ship-now slice; the full seed-phase → compendium → scout design is PR-A (seed phase + Mirror + baton) / PR-B (compendium + scout) with one-way-door decisions (new bootstrap phase, consent key, `skills/first-light/`, a one-time Gate-3 narration exemption). Priority: P2.
|
||||
|
||||
## Ambient recall follow-ups (filed v0.45.7.0, issue #1)
|
||||
|
||||
Deferred from the ambient-recall wave (`context_pack` + `delta` frozen verbs +
|
||||
boundary runtime; CEO+ENG cleared, plan at
|
||||
`~/.claude/plans/system-instruction-you-are-working-vectorized-gem.md`). Each was
|
||||
explicitly scoped OUT with a one-line rationale — none is a bug, all are additive.
|
||||
|
||||
- [ ] **Autonomous transcript watchers (D3=B).** The shipped event contract covers session boundaries (start, compaction, heartbeat) but relies on the harness emitting a lifecycle event. A per-harness transcript watcher would drive ambient recall for harnesses that can't emit — but watchers are fragile and compaction is often invisible on disk. Add per harness that proves it can't emit a boundary event. Priority: P3.
|
||||
- [ ] **Materialized `thread_state` table.** `delta`'s thread-change arm derives open-thread deltas from facts/timeline `updated_at` scans. If a perf gate ever forces it, materialize a `thread_state` table instead of deriving. Not needed until the derive-path SLO is threatened. Priority: P3.
|
||||
- [ ] **Codex native boundary hooks.** Codex has no hooks upstream (`CODEX_HAS_HOOKS=false`), so its ambient path is pull-only (AGENTS.md gate tells it to call `context_pack`/`delta` at boundaries). When Codex ships a hook mechanism, register the boundary events the way the Claude Code lane does; the IPC `context_pack` kind + `--harness codex` attribution channel are already reserved for it. Priority: P3.
|
||||
## Brain-currency harness-e2e follow-ups (filed with the PR-A wave)
|
||||
|
||||
- [ ] **P1 — Extend engine-identity convergence to the other long-lived planes.**
|
||||
The autopilot daemon now detects a post-migration engine flip
|
||||
(`autopilotEngineIdentity` per-tick compare → clean exit for supervisor
|
||||
relaunch), but `gbrain serve` (MCP) and a standalone `gbrain jobs work`
|
||||
worker hold their engine handle indefinitely and keep writing into the
|
||||
abandoned source engine after a flip — the same silent-divergence class,
|
||||
still open on those planes (adversarial-review catch). Fix shape: the same
|
||||
boot-identity compare in their main loops.
|
||||
- [ ] **P2 — DB-visible pause for cross-host workers.** The pause marker now
|
||||
fences local job pickup (pre-claim check + post-claim release-back in
|
||||
`src/core/minions/worker.ts`), but the marker is a local file: a worker on
|
||||
ANOTHER host or container pointed at the same Postgres brain never sees it
|
||||
and keeps claiming jobs during a migration copy (its in-flight work IS
|
||||
visible to the drain via `minion_jobs`/lock rows; new claims are the gap).
|
||||
Fix shape: a row in a control table (or a pause flag in `gbrain_cycle_locks`)
|
||||
that the claim query itself honors — atomic with claiming, visible
|
||||
cluster-wide.
|
||||
- [ ] **P2 — Route file→symlink typechanges to delete.** `buildSyncManifest`
|
||||
maps git status `T` to modified, but import-file deliberately SKIPS symlinks
|
||||
(the exfil guard), so replacing an indexed file with a symlink leaves the
|
||||
old content indexed forever with no delete. Fix shape: when the post-change
|
||||
path is a symlink, emit a delete instead of a modify.
|
||||
- [ ] **P3 — Surface daemon-internal degradation in status.** A daemon stuck
|
||||
in the reconnect-retry loop (crash-classified errors) keeps heartbeating,
|
||||
so `--status` reads fresh while zero work happens. Fix shape: a breadcrumb
|
||||
file with consecutive-failure count that showStatus reads.
|
||||
|
||||
- [ ] **P3 — Extract a shared `seedBrain` test helper.** The keyless-PGLite +
|
||||
tmp-HOME + shimmed-PATH setup is duplicated between
|
||||
`test/autopilot-launchd-lifecycle.serial.test.ts` and
|
||||
`test/agent-scheduler-contract.serial.test.ts` (review-army maintainability
|
||||
finding). A third harness-e2e file (the PR-B tier) should force the
|
||||
extraction into `test/helpers/`; don't extract before then — two instances
|
||||
is a coincidence, three is a pattern.
|
||||
- [ ] **P3 — Name the quiesce protocol's magic numbers.** `migrate-engine.ts`
|
||||
and `autopilot.ts` share three constants by value, not by name: the 600s
|
||||
heartbeat-freshness window, the 35s default grace, and the daemon's paused
|
||||
fast-poll interval. Hoist into `src/core/autopilot-paths.ts` (the shared
|
||||
leaf) as named exports so the two planes can't drift.
|
||||
- [ ] **P3 — Migration manifest rows don't carry content_hash.** A resume
|
||||
trusts `(source_id, slug)` membership in `completed_slugs`; a page edited
|
||||
BETWEEN the failed run and the resume is skipped with its stale copy left on
|
||||
the target (review-army data-migration finding; pre-existing design, not a
|
||||
regression). Fix shape: stamp `content_hash` per completed entry and re-copy
|
||||
on mismatch during resume.
|
||||
|
||||
- [ ] **P2 — Keyless `gbrain dream` contract test.** The documented nightly cron
|
||||
(INSTALL_FOR_AGENTS.md Step 7) runs `gbrain dream` unconditionally, and the cycle's
|
||||
embed phase hits the same `EmbeddingDisabledError` class that broke the documented
|
||||
sync-and-embed chain on keyless brains (fixed in `runEmbed` for the `--stale`
|
||||
spelling; `test/agent-scheduler-contract.serial.test.ts` pins it). Nobody has verified that a
|
||||
full keyless dream exits 0 — if any phase surfaces the disabled-embeddings error as a
|
||||
phase failure, the documented nightly cron is broken identically for every
|
||||
`init --no-embedding` install. **Where to start:** `src/core/cycle.ts` embed phase +
|
||||
`src/commands/dream.ts` exit-code handling; test shape mirrors
|
||||
`test/agent-scheduler-contract.serial.test.ts` (keyless PGLite brain, real CLI spawn, exit-code
|
||||
assertion). Surfaced by the harness-e2e outside-voice review.
|
||||
|
||||
## BrainBench follow-ups (filed v0.44.0.0, Cathedral 2)
|
||||
|
||||
Deferred from the BrainBench wave (eng-reviewed; plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-bright-firefly.md`).
|
||||
|
||||
- [ ] **`--live` agent-in-the-loop know-to-ask.** Replay fixtures with a real model deciding whether to issue retrieval calls; grade the agent, not just the deterministic reflex. Pre-registered in `docs/eval/BRAINBENCH.md` (the v1 metric grades the injection decision, which IS the shipped mechanism). Needs: seeded N-repeat methodology for model stochasticity + budget rails. Priority: P2.
|
||||
- [ ] **Intrusion-budget gating calibration.** `avg_injected_tokens` is reported, non-gating (decision 18) — a wrong threshold is worse than none. After a few weeks of scoreboard data across PRs, pick calibrated per-seam thresholds and promote it to a gated metric. Priority: P2.
|
||||
- [ ] **Flip contract adapters to production — claude-code half now unblocked.** `adapters/claude-code.ts` exports the UserPromptSubmit hook wire types; the real hook (`gbrain hook user-prompt`, shipped with the bootstrap lane and extended with cross-turn dedupe + the channel feedback loop in the cathedral-3 convergence) swaps the in-process transport for an exec of the hook script and flips `seam: 'contract'` → `'production'` with continuous bench numbers. Note the production hook also exercises transcript-based dedupe, which the memoryless contract row deliberately doesn't. Same for codex fragments when that integration lands. Priority: P1 (the claude-code integration has landed; this is now standalone-actionable).
|
||||
- [ ] **Cathedral 1 conformance-kit fixture import.** The memory-verbs conformance scenarios convert to BrainBench fixtures via the published `evals/brainbench/schema/fixture.schema.json` once `garrytan/cathedral-1` merges ("conformance tests double as BrainBench seed fixtures", decision log 2026-06-12). Free corpus growth from already-reviewed scenarios. Blocked by: cathedral-1 on master. Priority: P2.
|
||||
- [ ] **Live-embeddings fidelity mode (`--embeddings`).** Hermetic CI grades the keyword/alias arms only (disclosed); an opt-in mode seeding real embeddings would grade write-back/continuity retrieval through the vector path. Same budget rails as `--llm`. Priority: P3.
|
||||
- [ ] **Community fixture intake + competitor adapters.** The TD1 remainder after the generated corpus absorbed in-PR growth: an `external-authors/`-style intake path for contributed fixtures (validator + privacy guard already gate them) and adapters for non-gbrain memory systems against the published schemas, enabling true head-to-head rows in the gbrain-evals scorecard. Priority: P3.
|
||||
- [ ] **JSON-Schema ↔ validator parity test.** `evals/brainbench/schema/*.schema.json` is the foreign-runner contract but only existence/parse is pinned — the schemas could drift from `fixtures.ts`'s validator silently. Needs a JSON-schema validator dep (ajv) or a hand-rolled subset checker; deferred to avoid a new dependency mid-wave. Priority: P3.
|
||||
- [ ] **`eval compare` markdown rendering for `mode: 'n/a'` rows.** EvalRunRecord v3 records brainbench under 'n/a'; the markdown renderer iterates SEARCH_MODES only, so those rows surface in `--json` output but not the mode table (documented in the v3 docstring). Add an un-grouped section to `renderMarkdown`. Priority: P3.
|
||||
- [ ] **Periodic re-baselining (the ratchet doesn't auto-tighten).** Improvements aren't banked into master's baseline until a PR updates it, so a regression back to a stale baseline level passes. Documented as an accepted residual in `docs/eval/BRAINBENCH.md`; the fix is an operator habit or a scheduled job that re-runs `--update-baseline` after metric-improving merges. Priority: P3.
|
||||
|
||||
- [ ] **Hermetic-ize the 7 env-sensitive LLM-availability tests.** `test/think-gateway-adapter.test.ts`, `test/conversation-parser/llm-base.test.ts`/`llm-fallback.test.ts`, `test/doctor-ze-checks.test.ts` assert behavior "when ANTHROPIC_API_KEY is unset" by reading the live process env — they fail on any dev shell that exports provider keys (verified failing on clean master in such a shell; green in keyless CI). Stub/save-restore the env per test so local runs match CI. Priority: P2.
|
||||
## #2416 follow-ups (query-steering wave)
|
||||
|
||||
- [ ] **P2 — MCP-envelope `hint` field for concept-shaped `search` calls.**
|
||||
**What:** surface the concept→query nudge to remote/MCP agent callers, not
|
||||
just the CLI. **Why:** MCP agents are the primary misrouting class the
|
||||
#2416 issue describes; the shipped CLI stderr nudge covers the caller class
|
||||
*least* at risk. **Context:** the `search` op returns a bare
|
||||
`SearchResult[]` (`src/core/operations.ts` — both return sites), so a hint
|
||||
needs an envelope change that ripples into `formatResult`, MCP
|
||||
serialization, and array-shape tests — deliberately kept out of the atomic
|
||||
#2416 commit. The pure classifier already exists
|
||||
(`looksConceptShaped`/`conceptNudge` in `src/core/search/query-intent.ts`);
|
||||
only the transport is missing. Consider a sibling metadata channel (like
|
||||
`_meta.metric_glossary`) rather than changing the array shape.
|
||||
**Depends on:** agreeing an envelope pattern that doesn't break existing
|
||||
MCP consumers.
|
||||
|
||||
## MEMORY_VERBS v1 follow-ups (filed v0.43.0.0 — Cathedral 1)
|
||||
|
||||
Deferred from the Cathedral 1 ship (CEO review, EXPANSION mode). Both are
|
||||
additive to the frozen v1 contract — neither breaks it. See plan + GSTACK
|
||||
REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-agile-iverson.md`
|
||||
and the scope record at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-06-12-memory-verbs-protocol.md`.
|
||||
|
||||
- [ ] **P3 — external-implementation certification PROGRAM.** The conformance
|
||||
TOOLING shipped (`gbrain protocol conformance --target <endpoint>`); the
|
||||
PROGRAM around it (badges, a registry of conformant implementations, listed
|
||||
third-party servers) waits for a second implementation to exist. **Why:** the
|
||||
protocol-not-product thesis only pays off once someone else implements
|
||||
MEMORY_VERBS; until then a certification program certifies an empty set.
|
||||
**Where:** new — would build on `src/commands/protocol.ts` conformance output.
|
||||
- [ ] **P3 — persistent open-threads model for the entity card.** v1 derives
|
||||
`entity.open_threads` from active commitment-kind facts + recent timeline
|
||||
entries (best-effort, possibly empty). A richer model (a real threads table:
|
||||
conversation id, opened/closed state, last activity) would make open-threads
|
||||
authoritative. **Why:** the card's open-threads field is the weakest signal
|
||||
in v1; a first-class threads store would make it load-bearing. **Where:**
|
||||
`src/core/verbs/entity-card.ts` open-threads assembly + a new schema table
|
||||
(additive — the card field already exists, so this is a quality upgrade, not
|
||||
a contract change).
|
||||
- [ ] **P2 — `recall` filter composition vs the spec (found by the v0.43.0.0
|
||||
cross-model doc review).** The handler dispatch is first-match
|
||||
(`supersessions` > `entity` > `session_id` > `since`), so `since` is
|
||||
silently ignored when `entity`/`session_id` is supplied, and `limit` has no
|
||||
server-side cap. Either compose the filters (additive — the spec's "filters
|
||||
the FACTS arm" wording already reads that way) or spell the precedence out
|
||||
in `docs/protocol/MEMORY_VERBS_v1.md`. **Where:** the `recall` handler in
|
||||
`src/core/operations.ts`.
|
||||
- [ ] **P3 — widen `synthesize`'s `unavailable` mapping.** Only the
|
||||
missing-key gateway warning maps to the `unavailable` error today; other
|
||||
no-usable-model failures can surface as `internal` (contract-legal but less
|
||||
actionable) or, worst case, a stubbed success. Audit the gateway failure
|
||||
modes and map every model-unusable path to `unavailable` with a fix
|
||||
suggestion. **Where:** the `synthesize` handler in `src/core/verbs.ts`.
|
||||
## Fix-wave 1 follow-ups (upgrade-wedge + trust-seam wave, 2026-08)
|
||||
|
||||
Deferred from the un-wedge-v121 hotfix wave (eng review + codex outside voice
|
||||
CLEARED; every item an explicit review decision). Waves 2–6 of the sequence are
|
||||
planned separately (provider-compat rescue is next; its original 2026-07-24
|
||||
DeepSeek-deprecation deadline has now PASSED — re-verify each cluster against
|
||||
master before starting, several fixes landed independently).
|
||||
|
||||
- [ ] **P2 — Shared strict `parseFlags` helper as the #2185 end-state (eng
|
||||
review 2B).** This wave ships the generated known-flags registry +
|
||||
pre-dispatch validator (parser and registry can drift only until the
|
||||
freshness guard fires). The structural end-state migrates commands onto one
|
||||
shared strict parser so parser == registry by construction; mechanical but
|
||||
touches 60+ command files — its own PR. Where: `src/commands/*.ts`,
|
||||
`src/cli.ts`, `scripts/generate-flag-registry.ts` (retires).
|
||||
- [ ] **P2 — `whoknows` CLI routing (surfaced by the #2035-class sweep).**
|
||||
`handleCliOnly`'s `whoknows` case (the dedicated CLI renderer with
|
||||
thin-client routing) is dead code — the command resolves via the
|
||||
`find_experts` op alias, and adding it to CLI_ONLY trips the alias-collision
|
||||
guard. Decide the intended surface alongside PR #2509 (whoknows --explain
|
||||
per-result factor breakdown) and delete whichever lane loses. Where:
|
||||
`src/cli.ts`, `src/commands/whoknows.ts`, PR #2509.
|
||||
- [ ] **P3 — #2544 second half: per-put_page `getAllSlugs` full scan.** The
|
||||
getChunks egress half shipped in this wave (explicit non-vector column
|
||||
list). The remaining Postgres-egress cost is put_page's per-call
|
||||
`getAllSlugs` table scan — needs a targeted existence probe or cached slug
|
||||
set. Where: `src/core/operations.ts` put_page path, both engines.
|
||||
- [ ] **P3 — #1558 admin-UI register form.** The `/admin/api/register-client`
|
||||
API now accepts `source` + `federatedRead` (this wave, PR #2016 absorbed);
|
||||
the admin SPA form fields + `/admin/api/sources` picker are the UI layer.
|
||||
Where: `src/commands/serve-http.ts` admin SPA blob.
|
||||
- [ ] **P3 — jsonb-integrity surfaces: batch + share (ship-review follow-up).**
|
||||
doctor's jsonbIntegrityCheck runs 2 queries per target (16 round-trips) and
|
||||
duplicates the TARGETS table with repair-jsonb (already drifted once on the
|
||||
jsonPayloadOnly predicate before being mirrored by hand). Batch the counts
|
||||
into one UNION ALL query and extract a shared targets constant
|
||||
(src/core/jsonb-integrity-targets.ts) consumed by both. Where:
|
||||
`src/commands/doctor.ts` jsonbIntegrityCheck, `src/commands/repair-jsonb.ts`.
|
||||
- [ ] **P3 — register-client HTTP-level e2e (ship-review follow-up).** The
|
||||
source/federatedRead lane is covered by unit normalizers + a structural
|
||||
route pin; a DATABASE_URL-gated serve-http e2e (register with bindings →
|
||||
assert stored client via /admin/api/agents; invalid source → 400
|
||||
invalid_source) closes the wire-level gap. Where:
|
||||
`test/e2e/serve-http-oauth.test.ts`.
|
||||
- [ ] **P3 — get_chunks `__all__` sentinel narrows to 'default' (red-team,
|
||||
Wave 3 territory).** `sourceScopeOpts` returns `{}` for a trusted local
|
||||
`--source __all__` caller (documented "spans the brain"), but both engines'
|
||||
getChunks map empty scope to the 'default' floor — the one read op where
|
||||
`{}` is reinterpreted. Fold into the Wave 3 source-federation cluster's
|
||||
`__all__` work (an explicit unscoped signal in the engine signature, or
|
||||
handler-side expansion for trusted callers). Where: `src/core/operations.ts`
|
||||
get_chunks, both engines' getChunks.
|
||||
- [ ] **P3 — #2536 wedged-migration diagnostics.** The v121 wedge aborted
|
||||
initSchema BEFORE runMigrations, so the wedged-migration diagnostics row was
|
||||
never written — operators got a bare SQL error with no remediation hint.
|
||||
Write the diagnostics row (or a stderr remediation block) from the blob-replay
|
||||
catch path too. Where: `src/core/migrate.ts`, `src/commands/apply-migrations.ts`.
|
||||
## WAL-repair wave follow-ups (#223/#1670/#2575)
|
||||
|
||||
- [ ] **P2 — gate auto-repair on an unclean-shutdown marker (adversarial F7).** The classifier
|
||||
deliberately over-matches (`RuntimeError`/`unreachable` → `wasm-abort`). If an unclean
|
||||
shutdown leaves a REPLAYABLE WAL tail (normal crash recovery would restore those committed
|
||||
txns) and the reopen then fails on a transient WASM error (OOM), auto-repair fires, layout
|
||||
validation can't tell torn from replayable, and resetWal discards the tail while the notice
|
||||
says "data preserved." Bounded today (backup always taken + restore + honest failure + repeated
|
||||
attempts capped), but a false-positive-with-successful-retry silently drops committed data.
|
||||
Fix direction (probe-verified): PGLite removes `postmaster.pid` on clean close, so gate AUTO
|
||||
repair (not the manual command) on `postmaster.pid` presence — a clean dir that aborts is not
|
||||
torn-WAL. Requires making the serial regression test stamp a `postmaster.pid` before corrupting
|
||||
(it currently clean-disconnects then corrupts, which the red-team flagged as unfaithful anyway).
|
||||
Needs a recall/precision call before landing.
|
||||
- [ ] **P3 — live non-gbrain PGLite consumer not caught by the postmaster.pid liveness guard
|
||||
(adversarial F8).** PGLite writes a sentinel `postmaster.pid` of `-42`; the liveness refusal in
|
||||
`validateWalRepairTarget` requires `pid > 0`, so it protects native Postgres dirs but not a
|
||||
non-gbrain pglite app that has the dir open (such an app writes no `.gbrain-lock`). Deliberate
|
||||
misuse of `pglite-repair --path <foreign pglite dir>` required. Option: refuse when
|
||||
postmaster.pid holds pid ≤ 0 with a very recent mtime, or document the boundary.
|
||||
- [ ] **P3 — mixed-version torn-lock read (adversarial F10 residual).** The heartbeat + initial
|
||||
lock writes are atomic (tmp+rename) now, but an OLD gbrain binary writing heartbeats IN PLACE
|
||||
while a NEW binary poll-reads can still catch a torn read → corrupt-lock verdict → a live
|
||||
holder's lock reaped → two writers (the #2348 class, version-skew-triggered). The reap marker
|
||||
quarantines repair, not the concurrent open. Cheap hardening: double-read the lock file (~50ms
|
||||
apart) before declaring it corrupt.
|
||||
|
||||
|
||||
- [ ] **P2 — graceful PGLite close on SIGTERM for the remaining long-running paths.**
|
||||
The torn-WAL genesis this wave repairs is an unclean shutdown: `src/core/process-cleanup.ts`
|
||||
releases locks on SIGTERM but never closes the PGlite handle, so `serve` / `jobs work` /
|
||||
`sync` killed mid-write (macOS-upgrade reboot, `systemctl stop`) leave the WAL torn.
|
||||
Autopilot already ships the pattern (d2fd1f29, #3178/#1872: `registerCleanup('autopilot-engine-close', ...)`
|
||||
— abort in-flight work → ≤2s bounded wait inside the 3s cleanup deadline →
|
||||
`engine.disconnect()`, double-call safe; rationale comment at autopilot.ts:438-452).
|
||||
Extend that exact pattern to the remaining long-running PGLite paths (register in
|
||||
connect()/command scope; dedupe so autopilot doesn't double-close), pinned by a serial
|
||||
lifecycle test. Interacts with #2084 exitCode containment + #1337 close ordering — read
|
||||
those comments in pglite-engine.ts first. Auto-repair makes recurrence self-healing
|
||||
meanwhile, so this is prevention, not recovery.
|
||||
- [ ] **P3 — pglite upgrade blocker tracker.** Two couplings make a "routine" pglite bump a
|
||||
breaking change: (a) pglite ≥0.5 removes the `@electric-sql/pglite/vector` export that
|
||||
`pglite-engine.ts` imports (verified against npm); (b) the pg_resetwal port
|
||||
(`src/core/pglite-resetwal.ts`) is coupled to the PG17 pg_control layout
|
||||
(`PG_CONTROL_VERSION` 1700 — guarded at runtime by `WalResetUnsupportedError`, so a
|
||||
mismatched bump makes the repair tool refuse every dir rather than corrupt, but it still
|
||||
means the repair feature silently dies). Any future pglite upgrade wave must revisit BOTH
|
||||
together and re-derive the ControlFileData offset table for the new PG major.
|
||||
|
||||
## serve --http takes-holders + agent-voice hardening follow-ups (filed v0.42.74.0)
|
||||
|
||||
Deferred from the #2529/#2477 security-fix wave (plan-eng-review + codex outside
|
||||
voice CLEARED). None block the wave.
|
||||
|
||||
- [ ] **P2 — Per-OAuth-client `takes_holders` storage (#2529 follow-up).** Legacy
|
||||
bearer tokens honor `access_tokens.permissions.takes_holders` through
|
||||
`verifyAccessToken`; OAuth clients have no equivalent column on `oauth_clients`,
|
||||
so OAuth-minted tokens fail closed to `['world']`. Needs a schema migration
|
||||
(`oauth_clients.takes_holders` JSONB or TEXT[]) + a `register-client` flag +
|
||||
the `verifyAccessToken` JOIN projection. Include surfacing the EFFECTIVE
|
||||
takes-holder scope in `whoami` output as part of this follow-up, so operators
|
||||
can self-diagnose the legacy-vs-OAuth semantic split instead of reading docs.
|
||||
Where: `src/schema.sql`, `src/core/migrate.ts`, `src/core/oauth-provider.ts`,
|
||||
`src/commands/auth.ts`, `src/core/operations.ts` (whoami).
|
||||
- [ ] **P3 — agent-voice Host-header allowlist (DNS-rebinding hardening).** The
|
||||
#2477 fix ships default-deny CORS + an Origin gate on `/session`/`/tool`, but
|
||||
the gate derives self-origin from the `Host` header, so a DNS-rebound page
|
||||
(attacker origin whose host resolves to the operator's loopback) still passes.
|
||||
Validate `Host` against `localhost`/`127.0.0.1`/operator-configured hosts and
|
||||
403 otherwise; slots beside `originAllowed()` in the router. Issue #2477
|
||||
explicitly deferred this. Where: `recipes/agent-voice/code/server.mjs`.
|
||||
- [ ] **P3 — Debounce `last_used_at` in the oauth-provider legacy path.** The
|
||||
legacy branch of `verifyAccessToken` fires an unconditional
|
||||
`UPDATE access_tokens SET last_used_at = now()` on EVERY request, while the
|
||||
legacy HTTP transport debounces the same write to once per 60s via a WHERE
|
||||
clause (`src/mcp/http-transport.ts` validateToken). Apply the same pattern —
|
||||
one fewer write per request on the `serve --http` hot path.
|
||||
Where: `src/core/oauth-provider.ts`.
|
||||
|
||||
## v0.42.67.0 follow-ups (Windows build tooling)
|
||||
|
||||
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
|
||||
@@ -285,6 +572,42 @@ Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at
|
||||
filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates
|
||||
through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan.
|
||||
|
||||
## Harness hook lane follow-ups (filed from the cathedral-3 convergence)
|
||||
|
||||
Filed when the cathedral-3 branch converged its push-adapter work into the
|
||||
#3975 hook lane (feedback loop + cross-turn dedupe for `gbrain hook
|
||||
user-prompt`). Context: the hook lane now logs channel-attributed volunteer
|
||||
events at the IPC delivery point and dedupes via the transcript's
|
||||
`hook_additional_context` attachments.
|
||||
|
||||
- [ ] **P3 — PostToolUse / mid-turn push adapter, evaluated against per-channel stats.**
|
||||
The user-prompt hook fires at prompt time only; entities that first appear mid-turn
|
||||
in tool output (a file opened, a person named in a search result) get no pointer
|
||||
until the NEXT prompt. Harnesses expose a PostToolUse hook, but it fires dozens of
|
||||
times per turn (one `gbrain hook` process spawn each). Now that the feedback loop
|
||||
exists, the per-channel `--stats` precision + volume data is exactly the evidence
|
||||
needed to decide. **Trigger:** claude-code channel stats showing healthy precision
|
||||
plus user reports of "it only noticed on my next message". **Start:**
|
||||
`src/commands/hook.ts` (the event already has a dispatch slot pattern),
|
||||
`src/core/bootstrap/hooks.ts` registration writers.
|
||||
- [ ] **P3 — engine-uniform IPC listener (Postgres serves).** serve's resolve/turn_context
|
||||
socket is PGLite-gated (`src/mcp/server.ts`: `cfg?.engine === 'pglite'`), so on a
|
||||
Postgres brain `gbrain hook user-prompt` short-circuits (`no_pglite_path`) and the
|
||||
hook lane is PGLite-only. Extending the listener needs (a) a canonical per-connection
|
||||
socket path for brains with no data dir (e.g. `~/.gbrain/run/resolve-<hash12(database_url)>.sock`,
|
||||
0700 dir) and (b) a secret-file home for `turn_context` auth (same hash-keyed run dir).
|
||||
The cathedral-3 branch prototyped (a) as `resolveSocketPathForConfig` (see branch
|
||||
history at commit 2350294c) before the convergence dropped it pending the secret
|
||||
design. **Trigger:** a Postgres-brain user asking why hooks stay silent. **Start:**
|
||||
`src/core/context/resolve-ipc.ts` socket-path helpers + `src/mcp/server.ts` listener gate
|
||||
+ `src/commands/hook.ts:no_pglite_path` branch.
|
||||
- [ ] **P3 — thin-client remote push route.** Thin-client installs (remote_mcp) have no
|
||||
local engine and no serve socket — every push channel is dead there and only the
|
||||
hook's typed heartbeat reason says why. The natural route is `volunteer_context`
|
||||
over the remote MCP transport (`callRemoteTool`), rate-limited per prompt.
|
||||
**Trigger:** thin-client adoption of bootstrap. **Start:** `src/commands/hook.ts`
|
||||
user-prompt branch + `src/cli.ts` remote-tool plumbing.
|
||||
|
||||
## gbrain#2095 push-based context follow-ups (v0.43+)
|
||||
|
||||
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
|
||||
@@ -298,20 +621,22 @@ are the bar). Plan + GSTACK REVIEW REPORT at
|
||||
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
|
||||
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
|
||||
**Blocked by:** a real consumer (revisit when one exists).
|
||||
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
|
||||
needed doctor visibility because silent failure was invisible; volunteer is
|
||||
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
|
||||
agents not discovering the surface, ship a `push-context` recipe (mirror
|
||||
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
|
||||
**Where:** `recipes/`, `src/commands/doctor.ts`.
|
||||
- [ ] **P3 — policy skill (recipe) for push-context.** The doctor-check half of
|
||||
this item shipped with the harness hook lane: `volunteer_channels`
|
||||
(`src/commands/doctor.ts:checkVolunteerChannels`) reads the events table
|
||||
per-channel on both the local and remote doctor. Remaining scope: if
|
||||
`volunteer-context --stats` adoption shows agents not discovering the
|
||||
surface, ship a `push-context` recipe (mirror `recipes/retrieval-reflex/`).
|
||||
**Where:** `recipes/`.
|
||||
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
|
||||
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
|
||||
If MCP callers accumulate parsing bugs, add a structured array param beside it.
|
||||
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
|
||||
- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver
|
||||
(`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE
|
||||
ANY('%/...')`) predates #2095 but now runs per turn on three channels
|
||||
(reflex window, volunteer_context, watch) federated across sources. Neither
|
||||
ANY('%/...')`) predates #2095 but now runs per turn on four channel surfaces
|
||||
(reflex window, volunteer_context, watch, and the harness-hook `turn_context`
|
||||
lane) federated across sources. Neither
|
||||
the leading-wildcard suffix arm nor `lower(title)` is index-served. If
|
||||
per-turn latency telemetry on large brains comes back hot: add
|
||||
`(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or
|
||||
@@ -2383,8 +2708,9 @@ contributor traps.
|
||||
- [ ] **P2: Document `FREE_LOCAL_RERANK_PROVIDERS` invariant.** `src/core/budget/budget-tracker.ts:lookupPricing`
|
||||
returns `{input:0, output:0}` for any model id under the `llama-server-reranker:`
|
||||
provider on the rerank kind. The contract relies on all callers going through
|
||||
`gateway.rerank()`'s `assertTouchpoint`-with-extended-models check (which validates
|
||||
the model exists before pricing fires). Theoretical bypass: a future caller that
|
||||
`gateway.rerank()`'s own model-list check (rerank-specific; it validates the
|
||||
model exists before pricing fires — note this was never `assertTouchpoint`,
|
||||
which checks provider touchpoints only). Theoretical bypass: a future caller that
|
||||
reserves directly against BudgetTracker with `kind: 'rerank'` and an arbitrary
|
||||
`llama-server-reranker:<anything>` model id gets free pricing. Fix: code comment
|
||||
documenting the invariant, OR move the freeness check to gateway.rerank() where
|
||||
@@ -3514,28 +3840,99 @@ After the sweep, both should be fixable and renameable back to plain `*.test.ts`
|
||||
|
||||
## claw-test E2E (v0.22.16 follow-ups)
|
||||
|
||||
### Hermes runner — `src/core/claw-test/runners/hermes.ts`
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Add a Hermes implementation of the `AgentRunner` interface. v1 ships only OpenClaw; v1.1 lands hermes once we have real friction reports from openclaw to validate the contract against.
|
||||
|
||||
**Why:** Cross-agent diff (`gbrain friction diff --base openclaw --compare hermes`) is the highest-leverage next signal. Friction unique to one agent vs common-to-both separates "agent contract bug" from "gbrain bug" automatically.
|
||||
|
||||
**Effort:** S (CC ~30m). Depends on: v1 openclaw runner producing real friction reports first.
|
||||
### ~~Hermes runner — `src/core/claw-test/runners/hermes.ts`~~ DONE (hermes-harness wave)
|
||||
Shipped: `HermesRunner` (`hermes -z <brief>`, `$HERMES_BIN` > `which hermes`,
|
||||
`HERMES_HOME` env-allowlist delta) + the full hermes install door
|
||||
(`test/e2e/install-real-hermes.serial.test.ts`, opt-in-gated) + the label-gated
|
||||
`hermes-door` CI job in heavy-tests.yml. The cross-agent
|
||||
`gbrain friction diff --base openclaw --compare hermes` payoff shipped in the
|
||||
same wave (below). Observed-CLI pins live in `docs/mcp/HERMES-CLI-PIN.md` and
|
||||
`docs/mcp/HERMES.md`.
|
||||
|
||||
---
|
||||
|
||||
### Friction analytics suite — `diff` / `trend` / `migration-stub`
|
||||
### Friction analytics suite — `trend` / `migration-stub` (diff SHIPPED)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Three new `gbrain friction` subcommands deferred from v1:
|
||||
- `gbrain friction diff --base <run-or-agent> --compare <run-or-agent>` (cross-agent comparison; ~80 LOC)
|
||||
**What:** Two remaining `gbrain friction` subcommands deferred from v1
|
||||
(`diff` shipped in the hermes-harness wave — see `src/commands/friction.ts`):
|
||||
- `gbrain friction trend [--since <version-or-date>] [--phase <name>]` (time-series across runs; ~60 LOC)
|
||||
- `gbrain friction migration-stub [--threshold N]` (clusters friction by phase + tokens, emits `skills/migrations/v[N+1].md` stub; ~150 LOC)
|
||||
|
||||
**Why:** Turns point-in-time reports into a slope. Pairs with the v1.1 public scoreboard.
|
||||
|
||||
**Effort:** M (CC ~2h total).
|
||||
**Effort:** M (CC ~1.5h total).
|
||||
|
||||
---
|
||||
|
||||
### Promote hermes-door soft probes to hard assertions + build the REAL cron test
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Two follow-ups now that the hermes CLI surface is pinned (v0.20.0,
|
||||
`docs/mcp/HERMES-CLI-PIN.md`): (1) promote the door's logged-evidence probes
|
||||
(`hermes mcp list` output shape; session-artifact tool-call traces under
|
||||
`<home>/.hermes/`) to hard assertions once a couple of CI runs confirm their
|
||||
stability across hermes releases; (2) build the real cron pairing test — the
|
||||
surface is fully non-interactive (`hermes cron create [--name N] [--no-agent]
|
||||
[--script PATH] <schedule> [prompt]` + `hermes cron tick` runs due jobs once
|
||||
and exits) — create a job that runs `gbrain sync --json`, tick, and assert the
|
||||
sync actually executed against the run's brain. (A self-skipping probe was
|
||||
deliberately CUT in review: a test that cannot fail is not coverage.)
|
||||
|
||||
**Why:** INSTALL_FOR_AGENTS.md's recurring-jobs step has zero coverage; the
|
||||
evidence sweep is the promotion signal the door already logs.
|
||||
|
||||
**Effort:** S-M (CC ~45m). Depends on: first labeled hermes-door CI runs.
|
||||
|
||||
---
|
||||
|
||||
### Wire the orphaned `voice-agent-install` ScenarioKind
|
||||
**Priority:** P2
|
||||
|
||||
**What:** `test/fixtures/claw-test-scenarios/voice-agent-install/` carries the
|
||||
richest install-assertion template in the repo (60-line expected.json:
|
||||
filesystem manifest, `.gbrain-source.json` sha256s, resolver rows, PII
|
||||
blocklist, health probe, tiered soft-fail) but `scenario.json` declares
|
||||
`kind: "voice-agent-install"`, which `ScenarioKind` rejects — the fixture
|
||||
cannot load. Extend `ScenarioKind` + `loadScenario` + a `postInstallHook`
|
||||
implementation so the scenario runs.
|
||||
|
||||
**Why:** Integrations-recipe install coverage (the `gbrain integrations
|
||||
install` path) has a fully-designed scenario sitting dead.
|
||||
|
||||
**Effort:** M (CC ~1h). Integrations-lane work, deliberately kept out of the
|
||||
hermes-harness wave.
|
||||
|
||||
---
|
||||
|
||||
### Cold-install container test — fill the `tests/docker/bootstrap-e2e.sh` placeholder
|
||||
**Priority:** P3
|
||||
|
||||
**What:** heavy-tests.yml carries a gated no-op step for
|
||||
`tests/docker/bootstrap-e2e.sh` (networkless cold-machine container install of
|
||||
gbrain itself: global install, PATH discovery, migrations). The file doesn't
|
||||
exist. Write it.
|
||||
|
||||
**Why:** The agent-platform door tests (claude/codex/hermes) all deliberately
|
||||
run gbrain from the dev tree / compiled binary — none of them proves gbrain's
|
||||
own cold install. That gap was re-flagged in the hermes-harness wave's outside
|
||||
review and scoped OUT of that wave on purpose.
|
||||
|
||||
**Effort:** M (CC ~1-2h, docker).
|
||||
|
||||
---
|
||||
|
||||
### BrainBench hermes adapter
|
||||
**Priority:** P3
|
||||
|
||||
**What:** ~50-100 lines in `src/eval/brainbench/adapters/hermes.ts` + an
|
||||
`ALL_HARNESSES` entry + baseline cells in `evals/brainbench/baselines/main.json`.
|
||||
|
||||
**Why:** Cross-harness memory-conformance coverage for the third platform.
|
||||
Eval seam (memory conformance), NOT install — kept out of the install wave on
|
||||
purpose; needs baseline-governance care per the BrainBench gate rules.
|
||||
|
||||
**Effort:** S-M (CC ~1h + baseline runs).
|
||||
|
||||
---
|
||||
|
||||
@@ -3555,7 +3952,7 @@ After the sweep, both should be fixable and renameable back to plain `*.test.ts`
|
||||
### Real v0.18 SQL dump for upgrade scenario
|
||||
**Priority:** P2
|
||||
|
||||
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. The harness gracefully no-ops the seed phase when absent, so the scenario currently behaves like fresh-install. v1.1: generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
|
||||
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. Both scripted and live runs now FAIL LOUDLY on the missing dump (a silent skip used to init a current database and false-green the "upgrade"), so the shipped scenario is unrunnable until the dump lands. Generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
|
||||
|
||||
**Why:** Without a real seed, the scenario doesn't actually exercise the migration chain forward-walk. That's the whole point of the upgrade scenario — proves issue #239/#243/#266/#357 class regressions stay fixed.
|
||||
|
||||
@@ -4808,3 +5205,312 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
+ the models-doctor tests.
|
||||
|
||||
**Depends on:** nothing.
|
||||
|
||||
## Agent-bootstrap wave follow-ups (filed at build time)
|
||||
|
||||
- [ ] **P2 — repoPhaseComplete is single-workspace (one global receipt).** The
|
||||
no-daemon push gate binds to the one `receipt.repo_url`, so with two bootstrap
|
||||
workspaces sharing a gbrain home, workspace B's `bootstrap repo` overwrites the
|
||||
receipt and permanently leaves A's per-turn/session-end pushes at
|
||||
`push_deferred_repo_pending`. Fails CLOSED (defers, never mis-pushes) and
|
||||
matches the v1 single-workspace contract, but the per-turn push made it more
|
||||
visible. Fix = per-root repo binding (a receipt map or a per-root marker).
|
||||
Surfaced by both v0.45.9.0 adversarial reviewers.
|
||||
- [ ] **P2 — visibility ladder subprocess/body bounds.** `runWithTimeout`
|
||||
(`src/core/repo-visibility.ts`) races the `gh`/`git` probe against a timer but
|
||||
doesn't kill the raced child, and the anon-probe `res.text()` buffers the whole
|
||||
(operator-configured-origin) body before slicing. Bounded in practice by the
|
||||
detached push child's lifetime, but a proper fix kills the raced process and
|
||||
caps the body read. Filed from the v0.45.9.0 Codex adversarial pass.
|
||||
- [ ] **P3 — `config set` for the file-plane hook-lane keys is engine-bound.**
|
||||
`runConfig` dispatches through the engine path, so `gbrain config set
|
||||
push.allow_unverified_remote true` can fail while a live PGLite serve holds the
|
||||
writer lock — the documented recovery command, unavailable exactly when needed.
|
||||
The env-var form (`GBRAIN_ALLOW_UNVERIFIED_REMOTE=1`) is the cloud path and needs
|
||||
no engine, so this is convenience-only; fix = route these two keys through the
|
||||
no-engine CLI dispatch. Filed from the v0.45.9.0 Codex adversarial pass.
|
||||
|
||||
|
||||
- [ ] **P3 — plugin-based hook distribution for Claude Code.** Ship gbrain's
|
||||
hooks as a Claude Code plugin (`hooks/hooks.json` + `.claude-plugin/plugin.json`
|
||||
manifest, installed via the plugin marketplace flow) instead of two settings
|
||||
files. Plugins merge hooks first-class across scopes and update centrally —
|
||||
it would REPLACE both current carriers (repo-committed `.claude/settings.json`
|
||||
for cloud installs + gitignored `settings.local.json` for local), so it must
|
||||
migrate, not join; a third simultaneous carrier would double-fire events.
|
||||
Cons: needs marketplace repo hosting; enterprise `allowManagedHooksOnly`
|
||||
policies can block plugin hooks entirely. Start at
|
||||
`src/core/bootstrap/hooks.ts` (both writers + the dedupe rule live there).
|
||||
Filed from the cloud-DX eng review (v0.46.x wave).
|
||||
- [ ] **P3 — watch Claude Code Channels as the push path for
|
||||
volunteer_context/signals.** Channels (research preview) push external events
|
||||
into a LIVE session — the native version of gbrain's push-context lane
|
||||
(`docs/guides/push-context.md`). Not actionable today: delivery requires an
|
||||
always-on session plus an Anthropic-allowlisted channel plugin. Revisit when
|
||||
channel-plugin distribution opens; the win is replacing per-turn pull with
|
||||
event push for signals/reflex windows. Filed from the cloud-DX eng review.
|
||||
|
||||
- [ ] **P1 — enforce op scope/localOnly on the stdio MCP dispatch when no auth
|
||||
context is present, and consider a narrower default surface for pull-mode
|
||||
harness registrations.** HTTP dispatch enforces `scope`/`localOnly` before
|
||||
handlers run; the stdio surface should reach parity so a registration that is
|
||||
user-global by host design (no per-project scoping available) does not expose
|
||||
more authority than the session needs. Surfaced by the v0.45.x ship
|
||||
adversarial pass (cross-model); pre-existing behavior, not introduced by the
|
||||
Codex scope-consent fix — that fix's prose now states the read+write reality
|
||||
honestly. Needs its own design pass (interaction with `--surface` pinning,
|
||||
MEMORY_VERBS, and the trust-boundary invariant in CLAUDE.md).
|
||||
- [ ] **P2 — consent-key answers vs the A8 confirm gate.** Decide whether
|
||||
`consent: true` bank keys should be exempt from `setAnswer`'s confirmation
|
||||
invalidation (`src/core/bootstrap/interview.ts:308-309` `[A8]` deletes
|
||||
`state.confirmed` on ANY set) so operational consents can be recorded at their
|
||||
designed phase-contextual moment post-confirm without regressing
|
||||
`bootstrap status` to "answers complete but not confirmed" (status.ts
|
||||
interview detector). Deferred from the Codex MCP-scope fix (eng review option
|
||||
3B chose prose realignment instead: the runbook now records `MCP_SCOPE` in
|
||||
phase 3, pre-confirm, so the confirm hash covers it). An exemption touches a
|
||||
tamper-tripwire — a post-confirm flip of `PERSIST_CRON` (background-push
|
||||
consent) would no longer invalidate anything — so it needs its own
|
||||
adversarial review before landing. Also cover the healing half: pre-fix
|
||||
installs that recorded `MCP_SCOPE` at the old wire-phase moment have a
|
||||
permanently-invalidated confirm, and `bootstrap status` can't distinguish a
|
||||
consent-key invalidation from a tampered answer set — a status detail for
|
||||
that case would stop resumed installs being steered into a redundant
|
||||
re-confirm loop (ship-review data-migration finding). Context: eng review +
|
||||
codex consult of the Codex scope fix, 2026-08-11.
|
||||
- [ ] **P2 — bootstrap first-push secret scan reads the working tree, not the
|
||||
index blobs; fail-open on binary/large files.** `secretScanOrThrow` /
|
||||
`scanFiles` (src/core/bootstrap/repo.ts + src/core/secret-scan.ts) read
|
||||
working-tree bytes and silently skip unreadable, binary, and >25 MiB files, so
|
||||
a git clean filter could commit a secret whose working-tree copy scans clean,
|
||||
and a secret in a binary/large file is never seen. Pre-existing across ALL
|
||||
bootstrap pushes (create + adopt), not specific to create-repo-first. Fix:
|
||||
scan the staged index blobs (`git show :file` / `git cat-file`) fail-closed,
|
||||
or reuse the hardened scanner path from `workspacePush`. Filed from the
|
||||
v0.45.2.0 /ship Codex adversarial pass (P0 there; scoped to P2 here as a
|
||||
shared-scanner hardening that needs its own tests, deliberately out of the
|
||||
create-repo-first change).
|
||||
- [x] **P2 — compiled `gbrain` binary can now `serve` a PGLite brain.** FIXED:
|
||||
`src/core/pglite-embedded-assets.ts` embeds PGLite's runtime payload
|
||||
(`pglite.wasm`, `initdb.wasm`, `pglite.data`, `vector.tar.gz`,
|
||||
`pg_trgm.tar.gz`) with Bun's `import ... with { type: 'file' }` and hands them
|
||||
to PGLite via `PGliteOptions` (`pgliteWasmModule` / `initdbWasmModule` /
|
||||
`fsBundle` + custom `vector`/`pg_trgm` extensions whose `setup()` returns a
|
||||
`bundlePath` pointing at the embedded tarball, materialized to a temp file
|
||||
because `createReadStream` — unlike `readFileSync` — can't read `/$bunfs`
|
||||
paths). `src/core/pglite-engine.ts` spreads `getEmbeddedPgliteOptions()` at
|
||||
both `PGlite.create()` sites, so the Bun-vfs #1340 ENOENT no longer fires for
|
||||
a correctly-built binary. Guarded by `scripts/check-pglite-embedded.sh`
|
||||
(compiles a smoketest and asserts a real PGLite query round-trips), wired into
|
||||
`bun run verify` + `check:all` + `check:pglite-embedded`. The real-agent e2e
|
||||
harness (`test/helpers/agent-harness.ts`) now resolves to the fast compiled
|
||||
MCP server; the `bun run` fallback stays as a safety net. Upstream Bun issue
|
||||
(still open since Nov 2024) is now moot for gbrain.
|
||||
- [ ] **P1 — `--background --follow` spawns a nonexistent subcommand.**
|
||||
`src/core/cli-options.ts` (~:391) spawns `gbrain jobs follow <id>` after a
|
||||
background submit, but jobs.ts has no `follow` subcommand (`jobs watch
|
||||
--follow` exists and takes no id; `jobs submit --follow` is inline-only).
|
||||
The spawned child hits the unknown-subcommand path, so `--background
|
||||
--follow` submits fine but the live stream never attaches. Fix: implement
|
||||
`jobs follow <id>` (poll get_job + stream progress) or retarget the spawn.
|
||||
Found during the v0.45.0.0 doc audit; KEY_FILES documents current behavior.
|
||||
- [ ] **P1 — bootstrap status is not workspace-scoped.** status.ts reads the
|
||||
newest GLOBAL verify snapshot + global push-status, so a green verify in
|
||||
workspace A can report workspace B as verified/healthy. Key the verify
|
||||
snapshots + push-status by workspace path (or store them under the
|
||||
workspace's own state dir) so status reflects the workspace it runs in.
|
||||
Found by the v0.45.0.0 adversarial pass (Codex P2, raised to P1 — multi-
|
||||
workspace is the graduation/multi-device story).
|
||||
- [ ] **P2 — mkdir-lock ABA steal race (folds into the shared-primitive TODO).**
|
||||
Both acquirePushLock (workspace-push.ts) and acquireBootstrapLock (lock.ts)
|
||||
steal a dead+stale lock as rmSync-then-mkdir with no re-verify between, so
|
||||
two simultaneous stealers can both believe they hold it (same known class as
|
||||
pglite-lock; git index.lock bounds the push damage). When extracting the
|
||||
shared mkdir-lock primitive, harden the steal: re-read + token-compare the
|
||||
stale owner immediately before rmSync, or use an atomic rename-based steal.
|
||||
Cross-model finding (Claude + Codex) v0.45.0.0 adversarial.
|
||||
- [ ] **P2 — uninstall TOCTOU on a live PGLite DB.** uninstall probes the serve
|
||||
lock then recursively deletes later; a serve starting in between wins the
|
||||
race. Gated today by the lock probe + refuse-on-live-serve, so this is the
|
||||
residual known lock-class window — close it when the shared lock primitive
|
||||
lands (hold the lock across the probe→delete span). Codex P1 v0.45.0.0.
|
||||
- [ ] **P2 — sweep fairness: recency-only selection starves old pages.**
|
||||
sweep.ts pass 2 repeatedly takes the newest batchLimit pages with no
|
||||
extraction-watermark filter, so frequently-updated pages monopolize every
|
||||
sweep and older pages never get link/timeline extraction. Add an
|
||||
extracted-watermark (or round-robin) so the backlog drains. Codex P2.
|
||||
- [ ] **P2 — untracked deny-glob exclusion is invisible under detached push.**
|
||||
A first-time deny-glob file (e.g. a fresh .env) is dropped from staging and
|
||||
reported only via the logger callback + excludedUntracked — but session-end/
|
||||
session-start auto-push runs via spawnDetachedPush with stdio:'ignore', so
|
||||
the warning is discarded. Fail-safe (never leaves the machine) but the user
|
||||
gets a false "synced" impression. Surface excluded deny-matches via the
|
||||
heartbeat/push-status file so doctor + status can report them. Claude
|
||||
adversarial (minor) v0.45.0.0.
|
||||
|
||||
- [ ] **P2 — shared receipt upsert helper.** InstallReceipt construct/merge
|
||||
logic is quadruplicated (bootstrap.ts writeRenderReceipt +
|
||||
appendReceiptRegistration, attach.ts, repo.ts recordRepoInReceipt) with
|
||||
slightly different defaults; guardReceiptOverwrite is wired at each site
|
||||
individually. One `upsertReceipt(home, ws, patch)` in format.ts.
|
||||
- [ ] **P2 — promote the atomic tmp+rename write idiom to one core helper.**
|
||||
~7 hand-rolled copies across format.ts/interview.ts/lock.ts/render.ts/
|
||||
hook.ts/workspace-push.ts/verify.ts; hooks.ts already has a private
|
||||
atomicWriteJson to promote (plus a text variant).
|
||||
- [ ] **P2 — extract a shared mkdir-lock primitive.** bootstrap/lock.ts and
|
||||
workspace-push.ts both implement atomic-mkdir + PID/age/token steal rules
|
||||
citing the same pglite-lock learnings (#2058/#2348); three parallel
|
||||
implementations counting pglite-lock. One primitive with injectable stale
|
||||
window + throw-vs-result adapters.
|
||||
- [ ] **P2 — move the hook heartbeat read surface into core.** core/bootstrap/
|
||||
status.ts and core/bootstrap/verify.ts dynamic-import readHeartbeatTail
|
||||
from commands/hook.ts (core→commands inversion). A core/hooks-telemetry.ts
|
||||
owning the read/write surface removes the reach-ins.
|
||||
- [ ] **P2 — consolidate the two GitHub remote parsers + privacy probes.**
|
||||
workspace-push.ts parseGithubOwnerRepo (accepts ssh:// + trailing /) vs
|
||||
repo.ts parseGithubRemote (scp-form only), and verifyRemotePrivacy (gh repo
|
||||
view) vs verifyRepoPrivate (gh api): grammars/failure classes can drift.
|
||||
One exported parser + one privacy-probe core returning the verdict union.
|
||||
- [ ] **P2 — connector-ingest capability probe (with v1.1 connector ingest).**
|
||||
The plan's build-order-2 probe (email/calendar connector availability wired
|
||||
into install output) deferred with the feature it gates; the capability
|
||||
report today covers embeddings + extraction only.
|
||||
- [ ] **P2 — G15 retention warnings in doctor.** MEMORY.md size cap + corpus
|
||||
retention + orphaned stop-hook buffers are enforced/pruned but doctor never
|
||||
warns when they accumulate; add the three checks to the bootstrap group.
|
||||
- [ ] **P2 — NER-based graph floor in verify.** The graph-floor check asserts
|
||||
wikilink-edge/backlink link-table rows; a true entity-extraction floor
|
||||
(≥1 NER entity + one edge-only query op) needs the extraction path or a
|
||||
keyed provider, so it rides with a keyed-mode verify extension.
|
||||
- [ ] **P2 — `--isolated` flag productization.** GBRAIN_HOME threading through
|
||||
MCP registration env, hooks, and uninstall guards ships in v1; the
|
||||
documented `bootstrap --isolated` one-flag wrapper (set env + assert
|
||||
database_path containment + gitignore check, per D2=C) is not yet a flag.
|
||||
- [ ] **P2 — sweep N+1 page fetch.** The links/timeline pass getPage-per-slug
|
||||
loop is bounded (batchLimit 20) but serializes round-trips on the live
|
||||
serve connection; add a getPagesBySlugs batch read to both engines (engine
|
||||
parity + bootstrap-probe obligations) and use it.
|
||||
|
||||
- [ ] **P2 — `gbrain quota` meter command.** Productize the release-gate quota
|
||||
measurement (script+doc ship with the bootstrap wave) into a command that reports
|
||||
a session/day's token burn against the harness subscription allowance. Blocked on
|
||||
a proven per-harness token-count method — an inaccurate meter erodes the trust it
|
||||
exists to build (CEO review D3.4; not promoted by a passing quota gate, by
|
||||
decision). Start: the spike's measurement scripts under docs/designs/.
|
||||
- [ ] **P2 — Networked Docker paste-flow e2e.** The offline container e2e (interview
|
||||
→ render → verify with fake gh) ships with the wave; the full networked flow (bun
|
||||
install, gh auth, repo create, MCP registration) stays manual. Unblocks after 10
|
||||
consecutive green offline runs in heavy-tests (CEO review D3.3b). Start:
|
||||
tests/docker/ harness + the fake-gh recording shim.
|
||||
- [ ] **P2 — Real-agent door e2e needs a provisioned runner.** The real-binary door
|
||||
tests (`test/e2e/bootstrap-real-{claude,codex}.serial.test.ts`) drive the ACTUAL
|
||||
`claude`/`codex` binaries against a real gbrain and pay API cost. They self-SKIP
|
||||
(`describe.skipIf` on binary/auth) everywhere else, so the `real-agent-e2e` job in
|
||||
`.github/workflows/heavy-tests.yml` is a green no-op on stock GitHub runners. To
|
||||
actually EXERCISE them we need a self-hosted / manually-provisioned runner with
|
||||
authed `claude` + `codex` and provider creds exported
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`). Until then they
|
||||
run locally on an operator machine only. Start: stand up a labeled runner with the
|
||||
binaries pre-authed, or a scheduled self-hosted lane.
|
||||
|
||||
- [ ] **P1 — unit-shard exit hang: bun test process leaks a ref'd handle and never
|
||||
exits after all tests pass.** Probabilistic (scales with file count/duration),
|
||||
PRE-EXISTING: reproduces on origin/master with the same 274-file combination
|
||||
(`bun test $(cat list) --max-concurrency=2` — shard-2 composition from the
|
||||
agent-bootstrap wave's reshuffled split; solo halves/eighths pass, full list
|
||||
hangs on both branches; a repeat of the same 68-file quarter passed, so it is a
|
||||
race, not a single file). Forensics: the "[process-cleanup] db-lock:test:r1:
|
||||
PGlite is closed" tail line is printed BY the SIGTERM cleanup at kill time, not
|
||||
pre-hang; db-lock refresh + process-watchdog intervals are already unref'd —
|
||||
the leak is elsewhere (candidate class: a ref'd handle in a PGLite/worker
|
||||
teardown race). run-unit-parallel.sh now classifies the exact signature
|
||||
(killed at cap + all assigned files started + zero fail markers) as a loud
|
||||
warn-pass so it can't red-X unrelated work; fixing the leak removes the
|
||||
classifier's reason to exist. Start: reproduce with `--inspect` /
|
||||
Bun.unsafe process handle dumps on the full shard-2 list.
|
||||
- [ ] **P2 — WAL-repair unit tests flake under parallel shard load.**
|
||||
`test/pglite-repair.test.ts` (`repairPgliteWal rename-based backup`,
|
||||
`inspectPgliteDataDir verdicts`) failed once in a full 4-shard run and pass
|
||||
44/44 standalone — load-sensitive, from the v0.42.75.0 WAL auto-repair wave
|
||||
(pre-existing on master; observed during the agent-bootstrap gate runs).
|
||||
Start: run the file under `--max-concurrency=4` alongside PGLite-heavy
|
||||
neighbors to reproduce; suspect tmp-dir or timing assumptions.
|
||||
|
||||
## Giftable-import wave follow-ups (filed at build time)
|
||||
|
||||
- [ ] **P1 — Wire citation edge types into relational retrieval.** `relational-intent.ts`
|
||||
recognizes a hardcoded edge-type set that excludes `overrules`/`distinguishes`/
|
||||
`relies-on` (the types citation-graph-ingest creates). Until they're walked by
|
||||
natural-language relational recall, the skill's value is explicit `graph-query`
|
||||
only. Add the types + an eval fixture proving a relational question traverses a
|
||||
citation edge. Files: `src/core/search/relational-intent.ts`,
|
||||
`src/core/search/relational-recall.ts`.
|
||||
- [ ] **P2 — Native operation-boundary confirm for destructive ops.** data-loss-gate
|
||||
is routing prose; destructive paths (bulk forget, `delete_page` sweeps, source
|
||||
removal, mounts remove) can bypass it via CLI/MCP/jobs. Add a native confirm
|
||||
(TTY prompt / `--yes` flag / MCP scope) at the operation boundary.
|
||||
- [ ] **P2 — `gbrain ingest feed`: native feed adapter.** blog-ingest ships the
|
||||
agent-procedure layer; the durable path is a deterministic RSS/Atom adapter
|
||||
(discovery, pagination, canonical-URL dedup, 429 backoff) behind one command.
|
||||
- [ ] **P2 — Native AI-chat export importer.** conversation-archive converts
|
||||
ChatGPT/Claude/Perplexity exports via agent procedure; a native importer
|
||||
(export JSON → conversations/ pages) makes it deterministic. Pairs with the
|
||||
existing conversation-parser surface.
|
||||
- [ ] **P2 — Entity-guard as a native op.** phonetic-name-guard's own changelog
|
||||
proves prose-only failed: ASR-variant entity collisions need a native check
|
||||
(registry + alias table consulted at put/import time). The wave shipped the
|
||||
registry-first discipline in brain-ingest-gate; this hardens it.
|
||||
- [ ] **P2 — Premiere-repo program ① distribution:** list gbrain on skills.sh +
|
||||
Claude Code plugin marketplace + agentskills.io conformance; README cross-
|
||||
harness matrix (CI-verified). First fast-follow PR after this wave.
|
||||
- [ ] **P2 — Premiere-repo program ② receipts:** public BrainBench receipts page
|
||||
pairing accuracy with token cost per query, regenerated per release; "trust
|
||||
layer" framing (data-loss-gate + brain-ingest-gate + correction-pipeline).
|
||||
- [ ] **P3 — Premiere-repo program ③ protocol moat:** Anthropic memory-tool
|
||||
(`memory_20250818`) adapter backed by recall/remember; publish MEMORY_VERBS_v1
|
||||
as an open spec with BrainBench as its conformance suite. Own cathedral.
|
||||
- [ ] **P3 — Premiere-repo program ④ badges:** per-skill conformance badges
|
||||
(security-scan + eval-receipt + provenance hash) surfaced in manifest/README;
|
||||
generalize the functional-area-resolver A/B harness into `evals/skills/`.
|
||||
- [ ] **P3 — RESOLVER two-layer compression as its own PR.** Deferred out of the
|
||||
wave at eng review: requires arrow-form dispatcher entries, the A/B run at
|
||||
>=95% (per the functional-area-resolver contract), resolver.test.ts updates,
|
||||
and fixture backfill for fixture-less skills. RESOLVER.md is now past the 12KB
|
||||
gate, so the skill's precondition is satisfied.
|
||||
- [ ] **P3 — extract-atoms quality-gate prompt patch.** Fold the donor pack's
|
||||
truism filter / statistic-punchline test / entity-page routing test / named-
|
||||
attribution rule into `src/core/cycle/extract-atoms.ts`'s EXTRACT_PROMPT,
|
||||
eval-gated (the native prompt's only bar today is "not a generic platitude").
|
||||
- [ ] **P3 — cross-modal eval `--corpus` hub-and-spoke mode + judge-leniency
|
||||
normalization.** Follow relative .md links from a hub page so multi-page brain
|
||||
artifacts aren't falsely penalized; normalize per-judge leniency in
|
||||
`src/core/cross-modal-eval/aggregate.ts` (mean+floor only today).
|
||||
- [ ] **P3 — Advisor collectors: freshness-monitor + context-audit token drift.**
|
||||
Two new collectors: per-source staleness SLA (the donor freshness-monitor
|
||||
kernel) and a deterministic loaded-context token-drift check feeding the
|
||||
context-audit skill.
|
||||
- [ ] **P3 — idea-miner import (deferred at CEO review, fit 6).** Daily brain-
|
||||
grounded "what could I build" mining feeding skill-creator; below the wave's
|
||||
fit bar but a strong self-improvement story.
|
||||
- [ ] **P3 — public-repo-guard revisit.** Only egress leak-gate candidate; its
|
||||
upstream scan script fails open (`SCAN_EXIT` captured after `|| true`). Fix
|
||||
upstream first; template-ize the patterns file; mind the gstack cso boundary.
|
||||
- [ ] **P3 — `search --fm` + schema-pack fragment** from the social-json-store
|
||||
audit disposition: frontmatter-ID/JSONB query kernel as a native search flag
|
||||
+ a schema-pack fragment, not a skill.
|
||||
- [ ] **P3 — back-catalog-check kernel.** Optional pre-publish own-corpus
|
||||
consistency pass folding into fact-check (per-claim own-record search);
|
||||
`find_contradictions` + idea-lineage cover the rest today.
|
||||
|
||||
## Skill self-knowledge — semantic skill search (deferred subsystem, from the migration-harness build)
|
||||
|
||||
- [ ] **P2 — Make built-in skills semantically searchable in the brain.** Today skills
|
||||
are markdown the harness routes to via triggers + a host catalog (`list_skills`/
|
||||
`get_skill`); `gbrain search "how do I verify claims"` can't surface `fact-check`.
|
||||
Making skills first-class searchable content needs a real design pass (tenancy +
|
||||
source-isolation: skill pages must not pollute user-source query results; embedding
|
||||
storage + backfill; search-steering to include/exclude the skill catalog; engine
|
||||
parity). Deliberately NOT built in the currency/preconditions wave — it is a
|
||||
subsystem that deserves its own eng + CEO review, not a rider. The currency work
|
||||
(`skillpack status`/`sync`, doctor `skill_currency`) already keeps the brain's skill
|
||||
set current on upgrade; this item is purely about semantic retrieval of skills.
|
||||
|
||||
+4
-3
@@ -19,7 +19,8 @@
|
||||
},
|
||||
"overrides": {
|
||||
"@babel/core": "^7.29.6",
|
||||
"postcss": "^8.5.10",
|
||||
"nanoid": "^3.3.17",
|
||||
"postcss": "^8.5.23",
|
||||
},
|
||||
"packages": {
|
||||
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
|
||||
@@ -224,7 +225,7 @@
|
||||
|
||||
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
|
||||
|
||||
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
|
||||
"nanoid": ["nanoid@3.3.18", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w=="],
|
||||
|
||||
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
|
||||
|
||||
@@ -232,7 +233,7 @@
|
||||
|
||||
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
|
||||
|
||||
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
|
||||
"postcss": ["postcss@8.5.25", "", { "dependencies": { "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw=="],
|
||||
|
||||
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
|
||||
|
||||
|
||||
+2
-1
@@ -20,6 +20,7 @@
|
||||
},
|
||||
"overrides": {
|
||||
"@babel/core": "^7.29.6",
|
||||
"postcss": "^8.5.10"
|
||||
"postcss": "^8.5.23",
|
||||
"nanoid": "^3.3.17"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
"express-rate-limit": "^7.5.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"heic-decode": "^2.1.0",
|
||||
"js-yaml": "^3.15.0",
|
||||
"js-yaml": "^3.15.1",
|
||||
"marked": "^18.0.2",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
@@ -53,13 +53,13 @@
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"body-parser": "^2.3.0",
|
||||
"fast-uri": "^3.1.4",
|
||||
"fast-uri": "^3.1.5",
|
||||
"fast-xml-builder": "^1.1.7",
|
||||
"fast-xml-parser": "^5.7.0",
|
||||
"form-data": "^4.0.6",
|
||||
"hono": "^4.12.25",
|
||||
"ip-address": "^10.1.1",
|
||||
"js-yaml": "^3.15.0",
|
||||
"hono": "^4.12.34",
|
||||
"ip-address": "^10.3.1",
|
||||
"js-yaml": "^3.15.1",
|
||||
"qs": "^6.15.2",
|
||||
},
|
||||
"packages": {
|
||||
@@ -163,7 +163,7 @@
|
||||
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@2.0.11", "", { "peerDependencies": { "hono": "^4" } }, "sha512-bjD221KPLoJTWUwso1J6fGKiTXEUFedG/s0visavY4zakFPkeGURMRNly+FhBHs7T8Dz4qHaZIMX9ZoJHSJtKA=="],
|
||||
"@hono/node-server": ["@hono/node-server@2.1.0", "", { "peerDependencies": { "hono": "^4" } }, "sha512-XovyyCCnBzW+zKu+z/zq8hwNs4KOR5rEMAOxo2f40Q5xoOI37IMm6MIg2COOUtUApo0i6850MTBKH2u4QLGIqg=="],
|
||||
|
||||
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
|
||||
|
||||
@@ -401,7 +401,7 @@
|
||||
|
||||
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
|
||||
|
||||
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
|
||||
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
|
||||
|
||||
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
|
||||
|
||||
@@ -437,7 +437,7 @@
|
||||
|
||||
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
|
||||
|
||||
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
|
||||
"hono": ["hono@4.13.0", "", {}, "sha512-jhunvfHWxd7J5EFfSgH4xsYJzSe/lfqbUCxiyyeaQasUsXeEHXtzVid+7EOGByc5JnFa23SSFL3Y2RV/z1T+eQ=="],
|
||||
|
||||
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
|
||||
|
||||
@@ -447,7 +447,7 @@
|
||||
|
||||
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
|
||||
|
||||
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
|
||||
"ip-address": ["ip-address@10.4.0", "", {}, "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ=="],
|
||||
|
||||
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
|
||||
|
||||
@@ -461,7 +461,7 @@
|
||||
|
||||
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
|
||||
|
||||
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
|
||||
"js-yaml": ["js-yaml@3.15.1", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag=="],
|
||||
|
||||
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
|
||||
|
||||
|
||||
+6
-1
@@ -18,4 +18,9 @@ timeout = 60_000
|
||||
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
|
||||
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
|
||||
# test/helpers/audit-dir-preload.ts for the full rationale.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
|
||||
#
|
||||
# Same treatment for the sync failure ledger: broken-fixture import/sync tests
|
||||
# were appending rows into the operator's real ~/.gbrain/sync-failures.jsonl,
|
||||
# which `gbrain doctor` reads and warns on. See
|
||||
# test/helpers/sync-failures-preload.ts.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts", "./test/helpers/sync-failures-preload.ts"]
|
||||
|
||||
+123
-82
@@ -4,7 +4,7 @@
|
||||
|
||||
Every GBrain operation goes through `BrainEngine`. The engine is the contract between "what the brain can do" and "how it's stored." Swap the engine, keep everything else.
|
||||
|
||||
v0 shipped `PostgresEngine` backed by Supabase. v0.7 adds `PGLiteEngine` -- embedded Postgres 17.5 via WASM (@electric-sql/pglite), zero-config default. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
|
||||
Two engines ship today: `PGLiteEngine` — embedded Postgres via WASM (@electric-sql/pglite), the zero-config default — and `PostgresEngine`, backed by Supabase or any Postgres + pgvector. The interface is designed so a `DuckDBEngine`, `TursoEngine`, or any custom backend could slot in without touching the CLI, MCP server, skills, or any consumer code.
|
||||
|
||||
## Why this matters
|
||||
|
||||
@@ -12,7 +12,7 @@ Different users have different constraints:
|
||||
|
||||
| User | Needs | Best engine |
|
||||
|------|-------|-------------|
|
||||
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (default since v0.7) |
|
||||
| Getting started | Zero-config, no accounts, no server | PGLiteEngine (the default) |
|
||||
| Power user (you) | World-class search, 7K+ pages, zero-ops | PostgresEngine + Supabase |
|
||||
| Open source hacker | Single file, no server, git-friendly | PGLiteEngine |
|
||||
| Team/enterprise | Multi-user, RLS, audit trail | PostgresEngine + self-hosted |
|
||||
@@ -23,72 +23,30 @@ The engine interface means we don't have to choose. PGLite is the zero-friction
|
||||
|
||||
## The interface
|
||||
|
||||
```typescript
|
||||
// src/core/engine.ts
|
||||
**The single source of truth is `export interface BrainEngine` in
|
||||
`src/core/engine.ts`.** It is large (100+ methods) and grows with every
|
||||
feature wave — do NOT work from any snapshot of it, including an old copy of
|
||||
this doc. Read the interface itself, and let
|
||||
`test/e2e/engine-parity.test.ts` + `test/pglite-engine.test.ts` tell you
|
||||
whether both engines agree.
|
||||
|
||||
export interface BrainEngine {
|
||||
// Lifecycle
|
||||
connect(config: EngineConfig): Promise<void>;
|
||||
disconnect(): Promise<void>;
|
||||
initSchema(): Promise<void>;
|
||||
transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T>;
|
||||
The method families, to orient you before opening the file:
|
||||
|
||||
// Pages CRUD
|
||||
getPage(slug: string): Promise<Page | null>;
|
||||
putPage(slug: string, page: PageInput): Promise<Page>;
|
||||
deletePage(slug: string): Promise<void>;
|
||||
listPages(filters: PageFilters): Promise<Page[]>;
|
||||
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
|
||||
// Chunks
|
||||
upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void>;
|
||||
getChunks(slug: string): Promise<Chunk[]>;
|
||||
|
||||
// Links
|
||||
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
|
||||
removeLink(from: string, to: string): Promise<void>;
|
||||
getLinks(slug: string): Promise<Link[]>;
|
||||
getBacklinks(slug: string): Promise<Link[]>;
|
||||
traverseGraph(slug: string, depth?: number): Promise<GraphNode[]>;
|
||||
|
||||
// Tags
|
||||
addTag(slug: string, tag: string): Promise<void>;
|
||||
removeTag(slug: string, tag: string): Promise<void>;
|
||||
getTags(slug: string): Promise<string[]>;
|
||||
|
||||
// Timeline
|
||||
addTimelineEntry(slug: string, entry: TimelineInput): Promise<void>;
|
||||
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
|
||||
|
||||
// Raw data
|
||||
putRawData(slug: string, source: string, data: object): Promise<void>;
|
||||
getRawData(slug: string, source?: string): Promise<RawData[]>;
|
||||
|
||||
// Versions
|
||||
createVersion(slug: string): Promise<PageVersion>;
|
||||
getVersions(slug: string): Promise<PageVersion[]>;
|
||||
revertToVersion(slug: string, versionId: number): Promise<void>;
|
||||
|
||||
// Stats + health
|
||||
getStats(): Promise<BrainStats>;
|
||||
getHealth(): Promise<BrainHealth>;
|
||||
|
||||
// Ingest log
|
||||
logIngest(entry: IngestLogInput): Promise<void>;
|
||||
getIngestLog(opts?: IngestLogOpts): Promise<IngestLogEntry[]>;
|
||||
|
||||
// Config
|
||||
getConfig(key: string): Promise<string | null>;
|
||||
setConfig(key: string, value: string): Promise<void>;
|
||||
|
||||
// Migration + advanced (added v0.7)
|
||||
runMigration(sql: string): Promise<void>;
|
||||
getChunksWithEmbeddings(slug: string): Promise<ChunkWithEmbedding[]>;
|
||||
}
|
||||
```
|
||||
- **Lifecycle + identity** — `connect` / `disconnect` / `reconnect`,
|
||||
`initSchema`, `transaction`, `withReservedConnection`, and the `kind`
|
||||
discriminator (`'pglite' | 'postgres'`) for the rare engine-specific branch.
|
||||
- **Pages CRUD** — `getPage`, `putPage`, `deletePage`, `listPages`, slug
|
||||
resolution.
|
||||
- **Search** — `searchKeyword`, `searchVector`, chunk-level variants, takes
|
||||
search (keyword + vector), and `relationalFanout` (the typed-edge recall
|
||||
arm).
|
||||
- **Chunks + embeddings** — upsert/get, embedding-bearing variants.
|
||||
- **Graph** — links (single + batch writers), backlinks, `traverseGraph`,
|
||||
`traversePaths`.
|
||||
- **Tags, timeline (single + batch), raw data, versions.**
|
||||
- **Takes / facts / eval / salience** — the epistemological layer and the
|
||||
instruments over it.
|
||||
- **Stats, health, ingest log, config, migrations.**
|
||||
|
||||
### Key design choices
|
||||
|
||||
@@ -131,7 +89,7 @@ export interface BrainEngine {
|
||||
|
||||
RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They operate on `SearchResult[]` arrays. Only the raw keyword and vector searches are engine-specific.
|
||||
|
||||
## PostgresEngine (v0, ships)
|
||||
## PostgresEngine
|
||||
|
||||
**Dependencies:** `postgres` (porsager/postgres), `pgvector`
|
||||
|
||||
@@ -144,9 +102,7 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
- JSONB for frontmatter with GIN index
|
||||
- Connection pooling via Supabase Supavisor (port 6543)
|
||||
|
||||
**Hosting:** Supabase Pro ($25/mo). Zero-ops. Managed Postgres with pgvector built in.
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
**Hosting:** Supabase Pro ($25/mo, zero-ops, pgvector built in) is the managed path; self-hosted Postgres + pgvector (Docker or Homebrew — recipe in the troubleshooting section below) works the same.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
@@ -193,17 +149,17 @@ run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
## PGLiteEngine
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
**Dependencies:** `@electric-sql/pglite`
|
||||
|
||||
**What it is:** Embedded Postgres 17.5 compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. All 37 BrainEngine methods implemented.
|
||||
**What it is:** Embedded Postgres compiled to WASM via ElectricSQL's PGLite. Runs in-process, no server, no Docker, no accounts. Same SQL as PostgresEngine -- not a separate dialect. Implements the full `BrainEngine` interface; `test/e2e/engine-parity.test.ts` pins that the two engines move in lockstep.
|
||||
|
||||
**PGLite-specific details:**
|
||||
- Uses `pglite-schema.ts` for DDL (pgvector extension, pg_trgm, triggers, indexes)
|
||||
- Parameterized queries throughout (shared utilities in `src/core/utils.ts`)
|
||||
- `hybridSearch` keyword-only fallback when `OPENAI_API_KEY` is not set
|
||||
- Data stored at `~/.gbrain/brain.db` (configurable)
|
||||
- Data stored at `~/.gbrain/brain.pglite` (configurable)
|
||||
- pgvector HNSW index for cosine similarity vector search (same as Postgres)
|
||||
- tsvector + ts_rank for full-text search (same as Postgres)
|
||||
- pg_trgm for fuzzy slug resolution (same as Postgres)
|
||||
@@ -221,6 +177,85 @@ live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
The migration and the autopilot daemon do not race: `migrate --to` claims a
|
||||
cooperative pause marker before touching the target. The marker doubles as a
|
||||
migration mutex — a second concurrent migrate refuses to run, and a marker
|
||||
that cannot be written refuses the migration outright. Background job workers
|
||||
stop picking up new work while it is parked, and the migration waits for
|
||||
in-flight sync/embed/cycle work and running jobs to actually drain (watching
|
||||
the DB lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS` — default 300;
|
||||
`0` skips the wait). Cleanup registers the moment the claim lands, so the
|
||||
marker is released on failure and on catchable signals; a marker orphaned by
|
||||
an uncleanly killed run is adopted by a later migrate only after a
|
||||
pid-liveness check (a live migrate's marker is never stolen), and the daemon
|
||||
clears an orphan whose owning process died on its next poll. `gbrain
|
||||
autopilot --status` reports `paused` (exit 1) while the marker is parked and
|
||||
prints the marker path; on a host with no daemon running to self-heal,
|
||||
remove an orphan by hand only after confirming the pid it names is dead.
|
||||
After a clean flip the daemon detects the engine change on its next
|
||||
tick and relaunches onto the new engine, and the migration warns if an
|
||||
exported connection-string env var would override the new config.
|
||||
|
||||
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
|
||||
|
||||
**Symptom:** every PGLite-touching command dies at startup with
|
||||
`PGLite failed to initialize its WASM runtime … Aborted(). Build with
|
||||
-sASSERTIONS for more info.` — commonly first seen right after a macOS
|
||||
upgrade.
|
||||
|
||||
**Real root cause:** corrupt WAL/checkpoint state in the data dir after an
|
||||
unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the
|
||||
write-ahead log; every subsequent open fails WAL replay inside WASM and
|
||||
Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM
|
||||
incompatibility — the same signature reproduces across macOS versions and on
|
||||
Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun
|
||||
version bump changes it.
|
||||
|
||||
**Recovery ladder** (top rung first):
|
||||
|
||||
1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort,
|
||||
backs up `pg_wal/` + `pg_control` into a sibling
|
||||
`<dataDir>.wal-repair-backup-<ts>/` dir, resets the WAL in place
|
||||
(pg_resetwal semantics — data files preserved; transactions not
|
||||
checkpointed before the corruption may be lost), and retries once. On
|
||||
success it prints a loud stderr notice naming the backup and recommending
|
||||
`gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired
|
||||
data-dir lock (never after reaping another process's lock), skips for a
|
||||
cooldown window after a failed attempt
|
||||
(`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one
|
||||
backup per corruption episode (newest 3 episodes retained), and restores
|
||||
the original files if the retry still fails. Kill-switch:
|
||||
`GBRAIN_PGLITE_WAL_REPAIR=off`.
|
||||
2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir
|
||||
(read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset
|
||||
deliberately. Refuses when another gbrain process holds the brain (a live
|
||||
`gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`.
|
||||
3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default
|
||||
from your config) wipes and re-creates the brain from your brain repo, or
|
||||
manually: back up `~/.gbrain`, move `brain.pglite` aside,
|
||||
`gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`.
|
||||
Required for *catalog* corruption (58P01 / pgvector load failure) — WAL
|
||||
repair cannot fix that class.
|
||||
4. **Switch engines.** `gbrain init --supabase`, or native Postgres +
|
||||
pgvector (recipe below, contributed by @roysaurav):
|
||||
|
||||
```bash
|
||||
brew install postgresql@17
|
||||
brew services start postgresql@17
|
||||
createdb gbrain
|
||||
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
|
||||
cd pgvector && make && make install
|
||||
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
# ~/.gbrain/config.json: { "engine": "postgres",
|
||||
# "database_url": "postgresql://localhost:5432/gbrain" }
|
||||
gbrain apply-migrations --yes && gbrain doctor
|
||||
```
|
||||
|
||||
`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails
|
||||
to connect: it diagnoses the dir from disk, names the repair command, reports
|
||||
retained repair backups, and escalates when repairs keep recurring (that
|
||||
means the unclean-shutdown genesis is still active — see the ladder's rung 4).
|
||||
|
||||
## JSONB writes: never double-encode (the #2339 trap)
|
||||
|
||||
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
|
||||
@@ -259,16 +294,22 @@ and assert `jsonb_typeof` — the assertion PGLite cannot make.
|
||||
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
|
||||
2. Add to engine factory in `src/core/engine-factory.ts`:
|
||||
```typescript
|
||||
export function createEngine(type: string): BrainEngine {
|
||||
switch (type) {
|
||||
case 'pglite': return new PGLiteEngine();
|
||||
case 'postgres': return new PostgresEngine();
|
||||
case 'myengine': return new MyEngine();
|
||||
default: throw new Error(`Unknown engine: ${type}`);
|
||||
export async function createEngine(config: EngineConfig): Promise<BrainEngine> {
|
||||
switch (config.engine || 'postgres') {
|
||||
case 'pglite': {
|
||||
const { PGLiteEngine } = await import('./pglite-engine.ts');
|
||||
return new PGLiteEngine();
|
||||
}
|
||||
case 'myengine': {
|
||||
const { MyEngine } = await import('./my-engine.ts');
|
||||
return new MyEngine();
|
||||
}
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
The factory uses dynamic imports so engines are only loaded when selected.
|
||||
The factory uses dynamic imports so an engine's dependencies (e.g. the
|
||||
PGLite WASM blob) are only loaded when that engine is selected.
|
||||
3. Store engine type in `~/.gbrain/config.json`: `{ "engine": "myengine", ... }`
|
||||
4. Add tests. The test suite should be engine-agnostic where possible... same test cases, different engine constructor.
|
||||
5. Document in this file + add a design doc in `docs/`
|
||||
@@ -299,7 +340,7 @@ Every method in `BrainEngine`. The full interface. No optional methods, no featu
|
||||
| JSONB queries | GIN index | GIN index | Identical |
|
||||
| Concurrent access | Connection pooling | Single process | PGLite limitation |
|
||||
| Hosting | Supabase, self-hosted, Docker | Local file | |
|
||||
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Added v0.7 |
|
||||
| Migration methods | runMigration, getChunksWithEmbeddings | Same | Identical |
|
||||
|
||||
## Future engine ideas
|
||||
|
||||
|
||||
@@ -6,6 +6,14 @@ A system prompt for any AI agent that wants to build and maintain a personal kno
|
||||
|
||||
Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest.
|
||||
|
||||
> **Relationship to schema packs:** this document is the prose, paste-in
|
||||
> version of the schema pattern. gbrain also ships a machine-enforced
|
||||
> counterpart — schema packs (`gbrain schema`, typed pages, extraction,
|
||||
> aliases, lint) — documented in `docs/architecture/schema-packs.md` and
|
||||
> `docs/schema-author-tutorial.md`. The prose schema here and the active
|
||||
> schema pack should describe the same brain; when you evolve one, evolve
|
||||
> the other.
|
||||
|
||||
---
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
<!-- skillpack-version: 0.7.0 -->
|
||||
<!-- source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md -->
|
||||
# GBrain Skillpack: Reference Architecture for AI Agents
|
||||
|
||||
This is a reference architecture for how a production AI agent uses gbrain as its
|
||||
knowledge backbone. Based on patterns from a real deployment with 14,700+ brain
|
||||
files, 40+ skills, and 20+ cron jobs running continuously.
|
||||
files, the 50+ bundled skills (`skills/manifest.json`), and 20+ cron jobs running
|
||||
continuously.
|
||||
|
||||
**The memex vision, realized.** Vannevar Bush imagined a device where an individual
|
||||
stores everything, mechanized so it may be consulted with exceeding speed. GBrain is
|
||||
@@ -25,6 +25,7 @@ The foundational read-write loop and data model.
|
||||
| [Entity Detection](guides/entity-detection.md) | Run it on every message. Capture original thinking + entity mentions |
|
||||
| [The Originals Folder](guides/originals-folder.md) | Capturing WHAT YOU THINK, not just what you found |
|
||||
| [Brain-First Lookup](guides/brain-first-lookup.md) | Check the brain before calling any external API |
|
||||
| [Push-Based Context](guides/push-context.md) | volunteer_context: the brain volunteers relevant pages instead of waiting to be asked |
|
||||
| [Compiled Truth + Timeline](guides/compiled-truth.md) | Above the line: current synthesis. Below: append-only evidence |
|
||||
| [Source Attribution](guides/source-attribution.md) | Every fact needs a citation. Format and hierarchy |
|
||||
|
||||
@@ -99,6 +100,7 @@ Keeping it running and up to date.
|
||||
|
||||
| Guide | What It Covers |
|
||||
|-------|---------------|
|
||||
| [Agent Bootstrap](guides/bootstrap.md) | The paste-in install: `gbrain bootstrap`, hooks, `bootstrap verify`, keyless mode |
|
||||
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
|
||||
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
|
||||
|
||||
|
||||
+28
-14
@@ -1,5 +1,13 @@
|
||||
# GBrain Installation Verification Runbook
|
||||
|
||||
> **One-command equivalent:** `gbrain bootstrap verify` runs the whole install
|
||||
> contract (round-trip, graph floor, and more) automatically and exits non-zero
|
||||
> on failure — it is the modern first thing to run after any install. See
|
||||
> [docs/guides/bootstrap.md](guides/bootstrap.md). This runbook is the
|
||||
> **manual, deep-verification** companion: use it when `bootstrap verify` fails
|
||||
> and you need to isolate which layer broke, or when you want to understand
|
||||
> what "healthy" looks like check by check.
|
||||
|
||||
Run these checks after install to confirm every part of GBrain is working.
|
||||
Each check includes the command, expected output, and what to do if it fails.
|
||||
|
||||
@@ -20,7 +28,8 @@ gbrain doctor --json
|
||||
**Expected:** All checks return `"ok"`:
|
||||
- `connection`: connected, N pages
|
||||
- `pgvector`: extension installed
|
||||
- `rls`: enabled on all tables
|
||||
- `rls`: enabled on all tables (Postgres/Supabase brains only — PGLite brains
|
||||
skip this check; the embedded engine has no remote surface)
|
||||
- `schema_version`: current
|
||||
- `embeddings`: coverage percentage
|
||||
|
||||
@@ -33,12 +42,12 @@ check. See `skills/setup/SKILL.md` Error Recovery table.
|
||||
|
||||
**Check:** Ask the agent: "What is the brain-agent loop?"
|
||||
|
||||
**Expected:** The agent references GBRAIN_SKILLPACK.md Section 2 and describes
|
||||
the read-write cycle: detect entities, read brain, respond with context, write
|
||||
brain, sync.
|
||||
**Expected:** The agent describes the read-write cycle documented in
|
||||
[docs/guides/brain-agent-loop.md](guides/brain-agent-loop.md): detect entities,
|
||||
read brain, respond with context, write brain, sync.
|
||||
|
||||
**If it fails:** The agent hasn't loaded the skillpack. Run step 6 from the
|
||||
install paste (read `docs/GBRAIN_SKILLPACK.md`).
|
||||
**If it fails:** The agent hasn't loaded the skillpack. Have it read
|
||||
`docs/GBRAIN_SKILLPACK.md` (the index) and follow the Core Patterns links.
|
||||
|
||||
---
|
||||
|
||||
@@ -53,8 +62,8 @@ gbrain check-update --json
|
||||
**Expected:** Returns JSON with `current_version`, `latest_version`,
|
||||
`update_available` (boolean). The cron `gbrain-update-check` is registered.
|
||||
|
||||
**If it fails:** Run step 7 from the install paste. See GBRAIN_SKILLPACK.md
|
||||
Section 17.
|
||||
**If it fails:** See [docs/guides/upgrades-auto-update.md](guides/upgrades-auto-update.md)
|
||||
for how to register the update-check cron.
|
||||
|
||||
---
|
||||
|
||||
@@ -88,8 +97,9 @@ find /data/brain -name '*.md' \
|
||||
Some difference is normal (files added since last sync), but if page count is
|
||||
less than half the file count, sync is silently skipping pages.
|
||||
|
||||
**If page count is way too low:** The #1 cause is an unreachable direct
|
||||
connection on an IPv4-only host. GBrain uses the Transaction pooler (port 6543)
|
||||
**If page count is way too low (Supabase/Postgres brains):** The #1 cause is an
|
||||
unreachable direct connection on an IPv4-only host. (PGLite brains have no
|
||||
network layer — for them, check that the sync cron/watch is actually running.) GBrain uses the Transaction pooler (port 6543)
|
||||
for reads, but routes migrations, DDL, and sync transactions to a derived direct
|
||||
connection (`db.<ref>.supabase.co:5432`), which is IPv6-only.
|
||||
- On an IPv4-only host, reads work but sync transactions fail and silently skip
|
||||
@@ -122,7 +132,7 @@ This is the real test. Edit a brain page, push, wait, search.
|
||||
1. Edit a page in the brain repo (e.g., correct a fact on a person's page):
|
||||
|
||||
```bash
|
||||
# Example: fix a line in Gustaf's page
|
||||
# Example: fix a line in alice-example's page
|
||||
cd /data/brain
|
||||
# Make a small edit to any .md file
|
||||
git add -A && git commit -m "test: verify live sync" && git push
|
||||
@@ -253,19 +263,23 @@ gbrain repair-jsonb
|
||||
|
||||
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
|
||||
|
||||
**Bonus check** — frontmatter-keyed queries actually resolve:
|
||||
**Bonus check** — the doctor's dedicated JSONB scan agrees:
|
||||
|
||||
```bash
|
||||
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
|
||||
gbrain doctor --json | grep -o '"name":"jsonb_integrity"[^}]*'
|
||||
```
|
||||
|
||||
If this returns rows on a brain with person pages, the JSONB path is healthy.
|
||||
**Expected:** the fragment contains `"status":"ok"` ("All JSONB columns store
|
||||
objects/arrays"). If it reports double-encoded rows, run `gbrain repair-jsonb`.
|
||||
|
||||
---
|
||||
|
||||
## Quick Verification (all checks in one pass)
|
||||
|
||||
```bash
|
||||
# 0. The one-command contract check (exits non-zero on failure)
|
||||
gbrain bootstrap verify
|
||||
|
||||
# 1. Schema
|
||||
gbrain doctor --json
|
||||
|
||||
|
||||
+38
-39
@@ -1,15 +1,24 @@
|
||||
# Install
|
||||
|
||||
Three install paths. Pick one. Mix later if needed.
|
||||
**Recommended door: the agent bootstrap.** Open your agent (Codex, Claude Code,
|
||||
or any harness) in the folder that will become its home and paste the block
|
||||
from the [README's install section](../README.md) — the agent fetches
|
||||
`BOOTSTRAP_FOR_AGENTS.md` from the `latest-stable` tag, installs the CLI,
|
||||
initializes a local PGLite brain, wires MCP, and isn't done until
|
||||
`gbrain bootstrap verify` exits 0. Full contract, security posture, and
|
||||
uninstall: [docs/guides/bootstrap.md](guides/bootstrap.md).
|
||||
|
||||
## 1. Run with an agent platform (recommended)
|
||||
The paths below are the manual equivalents and deep-dive detail. Pick one.
|
||||
Mix later if needed.
|
||||
|
||||
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
|
||||
## 1. Run with an agent platform
|
||||
|
||||
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/NousResearch/hermes-agent)?
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
bun install -g github:garrytan/gbrain#latest-stable
|
||||
gbrain init --pglite # 2 seconds; no server
|
||||
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
|
||||
gbrain skillpack scaffold --all # scaffolds every bundled skill (skills/manifest.json) into your agent workspace
|
||||
gbrain doctor # green checks all the way down
|
||||
```
|
||||
|
||||
@@ -24,7 +33,7 @@ To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts
|
||||
No agent platform, just shell + MCP-aware editor.
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
bun install -g github:garrytan/gbrain#latest-stable
|
||||
gbrain init --pglite
|
||||
```
|
||||
|
||||
@@ -58,16 +67,17 @@ gbrain autopilot --install # background daemon for nightly enrichment
|
||||
**Wire this same local brain into your coding agent** — zero server, zero token:
|
||||
|
||||
```bash
|
||||
claude mcp add gbrain -- gbrain serve # Claude Code
|
||||
codex mcp add gbrain -- gbrain serve # Codex
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs # Claude Code
|
||||
codex mcp add gbrain -- gbrain serve --surface verbs # Codex
|
||||
```
|
||||
|
||||
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
|
||||
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. `--surface verbs` gives the agent the seven-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget`, `context_pack`, `delta` — [MEMORY_VERBS v1](protocol/MEMORY_VERBS_v1.md)) instead of the full tool catalog; drop the flag (default `full`) for every operation. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
|
||||
|
||||
## 3. MCP server (any MCP client)
|
||||
|
||||
```bash
|
||||
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
|
||||
gbrain serve --surface verbs # stdio MCP, just the 7 memory verbs (quickstart)
|
||||
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
|
||||
```
|
||||
|
||||
@@ -88,6 +98,8 @@ Per-client setup guides live in [`docs/mcp/`](mcp/):
|
||||
- [`docs/mcp/CODEX.md`](mcp/CODEX.md)
|
||||
- [`docs/mcp/CLAUDE_DESKTOP.md`](mcp/CLAUDE_DESKTOP.md)
|
||||
- [`docs/mcp/CHATGPT.md`](mcp/CHATGPT.md)
|
||||
- [`docs/mcp/HERMES.md`](mcp/HERMES.md)
|
||||
- [`docs/mcp/OPENCLAW.md`](mcp/OPENCLAW.md)
|
||||
- [`docs/mcp/PERPLEXITY.md`](mcp/PERPLEXITY.md)
|
||||
- [`docs/mcp/DEPLOY.md`](mcp/DEPLOY.md) — production deploy patterns
|
||||
|
||||
@@ -106,44 +118,31 @@ Useful for: team mounts, brain-as-a-service deployments, dev machines without di
|
||||
## Verifying the install
|
||||
|
||||
```bash
|
||||
gbrain bootstrap verify # the whole install contract; exits non-zero on failure
|
||||
gbrain doctor --json # full health check
|
||||
gbrain models # which AI models are configured for what
|
||||
gbrain models doctor # 1-token probe per configured model
|
||||
```
|
||||
|
||||
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
|
||||
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`). For the manual check-by-check runbook, see [docs/GBRAIN_VERIFY.md](GBRAIN_VERIFY.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### PGLite crashes on macOS 26.x (Tahoe)
|
||||
### PGLite crashes at startup (`RuntimeError: Aborted()`)
|
||||
|
||||
PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL:
|
||||
This crash (typically first seen after a macOS upgrade) is **not** a
|
||||
macOS/WASM incompatibility — an unclean shutdown tore the data dir's
|
||||
write-ahead log, and every subsequent open fails WAL replay. The short
|
||||
version of the recovery ladder:
|
||||
|
||||
```bash
|
||||
# Install PostgreSQL + pgvector
|
||||
brew install postgresql@17
|
||||
brew services start postgresql@17
|
||||
createdb gbrain
|
||||
1. **Auto-repair (default):** run any gbrain command — gbrain detects the
|
||||
abort, resets the WAL in place (data preserved, backup kept), and
|
||||
continues. Then run `gbrain doctor`.
|
||||
2. **Manual repair:** `gbrain pglite-repair --dry-run`, then
|
||||
`gbrain pglite-repair --yes`.
|
||||
3. **Rebuild:** `gbrain reinit-pglite`.
|
||||
4. **Switch engines:** Supabase or native Homebrew Postgres + pgvector.
|
||||
|
||||
# Build pgvector from source (required for vector search)
|
||||
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
|
||||
cd pgvector && make && make install
|
||||
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
|
||||
# Point gbrain at your local Postgres
|
||||
cat > ~/.gbrain/config.json << 'EOF'
|
||||
{
|
||||
"engine": "postgres",
|
||||
"database_url": "postgresql://localhost:5432/gbrain",
|
||||
"schema_pack": "gbrain-base-v2"
|
||||
}
|
||||
EOF
|
||||
|
||||
# Run migrations and verify
|
||||
gbrain apply-migrations --yes
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend.
|
||||
|
||||
> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again.
|
||||
The full ladder — safety bounds, kill-switches, when WAL repair can't help,
|
||||
and the Homebrew Postgres recipe — lives in
|
||||
[docs/ENGINES.md](ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted).
|
||||
|
||||
+57
-11
@@ -26,19 +26,21 @@ Two equivalent paths:
|
||||
|
||||
**Path B — manual lifecycle (still supported):**
|
||||
- `bun test` — unit tests (no database required)
|
||||
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
|
||||
run `bun run test:e2e`, then tear it down.
|
||||
- Follow the "E2E test DB lifecycle" steps in
|
||||
[docs/TESTING.md](TESTING.md) to spin up the test DB, run
|
||||
`bun run test:e2e`, then tear it down.
|
||||
|
||||
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
|
||||
|
||||
**Always run typecheck before pushing.** `bun test` (the bun runner)
|
||||
skips TypeScript type checking — it only enforces runtime behavior.
|
||||
**Always run typecheck before pushing.** Neither `bun test` (the bun runner)
|
||||
nor `bun run test` gates on types — `bun run test` is just
|
||||
`bash scripts/run-unit-parallel.sh` (the sharded unit runner; no typecheck,
|
||||
no shell pre-checks — see the test-tier table in [docs/TESTING.md](TESTING.md)).
|
||||
Three ways to actually gate on types:
|
||||
|
||||
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
|
||||
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
|
||||
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
|
||||
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
|
||||
1. `bun run verify` — runs the shell guard checks (privacy, jsonb, source-id,
|
||||
progress-to-stdout, …) plus `bun run typecheck` in parallel
|
||||
(`scripts/run-verify-parallel.sh`). Use this mid-branch.
|
||||
2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
|
||||
3. `bun run ci:local` — the full local CI gate from Path A.
|
||||
|
||||
@@ -290,7 +292,8 @@ matter" with BrainBench-style before/after table, "what this means" closer, then
|
||||
|
||||
Create a migration file at `skills/migrations/v[version].md` when a release
|
||||
includes changes that existing users need to act on. The auto-update agent
|
||||
reads these files post-upgrade (Section 17, Step 4) and executes them.
|
||||
reads these files post-upgrade (see `docs/guides/upgrades-auto-update.md`)
|
||||
and executes them.
|
||||
|
||||
**You need a migration file when:**
|
||||
- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
|
||||
@@ -343,8 +346,8 @@ canonical.
|
||||
## Schema state tracking
|
||||
|
||||
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
|
||||
adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
|
||||
reads this during upgrades to suggest new schema additions without re-suggesting
|
||||
adopted, declined, or added custom. The auto-update agent
|
||||
(`docs/guides/upgrades-auto-update.md`) reads this during upgrades to suggest new schema additions without re-suggesting
|
||||
things the user already declined. The setup skill writes the initial state during
|
||||
Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
|
||||
|
||||
@@ -373,6 +376,46 @@ the release the same, uses that version's `CHANGELOG.md` entry as the notes
|
||||
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
|
||||
missing), and attaches the compiled binaries.
|
||||
|
||||
### The `latest-stable` tag
|
||||
|
||||
The **final step of the release job** force-advances the `latest-stable` tag to
|
||||
the release commit (`git push origin "+${GITHUB_SHA}:refs/tags/latest-stable"`).
|
||||
`latest-stable` is the single sanctioned distribution ref: the README paste
|
||||
block, the `BOOTSTRAP_FOR_AGENTS.md` fetch URL, and
|
||||
`bun install -g github:garrytan/gbrain#latest-stable` all reference it
|
||||
permanently, so paste blocks copied into the wild never rot and there is no 404
|
||||
window between VERSION landing and assets publishing.
|
||||
`scripts/check-bootstrap-tag.sh` keeps the entry docs pinned to this ref.
|
||||
|
||||
Because it moves ONLY after binaries + provenance attestation have fully
|
||||
published, a half-built release never advances it. If the tag-advance step
|
||||
alone fails, re-advance by hand (a full workflow re-run would skip — the
|
||||
release already exists with all assets):
|
||||
|
||||
```bash
|
||||
git push origin "+refs/tags/v<VERSION>^{commit}:refs/tags/latest-stable"
|
||||
```
|
||||
|
||||
### The `publish-template` job
|
||||
|
||||
After the release job, a `publish-template` job force-pushes the rendered
|
||||
agent-workspace template repo (the GitHub "Use this template" door,
|
||||
`vars.TEMPLATE_REPO`, default `garrytan/gbrain-agent-template`) from CI only —
|
||||
no human pushes it by hand, so what adopters clone is exactly what this repo
|
||||
reviewed. It is guarded three ways: the release above fully published; the
|
||||
vendored tree `templates/bootstrap/template-repo/` exists (skip, never fail,
|
||||
if not); and the `TEMPLATE_REPO_PAT` secret is configured (skip if not).
|
||||
Before pushing, it regenerates the template tree
|
||||
(`bun run scripts/generate-template-repo.ts`) and byte-diffs it against the
|
||||
vendored copy — a mismatch fails the job; regenerate + commit the vendored
|
||||
tree (`scripts/check-bootstrap-templates.sh` runs the same diff offline in
|
||||
`bun run verify`).
|
||||
|
||||
**`TEMPLATE_REPO_PAT` scope:** a fine-grained PAT with `contents: write` on
|
||||
the template repository ONLY — no other repositories, no other permissions.
|
||||
Configure it as a repo secret; when absent, template publishing is disabled
|
||||
and the job skips cleanly.
|
||||
|
||||
Why every bump, not selective: `gbrain check-update` resolves the latest
|
||||
version from `VERSION` on master, while binary self-update
|
||||
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
|
||||
@@ -395,6 +438,9 @@ Invariants:
|
||||
history; every new 4-segment `VERSION` mints a fresh tag.
|
||||
- **Permissions stay scoped.** `contents: write` lives on the release job
|
||||
only; everything else runs read-only.
|
||||
- **Never advance `latest-stable` on a partial release.** The tag moves only
|
||||
as the final release-job step, after every asset has published. Manual
|
||||
re-advances must point at a fully published `v<VERSION>` release.
|
||||
|
||||
## PR descriptions cover the whole branch
|
||||
|
||||
|
||||
+23
-8
@@ -11,13 +11,13 @@ Seven test command tiers, each with a clear scope:
|
||||
|
||||
| Command | What it runs | Wallclock | When to use |
|
||||
|---|---|---|---|
|
||||
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
@@ -47,10 +47,10 @@ there even though they pass on Linux and macOS.
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. CI is the ground truth for "did everything pass."
|
||||
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
|
||||
|
||||
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
|
||||
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
|
||||
|
||||
### Failure-first logging
|
||||
|
||||
@@ -61,11 +61,16 @@ When `bun run test` finds any failure, the wrapper:
|
||||
3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
|
||||
4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
|
||||
|
||||
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
|
||||
If a shard hits the per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap (default 3000s — sized so the heaviest count-balanced shard finishes under 4-way contention; `GBRAIN_TEST_SHARD_KILL_AFTER` sets the grace after TERM before KILL, default 30s), the wrapper classifies the kill one of two ways:
|
||||
|
||||
- **EXIT-HANG → warn-pass.** If the shard's log had been silent for ≥300s at kill time AND shows zero `(fail)` markers, the shard finished all its work, leaked a handle, and never exited (a pre-existing, master-reproducible PGLite-adjacent leak — see TODOS.md "unit-shard exit hang"). The wrapper prints a `⚠️ shard N/M: EXIT-HANG ... Treating as pass-with-warning` banner, writes `EXIT-HANG (idle Ns, 0 fails) ... warn-pass` to the summary, and does NOT fail the run. Its pass counts are undercounted (bun never printed its final summary). Bun's per-test `--timeout` turns a genuinely hung TEST into a printed `(fail)` — new output — so this classification cannot mask a hung test; the residual maskable case is a file-level import hang in the very last file, which the banner keeps visible.
|
||||
- **WEDGED → hard failure.** Anything else (failures present, or the log was still growing) writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log with the last 50 lines of the shard log, marks the run failed, and proceeds with other shards' results.
|
||||
|
||||
Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT a test failure — don't burn time bisecting it; a `WEDGED` line is.
|
||||
|
||||
### File taxonomy
|
||||
|
||||
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
|
||||
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
|
||||
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
|
||||
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
|
||||
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
|
||||
@@ -83,6 +88,8 @@ Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-
|
||||
|
||||
### Test-isolation lint and helpers
|
||||
|
||||
**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules.
|
||||
|
||||
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
|
||||
|
||||
| Rule | What it bans | Fix |
|
||||
@@ -168,6 +175,9 @@ Unit tests and what they cover:
|
||||
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
|
||||
- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
|
||||
- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
|
||||
- `test/autopilot-launchd-lifecycle.serial.test.ts` — autopilot lifecycle behavior, not generated-string assertions: the full install → self-disable → status → reinstall → uninstall arc with `launchctl` replaced by an argv recorder and the generated wrapper executed by a REAL bash against a genuinely deleted repo (every platform), plus a darwin-only fail-SKIP describe against the real launchd under a per-run unique label (`GBRAIN_AUTOPILOT_LABEL`) so it can never collide with — or tear down — a real install on the host. Serial: spawns subprocesses and pins HOME/GBRAIN_HOME for the whole file.
|
||||
- `test/autopilot-fanout.test.ts` — Autopilot fan-out and #4046 policy regression: targeted idempotency keys reopen per dispatch interval while stable doctor/remediate keys remain unchanged; the 60-minute full-cycle floor wins with a remaining small plan, and an all-fresh restart check advances the process-local clock without masking failed stale-source submissions.
|
||||
- `test/agent-scheduler-contract.serial.test.ts` — the documented external agent-scheduler shell chain (`gbrain sync --repo X && gbrain embed --stale`, live-sync.md / INSTALL_FOR_AGENTS.md Step 7) driven end-to-end through a real `/bin/sh` against a keyless PGLite brain: the `&&` short-circuit IS the contract (argv arrays can't exercise it), the keyless bare stale embed exits 0, and the pull-failure case that must break the chain does. Anti-vacuity: the fixture commits a real page and every read-back asserts pages >= 1. Serial: real spawned CLI + tmpdir HOME.
|
||||
- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
|
||||
- `test/config.test.ts` — config redaction.
|
||||
- `test/files.test.ts` — MIME/hash.
|
||||
@@ -201,6 +211,8 @@ Unit tests and what they cover:
|
||||
- `test/dedup.test.ts` — source-aware dedup, compiled truth guarantee, layer interactions.
|
||||
- `test/intent.test.ts` — query intent classification: entity/temporal/event/general.
|
||||
- `test/eval.test.ts` — retrieval metrics: `precisionAtK`, `recallAtK`, `mrr`, `ndcgAtK`, `parseQrels`.
|
||||
- `test/brainbench-fixtures.test.ts` / `test/brainbench-generator.test.ts` / `test/brainbench-metrics.test.ts` / `test/brainbench-continuity.test.ts` / `test/brainbench-writeback.test.ts` / `test/brainbench-adapters.test.ts` / `test/brainbench-scoreboard.test.ts` — the BrainBench memory-conformance unit suites (`src/eval/brainbench/`): fixture loader/validator + the sealed-gold seal (a `gold` key inside a fixture must reject) and committed-corpus integrity; generator determinism (the committed corpus is exactly what `gen.ts` produces, holdout discipline, category counts); metric formulas over hand-built turn rows (zero should-retrieve turns, empty injections, acceptable-vs-gold asymmetry, micro-averaging); cross-harness continuity (writer's decision persists through the production write-back pipeline, reader recalls on the SAME brain); write-back grading the PRODUCTION conversation→facts pipeline via the injected gold extractor; adapter seam contracts over hermetic PGLite (budget caps, suppression modes); scoreboard + gate governance (baseline determinism, count-aware gating, corpus-bless modes, justification flow, isolation gates-at-zero).
|
||||
- `test/eval-brainbench-e2e.test.ts` — BrainBench CLI end-to-end via subprocess against a small tmp corpus: the literal exit codes (0 pass / 1 regression / 2 error-or-inconclusive — the CI product), `--out` artifact validity incl. `_meta.metric_glossary`, byte-deterministic `--update-baseline`, anti-vacuous-pass, and the `eval run-all` in-process wiring.
|
||||
- `test/check-resolvable.test.ts` — resolver reachability, MECE overlap, gap detection, proximity-based DRY detection, `extractDelegationTargets` coverage.
|
||||
- `test/dry-fix.test.ts` — auto-fix: three shape-aware expander pure-function tests; five guards (working-tree-dirty, no-git-backup, inside-code-fence, already-delegated within 40 lines, ambiguous-multi-match, block-is-callout).
|
||||
- `test/doctor-fix.test.ts` — `gbrain doctor --fix` CLI integration: dry-run preview, apply path, JSON output shape.
|
||||
@@ -270,7 +282,9 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
|
||||
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
|
||||
- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + skillpack install-model against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
|
||||
- `test/e2e/workspace-generic-compat.test.ts` — always-on (PGLite, no binary): pins the INSTALL_FOR_AGENTS.md "any repo with a workspace" contract against `test/fixtures/generic-agents-workspace/` (Hermes is the motivating consumer): `cwd_walk_up` detection, the `GBRAIN_SKILLS_DIR` override, `check-resolvable` on a root AGENTS.md, and scaffold additivity + refuse-overwrite. The real Hermes-behavior proof is the door suite below.
|
||||
- `test/e2e/install-real-hermes.serial.test.ts` — the hermes "door": real `hermes` binary + real `hermes mcp add` handshake (full-catalog tool discovery; the count tracks the op catalog, so the test asserts discovery happened, not a number) + a paid `hermes -z` recall turn against a seeded brain. Triple-gated: `GBRAIN_REAL_HERMES_E2E=1` (explicit opt-in — run-e2e.sh scrubs GBRAIN_*, so it can never fire under `bun run test:e2e`) + resolvable binary + non-empty ANTHROPIC key (anthropic-pinned on purpose: a second provider key flips hermes provider-auto into a mis-routed 401). Hermetic HOME + HERMES_HOME with a tripwire on the operator's real config; evidence copies to `GBRAIN_E2E_EVIDENCE_DIR` for CI upload. Venue: heavy-tests.yml (`real-agent-e2e` + `hermes-door` jobs).
|
||||
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
|
||||
- `test/e2e/engine-parity.test.ts` — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector` (Postgres ranks pages then picks best chunk while PGLite returns chunks directly, so the source-boost behavior needs parity coverage). Skips without `DATABASE_URL`.
|
||||
@@ -285,6 +299,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/think-source-isolation-pglite.test.ts` — PGLite in-memory suite pinning the `think` gather stage's source scope: seeds three sources with cross-source links and embedded takes, then asserts `runGather` under a federated `sourceIds` grant (and under a scalar `sourceId`) keeps every stream — hybrid retrieval, takes keyword + vector (`searchTakes`/`searchTakesVector`), and the `traversePaths` graph walk — inside the grant while still reaching authorized neighboring sources. No `DATABASE_URL` needed.
|
||||
- `test/e2e/skill-brain-first.test.ts` — doctor reports `skill_brain_first` check with structured issues; `--fix --dry-run` previews insertion without writing; `--fix` applies the canonical Convention callout idempotently; `brain_first: exempt` frontmatter resolves the warn; `brain_first_typo` surfaces a paste-ready hint; audit JSONL records `detected` / `resolved` / `fixed` transitions; stable brain emits 0 audit lines/run.
|
||||
- Tier 2 (`test/e2e/skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI.
|
||||
- `test/e2e/claw-test.test.ts` also covers live mode token-free via shim agents (`OPENCLAW_BIN=<sh script>`): the success-oracle break path (a do-nothing agent now FAILS), the E0 child-friction merge surviving tempdir cleanup, and the upgrade staging + schema-version-probe regression.
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
- **Run E2E tests without asking permission.** When you want to verify behavior, there's a relevant E2E test, or you're shipping anything covered by an E2E suite — spin up the test DB, run the tests, tear down. Don't ask, don't propose it, don't defer. The lifecycle is short (~2-30s startup, sub-minute tests, instant teardown) and the gate value is high. Skipping with "DATABASE_URL unset" is silent regression, not caution.
|
||||
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# Upgrading Downstream Agents
|
||||
|
||||
> **Currency note:** this file is an append-only historical log and stopped
|
||||
> receiving new sections after v0.36.5.0. **The canonical, maintained upgrade
|
||||
> channel is `skills/migrations/v*.md`** (the agent-executed migration files
|
||||
> that `gbrain upgrade` / `gbrain post-upgrade` route through), plus
|
||||
> `CHANGELOG.md` for what each release changed. Use this file only to catch a
|
||||
> long-diverged fork up through the versions it covers; for anything after
|
||||
> v0.36.5.0, walk the migration files and CHANGELOG instead.
|
||||
|
||||
GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments,
|
||||
agent forks of any kind) often **copy** these skill files into their own workspace and
|
||||
diverge over time — adding agent-specific phases, removing irrelevant ones, tightening
|
||||
|
||||
@@ -1,5 +1,23 @@
|
||||
# ZeroEntropy — zembed-1 + zerank-2
|
||||
|
||||
> **Hosted API shutdown: 2026-09-04.** ZeroEntropy announced (2026-07-24)
|
||||
> that its hosted endpoints — `/models/embed` and `/models/rerank` — shut
|
||||
> down on that date. A brain still embedding through the hosted API loses
|
||||
> semantic retrieval entirely on that date: query embedding uses the same
|
||||
> endpoint, so **existing vectors become unqueryable**, not just new
|
||||
> content. Two fixes, either works:
|
||||
>
|
||||
> 1. **Self-host the same model** — zembed-1 weights are Apache-2.0. Serve
|
||||
> them via `llama-server` or Ollama and point the config at the local
|
||||
> endpoint. Keeps every existing vector; no re-embed at all.
|
||||
> 2. **Migrate to another provider** — `gbrain migrate embeddings --to
|
||||
> <provider:model> --dim <N> --dry-run` (resumable; see
|
||||
> [the migration guide](../guides/embedding-migration.md)). `gbrain
|
||||
> doctor` (check `provider_sunset`) prints this command with your
|
||||
> brain's actual `--dim` filled in.
|
||||
>
|
||||
> The hosted setup below remains accurate until the shutdown date.
|
||||
|
||||
[ZeroEntropy](https://zeroentropy.dev) ships two specialized small models
|
||||
for retrieval pipelines:
|
||||
|
||||
|
||||
+130
-127
File diff suppressed because one or more lines are too long
@@ -4,14 +4,14 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
|
||||
|
||||
## The four strategies in concert
|
||||
|
||||
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
|
||||
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at acme-example?" → pages mentioning "alice-example + retrieval" even when the user never typed "acme".
|
||||
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
|
||||
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
|
||||
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
|
||||
|
||||
## Why each one alone fails
|
||||
|
||||
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
|
||||
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in alice-example's portfolio" returns essays about portfolios, not company pages.
|
||||
|
||||
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
|
||||
|
||||
@@ -21,7 +21,7 @@ Vector search alone underdelivers on real personal-knowledge queries. This doc e
|
||||
|
||||
## The benchmark
|
||||
|
||||
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus.
|
||||
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus. (This is the retrieval-ranking benchmark; the in-repo `gbrain eval brainbench` suite — [`docs/eval/BRAINBENCH.md`](../eval/BRAINBENCH.md) — gates the memory behaviors *above* retrieval: unprompted context push, write-back fidelity, cross-session continuity.)
|
||||
|
||||
| Strategy | P@5 | R@5 | Notes |
|
||||
|---|---|---|---|
|
||||
@@ -36,8 +36,8 @@ BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/ga
|
||||
|
||||
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
|
||||
|
||||
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
|
||||
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
|
||||
- Standard markdown links: `[Alice Example](wiki/people/alice-example)`
|
||||
- Obsidian wikilinks: `[[wiki/people/alice-example|Alice Example]]`
|
||||
- Typed-link blockquotes: `> **Convention:** see [path](path).`
|
||||
|
||||
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') JOIN pages ON CONFLICT DO NOTHING RETURNING 1` (free-text-safe; the prior `unnest(${arr}::text[])` form crashed on calendar/Zoom context per gbrain#1861). The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
|
||||
@@ -46,7 +46,7 @@ Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`,
|
||||
|
||||
## ZeroEntropy as reranker: 60% top-1 reshuffle
|
||||
|
||||
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
|
||||
ZeroEntropy's `zerank-2` is the default reranker (on for the `balanced` and `tokenmax` mode bundles, off for `conservative`). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
|
||||
|
||||
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
|
||||
|
||||
@@ -62,8 +62,9 @@ The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `Sea
|
||||
|
||||
## Named-thing retrieval (per-page pool + title + alias + evidence)
|
||||
|
||||
A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than
|
||||
embedding proximity. Four layers, added after the incident in
|
||||
A brain organized around *chosen names* (project codenames, place nicknames —
|
||||
say a project named "Helios" whose page is also known as "the Sun Room") needs
|
||||
more than embedding proximity. Four layers, added after the incident in
|
||||
[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md):
|
||||
|
||||
- **Per-page max-pool** — `searchVector` (both engines) collapses chunk-grain
|
||||
@@ -79,7 +80,7 @@ embedding proximity. Four layers, added after the incident in
|
||||
`page_aliases` table (separate from the `slug_aliases` wikilink redirect) and
|
||||
consulted at query time: a full normalized-query match injects/boosts the
|
||||
canonical page (`applyAliasHop`). The only layer that bridges true synonyms
|
||||
with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill
|
||||
with zero surface overlap ("the Sun Room" → the Helios page). Backfill
|
||||
existing pages with `gbrain reindex --aliases`.
|
||||
- **Evidence contract** — every result carries `evidence`
|
||||
(`alias_hit | exact_title_match | high_vector_match | keyword_exact |
|
||||
@@ -97,13 +98,17 @@ Promote or reject them via `gbrain extraction-pending` / `gbrain
|
||||
extraction-review`.
|
||||
|
||||
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
|
||||
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
|
||||
title + alias, expansion off); `query` is the full-control variant. Route
|
||||
concept / landscape / "all-of-X" questions to `query` — expansion recovers
|
||||
synonym-phrased matches `search` can miss, and a populated `search` result set
|
||||
is not proof of coverage (both are top-K; exhaustive enumeration belongs to
|
||||
`list_pages`). NamedThingBench
|
||||
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
|
||||
specific miss with `gbrain search diagnose "<q>" --target <slug>`.
|
||||
|
||||
## Intent-aware query rewriting
|
||||
|
||||
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
|
||||
`src/core/search/query-intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
|
||||
|
||||
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
|
||||
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
|
||||
@@ -116,42 +121,68 @@ The classifier is deterministic (no LLM call). Wrong classification degrades gra
|
||||
|
||||
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
|
||||
|
||||
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
|
||||
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale. The `query` op is the exception: it defaults `expand: true` per call (pass `expand: false` to opt out) — expansion-by-default is what makes it the concept/landscape verb.
|
||||
|
||||
## Putting it together
|
||||
|
||||
The full pipeline for a `query` op:
|
||||
|
||||
```
|
||||
intent classify
|
||||
intent classify (query-intent.ts — deterministic, no LLM)
|
||||
│
|
||||
▼
|
||||
expansion (if enabled)
|
||||
expansion (if enabled — tokenmax only by default)
|
||||
│
|
||||
▼
|
||||
hybrid search:
|
||||
├── vector (HNSW on chunk embeddings)
|
||||
hybrid recall + fusion:
|
||||
├── vector (HNSW on chunk embeddings, per-page max-pool)
|
||||
├── keyword (BM25 via tsvector)
|
||||
├── relational (v0.42.34.0: typed-edge recall arm — relational queries only)
|
||||
├── title-phrase arm
|
||||
├── relational (typed-edge recall arm — relational queries only)
|
||||
├── source-aware re-rank (CASE in SQL)
|
||||
└── RRF fusion → top 30
|
||||
└── RRF fusion → cosine re-score → post-fusion boosts
|
||||
(backlink / salience / recency / graph signals / exact-match)
|
||||
│
|
||||
▼
|
||||
graph augment (typed-edge traversal from any seed)
|
||||
graph augment (optional two-pass structural expansion — walkDepth > 0)
|
||||
│
|
||||
▼
|
||||
reranker (zerank-2 cross-encoder, top 30 → reordered)
|
||||
deduplication (4-layer: per-page cap, Jaccard, type diversity)
|
||||
│
|
||||
▼
|
||||
token-budget enforcement (per mode bundle)
|
||||
reranker (zerank-2 cross-encoder — balanced/tokenmax; fail-open)
|
||||
│
|
||||
▼
|
||||
deduplication (same slug, different chunks → keep best)
|
||||
alias hop (exact alias match injects/boosts the canonical page)
|
||||
│
|
||||
▼
|
||||
evidence stamp → adaptive return (opt-in) → autocut (reranked modes)
|
||||
│
|
||||
▼
|
||||
limit slice → token-budget enforcement (per mode bundle)
|
||||
│
|
||||
▼
|
||||
results
|
||||
```
|
||||
|
||||
The stage order is pinned by `hybridSearch` in `src/core/search/hybrid.ts`:
|
||||
dedup runs BEFORE the reranker (so the reranker sees a diverse candidate pool,
|
||||
capped by its own `topNIn`), the alias hop runs AFTER the reranker (so a query
|
||||
that is a page's declared name reliably surfaces that page regardless of how
|
||||
the reranker scored body chunks), and the token budget is enforced last, on
|
||||
the final slice.
|
||||
|
||||
### Autocut: score-discontinuity result-sizing
|
||||
|
||||
Default-on for `balanced` and `tokenmax` (off for `conservative`, which has no
|
||||
reranker and therefore no trustworthy cliff signal). `applyAutocut`
|
||||
(`src/core/search/autocut.ts`) cuts the ranked set at the largest
|
||||
cross-encoder rerank-score cliff, before the limit slice, first page only.
|
||||
Never-empty failsafe (`minKeep`), no-op when fewer than 2 results carry a
|
||||
finite rerank score (covers the fail-open reranker path), and alias-hop exact
|
||||
matches are preserved through the cut. Knobs: per-call `SearchOpts.autocut` →
|
||||
`search.autocut` / `search.autocut_jump` config → mode bundle.
|
||||
|
||||
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
|
||||
|
||||
## How to verify on your own brain
|
||||
|
||||
@@ -19,18 +19,18 @@ need to understand both of them, or queries misroute silently.
|
||||
A **brain** is one database — PGLite file, self-hosted Postgres, or Supabase.
|
||||
Each brain has:
|
||||
- Its own `pages` table, `chunks` table, `embeddings`, etc.
|
||||
- Its own OAuth surface if served over HTTP MCP (v0.19+, PR 2).
|
||||
- Its own OAuth surface if served over HTTP MCP.
|
||||
- Its own separate lifecycle, backup, access control.
|
||||
|
||||
Brains are enumerated by:
|
||||
- **host** — your default brain, configured in `~/.gbrain/config.json`.
|
||||
- **mounts** — additional brains registered in `~/.gbrain/mounts.json` via
|
||||
`gbrain mounts add <id>` (v0.19+).
|
||||
`gbrain mounts add <id>`.
|
||||
|
||||
Routing: `--brain <id>`, `GBRAIN_BRAIN_ID`, `.gbrain-mount` dotfile, or
|
||||
longest-path match against registered mount paths. Falls back to `host`.
|
||||
|
||||
### Sources (the repo axis, v0.18.0+)
|
||||
### Sources (the repo axis)
|
||||
|
||||
A **source** is a named content repo *inside* one brain. Every `pages` row
|
||||
carries a `source_id`. Slugs are unique per source, not globally.
|
||||
@@ -142,7 +142,7 @@ Use this topology when:
|
||||
|
||||
You're senior enough to sit across multiple teams. You maintain your personal
|
||||
brain (with N sources inside) AND mount several work team brains. Each team
|
||||
brain is itself a multi-source brain in the v0.18.0 sense — organized
|
||||
brain is itself a multi-source brain — organized
|
||||
internally however the team owner chose.
|
||||
|
||||
```
|
||||
@@ -181,7 +181,7 @@ Use this topology when:
|
||||
- You need latent-space federation (agent decides when to query across
|
||||
brains), not SQL federation.
|
||||
|
||||
Cross-brain queries are **not deterministic** in v0.19. The agent sees the
|
||||
Cross-brain queries are **not deterministic**. The agent sees the
|
||||
brain list and re-queries as needed. That's the feature — it keeps debugging
|
||||
sane and access control clean.
|
||||
|
||||
@@ -202,6 +202,13 @@ WHICH BRAIN (DB)? WHICH SOURCE (repo in DB)?
|
||||
Both axes follow the same layered pattern on purpose. If you know one, you
|
||||
know the other.
|
||||
|
||||
One addition on the source axis for remote (MCP/OAuth) callers: a client
|
||||
registered with federated reads carries `ctx.auth.allowedSources` — an
|
||||
ARRAY of readable sources that takes precedence over the scalar
|
||||
`ctx.sourceId` on every read path (`sourceScopeOpts(ctx)` in the
|
||||
operations layer). Local CLI callers never set it; the scalar chain above
|
||||
is the whole story for them.
|
||||
|
||||
---
|
||||
|
||||
## For agents reading this
|
||||
@@ -236,7 +243,7 @@ know the other.
|
||||
|
||||
## Further reading
|
||||
|
||||
- v0.18.0 CHANGELOG — introduced `sources` primitive.
|
||||
- v0.19.0 CHANGELOG (TBD after PR 0+1+2 ship) — introduces `mounts`.
|
||||
- `docs/mounts/publishing-a-team-brain.md` (PR 2) — how to be the brain
|
||||
publisher, not just the subscriber.
|
||||
- [`topologies.md`](./topologies.md) — where the DB lives (operator recipes
|
||||
for each deployment shape).
|
||||
- `skills/conventions/brain-routing.md` — the agent-facing decision table.
|
||||
- `CHANGELOG.md` — release history for the `sources` and `mounts` primitives.
|
||||
|
||||
@@ -10,14 +10,16 @@
|
||||
> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80
|
||||
> during successive master merges — v0.37.0.0's autonomous-remediation wave
|
||||
> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79.
|
||||
> - **Follow-up minor** (forthcoming): falsifiability + category extraction at
|
||||
> `propose_takes`, SQL-side grade gate, per-category calibration scorecards,
|
||||
> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation
|
||||
> against the v0.36.1.0 fixtures.
|
||||
> - **Follow-up minor — NEVER IMPLEMENTED.** The falsifiability + category
|
||||
> extraction at `propose_takes`, SQL-side grade gate, per-category
|
||||
> calibration scorecards, and pg_trgm-based proposal dedup described in the
|
||||
> sections below remain UNSHIPPED design. Do not read §§1–4 as current
|
||||
> behavior; only the `unresolvable` hotfix above landed.
|
||||
>
|
||||
> Preserved here per the hotfix plan's PR #1191 close protocol so the
|
||||
> production context (96K-page brain, 6.8% falsifiability rate, category
|
||||
> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation.
|
||||
> production context (falsifiability rate + category breakdown observed on a
|
||||
> large real brain) doesn't get lost in the CHANGELOG → release-notes
|
||||
> condensation.
|
||||
|
||||
## Problem
|
||||
|
||||
|
||||
@@ -68,11 +68,13 @@ existing.
|
||||
## Migration shape
|
||||
|
||||
```ts
|
||||
// src/core/migrate.ts — append after the v80 entry
|
||||
// src/core/migrate.ts — append after the CURRENT last entry in the
|
||||
// MIGRATIONS array (take the next unused version number at implementation
|
||||
// time; the numbers below are placeholders, not a reserved slot)
|
||||
const migrations = [
|
||||
// ...existing v1-v80...
|
||||
// ...existing entries...
|
||||
{
|
||||
version: 81,
|
||||
version: NEXT_VERSION, // next unused number in the MIGRATIONS array
|
||||
name: 'frontmatter_scan_state',
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
|
||||
@@ -194,7 +196,7 @@ stale data as authoritative.
|
||||
```
|
||||
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
|
||||
Design lives at docs/architecture/frontmatter-scan-incremental.md.
|
||||
Schema migration v81 + sync-side UPSERT + incremental scan command
|
||||
New schema migration + sync-side UPSERT + incremental scan command
|
||||
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
|
||||
table + writes first; flip the reader one release later.
|
||||
```
|
||||
|
||||
@@ -1,105 +1,33 @@
|
||||
# GBrain Infrastructure Layer
|
||||
# GBrain Infrastructure Layer (orientation pointer)
|
||||
|
||||
The shared foundation that all skills, recipes, and integrations build on.
|
||||
This page is a router — the detailed, current-state references live in the
|
||||
docs below (this file once carried its own copies of the pipeline and schema;
|
||||
those rotted, so each concept now has exactly one home).
|
||||
|
||||
## Data Pipeline
|
||||
## Where things live
|
||||
|
||||
```
|
||||
INPUT (markdown files, git repo)
|
||||
↓
|
||||
FILE RESOLUTION (local → .redirect → .supabase → error)
|
||||
↓
|
||||
MARKDOWN PARSER (gray-matter frontmatter + body)
|
||||
→ compiled_truth + timeline separation
|
||||
↓
|
||||
CONTENT HASH (SHA-256 idempotency check — skip if unchanged)
|
||||
↓
|
||||
CHUNKING (3 strategies, configurable)
|
||||
├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy
|
||||
├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing
|
||||
└── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates
|
||||
↓
|
||||
EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions)
|
||||
→ batch 100, exponential backoff, non-fatal if fails
|
||||
↓
|
||||
DATABASE TRANSACTION (atomic: page + chunks + tags + version)
|
||||
↓
|
||||
SEARCH (hybrid, available immediately)
|
||||
```
|
||||
|
||||
## Search Architecture
|
||||
|
||||
GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search:
|
||||
|
||||
```
|
||||
User Query
|
||||
↓
|
||||
EXPANSION (optional: Claude Haiku generates 2 alternative phrasings)
|
||||
↓
|
||||
├── VECTOR SEARCH (pgvector HNSW, cosine distance)
|
||||
│ → 2x limit results per query variant
|
||||
│
|
||||
└── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank)
|
||||
→ 2x limit results
|
||||
↓
|
||||
RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly)
|
||||
↓
|
||||
4-LAYER DEDUP
|
||||
├── Best 3 chunks per page (source dedup)
|
||||
├── Jaccard similarity > 0.85 (text dedup)
|
||||
├── No type exceeds 60% (diversity)
|
||||
└── Max 2 chunks per page (page cap)
|
||||
↓
|
||||
TOP N RESULTS (default 20)
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) |
|
||||
| `src/core/postgres-engine.ts` | Postgres + pgvector implementation |
|
||||
| `src/core/import-file.ts` | importFromFile + importFromContent pipeline |
|
||||
| `src/core/sync.ts` | Git-based incremental change detection |
|
||||
| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing |
|
||||
| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff |
|
||||
| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) |
|
||||
| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection |
|
||||
| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking |
|
||||
| `src/core/search/hybrid.ts` | RRF merge of vector + keyword |
|
||||
| `src/core/search/dedup.ts` | 4-layer result deduplication |
|
||||
| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku |
|
||||
| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) |
|
||||
| `src/core/operations.ts` | Contract-first operation definitions (31 ops) |
|
||||
| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) |
|
||||
|
||||
## Schema Overview
|
||||
|
||||
10 tables in Postgres:
|
||||
|
||||
- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB)
|
||||
- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline)
|
||||
- **links** — typed edges (knows, works_at, invested_in, founded, etc.)
|
||||
- **tags** — many-to-many page tagging
|
||||
- **timeline_entries** — structured events (date, source, summary, detail)
|
||||
- **page_versions** — snapshot history for diff/revert
|
||||
- **raw_data** — sidecar JSON from external APIs (preserves provenance)
|
||||
- **files** — binary attachments in storage backend
|
||||
- **ingest_log** — audit trail of import operations
|
||||
- **config** — brain-level settings (version, embedding model, chunk strategy)
|
||||
|
||||
Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C).
|
||||
Vector search uses HNSW index with cosine distance on content_chunks.embedding.
|
||||
| Topic | Home |
|
||||
|---|---|
|
||||
| Ingest pipeline (file resolution → frontmatter parse → content-hash idempotency → chunking → embedding → atomic write) | per-file entries in [`KEY_FILES.md`](./KEY_FILES.md): `src/core/import-file.ts`, `src/core/sync.ts`, `src/core/markdown.ts`, `src/core/embedding.ts`, `src/core/chunkers/*` |
|
||||
| Chunking strategies (recursive / semantic / LLM-guided) | `src/core/chunkers/{recursive,semantic,llm}.ts` entries in [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Search pipeline (hybrid RRF, graph, reranker, autocut, dedup, budgets) | [`RETRIEVAL.md`](./RETRIEVAL.md) |
|
||||
| Search modes + cost knobs | `docs/guides/search-modes.md` + the CLAUDE.md Search Mode table |
|
||||
| Per-file index of `src/` (what each file does + its invariants) | [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Schema DDL | the `MIGRATIONS` array in `src/core/migrate.ts` (source of truth) + `src/schema.sql`; per-table classification in [`system-of-record.md`](./system-of-record.md) |
|
||||
| Engines (PGLite vs Postgres, parity rules) | `docs/ENGINES.md` + the engine entries in [`KEY_FILES.md`](./KEY_FILES.md) |
|
||||
| Operations contract (CLI + MCP generated from one source) | `src/core/operations.ts` (100+ operations; run `gbrain --tools-json` for the live list) |
|
||||
| Brains vs sources (which database vs which repo inside it) | [`brains-and-sources.md`](./brains-and-sources.md) |
|
||||
|
||||
## The Thin Harness Principle
|
||||
|
||||
GBrain is the deterministic layer. Skills and recipes are the latent space layer.
|
||||
GBrain is the deterministic layer. Skills and recipes are the latent-space layer.
|
||||
|
||||
See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full
|
||||
architecture philosophy.
|
||||
|
||||
- **GBrain CLI** = thin harness (same input → same output)
|
||||
- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills
|
||||
- **Skills** (the bundled set routed by `skills/RESOLVER.md`) = fat skills
|
||||
- **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure
|
||||
|
||||
The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Lens packs (v0.41.2.0)
|
||||
# Lens packs
|
||||
|
||||
Four bundled schema packs that turn the gbrain dream cycle into a multi-lens
|
||||
brain. Activate one with `gbrain config set schema_pack <name>` and the cycle
|
||||
@@ -7,7 +7,7 @@ picks up the pack's declared phases on the next `gbrain dream` run.
|
||||
## The four packs
|
||||
|
||||
```
|
||||
gbrain-base (shipped v0.38)
|
||||
gbrain-base
|
||||
▲
|
||||
│ extends
|
||||
┌──────────────┼──────────────────────┐
|
||||
@@ -60,37 +60,33 @@ conviction so high-stakes misses cost more).
|
||||
|
||||
### gbrain-engineer
|
||||
Bridge-only pack. Declares `learning` page type + reuses base `code`.
|
||||
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource
|
||||
(T8) watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
|
||||
No new cycle phases — the daemon-side `gstack-learnings` IngestionSource watches `~/.gstack/projects/{repo}/learnings.jsonl` and emits
|
||||
each JSONL line as a `learning` page when this pack is active. Three
|
||||
calibration domains: `architecture_calls` (scalar_brier),
|
||||
`effort_estimates` (weighted_brier), `risk_assessment` (scalar_brier).
|
||||
|
||||
Speculative ADR/postmortem/refactor_thesis/tech_debt types deferred
|
||||
to v0.42+ — they'll ship when a real user authors the first one (D8).
|
||||
Speculative ADR/postmortem/refactor_thesis/tech_debt types are
|
||||
deferred — they'll ship when a real user authors the first one.
|
||||
|
||||
### gbrain-everything
|
||||
Meta-pack stacking creator + investor + engineer via the v0.38
|
||||
Meta-pack stacking creator + investor + engineer via the
|
||||
`extends` + `borrow_from` chain. Single-active-pack constraint
|
||||
preserved — this IS the active pack; the registry walks extends +
|
||||
borrow to materialize the merged view.
|
||||
|
||||
**Merge contract (T20 / #1749).** `resolvePack` merges parent → child
|
||||
(child-wins) for the six ingest/query-shaping fields: `page_types`,
|
||||
`link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`,
|
||||
and `takes_kinds` (unioned — a child cannot narrow it). `phases` and
|
||||
`calibration_domains` are **NOT** inherited: they gate cycle execution,
|
||||
so each pack must declare its own participation explicitly. That is why
|
||||
`gbrain-everything` re-declares all its phases and all 7
|
||||
`calibration_domains` — inheritance does not carry them.
|
||||
**Merge contract.** The full `extends` + `borrow_from` merge rules live in
|
||||
[`schema-packs.md` § Merge contract](./schema-packs.md#merge-contract-extends--borrow_from).
|
||||
The one rule that matters here: `phases` and `calibration_domains` are
|
||||
**NOT** inherited (they gate cycle execution, so each pack must declare
|
||||
its own participation explicitly) — which is why `gbrain-everything`
|
||||
re-declares all its phases and all 7 `calibration_domains`.
|
||||
|
||||
Activate via `gbrain config set schema_pack gbrain-everything` and
|
||||
calibration_profile produces all 7 domain scorecards in one JSONB.
|
||||
|
||||
## Calibration profile widening (T10)
|
||||
## Calibration profile domains
|
||||
|
||||
Before v0.41.2.0, `calibration_profiles.domain_scorecards` was a
|
||||
`JSON.stringify({})` placeholder. v0.41.2.0 widens it: each declared
|
||||
Each declared
|
||||
domain produces a `{n, brier, accuracy, aggregator, page_types,
|
||||
extras}` entry. Four aggregator algorithms (closed enum):
|
||||
|
||||
@@ -109,20 +105,20 @@ Domain names are OPEN (third-party packs can declare new domain labels
|
||||
without a gbrain release). Aggregator algorithms are CLOSED (safe SQL
|
||||
stays in code, validated at pack-load).
|
||||
|
||||
## take_domain_assignments table (T1)
|
||||
## take_domain_assignments table
|
||||
|
||||
New JOIN table (migration v94):
|
||||
JOIN table (migration v94):
|
||||
`take_domain_assignments(take_id BIGINT FK, domain TEXT, pack TEXT,
|
||||
source TEXT, confidence REAL, assigned_at TIMESTAMPTZ, PK(take_id,
|
||||
domain))`. Multi-domain assignment honest — a take about "Sequoia's
|
||||
investment in Anthropic" can land in BOTH `deal_success` AND
|
||||
domain))`. Multi-domain assignment honest — a take about "fund-a's
|
||||
investment in acme-example" can land in BOTH `deal_success` AND
|
||||
`market_call` rather than being force-bucketed.
|
||||
|
||||
## What this enables for the user
|
||||
|
||||
- **Atoms + concepts ship in the binary.** Your OpenClaw's parallel
|
||||
atom-pipeline-coordinator + atom-backfill-coordinator + concept-
|
||||
synthesis crons can retire (T12 follow-up). One `gbrain dream` cron
|
||||
synthesis crons can retire. One `gbrain dream` cron
|
||||
covers everything.
|
||||
- **gstack learnings reach gbrain.** Engineer-pack-active brains
|
||||
surface every gstack-logged learning as a queryable page within
|
||||
@@ -131,22 +127,22 @@ investment in Anthropic" can land in BOTH `deal_success` AND
|
||||
often you're wrong on deals AND market calls AND architecture
|
||||
AND effort estimates in one `gbrain calibration --json` call.
|
||||
- **Lossless OpenClaw migration.** The `markdown-greenfield`
|
||||
importer (T7, mode='migration') re-ingests existing OpenClaw
|
||||
importer (mode='migration') re-ingests existing OpenClaw
|
||||
pages with permanent slug-keyed idempotency + per-row JSONL audit
|
||||
+ the `imported_from` marker so extract_atoms + synthesize_concepts
|
||||
don't re-extract already-atomized material.
|
||||
|
||||
## v0.41.2.1 follow-ups (filed in plan)
|
||||
## Known gaps / deferred follow-ups
|
||||
|
||||
- Per-page-type `frontmatter_validators` on PageTypeSchema so the
|
||||
atom_type enum (currently hardcoded in extract_atoms.ts) reads from
|
||||
the active pack manifest at runtime per D11.
|
||||
the active pack manifest at runtime.
|
||||
- 3-check quality gate (truism / punchline / entity-page reject) as
|
||||
a multi-pass extract_atoms refinement.
|
||||
- Embedding-similarity dedup in synthesize_concepts (currently
|
||||
exact-string concept ref match only).
|
||||
- Voice gate integration for T1 Canon narratives.
|
||||
- Voice gate integration for concept narratives.
|
||||
- op_checkpoint resumability for cross-cycle continuation in both
|
||||
phases.
|
||||
- Parity-baseline eval gates against your OpenClaw's existing 13K atoms
|
||||
+ 11K concepts on a 500-page sample subset.
|
||||
- Parity-baseline eval gates against a pre-existing downstream
|
||||
atom/concept corpus on a sample subset.
|
||||
|
||||
@@ -127,10 +127,10 @@ candidate ≠ the active pack name, loads the manifest via
|
||||
migration_from.version)`. Returns matching packs sorted by version
|
||||
descending.
|
||||
|
||||
v0.41.22 covers bundled packs only. v0.43+ TODO: enumerate user-installed
|
||||
packs at `~/.gbrain/schema-packs/*/pack.yaml` (defer to v0.43 since the
|
||||
filesystem-scan cost needs the cache invalidation strategy from
|
||||
`registry.ts`).
|
||||
Successor detection covers bundled packs only. Future work: enumerate
|
||||
user-installed packs at `~/.gbrain/schema-packs/*/pack.yaml` (deferred
|
||||
because the filesystem-scan cost needs the cache invalidation strategy
|
||||
from `registry.ts`).
|
||||
|
||||
## The manual_only apply policy
|
||||
|
||||
@@ -173,8 +173,8 @@ migration_from:
|
||||
version: "1.x"
|
||||
|
||||
page_types:
|
||||
# Inherit gbrain-base-v2's 15 types here (or use extends to merge
|
||||
# automatically once v0.43+ extends-chain composition lands)
|
||||
# Inherit gbrain-base-v2's 15 types here (or declare `extends:
|
||||
# gbrain-base-v2` and let the merge contract in schema-packs.md merge them)
|
||||
- { name: person, primitive: entity, path_prefixes: [people/], expert_routing: true }
|
||||
- { name: company, primitive: entity, path_prefixes: [companies/], expert_routing: true }
|
||||
# ... all 13 other v2 canonicals ...
|
||||
@@ -222,17 +222,17 @@ Every unify run writes to `~/.gbrain/audit/schema-unify-YYYY-Www.jsonl`
|
||||
identities (before + after), per-phase counts (would_apply + applied),
|
||||
warnings, completion timestamp. Privacy: page slugs are NOT logged in
|
||||
bulk (only the per-rule sample_slugs[≤10]); for forensic debugging
|
||||
add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
a `GBRAIN_AUDIT_FULL=1` escape hatch has been proposed but is not yet wired.
|
||||
|
||||
## What's NOT yet supported
|
||||
|
||||
- Subprocess sandbox for the publish-gate (v0.43+ TODO)
|
||||
- Subprocess sandbox for the publish-gate
|
||||
- Per-source pack-upgrade (the handler accepts `sourceId` but
|
||||
`findPackSuccessors` doesn't yet pass it through)
|
||||
- Cross-brain federated mounts that disagree on canonical packs
|
||||
- Automatic rollback (today: manual SQL or `gbrain restore`)
|
||||
- LLM-assisted mapping_rules codegen from production data (`gbrain
|
||||
schema detect-mappings`; deferred to v0.43+)
|
||||
- LLM-assisted mapping_rules codegen from production data (a proposed
|
||||
`gbrain schema detect-mappings`)
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -242,6 +242,6 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
- Onboard check: `src/core/onboard/checks.ts:checkPackUpgradeAvailable`
|
||||
- Render allowlist: `src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS`
|
||||
- Handler: `src/core/schema-pack/unify-types-handler.ts`
|
||||
- Migration: `src/core/migrate.ts:105` (slug_aliases table)
|
||||
- Migration: the `slug_aliases` entry in `src/core/migrate.ts`'s `MIGRATIONS` array
|
||||
- Type taxonomy doc: `docs/architecture/type-taxonomy.md`
|
||||
- Skill: `skills/schema-unify/SKILL.md`
|
||||
|
||||
@@ -7,20 +7,25 @@ paths, and which link verbs connect what to what. The schema pack is the
|
||||
querying, or routing experts. It is the single source of truth for
|
||||
"what's in your brain."
|
||||
|
||||
The v0.39.0.0 wave shipped a full schema-pack cathedral. This doc is the
|
||||
user-facing reference; for implementation details see
|
||||
`docs/designs/V038_SCHEMA_PACKS.md` (CEO plan) and the engine layer in
|
||||
`src/core/schema-pack/`.
|
||||
This doc is the user-facing reference; for implementation details see
|
||||
`docs/designs/V038_SCHEMA_PACKS.md` (the original design) and the engine
|
||||
layer in `src/core/schema-pack/`.
|
||||
|
||||
## What ships in the box
|
||||
|
||||
Two bundled packs:
|
||||
Seven bundled packs (`src/core/schema-pack/base/`):
|
||||
|
||||
- **`gbrain-base`** (default) — reproduces pre-v0.38 hardcoded behavior
|
||||
byte-for-byte. Existing brains see zero behavior change after upgrade.
|
||||
Covers: person, company, deal, meeting, project, place, concept, writing,
|
||||
analysis, guide, hardware, architecture, etc. (the original
|
||||
`ALL_PAGE_TYPES` list).
|
||||
- **`gbrain-base-v2`** — the 15-type canonical taxonomy. Fresh installs
|
||||
(`gbrain init`) activate this by default. See
|
||||
[`type-taxonomy.md`](./type-taxonomy.md) for the full type list and the
|
||||
upgrade path from `gbrain-base`.
|
||||
|
||||
- **`gbrain-base`** — the original hardcoded behavior, byte-for-byte
|
||||
(person, company, deal, meeting, project, place, concept, writing,
|
||||
analysis, guide, hardware, architecture, etc. — the original
|
||||
`ALL_PAGE_TYPES` list). Still the resolution-chain fallback (tier 7)
|
||||
for brains with no pack configured anywhere, so pre-existing brains see
|
||||
zero behavior change until they opt in to something newer.
|
||||
|
||||
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional
|
||||
directories described in `docs/GBRAIN_RECOMMENDED_SCHEMA.md`: deal,
|
||||
@@ -32,12 +37,17 @@ Two bundled packs:
|
||||
gbrain schema use gbrain-recommended
|
||||
```
|
||||
|
||||
- **`gbrain-creator`**, **`gbrain-investor`**, **`gbrain-engineer`**,
|
||||
**`gbrain-everything`** — the lens packs, which add cycle phases and
|
||||
calibration domains on top of the base taxonomy. See
|
||||
[`lens-packs.md`](./lens-packs.md).
|
||||
|
||||
Plus user-installed packs at `~/.gbrain/schema-packs/<name>/pack.yaml`
|
||||
that you author with `gbrain schema init` or `gbrain schema fork`.
|
||||
|
||||
## CLI surface
|
||||
|
||||
Five inspection verbs (shipped in v0.38):
|
||||
Inspection verbs:
|
||||
|
||||
```bash
|
||||
gbrain schema active # show resolved pack + which tier set it
|
||||
@@ -47,7 +57,7 @@ gbrain schema validate # validate a manifest's shape
|
||||
gbrain schema use <pack> # activate a pack (writes ~/.gbrain/config.json)
|
||||
```
|
||||
|
||||
Eight authoring + discovery verbs (shipped in v0.39):
|
||||
Authoring + discovery verbs:
|
||||
|
||||
```bash
|
||||
gbrain schema detect # propose types matching brain shape
|
||||
@@ -62,12 +72,12 @@ gbrain schema graph # ASCII type listing (experimental)
|
||||
gbrain schema lint # flag duplicates + missing prefixes
|
||||
gbrain schema explain <type> # plain-English type description (experimental)
|
||||
gbrain schema downgrade --to <p> # restore previous pack (recovery)
|
||||
gbrain schema usage --since 30d # per-verb invocation counts (D14 telemetry)
|
||||
gbrain schema usage --since 30d # per-verb invocation counts (telemetry)
|
||||
```
|
||||
|
||||
The verbs marked `experimental` are demand-gated per D14: their usage is
|
||||
tracked via T15's schema-events audit, and v0.40+ retro decides whether
|
||||
to deprecate any that stay <5% usage.
|
||||
The verbs marked `experimental` are demand-gated: usage is tracked via the
|
||||
schema-events audit (`gbrain schema usage`), which informs whether
|
||||
rarely-used verbs get deprecated.
|
||||
|
||||
## Resolution chain (7 tiers)
|
||||
|
||||
@@ -78,10 +88,10 @@ this chain top-down. First match wins.
|
||||
|------|--------|-------|
|
||||
| 1 | Per-call `schema_pack` opt | CLI only (`ctx.remote === false`); MCP rejected. |
|
||||
| 2 | `GBRAIN_SCHEMA_PACK` env | Process-scope override. |
|
||||
| 3 | Per-source DB config key `schema_pack:source:<id>` | New in v0.38. |
|
||||
| 3 | Per-source DB config key `schema_pack:source:<id>` | |
|
||||
| 4 | Brain-wide DB config key `schema_pack` | |
|
||||
| 5 | `gbrain.yml schema:` section | Repo-checked. |
|
||||
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` writes. |
|
||||
| 6 | `~/.gbrain/config.json` `schema_pack` field | What `gbrain schema use` (and `gbrain init`, which sets `gbrain-base-v2`) writes. |
|
||||
| 7 | Default: `gbrain-base` | Always present. |
|
||||
|
||||
## How the agent uses the active pack
|
||||
@@ -97,18 +107,18 @@ Every read + write path consults the active pack at runtime:
|
||||
- **`extract_facts`** runs only on `extractable: true` types.
|
||||
- **`enrichment-service`** routes person/company enrichment based on the
|
||||
pack's primitive declarations.
|
||||
- **Search hybrid cache** (`knobsHash`) folds in pack name + version
|
||||
(v0.39 T21). A cache row written under pack A is unreachable when pack
|
||||
- **Search hybrid cache** (`knobsHash`) folds in pack name + version.
|
||||
A cache row written under pack A is unreachable when pack
|
||||
B is active. Cross-pack contamination is structurally impossible.
|
||||
|
||||
## The magical moment (T2-T4 + T10)
|
||||
## The magical moment
|
||||
|
||||
Persona A (Notion refugee) installs gbrain, imports her exports, and the
|
||||
brain looks unfamiliar — the default `gbrain-base` pack expects
|
||||
`people/`, `companies/`, etc., but her files live under `Projects/`,
|
||||
`Reading/`, `Daily Notes/`. The friction signal fires in two places:
|
||||
|
||||
1. **Import warn (T7):** the end of `gbrain import` prints
|
||||
1. **Import warn:** the end of `gbrain import` prints
|
||||
`[schema] X of Y pages (Z%) have no type matching the active schema
|
||||
pack. Run gbrain schema detect to propose a pack matching your
|
||||
content shape.`
|
||||
@@ -124,7 +134,7 @@ gbrain schema review-candidates # human gate on promotion
|
||||
gbrain schema review-candidates --apply Projects/ # accept
|
||||
```
|
||||
|
||||
The agent (via the new EIIRP skill) automates phases 1-3 of this for any
|
||||
The agent (via the EIIRP skill, `skills/eiirp/SKILL.md`) automates phases 1-3 of this for any
|
||||
significant work session. The brain's schema becomes a living artifact
|
||||
the agent maintains, not a hardcoded ceremony the user authors.
|
||||
|
||||
@@ -172,9 +182,10 @@ filing_rules: []
|
||||
|
||||
## Merge contract (`extends` + `borrow_from`)
|
||||
|
||||
This section is the single home for the merge rules (other docs link here).
|
||||
`resolvePack` composes a pack against its `extends` chain (and any
|
||||
`borrow_from` targets) into the `resolved.manifest` every consumer reads
|
||||
(T20 / #1749). The rules:
|
||||
`borrow_from` targets) into the `resolved.manifest` every consumer reads.
|
||||
The rules:
|
||||
|
||||
- **Six fields inherit, child-wins:** `page_types`, `link_types`,
|
||||
`frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`.
|
||||
@@ -200,9 +211,8 @@ filing_rules: []
|
||||
|
||||
## Recovery + revert
|
||||
|
||||
The single-PR cathedral is hard to revert atomically. Per codex finding
|
||||
#4 from plan-eng-review, T20 ships `gbrain schema downgrade` to restore
|
||||
the active-pack config field:
|
||||
A pack activation is config, not code, so reverting code alone doesn't
|
||||
undo it. `gbrain schema downgrade` restores the active-pack config field:
|
||||
|
||||
```bash
|
||||
gbrain schema downgrade --to gbrain-base
|
||||
@@ -214,19 +224,19 @@ gbrain schema downgrade
|
||||
|
||||
1. `git revert <merge-commit>` — restores the code.
|
||||
2. `gbrain schema downgrade --to gbrain-base` — restores config.
|
||||
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
|
||||
v0.39-typed pages that no longer have a matching type in the active
|
||||
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — hard-deletes
|
||||
soft-deleted pages that no longer have a matching type in the active
|
||||
pack.
|
||||
|
||||
The cache + eval rows that pack-aware code wrote are isolated by the
|
||||
`knobsHash` pack-folding (T21) — they become unreachable under the
|
||||
`knobsHash` pack-folding — they become unreachable under the
|
||||
restored pack so no eviction is needed.
|
||||
|
||||
## Distribution
|
||||
|
||||
`.gbrain-schema` tarballs ride the same v0.37 skillpack pipeline as
|
||||
`.gbrain-skillpack` tarballs (T14 artifact abstraction). The
|
||||
discriminator is `api_version` in the manifest:
|
||||
`.gbrain-schema` tarballs ride the same distribution pipeline as
|
||||
`.gbrain-skillpack` tarballs. The discriminator is `api_version` in the
|
||||
manifest:
|
||||
|
||||
- `gbrain-schema-pack-v1` → schemapack
|
||||
- `gbrain-skillpack-v1` → skillpack
|
||||
@@ -237,22 +247,17 @@ respectively.
|
||||
|
||||
Publication to the public registries (`garrytan/gbrain-schema-registry`,
|
||||
`garrytan/gbrain-skillpack-registry`) follows the same publish-as-PR
|
||||
workflow as v0.37 skillpack publishing.
|
||||
workflow as skillpack publishing.
|
||||
|
||||
## What's deferred to v0.40+
|
||||
## Known limits / deferred work
|
||||
|
||||
- **Per-source pack federation across mounts.** A query crossing multiple
|
||||
sources currently rejects with `permission_denied` when those sources
|
||||
have divergent active packs (T19 + codex finding #2). The v0.40+ work
|
||||
computes a true per-source closure via the existing
|
||||
`buildSourceClosureCte` engine surface.
|
||||
- **`extends` chain semver compatibility checks** between pack versions.
|
||||
- **`skillpack ↔ schemapack` cross-reference declarations** — a skillpack
|
||||
can declare "I work best with these primitives present in your pack."
|
||||
- **Live schema migration helpers** — when you add a type, auto-suggest
|
||||
backfill of existing pages.
|
||||
- **Authoring vs derivation thesis reframe (D14).** v0.39.0.0 ships the
|
||||
full 11-verb cathedral with 6 verbs marked experimental-tier. v0.40+
|
||||
retro reads T23 usage telemetry to decide which to deprecate.
|
||||
sources rejects with `permission_denied` when those sources have
|
||||
divergent active packs (`src/core/schema-pack/op-trust-gate.ts`). A true
|
||||
per-source closure via the existing `buildSourceClosureCte` engine
|
||||
surface remains future work.
|
||||
- **Pack version upgrades** (e.g. `gbrain-base` → `gbrain-base-v2`) are
|
||||
handled by the successor-detection + unify-types mechanism — see
|
||||
[`pack-upgrade-mechanism.md`](./pack-upgrade-mechanism.md).
|
||||
|
||||
See `TODOS.md` v0.40+ section for the full deferred list.
|
||||
The live deferred list is in `TODOS.md`.
|
||||
|
||||
@@ -52,3 +52,13 @@ gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
|
||||
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
|
||||
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
|
||||
|
||||
The manual diagnosis above has an automated cousin: the progress-aware stall
|
||||
watchdog. If the import drain makes no forward progress for
|
||||
`GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900; keyed on file-import
|
||||
progress, not the lock heartbeat), the run aborts with
|
||||
`reason: 'stall_timeout'` and releases the per-source lock so the next
|
||||
`gbrain sync` resumes from the checkpoint. It fires BETWEEN files — a hang
|
||||
inside one file's import runs until the wall-clock hard deadline. `0`
|
||||
disables it. The full sync-resumability knob table lives in CLAUDE.md
|
||||
("Sync resumability + lock tuning").
|
||||
|
||||
@@ -88,6 +88,7 @@ the repo. The architectural rule still holds — these aren't
|
||||
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
|
||||
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
|
||||
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
|
||||
| `op_checkpoint_paths` | Sync-resume checkpoint. Append-only progress banking; a completed sync makes it irrelevant. |
|
||||
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
|
||||
|
||||
A new derived table that holds user-knowledge MUST land FS-first.
|
||||
@@ -189,9 +190,6 @@ reconciler / migration layer without the explicit allow-list comment.
|
||||
|
||||
## Related
|
||||
|
||||
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
|
||||
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
|
||||
and round 2 finds)
|
||||
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
|
||||
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
|
||||
- `scripts/check-system-of-record.sh` — the CI gate that enforces
|
||||
|
||||
@@ -3,68 +3,63 @@
|
||||
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
|
||||
only; release history lives in `CHANGELOG.md` + git.
|
||||
|
||||
`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local
|
||||
brain content, just an OAuth client pointing at a remote `gbrain serve --http`.
|
||||
v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25
|
||||
silently fell through to `connectEngine()` and opened the empty local PGLite,
|
||||
returning "No results." against a populated remote brain. v0.31.1 fixes the
|
||||
silent-empty-results bug class for every operation surface.
|
||||
`gbrain init --mcp-only` sets up a thin-client install: no local brain content,
|
||||
just an OAuth client pointing at a remote `gbrain serve --http`. Every operation
|
||||
surface routes through the remote brain — a thin-client install never opens the
|
||||
empty local PGLite, so a populated remote brain can't silently return
|
||||
"No results." Local-only commands refuse with a pinpoint hint instead of
|
||||
falling through.
|
||||
|
||||
Key files:
|
||||
Key files (per-file detail lives in each file's `KEY_FILES.md` entry; this doc
|
||||
carries the routing-seam picture):
|
||||
|
||||
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no
|
||||
- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (no
|
||||
parallel `src/core/thin-client/` module; routing is a ~80-line conditional
|
||||
in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine`
|
||||
so thin-client installs never open the empty PGLite. localOnly ops on
|
||||
thin-client refuse via `refuseThinClient` (with pinpoint hint table
|
||||
`THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort`
|
||||
`THIN_CLIENT_REFUSE_HINTS`, which covers the full DB-bound command surface —
|
||||
sync, embed, extract, migrate, enrich, dream, jobs, sources, pages, files,
|
||||
eval, code-*, and more). Banner via `printIdentityBannerBestEffort`
|
||||
before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`,
|
||||
non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason`
|
||||
for canned, actionable error messages. ENG-2 renderer parity: local-engine
|
||||
for canned, actionable error messages. Renderer parity: the local-engine
|
||||
path runs `JSON.parse(JSON.stringify(result))` so renderers see the same
|
||||
shape on both paths (kills Date/bigint/Buffer drift class).
|
||||
- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`.
|
||||
Hardened in v0.31.1 (CDX-4): all transport errors normalized to
|
||||
`RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions
|
||||
{timeoutMs, signal}`; `buildAbortController` composes external signal with
|
||||
timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind`
|
||||
('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code`
|
||||
field carrying server-supplied error codes (e.g. `missing_scope`).
|
||||
`extractToolErrorCode` parses JSON envelopes first, falls back to substring
|
||||
detection for legacy server messages. `unpackToolResult<T>(res)` unchanged
|
||||
(parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape.
|
||||
- `src/core/cli-options.ts` — `parseGlobalFlags` adds `--timeout=Ns` (accepts
|
||||
`30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s
|
||||
for most ops, 180s for `think`). `parseTimeout(s)` exported helper.
|
||||
- `src/core/doctor-remote.ts` — `gbrain remote doctor` adds the
|
||||
`oauth_client_scopes_probe` check (CDX-5). Probes the read tier via
|
||||
`get_brain_identity` and admin tier via `get_health`; reports per-tier
|
||||
shape on both paths (kills the Date/bigint/Buffer drift class).
|
||||
- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`,
|
||||
the transport under the routing seam. All transport errors normalize to
|
||||
`RemoteMcpError` via the `toRemoteMcpError` funnel, with a stable
|
||||
`RemoteMcpErrorReason` union the dispatcher's `never` switch keys off.
|
||||
Full symbol-level detail: the `src/core/mcp-client.ts` entry in
|
||||
[`KEY_FILES.md`](./KEY_FILES.md).
|
||||
- `src/core/cli-options.ts` — `parseGlobalFlags` supports `--timeout=Ns`
|
||||
(accepts `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command
|
||||
default (30s for most ops, 180s for `think`). `parseTimeout(s)` exported
|
||||
helper.
|
||||
- `src/core/doctor-remote.ts` — `gbrain remote doctor` includes the
|
||||
`oauth_client_scopes_probe` check. Probes the read tier via
|
||||
`get_brain_identity` and the admin tier via `get_health`; reports per-tier
|
||||
status with pinpoint remediation when admin is missing. `buildScopeCheck`
|
||||
+ `ScopeProbeResult` exported for test access. Skippable via
|
||||
`GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC
|
||||
initialize level only (MCP SDK Client hangs on shape mismatch).
|
||||
- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
|
||||
- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries.
|
||||
- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller).
|
||||
- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality` → `'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error.
|
||||
- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).
|
||||
- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex).
|
||||
- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed.
|
||||
- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on).
|
||||
- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command).
|
||||
- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows.
|
||||
- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)` — `date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query.
|
||||
- `src/core/operations.ts` — `get_brain_identity` op (read scope, no params,
|
||||
banner-only): cheap counter packet `{version, engine, page_count,
|
||||
chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses
|
||||
`engine.getStats()`; banner's 60s client-side TTL bounds frequency to
|
||||
≤1/60s per CLI process (well below the Fly.io health-check cadence that
|
||||
motivated the original `getStats` cost warning).
|
||||
`engine.getStats()`; the banner's 60s client-side TTL bounds frequency to
|
||||
≤1/60s per CLI process.
|
||||
- `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command
|
||||
thin-client routing branches. These commands bypass the operation-layer
|
||||
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
|
||||
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
|
||||
to op params. `think` is a special case: the server's `think` op
|
||||
intentionally disables `--save`/`--take` for remote callers
|
||||
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
|
||||
loudly when those flags are set.
|
||||
to op params. `think` is a special case: the server's `think` op is
|
||||
read-scoped for OAuth/MCP and intentionally disables `--save`/`--take` for
|
||||
remote callers (the `safeSave`/`safeTake` trust-boundary gate in the `think`
|
||||
handler in `operations.ts`); thin-client `think` warns loudly when those
|
||||
flags are set.
|
||||
|
||||
Cross-modal search files (image query, SSRF-guarded image loading, spend
|
||||
tracking, multimodal reindex) are indexed per-file in
|
||||
[`KEY_FILES.md`](./KEY_FILES.md) and described behaviorally in
|
||||
[`RETRIEVAL.md`](./RETRIEVAL.md) — they are not part of the thin-client
|
||||
routing seam.
|
||||
|
||||
@@ -108,11 +108,13 @@ instead of a local DB connection:
|
||||
}
|
||||
```
|
||||
|
||||
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
|
||||
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
|
||||
`integrity`, `serve`) on a thin-client install with a clear error pointing
|
||||
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
|
||||
(OAuth discovery, token round-trip, MCP smoke).
|
||||
The CLI dispatch guard refuses every DB-bound command (`sync`, `embed`,
|
||||
`extract`, `migrate`, `serve`, `enrich`, `jobs`, `sources`, `pages`,
|
||||
`files`, `eval`, and the rest of the local-only surface — the full hint
|
||||
table is `THIN_CLIENT_REFUSE_HINTS` in `src/cli.ts`) on a thin-client
|
||||
install with a clear error pointing at the remote host. `gbrain doctor`
|
||||
runs a dedicated thin-client check set (OAuth discovery, token round-trip,
|
||||
MCP smoke). See [`thin-client.md`](./thin-client.md) for the routing seam.
|
||||
|
||||
### Setup
|
||||
|
||||
@@ -394,6 +396,9 @@ simultaneously — that's by design.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/guides/bootstrap.md` — `gbrain bootstrap`, the paved-road paste-in
|
||||
install for Topology 1 with a desktop coding agent (interview, hooks,
|
||||
MCP registration, verify).
|
||||
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
|
||||
vs sources axes).
|
||||
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Type Taxonomy (v0.41.22: gbrain-base-v2)
|
||||
# Type Taxonomy (gbrain-base-v2)
|
||||
|
||||
> The 14-canonical-type DRY/MECE taxonomy shipped in v0.41.22. Predecessor
|
||||
> `gbrain-base` (24 types) stays bundled for back-compat; v0.42+ installs
|
||||
> The 14-canonical-type DRY/MECE taxonomy. Predecessor
|
||||
> `gbrain-base` (24 types) stays bundled for back-compat; fresh installs
|
||||
> default to `gbrain-base-v2`.
|
||||
|
||||
## Why
|
||||
@@ -79,7 +79,7 @@ gbrain jobs submit unify-types \ # PROTECTED + manual_only
|
||||
--params '{"target_pack":"gbrain-base-v2","apply":true}'
|
||||
# omit "apply":true → dry-run (default)
|
||||
↓
|
||||
Handler runs 4 phases:
|
||||
Handler runs 8 phases:
|
||||
┌─────────────────────────────────────┐
|
||||
│ Phase 1: Preflight + lock │ → gbrain-unify db-lock (60min TTL)
|
||||
├─────────────────────────────────────┤
|
||||
@@ -165,8 +165,10 @@ explicitly disambiguated this as canonical, so it should outrank fuzzy
|
||||
matches that hit aliases by accident."
|
||||
|
||||
`SearchResult.alias_resolved_boost` is stamped on touched results for
|
||||
`--explain` formatter visibility. KNOBS_HASH_VERSION bumped 5→6 to
|
||||
invalidate pre-v0.42 cache rows that don't reflect the new stage.
|
||||
`--explain` formatter visibility. The stage participates in the search
|
||||
cache key (`KNOBS_HASH_VERSION` in `src/core/search/mode.ts` is the
|
||||
single source of truth for the current cache-key version), so cache rows
|
||||
written before the stage existed are unreachable.
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -176,4 +178,3 @@ invalidate pre-v0.42 cache rows that don't reflect the new stage.
|
||||
- Migration handler: `src/core/schema-pack/unify-types-handler.ts`
|
||||
- Onboard checks: `src/core/onboard/checks.ts`
|
||||
- Skill: `skills/schema-unify/SKILL.md`
|
||||
- Plan + decisions: `~/.claude/plans/system-instruction-you-are-working-transient-elephant.md`
|
||||
|
||||
+21
-19
@@ -1,4 +1,4 @@
|
||||
# gbrain eval suspected-contradictions (v0.32.6)
|
||||
# gbrain eval suspected-contradictions
|
||||
|
||||
The contradiction probe samples retrieval results, asks an LLM judge whether
|
||||
any pair contradicts on a factual claim relevant to the user's query, and
|
||||
@@ -142,25 +142,27 @@ pay near-zero on re-runs (until you bump PROMPT_VERSION).
|
||||
gate makes accidental private-data commits hard, but the operator MUST
|
||||
inspect every redaction before commit.
|
||||
|
||||
## Temporal axis
|
||||
|
||||
The judge distinguishes real contradictions from legitimate change-over-time.
|
||||
The verdict enum has six members (`no_contradiction | contradiction |
|
||||
temporal_supersession | temporal_regression | temporal_evolution |
|
||||
negation_artifact`), and `pages.effective_date` is threaded into the judge
|
||||
prompt so the probe doesn't cry wolf on facts that simply changed.
|
||||
|
||||
The trajectory substrate builds on the same signal:
|
||||
`gbrain eval trajectory <entity>` shows the chronological typed-claim
|
||||
history with regressions flagged inline; `gbrain founder scorecard
|
||||
<entity>` rolls up four signals (accuracy, consistency, growth
|
||||
direction, red flags) into a stable JSON contract. MCP op
|
||||
`find_trajectory` (read scope, visibility-filtered for remote callers)
|
||||
exposes the same data to agents. The probe's `temporal_supersession`
|
||||
verdict and the consolidate phase's `valid_until` writeback both
|
||||
preserve the `auto-supersession.ts` "NEVER auto-applies" invariant
|
||||
— the probe only emits paste-ready commands; only `consolidate`
|
||||
writes `valid_until` (a grep guard pins this).
|
||||
|
||||
## See also
|
||||
|
||||
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
|
||||
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
|
||||
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
|
||||
+ trend-tracking workflow.
|
||||
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
|
||||
six-member verdict enum (`no_contradiction | contradiction |
|
||||
temporal_supersession | temporal_regression | temporal_evolution |
|
||||
negation_artifact`) and threaded `pages.effective_date` into the judge
|
||||
prompt so the probe stops crying wolf on legitimate change-over-time.
|
||||
v0.35.7 lands the trajectory substrate the probe pointed at:
|
||||
`gbrain eval trajectory <entity>` shows the chronological typed-claim
|
||||
history with regressions flagged inline; `gbrain founder scorecard
|
||||
<entity>` rolls up four signals (accuracy, consistency, growth
|
||||
direction, red flags) into a stable JSON contract. MCP op
|
||||
`find_trajectory` (read scope, visibility-filtered for remote callers)
|
||||
exposes the same data to agents. The probe's `temporal_supersession`
|
||||
verdict and the consolidate phase's `valid_until` writeback both
|
||||
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
|
||||
— the probe still emits paste-ready commands, only `consolidate`
|
||||
writes `valid_until` (R1+R8 grep guard pins this).
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# Agent Bootstrap — Product Design (normative for scope & sequencing)
|
||||
|
||||
**Status:** APPROVED (product review, 3 adversarial rounds; CEO review; eng review —
|
||||
0 unresolved decisions). Implementation source of truth:
|
||||
[AGENT_BOOTSTRAP_PLAN.md](AGENT_BOOTSTRAP_PLAN.md).
|
||||
|
||||
## Problem
|
||||
|
||||
Agent platforms (OpenClaw, Hermes) deliver the full personal-agent experience —
|
||||
scheduled work, skill files, SOUL.md identity, persistent memory — but setting one up
|
||||
means deploying a server, wiring API keys, and paying raw token prices. Meanwhile
|
||||
nearly everyone already runs Claude Code or Codex, both subsidized by subscriptions,
|
||||
both capable of executing an install protocol themselves.
|
||||
|
||||
**The feature:** a paste-in install that adds memory + skills + identity +
|
||||
session-triggered schedules + knowledge persistence to a private GitHub repo,
|
||||
mimicking the agent-platform experience on the desktop apps people already have.
|
||||
"Just use your local harness as your agent."
|
||||
|
||||
## Target surfaces & order
|
||||
|
||||
| Surface | In v1? | Order | Per-turn context seam | Persistence (write) seam |
|
||||
|---|---|---|---|---|
|
||||
| Codex in ChatGPT desktop | YES | 1st | spike exit question #2 | spike exit question #1 (blocking); fallback: mandated MCP writes + end-of-session sweep |
|
||||
| Codex CLI | YES | with 1st | AGENTS.md pull protocol + `volunteer_context` | mandated MCP write ops |
|
||||
| Claude Code desktop | YES | 2nd | hooks: SessionStart / UserPromptSubmit via serve IPC | SessionEnd hook: transcript ingest + scan-gated push |
|
||||
| Claude Code CLI | YES | with 2nd | same hooks | same hooks |
|
||||
|
||||
The wedge is NOT platform feature-parity on day one. It is: paste → interview → an
|
||||
agent that knows who it is and who you are → recalls what you told it in the next
|
||||
session → visibly compounds during week one.
|
||||
|
||||
## The repo format (the contract)
|
||||
|
||||
A private GitHub repo is the product artifact — the agent's portable body:
|
||||
|
||||
- **Manifest:** `agent.json` (`format_version: 1`, provisional; `initialized`
|
||||
sentinel distinguishes a template clone from a bootstrapped workspace).
|
||||
- **Identity:** SOUL.md, USER.md, MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md,
|
||||
ACCESS_POLICY.md, GITHUB.md — rendered ONLY from interview answers, never invented.
|
||||
- **Content:** `brain/` (the gbrain source), `memory/` (daily notes), `skills/`,
|
||||
`state/` (committed: interview.json, portable mcp.json; local-only files gitignored).
|
||||
- **Compatibility promise:** hosted gbrain mounts `format_version: 1` repos natively;
|
||||
compatibility is a test against this spec.
|
||||
|
||||
## Premises (all settled)
|
||||
|
||||
1. **Free desktop tier of the hosted ladder** — the desktop ceiling (laptop asleep =
|
||||
agent asleep; data outgrows the disk) is the graduation mechanic to hosted gbrain.
|
||||
2. **One agent-body format, two doors.** Portability up the ladder is moat #1.
|
||||
3. **Awake-when-you-are is the honest desktop contract.** Session-triggered schedules
|
||||
(jobs run at turn/session boundaries while the harness is open); true 24/7 crons
|
||||
are hosted-tier, stated in-product.
|
||||
4. **Day-one-empty-brain is the #1 churn risk.** Magic moment with zero corpus
|
||||
(interview → next-session recall) + fast ramp (file import; connector ingest v1.1).
|
||||
5. **Wire-level truth before build:** a clean-machine spike gates door 1 (write-seam
|
||||
pass bar: 0 durable-write failures in 20 sessions over ≥3 days, else extend to 50).
|
||||
6. **The graph is moat #2:** v1 exercises entity extraction, backlinks, and
|
||||
graph-aware recall; verify enforces a graph floor.
|
||||
7. **Keyless mode is first-class:** the harness agent IS the subsidized LLM — with
|
||||
zero API keys, memory is agent-authored through write ops, search is keyword-only,
|
||||
and the magic moment still passes. One optional key unlocks embeddings +
|
||||
auto-extraction.
|
||||
|
||||
## Build order (one cathedral PR; size trip-wire: split at build order 2 if PR is
|
||||
open >10 days from first code commit)
|
||||
|
||||
0. **Spike + quota gate** (manual, gates door-1 ship; per-harness quota measured; a
|
||||
p90 day must fit ≤10% of weekly subscription quota or schedule scope is cut).
|
||||
1. **Shared body + engine machinery:** `gbrain bootstrap` family, templates, format
|
||||
spec, secret-scan-gated persistence, verify, uninstall (v1 via CEO-review
|
||||
expansion; receipt-keyed scope per the PLAN's CX2-12).
|
||||
2. **Codex door ships first** (runbook variant + approvals preflight + capability
|
||||
probe; CLI path not spike-gated).
|
||||
3. **Claude Code door:** hooks, IPC turn_context, transcript ingestion, greeting
|
||||
digest, schedule mechanism.
|
||||
4. **Graduation seam, desktop half:** format spec + documented upgrade path (advisor
|
||||
nudge ships with the hosted mount in v1.1).
|
||||
|
||||
## Out of scope for v1
|
||||
|
||||
| Deferred | Lands | Why |
|
||||
|---|---|---|
|
||||
| Connector-driven ingest (email/calendar) | v1.1 (keyed to probe) | unverified host capability |
|
||||
| Hosted mount + "outgrowing this laptop" nudge | v1.1 together | never point at a destination that can't accept the repo |
|
||||
| `serve --attach` (simultaneous multi-harness) | fast-follow | v1 documents one-live-serve politely |
|
||||
| Codex `notify` transcript sweeper | fast-follow | mandated MCP writes cover v1 |
|
||||
| Windows (named-pipe IPC) | deferred | v1 = macOS + Linux |
|
||||
| Networked Docker paste-flow e2e | fast-follow | offline container e2e covers 80% at 20% of the flake |
|
||||
| `gbrain quota` meter command | TODOS | measurement ships as script+doc; productize when per-harness token counting is proven |
|
||||
| True 24/7 crons on desktop | never | hosted-tier by design |
|
||||
|
||||
## Success criteria
|
||||
|
||||
- **TTFM ≤15 min** paste→verified install, excluding first-run toolchain downloads
|
||||
(published separately); every human action counted.
|
||||
- **Magic moment, deterministically:** verify asserts an interview fact is retrievable
|
||||
through the agent's own MCP path; fresh-session end-to-end is a scripted human
|
||||
confirmation. Must pass keyless.
|
||||
- **Pilot tests:** a non-developer pilot completes the door-1 install unaided and
|
||||
still uses it in week two; a developer completes the door-2 README install ≤10 min
|
||||
with a week-two recall check.
|
||||
- **Graph floor:** ≥1 entity extracted, ≥1 backlink resolved, one edge-only query
|
||||
answered — via the real MCP write path.
|
||||
- **Ladder proof:** a desktop-grown repo validates against `format_version: 1`.
|
||||
- **Honesty checks:** quota number published; desktop contract stated in-product;
|
||||
door 1 demotes to documented-beta on its trigger rather than shipping flaky.
|
||||
|
||||
## Distribution
|
||||
|
||||
Paste block + tag-pinned runbook (`BOOTSTRAP_FOR_AGENTS.md`, fetched at the
|
||||
`latest-stable` ref — advanced by the release job only after assets publish, so
|
||||
published copies never rot); optional GitHub template repo (generated at release from
|
||||
the same renderer); binary via `bun install -g github:garrytan/gbrain#latest-stable`
|
||||
(never npm). The paste block lives in the README's `## Install` section, as
|
||||
per-harness subsections ordered "For Codex — the recommended first step" → "For
|
||||
Claude Code" → "For OpenClaw or Hermes" (the 2026-08-09 ordering decision, recorded
|
||||
in the PLAN's artifact table). `INSTALL_FOR_AGENTS.md` remains the paste path for
|
||||
agent platforms and lives inside the OpenClaw/Hermes subsection.
|
||||
|
||||
## Threat model (v1 summary)
|
||||
|
||||
Tag-pinned fetch + version-stamp skew check + runbook phase allowlist; secret-scan
|
||||
gates every commit AND corpus write (loud block, per-finding override); repo privacy
|
||||
verified via API after create; hooks in gitignored local settings with a kill switch;
|
||||
interview answers rendered as fenced data (never instructions) with escaping and
|
||||
caps; uninstall keyed to a machine-local receipt, never deletes a brain it didn't
|
||||
create; provider-policy drift acknowledged as residual risk — posture: measured
|
||||
sustainable load, no absent-user background burn, portable body as the exit plan.
|
||||
Full posture: `docs/guides/bootstrap.md`.
|
||||
@@ -0,0 +1,820 @@
|
||||
# Agent Bootstrap — Implementation Plan (normative)
|
||||
|
||||
The engineering source of truth for `gbrain bootstrap` — the paste-in install that
|
||||
turns Claude Code / Codex desktop apps into a persistent personal agent (identity +
|
||||
memory + skills + session-triggered schedules + private-repo persistence + local
|
||||
PGLite brain). Product scope/sequencing source of truth:
|
||||
[AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md). Where they disagree, the
|
||||
design doc wins on scope; this doc wins on implementation.
|
||||
|
||||
Reviews absorbed: 3-designer panel + adversarial critique; product design review (3
|
||||
rounds); CEO review (SELECTIVE EXPANSION, ~55 findings); outside voice x2 (37
|
||||
findings, 35 accepted); eng review (32 findings). All accepted fixes are inlined
|
||||
below with their finding IDs. 0 unresolved decisions.
|
||||
|
||||
---
|
||||
|
||||
## As-shipped deltas (read this first — where the code moved after the plan froze)
|
||||
|
||||
This plan is layered: later absorption sections (the post-design-review deltas, the
|
||||
CX2 series) override earlier prose, and THIS section overrides everything below it.
|
||||
The shipped implementation matches the plan except for these deltas:
|
||||
|
||||
1. **Verify runs LAST, not before host registration.** [CX2-5]'s determinism goal
|
||||
survived, but the shipped phase order (single TS source:
|
||||
`src/core/bootstrap/status.ts` `PHASES`) is
|
||||
preflight → engine → interview → render → skills → wire → repo → **verify**,
|
||||
and verify runs in-process on the caller-held engine, calling
|
||||
`runMaintenanceSweep` directly — no transient serve. It works pre-registration
|
||||
AND as the weekly re-run (`src/core/bootstrap/verify.ts`).
|
||||
2. **Uninstall scope: [CX2-12] wins over the CEO-expansion bullet.** `~/.gbrain` is
|
||||
NEVER deleted wholesale — only receipt-enumerated bootstrap-created state
|
||||
(`src/core/bootstrap/uninstall.ts`).
|
||||
3. **Module naming/layout:** `private-repo.ts` shipped as `repo.ts`; additional
|
||||
shipped modules the artifact table doesn't list: `attach.ts, assets.ts,
|
||||
format.ts, host-specs.ts, hooks.ts, lock.ts, status.ts, template-repo.ts,
|
||||
uninstall.ts`.
|
||||
4. **Templates layout:** all bootstrap templates live under `templates/bootstrap/`
|
||||
(not at `templates/` root).
|
||||
5. **Test filenames:** `test/hook-command.serial.test.ts` and
|
||||
`test/e2e/bootstrap-*.serial.test.ts` — the `.serial` variants the plan's own
|
||||
[A7] mandated; the artifact table predates that.
|
||||
6. **README ordering:** the D5 placement was superseded by the 2026-08-09 user
|
||||
decision — per-harness `## Install` sections ordered Codex → Claude Code →
|
||||
OpenClaw/Hermes, with `INSTALL_FOR_AGENTS.md` living inside the OpenClaw/Hermes
|
||||
section (annotated in the artifact table; the D5 prose at the bottom is stale).
|
||||
|
||||
## Post-design-review deltas (2026-08-07, /office-hours APPROVED — these override below)
|
||||
|
||||
Product: **"GBrain for Codex" + "GBrain for Claude Code"** (names contingent on
|
||||
trademark review) — the free desktop tier of the hosted-gbrain ladder (the hosted service is the
|
||||
graduation path; the desktop ceiling is the mechanic, not a bug). Target surfaces are
|
||||
the DESKTOP APPS; **Codex door ships first** (named pilot user is a non-developer
|
||||
ChatGPT-app user). CLIs come along via shared machinery.
|
||||
|
||||
**Build order (replaces the PR1/FF phasing below where they conflict):**
|
||||
0. **Spike (gates door-1 ship; starts immediately, before build):** clean-machine
|
||||
reproduction of Garry's working Codex-in-ChatGPT + gbrain prototype. Exit questions
|
||||
with pass bars: (#1 blocking) reliable per-turn/per-session WRITE trigger — 0
|
||||
durable-write failures in 20 sessions across ≥3 days on a non-founder machine, else
|
||||
extend to 50; causes logged. (#2) per-turn READ seam — context block present at turn
|
||||
start or degraded pull-mode documented. Also: folder access, MCP registration path,
|
||||
approval-tap count, connector availability (yes/no/degraded). PLUS in parallel: the
|
||||
**quota release gate** measured against Garry's production usage — pass: p90 day
|
||||
≤10% of weekly Max/Pro quota; failure = scope change to "session-agent, no schedules
|
||||
by default" BEFORE schedule build effort. Codex CLI is NOT spike-gated.
|
||||
1. **Shared body + engine machinery (PR1):** everything in this plan's PR1 (bootstrap
|
||||
family, gbrain hook + IPC turn_context, templates, question bank, secret scan,
|
||||
sources push, skillpack scaffold wiring, doctor checks) PLUS: `agent.json` repo
|
||||
manifest with `format_version: 1` (the ladder contract — the hosted gbrain service mounts this
|
||||
format; compatibility is a test, not a hope); wire EXISTING import + entity
|
||||
extraction + backlinks + graph-aware query into install/verify (funds design
|
||||
Premises 4/6 — new verify check: ≥1 entity extracted, ≥1 backlink resolved, one
|
||||
edge-only query answered); byte floors scaled to answered-question count (they catch
|
||||
skipped interviews, NOT invention — never pressure padding); commit this plan to
|
||||
docs/designs/AGENT_BOOTSTRAP_PLAN.md.
|
||||
2. **Codex door (ships first):** runbook variant for Codex desktop approvals/sandbox
|
||||
(seam per spike; fallback = mandated MCP writes via AGENTS.md gates + end-of-session
|
||||
sweep, shipped only with measured reliability) + Codex CLI path (AGENTS.md pull
|
||||
protocol; session-start gate includes "run due jobs"). Connector-ingest capability
|
||||
probe wired into install output; connector ingest itself is v1.1.
|
||||
3. **Claude Code door:** hooks per this plan (SessionStart/UserPromptSubmit/SessionEnd
|
||||
via serve IPC), MCP project-scope default, transcript ingestion. Includes the v1
|
||||
schedule mechanism: hook layer checks HEARTBEAT.md due-job list at session start +
|
||||
turn boundaries. If the spike finds no turn-boundary trigger on ChatGPT desktop,
|
||||
schedules are Claude-Code-only in v1 and per-surface copy says so.
|
||||
4. **Graduation seam, desktop half only:** format spec + documented upgrade path +
|
||||
honest-contract copy. The "outgrowing this laptop" advisor nudge moves to v1.1
|
||||
WITH the hosted mount (never point at a destination that can't accept the repo).
|
||||
|
||||
**New v1 requirements from the design review (fold into implementation):**
|
||||
- Binary install is tag-pinned: `bun install -g github:garrytan/gbrain#<release-tag>`,
|
||||
same stamp as the runbook, so `bootstrap status` skew check is meaningful.
|
||||
- Non-terminal-buyer error channel: every blocking condition (scan block, lock
|
||||
collision, partial install, verify fail) surfaces through the agent's reply channel
|
||||
(agent-readable status/verify/doctor output the runbook + AGENTS.md instruct the
|
||||
agent to relay; hook failures write a status file surfaced at session start).
|
||||
- Both doors, one machine: shared brain, one live serve; installer detects + reuses
|
||||
existing registration; simultaneous-session collision fails politely, doctor names
|
||||
it, agent relays it. serve --attach lifts the limit later.
|
||||
- state/ boundary: interview.json + mcp.json COMMITTED (machine-2 re-render must not
|
||||
hard-fail); *.local/caches/heartbeat files gitignored; DB never committed.
|
||||
- OS scope v1: macOS + Linux (unix-socket IPC). Windows deferred (named-pipe = spike
|
||||
question if prioritized).
|
||||
- Success criteria additions: TTFM ≤15 min EXCLUDING first-run toolchain downloads
|
||||
(published separately); door-2 developer test (README paste → ≤10 min install,
|
||||
week-two recall check); graph floor verify check; magic moment = deterministic
|
||||
retrieval assertion + scripted human confirmation (never fake-automated).
|
||||
- Provider-policy drift acknowledged as unmitigable external risk; posture = quota
|
||||
gate + no absent-user background burn + portable body as exit plan.
|
||||
- README: bootstrap paste block is the "full agent" option INSIDE the existing "Quick
|
||||
start: Claude Code or Codex" section; the 2-line memory-only quick start STAYS,
|
||||
labeled "just memory, no agent"; INSTALL_FOR_AGENTS.md remains the platform headline.
|
||||
- Consolidated v1 non-goals (from design doc, amended by CEO review): connector ingest
|
||||
(v1.1), hosted mount + advisor nudge (v1.1), serve --attach, notify sweeper, Windows,
|
||||
networked-paste Docker e2e (fast-follows); 24/7 crons never on desktop.
|
||||
|
||||
**CEO-review accepted expansions (2026-08-07, SELECTIVE EXPANSION — now v1 scope):**
|
||||
- **GitHub template repo** ("Use this template") as a second distribution artifact for
|
||||
the ChatGPT door (resolves design OQ2 = yes). Same rendered file set, published;
|
||||
kept in sync with templates/ by extending scripts/check-bootstrap-templates.sh to
|
||||
diff the template repo content. Build order 2.
|
||||
- **`gbrain bootstrap uninstall`** in v1 (was fast-follow): removes MCP registration +
|
||||
hooks + bootstrap-created state (confirm-gated), leaves the repo ("the body remains
|
||||
yours"). [Scope superseded by CX2-12 + as-shipped delta 2: `~/.gbrain` is never
|
||||
deleted wholesale — only receipt-enumerated bootstrap-created state.]
|
||||
- **Docker cold-machine e2e (offline parts) in CI** in v1: networkless read-only
|
||||
container running interview → render → verify with fake gh (codex-as-agent
|
||||
tests/docker shape). The full networked paste flow stays a fast-follow (flake).
|
||||
- **Hot-memory greeting digest in SessionStart**: top facts via IPC turn_context when
|
||||
serve is up; file-plane MEMORY.md digest fallback when not (session start often
|
||||
precedes serve spawn); fail-open. Claude door in v1; door 1 iff spike finds a read
|
||||
seam.
|
||||
- **First-run tour**: `bootstrap verify` success output ends with three scripted magic
|
||||
prompts (who am I to you / remember X then restart / what do you know about
|
||||
<project>) — makes week-one compounding visible (design Premise 4).
|
||||
- **Deferred to TODOS.md in the PR** (must land as TODOS entries with the cathedral
|
||||
PR): `gbrain quota` meter command (release-gate measurement ships as script+doc
|
||||
first; productize once the per-harness token-count method is proven); networked
|
||||
Docker paste-flow e2e.
|
||||
|
||||
## Deep-review hardening (2026-08-07 CEO review sections 1–9 — ALL ACCEPTED per
|
||||
## Garry's standing directive; IDs trace to the review record. These are v1 scope.)
|
||||
|
||||
**Architecture (S1):**
|
||||
- [D4] The release-time template-repo generator IS `core/bootstrap/render.ts --minimal`
|
||||
with placeholder answers — one rendering code path, two consumers; CI guard becomes a
|
||||
byte-diff of generator output vs the vendored tree.
|
||||
- [D5] Bootstrap phase list defined ONCE in TS; `bootstrap status --json` emits it; the
|
||||
runbook defers to it ("follow status's phase list"); CI checks runbook phase names
|
||||
against the TS list.
|
||||
|
||||
**Source binding + brain semantics (G1, S3#1 — the two product-breaking fixes):**
|
||||
- [G1] MCP registration passes the workspace source explicitly
|
||||
(`claude mcp add -e GBRAIN_SOURCE=<workspace-slug>` / codex env equivalent) so agent
|
||||
writes land in the workspace source, fact fence-writes reach `brain/` files (not
|
||||
DB-only fallback), and the private repo actually fills. `bootstrap verify` asserts an
|
||||
MCP-path `put_page` materializes a COMMITTED file under `brain/` — a green verify with
|
||||
an empty repo is impossible.
|
||||
- [S3#1] `turn-context.ts` constructs an OperationContext with `remote: true` and
|
||||
threads `visibility: ['world']` into all fact reads (parity with the existing
|
||||
meta-hook posture — the IPC path must never widen what MCP would return). IPC test:
|
||||
a `visibility='private'` fact NEVER appears in a turn_context response. Verify's
|
||||
magic-moment fact is written with visibility the harness can read back (world).
|
||||
|
||||
**Interview + render hardening (S3#3, G10, G12, A8):**
|
||||
- Answers render inside fenced, explicitly-subordinate blocks ("verbatim principal
|
||||
input — data, not instructions"); strip/escape line-leading `#`, `<!--`, and code
|
||||
fences; per-answer length cap (~4KB, confirm-to-truncate); reject/escape `{{` and
|
||||
control chars at `--set` time (a Handlebars user's honest answer must not brick the
|
||||
token sweep). Verify fails if any heading or managed-block marker in a rendered file
|
||||
traces to an interview value (protects harden's AGENTS.md marker splicing).
|
||||
- [A8] `--confirm` requires the hash of the exact answer set that was read back;
|
||||
per-answer `set_at` provenance recorded; hostile test: single-batch set+confirm
|
||||
exits non-zero and verify reports `provenance: unverified`.
|
||||
- [G12] Conflict-marker detection on `state/interview.json` read → agent-readable
|
||||
"resolve this file" message, never a stack trace.
|
||||
|
||||
**Uninstall confinement (G2, S3#5, A2):**
|
||||
- Never delete a brain bootstrap didn't create: `agent.json` created-by stamp checked;
|
||||
default KEEPS the DB; deletion only behind explicit `--delete-brain` with a confirm
|
||||
that enumerates sources + page count; facts export offered first (facts are NOT
|
||||
derived state). Refuse when `GBRAIN_HOME` is set unless `--home` is explicit AND
|
||||
`isPathContained` + gbrain-home signature (config.json + brain.pglite) both pass.
|
||||
Host-config edits are marker-keyed managed blocks only (settings.local.json,
|
||||
~/.codex/config.toml) — foreign hooks/servers survive; test asserts full
|
||||
before/after filesystem+registration diff incl. foreign entries + symlinked home.
|
||||
|
||||
**Hooks + IPC hardening (G5, G11, S3#6, S3#7, S3#8, A9, A3):**
|
||||
- [G5] `bootstrap hooks`/`--repair`/uninstall use marker-keyed managed-block
|
||||
read-merge-write on settings.local.json — never overwrite `permissions.allow` or
|
||||
foreign hooks; never append duplicates.
|
||||
- [G11] IPC client timeout parameterized per kind (turn_context > 250ms default);
|
||||
window payload + assembled block clamped below the 256KB message cap before send.
|
||||
- [ENG-1] **Claude Code hook-output cap:** stdout/additionalContext is capped at
|
||||
10,000 chars by the harness (overflow is diverted to a file and NOT injected) —
|
||||
the assembled turn_context block is budgeted to ≤8KB (pointers + facts trimmed by
|
||||
confidence to fit), asserted in the hook snapshot test. Docs: code.claude.com hooks
|
||||
reference (verified 2026-08-08; the hook writers are dated spec-targets, so this cap
|
||||
lives with the settings-shape module).
|
||||
|
||||
## Eng-review hardening (2026-08-08 /plan-eng-review — ALL ACCEPTED per standing
|
||||
## directive; seams verified against source with file:line quotes)
|
||||
|
||||
- [ENG-2] (9/10) **cli.ts registration is THREE touchpoints, not one:** `CLI_ONLY` set
|
||||
(cli.ts:58), the engine-free if-chain inside `handleCliOnly` (add before the
|
||||
`connectEngine` terminator at cli.ts:1840), and `CLI_ONLY_SELF_HELP` (cli.ts:61 —
|
||||
omit it and the subcommands' `--help` is dead code, the documented init.ts:117 trap).
|
||||
`bootstrap`/`hook` must NOT enter `THIN_CLIENT_REFUSED_COMMANDS`. Membership test per
|
||||
the #2035 precedent (test/cli-bigint-normalize.test.ts:46 shape).
|
||||
- [ENG-3] (9/10) **IPC widening = handler restructure, not a field add.** Correct path
|
||||
is `src/core/context/resolve-ipc.ts` (constants at :26-28). `ResolveHandler` is a
|
||||
single-function type (:42) and the server handler dispatches unconditionally
|
||||
(:127-128) — turn_context needs a discriminated-union request + handler map
|
||||
(restructuring the closure at src/mcp/server.ts:90-108), NAMED response types
|
||||
(today's responses are inline literals), and per-kind MAX_MSG_BYTES/CLIENT_TIMEOUT.
|
||||
Absent `kind` defaults to 'resolve'; old-serve grace confirmed (malformed → client
|
||||
IPC_UNAVAILABLE fail-soft, resolve-ipc.ts:80-84).
|
||||
- [ENG-4] (8/10) **Postgres-hook silent degrade on unmigrated brains:** volunteer.ts
|
||||
(:247-249) and retrieval-reflex.ts (:180-183) swallow missing-table errors
|
||||
(pre-v110/v117), so a direct-engine hook returns empty, not an error. Fix: hook
|
||||
heartbeat records `degraded_reason: 'schema_pre_vNNN'`; doctor pairs
|
||||
hook-in-use + unmigrated-brain into a named warning.
|
||||
- [ENG-5] (8/10) **Sweep layer ownership decided:** startup sweep attaches in
|
||||
src/mcp/server.ts immediately after `server.connect` (:77) in the same best-effort
|
||||
try/catch shape as the resolve-IPC block (:85-112), with cleanup added to shutdown
|
||||
(:122); the idle sweep lives in src/commands/serve.ts reusing the `armIdle` pattern
|
||||
(:437-451) through the injectable `deps.setInterval` seam (:281,297), every timer
|
||||
`unref()`d (the :424/:444 convention) so the sweep can never hold the process open.
|
||||
- [ENG-6] (9/10) **Compiled-binary asset rule:** `bin/gbrain` ships via
|
||||
`bun build --compile`; `dirname(dirname(__dirname))` template resolution
|
||||
(init.ts:1514 pattern) breaks in the binary. Templates + questions.json + runbook
|
||||
stamps are STATICALLY IMPORTED (bundled) — same mechanism as the existing
|
||||
`skills/_brain-filing-rules.json` static import in brain-repo-durability.ts. A
|
||||
compiled-binary e2e asserts `bootstrap render` works with NO repo checkout present.
|
||||
- [ENG-7] (7/10) **Host-format module precedents named:** registration strings follow
|
||||
connect.ts `AGENT_SPECS` + argv builders (:61-67, :262-267); file-writing hook/
|
||||
settings writers follow the integrations.ts / frontmatter-install-hook.ts idiom
|
||||
(backup + marker + restore). The dated-spec-target scaffold is imported from
|
||||
codex-as-agent (greenfield here — docs/plans/ has n=1 file); it becomes
|
||||
`src/core/bootstrap/host-specs.ts` with TARGETS entries carrying id/status/
|
||||
verifiedAt/references.
|
||||
- [ENG-8] (9/10) **Facts visibility knob = ONE resolver helper.** The 'private'
|
||||
default is duplicated at backstop.ts:185, :352, operations.ts:4468, :5812 — and the
|
||||
:4468 ternary coerces any non-'world' to 'private', so a config default needs an
|
||||
explicit caller-unset check. Implement `resolveDefaultVisibility(engine)` (reads
|
||||
`facts.default_visibility` via the getConfig precedent, extract.ts:44) feeding
|
||||
ctx.visibility at ALL FOUR sites; no schema change (CHECK already permits 'world',
|
||||
migrate.ts:2319). Documented as security-relevant: it widens what remote/MCP
|
||||
callers read via meta-hook.ts:66 — the intended effect, stated as such.
|
||||
- [ENG-9] (8/10) **Secret-scan module reuses residents:** seed exclusion list from
|
||||
`.gitleaks.toml` allowlist paths (test/, skills/, .claude/skills/) so the scanner
|
||||
doesn't fire on fixtures CI already ignores; findings render through
|
||||
`redactSecretsInText` (shell-redact.ts:36) for consistent `<REDACTED:name>` output.
|
||||
- [ENG-10] (7/10) **Renderer must not eat intentional literals:** skillpack scaffolds
|
||||
carry a literal `{{output-from-skill}}` token (init-scaffold.ts:269) that must
|
||||
survive to disk — the renderer is never pointed at skillpack scaffold paths, pinned
|
||||
by a negative test.
|
||||
- [ENG-11] (7/10) **turn_context reuses the existing hot-memory cache** (30s TTL keyed
|
||||
on source+session, meta-hook.ts) instead of a fresh facts query per turn — the
|
||||
per-turn cost profile is then identical to what every MCP tool call already pays.
|
||||
- [ENG-tests] New tests from this review: CLI_ONLY membership (bootstrap, hook);
|
||||
compiled-binary render e2e (no repo checkout); Postgres-hook degraded_reason;
|
||||
resolveDefaultVisibility across all 4 call sites (unset/world/private × config);
|
||||
IPC handler-map dispatch (kind absent/resolve/turn_context/unknown); renderer
|
||||
negative test on scaffold literals; 8KB block-budget snapshot assert.
|
||||
- [ENG-tests-2] Coverage-trace gaps (Section 3): **attach-mode e2e** (clone a
|
||||
bootstrap-created fixture repo on "machine 2" → attach → hooks repair → verify);
|
||||
**sweep tests** (bounded batch per idle tick, spend-gate off ⇒ no LLM calls,
|
||||
unref/shutdown never held open, corpus file marked processed exactly once);
|
||||
**keyless-mode e2e** (zero API keys: install → agent-authored fact via ops → BM25
|
||||
recall → magic-moment passes → capability report says keyless);
|
||||
**decline-everything e2e** (no gh, no keys, hooks declined: install completes
|
||||
local-only with honest warnings, verify exits 0-with-warnings, nothing silently
|
||||
broken).
|
||||
|
||||
## Eng outside-voice absorption (Codex round 2, 2026-08-08 — 17/17 ACCEPTED,
|
||||
## all with file:line evidence; CX2 ids)
|
||||
|
||||
- [CX2-1 P0] **Template-vs-attach discriminator:** agent.json presence cannot
|
||||
discriminate a template clone from a machine-2 clone. The template ships
|
||||
`agent.json` with `initialized: false`; `bootstrap render` flips it true atomically
|
||||
AND writes a machine-local install receipt (`~/.gbrain/bootstrap/receipt.json`);
|
||||
`attach` requires `initialized: true`. [pairs with CX2-12]
|
||||
- [CX2-2 P0] **One-live-serve wording clarified:** each door spawns its OWN stdio
|
||||
serve via its MCP registration (process-bound transport, server.ts:76); "reuse
|
||||
registration" means config, never process. v1 contract restated: one live serve at
|
||||
a time per brain; sequential across doors works; simultaneous fails politely.
|
||||
- [CX2-3 P0] **Durability must be parent-repo-aware:** write-through targets
|
||||
`repo/brain` but hardening asserts `.git` in that exact dir (sources-harden.ts:83,
|
||||
brain-repo-durability.ts:621) — would fail on the workspace layout. Fix: durability
|
||||
resolves the repo root via `git rev-parse --show-toplevel` (the sync.ts:1002
|
||||
precedent); commit/push operate on the parent repo; integration test on the
|
||||
workspace fixture.
|
||||
- [CX2-4 P0] **Keyless facts ingestion made deterministic:** put_page only queues
|
||||
extraction (operations.ts:1337); facts-fence reconciliation lives in the cycle
|
||||
extractor (extract-facts.ts:337). The serve sweep INCLUDES the zero-LLM facts-fence
|
||||
reconciliation pass, so agent-authored `## Facts` fences populate the facts table
|
||||
with no API key. Keyless e2e asserts it.
|
||||
- [CX2-5 P0] **Graph-floor verify made deterministic:** verify cannot command the
|
||||
host's serve (stdio owned by the desktop app). Fix: the sweep gets a trusted
|
||||
local-only CLI entry (`gbrain sweep --once`, CLI_ONLY, never over MCP), and
|
||||
`bootstrap verify` runs BEFORE host registration on its own transient serve/engine:
|
||||
write via op → `sweep --once` → edge query. No timing nondeterminism.
|
||||
[Sequencing superseded by as-shipped delta 1: verify shipped as the LAST phase,
|
||||
in-process on the caller-held engine; the determinism goal is unchanged.]
|
||||
- [CX2-6 P1] **Cross-platform lock replaces flock dependence:** flock(1) absent ⇒
|
||||
locking silently disabled (brain-repo-durability.ts:137) — macOS is the v1 target.
|
||||
One cross-platform lock (atomic mkdir/lockfile with PID+age+token semantics) spans
|
||||
scan → stage → commit → pull → push, coordinated with the post-commit hook.
|
||||
[As-shipped delta: a TOCTOU fix reordered the scanned phase to stage FIRST, then
|
||||
secret-scan the STAGED index blobs (`git cat-file`), so scanned bytes == committed
|
||||
bytes; unscannable staged blobs fail closed (`blocked_unscannable`). Lock span
|
||||
otherwise unchanged.]
|
||||
- [CX2-7 P1] **Push ordering pinned:** commit FIRST, then divergenceSafePull, then
|
||||
push (the existing durability ordering, brain-repo-durability.ts:200 —
|
||||
divergenceSafePull returns skipped_dirty on a dirty tree, git-remote.ts:489); test:
|
||||
dirty local + advanced remote.
|
||||
- [CX2-8 P1] **GBRAIN_HOME dual semantics normalized:** config appends `.gbrain`
|
||||
(config.ts:1210) while durability uses the value directly
|
||||
(brain-repo-durability.ts:95) — the S3#10 `ensureGbrainHome()` choke point is also
|
||||
the single semantic resolver; the --isolated e2e asserts the credential store's
|
||||
ACTUAL location is gitignored, not just the expected path.
|
||||
- [CX2-9 P1] **Provider-key resolution re-specified honestly:** env legitimately
|
||||
overrides file config (config.ts:568) — the CX-P1.4 claim "never from shell env" is
|
||||
wrong. Real contract: interview-provided keys go to the 0600 config file so
|
||||
GUI-spawned serves (which lack shell env) find them; normal env>file precedence
|
||||
stands; test = GUI-launch simulation with empty env + file key.
|
||||
- [CX2-10 P1] **IPC authorization, not just authentication:** turn_context binds
|
||||
server-side to the registered GBRAIN_SOURCE and rejects caller-supplied cross-source
|
||||
requests (existing handler accepts caller sourceId, server.ts:89); cross-source
|
||||
rejection test.
|
||||
- [CX2-11 P1] **Hot-memory cache session key made real:** meta-hook reads an ad-hoc
|
||||
`source_session` that dispatch never sets (meta-hook.ts:49, dispatch.ts:208) — all
|
||||
callers collapse to the null-session cache key today. Session identity becomes
|
||||
typed OperationContext state set from MCP `_meta.session_id`; two-session isolation
|
||||
test.
|
||||
- [CX2-12 P1] **Uninstall ownership = machine-local receipt** (never the repo-carried
|
||||
agent.json, which template/attach clones inherit); uninstall stops/refuses a live
|
||||
serve before touching state; `~/.gbrain` global config/sources/clones are NEVER
|
||||
deleted wholesale — only receipt-enumerated bootstrap-created state.
|
||||
- [CX2-13 P1] **Committed-state hygiene:** the optional interview API key bypasses
|
||||
answers/hashes/provenance/logs entirely → written only to the 0600 config sink
|
||||
(config.ts:1138); committed `state/mcp.json` is the PORTABLE snippet (no absolute
|
||||
paths, no machine GBRAIN_HOME) — machine-specific wiring lives in local state and
|
||||
is regenerated by attach/`hooks --repair`.
|
||||
- [CX2-14 P1] **Generator determinism:** template-repo renders use canonical
|
||||
placeholder provenance (frozen timestamps), exclude runtime state; guard test =
|
||||
two independent renders are byte-identical.
|
||||
- [CX2-15 P1] **Two scan policies, not one:** the `.gitleaks.toml` allowlist is a CI
|
||||
fixture policy for a PUBLIC repo — importing it into the personal-repo runtime
|
||||
scanner creates blind spots (skills/ trees). Runtime scanner ships its own minimal
|
||||
allowlist + per-finding override; ENG-9 amended.
|
||||
- [CX2-16 P1] **Bootstrap lock done properly:** atomic acquisition + PID liveness +
|
||||
age guard + ownership token (the pid-reuse learning applied); kill→immediate-rerun
|
||||
recovery test; export `LiveServeLockError` (currently unexported, pglite-lock.ts:28)
|
||||
so the planned class assertion can import it.
|
||||
- [CX2-17 P2] **Format-aware host-config writers:** JSON has no comment-marker
|
||||
boundary — settings.local.json gets a structural JSON merger (gbrain entries keyed
|
||||
by a `_gbrain` marker property, semantic dup detection surviving reordering);
|
||||
config.toml gets a TOML-aware writer; both atomic write+backup; the
|
||||
frontmatter-install-hook replace/backup idiom applies only to whole-file targets.
|
||||
G5 amended.
|
||||
- [S3#6] Socket binds in a 0700 dir with mode set BEFORE exposure; turn_context
|
||||
requires a shared secret from a 0600 file in the data dir; heartbeat counts
|
||||
turn_context serves so doctor can flag unexplained callers.
|
||||
- [S3#7] Heartbeat JSONL schema pinned to counters + durations + error codes — NO
|
||||
prompt/fact/slug text; dir 0700; line cap; CI test greps fixture for keys outside
|
||||
the allowlist.
|
||||
- [S3#8] `transcript_path` confined: `isPathContained(path, ~/.claude/projects)`,
|
||||
`.jsonl` + valid envelope on line 1, lstat-reject symlinks, byte cap.
|
||||
- [A9] IPC request carries `protocol: 2`; hook treats a response lacking the protocol
|
||||
echo as "stale serve" and degrades LOUDLY (heartbeat entry + doctor warn); live test
|
||||
against a v1-shaped server.
|
||||
- [A3] Session-start digest: explicit list of digest-eligible MEMORY.md sections
|
||||
(respecting the template's own security-boundary note); three-case test (socket up /
|
||||
socket absent / malformed MEMORY.md).
|
||||
|
||||
**Persistence + sync (G4, G6, G8, G9, G13, G14/A5, G15, S3#2, S3#10):**
|
||||
- [G4] `hook session-start` checks for unpushed commits / dirty tree from crashed
|
||||
sessions, pushes, and names it in the greeting digest (SessionEnd is not the only
|
||||
persistence path anymore).
|
||||
- [G6] Verify + every push gate run `git ls-files` against a deny-glob list
|
||||
(`*.pglite`, `.env*`, keys) — a truncated or pre-existing .gitignore can't leak.
|
||||
- [G8] `bootstrap repo` creates a dedicated repo, OR adopts a pre-existing `origin`
|
||||
when the authed gh user owns it, no `repo_url` is recorded yet, and it is empty (or
|
||||
already carries our history) — the create-repo-first path; a foreign-content or
|
||||
org-owned origin is refused and pointed at attach. "couldn't verify visibility" is
|
||||
refuse-and-name-the-reason, never fail-open; idempotency keys off the remote URL,
|
||||
not the name probe.
|
||||
- [G9] Workspace lockfile (pid+timestamp) makes concurrent `bootstrap` runs impossible;
|
||||
second run exits "bootstrap already running (pid N)".
|
||||
- [G13] Fixed verify probe slug; sweep any prior probe before writing; excluded from
|
||||
retrieval; delete failure reported as a verify warning.
|
||||
- [G14/A5] The ENTIRE `sources push` (add+commit+push) runs under the existing
|
||||
durability flock; single-flight test: N concurrent pushes → 1 winner, N-1 clean
|
||||
"skipped, push in flight" exits, zero leftover locks.
|
||||
- [G15] Retention policy everywhere: MEMORY.md size cap in the template contract with
|
||||
rotation into `memory/reference/`; corpus pruned via
|
||||
`dream.synthesize.corpus_retention_days` (default 30); orphaned stop-hook buffers
|
||||
GC'd; doctor warns on all three.
|
||||
- [S3#2] Secret scan runs at corpus-WRITE time (redact matched span + log redaction);
|
||||
bootstrap adds one consent line naming the extraction/embedding provider that will
|
||||
see session text.
|
||||
- [S3#10] One `ensureGbrainHome()` choke point creates ~/.gbrain 0700 (all callers);
|
||||
brain-push.log 0600 + rotated; `--isolated` e2e asserts `git check-ignore
|
||||
.gbrain/git-credentials` passes and push never stages it.
|
||||
|
||||
**Transcripts (G3, A6):**
|
||||
- Parser registered as a dated spec-target (same pattern as host-format writers);
|
||||
heartbeat logs parsed-turns/bytes; `bytes>0 && turns==0` raises a LOUD status-file
|
||||
failure ("the agent stopped learning" is never silent); scrubbed
|
||||
`claude-code.jsonl` fixture (tool_use/tool_result/thinking/image/sidechain/summary/
|
||||
compact-boundary shapes) added to `test/fixtures/conversation-formats/` and wired
|
||||
into `check:conversation-parser` + fixture-privacy guard; session-id-keyed corpus
|
||||
filenames prevent double-ingest on resume.
|
||||
|
||||
**Template repo (G7, S3#4, A1, C3):**
|
||||
- Published ONLY from a CI release job (branch-protected, no direct pushes) behind the
|
||||
placeholder assertion + secret scan + privacy scripts run against the RENDERED
|
||||
artifact; fine-grained PAT secret scoped `contents:write` to the template repo only,
|
||||
documented in docs/RELEASING.md; version-job completeness check verifies template
|
||||
repo HEAD tree hash == vendored tree. Vendored rendered tree lives at
|
||||
`templates/bootstrap/template-repo/` — the CI guard diffs generator↔vendored OFFLINE
|
||||
(network comparison happens only in the release workflow). Template's first runbook
|
||||
step is `bootstrap status`, which hard-fails on a public `origin` before any
|
||||
identity file lands (closes the no-privacy-gate-on-this-door hole).
|
||||
|
||||
**Toolchain trust (S3#9):**
|
||||
- Prefer platform package managers for bun/gh; curl fallback downloads to a file,
|
||||
verifies against the pinned bun release's SHASUMS256.txt, then executes; the
|
||||
runbook's phase allowlist names the verified form as the only permitted variant.
|
||||
|
||||
**Observability (B1–B5):**
|
||||
- [B1] Append-only `~/.gbrain/bootstrap/install.jsonl` ({ts, phase, outcome,
|
||||
duration_ms, binary_version, harness, workspace}) written by every subcommand;
|
||||
`status` renders the tail.
|
||||
- [B2] Every verify run persisted to `~/.gbrain/bootstrap/verify-<ts>.json` (keep 5);
|
||||
doctor reports last verify timestamp/status/deltas.
|
||||
- [B3] Hook heartbeat records outcome + reason on EVERY invocation; trailing-20
|
||||
failure rate over threshold → one visible line inside the injected context block
|
||||
("brain context unavailable for the last N turns — run `gbrain doctor`").
|
||||
- [B4] `~/.gbrain/bootstrap/push-status.json`; doctor fails when last successful push
|
||||
>48h old with a dirty tree; surfaced in the SessionStart digest.
|
||||
- [B5] `bootstrap status --json` emits a support blob (workspace, binary version,
|
||||
harness, engine, last verify, last push, hook failure rate); AGENTS.md instructs
|
||||
the agent to relay it verbatim on any "something's broken" report.
|
||||
|
||||
**Tests (A1–A9 not already covered above):**
|
||||
- [A4] First-run tour prompts pinned in the questions.json↔template bijection guard +
|
||||
verify success-output snapshot test.
|
||||
- [A7] Flake pinning: hook deadline asserted against an injected slow-IPC stub (real
|
||||
latency kept as non-gating benchmark); kill-mid-phase via deterministic
|
||||
`GBRAIN_BOOTSTRAP_ABORT_AFTER=<phase>` injection; lock-contention asserts on
|
||||
LiveServeLockError class; all subprocess tests named `*.serial.test.ts` with
|
||||
explicit --timeout; offline Docker e2e runs from heavy-tests.yml (nightly + label),
|
||||
not the PR shard matrix.
|
||||
|
||||
**Release mechanics (C1 RESOLVED + C2, C4, C5, C6, C8):**
|
||||
- [C1 = D6-A, decided by Garry] **Distribution ref: single `latest-stable` ref.** The
|
||||
release job force-updates `latest-stable` to the just-verified release commit as its
|
||||
FINAL step (after binary assets publish + provenance attestation). README paste
|
||||
block, runbook URL, and `bun install -g github:garrytan/gbrain#latest-stable` all
|
||||
reference it permanently — no per-release tag pins, no 404 window, no per-ship
|
||||
README edits. The fetched runbook embeds the concrete VERSION it was cut from;
|
||||
`bootstrap status` compares that stamp to the installed binary (skew check intact).
|
||||
New guard `scripts/check-bootstrap-tag.sh`: README/runbook reference ONLY the
|
||||
sanctioned ref AND the runbook's embedded stamp equals VERSION. Modeled on
|
||||
test/release-workflow.test.ts.
|
||||
- [C2] README.md + BOOTSTRAP_FOR_AGENTS.md re-admitted to `scripts/ci-cache-hash.sh`
|
||||
ALLOW_PATTERNS (a README-only paste-block change must never skip CI green).
|
||||
- [C4/C5] `docs/designs/AGENT_BOOTSTRAP_PLAN.md` + a SCRUBBED
|
||||
`docs/designs/AGENT_BOOTSTRAP_DESIGN.md` (banned names → capability-class phrasing,
|
||||
founder quotes/pilot identifiers/pricing-funnel strategy removed) land in the SAME
|
||||
commit; normativity paragraphs point in-repo; `bash scripts/check-privacy.sh` run
|
||||
against the staged index before that commit.
|
||||
- [C6] CHANGELOG states capabilities functionally ("installs hooks that run on each
|
||||
prompt; installs an opt-in background push job") linking to docs/guides/bootstrap.md
|
||||
which owns the full security/consent posture together with the rendered
|
||||
ACCESS_POLICY.md; zero plan IDs / review-round references.
|
||||
- [C7] Public product names ("GBrain for Codex"/"GBrain for Claude Code") are a
|
||||
pre-merge checklist item owned by Garry (trademark review); all committed copy uses
|
||||
the neutral `gbrain bootstrap` verb until sign-off.
|
||||
- [C8] Zero-migration release confirmed (facts/context_volunteer_events already
|
||||
exist); ALL new bootstrap telemetry stays on the filesystem — the moment it moves
|
||||
into a table it becomes migration v126 + bootstrap-coverage + engine-parity
|
||||
obligations.
|
||||
- [G16] User-scope consent question names the project-scope tradeoff explicitly;
|
||||
CLAUDE.md renders a one-line "this agent lives in <dir>" note.
|
||||
|
||||
## Outside-voice absorption (Codex, 2026-08-07 — dispositions under Garry's
|
||||
## auto-accept directive; CX ids trace to the codex output)
|
||||
|
||||
**ACCEPTED — product-critical (the two the whole review chain missed):**
|
||||
- [CX-P0.5 **Keyless mode is a first-class design requirement.**] The named pilot user
|
||||
has ChatGPT Pro but NO API keys — and gbrain's embeddings + extract_facts (Haiku)
|
||||
are API-metered. Bootstrap MUST work with zero API keys: the HARNESS AGENT is the
|
||||
subsidized LLM, so AGENTS.md instructs it to author facts/timeline/links explicitly
|
||||
through write ops (zero API cost); search degrades to BM25 keyword (no embeddings);
|
||||
verify prints an honest capability report ("keyless mode: keyword search, agent-
|
||||
authored memory; add ONE key to unlock embeddings + auto-extraction") and the
|
||||
interview offers the optional key question. The magic-moment check must pass in
|
||||
keyless mode (fact written by the agent via ops, recalled next session). Quota-gate
|
||||
copy amended: API spend (embeddings/extraction) is metered separately from
|
||||
subscription quota and is ZERO in keyless mode; with a key, existing spend gates
|
||||
(docs/operations/spend-controls.md) govern. [also resolves CX-P0.6]
|
||||
- [CX-P0.1+P0.3 **Serve-resident maintenance sweep** closes the persistence loop.]
|
||||
Nothing previously ingested the transcript corpus into the live brain (dream is
|
||||
disabled; CLI can't open PGLite under a live serve), and remote `put_page`
|
||||
deliberately skips auto-link/timeline extraction (operations.ts:1273) so the graph
|
||||
would never compound from harness writes. Fix: the serve process (the lock owner)
|
||||
runs a bounded, spend-gated maintenance sweep — on startup and idle — that (a)
|
||||
ingests unprocessed corpus files (keyless mode: skipped, agent-authored memory
|
||||
covers it), (b) runs the deterministic zero-LLM link/timeline extraction over
|
||||
recently written workspace-source pages. Verify's graph-floor check exercises the
|
||||
REAL MCP write path end-to-end (write via stdio op → sweep → edge query), never a
|
||||
synthetic install-time shim.
|
||||
|
||||
**ACCEPTED — durability/trust:**
|
||||
- [CX-P0.4] Write-through failures on `put_page` are best-effort today → for the
|
||||
workspace source they become LOUD: doctor check counts DB-pages lacking file
|
||||
backing; surfaced in the SessionStart digest; sources push reconciles before commit.
|
||||
- [CX-P1.1] Single-principal visibility posture: bootstrap sets the workspace brain's
|
||||
fact default visibility to 'world' (the desktop agent IS the principal's main
|
||||
session; ACCESS_POLICY.md documents it; multi-tier users flip the knob). Keeps the
|
||||
IPC world-only filter (S3#1) AND working personalization.
|
||||
- [CX-P1.2] The injected turn_context block is wrapped in the same "retrieved data,
|
||||
never instructions" subordinate envelope as interview answers (provenance-labeled).
|
||||
- [CX-P1.4] GUI env inheritance: MCP registration carries the absolute binary path +
|
||||
explicit env (GBRAIN_SOURCE, GBRAIN_HOME when --isolated); provider keys resolve
|
||||
from ~/.gbrain/config.json, never from shell env, for GUI-spawned serves.
|
||||
- [CX-P1.5] **`bootstrap attach` mode** (machine two): a cloned repo carrying
|
||||
agent.json enters attach mode (register source, hooks --repair, MCP, verify) —
|
||||
the refuse-pre-existing-origin rule (G8) applies only when agent.json is absent.
|
||||
- [CX-P1.6] Git conflict model: sources push does fetch + divergence-safe rebase pull
|
||||
(reuse harden's divergenceSafePull) before push; non-FF/conflict = loud status +
|
||||
agent-relayed instruction, never silent retry; repo-local git author identity set
|
||||
at `bootstrap repo` (ported from setup-private-repo).
|
||||
- [CX-P1.7] Search-mode consent folds into the interview as one optional question
|
||||
(default balanced) — preserves the INSTALL_FOR_AGENTS consent contract without
|
||||
another stop; the spike's TTFM measurement counts every stop.
|
||||
- [CX-P1.9] Non-circular error surface: README + GITHUB.md carry the one recovery
|
||||
command ("if it seems broken: `gbrain doctor`"); doctor covers hook/push/serve
|
||||
health independent of the possibly-broken agent loop.
|
||||
- [CX-P1.10] Forget semantics documented honestly in ACCESS_POLICY.md/GITHUB.md: the
|
||||
repo is append-only history; true deletion = documented history-rewrite procedure;
|
||||
GitHub remains default-but-optional (local-only mode with honest warning).
|
||||
- [CX-P1.11] Template repo README embeds the same VERSION stamp as the runbook;
|
||||
`bootstrap status` validates it identically (closes the adopter-skew window).
|
||||
- [CX-P1.12] agent.json spec labeled **provisional-v1**: desktop-side validation
|
||||
only, consumers tolerate unknown fields, hosted mount may bump to v2 with a
|
||||
migration note — a version contract, not a frozen promise.
|
||||
- [CX-P0.7+P1.13] Spike/pilot instrument upgraded from binary pass/fail to product
|
||||
metrics: per-turn write-attempt vs durable-write precision, correct-recall rate,
|
||||
false-memory incidents, correction round-trips — measured through the 2-week pilot
|
||||
(the pilot IS the extended sample); the 0-failures-in-20 bar remains only the
|
||||
minimum to START the pilot.
|
||||
|
||||
**REJECTED (with reasons, recorded):**
|
||||
- [CX-P0.2] "Two desktop apps cannot share the brain" — TRUE for simultaneous
|
||||
sessions and already the documented v1 limit (polite collision + doctor + attach
|
||||
proxy fast-follow); sequential use works. No change beyond what's accepted.
|
||||
- [CX-P1.8] "Networked paste flow untested" — known; manual clean-machine acceptance
|
||||
in v1 + networked Docker e2e deliberately deferred (D3.3b). Stands.
|
||||
|
||||
**CROSS-MODEL TENSION (recorded, not re-litigated):** Codex's "fundamental
|
||||
simplification" (Codex-only, 3-question interview, no hooks/template/GitHub, 2-week
|
||||
manual pilot before building) is a REDUCTION-mode argument against the settled
|
||||
cathedral decision (D1-A) and accepted Approach B scope. Disposition: rejected as
|
||||
sequencing (Garry chose the cathedral 30 minutes prior, with the two-wave case
|
||||
recorded for revisit + a size trip-wire), but its measurement substance was absorbed
|
||||
via CX-P0.7/P1.13, and the spike + pilot ARE the "prove the loop" instrument, run
|
||||
before the doors ship rather than instead of building them.
|
||||
|
||||
|
||||
## Design (synthesized)
|
||||
|
||||
Synthesis of three independent designs (DX lens, runtime-parity lens, architecture lens) +
|
||||
an adversarial critique that verified the load-bearing claims against both repos. Where the
|
||||
designs disagreed, the critique's evidence-checked winner is taken.
|
||||
|
||||
### The experience (end state)
|
||||
|
||||
One paste block (README + tweet) → the agent fetches `BOOTSTRAP_FOR_AGENTS.md` (raw GitHub
|
||||
URL **pinned to a release tag**, same mechanism as INSTALL_FOR_AGENTS.md) and drives:
|
||||
|
||||
preflight → interview (chat, 12Q/6-required, hard gate) → render identity files → skills →
|
||||
MCP + hooks wiring → sources add/sync/embed → private GitHub repo (created, privacy-verified,
|
||||
pushed) → `gbrain bootstrap verify` (exit-code contract) → completion manifest.
|
||||
|
||||
Human actions: paste, answer interview in chat, `gh auth login` if needed, ~2 consent
|
||||
questions. Target ≤15 min. Definition of done includes the magic moment: after restart, the
|
||||
agent recalls a fact the human said during the interview.
|
||||
|
||||
Second session: SessionStart hook injects identity digest + "since last time"; every prompt
|
||||
gets Live Context + ≤3 volunteered brain pointers + hot memory injected via hook; the agent
|
||||
writes facts back same-turn through MCP ops; SessionEnd ingests the transcript into the dream
|
||||
corpus and fires a scan-gated commit+push. "Your local harness IS your agent."
|
||||
|
||||
### Decisions (settled by panel + critique)
|
||||
|
||||
- **D1 Topology: single private repo.** Workspace root = identity files (SOUL.md, USER.md,
|
||||
MEMORY.md, AGENTS.md, CLAUDE.md, HEARTBEAT.md, GITHUB.md, ACCESS_POLICY.md) + `skills/` +
|
||||
`memory/` (daily notes + README) + `state/` + **`brain/`** (people/ companies/ meetings/
|
||||
concepts/ daily/). **Only `brain/` is registered as the gbrain source** (critique: indexing
|
||||
the whole repo puts the contract files into the retrieval corpus — noise + self-referential
|
||||
injection surface). `.gbrain-source` dotfile at root routes CLI calls. PGLite DB is NEVER
|
||||
in the repo (`.gitignore` renders first: `*.pglite`, `.env*`, `state/*.local`, corpus paths,
|
||||
keys/PEM). Two-repo model stays documented as the graduation path in GITHUB.md/docs.
|
||||
- **D2 Workspace = the cwd the user pasted in.** Guard: if cwd is an existing code project
|
||||
(tracked files/remote), ask ONE relocation question. DB default: global `~/.gbrain` host
|
||||
brain (workspace is a source); `--isolated` escape hatch per resolved D2.
|
||||
- **D3 Delivery: hybrid.** New `gbrain bootstrap` command family (deterministic, idempotent,
|
||||
exit-coded) + fetched runbook (judgment: interview wording, read-backs, consent). Bootstrap
|
||||
is **CLI-only like init/connect — NOT an operation**; zero new `ctx.remote` surface (a
|
||||
remote-triggerable `gh repo create` + filesystem render is forbidden by the trust model).
|
||||
Subcommands: `status` (resume entrypoint; also verifies runbook version stamp vs binary —
|
||||
supply-chain skew check), `interview --init|--set K V|--skip K|--status|--confirm|--show`,
|
||||
`render [--force] [--only F] [--minimal]`, `repo`, `hooks --harness claude-code|codex
|
||||
[--repair]`, `verify [--json]`.
|
||||
- **D4 Interview: one question bank, two entry points.** Port codex-as-agent's
|
||||
`questions.json` (scrubbed) to `templates/bootstrap/questions.json`: 12 asked / 6 required
|
||||
(AGENT_NAME, PRINCIPAL_NAME, AGENT_PURPOSE, AGENT_TOP_JOBS, PRINCIPAL_CONTEXT,
|
||||
VOICE_REGISTER — the wince question and vibe samples from soul-audit Phase 2 fold in as
|
||||
optional keys). Answers → `<ws>/state/interview.json` (committed; identity source of truth,
|
||||
same sensitivity as the rendered USER.md; makes re-render-on-new-machine work).
|
||||
`--status` exits non-zero until required present; `render` additionally refuses until
|
||||
`--confirm` (set only after the read-back "Is this the thing you want in the room?").
|
||||
Hard rules verbatim in runbook: NEVER INVENT ANSWERS, accept "skip", quote literal words.
|
||||
`skills/soul-audit/SKILL.md` becomes the re-run/deepen surface over the same bank
|
||||
(`interview --set` + `render --only SOUL.md`) — one bank, CI-guarded against drift.
|
||||
- **D5 Per-turn context (critique-decided; the PGLite lock is the constraint).**
|
||||
`gbrain serve` holds the PGLite single-writer lock for its lifetime (#2348: live holders
|
||||
are never stolen) — so hooks must NEVER open the engine on PGLite. New engine-free
|
||||
`gbrain hook <event>` command (no-engine dispatch branch in cli.ts) talks ONLY to serve's
|
||||
existing resolve-IPC unix socket (`src/core/context/resolve-ipc.ts`), widened with
|
||||
back-compat `kind: 'resolve' | 'turn_context'`. `turn_context` request carries
|
||||
{window, priorContextText, sourceId}; serve assembles server-side (new
|
||||
`src/core/context/turn-context.ts`): reflex pointers + volunteerContext pages (≤3) +
|
||||
hot-memory facts (the same content `_meta.brain_hot_memory` carries — this ROUTES AROUND
|
||||
the _meta invisibility rather than fixing the harness). Postgres fallback: hook opens the
|
||||
engine directly (multi-connection safe) since the IPC socket only exists on PGLite.
|
||||
- Claude Code hooks (written to **`.claude/settings.local.json`** — gitignored; committed
|
||||
hooks with absolute paths are a portability trap + persistence-of-execution surface;
|
||||
`bootstrap hooks --repair` re-renders on a new machine):
|
||||
- SessionStart → `gbrain hook session-start`: zero-DB file reads (date/tz, MEMORY.md
|
||||
open-commitments, ops/tasks.md if present) + best-effort socket warm probe. ≤1.5s.
|
||||
- UserPromptSubmit → `gbrain hook user-prompt`: stdin hook JSON → tail last 4 turns from
|
||||
transcript_path → socket turn_context → stdout additionalContext block.
|
||||
**800ms hard self-deadline**, fail-open (exit 0, empty stdout), heartbeat JSONL at
|
||||
`~/.gbrain/integrations/hooks/` for doctor.
|
||||
- Stop → `gbrain hook stop`: append turn to live per-session buffer (pure file append).
|
||||
- SessionEnd → `gbrain hook session-end`: parse full transcript .jsonl → corpus .txt in
|
||||
`dream.synthesize.session_corpus_dir` (default `~/.gbrain/transcripts/corpus/`, 0700,
|
||||
never in the repo) + fire scan-gated commit+push (D6). Closes the transcript gap in PR1.
|
||||
- Absolute binary path everywhere; `GBRAIN_HOOKS=0` kill switch.
|
||||
- Codex (no hooks): honest pull model. Rendered AGENTS.md carries adapted per-message
|
||||
gates 0–7 (entity lookup = "call recall/volunteer_context with recent window";
|
||||
receipts; WRITE IT DOWN same turn via extract_facts/put_page) +
|
||||
`codex mcp add gbrain -- gbrain serve`. FF2: `notify` hook in ~/.codex/config.toml as
|
||||
transcript sweeper (validate the event semantics first). Runbook/verify state the
|
||||
degradation plainly: Claude Code = push-on-hook, Codex = pull-on-protocol.
|
||||
- **Write-path rule rendered into AGENTS.md** (critique hole 2): on PGLite, durable
|
||||
knowledge is written through MCP ops (put_page/extract_facts/add_timeline_entry), never
|
||||
by editing brain/ files directly — file edits are invisible to retrieval until a sync
|
||||
can run, and sync can't run while serve holds the lock. MEMORY.md/memory/ file edits are
|
||||
fine (file-plane, loaded by path not retrieval).
|
||||
- **D6 Private repo + persistence.** `gbrain bootstrap repo` = TS port of
|
||||
setup-private-repo.mjs: gh-auth exit-2 gate (only human step: `gh auth login`), slugified
|
||||
`<agent-name>-workspace` collision probe, `gh repo create --private --source . --push`,
|
||||
**privacy verified via `gh api ... --jq .private` (hard fail)**. Sync: extract secret scan
|
||||
to `src/core/secret-scan.ts` (sk-/gh[pousr]_/github_pat_/xox[baprs]-/PEM; blocks commit);
|
||||
new `gbrain sources push [<id>|--path]` = scan-gated add+commit+push, refuses public
|
||||
remotes, pushes even on clean tree. `hardenBrainRepo` gains the scan as a step (its
|
||||
existing post-commit hook + cron machinery is reused, NOT a parallel sync system).
|
||||
Cadence per resolved D3: 15-min cron installed after an explicit consent question;
|
||||
SessionEnd-hook push always on as the no-daemon backstop/fallback. GITHUB.md persistence
|
||||
contract rendered.
|
||||
- **D7 Rendered files.** AGENTS.md (adapted gates + hard gates: WRITE IT DOWN, NO SILENT
|
||||
FAILURE, VERIFY BEFORE CLAIMING DONE, RED LINES, PRIVATE REPO PERSISTENCE + brain-first
|
||||
protocol from docs/tutorials/connect-coding-agent.md + brain filing contract rendered from
|
||||
skills/_brain-filing-rules.md), CLAUDE.md (thin: @AGENTS.md @SOUL.md @USER.md @MEMORY.md +
|
||||
hooks note), SOUL.md (codex-as-agent section skeleton: Identity/Mission/Worldview/The
|
||||
Standard/Honesty/Voice+wince/Good vs bad output/High agency/Never — IDENTITY.md merged in),
|
||||
USER.md ("their literal words are ground truth"), **MEMORY.md (new template**: hot state,
|
||||
corrections format `- YYYY-MM-DD — rule (Bug: ...)`, open commitments, security-boundary
|
||||
note), ACCESS_POLICY.md, HEARTBEAT.md (quiet hours, verify-time-first, silence contract,
|
||||
jobs disabled), GITHUB.md, memory/README.md, brain/ dirs + READMEs, .gitignore,
|
||||
state/interview.json, state/mcp.json (portable snippet). Skills via existing
|
||||
`gbrain skillpack scaffold --all` (resolved D4) + inert-skill report (skillpack check
|
||||
wired into verify AND into the completion manifest's next-steps block with the exact API
|
||||
keys to add). ALL templates REWRITTEN generic (privacy IRON RULE —
|
||||
never copy codex-as-agent prose verbatim; CI placeholder assertion). TOOLS.md deferred
|
||||
(essential lines fold into AGENTS.md). `installDefaultTemplates` (init.ts:1513) finally
|
||||
gets its caller via `render --minimal`.
|
||||
- **D8 Verify (union).** (1) doctor green; (2) DB round-trip: put_page → get → query-with-
|
||||
score → DELETE probe page; (3) MCP registered (`claude|codex mcp list`) + probeBrainIdentity
|
||||
smoke; (4) {{TOKEN}} sweep hard-fail; (5) byte floors (SOUL.md ≥3000B, USER.md ≥1000B);
|
||||
(6) secret scan clean; (7) repo private via API; (8) **hooks smoke UNDER LIVE SERVE**
|
||||
(spawn real serve, pipe fixture UserPromptSubmit stdin, assert non-empty block + <800ms +
|
||||
never acquires the lock — bootstrap-time-only smoke is a false green); (9) one manual
|
||||
sources push succeeded; (10) inert-skill report; (11) transcript parser dry-run on fixture.
|
||||
Prints ranked completion manifest. Re-runnable weekly as the workspace rot self-check.
|
||||
- **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
|
||||
daemon). OPT-IN: 15-min harden cron. Autopilot NOT default on PGLite (verified: its
|
||||
sync/embed children would contend with every live serve for the single-writer lock, and
|
||||
nothing handles LiveServeLockError politely today) — recommended on Postgres; any future
|
||||
scheduled job must treat lock-held as skip-silently-and-log. LLM crons (briefing, dream via
|
||||
`claude -p`/`codex exec`) ship rendered-but-disabled with the enable-one-at-a-time ritual.
|
||||
- **D10 Phasing.**
|
||||
- **PR1 (complete usable experience):** BOOTSTRAP_FOR_AGENTS.md + README paste block;
|
||||
`gbrain bootstrap` family; `gbrain hook` + IPC v2 turn_context; new templates + question
|
||||
bank; secret-scan + `sources push` + harden integration; SessionEnd transcript ingest
|
||||
(Claude Code .jsonl parser); skillpack scaffold wiring; soul-audit re-run update; doctor
|
||||
checks (hooks heartbeat, dual-serve report, sync check); docs; unit + e2e (incl.
|
||||
hook-under-serve + lock-contention pins); CI guards.
|
||||
- **FF2:** Codex notify sweeper + ~/.codex/sessions parser. **FF3:** `gbrain serve
|
||||
--attach` stdio proxy (two harnesses fully concurrent on one PGLite brain). **FF4:**
|
||||
cron fleet + cron-doctor port + heartbeat activation. **FF5:** Docker fresh-machine e2e
|
||||
in CI, _meta surfacing experiments, per-turn-context BrainBench eval, upgrade re-render
|
||||
nudge via runPostUpgrade (`bootstrap render --diff`).
|
||||
|
||||
### Security/trust invariants (critique holes, addressed)
|
||||
|
||||
1. Paste block pinned to release tag; runbook version-stamped; `bootstrap status` compares
|
||||
stamp vs binary and warns; runbook instructs refusing steps outside its phase list.
|
||||
2. Write-through-ops rule in AGENTS.md (above).
|
||||
3. MCP scope: project scope default, user-scope opt-in (resolved D1); threat named in
|
||||
ACCESS_POLICY.md either way.
|
||||
4. Codex sandbox reality: runbook carries a Codex-specific preflight (approval mode /
|
||||
workspace-write + network consent) — the paste block warns the human they'll be asked.
|
||||
5. Hooks in settings.local.json (gitignored) + --repair.
|
||||
6. Upgrade story: FF5 re-render nudge; render never clobbers (backup on --force).
|
||||
7. Transcript corpus + answers privacy: corpus 0700 outside repo; interview.json committed
|
||||
(same sensitivity as rendered USER.md, which is committed); responsible-disclosure
|
||||
phrasing in CHANGELOG (functional, no attack-surface enumeration).
|
||||
8. Process invariants: CLAUDE.md dispatcher row edit → `bun run build:llms` same commit;
|
||||
KEY_FILES.md entries current-state prose; version-first PR title; ship via /ship;
|
||||
/document-release after.
|
||||
|
||||
### New/changed artifacts (paths)
|
||||
|
||||
| Path | New/changed |
|
||||
|---|---|
|
||||
| `BOOTSTRAP_FOR_AGENTS.md` | NEW root runbook (fetched by paste block) |
|
||||
| `README.md` | dedicated "For Codex" / "For Claude Code" paste-block sections, ordered Codex → Claude Code → OpenClaw/Hermes at equal weight (user decision 2026-08-09, supersedes D5's ordering; both platform paths preserved) |
|
||||
| `src/commands/bootstrap.ts` (+ `src/commands/bootstrap/*.ts`) | NEW dispatcher + subcommands |
|
||||
| `src/commands/hook.ts` | NEW `gbrain hook session-start|user-prompt|stop|session-end` |
|
||||
| `src/cli.ts` | CHANGED: `bootstrap`+`hook` in no-engine dispatch branch |
|
||||
| `src/core/bootstrap/{interview,render,private-repo,verify}.ts` | NEW (TS ports) |
|
||||
| `src/core/secret-scan.ts` | NEW (shared: sources push, harden, verify) |
|
||||
| `src/commands/sources.ts` + `src/core/brain-repo-durability.ts` | CHANGED: `sources push`, scan-gated hook/cron |
|
||||
| `src/core/context/resolve-ipc.ts` + `src/mcp/server.ts` | CHANGED: IPC v2 `turn_context` (back-compat) |
|
||||
| `src/core/context/turn-context.ts` | NEW server-side block assembly |
|
||||
| `src/core/transcripts/claude-code-jsonl.ts` | NEW parser + corpus writer |
|
||||
| `templates/{MEMORY,AGENTS,CLAUDE,GITHUB,memory-README}.md.template` + `gitignore.template` + enriched SOUL/USER/HEARTBEAT/ACCESS_POLICY | NEW/CHANGED (generic, scrubbed) |
|
||||
| `templates/bootstrap/questions.json` | NEW shared question bank |
|
||||
| `skills/soul-audit/SKILL.md` | CHANGED: re-run surface over the bank |
|
||||
| `src/commands/doctor.ts` | CHANGED: hooks heartbeat, dual-serve, sync checks |
|
||||
| `docs/guides/bootstrap.md` + docs/mcp/ + connect-coding-agent.md cross-links | NEW/CHANGED |
|
||||
| `scripts/check-bootstrap-templates.sh` | NEW CI guard (token↔bank bijection + placeholder-only assertion) |
|
||||
| `docs/architecture/KEY_FILES.md`, `CLAUDE.md` (+build:llms) | CHANGED |
|
||||
| `test/bootstrap-*.test.ts`, `test/hook-command.test.ts`, `test/secret-scan.test.ts`, `test/e2e/bootstrap-lifecycle.test.ts`, IPC back-compat tests | NEW |
|
||||
|
||||
### Port map (codex-as-agent → gbrain)
|
||||
|
||||
interview.mjs → core/bootstrap/interview.ts · render-templates.mjs → core/bootstrap/render.ts
|
||||
· setup-private-repo.mjs → core/bootstrap/private-repo.ts · verify-install.mjs (+
|
||||
install-gbrain.mjs round-trip) → core/bootstrap/verify.ts · git-sync.mjs → secret-scan.ts +
|
||||
sources push · questions.json → templates/bootstrap/questions.json (scrubbed) · AGENTS.md
|
||||
gates 0–7 / SOUL/USER/MEMORY/HEARTBEAT/GITHUB templates → REFERENCE structure, REWRITE
|
||||
content (privacy rule) · install-skills.mjs → REFERENCE (skillpack scaffold exists) ·
|
||||
cron fleet/cron-doctor → FF4 · codex-plugin-spec one-file-owns-format pattern → governs the
|
||||
settings.local.json + config.toml writers (single module owns each host format).
|
||||
|
||||
### Verification (how we know it works end-to-end)
|
||||
|
||||
- Unit: interview gate exit codes; render token hard-fail/no-clobber/backup; secret-scan
|
||||
fixture corpus (positives + benign lookalikes); hook stdin→JSON contract; IPC v1↔v2
|
||||
back-compat both directions; questions.json↔template token bijection.
|
||||
- E2E: full lifecycle in temp dir (sandboxed GBRAIN_HOME, PATH-shimmed fake gh/claude/codex
|
||||
recording invocations): render → repo → verify exit 0; idempotency (second run no-op);
|
||||
kill-mid-phase → `status` resumes. **hook-under-serve** and **lock-contention** pins
|
||||
(permanent). Engine parity for turn-context on both engines.
|
||||
- Manual acceptance at ship: fresh macOS account, real paste, both harnesses, timed
|
||||
(≤15 min, ≤3 human actions).
|
||||
|
||||
### Resolved user decisions (Garry, 2026-08-07)
|
||||
|
||||
- **D1 = A. MCP scope: project scope default, user-scope opt-in** (consent question during
|
||||
bootstrap; threat named in ACCESS_POLICY.md).
|
||||
- **D2 = C. DB: global `~/.gbrain` default + documented `--isolated` escape hatch.**
|
||||
`gbrain bootstrap --isolated` threads `GBRAIN_HOME=<workspace>/.gbrain` through init, MCP
|
||||
registration env (`claude mcp add -e` / codex config env), and the hook commands in
|
||||
settings.local.json. Port install-gbrain.mjs's guards: GBRAIN_HOME does NOT isolate
|
||||
`sync.repo_path` (set it explicitly), strip ambient GBRAIN_DATABASE_URL/DATABASE_URL/
|
||||
GBRAIN_BRAIN_ID, assert database_path is inside the workspace. `.gbrain/` already
|
||||
gitignored by the rendered template.
|
||||
- **D3 = B with consent. 15-min scan-gated commit+push cron is the default posture, but the
|
||||
runbook ASKS PERMISSION before installing it** ("Enable background persistence? Installs a
|
||||
15-min launchd/cron job that commits + pushes this workspace, secret-scan-gated"). The
|
||||
SessionEnd-hook push stays as the always-available, no-daemon backstop and is the fallback
|
||||
when the cron is declined — persistence never silently disappears. Autopilot posture
|
||||
unchanged: off on PGLite, recommended on Postgres.
|
||||
- **D4 = B. ALL bundled skills scaffold in** (`skillpack scaffold --all`). The onboarding
|
||||
ENDS with a next-steps block in the completion manifest: the inert-skill report (which
|
||||
installed skills are dormant for lack of which API key), the exact keys to add and where,
|
||||
and pointers to soul-audit (deepen identity this week) + cold-start (fill the brain with
|
||||
your data). The routing-table-size concern is mitigated by frontmatter-trigger routing
|
||||
(authoritative since v0.36) and noted for a future curated-profile fast-follow if dispatch
|
||||
accuracy suffers in practice.
|
||||
- **D5 = Codex/Claude-Code-scoped placement.** [Superseded by the 2026-08-09 user
|
||||
decision — see as-shipped delta 6 and the artifact table's README row.] This is NOT the new headline install —
|
||||
most users still use GBrain with OpenClaw/Hermes, so `INSTALL_FOR_AGENTS.md` remains the
|
||||
primary paste path at the top of the README. The bootstrap paste block becomes the
|
||||
flagship "For Codex" / "For Claude Code" README sections, ahead of the OpenClaw/Hermes path at equal weight (and
|
||||
docs/tutorials/connect-coding-agent.md cross-links it). Command name stays
|
||||
`gbrain bootstrap`; paste block is pinned to a release tag (supply-chain integrity).
|
||||
BOOTSTRAP_FOR_AGENTS.md opens with a scope note: "For Claude Code / Codex. Running
|
||||
OpenClaw or Hermes? Use INSTALL_FOR_AGENTS.md instead."
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Agent Bootstrap — Spike Instrument (build order 0)
|
||||
|
||||
Manual validation that gates door-1 (Codex/ChatGPT desktop) ship. Run on a machine
|
||||
the maintainer does not own, with fresh accounts. Owner: the maintainer. Timebox:
|
||||
~1 week wall-clock. Outcomes feed the design doc's gate
|
||||
([AGENT_BOOTSTRAP_DESIGN.md](AGENT_BOOTSTRAP_DESIGN.md)).
|
||||
|
||||
## Exit questions and pass bars
|
||||
|
||||
**#1 (blocking) — the write seam.** What reliably persists memory per turn/session
|
||||
on the ChatGPT-desktop surface?
|
||||
- Protocol: run 20 sessions across ≥3 days of ordinary use. Each session must
|
||||
produce at least one durable write (a page/fact retrievable in the NEXT session).
|
||||
- Pass: 0 durable-write failures in 20 sessions; else extend to 50 and log every
|
||||
failure's cause (crash, sleep, approval friction, format drift, model forgot).
|
||||
- Below the bar → door 1 demotes to documented beta (Codex CLI unaffected).
|
||||
|
||||
**#2 — the read seam.** Is injected/pulled context demonstrably present at turn
|
||||
start? Pass: context block present (or the degraded pull-mode documented as
|
||||
door-1's v1 behavior). Pass expands the greeting digest to door 1.
|
||||
|
||||
**#3 — capability surface.** Record, with screenshots: local folder access
|
||||
(yes/no/how), MCP registration path (config file? `codex mcp add`? UI?), approval
|
||||
taps for each toolchain step (count them), connector availability for
|
||||
email/calendar (yes/no/degraded).
|
||||
|
||||
**#4 — quota.** Per harness the doors run on (Claude Code: Max plan; Codex: the
|
||||
ChatGPT plan): log each day's usage-meter readings during the pilot.
|
||||
- Load model: ordinary sessions + hooks + one session-triggered schedule.
|
||||
- Pass: a p90 day consumes ≤10% of the weekly allowance (per-door; one harness
|
||||
failing cuts schedule scope for that door only).
|
||||
- Measurement: the harness's own usage UI (screenshot at day start/end) + a tally
|
||||
of sessions/turns from the transcript dir. No telemetry — this is a manual
|
||||
instrument by design.
|
||||
|
||||
**#5 — TTFM baseline.** One full paste-to-verified install, timed. Count every
|
||||
human action (paste / auth click / interview answers / consents / approval taps).
|
||||
Toolchain download time recorded separately (excluded from the 15-minute target).
|
||||
|
||||
## Pilot metrics (continue for 2 weeks after the spike)
|
||||
|
||||
Per week, from session review (screen recordings + self-report — no telemetry):
|
||||
correct-write rate (things worth remembering that got written), correct-recall
|
||||
rate (recalls that were right), false-memory incidents (recalled things that were
|
||||
wrong), correction round-trips (corrections that stuck as standing rules). The
|
||||
0-failures-in-20 bar is the minimum to START the pilot, not the proof — the pilot
|
||||
is the sample.
|
||||
|
||||
## Log template (one row per session)
|
||||
|
||||
| # | date | door | duration | writes attempted | writes durable | recalls right/wrong | approvals | notes |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
|
||||
## Deliverable
|
||||
|
||||
A filled copy of this doc committed as `AGENT_BOOTSTRAP_SPIKE_RESULTS.md`
|
||||
(scrubbed: no real names beyond the maintainer, no account identifiers), plus the
|
||||
gate decision recorded in the design doc: door-1 ships full / ships as documented
|
||||
beta / schedule scope cut per quota.
|
||||
|
||||
**Gate status:** not yet run — no `AGENT_BOOTSTRAP_SPIKE_RESULTS.md` is committed,
|
||||
so no gate decision is recorded and door 1 has not been promoted past the
|
||||
documented-beta bar by this instrument. Update this line when the results land.
|
||||
@@ -0,0 +1,406 @@
|
||||
# Brain currency — fix the incident, then build the ladder
|
||||
Generated by /plan-ceo-review on 2026-08-10
|
||||
Rev 3, after two adversarial spec-review rounds (6/10 → 7/10) and an independent outside voice.
|
||||
Branch: garrytan/gbrain-commit-indexing | Mode: SELECTIVE EXPANSION
|
||||
Repo: garrytan/gbrain
|
||||
|
||||
**Citation convention:** repo-relative paths. `src/core/sync.ts` (540 lines) and
|
||||
`src/commands/sync.ts` (5804 lines) are different files; both are cited.
|
||||
|
||||
## Origin
|
||||
|
||||
An investigation into "how does gbrain pick up new commits from GitHub" found it never
|
||||
talks to GitHub. It diffs `git diff last_commit..HEAD` against a **local checkout**
|
||||
(`src/core/sync-delta.ts:113`). Getting remote commits into that checkout is a separate,
|
||||
opt-in concern.
|
||||
|
||||
It then found worse: on the founder's machine `gbrain autopilot` was installed, died
|
||||
2026-05-31, and stayed dead **71 days** while three surfaces reported healthy.
|
||||
|
||||
**1. `autopilot --status` is an artifact-presence check.**
|
||||
`src/commands/autopilot.ts:1775-1786` — plist `existsSync` on darwin, crontab grep
|
||||
elsewhere. Never asks whether the job is loaded, the process alive, the baked `--repo`
|
||||
present, or the log fresh. Always exits 0.
|
||||
|
||||
**2. `doctor`'s `sync_freshness` computes the 71-day number and throws it away.**
|
||||
`src/core/source-health.ts:182-194`:
|
||||
|
||||
```ts
|
||||
const wallClockSeconds = Math.floor((nowMs - lastSyncMs) / 1000); // ← the 71 days
|
||||
if (wallClockSeconds < 0) return wallClockSeconds;
|
||||
if (contentMs !== null && Number.isFinite(contentMs)) {
|
||||
return contentMs <= lastSyncMs ? 0 : wallClockSeconds; // ← discarded
|
||||
}
|
||||
```
|
||||
|
||||
When the clone is unreachable, `src/commands/doctor.ts:4306-4344` routes the verdict here.
|
||||
The function measures *drain completeness*, not *staleness*. "We caught up when we last
|
||||
looked" and "we have not looked in 71 days" both return 0.
|
||||
|
||||
**3. `gbrain status` inherits it.** `src/commands/sync.ts:5440-5453` → `'fresh'` beside a
|
||||
71-day-old date, exit 0. (`gbrain sources status` does report the real lag in its LAG
|
||||
column, but has no warn line for it and no exit contract.)
|
||||
|
||||
**Root cause of the death:** `src/commands/migrate-engine.ts` (22,733 bytes) contains
|
||||
**zero** autopilot references. The Supabase-to-PGLite migration rewrote
|
||||
`~/.gbrain/config.json` while a daemon built on the old config kept running and died on
|
||||
`config.database_url`.
|
||||
|
||||
## The key insight the reviews converged on
|
||||
|
||||
The content comparison in #2 is not a bug someone forgot. `src/commands/doctor.ts:4288-4305`
|
||||
documents why it exists:
|
||||
|
||||
> a container restart wipes `local_path` ... **and since a no-op sync doesn't advance
|
||||
> `last_sync_at`**, every QUIET source read as stale/FAIL after a restart (score-sinking
|
||||
> alert storm; observed live: 16-source brain, 12 clones gone after a config-update
|
||||
> restart, doctor 70→30).
|
||||
|
||||
**The premise in bold was invalidated after that code was written.** v0.42.52.0 added a
|
||||
heartbeat at `src/commands/sync.ts:2287-2298`:
|
||||
|
||||
```ts
|
||||
// bump last_sync_at as a heartbeat on every successful 0-changes sync...
|
||||
if (opts.sourceId) {
|
||||
await engine.executeRaw(`UPDATE sources SET last_sync_at = now() WHERE id = $1`, [opts.sourceId]);
|
||||
}
|
||||
```
|
||||
|
||||
A no-op sync **does** advance `last_sync_at` now. So a quiet source that is being checked
|
||||
has a recent `last_sync_at` and survives a wall-clock ceiling; the 71-day case has an old
|
||||
one because **no sync ran at all**. The two cases are now distinguishable, and the
|
||||
fallback's justification has expired.
|
||||
|
||||
That is the whole incident: a wall-clock ceiling on the discard branch, in one pure
|
||||
function that `doctor`, `gbrain status`, and `sources status` all call. It fixes all three
|
||||
by construction, with no new table, no new command, and no migration.
|
||||
|
||||
It also means **the heartbeat this plan originally proposed to build already ships.** A
|
||||
separate `live_ticks` table would be a fourth status surface on a fifth data source,
|
||||
curing "three surfaces disagreed" by adding one more that can disagree.
|
||||
|
||||
## Base branch
|
||||
|
||||
The whole wave (PR-A, PR-B, PR-C) is based on
|
||||
`garrytan/codex-as-agent-default-install`, not `master`. That branch carries the
|
||||
bootstrap surface (`src/core/bootstrap/{host-specs,hooks}.ts`, `detectHarness()`) that
|
||||
PR-B's harness tier needs, so **PR-B is not blocked** — an earlier revision of this doc
|
||||
assumed it was.
|
||||
|
||||
That branch moves frequently; re-fetch before comparing anything against it. A stale
|
||||
remote-tracking ref is an easy way to reach a confidently wrong conclusion here.
|
||||
|
||||
## Sequencing (decided)
|
||||
|
||||
Three PRs. Nothing is cut; the order changed.
|
||||
|
||||
### PR-A — close the incident (ships first)
|
||||
|
||||
1. **Wall-clock ceiling** in `lagFromContentMs` (`src/core/source-health.ts:189`): return
|
||||
`wallClockSeconds` once it exceeds an absolute bound regardless of the content
|
||||
comparison. Bound is a named env knob per repo convention
|
||||
(`GBRAIN_STALENESS_CEILING_HOURS`, default 72, matching the existing
|
||||
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`).
|
||||
2. **Regression test** (acceptance criterion 1 below).
|
||||
3. **E3** — `src/commands/migrate-engine.ts` reconciles the running daemon.
|
||||
4. **Wrapper self-disable** — `src/commands/autopilot.ts:1314-1359`. Adapted from
|
||||
`src/core/brain-repo-durability.ts:509-512`, NOT copied: two corrections the
|
||||
engineering review established.
|
||||
- Predicate is `[ ! -d "$repo" ]`, not `[ ! -d "$repo/.git" ]`. `--repo` may be a
|
||||
subdirectory of the checkout (sync resolves the root itself by walking up), and
|
||||
`.git` is a FILE in worktrees and submodules — either shape would self-disable a
|
||||
healthy install.
|
||||
- `exit 0` is correct for the durability wrapper because launchd fires it on
|
||||
`StartInterval` (one shot). Autopilot runs under `KeepAlive=true` +
|
||||
`ThrottleInterval=60` and systemd `Restart=always`, where exiting disables nothing
|
||||
and instead produces a silent respawn-every-60s loop. The wrapper must
|
||||
`launchctl bootout` / `systemctl --user disable --now` itself on those targets and
|
||||
drop a marker that `--status` surfaces.
|
||||
5. **Reconnect classifier** — `src/commands/autopilot.ts:58-78`; a JS `TypeError` must not
|
||||
substring-match as a config verdict.
|
||||
6. **`autopilot --status` reads the heartbeat** instead of `existsSync`, and exits nonzero
|
||||
when stale.
|
||||
7. **E8 hygiene** — test-run pollution of `~/.gbrain/sync-failures.jsonl`;
|
||||
`buildSyncManifest` (`src/core/sync.ts:105-140`) dropping git **`T`** (typechange).
|
||||
Narrowed: `src/core/sync-delta.ts:130` passes `-M` only, so `C` is unreachable without
|
||||
`--find-copies` and `U` needs a conflicted worktree. `C`/`U` handled defensively.
|
||||
|
||||
**Not in PR-A:** the lockfile-leak fix. Removing the leaked `~/.gbrain/autopilot.lock`
|
||||
deletes the signal that distinguishes *crashed* from *never installed*
|
||||
(`src/commands/status.ts:595-598`) before its replacement exists. It lands in PR-C
|
||||
alongside `live status`.
|
||||
|
||||
### PR-B — the `harness` tier alone
|
||||
|
||||
The harness tier is the only tier the modal gbrain user can actually run (PGLite default,
|
||||
desktop harness, behind NAT), so it ships alone and early rather than buried inside the
|
||||
cathedral. Reuses `src/core/bootstrap/{host-specs,hooks}.ts`, which the base branch
|
||||
already provides.
|
||||
|
||||
### PR-C — the ladder
|
||||
|
||||
`live` command family, `live.mode` bundle, shape detection, `cron`/`daemon`/`webhook`
|
||||
tiers, advisor collector, `init` offer, watch tier, shared `os-scheduler.ts`, and the
|
||||
lockfile-leak fix. **`live_ticks` is re-examined here against the shipped
|
||||
`last_sync_at` heartbeat rather than assumed** — the burden is on the new table to justify
|
||||
itself.
|
||||
|
||||
## The constraint being satisfied (quoted so it can be checked)
|
||||
|
||||
`docs/designs/AGENT_BOOTSTRAP_PLAN.md` on `origin/garrytan/codex-as-agent-default-install`,
|
||||
decision **D9**:
|
||||
|
||||
> **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
|
||||
> daemon). OPT-IN: 15-min harden cron. **Autopilot NOT default on PGLite** (verified: its
|
||||
> sync/embed children would contend with every live serve for the single-writer lock, and
|
||||
> nothing handles `LiveServeLockError` politely today) — **recommended on Postgres**; any
|
||||
> future scheduled job must treat lock-held as skip-silently-and-log.
|
||||
|
||||
*Reconciliation:* D9 says "15-min harden cron"; the shipped default is **1800s / 30 min**
|
||||
(`src/core/brain-repo-durability.ts:76`, `:659`). D9's figure is stale. This plan uses 30.
|
||||
|
||||
This plan's decisions are labelled **L1..L14** to avoid collision with that document.
|
||||
|
||||
## PR-C design (carried forward, not yet committed to a diff)
|
||||
|
||||
### Tiers — five active plus `off`
|
||||
|
||||
| tier | mechanism | expected cadence | engine gate |
|
||||
|---|---|---|---|
|
||||
| `off` | nothing | n/a — `live status` exits **0** | — |
|
||||
| `harness` | agent hook / session boundary | event-driven, **age-exempt** | any (incl. Windows, containers) |
|
||||
| `webhook` | HMAC push from GitHub | event-driven, **age-exempt**; paired keepalive `cron` supplies the age signal | any + reachable `serve --http` |
|
||||
| `cron` | OS scheduler | declared `expected_cadence_seconds` | any; **PGLite floor 1800s + lock-aware skip** |
|
||||
| `daemon` | resident autopilot, `runCycle` | 300s | **Postgres only** (D9) |
|
||||
| `watch` | daemon + chokidar | **floor 300s for freshness purposes**, not the ~1s event latency | **Postgres only** (D9) |
|
||||
|
||||
Event-driven tiers are exempt from age-based failure; a webhook repo with no pushes for
|
||||
three days is healthy, not failed. `watch`'s freshness cadence is decoupled from its event
|
||||
latency so a GC pause is not a FAIL.
|
||||
|
||||
`off` is a first-class bundle member with `enabled: false`, copied from
|
||||
`src/core/pace-mode.ts:65-71`.
|
||||
|
||||
### L1 — Shape detection predicate
|
||||
|
||||
| Signal | Source | Meaning |
|
||||
|---|---|---|
|
||||
| engine | `config.engine` | `postgres` required for `daemon`/`watch` |
|
||||
| interactive desktop harness | `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT`, `CODEX_HOME`, `CODEX_SANDBOX`, `CODEX_CI` (**env only**) | any present → cap at `harness` |
|
||||
| long-lived host | `detectInstallTarget()` ∈ {`macos`, `linux-systemd`, `ephemeral-container`+injection point} | a reboot-surviving scheduler exists |
|
||||
| server posture | `serve --http` configured, or `minion_mode != 'off'` | corroborating, never sufficient alone |
|
||||
|
||||
`macos` is in the long-lived row deliberately: `detectInstallTarget()` returns `'macos'`
|
||||
unconditionally on darwin (`src/commands/autopilot.ts:1277`), and darwin is the platform
|
||||
of the origin incident. Omitting it would make the incident host permanently
|
||||
shape-ineligible.
|
||||
|
||||
**No filesystem probes for harness identity.** The `~/.claude/hooks/...` class of probe
|
||||
(`src/commands/autopilot.ts:1304`) is what false-positives today. Env vars only.
|
||||
|
||||
Any inconclusive read falls to `harness`, never `daemon`.
|
||||
|
||||
### L2 — `live status` exit codes
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| fresh, or `live.mode == off` | 0 |
|
||||
| PGLite lock held by a live `serve` (`blocked_by_serve`) | 0 |
|
||||
| tier enabled + heartbeat missing or stale | 1 |
|
||||
| drifted install, or DB **connect failure** | 2 |
|
||||
|
||||
`live.mode == off` exiting 0 is load-bearing: otherwise every fresh install exits nonzero,
|
||||
which is the `cycle_freshness` #2540 lesson (never-configured must not turn the surface
|
||||
red). And lock-held is **not** an outage: `src/core/pglite-engine.ts:444` acquires the file
|
||||
lock on every `connect()` and throws if it fails, so on the default engine with a resident
|
||||
`serve`, treating that as exit 2 would make FAIL the steady state.
|
||||
|
||||
### L3 — `skipped_locked` semantics
|
||||
|
||||
A tick that cannot acquire the PGLite lock **does not satisfy freshness and does not
|
||||
degrade it**. It is neutral: logged, not recorded as work-done, and not counted toward
|
||||
staleness for a grace window of 3 consecutive skips, after which the surface reports
|
||||
`blocked_by_serve` with the remediation inline. Treating it as work-done rebuilds the
|
||||
71-day false-green; treating it as failure makes the default engine permanently red.
|
||||
|
||||
### L4 — Scheduler ownership
|
||||
|
||||
Ownership lives in a sidecar `~/.gbrain/live-ownership.json`, **not** in an entry comment.
|
||||
On darwin both harden and autopilot install launchd **plists** (files, not comment-bearing
|
||||
crontab lines), so the `# gbrain:autopilot v0.11.0` marker convention does not generalize.
|
||||
The sidecar covers all install targets uniformly.
|
||||
|
||||
Three enumerated cases:
|
||||
|
||||
1. **Harden cron exists + pull opted in** → rewrite through `os-scheduler.ts`,
|
||||
`ownership=live-adopted`.
|
||||
2. **Harden cron exists + pull declined** → leave it entirely alone; install a separately
|
||||
labelled `live` entry. **This is the default and lands first**, so PR-C's `live on`
|
||||
never meets an existing harden cron without a rule.
|
||||
3. **Neither exists** → install a `live` entry, `ownership=live`.
|
||||
|
||||
`live off` removes only entries `live` created and reverts adopted ones to harden.
|
||||
|
||||
Pre-existing `gbrain autopilot` installs are **migrated, not orphaned**: first `live
|
||||
status` after upgrade reports `tier: daemon (legacy autopilot)` and offers one-time
|
||||
adoption.
|
||||
|
||||
### L5 — Op scopes
|
||||
|
||||
| Op | scope | localOnly | remote |
|
||||
|---|---|---|---|
|
||||
| `live_status` | `read` | no | allowed; omits `local_path`, scheduler artifact paths, and log tail |
|
||||
| `live_tick` | `write` | **yes** | reject |
|
||||
| `live_on` / `live_off` | `admin` | **yes** | reject |
|
||||
| `live_self_heal` | `admin` | **yes** | reject |
|
||||
|
||||
Self-heal walks a **DB-supplied** `local_path` and then writes a scheduler entry.
|
||||
`src/commands/doctor.ts` already gates its git short-circuit on `localOnly === true`
|
||||
(*"a remote-callable code path must NOT walk DB-supplied `local_path` values with
|
||||
subprocess calls"*). Self-heal honors that and additionally requires a realpath match
|
||||
against the anchor via `isAnchorOwnedSyncPath` (`src/commands/sync.ts:1296`).
|
||||
|
||||
**Bootstrap paradox, acknowledged:** if the broken thing is the scheduler entry, a
|
||||
scheduled self-heal never runs. Non-scheduled triggers are the `harness` tier (PR-B) and
|
||||
an explicit `gbrain live doctor`. PR-C ships self-heal with both, not with a scheduled
|
||||
trigger alone.
|
||||
|
||||
### L6 — Revert
|
||||
|
||||
A code revert leaves plists, crontab lines, systemd units, and (E1) a GitHub webhook
|
||||
installed and unowned. Therefore:
|
||||
|
||||
- **Revert requires `gbrain live off` first** on any enabled host. Stated in the PR body.
|
||||
- The generated wrapper self-disables on a **marker file** written by `live on` and removed
|
||||
by `live off`. Not a `gbrain live --help` probe: that adds a process spawn per tick and
|
||||
assumes an exit code the CLI does not guarantee.
|
||||
- The migration, if `live_ticks` survives PR-C's re-examination, is additive and uses the
|
||||
**next free version at implementation time** (125 is the current max; two waves may land
|
||||
first).
|
||||
|
||||
### L7 — E5 must not use `nag-state.ts`
|
||||
|
||||
`src/core/skillpack/nag-state.ts` is skillpack-scoped (schema `gbrain-skillpack-nag-v1`,
|
||||
entries keyed on `pack_version`, `DEFAULT_NAG_CEILING = 3`, suppressed thereafter). Wiring
|
||||
a dead-sync alarm through it means a genuinely broken brain goes silent after three
|
||||
notices, which is a suppression mechanism for the exact failure mode whose defining
|
||||
property was 71 days of silence.
|
||||
|
||||
E5 instead uses a **rate limit, not a ceiling**: at most once per session, never
|
||||
suppressed permanently, escalating in terseness rather than disappearing.
|
||||
|
||||
### L8 — E1 webhook dependencies (previously unpriced)
|
||||
|
||||
Creating a GitHub webhook programmatically needs an `admin:repo_hook` token. No
|
||||
acquisition, storage, scope, or rotation story existed. Therefore E1 ships in **manual
|
||||
mode only**: `live on --tier webhook` generates the secret, resolves and prints the payload
|
||||
URL, and the user pastes it into GitHub, matching what `gbrain sources webhook set`
|
||||
(`src/commands/sources.ts:909-916`) already does. No token, no remote hook creation, no
|
||||
`live off` remote deletion problem.
|
||||
|
||||
The "verified test ping" must originate **from GitHub**, not locally. A local ping proves
|
||||
nothing through NAT and would be an artifact-presence check, the precise anti-pattern in
|
||||
the Origin section.
|
||||
|
||||
### L9 — `live_ticks` retention
|
||||
|
||||
If the table survives PR-C, the sweep runs **inside `live tick`** (bounded best-effort
|
||||
DELETE on a TTL), not only in the cycle's `purge` phase. `purge` is a `runCycle` phase
|
||||
(`src/core/cycle.ts:1434`), and `runCycle` runs only on `daemon`/`watch` — the `cron`,
|
||||
`webhook`, and `harness` tiers would accumulate forever.
|
||||
|
||||
## Scope decisions (all accepted; PR assignment added)
|
||||
|
||||
| # | Item | PR | Note |
|
||||
|---|---|---|---|
|
||||
| L10 | Approach C: full ladder | A/B/C | user chose the cathedral; resequenced, not cut |
|
||||
| L11 | Tier default keys on deployment shape, not vendor | C | Hermes has zero detectable signal |
|
||||
| E1 | Webhook tier, **manual mode** (L8) | C | |
|
||||
| E2 | Self-heal with `.tmp`+rename+`.bak` rollback | C | bootstrap paradox handled per L5 |
|
||||
| E3 | `migrate-engine` reconciles the daemon | **A** | the literal root cause |
|
||||
| E4 | Pull cron adoption per L4, separate opt-in per L12 | C | |
|
||||
| E5 | Agent-facing staleness, rate-limited not nag-ceilinged (L7) | C | |
|
||||
| E6 | Windows hard error naming `--tier harness` | C | `detectInstallTarget()` has no win32 branch |
|
||||
| E7 | `live_ticks` — **re-examined, not assumed** | C | the shipped `last_sync_at` heartbeat may suffice |
|
||||
| E8 | Hygiene, narrowed to git `T` | **A** | |
|
||||
|
||||
### L12 — E4's pull cron is an autonomy question
|
||||
|
||||
`docs/guides/upgrades-auto-update.md:41-43` states *"`auto` is deliberately NOT a default
|
||||
anywhere — it's an explicit autonomy grant, because applying code from GitHub unattended
|
||||
is, by design, remote code execution."* This plan does **not** flip `self_upgrade.mode`.
|
||||
|
||||
E4 schedules `git pull` every 30 minutes. That is content, not code, and durability keeps
|
||||
gbrain's hooks local and untracked so a pulled commit cannot rewrite executable hook code.
|
||||
But it is still unattended network fetch into a directory gbrain runs tooling against.
|
||||
Therefore the pull cron is a **separate opt-in from the tier**, proposed and explained by
|
||||
`live on`, never silently bundled.
|
||||
|
||||
### L13 — The directive's internal tension, stated
|
||||
|
||||
"OpenClaw and Hermes default to always-up-to-date" sits against L1's "shape detection
|
||||
recommends, never installs" and D9's "almost nothing on by default." These are reconciled
|
||||
by scope: shape detection sets the **recommended tier** and pre-selects it in the `init`
|
||||
consent prompt, so a shape-matching host is one keystroke from always-on rather than
|
||||
silently converted. Whether that consent is required on **upgrade** as well as fresh
|
||||
install is **open decision F1** below.
|
||||
|
||||
## L14 — Acceptance criteria
|
||||
|
||||
1. **Three-surface honesty.** A source whose `local_path` is deleted, whose `last_sync_at`
|
||||
is 71 days old, whose `newest_content_at` is **non-NULL**, and whose `chunker_version`
|
||||
**matches** must report stale/fail from `doctor` and `gbrain status`, and must surface
|
||||
the lag in `sources status`. Both fixture preconditions are required: a NULL
|
||||
`newest_content_at` already falls through to wall-clock
|
||||
(`src/commands/doctor.ts:4335-4342`) and a chunker mismatch already disables the
|
||||
fallback (`:4318`), so a naive fixture passes against unfixed code.
|
||||
*`sources status` is held to output, not exit code — it has no exit contract today and
|
||||
adding one is an undeclared breaking change to a read-only dashboard.*
|
||||
2. **Quiet-source non-regression.** A source with a recent `last_sync_at`, an unreachable
|
||||
clone, and no new content must still report **OK**. This is the 16-source / doctor
|
||||
70→30 incident; the ceiling must not re-light it.
|
||||
3. **Install honesty** (PR-C). `live on --tier cron` verifies the job loaded and exits
|
||||
nonzero if not; deleting the repo makes `live status` exit nonzero and name the path;
|
||||
`live off` leaves nothing.
|
||||
4. **Concurrency** (PR-C, **Postgres only**). Two tiers ticking produce one import and one
|
||||
neutral skip record. On PGLite the second process cannot open the DB at all, so the
|
||||
defined outcome is a log line and no row.
|
||||
5. **Watch tier** (PR-C). E2E expects **queued-job-failure**, not synchronous rejection —
|
||||
`ingest_capture` enqueues and returns.
|
||||
6. **Engine parity** (PR-C, if `live_ticks` survives). DDL identical in both engines,
|
||||
pinned by `test/e2e/engine-parity.test.ts`; bootstrap probe-set entry pinned by
|
||||
`test/schema-bootstrap-coverage.test.ts`.
|
||||
|
||||
## Open decisions (unanswered — do not silently default)
|
||||
|
||||
- **F1.** Does shape-detected always-on apply on **upgrade** as well as fresh install?
|
||||
Codebase precedent (`src/commands/upgrade.ts:513-516`, `mcp.publish_skills`) is
|
||||
new-installs-only with a one-time prompt for existing. Gates PR-C only.
|
||||
- **F2.** Command noun and config key: `gbrain live` + `live.mode` (requires renaming the
|
||||
existing `liveSyncStatus` helper at `src/core/db-lock.ts:749` to `syncInProgress`, two
|
||||
call sites) vs `gbrain sync live` + `sync.live.mode`. Gates PR-C only.
|
||||
|
||||
## Deferred to TODOS.md
|
||||
|
||||
- Full Windows `schtasks` tier — no test machine; `harness` covers it
|
||||
- Per-tier cost meter for `daemon` / `watch`
|
||||
- Cross-OS scheduler probing as a `live status` diagnostic (TODO-V19-D stays open; the
|
||||
heartbeat makes it optional rather than load-bearing)
|
||||
- Centralize the three freshness call sites onto one `freshnessVerdict()` helper
|
||||
(existing filed P3, now partially satisfied by PR-A's single-function fix)
|
||||
|
||||
## Dream state delta
|
||||
|
||||
PR-A leaves brain currency *honest*. PR-B leaves it *workable for the modal user*. PR-C
|
||||
leaves it *a product feature*. Remaining gap to the 12-month ideal: currency is still
|
||||
something the user turns on, not something simply true of a configured brain. F1 is the
|
||||
decision that closes or preserves that gap.
|
||||
|
||||
## Reviewer concerns (unresolved after 3 iterations)
|
||||
|
||||
- **Scope, from both reviewers:** PR-C remains large (command family, mode bundle, shape
|
||||
detector, three tiers, advisor collector, init prompt, webhook, watch tier, scheduler
|
||||
extraction). The PR-A/B/C split answers the sequencing objection but not the size of C
|
||||
itself. Revisit at PR-C planning with the incident already fixed.
|
||||
- **`live_ticks` necessity** is explicitly unresolved and assigned to PR-C rather than
|
||||
decided here.
|
||||
+31
-29
@@ -8,11 +8,19 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
|
||||
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
||||
that lives on top of that format.
|
||||
|
||||
## v0.41 update — the LOOP is now real
|
||||
If you're touching **memory behavior** rather than retrieval ranking — the
|
||||
Retrieval Reflex push path, conversation→facts write-back, cross-session
|
||||
continuity, source isolation — the gate for that layer is **BrainBench**
|
||||
(`gbrain eval brainbench`): see [`eval/BRAINBENCH.md`](./eval/BRAINBENCH.md).
|
||||
The two stack: this doc's capture→baseline→replay loop gates query-level
|
||||
result sets; BrainBench gates the memory behaviors above them, with its own
|
||||
committed baseline (`evals/brainbench/baselines/main.json`) compared against
|
||||
MAIN's copy in CI so a PR can't self-approve a regression.
|
||||
|
||||
Before v0.41, you could capture eval rows and replay them but nothing
|
||||
stitched them into a gate. `gbrain bench publish` + `gbrain eval gate`
|
||||
close the loop. Two gates:
|
||||
## The eval gate loop
|
||||
|
||||
`gbrain bench publish` + `gbrain eval gate` stitch captured eval rows into
|
||||
a pass/fail gate. Two gates:
|
||||
|
||||
- **Regression gate** (`--baseline X.baseline.ndjson`): replays a baseline
|
||||
you captured against your current brain. Catches: "did my refactor break
|
||||
@@ -39,7 +47,7 @@ gbrain bench publish --from /tmp/captured.ndjson --to ~/.gbrain/baselines/person
|
||||
gbrain eval gate --baseline ~/.gbrain/baselines/personal.baseline.ndjson
|
||||
```
|
||||
|
||||
### Privacy posture (D9)
|
||||
### Privacy posture
|
||||
|
||||
**Public baselines in `gbrain-evals` are hermetic-synthetic ONLY.** Real
|
||||
user captures stay local in `~/.gbrain/baselines/`. The boundary is
|
||||
@@ -131,14 +139,9 @@ gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
|
||||
```
|
||||
|
||||
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": true}} // force on
|
||||
{"eval": {"capture": false}} // force off
|
||||
```
|
||||
|
||||
Explicit config beats the env var both directions.
|
||||
The full on/off resolution order (config beats env var, both directions) is
|
||||
documented once in [`eval-capture.md`](./eval-capture.md) — that file is the
|
||||
capture contract.
|
||||
|
||||
## The 4-command loop
|
||||
|
||||
@@ -205,7 +208,7 @@ retrieval, and which queries did it move most?"
|
||||
|
||||
For a third evaluation axis — public benchmark, ground-truth labels, full
|
||||
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
|
||||
<dataset.jsonl>` runs the LongMemEval benchmark against gbrain's
|
||||
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
|
||||
imported, the question asked, the hypothesis emitted as JSONL — exactly the
|
||||
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
|
||||
@@ -337,7 +340,7 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
|
||||
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
|
||||
|
||||
## Public benchmarks: LongMemEval (v0.28.8)
|
||||
## Public benchmarks: LongMemEval
|
||||
|
||||
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
|
||||
benchmark directly against gbrain's hybrid retrieval. Different evaluation
|
||||
@@ -398,7 +401,7 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
|
||||
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
|
||||
LLM latency.
|
||||
|
||||
## Measuring brain consistency over time (v0.32.6)
|
||||
## Measuring brain consistency over time
|
||||
|
||||
`gbrain eval suspected-contradictions` is a complementary measurement
|
||||
instrument: it samples retrieval results for unmarked semantic
|
||||
@@ -435,20 +438,19 @@ commands per high-severity finding.
|
||||
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
|
||||
decision criteria gated on Wilson CI lower-bound.
|
||||
|
||||
## v0.40.1.0 Track D — Eval infrastructure
|
||||
## Eval infrastructure: by-type breakdowns, the hermetic gate, batch scoring
|
||||
|
||||
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
|
||||
covers the dev loop that uses them and the gates they enforce.
|
||||
Three further eval surfaces, and the dev loop that uses them.
|
||||
|
||||
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
|
||||
|
||||
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
|
||||
surfaces it in machine-readable form. Two additive changes:
|
||||
LongMemEval computes per-question-type recall internally, and surfaces it in
|
||||
machine-readable form:
|
||||
|
||||
1. Every per-question JSONL row now includes a `question: string` field so the
|
||||
1. Every per-question JSONL row includes a `question: string` field so the
|
||||
`gbrain eval cross-modal --batch` consumer (below) can read it without
|
||||
joining back against the source dataset.
|
||||
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
|
||||
2. The `--by-type` flag emits a final aggregate line keyed by `question_type`:
|
||||
|
||||
```json
|
||||
{"schema_version": 1, "kind": "by_type_summary",
|
||||
@@ -480,11 +482,11 @@ echo "exit=$?" # 1 if any type fell below 0.80
|
||||
|
||||
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
|
||||
|
||||
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
|
||||
silently regress retrieval." Replaces the original "replay against captured
|
||||
eval_candidates" design (which Codex caught as non-functional in CI — see
|
||||
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
|
||||
real-query version).
|
||||
The structural fix for "PRs touching `src/core/search/` silently regress
|
||||
retrieval." A "replay against captured eval_candidates" design can't work in
|
||||
CI (CI has no captured production queries), so the gate is hermetic; see the
|
||||
`contributor-mode CI capture` TODO in `TODOS.md` for the deferred
|
||||
real-query version.
|
||||
|
||||
How it works:
|
||||
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
|
||||
@@ -499,7 +501,7 @@ How it works:
|
||||
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
|
||||
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
|
||||
|
||||
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
|
||||
#### Refreshing the qrels fixture (the `Why:` discipline)
|
||||
|
||||
When CI fails because a legitimate ranking change moved expected slugs, the
|
||||
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
|
||||
|
||||
@@ -28,7 +28,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|---|---|---|
|
||||
| `--limit N` | 100 | Random sample of N takes from the brain. |
|
||||
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
|
||||
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
|
||||
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud rather than silently blowing the budget. |
|
||||
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
|
||||
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
|
||||
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
|
||||
@@ -73,8 +73,8 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|
||||
- `schema_version` — locks the contract. Adding optional fields is additive
|
||||
and compatible. Renaming, removing, or changing semantics bumps the version.
|
||||
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
|
||||
(codex review #3). When the rubric definition changes, both fields update,
|
||||
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch.
|
||||
When the rubric definition changes, both fields update,
|
||||
and trend mode groups runs accordingly so a stricter rubric doesn't
|
||||
silently look like a quality drop.
|
||||
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
|
||||
@@ -83,7 +83,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
models in `--models` doesn't change the sha (sort is stable).
|
||||
- `successes_per_cycle` — count of contributing models per cycle. A model
|
||||
contributes when (a) its JSON parsed AND (b) every declared rubric dim
|
||||
has a finite score (codex review #5 — missing-dim drops the contribution).
|
||||
has a finite score (a missing dim drops the whole contribution).
|
||||
- `verdict` — `pass` if every dim mean >= 7 AND every dim min across
|
||||
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
|
||||
2/3 models contributed complete scores.
|
||||
@@ -93,11 +93,11 @@ receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|
||||
## Receipt persistence
|
||||
|
||||
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
|
||||
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
|
||||
Receipts persist to **`eval_takes_quality_runs`** (the DB is authoritative)
|
||||
AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
|
||||
as a best-effort artifact. The DB row carries the full receipt JSON in the
|
||||
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
|
||||
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
|
||||
can still reconstruct via `loadReceiptFromDb`.
|
||||
|
||||
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
|
||||
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
# BrainBench — cross-harness memory conformance methodology
|
||||
|
||||
BrainBench generalizes gbrain's internal eval surface into a reproducible,
|
||||
cross-harness benchmark for agent memory. It operationalizes the four failure
|
||||
modes of the agent-memory thesis: **know-to-ask** (nobody has a push path),
|
||||
**push precision/recall** (the intrusion budget must be enforced),
|
||||
**write-back fidelity** (memory write is even less solved than read), and
|
||||
**cross-session continuity** (continuity that survives the harness hop).
|
||||
Every subsequent memory PR must move — or hold, with a recorded justification —
|
||||
a BrainBench number to merge.
|
||||
|
||||
Operator quickstart, corpus layout, and fixture-authoring rules live in
|
||||
[`evals/brainbench/README.md`](../../evals/brainbench/README.md). This document
|
||||
is the methodology: what the numbers mean, what they deliberately do not mean,
|
||||
and how the gate governs change.
|
||||
|
||||
## Seam disclosure (read this before comparing rows)
|
||||
|
||||
Every scoreboard row carries a `seam` column:
|
||||
|
||||
| Harness | Seam | What the row actually measures |
|
||||
|---|---|---|
|
||||
| `openclaw` | **production** | The shipped OpenClaw context-engine pipeline, byte-for-byte (`extractCandidates` → `resolveEntitiesToPointers`, 3-pointer budget, prior-context suppression, markdown pointer block). |
|
||||
| `claude-code` | **contract** | gbrain's memory primitives driven through the UserPromptSubmit hook wire contract (`{prompt, session_id, cwd}` in → `{hookSpecificOutput.additionalContext}` out, exported from `src/eval/brainbench/adapters/claude-code.ts`). 2-pointer budget; NO conversation memory — this row deliberately models the memoryless wire contract (suppression off), so the re-injection cost is visible as `false_fire_rate`; the shipped `gbrain hook user-prompt` layers transcript-based cross-turn dedupe on top of this same contract. |
|
||||
| `codex` | **contract** | The fragments model: a static entity-index preamble (computed once, slugs not counted as injections) + at most ONE per-turn fragment. Measures how much push quality degrades when injection is mostly static. |
|
||||
|
||||
**Contract rows do NOT measure third-party harness behavior.** They measure
|
||||
gbrain's primitives under each harness's injection-shape constraints. The rows
|
||||
are comparable because fixtures, brain, and gold are identical — only the seam
|
||||
contract varies. The real Claude Code integration has landed (`gbrain hook
|
||||
user-prompt`, registered by `gbrain bootstrap`); flipping this adapter to exec
|
||||
the real hook and report `production` numbers is a filed follow-up (TODOS.md —
|
||||
"Flip contract adapters to production"). Same for codex fragments when that
|
||||
integration lands. Also not graded, by design: the production orchestrator's
|
||||
config gate, integration heartbeat, and 1500 ms timeout wrapper.
|
||||
|
||||
All three adapters drive ONE shared pipeline (`adapters/shared.ts`) with
|
||||
declarative configs — comparability is structural, not disciplined.
|
||||
|
||||
## Metrics (formulas)
|
||||
|
||||
All micro-averaged per (harness × suite) cell; registered in
|
||||
`src/core/eval/metric-glossary.ts` (plain-English in
|
||||
[`METRIC_GLOSSARY.md`](METRIC_GLOSSARY.md)); JSON output carries one
|
||||
`_meta.metric_glossary` block.
|
||||
|
||||
- `know_to_ask_failure_rate` = |should-retrieve turns where injected ∩ (gold ∪ acceptable) = ∅| / |should-retrieve turns|. Lower better.
|
||||
- `false_fire_rate` = |stay-silent turns with any injection| / |stay-silent turns|. Lower better. Anti-gaming companion: "always inject" cannot win both.
|
||||
- `push_precision` = Σ|injected ∩ (gold ∪ acceptable)| / Σ|injected| over turns with injection. `acceptable_slugs` count for precision, not recall.
|
||||
- `push_recall` = Σ|injected ∩ gold| / Σ|gold| over should-retrieve turns. Pointer budgets cap this by design.
|
||||
- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
|
||||
- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
|
||||
- `continuity_rate` = |decision probes recalled by the reader| / |probes|, per READER harness. The writer fixture's decisions persist through the production write-back pipeline — which is harness-INDEPENDENT in v1 — so each pair preps once and every harness replays the read-only reader against the same persisted state (an ordered writer×reader sweep would rebuild byte-identical brains for identical scores). A probe succeeds via pointer injection or stored-fact keyword lookup. The per-writer axis activates when harness-specific write paths land.
|
||||
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant. Granularity disclosure: detection is slug-keyed, so it catches injection of slugs seeded ONLY in a foreign source; a same-slug cross-source CONTENT leak would require the engine's source-scoped SQL itself to fail, which the engine-layer source-isolation fuzz (gbrain-evals Cat 22) covers directly.
|
||||
- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
|
||||
- `extraction_recall` / `extraction_precision` — `--llm` runs only: the real extractor's output vs gold keyword probes.
|
||||
|
||||
### What know-to-ask deliberately means in v1
|
||||
|
||||
It grades the **deterministic injection decision** — the Reflex pipeline that
|
||||
ships at the seam. The agent never "knows to ask"; the reflex pushes. An
|
||||
agent-LLM-in-the-loop replay (did the *model* issue a retrieval call when the
|
||||
reflex stayed silent?) is **pre-registered as the `--live` extension**:
|
||||
fixture-compatible, seeded, N-repeat methodology — and unimplemented. No LLM
|
||||
grading is faked in v1.
|
||||
|
||||
### Difficulty is stratified on purpose
|
||||
|
||||
Several know-to-ask variants exercise documented v1 reflex limits (lowercase
|
||||
mentions, surname-only references — `src/core/context/entity-salience.ts`).
|
||||
Gold records what SHOULD happen; the committed baseline records what the
|
||||
current system does (`know_to_ask_failure_rate` ≈ 0.15 at v1). The gap is the
|
||||
measured roadmap, not a bug in the bench.
|
||||
|
||||
## Pre-registered expectations (v1, recorded before the first published run)
|
||||
|
||||
1. The production seam (openclaw) leads `push_recall` strictly: 3-pointer > 2-pointer > 1-fragment budgets. *(Observed at landing: 0.81 / 0.65 / 0.45.)*
|
||||
2. The no-suppression contract (claude-code) is the only seam with `false_fire_rate` > 0. *(Observed: 0.02–0.03.)*
|
||||
3. `write_back_fidelity` = 1.0 and `provenance_accuracy` = 1.0 in deterministic mode — the production pipeline must not lose or mis-attribute gold facts it was handed. Anything below 1.0 is a pipeline bug, not benchmark noise.
|
||||
4. `source_isolation_violations` = 0 everywhere.
|
||||
5. `push_precision` = 1.0 at v1 (exact-match resolution arms cannot inject an irrelevant page on this corpus); expected to dip below 1.0 when fuzzy/semantic resolution lands — that dip is the precision/recall trade made visible.
|
||||
|
||||
## Determinism & statistical posture
|
||||
|
||||
The harness is deterministic end-to-end: regex extraction + SQL resolution
|
||||
(zero LLM, zero embeddings — facts seed with NULL embeddings; keyword/alias
|
||||
arms carry retrieval), seeded PRNG corpus, one in-memory PGLite reset between
|
||||
fixtures. Two runs produce identical metrics, so N-repeat error bars are
|
||||
meaningless here (stddev = 0 by construction, the gbrain-evals "deterministic
|
||||
adapters" convention) and the gate can be exact: **any flipped gold item is a
|
||||
real behavior change.** Bootstrap/CI discipline applies to the future `--live`
|
||||
and `--llm` published runs, which are model-stochastic.
|
||||
|
||||
## Gate governance (decision 4 — why a PR can't self-approve)
|
||||
|
||||
CI (`.github/workflows/test.yml` `brainbench` job, local parity
|
||||
`scripts/ci-brainbench-gate.sh`) fetches the baseline **from main**
|
||||
(`git show origin/master:evals/brainbench/baselines/main.json`) and compares
|
||||
HEAD's fresh run against it:
|
||||
|
||||
- **Same `fixtures_hash`** → count-aware gate: any newly-failed gold item, any
|
||||
adverse gated-metric move, or any isolation violation fails (exit 1).
|
||||
- **Different hash** (the PR changed fixtures) → **corpus-bless mode**: the
|
||||
PR's committed baseline must EXACTLY match HEAD's actual run (the file
|
||||
cannot lie; exit 2 until `--update-baseline` is re-run), and any adverse
|
||||
move vs main's baseline requires a `justification` string in the committed
|
||||
baseline — visible in the PR diff, judged by the reviewer.
|
||||
- `--allow-regression "reason"` is the local one-off escape hatch; the reason
|
||||
is recorded in the run output. It is not available to CI.
|
||||
|
||||
The committed baseline is diff-stable by construction (metrics rounded to 4
|
||||
decimals, keys sorted, receipts excluded; the run CONFIG — holdout/llm/
|
||||
harness/suite sets — is bound into it, and comparisons across mismatched
|
||||
configs are inconclusive). Same-hash hardening: any committed-baseline edit
|
||||
without a fixture change must byte-match the actual run (receipts-backed), a
|
||||
regressing receipts-backed update still needs a `justification`, gold_total
|
||||
may not move at all under an unchanged corpus, and the CI script refuses a
|
||||
working-tree baseline deletion. Holdout fixtures (~15%) are excluded from the
|
||||
gate and scored only in published runs (`--include-holdout`).
|
||||
|
||||
Accepted residuals (review-enforced, by design): a `justification` string is
|
||||
judged by the human reviewer, not parsed; count-preserving corpus dilution
|
||||
(replacing hard fixtures with easy ones at equal gold_total) is visible only
|
||||
in the fixture diff; and the ratchet does not auto-tighten — improvements
|
||||
aren't banked into main's baseline until a PR updates it (a regression back
|
||||
to the stale baseline level passes; periodic re-baselining is the operator's
|
||||
job, filed as a TODO).
|
||||
|
||||
## Gold methodology
|
||||
|
||||
Gold derives from the corpus generator (the same PRNG step that authors a turn
|
||||
authors its annotation, so gold-vs-text drift is structurally impossible for
|
||||
generated fixtures), plus hand-authored spike fixtures that froze the schema.
|
||||
A 10% double-label validation pass (independent agent review of fixture text vs
|
||||
gold, blind to the generator's intent) is run at corpus-change time; its
|
||||
receipt is recorded in the corpus `_ledger.json` and any disagreement is a
|
||||
fixture bug to fix, not a tolerance to average over.
|
||||
|
||||
## Interop
|
||||
|
||||
- **Foreign runners (gbrain-evals):** the subprocess contract is
|
||||
`gbrain eval brainbench --fixtures DIR --gold DIR --json --out FILE`;
|
||||
schemas in `evals/brainbench/schema/`. The sibling gbrain-evals repo wires
|
||||
this as `eval/runner/brainbench-memory.ts` with a published scorecard.
|
||||
- **Memory-verbs conformance kit (Cathedral 1):** conformance scenarios
|
||||
convert to BrainBench fixtures via the published fixture schema
|
||||
(`schema_version` 1) once that wave lands — the conversion path is the
|
||||
schema itself; no bespoke importer is required.
|
||||
- **Naming note:** "BrainBench" historically also names the in-house
|
||||
retrieval corpus in the sibling gbrain-evals repo (the 145-query relational
|
||||
suite, Cat taxonomy) and `test/cathedral-ii-brainbench.test.ts` (v0.20.0
|
||||
code-graph recall pins). This suite — the cross-harness memory conformance
|
||||
bench — is the generalization the name now primarily refers to; the older
|
||||
references stand unchanged.
|
||||
|
||||
## Extends docs/eval-bench.md
|
||||
|
||||
The capture → baseline → replay loop in [`eval-bench.md`](../eval-bench.md)
|
||||
gates *retrieval result sets* at the query level. BrainBench gates the
|
||||
*memory behaviors* above them. The two share the receipts discipline and the
|
||||
.gbrain-evals run ledger (`EvalRunRecord` v3; brainbench records once per
|
||||
sweep under `mode: 'n/a'`).
|
||||
@@ -168,6 +168,96 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli
|
||||
|
||||
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
|
||||
|
||||
## BrainBench — Cross-Harness Memory Conformance
|
||||
|
||||
### Know-to-ask failure rate (BrainBench)
|
||||
|
||||
**Key:** `know_to_ask_failure_rate`
|
||||
|
||||
**Plain English:** Of the conversation turns where memory SHOULD have surfaced something unprompted, the fraction where nothing relevant was injected. This is the thesis failure mode every agent harness shares: the agent can't ask for what it doesn't know it forgot — the memory layer has to volunteer it.
|
||||
|
||||
**Range:** 0..1, LOWER is better. 0.15 means memory stayed silent on 15% of the turns where it had the answer.
|
||||
|
||||
### False-fire rate (BrainBench)
|
||||
|
||||
**Key:** `false_fire_rate`
|
||||
|
||||
**Plain English:** Of the turns where memory should have stayed SILENT, the fraction where it injected anyway. The anti-gaming companion to the know-to-ask rate — "always inject" would ace one and bomb the other. Silence beats noise.
|
||||
|
||||
**Range:** 0..1, LOWER is better.
|
||||
|
||||
### Push precision (BrainBench)
|
||||
|
||||
**Key:** `push_precision`
|
||||
|
||||
**Plain English:** Of everything the memory layer volunteered into context, what fraction was actually relevant to the turn? Micro-averaged over injected pointers, so a 3-pointer turn weighs three times a 1-pointer turn — the way a token budget experiences it.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Push recall (BrainBench)
|
||||
|
||||
**Key:** `push_recall`
|
||||
|
||||
**Plain English:** Of everything that SHOULD have been volunteered (the gold pointers), what fraction actually was? Pointer budgets cap this by design: a seam that may inject only 1 fragment cannot reach full recall on a 3-entity turn — that constraint is what the per-harness rows measure.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Write-back fidelity (BrainBench)
|
||||
|
||||
**Key:** `write_back_fidelity`
|
||||
|
||||
**Plain English:** Of the facts stated in a conversation, what fraction survived the PRODUCTION conversation→memory pipeline (segmentation, insertion, dedup) and are findable afterward with the right entity attached? Measures the write path users actually run, not a test-only insert.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Provenance accuracy (BrainBench)
|
||||
|
||||
**Key:** `provenance_accuracy`
|
||||
|
||||
**Plain English:** Of the facts that survived write-back, what fraction carry correct provenance — the right source tag, session id, and origin page? A fact you can't trace is a fact you can't trust, audit, or expire.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Cross-session continuity rate (BrainBench)
|
||||
|
||||
**Key:** `continuity_rate`
|
||||
|
||||
**Plain English:** A decision is recorded in one session and persisted through the production write path; a different harness asks about it later on the same brain. What fraction of those decision probes were recalled — by pointer injection or stored-fact lookup? This is the continuity-that-survives-the-harness-hop moat, measured.
|
||||
|
||||
**Range:** 0..1, higher is better. Scored per reader harness (the v1 write path is harness-independent, disclosed in docs/eval/BRAINBENCH.md).
|
||||
|
||||
### Source-isolation violations (BrainBench)
|
||||
|
||||
**Key:** `source_isolation_violations`
|
||||
|
||||
**Plain English:** Count of injected pointers that belong to a source other than the active one. Cross-source leakage is gbrain's must-never-violate invariant (a missed source filter is a data leak), so this gates at ZERO — any baseline, any run.
|
||||
|
||||
**Range:** 0..n, count. MUST be 0; any value above 0 fails the gate.
|
||||
|
||||
### Average injected tokens per turn (BrainBench)
|
||||
|
||||
**Key:** `avg_injected_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens of volunteered context per replayed turn (chars/4 heuristic). The intrusion-budget diagnostic: two seams with equal precision can differ 3x in how much context they spend to get it. Reported, not gated, until calibration data exists.
|
||||
|
||||
**Range:** 0..n tokens, judgment call — lower is cheaper, but starving the agent has its own cost. Non-gating.
|
||||
|
||||
### Extraction recall (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_recall`
|
||||
|
||||
**Plain English:** With the real LLM extractor running (instead of the deterministic gold extractor), what fraction of the gold facts did it actually extract and persist? Only scored in --llm runs — the hermetic CI gate never calls a model.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
### Extraction precision (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_precision`
|
||||
|
||||
**Plain English:** Of everything the real LLM extractor persisted, what fraction matches a gold fact? Low precision means the extractor invents or over-extracts — junk memory that pollutes future recall.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Search Mode Evaluation Methodology
|
||||
|
||||
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
|
||||
_How gbrain measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible — pinned datasets, recorded seeds, and the exact run commands below._
|
||||
|
||||
## 1. What this measures and what it doesn't
|
||||
|
||||
@@ -21,14 +21,15 @@ If you want to know how a mode behaves on YOUR brain, run `gbrain search stats -
|
||||
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
|
||||
- **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
|
||||
|
||||
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
No private brain content is used in any reported result. The NDJSON run records under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
|
||||
## 3. Sample selection
|
||||
|
||||
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
|
||||
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode bundle is the only independent variable. A mode Δ therefore measures the joint effect of every knob the bundles differ on — today that's `tokenBudget`, `expansion`, `relationalRetrieval` (the typed-edge fourth recall arm, ON for balanced/tokenmax, OFF for conservative), and `searchLimit`; the canonical diff is `MODE_BUNDLES` in `src/core/search/mode.ts`.
|
||||
- **Cache comparability across upgrades.** The query cache keys on a versioned knobs hash (`KNOBS_HASH_VERSION` in `mode.ts`) that folds in the active knob set + embedding column/provider, so one mode's cached results can't be served to another mode's queries — and a version bump makes prior rows unreachable (one-time miss spike). Cross-run comparisons that straddle a knobs-hash bump see a cold cache on the first re-run.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from a run's `--output` dumps.
|
||||
|
||||
## 4. Run procedure
|
||||
|
||||
@@ -46,14 +47,14 @@ gbrain eval run-all \
|
||||
--limit 500 \
|
||||
--budget-usd-retrieval 5 \
|
||||
--budget-usd-answer 20 \
|
||||
--output docs/eval/results/v0.32.3/
|
||||
--output docs/eval/results/<version>/
|
||||
|
||||
# Render the comparison.
|
||||
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
|
||||
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
|
||||
gbrain eval compare --md > docs/eval/results/<version>/README.md
|
||||
gbrain eval compare --json > docs/eval/results/<version>/comparison.json
|
||||
```
|
||||
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. When a release publishes eval numbers, the `--output` dumps under `docs/eval/results/<version>/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation. **No dumps are committed in the repo right now** — reproduce by running the commands above; determinism (§3) means your re-run matches the reported orderings.
|
||||
|
||||
## 5. Threats to validity
|
||||
|
||||
@@ -68,7 +69,7 @@ Honest list. We name what would let a critic dismiss the numbers.
|
||||
|
||||
## 6. Per-question raw outputs
|
||||
|
||||
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
|
||||
Every reported metric is reproducible from the NDJSON dumps a run writes to its `--output` directory (`docs/eval/results/<version>/` when a release publishes numbers; none are committed right now — see §4). The commit SHA in the methodology footer pins the code version.
|
||||
|
||||
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
|
||||
|
||||
|
||||
@@ -12,13 +12,13 @@ surfaces**, and which one you pick depends on the operation.
|
||||
│ gbrain process │
|
||||
│ │
|
||||
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
|
||||
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
|
||||
│ │ (HTTP + OAuth) │ │ admin ops │ │
|
||||
openclaw, fork) ────┼──▶ MCP ops surface │ │ local-only │ │
|
||||
│ │ (HTTP + OAuth) │ │ commands │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ search, query, │ │ sync, embed, │ │
|
||||
│ │ put_page, │ │ dream, doctor,│ │
|
||||
│ │ get_page, │ │ autopilot, │ │
|
||||
│ │ find_experts, │ │ init, secrets │ │
|
||||
│ │ put_page, │ │ extract, │ │
|
||||
│ │ get_page, │ │ dream, │ │
|
||||
│ │ find_experts, │ │ enrich, ... │ │
|
||||
│ │ ... │ │ │ │
|
||||
│ └──────────────────┘ └────────────────┘ │
|
||||
│ ▲ ▲ │
|
||||
@@ -26,7 +26,7 @@ surfaces**, and which one you pick depends on the operation.
|
||||
│ │ │ │
|
||||
│ thin-client OAuth shell-job `inherit:`│
|
||||
│ (preferred for (only path for │
|
||||
│ MCP-equivalent ops) localOnly ops) │
|
||||
│ MCP-equivalent ops) local-only work) │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -45,7 +45,7 @@ the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
|
||||
The host runs gbrain as a long-lived HTTP server:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
The agent registers as an OAuth client (one-time):
|
||||
@@ -75,22 +75,27 @@ commands through the configured remote MCP. The agent can call
|
||||
agent to a specific source within a federated brain.
|
||||
- One audit surface (`mcp_request_log`) covers every op call uniformly.
|
||||
|
||||
## Surface 2 — localOnly admin ops via shell-job `inherit:`
|
||||
## Surface 2 — local-only work via shell-job `inherit:`
|
||||
|
||||
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
|
||||
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
|
||||
list (as of v0.36.5.0) includes:
|
||||
Two mechanisms keep local-only work off the remote surface, and they operate
|
||||
at different layers:
|
||||
|
||||
- `sync` (filesystem walks need local FS access)
|
||||
- `embed` (orchestrates the embed pipeline)
|
||||
- `extract` (walks markdown files)
|
||||
- `dream` (synthesis cycle)
|
||||
- `doctor` (filesystem hygiene checks)
|
||||
- `autopilot` (background daemon orchestration)
|
||||
- `init` (creates `~/.gbrain/`)
|
||||
- `secrets` (config management)
|
||||
- **Op layer:** operations flagged `localOnly: true` in
|
||||
`src/core/operations.ts` are filtered out of the HTTP MCP surface entirely
|
||||
— a remote caller never sees them.
|
||||
- **CLI layer:** on a thin-client install (remote MCP configured, no local
|
||||
engine), commands that require a local engine or the local filesystem are
|
||||
refused at dispatch with a pinpoint hint naming the closest alternative.
|
||||
The authoritative set is `THIN_CLIENT_REFUSED_COMMANDS` in `src/cli.ts` —
|
||||
read it there rather than trusting any list copied into a doc; it covers
|
||||
`sync`, `embed`, `extract`, `dream`, `enrich`, `serve`, `config`, and a
|
||||
couple dozen more.
|
||||
|
||||
For these, the agent cannot route through HTTP MCP. The only path is to run
|
||||
Notable non-members: `doctor` is NOT refused on a thin client — it reroutes
|
||||
to an outbound-HTTP probe set (`src/core/doctor-remote.ts`); `bootstrap` and
|
||||
`hook` are engine-free and work on any install shape.
|
||||
|
||||
For refused commands, the agent cannot route through HTTP MCP. The path is to run
|
||||
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
|
||||
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
|
||||
DLQ / audit trail all come for free.
|
||||
@@ -114,12 +119,11 @@ full validation rules and error catalog.
|
||||
|
||||
### Why this is preferred over writing secrets into `env:` per-job
|
||||
|
||||
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
|
||||
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
|
||||
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
|
||||
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
|
||||
validation. The error message names `inherit: ["database_url"]` as the
|
||||
replacement.
|
||||
- Passing `env: { GBRAIN_DATABASE_URL: "postgresql://..." }` per job would
|
||||
land the URL plaintext in `minion_jobs.data` and the shell-audit JSONL —
|
||||
visible to anyone with brain-DB read access (or a brain dump, or a shared
|
||||
brain via mounts). Pre-enqueue validation rejects it; the error message
|
||||
names `inherit: ["database_url"]` as the replacement.
|
||||
|
||||
### Worker setup (one-time, per host)
|
||||
|
||||
@@ -146,11 +150,11 @@ proxy for worker env.
|
||||
| `get_page` / `list_pages` | HTTP MCP | Same. |
|
||||
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
|
||||
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
|
||||
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
|
||||
| `sync` / `embed` / `extract` | Shell job + `inherit:` | Thin-client refused; needs local engine + FS. |
|
||||
| `dream` | Shell job + `inherit:` | Thin-client refused; synthesis runs on the host. |
|
||||
| `doctor` | Run directly (any install) | Not refused: thin clients get the remote probe set. |
|
||||
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
|
||||
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
|
||||
| `init` / `config` | One-time host setup | Operator action, not agent action. |
|
||||
|
||||
## Recommended patterns
|
||||
|
||||
@@ -166,16 +170,19 @@ proxy for worker env.
|
||||
- **`env:` still works** for non-secret values, or for cases where you
|
||||
WANT the value in the row (e.g. an opaque correlation token your audit
|
||||
flow needs to read back later). The validator doesn't second-guess you.
|
||||
- **Never try to route a `localOnly` op through thin-client MCP.** It will
|
||||
fail with `localOnly op refused in thin-client mode`. Use shell-job +
|
||||
`inherit:` (for secrets) or `env:` (for non-secrets).
|
||||
- **Never try to route a refused command through a thin client.** The CLI
|
||||
refuses it at dispatch with a hint. Use shell-job + `inherit:` (for
|
||||
secrets) or `env:` (for non-secrets) on the host instead.
|
||||
- **Push-based context.** Beyond request/response ops, MCP clients can
|
||||
receive volunteered context via the `volunteer_context` op — see
|
||||
[push-context.md](./push-context.md).
|
||||
|
||||
## Migration: from pre-v0.36.5.0
|
||||
## Migration: from `env:`-passed secrets
|
||||
|
||||
If your agent submits shell jobs that pass secrets via `env:`:
|
||||
|
||||
```jsonc
|
||||
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
|
||||
// Rejected at submit: the URL would persist in minion_jobs.data plaintext.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
@@ -186,7 +193,7 @@ If your agent submits shell jobs that pass secrets via `env:`:
|
||||
Switch to (recommended):
|
||||
|
||||
```jsonc
|
||||
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
|
||||
// Name in row, value resolved at child-spawn from worker config.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Ambient recall — placing retrieval at session boundaries
|
||||
|
||||
Long-lived agent harnesses (your OpenClaw, Hermes, Codex, Claude Code) get the
|
||||
most value from the brain not on every message, but at the moments where a fresh
|
||||
question rarely fires on its own: **session start, right after compaction, and
|
||||
on heartbeats.** This guide is the Pareto frontier of where to place each verb.
|
||||
|
||||
The bottleneck for a long-lived agent is not retrieval quality — the corpus
|
||||
answers well when asked. It is **placement**: the misses come from moments when
|
||||
no question fires. Two frozen verbs close that gap with 2-3 deterministic calls
|
||||
per session instead of per-message overhead.
|
||||
|
||||
## The frontier — which verb goes where
|
||||
|
||||
| Moment | Call | Why | Cost |
|
||||
|---|---|---|---|
|
||||
| Any entity-bearing message | `entity(name)` | Zero-LLM, p99 < 100ms. Safe to run synchronously almost anywhere. | negligible |
|
||||
| **Session start** | `context_pack(entities, budget_tokens)` | Warm the thread's 1-3 standing entities before the first message. | zero-LLM, sub-second |
|
||||
| **After compaction** | `context_pack(entities, budget_tokens)` | Rehydrate the verbatim detail the summary dropped. | zero-LLM, sub-second |
|
||||
| **Heartbeat / periodic wake** | `delta(session_id, budget_tokens)` | "What changed since my last wake" in O(changes), deduped. | zero-LLM, sub-second |
|
||||
| Explicit memory question | `recall(query \| entity, budget_tokens)` | The budget-packed read for "what do we know that we SAVED about X". | sub-second (+1 embedding if `query`) |
|
||||
| Answer needs cross-page reasoning | `synthesize(question)` | LLM-backed. **Never** on a hot or ambient path. | seconds-to-minutes, $$ |
|
||||
|
||||
Observed shape: per-message retrieval beyond `entity` cards adds latency faster
|
||||
than insight; session-start packs and post-compaction rehydration are nearly
|
||||
pure win. See the per-verb latency table in
|
||||
[`docs/protocol/MEMORY_VERBS_v1.md`](../protocol/MEMORY_VERBS_v1.md#latency-classes-per-verb).
|
||||
|
||||
## Two integration surfaces
|
||||
|
||||
- **Pull (works everywhere, including Codex + Postgres/Supabase):** the harness
|
||||
calls `context_pack` / `delta` over MCP (they are on `--surface verbs`) or the
|
||||
CLI (`gbrain context-pack`, `gbrain delta`) at the boundary and injects the
|
||||
returned `text` (or renders the structured arms). This is the portable path —
|
||||
no hooks required. It is the primary path for Codex (which has no hooks) and
|
||||
for Postgres brains (which have no local IPC socket).
|
||||
- **Push (PGLite + Claude Code):** the bundled hook framework fires
|
||||
automatically at `SessionStart` (injects a warm pack — including the
|
||||
post-compaction re-entry, `source=compact`) and `PreCompact` (banks the
|
||||
window's standing entities for that rehydration pack). Heartbeat deltas are
|
||||
the PULL path — there is deliberately no push heartbeat; call `delta` per
|
||||
the HEARTBEAT cadence table.
|
||||
|
||||
## Visibility — world-only by default
|
||||
|
||||
A pack is injected into an agent context window that may be logged or synced to a
|
||||
cloud model, so **every arm is world-visibility by default.** To pull private
|
||||
facts in, pass `include_private` — and it is honored ONLY for trusted-local
|
||||
callers (`remote === false`, i.e. the CLI/hook path). A remote MCP caller never
|
||||
widens, even if it asks (fail-closed). When it does widen, all arms widen
|
||||
together, so a pack is never a mix of private facts beside world-stripped
|
||||
synopses.
|
||||
|
||||
## Budgets
|
||||
|
||||
Every pack/delta call takes `budget_tokens`. The server packs highest-priority
|
||||
arms first (cards → facts for packs; pages → facts for deltas) and reports
|
||||
`budget_used` + `dropped_count`; the injectable `text` field is rendered from
|
||||
the packed sets, so it honors the same budget the structured arrays report. It
|
||||
never trims client-side — you always know what was left out (`dropped_count`,
|
||||
and `has_more` on deltas). Pick a budget to fit the boundary: a session-start
|
||||
pack can afford more than a heartbeat delta.
|
||||
|
||||
## Heartbeat cursor + dedup
|
||||
|
||||
Pass a stable `session_id` to `delta` and the brain keeps a per-session cursor:
|
||||
the first wake establishes it, each wake advances it. Dedup is **cursor-based**
|
||||
— a delivered page reappears only if it changes again after delivery (and then
|
||||
it should). Delivery is **at-least-once**: pages arrive oldest-first, and when
|
||||
a budget or the fetch limit drops some, the response sets `has_more: true` and
|
||||
the cursor advances only to the newest *delivered* page, so the tail surfaces
|
||||
on the next wake — nothing is silently lost. With no `session_id` you can still
|
||||
pass an explicit `since` for a stateless delta. The cursor is namespaced per
|
||||
caller (`(source_id, client_id, session_id)`; authenticated remotes use their
|
||||
client id, auth-less remotes share a `remote` namespace, and `local` is
|
||||
reserved for the trusted CLI/hook lane), so a remote harness can never read or
|
||||
advance the local lane's cursor. Idle session cursors are garbage-collected
|
||||
after **7 days** — a wake on an expired session re-establishes the cursor at
|
||||
now and returns an empty delta, so a harness returning from a long sleep
|
||||
should run one stateless `since`-based catch-up first.
|
||||
|
||||
## Example — a cold session start (pull)
|
||||
|
||||
```bash
|
||||
gbrain context-pack --entities "acme-example,alice-example" --budget-tokens 4000
|
||||
```
|
||||
|
||||
Returns entity cards + open threads + hot facts, budget-packed, world-only. Inject
|
||||
the `text` field into the model's context before the first user message.
|
||||
@@ -0,0 +1,247 @@
|
||||
# GBrain Bootstrap — your harness as your agent
|
||||
|
||||
`gbrain bootstrap` turns a Claude Code or Codex session into a persistent personal
|
||||
agent: identity files rendered from your own answers, a local PGLite brain,
|
||||
per-turn context, session-triggered schedules, and a private GitHub repo as the
|
||||
agent's durable, portable body. This guide is the full contract — what gets
|
||||
installed, what runs when, what it can and cannot do, and how to undo all of it.
|
||||
|
||||
Normative design docs: [AGENT_BOOTSTRAP_DESIGN.md](../designs/AGENT_BOOTSTRAP_DESIGN.md)
|
||||
(scope) and [AGENT_BOOTSTRAP_PLAN.md](../designs/AGENT_BOOTSTRAP_PLAN.md)
|
||||
(implementation). The paste block lives in the README; the runbook your agent
|
||||
follows is `BOOTSTRAP_FOR_AGENTS.md` at the repo root, fetched at the
|
||||
`latest-stable` ref.
|
||||
|
||||
## What gets installed, exactly
|
||||
|
||||
| Piece | Where | Runs when |
|
||||
|---|---|---|
|
||||
| Identity files (SOUL/USER/MEMORY/AGENTS/CLAUDE/HEARTBEAT/ACCESS_POLICY/GITHUB) | your workspace folder | loaded at session start |
|
||||
| `agent.json` manifest + `brain/`, `memory/`, `skills/`, `state/` | workspace | — |
|
||||
| Local brain (PGLite) | `~/.gbrain/` (never in the repo) | while a session's MCP serve is open |
|
||||
| MCP registration (`gbrain serve`) | Claude Code: project scope by default; Codex: user-global (no scope flag) | spawned by your harness per session |
|
||||
| Hooks (Claude Code, ON by default) | local installs: `.claude/settings.local.json` (gitignored); cloud sandboxes: the COMMITTED `.claude/settings.json` (PATH-resolved, fail-open commands) | each prompt; fail-open; `--no-hooks` opts out at install, `GBRAIN_HOOKS=0` disables at runtime |
|
||||
| Per-turn persistence | Stop hook → debounced, detached scan-gated push (per workspace; 5 min default, every turn in cloud sandboxes) | after each assistant turn; `GBRAIN_STOP_PUSH=0` disables; `GBRAIN_STOP_PUSH_DEBOUNCE_MIN` / config `hooks.stop_push_debounce_min` tune it |
|
||||
| Session persistence | SessionEnd hook → scan-gated commit+push | at session end (note: the harness never fires SessionEnd on `/exit` — the per-turn push is what covers that) |
|
||||
| Push-failure visibility | next turn's context + a user-visible notice; re-announces every 30 min while failing | whenever a background push fails |
|
||||
| Optional background job (consent-gated) | git post-commit auto-push + launchd/cron 30-min pull (pull job skipped honestly on hosts without a scheduler) | while logged in |
|
||||
| Private GitHub repo | your account, created by `bootstrap repo` (or an empty repo you made yourself, adopted) | privacy verified via API |
|
||||
| Machine receipt | `~/.gbrain/bootstrap/receipt.json` | uninstall is keyed to it |
|
||||
|
||||
**What does NOT run:** anything while the harness is closed. Session-triggered
|
||||
schedules fire at turn/session boundaries only. True 24/7 operation is what a
|
||||
hosted brain provides — this is the honest desktop contract.
|
||||
|
||||
## Cloud sandboxes (claude.ai/code and similar)
|
||||
|
||||
Cloud sessions run in a reclaimed-after-inactivity VM behind a
|
||||
credential-injecting egress proxy. `gbrain bootstrap status --json` reports
|
||||
`execution_environment: "cloud-sandbox"` there, and the install adapts:
|
||||
|
||||
- **Hooks live in the committed `.claude/settings.json`** with PATH-resolved,
|
||||
fail-open commands (no machine paths). The gitignored local settings file
|
||||
never survives into the next session's fresh clone, and hook config is
|
||||
snapshotted at session start — so hooks written mid-session go live on the
|
||||
NEXT session. Commit and push the file.
|
||||
- **The per-turn push runs every turn** (debounce 0) — a reclaimed VM's tail
|
||||
loss is permanent, so each turn banks to the private repo.
|
||||
- **Repo-privacy verification falls back to pure git protocol** when the proxy
|
||||
blocks the GitHub API (GraphQL is always pinned there; REST reaches only
|
||||
session-attached repos). Confirmed-public origins still always refuse.
|
||||
- **Repo creation is refused in cloud** with the flow that works: create the
|
||||
private repo from a normal machine or github.com, open the cloud session ON
|
||||
that repo, run `gbrain bootstrap attach`.
|
||||
- **The gbrain binary installs via the environment setup script** — print it
|
||||
with `gbrain bootstrap cloud-setup-script` and paste it into the environment
|
||||
config (npm-based; bun's package fetching is proxy-incompatible there).
|
||||
- **No scheduler exists** — the consent-gated pull job is skipped with an
|
||||
honest message; event-driven pushes cover persistence.
|
||||
|
||||
Escape hatch for self-hosted git you trust (every use warns loudly):
|
||||
the CLI flag on `sources push`, `GBRAIN_ALLOW_UNVERIFIED_REMOTE=1`, or
|
||||
`gbrain config set push.allow_unverified_remote true` (file-plane — the only
|
||||
form that reaches detached hook children inside a sandbox).
|
||||
|
||||
## Bring your own repo (create-repo-first)
|
||||
|
||||
By default bootstrap creates the private GitHub repo for you. If you prefer to own
|
||||
that step — pick the name/org-under-your-account, or just work the familiar way —
|
||||
create a new **empty** private repo **under your own GitHub account** (no
|
||||
README/.gitignore/license), clone it, open the clone in your harness, and run the
|
||||
bootstrap block. `gbrain bootstrap repo` detects the empty repo you created and
|
||||
**adopts** it: it verifies the repo is private, sets a repo-local git identity, and
|
||||
pushes your workspace. Two constraints, both enforced with a clear message rather
|
||||
than a silent failure:
|
||||
|
||||
- **Empty.** A repo that already has commits (a README, a license, an existing
|
||||
project) is refused — create it empty, or run `gbrain bootstrap attach` if it is
|
||||
an existing agent workspace. (A repo already carrying *this* workspace's history,
|
||||
e.g. from an interrupted run, is recognized as yours and resumed.)
|
||||
- **Personal account.** The repo must be owned by your authenticated GitHub user.
|
||||
Org-owned repos are refused today; create one under your own account, or let
|
||||
bootstrap make it.
|
||||
|
||||
Until the repo phase verifies the repo, the per-turn/session-end push stays
|
||||
deferred — bootstrap never publishes your workspace to an origin whose privacy it
|
||||
hasn't confirmed.
|
||||
|
||||
## The awake-when-you-are contract
|
||||
|
||||
Your agent is awake when your harness is. Laptop asleep = agent asleep. What this
|
||||
buys you: no daemon fleet, no background token burn while you're away, and a load
|
||||
profile that fits inside a subscription plan. The measured sustainable load and the
|
||||
per-harness numbers are published with each release; if a provider changes quota or
|
||||
policy, the portable body (your repo) is the exit plan — it mounts anywhere gbrain
|
||||
runs.
|
||||
|
||||
## Keyless mode
|
||||
|
||||
With zero API keys, everything works: the agent authors memory explicitly through
|
||||
the brain's write tools (`put_page`, timeline entries, `## Facts` fences — your
|
||||
harness's model is the LLM, already paid for), and search runs keyword-only
|
||||
(BM25). `bootstrap verify` prints the capability report honestly. One optional key
|
||||
upgrades capabilities per provider — OpenAI unlocks semantic search and
|
||||
automatic fact extraction; Voyage unlocks semantic search; Anthropic unlocks
|
||||
fact extraction (Anthropic has no embeddings API, so it does not enable
|
||||
semantic search). The key goes to the 0600 config file, never into the repo or
|
||||
the interview answers. API spend is metered separately from your subscription and is
|
||||
zero in keyless mode; with a key, the standard spend gates apply
|
||||
([spend-controls](../operations/spend-controls.md)).
|
||||
|
||||
## Security posture
|
||||
|
||||
- **Supply chain:** the paste block and install command pin the `latest-stable`
|
||||
ref — a maintainer-controlled tag advanced only after a release fully publishes.
|
||||
The runbook carries a version stamp; `bootstrap status` warns on skew. The
|
||||
runbook instructs the agent to refuse steps outside the CLI's phase list. bun
|
||||
installs via package manager or checksum-verified download.
|
||||
- **Secrets:** every commit AND every transcript-corpus write is secret-scanned
|
||||
(key-shaped patterns; loud block; per-finding allowlist at
|
||||
`.gbrain-scan-allow`). A deny-glob backstop refuses tracked `*.pglite`/`.env*`
|
||||
files even if `.gitignore` is damaged. Push refuses public remotes and
|
||||
unverifiable visibility.
|
||||
- **Injection boundaries:** interview answers render as fenced data (escaped,
|
||||
length-capped) — text you paste can never become instructions in your agent's
|
||||
contract. Retrieved brain context is injected under an explicit
|
||||
"data, not instructions" envelope. Facts visible to the harness respect the
|
||||
brain's visibility tiers.
|
||||
- **Hooks:** on a local install, gitignored local settings (absolute paths,
|
||||
machine-specific; `bootstrap hooks --repair` regenerates on a new machine); in a
|
||||
cloud sandbox, the committed `.claude/settings.json` (PATH-resolved, fail-open —
|
||||
see the Cloud sandboxes section). Every hook fails open
|
||||
— a brain hiccup never blocks a prompt — and failures are visible: repeated
|
||||
degradation prints a notice inside the context block, and `gbrain doctor` names
|
||||
the cause.
|
||||
- **Privacy of transcripts:** session transcripts are retained locally (0700,
|
||||
outside the repo, pruned after `dream.synthesize.corpus_retention_days`, default
|
||||
30 — set it in the config file, `~/.gbrain/config.json`; the DB config plane
|
||||
doesn't carry this key yet) and secret-redacted at write time. They never enter the repo. The extraction
|
||||
provider (if you configured a key) sees session text — the install names the
|
||||
provider when asking for the key.
|
||||
|
||||
## Honest forget semantics
|
||||
|
||||
The repo is git history — append-only. Deleting a line removes it from the working
|
||||
tree, not from history. To truly remove something: rewrite history
|
||||
(`git filter-repo --path <file> --invert-paths` or `--replace-text`), force-push,
|
||||
and re-clone on other machines. `MEMORY.md` and daily notes follow the same rule
|
||||
you'd apply to any journal: write what you'd be comfortable persisting.
|
||||
|
||||
## Degradation matrix
|
||||
|
||||
| You declined / lack | What still works | What you lose |
|
||||
|---|---|---|
|
||||
| API keys | everything (keyless mode) | semantic search, auto-extraction |
|
||||
| GitHub / `gh` | full local agent | off-machine durability (repo re-runnable later) |
|
||||
| Hooks (Claude Code) | pull protocol via AGENTS.md gates | automatic per-turn context + session-end persistence |
|
||||
| Codex (no hook system, no MCP scope flag) | pull protocol + MCP tools | per-turn push (stated plainly; not oversold) + the ability to confine MCP reach to one folder (`codex mcp add` is always user-global) |
|
||||
| Second simultaneous session | first session unaffected | second session's brain tools fail politely (one live serve per brain — v1 contract) |
|
||||
|
||||
## Multi-device
|
||||
|
||||
Clone your agent repo on machine two and run `gbrain bootstrap attach` — it
|
||||
validates the manifest, wires this machine (source registration, hooks repair,
|
||||
MCP), and verifies. The brain database is derived state, rebuilt from `brain/` +
|
||||
re-ingestion; hot facts extracted only on machine one arrive via the repo's pages
|
||||
and fences. Simultaneous editing from two machines is ordinary git conflict
|
||||
territory — `sources push` pulls divergence-safely (commit first, rebase pull,
|
||||
loud on conflicts).
|
||||
|
||||
## Uninstall
|
||||
|
||||
`gbrain bootstrap uninstall` removes exactly what this machine's install receipt
|
||||
records: hook wiring, MCP registrations (surgically — foreign servers and hooks
|
||||
survive), and bootstrap-created state. Your repo is never touched — the body
|
||||
remains yours. The brain database is KEPT by default; `--delete-brain` is offered
|
||||
only when bootstrap created the brain, offers a facts export first, and enumerates
|
||||
what it is about to remove. It refuses to run while a session's serve is live.
|
||||
|
||||
## If something seems broken
|
||||
|
||||
One command: `gbrain doctor`. It covers hook health, push staleness, serve/lock
|
||||
collisions, schema state, and prints fixes. `gbrain bootstrap status --json` emits
|
||||
a support blob (versions, harness, last verify/push, hook failure rate) your agent
|
||||
can relay verbatim when you report a problem.
|
||||
|
||||
## Real-agent e2e
|
||||
|
||||
Most bootstrap tests drive the dispatcher with PATH-shimmed `claude`/`codex`
|
||||
recorders — fast, hermetic, no API cost. Two additional "door" tests drive the
|
||||
ACTUAL binaries end to end so we catch real-world drift (a `codex mcp add` flag
|
||||
that changed shape, a harness that stopped calling our MCP server):
|
||||
|
||||
- `test/e2e/bootstrap-real-claude.serial.test.ts` — real `claude -p` over MCP.
|
||||
- `test/e2e/bootstrap-real-codex.serial.test.ts` — real `codex exec`. It runs the
|
||||
keyless-`init` → interview → render → `gbrain bootstrap hooks --harness codex`
|
||||
path (executing the real `codex mcp add` into a hermetic `~/.codex/config.toml`),
|
||||
asserts the rendered `AGENTS.md` carries the Gate-3 brain-first pull protocol
|
||||
(Codex has no hook system, so the pull protocol is its per-turn seam), then
|
||||
spends one live `codex exec` turn to prove real codex → gbrain MCP → brain →
|
||||
a seeded, brain-only fact (falling back to a shell `gbrain query` if headless
|
||||
stdio-MCP is unavailable).
|
||||
|
||||
These pay real API cost and take 30s–2min per turn, so they are NOT in the PR
|
||||
shard. Everything is hermetic (temp `HOME` / `CODEX_HOME` / `CLAUDE_CONFIG_DIR` /
|
||||
`GBRAIN_HOME` per test — the operator's real `~/.claude`, `~/.gbrain`, `~/.codex`
|
||||
are never touched; auth is copied read-only). Each file self-SKIPS via
|
||||
`describe.skipIf` when its binary or auth is absent, so on a machine without the
|
||||
tool it is a clean no-op that never fails. CI wires them into the `real-agent-e2e`
|
||||
job in `.github/workflows/heavy-tests.yml` (nightly + the `real-agent-e2e` /
|
||||
`heavy-tests` label); on a stock runner they self-skip. To actually exercise the
|
||||
binaries you need a runner with authed `claude`/`codex` and the provider creds
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`) exported.
|
||||
|
||||
Run locally (where both are installed + authed):
|
||||
|
||||
```bash
|
||||
bun test test/e2e/bootstrap-real-codex.serial.test.ts
|
||||
```
|
||||
|
||||
## DX exploration harness (developer instrument, not a test)
|
||||
|
||||
The door tests prove the install WORKS; they say nothing about how it FEELS.
|
||||
`test/helpers/tty-harness.ts` spawns any CLI (gbrain, `claude`, `codex`) under a
|
||||
real pseudo-terminal (Bun's `terminal:` spawn option) and records every output
|
||||
burst with a millisecond timestamp, so unnecessary pauses become a measurable
|
||||
artifact (`computeStalls` → `stalls.md`) instead of a vibe. Same hermetic env as
|
||||
`agent-harness.ts`; pure helpers are unit-tested in `test/tty-harness.test.ts`
|
||||
(zero subprocesses, PTY smokes self-skip where `terminal:` is unavailable).
|
||||
|
||||
`scripts/dx-explore.ts` drives it to capture the fresh-user funnel as timestamped
|
||||
transcripts under `.context/dx-runs/` (gitignored — nothing asserts, no CI):
|
||||
|
||||
```bash
|
||||
bun run scripts/dx-explore.ts help # comprehension surfaces (no keys)
|
||||
bun run scripts/dx-explore.ts init [--keyless] # interactive init, naive-user autopilot
|
||||
bun run scripts/dx-explore.ts claude-install # REAL claude running the paste-in bootstrap
|
||||
bun run scripts/dx-explore.ts codex-install # REAL codex, same
|
||||
bun run scripts/dx-explore.ts drive -- gbrain init # manual: steer a live TUI via a file channel
|
||||
```
|
||||
|
||||
`drive` mode is how an agent in a Conductor workspace explores a live TUI across
|
||||
separate tool calls: `cat <dir>/session/screen.txt` to watch, append
|
||||
`{"line":"..."}` / `{"key":"Down"}` / `{"stop":true}` to `<dir>/session/input.jsonl`
|
||||
to steer. Each run writes `meta.json`, `visible.txt`, `frames.jsonl`, and
|
||||
`stalls.md`. `--keyless` strips provider keys so the true no-key first-touch path
|
||||
is exercised (a Conductor session's ambient `ANTHROPIC_API_KEY` would otherwise
|
||||
leak in). Install scenarios pay real API cost — launch them as background tasks.
|
||||
@@ -27,6 +27,7 @@ READ: check brain FIRST (before responding)
|
||||
│ → gbrain search "{entity name}"
|
||||
│ → gbrain get {slug} (if you know it)
|
||||
│ → gbrain query "what do we know about {topic}"
|
||||
│ → full protocol: brain-first-lookup.md
|
||||
│
|
||||
▼
|
||||
RESPOND with brain context (every answer is better with context)
|
||||
@@ -100,12 +101,14 @@ on_message(text):
|
||||
Write immediately after the conversation, while the context is fresh.
|
||||
|
||||
3. **Sync after every write batch.** Without sync, the brain search index is
|
||||
stale. The next query won't find what you just wrote.
|
||||
stale. The next query won't find what you just wrote. On installs set up
|
||||
via `gbrain bootstrap`, per-turn context injection and session-end
|
||||
persistence hooks automate parts of this loop — see
|
||||
[bootstrap.md](bootstrap.md) and [push-context.md](push-context.md).
|
||||
|
||||
4. **External APIs are fallback, not primary.** `gbrain search` before
|
||||
Brave Search. `gbrain get` before Crustdata. The brain has relationship
|
||||
history, your own assessments, meeting transcripts, cross-references.
|
||||
No external API can provide that.
|
||||
4. **External APIs are fallback, not primary.** `gbrain search` before any
|
||||
web or enrichment API. The full brain-before-external protocol (and why)
|
||||
lives in [brain-first-lookup.md](brain-first-lookup.md).
|
||||
|
||||
## How to Verify It Works
|
||||
|
||||
|
||||
@@ -51,22 +51,25 @@ The brain has context no external API can provide:
|
||||
- Timeline (what changed recently, what's trending)
|
||||
|
||||
A LinkedIn scrape gives you their job title. The brain gives you: "co-founded
|
||||
Brex, you had coffee with him 3 times, last discussed the payments infrastructure
|
||||
thesis, he's interested in your take on AI agents."
|
||||
widget-co, you had coffee with her 3 times, last discussed the payments
|
||||
infrastructure thesis, she's interested in your take on AI agents."
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Try keyword first, then hybrid.** Keyword search works without embeddings
|
||||
(day one). Hybrid search needs embeddings but finds semantic matches. Try
|
||||
both in sequence.
|
||||
(day one — and it's ALL you get in keyless mode, see
|
||||
[bootstrap.md](bootstrap.md)). Hybrid search needs embeddings but finds
|
||||
semantic matches. Try both in sequence.
|
||||
|
||||
2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact
|
||||
slug doesn't exist, it suggests alternatives. Use this for name variants
|
||||
("Pedro" → "pedro-franceschi").
|
||||
2. **Fuzzy slug matching is opt-in.** Pass `--fuzzy` (the `fuzzy: true` param
|
||||
on `get_page`) and a near-miss slug resolves to the unique candidate, or
|
||||
returns an `ambiguous_slug` error listing the candidates. WITHOUT the flag
|
||||
a miss just throws `page_not_found` (with a hint to retry with
|
||||
`fuzzy: true`). Use it for name variants ("Alice" → "alice-example").
|
||||
|
||||
3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?"
|
||||
should check the brain first. The brain might have it, and the lookup adds
|
||||
no latency (< 100ms for keyword search).
|
||||
should check the brain first. The brain might have it, and a keyword
|
||||
lookup is fast enough to be effectively free.
|
||||
|
||||
4. **Load compiled truth + recent timeline.** The compiled truth gives you the
|
||||
state of play in 30 seconds. The timeline gives you what changed recently.
|
||||
|
||||
@@ -17,8 +17,8 @@ on new_information(info):
|
||||
# This is world knowledge -- facts about entities external to the agent
|
||||
gbrain put <slug> --content "..."
|
||||
# Examples:
|
||||
# "Pedro is CEO of Brex" -> gbrain (person page)
|
||||
# "Brex raised Series D at $12B" -> gbrain (company page)
|
||||
# "alice-example is CEO of widget-co" -> gbrain (person page)
|
||||
# "widget-co raised Series D at $12B" -> gbrain (company page)
|
||||
# "Tuesday's meeting covered Q2" -> gbrain (meeting page)
|
||||
# "The meatsuit maintenance tax" -> gbrain (originals page)
|
||||
|
||||
@@ -57,15 +57,16 @@ on user_asks(question):
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world.
|
||||
1. **Don't store people in agent memory.** "alice-example prefers email over Slack" feels like a preference, but it's a fact about Alice -- it goes in GBrain on her page. Agent memory is for the agent's own operational state, not facts about people in the world.
|
||||
2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration.
|
||||
3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory.
|
||||
4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything.
|
||||
On installs set up via `gbrain bootstrap`, "agent memory" has a concrete file form: MEMORY.md and the other identity files in the agent repo (see [bootstrap.md](bootstrap.md)). The routing rule is unchanged -- those files hold operational state and identity, not world knowledge.
|
||||
5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
|
||||
1. Ask the agent "Who is alice-example?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
|
||||
2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state.
|
||||
3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers.
|
||||
4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory).
|
||||
|
||||
@@ -44,12 +44,12 @@ Sharp technical leader. Under-appreciated internally. Watch for signs of burnout
|
||||
Ascending. Likely CTO track if the migration succeeds.
|
||||
|
||||
## Relationship
|
||||
Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis.
|
||||
Met through alice-example. Had coffee 3x. Last: discussed API architecture thesis.
|
||||
|
||||
## Contact
|
||||
sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
|
||||
|
||||
---
|
||||
<!-- timeline -->
|
||||
|
||||
## Timeline
|
||||
|
||||
@@ -58,7 +58,7 @@ sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
|
||||
[Source: Meeting notes, 2026-04-07 2:00 PM PT]
|
||||
- **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops.
|
||||
[Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT]
|
||||
- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background.
|
||||
- **2026-03-15** | First meeting. Intro from alice-example. Strong technical background.
|
||||
[Source: User, direct conversation, 2026-03-15 3:00 PM PT]
|
||||
```
|
||||
|
||||
@@ -113,9 +113,19 @@ support that claim.
|
||||
truth chunks with higher relevance than timeline chunks. This means the freshest
|
||||
synthesis surfaces first in search results.
|
||||
|
||||
4. **The --- separator matters.** GBrain uses the first standalone `---` after
|
||||
frontmatter to split compiled_truth from timeline. Everything above is compiled
|
||||
truth, everything below is timeline.
|
||||
4. **The timeline sentinel matters — and a bare `---` is NOT one.** GBrain
|
||||
splits compiled_truth from timeline at the first recognized sentinel, in
|
||||
order of precedence:
|
||||
1. `<!-- timeline -->` — preferred; unambiguous, and what GBrain itself
|
||||
emits when it writes a page.
|
||||
2. `--- timeline ---` — decorated separator.
|
||||
3. `---` ONLY when the next non-empty line is `## Timeline` or
|
||||
`## History` (backward-compat for older gbrain-written files).
|
||||
|
||||
A plain `---` line anywhere else is a markdown horizontal rule, not a
|
||||
separator. Author new pages with `<!-- timeline -->` (as in the example
|
||||
above); everything above it is compiled truth, everything below is
|
||||
timeline.
|
||||
|
||||
5. **Don't skip the Assessment section.** The assessment is the value. "Strong
|
||||
technical leader" is something no API can provide. It's YOUR read on this
|
||||
|
||||
@@ -8,6 +8,12 @@ Without this: media links are bookmarks that decay -- you remember watching a vi
|
||||
|
||||
## Implementation
|
||||
|
||||
gbrain's own media surfaces complement this pattern: the bundled
|
||||
`media-ingest` skill (`skills/media-ingest/`) ships the ingestion workflow,
|
||||
and `gbrain files` handles binary/file upload for attachments that should
|
||||
live alongside pages. For meeting recordings specifically, see
|
||||
[meeting-ingestion.md](meeting-ingestion.md).
|
||||
|
||||
```
|
||||
on user_shares_media(url_or_file):
|
||||
|
||||
@@ -119,7 +125,7 @@ on user_shares_media(url_or_file):
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it.
|
||||
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain.
|
||||
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "The speaker made a specific claim about compute scaling that contradicts what another researcher said in their NeurIPS talk -- see media/youtube/a-researcher-neurips-2025" is useful. The analysis connects the new media to the existing brain.
|
||||
3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page.
|
||||
4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry.
|
||||
5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power.
|
||||
|
||||
@@ -26,6 +26,27 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
|
||||
| Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) |
|
||||
| Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below |
|
||||
|
||||
### Prefer gbrain's native schedulers where they fit
|
||||
|
||||
System cron is the lowest common denominator, but gbrain ships its own
|
||||
scheduling surfaces — reach for these first:
|
||||
|
||||
- **`gbrain dream`** — the shipped nightly maintenance cycle (lint,
|
||||
backlinks, extract, sync, embed, synthesize). Schedule THIS instead of
|
||||
hand-rolling the dream cycle below.
|
||||
- **`gbrain jobs` / minions** — queue shell jobs or LLM subagents with retry,
|
||||
backoff, and an audit trail. See the `minion-orchestrator` skill.
|
||||
- **`gbrain autopilot`** — the long-lived background daemon that runs cycles
|
||||
on its own cadence.
|
||||
- **`cron-scheduler` skill** (`skills/cron-scheduler/`) — teaches an agent to
|
||||
manage its harness's scheduler.
|
||||
- **Bootstrap session-triggered schedules** — `gbrain bootstrap` installs
|
||||
HEARTBEAT.md-driven schedules that fire on session activity; see
|
||||
[bootstrap.md](bootstrap.md).
|
||||
|
||||
For scheduling `sync` + `embed --stale` specifically, the home doc is
|
||||
[live-sync.md](live-sync.md).
|
||||
|
||||
## Implementation: Setting Up Cron Jobs
|
||||
|
||||
```bash
|
||||
@@ -44,24 +65,24 @@ fixed. You wake up and the brain is smarter than when you went to sleep.
|
||||
# Brain health — weekly Mondays at 6 AM
|
||||
0 6 * * 1 gbrain doctor --json >> /tmp/gbrain-health.log 2>&1 && gbrain embed --stale
|
||||
|
||||
# Autopilot health gate — daily at 7 AM. The exit code is the signal:
|
||||
# 0 fresh (or nothing installed), 1 needs attention (stale heartbeat,
|
||||
# never ran, or paused), 2 the daemon took itself out of rotation.
|
||||
# Status is filesystem-only, so it works even during a DB outage.
|
||||
0 7 * * * gbrain autopilot --status >> /tmp/gbrain-autopilot-health.log 2>&1 || your-notify "gbrain autopilot needs attention"
|
||||
|
||||
# Dream cycle — nightly at 2 AM
|
||||
0 2 * * * /path/to/dream-cycle.sh
|
||||
```
|
||||
|
||||
### Quiet Hours Gate (MANDATORY)
|
||||
|
||||
Every cron job that sends notifications MUST check quiet hours first.
|
||||
See [Quiet Hours](quiet-hours.md) for the full pattern.
|
||||
|
||||
```bash
|
||||
# In every cron script:
|
||||
if ! bash scripts/quiet-hours-gate.sh; then
|
||||
mkdir -p /tmp/cron-held
|
||||
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
|
||||
exit 0
|
||||
fi
|
||||
# Not quiet hours — send normally
|
||||
```
|
||||
Every cron job that sends notifications MUST check quiet hours first. The
|
||||
gate is a small script YOU create (it doesn't ship with gbrain) and call at
|
||||
the top of every notification-sending cron script; held output goes to a
|
||||
holding directory that the morning briefing drains. See
|
||||
[Quiet Hours](quiet-hours.md) for the gate script and the full pattern —
|
||||
don't copy a snippet from here, that page is the single home.
|
||||
|
||||
### Travel-Aware Timezone Handling
|
||||
|
||||
@@ -88,6 +109,12 @@ morning briefing. Zero config change needed.
|
||||
|
||||
The most important cron job. Runs while you sleep.
|
||||
|
||||
**gbrain ships this**: `gbrain dream` runs the maintenance half of the cycle
|
||||
(lint, backlinks, extract, sync, embed, synthesize) as one command — schedule
|
||||
it nightly and Phase 4 below (plus most of Phase 2's hygiene checks) is
|
||||
covered. The pseudocode that follows is the harness-side variant for agents
|
||||
that also do LLM-driven entity sweeps and memory consolidation on top.
|
||||
|
||||
### What It Does
|
||||
|
||||
```
|
||||
@@ -150,11 +177,11 @@ echo "Dream cycle starting at $(date)"
|
||||
# Phase 1: Entity sweep (spawn sub-agent)
|
||||
# Read today's conversation logs, extract entities, update brain
|
||||
|
||||
# Phase 2: Citation hygiene
|
||||
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
|
||||
# Phase 2: Shipped maintenance cycle (lint, backlinks, extract, sync, embed, synthesize)
|
||||
gbrain dream
|
||||
|
||||
# Phase 3: Embed any stale content
|
||||
gbrain embed --stale
|
||||
# Phase 3: Surface anything the cycle flagged
|
||||
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
|
||||
|
||||
echo "Dream cycle complete at $(date)"
|
||||
```
|
||||
|
||||
@@ -143,4 +143,7 @@ the same pass.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The
|
||||
[email-to-brain recipe](../../recipes/email-to-brain.md) implements this
|
||||
collector pattern end-to-end; [cron-schedule.md](cron-schedule.md) covers
|
||||
scheduling it.*
|
||||
|
||||
@@ -53,6 +53,8 @@ Write extracted content to `brain/diligence/{company}/{doc-name}.md`:
|
||||
**Step 5: Save Raw Files.**
|
||||
Copy original PDFs/files to `brain/diligence/{company}/.raw/`
|
||||
Preserve originals for reference. The diarized version is for search.
|
||||
This is safe by design: `.raw/` directories are excluded from sync, so the
|
||||
originals never enter the search index — only your extracted markdown does.
|
||||
|
||||
**Step 6: Create or Update index.md.**
|
||||
Every diligence directory needs an `index.md`:
|
||||
@@ -103,7 +105,8 @@ cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} in
|
||||
|
||||
**Step 9: Publish (if asked).**
|
||||
When the user wants a shareable brief, create a password-protected
|
||||
published version. Strip internal notes and raw assessment language.
|
||||
published version via the `publish` skill (`skills/publish/`). Strip
|
||||
internal notes and raw assessment language.
|
||||
|
||||
### Quality Bar
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
|
||||
but it is provider-agnostic: any configured `provider:model` works as a
|
||||
target.
|
||||
|
||||
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
|
||||
README reference).
|
||||
Also reachable as `gbrain retrieval-upgrade` — the alias that `gbrain doctor`
|
||||
repair hints and the README point at.
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -27,6 +27,37 @@ gbrain migrate embeddings --to voyage:voyage-3-large --yes
|
||||
declared width and is required for recipes that don't declare one (litellm,
|
||||
llama-server, and other bring-your-own-model providers).
|
||||
|
||||
**Pick `--dim` = your brain's current column width when the target supports
|
||||
it.** A different width triggers the destructive schema transition (column +
|
||||
index rebuild across all three dim-pinned tables); the same width skips it
|
||||
entirely. `gbrain doctor` (check `provider_sunset`, for providers with an
|
||||
announced shutdown) prints the paste-ready command with your actual width
|
||||
already filled in — it reads the real `vector(N)` column, not the config
|
||||
value, which can drift.
|
||||
|
||||
## How affected brains find out (provider sunsets)
|
||||
|
||||
Two surfaces flag a brain whose embedding model (or reranker) is on a
|
||||
provider with an announced hosted-API shutdown, such as ZeroEntropy
|
||||
(2026-09-04):
|
||||
|
||||
- **`gbrain doctor`** — the `provider_sunset` check warns on every run until
|
||||
the brain is off the provider. After the shutdown date it escalates to
|
||||
`fail` only when embedded vectors actually exist on the dead provider
|
||||
(retrieval is genuinely down); a zero-vector brain whose config merely
|
||||
resolves to the dead default stays `warn`, so doctor-as-CI-gate setups
|
||||
don't start exiting 1 on the date. The reranker side resolves through the
|
||||
same plane search actually reranks with (the mode bundle +
|
||||
`search.reranker.*` overrides). The message carries the paste-ready
|
||||
migration command with the brain's actual `--dim`. Accepted the risk?
|
||||
`gbrain config set doctor.suppress_provider_sunset true` silences it.
|
||||
- **`gbrain upgrade`** — a one-shot banner (gated by
|
||||
`ze_sunset_notice_shown`) with the same two fixes.
|
||||
|
||||
Both state the full consequence: after the shutdown, **existing vectors
|
||||
become unqueryable** — query embedding uses the same endpoint as ingestion —
|
||||
not just new content.
|
||||
|
||||
## What it does, in order
|
||||
|
||||
1. **Plan.** Counts every chunk not already in the target embedding space —
|
||||
@@ -88,6 +119,12 @@ continues where it stopped. An in-flight marker (`embedding_migration.state`
|
||||
in DB config) records the target; it is cleared only when the backlog drains
|
||||
to zero.
|
||||
|
||||
One caveat after a HARD kill (SIGKILL, crash, power loss — not Ctrl-C): the
|
||||
run's per-source single-flight embed lock is left behind, and an immediate
|
||||
re-run skips the re-embed and reports the migration as paused. The command
|
||||
says so explicitly (`lock_skipped` in `--json`); the lock expires on its own
|
||||
after at most 60 minutes, then the same re-run resumes normally.
|
||||
|
||||
A page whose chunks straddle two stale batches is embedded correctly but not
|
||||
stamped by the embed loop (which only stamps all-or-nothing per batch), so the
|
||||
migration runs one reconcile pass after the drain that stamps every
|
||||
|
||||
@@ -8,6 +8,11 @@ Without this: brain pages are thin shells with only what the user manually typed
|
||||
|
||||
## Implementation
|
||||
|
||||
gbrain ships both halves of this: `gbrain enrich` is the batch enrichment
|
||||
primitive (finds thin pages and enriches at scale), and the `enrich` skill
|
||||
(`skills/enrich/`) is the agent-driven page-at-a-time workflow. The pipeline
|
||||
below is the pattern they implement — use it to customize or extend.
|
||||
|
||||
```
|
||||
on enrich(entity, trigger):
|
||||
# trigger: meeting mention, email thread, social interaction, user request
|
||||
@@ -68,9 +73,10 @@ on enrich(entity, trigger):
|
||||
gbrain link <person_slug> <deal_slug> # person -> deal
|
||||
# Every entity page links to every other entity page that references it
|
||||
|
||||
# People page sections (not a LinkedIn profile -- a living portrait):
|
||||
# Executive Summary, State, What They Believe, What They're Building,
|
||||
# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline
|
||||
# People page sections: use the person-page structure from compiled-truth.md
|
||||
# (Executive Summary, State, What They Believe, ... Timeline) -- that doc is
|
||||
# the single home for the section taxonomy. Enrichment can add texture
|
||||
# sections on top (What Motivates Them, Hobby Horses, Open Threads).
|
||||
# Facts are table stakes. TEXTURE is the value.
|
||||
|
||||
# Extract texture, not just facts:
|
||||
@@ -100,4 +106,4 @@ on enrich(entity, trigger):
|
||||
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Compiled Truth](compiled-truth.md) for the person-page section taxonomy, [Spend Controls](../operations/spend-controls.md) for gbrain's own embedding/LLM spend gates.*
|
||||
|
||||
@@ -7,10 +7,10 @@ Every inbound message gets scanned for original thinking AND entity mentions so
|
||||
## What the User Gets
|
||||
|
||||
Without this: the agent answers questions but forgets everything. You mention
|
||||
Pedro in a meeting, and next week the agent doesn't know who Pedro is.
|
||||
Alice in a meeting, and next week the agent doesn't know who Alice is.
|
||||
|
||||
With this: every person, company, and idea mentioned in conversation gets a
|
||||
brain page. Next time Pedro comes up, the agent already has context. The
|
||||
brain page. Next time Alice comes up, the agent already has context. The
|
||||
brain compounds.
|
||||
|
||||
## Implementation
|
||||
@@ -18,6 +18,11 @@ brain compounds.
|
||||
Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it
|
||||
to finish before responding. It runs in parallel.
|
||||
|
||||
This pattern is harness-side by design, but gbrain ships help on both ends:
|
||||
the `signal-detector` skill (`skills/signal-detector/`) is the bundled
|
||||
version of this detection loop, and `gbrain extract` runs gbrain's own
|
||||
extraction machinery (entities, facts) over already-synced content.
|
||||
|
||||
```
|
||||
on_every_message(message_text, source_context):
|
||||
|
||||
@@ -109,6 +114,7 @@ is_notable(entity):
|
||||
| Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions |
|
||||
| Hot takes with reasoning | Routine operational messages |
|
||||
| Metaphors that reveal new angles | Requests without embedded insight |
|
||||
| Emotional/psychological insights about self or others | |
|
||||
|
||||
### Filing Rules
|
||||
|
||||
@@ -116,11 +122,17 @@ is_notable(entity):
|
||||
|--------|-------------|
|
||||
| User generated the idea | `brain/originals/{slug}.md` |
|
||||
| User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
|
||||
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
|
||||
| World concept someone else coined | `brain/concepts/{slug}.md` |
|
||||
| Product or business idea | `brain/ideas/{slug}.md` |
|
||||
| Person mentioned | `brain/people/{slug}.md` |
|
||||
| Company mentioned | `brain/companies/{slug}.md` |
|
||||
| Media referenced | `brain/media/{type}/{slug}.md` |
|
||||
| Article ABOUT the user | `brain/media/writings/{slug}.md` |
|
||||
|
||||
This table is the single home for the capture/filing taxonomy. Other guides
|
||||
([idea-capture](idea-capture.md) especially) link here rather than carrying
|
||||
their own copy.
|
||||
|
||||
### The Iron Law of Back-Linking
|
||||
|
||||
@@ -128,21 +140,21 @@ Every entity mention MUST create a back-link FROM the entity page TO the
|
||||
source. This is not optional.
|
||||
|
||||
```
|
||||
// When message mentions "Pedro" and creates a meeting page:
|
||||
// When message mentions "Alice" and creates a meeting page:
|
||||
|
||||
// 1. Update the meeting page (normal)
|
||||
brain/meetings/2026-04-10-board-sync.md:
|
||||
- Pedro presented Q1 numbers
|
||||
- Alice presented Q1 numbers
|
||||
|
||||
// 2. ALSO update Pedro's page (back-link)
|
||||
brain/people/pedro-franceschi.md:
|
||||
// 2. ALSO update Alice's page (back-link)
|
||||
brain/people/alice-example.md:
|
||||
## Timeline
|
||||
- **2026-04-10** | Presented Q1 numbers at board sync
|
||||
[Source: User, board meeting, 2026-04-10]
|
||||
```
|
||||
|
||||
Without back-links, you can't traverse the graph. "Show me everything related
|
||||
to Pedro" only works if Pedro's page links back to every mention.
|
||||
to Alice" only works if Alice's page links back to every mention.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
@@ -163,7 +175,7 @@ to Pedro" only works if Pedro's page links back to every mention.
|
||||
|
||||
5. **Dedup before creating.** Always `gbrain search` before creating a page.
|
||||
Variant spellings, nicknames, and company abbreviations cause duplicates.
|
||||
"Pedro Franceschi" and "Pedro" might be the same person.
|
||||
"Alice Example" and "Alice" might be the same person.
|
||||
|
||||
## How to Verify
|
||||
|
||||
@@ -182,7 +194,7 @@ to Pedro" only works if Pedro's page links back to every mention.
|
||||
4. **Send a boring message.** Say "ok sounds good." Verify: nothing was
|
||||
created. The detector should report "No signals detected."
|
||||
|
||||
5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi."
|
||||
5. **Check for duplicates.** Mention "Alice" then later "Alice Example."
|
||||
Verify: one page, not two.
|
||||
|
||||
---
|
||||
|
||||
@@ -8,6 +8,11 @@ Without this: the agent triages email mechanically ("you have 12 unread"), preps
|
||||
|
||||
## Implementation
|
||||
|
||||
Before hand-rolling these: gbrain bundles the morning-briefing half of this
|
||||
pattern as the `briefing` skill (`skills/briefing/`) and the task-prep half
|
||||
as `daily-task-prep` (`skills/daily-task-prep/`). Use the workflows below to
|
||||
extend or customize what those skills already ship.
|
||||
|
||||
```
|
||||
# WORKFLOW 1: Email Triage
|
||||
on email_batch(emails):
|
||||
@@ -59,8 +64,8 @@ on upcoming_meeting(meeting):
|
||||
briefing[attendee] = "No brain page -- consider enriching"
|
||||
|
||||
# Surface: shared history, what to follow up on, what to watch for
|
||||
# "Last time you discussed the Series B timeline. Pedro was concerned
|
||||
# about burn rate. Here's the latest from his company page."
|
||||
# "Last time you discussed the Series B timeline. alice-example was
|
||||
# concerned about burn rate. Here's the latest from her company page."
|
||||
|
||||
# WORKFLOW 3: Post-Inbox Brain Updates
|
||||
on inbox_cleared():
|
||||
@@ -93,9 +98,9 @@ on schedule_request(meeting):
|
||||
|
||||
1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical.
|
||||
2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise.
|
||||
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate."
|
||||
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with alice-example -- last time you discussed the Series B, she was concerned about burn rate."
|
||||
4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip.
|
||||
5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
|
||||
5. **Scheduling nudges require timeline data.** "You haven't met with charlie-example in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
+13
-39
@@ -61,14 +61,13 @@ capture_idea(message_text, source_context):
|
||||
|
||||
### The Authorship Test
|
||||
|
||||
| Signal | Destination |
|
||||
|--------|-------------|
|
||||
| User generated the idea | `brain/originals/{slug}.md` |
|
||||
| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
|
||||
| World concept someone else coined | `brain/concepts/{slug}.md` |
|
||||
| Product or business idea | `brain/ideas/{slug}.md` |
|
||||
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
|
||||
| Article ABOUT user | `brain/media/writings/` |
|
||||
Who authored the idea determines where it files: user-generated ideas,
|
||||
syntheses, and ghostwritten work go to `brain/originals/`; borrowed world
|
||||
concepts to `brain/concepts/`; product ideas to `brain/ideas/`; articles
|
||||
ABOUT the user to `brain/media/writings/`. The full filing-rules table (and
|
||||
the what-counts-as-original-thinking criteria) lives in
|
||||
[entity-detection.md](entity-detection.md) — the single home for the capture
|
||||
taxonomy.
|
||||
|
||||
### Capture Standards
|
||||
|
||||
@@ -78,21 +77,6 @@ capture_idea(message_text, source_context):
|
||||
"tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase.
|
||||
The vivid version is the real version.
|
||||
|
||||
**What counts as worth capturing:**
|
||||
- Original observations about how the world works
|
||||
- Novel connections between disparate things
|
||||
- Frameworks and mental models
|
||||
- Pattern recognition moments ("I keep seeing X in every Y")
|
||||
- Hot takes with reasoning behind them
|
||||
- Metaphors that reveal new angles
|
||||
- Emotional/psychological insights about self or others
|
||||
|
||||
**What does NOT count:**
|
||||
- Routine operational messages ("ok", "do it")
|
||||
- Pure questions without embedded observations
|
||||
- Echoing back something the agent said
|
||||
- Acknowledgments and reactions
|
||||
|
||||
### The Depth Test
|
||||
|
||||
**Could someone unfamiliar with the user read this page and understand not
|
||||
@@ -137,21 +121,9 @@ Every original MUST link to:
|
||||
|
||||
### Notability Filtering
|
||||
|
||||
Before creating any entity page, check notability:
|
||||
|
||||
**Create a page for:**
|
||||
- People you know or discuss with specificity
|
||||
- Companies you're evaluating, working with, or investing in
|
||||
- Media you mention with personal reaction
|
||||
- Anyone you've explicitly engaged with
|
||||
|
||||
**Don't create pages for:**
|
||||
- Generic references or passing examples
|
||||
- Low-engagement accounts who mentioned you once
|
||||
- Pure metaphors ("like the Roman Empire...")
|
||||
- One-off encounters with no follow-up
|
||||
|
||||
**Decision:** If notable AND no page exists, create a full page with web
|
||||
Before creating any entity page, check notability — the full create/skip
|
||||
criteria live in [entity-detection.md](entity-detection.md#notability-filtering).
|
||||
The decision rule: if notable AND no page exists, create a FULL page with web
|
||||
search enrichment. No stubs. If you make a page, make it good.
|
||||
|
||||
## Tricky Spots
|
||||
@@ -187,4 +159,6 @@ search enrichment. No stubs. If you make a page, make it good.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). The bundled
|
||||
`idea-ingest` skill (`skills/idea-ingest/`) ships this workflow. See also:
|
||||
[Entity Detection](entity-detection.md).*
|
||||
|
||||
@@ -43,9 +43,18 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
|
||||
|
||||
- `gbrain sync --repo <path>` -- one-shot incremental sync. Detects changes via
|
||||
`git diff`, imports only what changed. For small changesets (<= 100 files),
|
||||
embeddings are generated inline during import.
|
||||
embeddings are generated inline during import — unless the inline cost gate
|
||||
intervenes: when the estimated embedding spend crosses the configured floor
|
||||
in a non-interactive session (cron, `--json`), sync auto-defers embeds to a
|
||||
capped `embed-backfill` job instead of spending silently. Either way the
|
||||
chunks get embedded; a deferred run just finishes asynchronously. See
|
||||
[spend controls](../operations/spend-controls.md).
|
||||
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
|
||||
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
|
||||
On a keyless brain (installed with `--no-embedding`), a bare stale embed
|
||||
refuses cleanly — exit 0 with a stderr note — so this chain is safe to
|
||||
schedule on keyless installs; keyword search keeps working. Explicit embed
|
||||
requests (a slug, `--slugs`, `--all`) still exit 1 on a keyless brain.
|
||||
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
|
||||
(configurable with `--interval N`). Embeds inline for small changesets. Exits
|
||||
after 5 consecutive failures, so run under a process manager or pair with a
|
||||
@@ -97,15 +106,27 @@ Triggers sync on push events for instant sync (<5s).
|
||||
### What Gets Synced
|
||||
|
||||
Sync only indexes "syncable" markdown files. These are excluded by design:
|
||||
- Hidden paths (`.git/`, `.raw/`, etc.)
|
||||
- The `ops/` directory
|
||||
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`
|
||||
- Hidden paths (`.git/`, `.raw/`, etc.) and vendored/generated trees
|
||||
(`node_modules/`, `dist/`, `build/`, `venv/`)
|
||||
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`, `RESOLVER.md`
|
||||
|
||||
### Sync is Idempotent
|
||||
Everything else is ordinary synced content — including `ops/` (the bundled
|
||||
daily-task-manager skill files its canonical page under `ops/tasks`).
|
||||
|
||||
### Sync is Idempotent — and Resumable
|
||||
|
||||
Concurrent runs are safe. Two syncs on the same commit no-op because content
|
||||
hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
|
||||
Long syncs also survive being killed: progress checkpoints into the database
|
||||
as files drain, so a killed or aborted run resumes from where it stopped, and
|
||||
the sync bookmark only advances on true completion. A progress-aware stall
|
||||
watchdog (`GBRAIN_SYNC_STALL_ABORT_SECONDS`, default 900, `0` disables) aborts
|
||||
a run that stops making forward progress and releases the per-source lock so
|
||||
the next `gbrain sync` picks up from the checkpoint. The checkpoint cadence
|
||||
and lock-steal grace are tunable via `GBRAIN_SYNC_*` / `GBRAIN_LOCK_*` env
|
||||
vars — incident-time escape hatches, not everyday knobs.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always chain sync + embed.** Running `gbrain sync` without
|
||||
@@ -134,7 +155,17 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
5. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
5. **Staleness can't read "fresh" forever.** A source whose content stopped
|
||||
moving (or whose local clone vanished) used to report fresh indefinitely
|
||||
off the stored content timestamp. Content-relative staleness now ramps
|
||||
toward stale once wall-clock time since the last sync passes a ceiling
|
||||
(default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune — it tracks
|
||||
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` unless set). The ramp is gradual, so
|
||||
the warn tier still fires before the fail tier. `gbrain status` source
|
||||
rows carry `hours_since_last_sync` (raw wall-clock truth) alongside the
|
||||
threshold-relative `staleness_hours` that drives the fresh/stale class.
|
||||
|
||||
6. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
Interrupted `gbrain import <dir>` runs may leave
|
||||
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
|
||||
checkpoint `dir` is the absolute, resolved import target captured when
|
||||
@@ -161,6 +192,15 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
`gbrain embed --stale` isn't running after sync, leaving chunks invisible
|
||||
to vector search.
|
||||
|
||||
4. **Gate on the daemon's heartbeat.** If the built-in daemon runs your sync
|
||||
(`gbrain autopilot --install`), wire your scheduler's health check to
|
||||
`gbrain autopilot --status`. The exit code is the signal: 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused by a migration), 2 the daemon took itself out of rotation.
|
||||
`--json` emits the full report, including `heartbeat_age_seconds`. Status
|
||||
reads only the filesystem — no database connection — so it keeps working
|
||||
during the exact outages it exists to diagnose.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -58,15 +58,20 @@ on new_meeting_transcript(meeting):
|
||||
# Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings
|
||||
# Source: Circleback (https://circleback.ai) or any service with
|
||||
# speaker diarization + API/webhook access
|
||||
|
||||
# Automation: the built-in `extract-timeline-from-meetings` Minion job
|
||||
# automates step 3 (entity timeline propagation) for already-ingested
|
||||
# meeting pages: gbrain jobs submit extract-timeline-from-meetings --follow
|
||||
```
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth.
|
||||
2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation.
|
||||
3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room.
|
||||
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
|
||||
3. **Mentioned people are not just attendees.** If the meeting discussed "Alice's team at widget-co," then Alice's page AND widget-co's page need updates -- even though Alice wasn't in the room.
|
||||
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Alice pushed back on the burn rate, Charlie didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
|
||||
5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always.
|
||||
6. **`--source` on `timeline-add` is the citation text, not source routing.** Because the op declares its own `source` parameter, the CLI binds `--source` to it. To write into a different registered source, use the `.gbrain-source` dotfile or `GBRAIN_SOURCE` env for routing instead.
|
||||
|
||||
## How to Verify
|
||||
|
||||
@@ -76,5 +81,11 @@ on new_meeting_transcript(meeting):
|
||||
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
|
||||
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/meeting-ingestion/SKILL.md` — the bundled, agent-executable skill
|
||||
for this workflow (the canonical step-by-step home; this guide is the
|
||||
pattern overview).
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -40,6 +40,10 @@ gbrain jobs supervisor status --json
|
||||
|
||||
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
|
||||
gbrain jobs supervisor stop
|
||||
|
||||
# Optional: cap worker memory in MB (--max-rss). Without the flag the RSS
|
||||
# watchdog is still on, at a RAM-relative auto-sized cap.
|
||||
gbrain jobs supervisor --concurrency 4 --max-rss 4096
|
||||
```
|
||||
|
||||
**Exit codes:**
|
||||
@@ -50,9 +54,11 @@ gbrain jobs supervisor stop
|
||||
| 1 | Max crashes exceeded (worker kept dying) |
|
||||
| 2 | Another supervisor holds the PID lock |
|
||||
| 3 | PID file unwritable (permission / path error) |
|
||||
| 4 | Queue-scoped DB lock lost mid-run (`LOCK_LOST` — exited rather than risk a split-brain) |
|
||||
|
||||
An agent seeing exit=2 can safely treat it as "one is already running";
|
||||
exit=1 should page a human.
|
||||
exit=4 as "restart me — the DB lock refresh failed"; exit=1 should page
|
||||
a human.
|
||||
|
||||
### Lowering scheduling priority (`--nice`)
|
||||
|
||||
@@ -233,7 +239,7 @@ use a dedicated queue name like `nightly-enrich` above.
|
||||
|
||||
## Upgrading from an older deployment
|
||||
|
||||
### From `minion-watchdog.sh` (pre-v0.20)
|
||||
### From `minion-watchdog.sh`
|
||||
|
||||
Earlier versions of this guide shipped a 68-line bash watchdog
|
||||
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
|
||||
@@ -270,10 +276,10 @@ Regardless of which deployment path you're upgrading from:
|
||||
in-flight job landing partial schema.
|
||||
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
|
||||
`gbrain doctor` reports any migration as `partial` or `pending`.
|
||||
3. **If you run shell jobs:** from v0.14 onward, pass
|
||||
`--allow-shell-jobs` to the supervisor (or keep
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
|
||||
need the flag; only the worker does.
|
||||
3. **If you run shell jobs:** pass `--allow-shell-jobs` to the
|
||||
supervisor (or keep `GBRAIN_ALLOW_SHELL_JOBS=1` in
|
||||
`/etc/gbrain.env`). Submitters don't need the flag; only the worker
|
||||
does.
|
||||
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
|
||||
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
|
||||
should show no unexplained growth in `dead` between pre- and
|
||||
@@ -283,29 +289,30 @@ Regardless of which deployment path you're upgrading from:
|
||||
|
||||
### Supabase connection drops
|
||||
|
||||
The worker uses a single Postgres connection. If Supabase drops it
|
||||
(maintenance, connection limits, network blip), lock renewal fails
|
||||
silently. The stall detector then dead-letters the job after
|
||||
`max_stalled` misses.
|
||||
If Supabase drops the worker's Postgres connection (maintenance,
|
||||
connection limits, network blip), this now self-heals under the
|
||||
supervisor: the worker's DB-liveness probe self-exits (`db_dead`) on a
|
||||
dead pool and the supervisor respawns it with a fresh pool, and the
|
||||
supervisor also restarts a worker that stops making progress while
|
||||
claimable work waits. The escalation commands and thresholds live in the
|
||||
[queue operations runbook](queue-operations-runbook.md) — that's the
|
||||
canonical home for wedge recovery.
|
||||
|
||||
**Current defaults that make this worse:**
|
||||
What can still bite: a *brief* blip during a long-running job can make
|
||||
lock renewal miss, and the stall detector dead-letters the job after
|
||||
`max_stalled` misses (schema column default 5; lock duration and stall
|
||||
check interval are both 30 s).
|
||||
|
||||
- `lockDuration: 30000` (30 s) — too short for long jobs during
|
||||
connection blips.
|
||||
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
|
||||
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
|
||||
- `stalledInterval: 30000` (30 s) — checks too aggressively.
|
||||
|
||||
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
|
||||
(since v0.13.1). These write onto the job row at submit time — which is
|
||||
what `handleStalled()` reads — so per-job tuning is the real knob today.
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags.
|
||||
These write onto the job row at submit time — which is what
|
||||
`handleStalled()` reads — so per-job tuning is the real knob.
|
||||
|
||||
### DO NOT pass `maxStalledCount` to `MinionWorker`
|
||||
|
||||
It's a no-op. The stall detector reads the row's `max_stalled` column
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts`.
|
||||
Use `gbrain jobs submit --max-stalled N` per-job instead.
|
||||
|
||||
### Zombie shell children
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Minions fix — repairing a half-migrated install
|
||||
|
||||
> **Historical repair guide** for the v0.11.0 → v0.11.1 migration. If you're
|
||||
> on any recent release, the canonical fix below (`gbrain apply-migrations
|
||||
> --yes`) is all you need; the stopgap sections exist for archaeology.
|
||||
|
||||
**tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially
|
||||
set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs
|
||||
still on `agentTurn`), run:
|
||||
@@ -34,17 +38,16 @@ stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`.
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
If the install is half-migrated, you'll see:
|
||||
If the install is half-migrated, you'll see the `minions_migration` check
|
||||
fail:
|
||||
|
||||
```
|
||||
[FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
(Missing `~/.gbrain/preferences.json` on a fresh install is a valid
|
||||
pre-`apply-migrations` state — doctor deliberately does NOT fail on that
|
||||
alone; the partial-migration record is the canonical half-migration signal.)
|
||||
|
||||
For a machine-readable report (cron-friendly):
|
||||
|
||||
@@ -118,6 +121,9 @@ cat ~/.gbrain/preferences.json
|
||||
cat ~/.gbrain/migrations/completed.jsonl
|
||||
|
||||
# 3. Autopilot is supervising a Minions worker child
|
||||
# (v0.46+: the exit code is the verdict — 0 fresh, 1 needs attention,
|
||||
# 2 self-disabled — so a nonzero exit here IS the finding, not a
|
||||
# broken verify step. Under `set -e`, append `|| true` to keep going.)
|
||||
gbrain autopilot --status
|
||||
ps aux | grep 'jobs work'
|
||||
|
||||
|
||||
@@ -129,11 +129,10 @@ JSONL records the same. Pre-enqueue validation rejects the submission if the
|
||||
worker can't resolve the requested key, with a paste-ready
|
||||
`gbrain config set database_url <value>` hint.
|
||||
|
||||
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
|
||||
wrote things like:
|
||||
**Why not just write the URL into `env:` directly?** You *can*:
|
||||
|
||||
```jsonc
|
||||
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
|
||||
// ❌ Works, but plants the secret in the job row. Prefer inherit:.
|
||||
{
|
||||
"cmd": "gbrain stats",
|
||||
"cwd": "/data/gbrain",
|
||||
@@ -141,14 +140,14 @@ wrote things like:
|
||||
}
|
||||
```
|
||||
|
||||
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
|
||||
This plants plaintext secrets in `minion_jobs.data` (DB row) and in the
|
||||
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
|
||||
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
|
||||
forbid that pattern — the validator trusts the agent — but **prefer
|
||||
or a shared brain via the mounts feature) sees the URL. The validator
|
||||
doesn't forbid the pattern — it trusts the agent — but **prefer
|
||||
`inherit:`** for any secret you want kept out of the row. Names land in the
|
||||
row; values resolve at child-spawn from the worker's config.
|
||||
|
||||
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
|
||||
**Scope:** `inherit:` is **free-form**. Pass any snake_case
|
||||
config-key name and the worker resolves the value from `loadConfig()` at
|
||||
child-spawn time:
|
||||
|
||||
|
||||
@@ -53,6 +53,11 @@ gbrain reindex-search-vector --dry-run # preview: language + row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The stamp survives later schema work: `initSchema()` — including the replay
|
||||
behind `gbrain init --migrate-only` on every upgrade — applies the schema
|
||||
template under the configured language, so it re-creates the trigger
|
||||
functions as they already are instead of reverting them to `english`.
|
||||
|
||||
The command recreates both trigger functions under the new language and
|
||||
backfills every existing `pages` and `content_chunks` row in batches,
|
||||
streaming progress to stderr. It is idempotent: re-running with the same
|
||||
|
||||
@@ -5,6 +5,11 @@ is a `source`: a logical brain-within-the-brain with its own slug
|
||||
namespace, its own sync state, and its own federation policy. The rest
|
||||
of this guide walks the three canonical scenarios.
|
||||
|
||||
(Sources are the *within-one-database* axis. If you want to connect a
|
||||
whole separate database — a team-published brain with its own access
|
||||
policy — that's the *brain* axis: `gbrain mounts add`. See
|
||||
`docs/architecture/brains-and-sources.md` for the two-axis topology.)
|
||||
|
||||
## The three scenarios
|
||||
|
||||
### 1. Unified knowledge recall (wiki + gstack)
|
||||
@@ -66,14 +71,14 @@ gbrain search "tech layoffs" --source yc-media,garrys-list
|
||||
### 3. Mixed (wiki federated + sessions isolated)
|
||||
|
||||
Your main wiki is federated with a few trusted sources. Your session
|
||||
transcripts (coming in v0.18) land in a separate isolated source so
|
||||
they don't dominate every search result.
|
||||
transcripts (`gbrain transcripts` ingests them) land in a separate
|
||||
isolated source so they don't dominate every search result.
|
||||
|
||||
```bash
|
||||
# Federated sources
|
||||
gbrain sources add gstack --path ~/.gstack --federated
|
||||
|
||||
# Isolated source (future v0.18 — sessions use this shape today for ingest)
|
||||
# Isolated source for session transcripts
|
||||
gbrain sources add sessions --path ~/.claude/sessions --no-federated
|
||||
```
|
||||
|
||||
@@ -104,14 +109,16 @@ Every source row stores `config.federated: boolean` in its JSONB config.
|
||||
| `true` | Source participates in unqualified `gbrain search "X"` results. |
|
||||
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
|
||||
|
||||
The seeded `default` source is `federated=true` so pre-v0.17 brains
|
||||
behave exactly as before — every page appears in search.
|
||||
The seeded `default` source is `federated=true` so single-source brains
|
||||
behave as you'd expect — every page appears in search.
|
||||
|
||||
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
|
||||
|
||||
## Commands
|
||||
|
||||
Full subcommand reference:
|
||||
The most-used subcommands (run `gbrain sources --help` for the full,
|
||||
always-current reference — it also covers `status`, `current`,
|
||||
`set-cr-mode`, and the `push`/`pull` durability surface):
|
||||
|
||||
```
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
|
||||
@@ -119,9 +126,17 @@ gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--
|
||||
--path must be a git repo (or a subdirectory of one) — see
|
||||
"The git requirement for --path sources" below. --force
|
||||
skips that check to register before git-init exists.
|
||||
gbrain sources add <id> --url <git-url> [--pat-file <p>] [--clone-dir <path>] [--no-harden]
|
||||
Clone + register a remote repo in one step; auto-hardens
|
||||
for durability when a PAT is provided (see "Durability" below).
|
||||
gbrain sources list [--json] List all sources with page counts + federation state.
|
||||
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
Cascade-delete a source (pages, chunks, timeline).
|
||||
gbrain sources archive <id> Soft-delete: hide from search, keep data for a TTL
|
||||
grace window. Prefer this over `remove`.
|
||||
gbrain sources restore <id> Un-archive. `gbrain sources archived` lists expiries;
|
||||
`gbrain sources purge` permanently deletes expired archives.
|
||||
gbrain sources remove <id> [--confirm-destructive] [--dry-run]
|
||||
Permanently cascade-delete a source (pages, chunks,
|
||||
timeline). Shows an impact preview first.
|
||||
gbrain sources rename <id> <new-name>
|
||||
Change display name only; id is immutable.
|
||||
gbrain sources default <id> Set the brain-level default.
|
||||
@@ -253,8 +268,8 @@ reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
|
||||
|
||||
## Upgrading an existing brain
|
||||
|
||||
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
|
||||
existing pages all move under `source_id='default'`. Behavior is
|
||||
`gbrain upgrade` runs the needed schema migrations automatically. Your
|
||||
existing pages all live under `source_id='default'`. Behavior is
|
||||
unchanged until you add a second source.
|
||||
|
||||
To add one:
|
||||
@@ -266,13 +281,13 @@ cd ~/.gstack && gbrain sources attach gstack && gbrain sync
|
||||
|
||||
Two commands. The existing default source is untouched.
|
||||
|
||||
## Not in v0.18.0
|
||||
## Related features that build on sources
|
||||
|
||||
- Session transcript ingest (`.jsonl`, raised size cap, session
|
||||
PageType) — v0.18.
|
||||
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
|
||||
- ACL enforcement via caller-identity — v0.17.1.
|
||||
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
|
||||
release after the core plumbing stabilizes.
|
||||
|
||||
All of these build on the `sources` primitive shipped here.
|
||||
- **Session transcript ingest** — `gbrain transcripts` (server-private:
|
||||
raw chat exports stay on the host machine).
|
||||
- **Per-source retention** — `gbrain sources archive` / `archived` /
|
||||
`purge` (soft-delete with a TTL grace window).
|
||||
- **One-shot remote bootstrap** — `gbrain sources add <id> --url <git-url>`
|
||||
(clone + register + auto-harden).
|
||||
- **Access control across brains** — the *brain* axis (`gbrain mounts`);
|
||||
see `docs/architecture/brains-and-sources.md`.
|
||||
|
||||
@@ -43,13 +43,14 @@ on information_needed(topic):
|
||||
# An agent that reaches for the web before checking its own brain
|
||||
# is wasting money and giving worse answers.
|
||||
|
||||
# DISCIPLINE 3: Sync After Every Write (MANDATORY)
|
||||
on brain_write_complete():
|
||||
# DISCIPLINE 3: Sync After Every Repo Write (MANDATORY)
|
||||
on brain_repo_files_changed():
|
||||
gbrain sync
|
||||
# Without this, search results are stale.
|
||||
# The page you just wrote won't appear in gbrain search or gbrain query
|
||||
# until sync runs. Skipping this means the next lookup misses the
|
||||
# most recent data.
|
||||
# `gbrain put` indexes immediately -- pages written through the CLI/MCP
|
||||
# are searchable the moment the command returns. No sync needed there.
|
||||
# But files written DIRECTLY to the brain repo (an editor, a script,
|
||||
# another agent committing markdown) are invisible to search until
|
||||
# `gbrain sync` imports them. If anything touched repo files, sync.
|
||||
|
||||
# DISCIPLINE 4: Daily Heartbeat Check
|
||||
on daily_schedule("09:00"):
|
||||
@@ -62,6 +63,8 @@ on daily_schedule("09:00"):
|
||||
on nightly_schedule("02:00"):
|
||||
# The dream cycle is the most important discipline.
|
||||
# The brain COMPOUNDS overnight.
|
||||
# gbrain ships this as a first-class command: `gbrain dream`.
|
||||
# The pseudocode below is the shape of the work it does.
|
||||
|
||||
# 5a: Entity sweep -- find unlinked mentions
|
||||
pages = gbrain list
|
||||
@@ -103,16 +106,16 @@ on nightly_schedule("02:00"):
|
||||
## Tricky Spots
|
||||
|
||||
1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots.
|
||||
2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes.
|
||||
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
|
||||
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context.
|
||||
2. **Skipping Discipline 3 (sync after repo writes) means stale search results.** A file lands in the brain repo, then you search for it -- and get nothing back. The file exists but isn't imported. Always sync after repo-file writes. (`gbrain put` is exempt: it indexes on write.)
|
||||
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Alice yesterday about the board seat" in passing -- that's a timeline entry on Alice's page, a potential update to her State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
|
||||
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Alice Example" returns a LinkedIn profile. The brain returns the full picture including private context.
|
||||
5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
|
||||
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
|
||||
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
|
||||
3. Write a markdown file directly into the brain repo (not via `gbrain put`), run `gbrain sync`, then `gbrain search` for it. Confirm it appears in results (verifies the sync discipline). A `gbrain put` page should appear in search immediately, with no sync.
|
||||
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Plugin authors guide (v0.15)
|
||||
# Plugin authors guide — subagent definitions
|
||||
|
||||
`gbrain` discovers subagent definitions from outside this repo via
|
||||
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
|
||||
@@ -7,6 +7,11 @@ subagents alongside it, drop a plugin directory on that env path.
|
||||
|
||||
This guide is for plugin authors. The CLI user doesn't need to read it.
|
||||
|
||||
> **Two plugin systems.** This doc covers *subagent definitions*
|
||||
> (markdown prompts the `subagent` job handler runs). Custom *job
|
||||
> handlers* (code the Minion worker executes) are a separate system —
|
||||
> see [plugin-handlers.md](plugin-handlers.md).
|
||||
|
||||
## Minimum viable plugin
|
||||
|
||||
```
|
||||
@@ -65,7 +70,7 @@ You control where your plugin lives on disk; `gbrain` doesn't guess.
|
||||
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
|
||||
with a warning naming both sources.
|
||||
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY:
|
||||
|
||||
- You **cannot** declare new tools.
|
||||
- You **cannot** extend the brain tool allow-list.
|
||||
@@ -76,8 +81,8 @@ with a warning naming both sources.
|
||||
your plugin gives you a loud startup error, not a silent "tool never
|
||||
fires" at 3am.
|
||||
|
||||
v0.16+ may open up plugin-declared tools with a separate contract. Don't
|
||||
expect it.
|
||||
Plugin-declared tools would require a new `plugin_version` contract;
|
||||
nothing under `gbrain-plugin-v1` opens that up.
|
||||
|
||||
## `gbrain.plugin.json`
|
||||
|
||||
@@ -85,9 +90,9 @@ expect it.
|
||||
|------------------|--------|----------|--------------------------------------------------------------------|
|
||||
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"`. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in a future plugin-listing command. |
|
||||
| `description` | string | no | Informational; appears in load/collision warnings. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
@@ -103,8 +108,7 @@ Recognized frontmatter fields:
|
||||
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
|
||||
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
|
||||
|
||||
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
|
||||
may consume more of them.
|
||||
Unknown frontmatter fields are preserved but ignored by the handler.
|
||||
|
||||
## Caveats that will bite you
|
||||
|
||||
@@ -115,8 +119,8 @@ may consume more of them.
|
||||
|
||||
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
|
||||
worker runs on a different host than the `gbrain agent logs` caller,
|
||||
the CLI won't see heartbeats from that worker. v0.16 will unify this;
|
||||
for now assume worker + CLI share a filesystem.
|
||||
the CLI won't see heartbeats from that worker. Assume worker + CLI
|
||||
share a filesystem.
|
||||
|
||||
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
|
||||
invocation. Tools that gate on `remote=true` (file_upload's strict
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
# Plugin handlers — registering host-specific Minion handlers
|
||||
|
||||
GBrain's Minion worker ships with seven built-in handlers: `sync`,
|
||||
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
|
||||
GBrain's Minion worker ships with a full set of built-in handlers,
|
||||
registered by `registerBuiltinHandlers` in `src/commands/jobs.ts` —
|
||||
that registry is the source of truth. Examples: `sync`, `embed`,
|
||||
`lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`, `shell`,
|
||||
`subagent`, `orphans`, `integrity`, plus dream-cycle phases and other
|
||||
maintenance jobs. Submitting an unknown job name with
|
||||
`gbrain jobs submit <name> --follow` prints the full registered list.
|
||||
These cover every background operation the gbrain CLI itself performs.
|
||||
|
||||
Host platforms (OpenClaw deployments, future hosts) register their own
|
||||
@@ -10,6 +15,11 @@ handlers via a plugin bootstrap that imports
|
||||
code, loaded by the worker, with the same trust model as any other
|
||||
code in the host's repo.
|
||||
|
||||
> **Two plugin systems.** This doc covers *job handlers* (code the Minion
|
||||
> worker runs). Custom *subagent definitions* (markdown prompts loaded via
|
||||
> `GBRAIN_PLUGIN_PATH`) are a separate system — see
|
||||
> [plugin-authors.md](plugin-authors.md).
|
||||
|
||||
## Why code, not data
|
||||
|
||||
An earlier design draft shipped `~/.claude/gbrain-handlers.json` where
|
||||
@@ -64,14 +74,23 @@ auto-loads on startup (configurable via a host-provided entry point).
|
||||
|
||||
## Handler contract
|
||||
|
||||
Every handler receives a `MinionJobContext`:
|
||||
Every handler receives a `MinionJobContext` (canonical definition:
|
||||
`src/core/minions/types.ts`). The load-bearing fields:
|
||||
|
||||
```ts
|
||||
interface MinionJobContext {
|
||||
data: Record<string, unknown>; // job params (whatever the cron submit passed)
|
||||
job: MinionJob; // full job row (id, queue, attempts, etc.)
|
||||
signal: AbortSignal; // set to aborted when the worker is shutting down
|
||||
inbox: MinionInbox; // read messages sent to this job while it runs
|
||||
id: number; // job id
|
||||
name: string; // job type
|
||||
data: Record<string, unknown>; // job params (whatever the cron submit passed)
|
||||
attempts_made: number;
|
||||
signal: AbortSignal; // fires on timeout, cancel, pause, or lock loss
|
||||
shutdownSignal: AbortSignal; // fires only on worker SIGTERM/SIGINT
|
||||
deadlineAtMs: number | null; // wall-clock deadline from timeout_at, if set
|
||||
updateProgress(progress: unknown): Promise<void>;
|
||||
updateTokens(tokens: TokenUpdate): Promise<void>;
|
||||
log(message: string | TranscriptEntry): Promise<void>;
|
||||
isActive(): Promise<boolean>; // is the job lock still held?
|
||||
readInbox(): Promise<InboxMessage[]>; // unread messages sent to this job
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -5,13 +5,14 @@ contributed anything. Push-based context inverts that — the brain volunteers
|
||||
relevant pages from the recent conversation, confidence-gated so push noise
|
||||
never becomes worse than pull silence.
|
||||
|
||||
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
|
||||
The push channels share one zero-LLM core (`src/core/context/volunteer.ts`):
|
||||
|
||||
| Channel | Surface | When to use |
|
||||
|---|---|---|
|
||||
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
|
||||
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
|
||||
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
|
||||
| `claude-code` / `codex` | `gbrain hook user-prompt` (registered by `gbrain bootstrap`) | per-prompt injection inside a harness; see "Harness hooks" below |
|
||||
|
||||
## How it decides
|
||||
|
||||
@@ -53,6 +54,42 @@ through a running serve's resolve socket rather than taking the lock. Routing
|
||||
watch through that same socket is a filed follow-up (TODOS.md). Postgres
|
||||
brains are unaffected.
|
||||
|
||||
## Harness hooks (the prompt-time channel)
|
||||
|
||||
`gbrain bootstrap` registers `gbrain hook user-prompt` as a Claude Code
|
||||
`UserPromptSubmit` hook: every prompt is assembled into a per-turn context
|
||||
block (reflex pointers + volunteered pages + hot facts) through a running
|
||||
serve's IPC socket and injected as `additionalContext`. Two properties make
|
||||
this channel production-grade rather than spammy-and-invisible:
|
||||
|
||||
- **Cross-turn dedupe.** The hook reads its OWN previous injections back out
|
||||
of the session transcript (Claude Code records them as structured
|
||||
`hook_additional_context` attachments; only gbrain-marked blocks count) and
|
||||
passes them as prior context — so a page is volunteered once per session,
|
||||
not once per mention. The dedupe horizon is bounded (the recent transcript
|
||||
window, byte-capped), so a marathon session can eventually re-volunteer its
|
||||
oldest injections. The extraction is structural, never substring matching
|
||||
over raw turn text, so a short slug appearing in a tool payload can't
|
||||
over-suppress.
|
||||
- **The feedback loop.** The serve logs each DELIVERED block's volunteered
|
||||
pages and pointers to `context_volunteer_events` under the hook's channel
|
||||
(`claude-code` by default; a codex hook registration passes
|
||||
`--harness codex`). `gbrain volunteer-context --stats` then shows
|
||||
per-harness precision, and `gbrain doctor`'s `volunteer_channels` check
|
||||
shows which channels actually fire, with guidance for the two quiet cases:
|
||||
"hook installed but never registered (restart the session)" and "registered
|
||||
but quiet". Logging happens at the delivery point only — a block abandoned
|
||||
before the serve responded is never counted — and because a delivered
|
||||
response still isn't proof of injection (the hook can trim or drop it
|
||||
client-side), the doctor check reconciles the counts against the hook's own
|
||||
heartbeat and cautions when they diverge.
|
||||
|
||||
The hook lane rides the PGLite serve's IPC socket: on a Postgres brain or a
|
||||
thin-client install the hook stays quiet by design (pull-mode retrieval covers
|
||||
those; extending the lane is a filed follow-up in TODOS.md).
|
||||
|
||||
Kill switch: `GBRAIN_HOOKS=0`. Install/uninstall: `docs/guides/bootstrap.md`.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | What it does |
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Queue operations runbook
|
||||
|
||||
"My queue looks wedged — what do I run?" The commands below are in the order
|
||||
you probably want them. Shipped with v0.19.1 after a production incident
|
||||
where the queue held for 90+ minutes before the operator noticed.
|
||||
you probably want them. Born from a production incident where the queue held
|
||||
for 90+ minutes before the operator noticed.
|
||||
|
||||
## First signal: jobs aren't running
|
||||
|
||||
@@ -23,16 +23,18 @@ container health), but its DB connection died (common behind a transaction
|
||||
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
|
||||
pile up with **0 active**. Liveness checks all pass; nothing crashes.
|
||||
|
||||
As of v0.42.22.0 this self-heals — you usually won't have to do anything:
|
||||
This self-heals — you usually won't have to do anything:
|
||||
|
||||
- **The worker exits on its own dead pool.** Under a supervisor, the worker's
|
||||
DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the
|
||||
supervisor respawns it with a fresh pool.
|
||||
- **The supervisor restarts a worker that stops making progress.** If a queue
|
||||
has claimable work, **0 live-lock active jobs**, and no completions for 15
|
||||
minutes while the child is alive, the supervisor restarts it (covers stuck
|
||||
handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
|
||||
`--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables).
|
||||
minutes across 3 consecutive health checks while the child is alive, the
|
||||
supervisor restarts it (covers stuck handlers too, not just dead pools).
|
||||
These thresholds are built in — there are no CLI flags to tune them. A
|
||||
restart-loop breaker caps wedge restarts at 3 per 30-minute window, then
|
||||
switches to a one-shot `wedge_restart_loop` alert in the audit log.
|
||||
|
||||
The signal is loud now — check either:
|
||||
|
||||
@@ -45,8 +47,12 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
|
||||
stale completions). Manual fix if you ever need it:
|
||||
|
||||
```bash
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool
|
||||
gbrain jobs retry <id> # dead-lettered jobs
|
||||
# Restart the supervisor with a fresh pool. `start` alone runs in the
|
||||
# FOREGROUND (blocks); use --detach to get your shell back.
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start --detach --json
|
||||
|
||||
# Re-queue any jobs that were dead-lettered during the wedge.
|
||||
gbrain jobs retry <id>
|
||||
```
|
||||
|
||||
## Triage commands
|
||||
@@ -101,9 +107,9 @@ claiming. Start one:
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
## Follow-ups tracked for v0.20+
|
||||
## Related
|
||||
|
||||
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
|
||||
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
|
||||
subcheck both need this).
|
||||
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
|
||||
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
|
||||
exit codes, and per-platform deployment (systemd / Fly / Render).
|
||||
- [Minions shell jobs](minions-shell-jobs.md) — the `shell` job type's
|
||||
security model and error table.
|
||||
|
||||
@@ -113,6 +113,22 @@ fi
|
||||
send_notification "$OUTPUT"
|
||||
```
|
||||
|
||||
### GBrain-native hooks
|
||||
|
||||
Two places gbrain already understands quiet hours natively — use these
|
||||
before rolling your own gate for the same job:
|
||||
|
||||
- **Self-upgrade** — `auto` mode only applies upgrades during quiet hours,
|
||||
configured via `gbrain config set self_upgrade.quiet_hours
|
||||
'{"start":23,"end":8,"tz":"US/Pacific"}'`. See
|
||||
[upgrades-auto-update.md](upgrades-auto-update.md).
|
||||
- **Cron prompts** — schedule-driven notification jobs should carry the
|
||||
gate described in this doc; [cron-schedule.md](cron-schedule.md) covers
|
||||
the scheduling side.
|
||||
|
||||
The shell pattern below is for everything else: your own cron jobs,
|
||||
collectors, and notification paths that gbrain doesn't gate for you.
|
||||
|
||||
### Configurable Hours
|
||||
|
||||
Some users want different quiet hours. Store the config:
|
||||
@@ -140,7 +156,11 @@ Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring)
|
||||
skill reads and clears the held directory. Orphaned held files mean the
|
||||
pickup integration is broken.
|
||||
|
||||
3. **Timezone auto-detection is fragile.** Calendar-based timezone detection
|
||||
3. **`/tmp` doesn't survive reboots (or, on macOS, periodic cleanup).** If a
|
||||
held message must not be lost across a restart, use a durable held
|
||||
directory (e.g. `~/.local/state/cron-held/`) instead of `/tmp/cron-held/`.
|
||||
|
||||
4. **Timezone auto-detection is fragile.** Calendar-based timezone detection
|
||||
relies on the user having airline/hotel events with location data. If the
|
||||
user books travel without calendar entries, the system won't detect the
|
||||
move. Fall back to activity-hour analysis (responding at 3 AM PT = probably
|
||||
|
||||
@@ -9,8 +9,9 @@ Separate agent behavior (replaceable) from world knowledge (permanent) into two
|
||||
Without this: agent config and world knowledge are mixed together. Switch agents
|
||||
and you lose your knowledge. Switch knowledge tools and you lose your agent setup.
|
||||
|
||||
With this: your brain (14,700+ files of people, companies, meetings, ideas)
|
||||
survives any agent swap. Your agent config survives any knowledge tool swap.
|
||||
With this: your brain (thousands of files of people, companies, meetings,
|
||||
ideas) survives any agent swap. Your agent config survives any knowledge
|
||||
tool swap.
|
||||
|
||||
## Implementation
|
||||
|
||||
@@ -119,7 +120,15 @@ without losing your agent setup.
|
||||
notes). The agent repo contains operational config. Different access controls.
|
||||
|
||||
**GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep
|
||||
the search index current. The agent repo is never indexed by GBrain.
|
||||
the search index current. The agent repo is not indexed by default.
|
||||
|
||||
**Multi-source nuance.** With multi-source brains
|
||||
([multi-source-brains.md](multi-source-brains.md)), "the brain repo" means
|
||||
"each registered source." You CAN deliberately register a non-brain repo
|
||||
(e.g. `~/.gstack`) as its own isolated or federated source — that's a
|
||||
conscious registration with its own slug namespace, not a violation of the
|
||||
boundary. The rule below is about *unregistered, accidental* indexing of
|
||||
agent config.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
@@ -133,9 +142,12 @@ the search index current. The agent repo is never indexed by GBrain.
|
||||
belongs in the brain. Agent configs, skills, cron jobs, and operational
|
||||
state are replaceable. People, companies, ideas, and meetings are not.
|
||||
|
||||
3. **Don't index the agent repo.** GBrain indexes the brain repo only.
|
||||
Running `gbrain sync` against the agent repo pollutes search results
|
||||
with operational config instead of world knowledge.
|
||||
3. **Don't casually index the agent repo.** Running `gbrain sync` against
|
||||
the agent repo pollutes search results with operational config instead
|
||||
of world knowledge. (Registering it deliberately as an isolated source
|
||||
is different — see the multi-source nuance above. Pin the working
|
||||
directory to the right source with a `.gbrain-source` dotfile via
|
||||
`gbrain sources attach <id>`.)
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
+43
-20
@@ -34,35 +34,35 @@ docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
|
||||
|
||||
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
|
||||
|
||||
## v0.26.7 — auto-RLS event trigger and one-time backfill
|
||||
## Auto-RLS: the event trigger and the one-time backfill
|
||||
|
||||
Starting in v0.26.7 (migration v35), gbrain ships two changes that close the
|
||||
gap where a table could exist in your `public` schema without RLS for any
|
||||
amount of time at all.
|
||||
gbrain ships two mechanisms (schema migration v35) that close the gap where a
|
||||
table could exist in your `public` schema without RLS for any amount of time
|
||||
at all.
|
||||
|
||||
**1. The event trigger.** A Postgres DDL event trigger named
|
||||
`auto_rls_on_create_table` runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY`
|
||||
on every newly created `public.*` table. It covers `CREATE TABLE`,
|
||||
`CREATE TABLE AS … SELECT`, and `SELECT … INTO` — every syntax Postgres
|
||||
reports as a table-creation command. Tables created by gbrain itself, by
|
||||
your other apps sharing the same Supabase project (Baku, Hermes, anything),
|
||||
or by a human running raw SQL all get RLS enabled the moment they exist.
|
||||
any other app sharing the same Supabase project, or by a human running raw
|
||||
SQL all get RLS enabled the moment they exist.
|
||||
Non-`public` schemas (`auth`, `storage`, `realtime`, etc.) are explicitly
|
||||
ignored — Supabase manages those, and we should not touch them.
|
||||
|
||||
**2. The one-time backfill.** When you upgrade to v0.26.7, the migration
|
||||
**2. The one-time backfill.** The first upgrade that applies migration v35
|
||||
walks every existing `public.*` base table whose RLS is off and whose comment
|
||||
doesn't carry the `GBRAIN:RLS_EXEMPT` exemption (see below) and enables RLS
|
||||
on each. After the upgrade, `gbrain doctor`'s `rls` check should be a no-op
|
||||
on every brain.
|
||||
|
||||
### Breaking change: read this before upgrading
|
||||
### Read this before upgrading a pre-auto-RLS brain
|
||||
|
||||
If you have public tables that are intentionally RLS-off and you want them
|
||||
to stay that way, you MUST add the `GBRAIN:RLS_EXEMPT` comment **before**
|
||||
running `gbrain upgrade` to v0.26.7. The backfill flips RLS on for any public
|
||||
table that doesn't carry the exact comment contract documented below. There
|
||||
is no `--dry-run` flag on the migration.
|
||||
the upgrade that applies migration v35. The backfill flips RLS on for any
|
||||
public table that doesn't carry the exact comment contract documented below.
|
||||
There is no `--dry-run` flag on the migration.
|
||||
|
||||
The minimum cost of getting this wrong is one round-trip: the operator runs
|
||||
the SQL to enable RLS on a table that should have been exempt, then
|
||||
@@ -71,7 +71,7 @@ prevent a re-flip on a later doctor run. No data is lost.
|
||||
|
||||
### Cross-app implications
|
||||
|
||||
If a non-gbrain app (Baku, Hermes, a script you wrote, anything) creates
|
||||
If a non-gbrain app (a side project, a script you wrote, anything) creates
|
||||
tables in the same Supabase project, the trigger will enable RLS on those
|
||||
tables too. Two ways to handle that:
|
||||
|
||||
@@ -90,17 +90,40 @@ ship a policy.
|
||||
|
||||
### What if the trigger gets dropped?
|
||||
|
||||
`gbrain doctor` includes a new `rls_event_trigger` check that verifies the
|
||||
`gbrain doctor` includes an `rls_event_trigger` check that verifies the
|
||||
trigger is installed and enabled. If you drop it manually for any reason
|
||||
(debugging, migration testing, anything), doctor warns and gives you the
|
||||
recovery command:
|
||||
(debugging, migration testing, anything), doctor warns and points you here.
|
||||
|
||||
```
|
||||
gbrain apply-migrations --force-retry 35
|
||||
Recreate it by re-running the trigger DDL from migration v35 — idempotent
|
||||
(`CREATE OR REPLACE` + `DROP EVENT TRIGGER IF EXISTS`), safe to paste into
|
||||
psql as a BYPASSRLS role (e.g. `postgres`):
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION auto_enable_rls()
|
||||
RETURNS event_trigger AS $$
|
||||
DECLARE
|
||||
obj record;
|
||||
BEGIN
|
||||
FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands()
|
||||
WHERE object_type = 'table'
|
||||
AND schema_name = 'public'
|
||||
LOOP
|
||||
EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', obj.object_identity);
|
||||
END LOOP;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP EVENT TRIGGER IF EXISTS auto_rls_on_create_table;
|
||||
CREATE EVENT TRIGGER auto_rls_on_create_table
|
||||
ON ddl_command_end
|
||||
WHEN TAG IN ('CREATE TABLE', 'CREATE TABLE AS', 'SELECT INTO')
|
||||
EXECUTE FUNCTION auto_enable_rls();
|
||||
```
|
||||
|
||||
Re-running migration v35 is idempotent — it `DROP EVENT TRIGGER IF EXISTS`
|
||||
and recreates cleanly.
|
||||
(This is the same DDL migration v35 runs — the canonical copy lives in the
|
||||
`MIGRATIONS` array in `src/core/migrate.ts`. There's no CLI shortcut:
|
||||
`gbrain apply-migrations --force-retry` targets the vX.Y.Z orchestrator
|
||||
registry, not numeric schema migrations like v35.)
|
||||
|
||||
### Why no FORCE ROW LEVEL SECURITY?
|
||||
|
||||
@@ -147,7 +170,7 @@ Rules:
|
||||
```sql
|
||||
ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY;
|
||||
COMMENT ON TABLE public.expenses_ramp IS
|
||||
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22';
|
||||
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=you, 2026-04-22';
|
||||
```
|
||||
|
||||
After that, `gbrain doctor` reports:
|
||||
|
||||
@@ -127,9 +127,9 @@ you can use as a reference shape. The skillpack story for distributing
|
||||
your own resolvers across machines is covered in
|
||||
[skillpacks as scaffolding](skillpacks-as-scaffolding.md).
|
||||
|
||||
## The compact list format (v0.41.7.0)
|
||||
## The compact list format
|
||||
|
||||
GBrain's resolver parser used to require markdown tables:
|
||||
GBrain's resolver parser originally required markdown tables:
|
||||
|
||||
```markdown
|
||||
| Trigger | Skill |
|
||||
@@ -146,14 +146,14 @@ format that scales better:
|
||||
- **flight-tracker**: track my flight | flight status | when does my flight land
|
||||
```
|
||||
|
||||
Before v0.41.7.0, `gbrain doctor` only spoke the table dialect. On a
|
||||
306-skill compact-format resolver, the doctor reported every skill as
|
||||
unreachable: **238 FAIL errors on every doctor run**. The parser was
|
||||
silently treating the compact dialect as zero skills.
|
||||
When `gbrain doctor` only spoke the table dialect, a 306-skill
|
||||
compact-format resolver reported every skill as unreachable: **238 FAIL
|
||||
errors on every doctor run**. The parser was silently treating the compact
|
||||
dialect as zero skills.
|
||||
|
||||
v0.41.7.0 ships dual-format support. The same `parseResolverEntries`
|
||||
function reads both table rows and list rows in the same file, with the
|
||||
v0.31.7 multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
|
||||
Today the parser supports both. The same `parseResolverEntries`
|
||||
function reads table rows and list rows in the same file, with the
|
||||
multi-resolver merge (skillpack `skills/RESOLVER.md` + workspace
|
||||
`../AGENTS.md`) folding everything into one unified view. Run `gbrain doctor`
|
||||
and the 238 FAILs collapse to 0.
|
||||
|
||||
@@ -267,8 +267,7 @@ I initially converted my resolver from a clean list format to a table
|
||||
format because the validator only spoke tables. That was wrong. When a
|
||||
tool fails against valid data, the right move is to fix the tool, not
|
||||
reshape the data. The list format was correct, compact, readable, easy
|
||||
to maintain. The parser needed to support both shapes. v0.41.7.0 is
|
||||
that fix.
|
||||
to maintain. The parser needed to support both shapes — and now it does.
|
||||
|
||||
The same principle applies everywhere in agent systems. Your SKILL.md is
|
||||
the source of truth. Your AGENTS.md is the source of truth. Your resolver
|
||||
@@ -295,6 +294,40 @@ architecture that gets you from 10 to 50. That's normal. Systems that
|
||||
scale change shape. The important thing is that each tier preserves full
|
||||
capability. You're organizing, not deleting.
|
||||
|
||||
## Plugin bundling is a curation decision
|
||||
|
||||
Not every skill in `skills/` reaches downstream installs. The plugin
|
||||
manifest (`openclaw.plugin.json`) is the bundled set; everything else is a
|
||||
recorded exclusion in `skills/plugin-exclusions.json`, each with a reason.
|
||||
The two are test-pinned in both directions: every manifest skill is either
|
||||
bundled or a recorded exclusion, and no skill is both. Adding a skill to
|
||||
the tree does NOT ship it — bundling is an explicit decision, and an
|
||||
unbundled skill never reaches a downstream install. When you write a new
|
||||
skill, decide (and record) which side of that line it lives on.
|
||||
|
||||
`bun run gate:skills` (`scripts/skills-commit-gate.sh`) is the per-commit gate
|
||||
for any change under `skills/`. It runs the conformance + resolver +
|
||||
plugin-manifest tests, `check-resolvable --strict`, the `skills.lock.json`
|
||||
regen + freshness check, and `check-skill-refs` in seconds — run it before
|
||||
committing a skills change so the membership/closure and `plugin.version`
|
||||
assertions fail locally instead of in CI.
|
||||
|
||||
## When a skill misroutes
|
||||
|
||||
Treat a misroute like a failing test, because it becomes one. First
|
||||
reproduce it as a fixture in the skill's `routing-eval.jsonl` — the utterance
|
||||
that misrouted, with the expected skill (or `null`). Rewrite the misrouted
|
||||
utterance onto placeholder entities (`alice-example`, `acme-example`) before
|
||||
committing the fixture — same rule as skill-autobench; a routing fixture is a
|
||||
public artifact and must not carry a real contact or company name. Only then
|
||||
fix the cause:
|
||||
usually a trigger in the skill's frontmatter or its row in
|
||||
`skills/RESOLVER.md`. Regenerate the lock (`bun run
|
||||
scripts/generate-skills-manifest.ts`) and the llms bundles (`bun run
|
||||
build:llms`), verify with `gbrain check-resolvable --strict`, and ship it as
|
||||
a MICRO release. Downstream installs heal on their next upgrade — the fix
|
||||
travels with the skillpack, not with a support thread.
|
||||
|
||||
## Related
|
||||
|
||||
- [Skill development cycle](skill-development.md) — the 5-step loop for
|
||||
|
||||
+128
-28
@@ -1,54 +1,152 @@
|
||||
# Search Modes
|
||||
|
||||
## Goal
|
||||
Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result.
|
||||
Two decisions shape every gbrain lookup, and this guide covers both:
|
||||
|
||||
## What the User Gets
|
||||
Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls.
|
||||
1. **Which mode bundle** your brain runs — `conservative` / `balanced` /
|
||||
`tokenmax`, the named cost-knob presets that control cache, token budget,
|
||||
query expansion, and result count. This is the config-level decision you
|
||||
make once (at `gbrain init` or via `gbrain config set search.mode`).
|
||||
2. **Which lookup verb** to use per call — `gbrain search` (keyword),
|
||||
`gbrain query` (hybrid), or `gbrain get` (direct). This is the
|
||||
per-lookup decision an agent makes on every question.
|
||||
|
||||
## Implementation
|
||||
## The three mode bundles
|
||||
|
||||
A search mode is a named preset that sets every search-cost knob at once.
|
||||
The bundles are frozen in `src/core/search/mode.ts` (`MODE_BUNDLES`):
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `relationalRetrieval` | false | **true** | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
- **`conservative`** — smallest payloads. Pairs naturally with a cheap
|
||||
downstream model (Haiku-class) or a high query volume.
|
||||
- **`balanced`** — the default and the fallback when no mode is set.
|
||||
- **`tokenmax`** — no token budget, LLM query expansion on, 50 results.
|
||||
Pairs with an expensive downstream model you want fully fed.
|
||||
|
||||
Two of the knobs deserve a sentence:
|
||||
|
||||
- **`expansion`** rewrites your query into multiple variants via a cheap
|
||||
LLM call per search (adds roughly $1.50 per 1K queries) — better recall,
|
||||
small extra cost.
|
||||
- **`relationalRetrieval`** adds a graph-walk recall arm for relational
|
||||
questions ("who invested in X", "what connects A and B"); it's a pure
|
||||
no-op for non-relational queries. The `query` op's `relational` flag
|
||||
forces it on/off per call.
|
||||
|
||||
### Setting and resolving the mode
|
||||
|
||||
```bash
|
||||
gbrain config set search.mode tokenmax
|
||||
```
|
||||
|
||||
Per-knob resolution (highest first):
|
||||
|
||||
per-call SearchOpts → per-key config override (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in bare `hybridSearch`, not just the cached wrapper,
|
||||
so eval replays test the same mode-affected behavior as the production
|
||||
`query` op. The query cache folds the active knobs into its key
|
||||
(`knobs_hash`), so switching modes never serves you a stale result set
|
||||
from a different configuration.
|
||||
|
||||
### Cost intuition
|
||||
|
||||
gbrain's own cost is rounding error; what the mode really controls is how
|
||||
many tokens your *downstream agent* pays to read per query. The
|
||||
corner-to-corner spread is ~25x once you pair mode with downstream model.
|
||||
Rough anchors at 10K queries/month, full payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku-class (\$1/M in) | Sonnet-class (\$3/M in) | Opus-class (\$5/M in) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K tok) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K tok) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K tok) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly with volume. Cache hits cut all numbers ~50%; disciplined
|
||||
prompt caching in the agent loop cuts further. Mismatched pairings waste
|
||||
capacity in both directions — a tokenmax payload overwhelms a cheap model,
|
||||
a conservative payload starves an expensive one. The full methodology and
|
||||
realistic-scale walkthrough live in
|
||||
[`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
### CLI surfaces
|
||||
|
||||
```bash
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
gbrain search diagnose "<query>" --target <slug>
|
||||
# trace where a page surfaces (or fails to)
|
||||
# across the keyword/vector/alias/hybrid layers
|
||||
```
|
||||
|
||||
The mode picker runs inside `gbrain init` (non-TTY auto-selects `balanced`).
|
||||
|
||||
## Choosing a lookup verb (search vs query vs get)
|
||||
|
||||
Independent of which bundle is active, every individual lookup should use
|
||||
the cheapest verb that answers the question.
|
||||
|
||||
```
|
||||
on user_asks_about(topic):
|
||||
# Decision tree: pick the right search mode
|
||||
# Decision tree: pick the right lookup verb
|
||||
|
||||
if know_exact_slug(topic):
|
||||
# MODE 3: Direct get -- instant, no search overhead
|
||||
# Direct get -- instant, no search overhead
|
||||
result = gbrain get <slug>
|
||||
# e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi
|
||||
# e.g., "Tell me about Alice" -> gbrain get alice-example
|
||||
# Returns the FULL page -- compiled truth + timeline
|
||||
|
||||
elif topic.is_exact_name or topic.is_keyword:
|
||||
# MODE 1: Keyword search -- fast, no embeddings needed, day-one ready
|
||||
# MODE 1: Cheap-hybrid search -- vector + keyword + RRF, NO LLM
|
||||
# expansion. Embeds the query when embeddings are configured; the
|
||||
# keyword arm still works day-one without them (keyword-only is
|
||||
# also available via the search.mcp_keyword_only opt-out).
|
||||
results = gbrain search "{name_or_keyword}"
|
||||
# e.g., "Find anything about Series A" -> gbrain search "Series A"
|
||||
# Returns CHUNKS, not full pages
|
||||
|
||||
# IMPORTANT: keyword search returns chunks
|
||||
# IMPORTANT: search returns chunks
|
||||
# If the chunk confirms relevance, THEN load the full page:
|
||||
if chunk.confirms_relevance:
|
||||
full_page = gbrain get <slug_from_chunk>
|
||||
|
||||
elif topic.is_semantic_question:
|
||||
# MODE 2: Hybrid search -- semantic + keyword, needs embeddings
|
||||
elif topic.is_semantic_question or topic.is_concept_or_landscape:
|
||||
# MODE 2: Full hybrid -- adds multi-query LLM expansion on top of
|
||||
# vector + keyword + RRF. Owns concept / landscape / "all-of-X"
|
||||
# questions: expansion recovers synonym- and outcome-phrased
|
||||
# matches a single embedding misses. Costs one LLM expansion call
|
||||
# per query -- worth it for these question shapes.
|
||||
results = gbrain query "{natural language question}"
|
||||
# e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts"
|
||||
# Returns ranked chunks via vector + keyword + RRF
|
||||
# e.g., "all the companies doing offshore wind" -> gbrain query "..."
|
||||
# Returns ranked chunks via vector + keyword + expansion + RRF
|
||||
|
||||
# Same rule: chunks first, then get full page if needed
|
||||
if chunk.confirms_relevance:
|
||||
full_page = gbrain get <slug_from_chunk>
|
||||
|
||||
# Quick reference:
|
||||
# | Mode | Command | Needs Embeddings | Speed | Best For |
|
||||
# |---------|----------------------|------------------|---------|---------------------------------|
|
||||
# | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches |
|
||||
# | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match |
|
||||
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
|
||||
# | Mode | Command | Needs Embeddings | Speed | Best For |
|
||||
# |-------------|----------------------|------------------|---------|-------------------------------------------|
|
||||
# | Cheap-hybrid| gbrain search "term" | Uses if present | Fastest | Known names, exact tokens |
|
||||
# | Full hybrid | gbrain query "..." | Yes | Fast | Concept / landscape / "all-of-X", synonyms |
|
||||
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
|
||||
|
||||
# Progression over time:
|
||||
# Day 1: keyword search (works without embeddings)
|
||||
# After first embed: hybrid search unlocked
|
||||
# Day 1: search (keyword arm works without embeddings)
|
||||
# After first embed: vector arm + full hybrid (query) unlocked
|
||||
# Once you know slugs: direct get for speed
|
||||
|
||||
# Precedence for conflicting information within a page:
|
||||
@@ -58,21 +156,23 @@ on user_asks_about(topic):
|
||||
# 4. External sources (web search, APIs)
|
||||
```
|
||||
|
||||
## Tricky Spots
|
||||
### Tricky Spots
|
||||
|
||||
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
|
||||
2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available.
|
||||
3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug.
|
||||
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page.
|
||||
5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
|
||||
2. **Search works without embeddings.** On day one before any embedding run, `gbrain search` still works (the keyword arm carries it; the vector arm joins once embeddings exist). Don't tell the user "search isn't available yet" -- search is always available.
|
||||
3. **Don't use full hybrid for known names.** `gbrain query "Alice Example"` wastes an LLM expansion call. Use `gbrain search "Alice Example"` or better yet `gbrain get alice-example` if you know the slug.
|
||||
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Alice" -- get the full page.
|
||||
5. **Full hybrid needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
|
||||
6. **A populated `gbrain search` result set is not proof you found everything.** Search runs without query expansion, so synonym- and outcome-phrased matches can be missed even when it returns plenty of hits. For "find every / all / the landscape of" questions, use `gbrain query`; for literal exhaustive enumeration ("list every page of type X"), use `list_pages` pagination. A nonzero count is not a completeness signal.
|
||||
|
||||
## How to Verify
|
||||
### How to Verify
|
||||
|
||||
1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references.
|
||||
1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references.
|
||||
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
|
||||
3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline.
|
||||
3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline.
|
||||
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
|
||||
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
|
||||
6. Run `gbrain search modes` -- confirm the active mode bundle and any per-key overrides are what you expect.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -40,7 +40,9 @@ Show the user the results. Get feedback.
|
||||
- Revise the process based on what you learned.
|
||||
|
||||
**Step 4: Codify into a Skill.**
|
||||
Write the SKILL.md. Either:
|
||||
Write the SKILL.md (`gbrain skillify scaffold <name>` scaffolds the tree for
|
||||
you; `gbrain skillopt` optimizes an existing skill against a benchmark).
|
||||
Either:
|
||||
- **New skill** -- genuinely new capability
|
||||
- **Add to existing skill** -- variation of something that exists (parameterize it)
|
||||
|
||||
@@ -63,7 +65,8 @@ Skills should be **Mutually Exclusive, Collectively Exhaustive**:
|
||||
- Each signal source has exactly ONE owner skill
|
||||
- Two skills creating the same brain page = MECE violation
|
||||
|
||||
**Example ownership (no overlap):**
|
||||
**Example ownership (no overlap — illustrative; your skill roster will
|
||||
differ):**
|
||||
|
||||
| Signal Source | Owner Skill | Creates |
|
||||
|--------------|-------------|---------|
|
||||
|
||||
@@ -65,14 +65,14 @@ For each step:
|
||||
more than 0.05.
|
||||
6. **Commit.** D8 history-intent-first 5-step atomic write — crash-safe.
|
||||
|
||||
After each epoch with no improvement: D6 slow-update fires one meta-edit
|
||||
proposal (this lives in v0.42 follow-up; v1 emits the audit event).
|
||||
After each epoch with no improvement: D6 slow-update fires. Today it emits
|
||||
the audit event only; the full meta-edit proposal is a tracked follow-up.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `--benchmark <path>` | `skills/<n>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
|
||||
| `--benchmark <path>` | `skills/<name>/skillopt-benchmark.jsonl` | Path to benchmark JSONL |
|
||||
| `--bootstrap-from-skill` | off | Generate a starter benchmark from SKILL.md (recommended; no routing-eval needed) |
|
||||
| `--bootstrap-tasks N` | 15 | How many starter tasks `--bootstrap-from-skill` generates (max 50) |
|
||||
| `--bootstrap-from-routing` | off | Auto-build benchmark from routing-eval.jsonl |
|
||||
@@ -136,7 +136,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
|
||||
|
||||
- **No benchmark.** Optimizing against guesses is worse than not optimizing.
|
||||
- **Write-flavored skills.** Skills whose job is to `put_page` heavily can't
|
||||
use the v1 read-only sandbox; mocked-write capture is a v0.42 follow-up.
|
||||
use the read-only sandbox; mocked-write capture is a tracked follow-up.
|
||||
- **Tiny benchmarks (<10 tasks).** D_sel < 5 refuses by default; meaningful
|
||||
validation needs ≥20 tasks total per the paper.
|
||||
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
# Skillpacks as scaffolding, not amber
|
||||
|
||||
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
|
||||
scaffold + reference library. This guide explains the model and the
|
||||
workflow.
|
||||
`gbrain skillpack` is a scaffold + reference library, not a package
|
||||
manager. This guide explains the model and the workflow.
|
||||
|
||||
## Why we changed it
|
||||
## Why it works this way
|
||||
|
||||
Pre-v0.33 (the "amber" model):
|
||||
An earlier design (the "amber" model):
|
||||
|
||||
- `gbrain skillpack install <name>` copied bundled skills into your
|
||||
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
|
||||
@@ -26,7 +25,16 @@ repo. You scaffold once, you own them, you fork and edit freely. When
|
||||
gbrain ships a new version, you ask "what changed?" — the agent reads
|
||||
the diff and decides what (if anything) to integrate.
|
||||
|
||||
## The five commands
|
||||
## The core workflow commands
|
||||
|
||||
The five commands below are the scaffold-and-own workflow. The full
|
||||
`gbrain skillpack` surface is larger (`list`, `diff`, `check`, `search`,
|
||||
`info`, `registry`, `doctor`, `init`, `pack`, `endorse`, …) — run
|
||||
`gbrain skillpack --help` for the always-current list. One worth calling
|
||||
out here: **`gbrain skillpack init-brain-pack <name>`** scaffolds a
|
||||
*brain-resident* pack inside a brain/source repo (`brain_resident: true`
|
||||
plus a machine-parseable README) that connecting harnesses discover on
|
||||
`gbrain sources add`.
|
||||
|
||||
### `gbrain skillpack scaffold <name> [--workspace PATH]`
|
||||
|
||||
@@ -77,15 +85,15 @@ gbrain skillpack reference book-mirror
|
||||
`reference <name> --apply-clean-hunks` is the auto-apply path. It
|
||||
parses the diff between gbrain's bundle and your local copy, applies
|
||||
every hunk whose pre-change context matches uniquely. **Two-way merge
|
||||
limitation**: without scaffold-time base tracking (intentionally
|
||||
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
|
||||
limitation**: without scaffold-time base tracking (intentionally out of
|
||||
scope), this cannot distinguish "gbrain changed X"
|
||||
from "you changed X." Applied hunks align everything to gbrain. Use
|
||||
`--dry-run` first to preview, or run plain `reference` to inspect the
|
||||
diff before letting auto-apply touch anything.
|
||||
|
||||
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
|
||||
|
||||
One-shot conversion for workspaces on the pre-v0.33 managed-block
|
||||
One-shot conversion for workspaces still on the legacy managed-block
|
||||
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
|
||||
markers and the manifest receipt comment from your resolver file.
|
||||
|
||||
@@ -157,7 +165,7 @@ Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
|
||||
frontmatter, and match the user's intent against every skill's
|
||||
`triggers:` array. When a match scores high enough, invoke that skill.
|
||||
|
||||
This replaces the v0.32 model where `gbrain skillpack install` wrote
|
||||
This replaces the legacy model where `gbrain skillpack install` wrote
|
||||
table rows into your `RESOLVER.md`. Rows are gone (or, for users
|
||||
migrating from the old model, preserved transitionally by
|
||||
`migrate-fence` until they run `scrub-legacy-fence-rows`).
|
||||
@@ -173,7 +181,7 @@ If you're a downstream agent author updating to this model:
|
||||
|
||||
## Removing a scaffolded skill
|
||||
|
||||
There's no `gbrain skillpack uninstall` command in v0.33. The files
|
||||
There's no `gbrain skillpack uninstall` command. The files
|
||||
in your `skills/<slug>/` are first-class members of your repo —
|
||||
delete them like any other code:
|
||||
|
||||
@@ -195,16 +203,17 @@ You own the files. There's no manifest to update, no fence to rebuild.
|
||||
## When to use which command (quick decision tree)
|
||||
|
||||
- **New host repo, want a gbrain skill** → `scaffold`
|
||||
- **Shipping a pack from inside a brain/source repo** → `init-brain-pack`
|
||||
- **gbrain shipped a new version, want to see what's changed**
|
||||
→ `reference` (read-only) or `reference --apply-clean-hunks` (auto)
|
||||
- **Upgrading from v0.32 or earlier** → `migrate-fence` (one-shot)
|
||||
- **Upgrading from the legacy managed-block model** → `migrate-fence` (one-shot)
|
||||
- **Cleanup after `migrate-fence`** → `scrub-legacy-fence-rows`
|
||||
- **Lift your fork's skill back into gbrain** → `harvest` + the
|
||||
`skillpack-harvest` editorial skill
|
||||
|
||||
## What about `install` and `uninstall`?
|
||||
|
||||
Both are removed in v0.33. Running either prints an error pointing at
|
||||
the replacement command. No deprecated alias — this is a clean break.
|
||||
Both are removed. Running either prints an error pointing at the
|
||||
replacement command. No deprecated alias — this is a clean break.
|
||||
If you have existing scripts referencing the old names, update them
|
||||
once and move on.
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
Every fact in the brain traces to where it came from -- who said it, in what context, and when.
|
||||
|
||||
## What the User Gets
|
||||
Without this: six months from now, someone reads a brain page and has no idea if "Pedro co-founded Brex" came from Pedro himself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
|
||||
Without this: six months from now, someone reads a brain page and has no idea if "Alice co-founded widget-co" came from Alice herself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
|
||||
|
||||
## Implementation
|
||||
|
||||
@@ -23,11 +23,11 @@ on brain_write(page, fact):
|
||||
# [Source: Crustdata LinkedIn enrichment, 2026-04-07 12:35 PM PT]
|
||||
elif source.type == "social_media":
|
||||
# MUST include full URL -- not just @handle
|
||||
# [Source: X/@pedroh96 tweet, product launch, 2026-04-07](https://x.com/pedroh96/status/...)
|
||||
# [Source: X/@alice_example tweet, product launch, 2026-04-07](https://x.com/alice_example/status/...)
|
||||
elif source.type == "email":
|
||||
# [Source: email from Sarah Chen re Q2 board deck, 2026-04-05 2:30 PM PT]
|
||||
# [Source: email from Alice Example re Q2 board deck, 2026-04-05 2:30 PM PT]
|
||||
elif source.type == "workspace":
|
||||
# [Source: Slack #engineering, Keith re deploy schedule, 2026-04-06 11:45 AM PT]
|
||||
# [Source: Slack #engineering, Charlie re deploy schedule, 2026-04-06 11:45 AM PT]
|
||||
elif source.type == "web":
|
||||
# [Source: Happenstance research, 2026-04-07 12:35 PM PT]
|
||||
elif source.type == "published":
|
||||
@@ -57,7 +57,7 @@ SOURCE_PRIORITY = [
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Compiled truth is NOT exempt from citations.** "Pedro co-founded Brex" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
|
||||
1. **Compiled truth is NOT exempt from citations.** "Alice co-founded widget-co" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
|
||||
2. **Tweet URLs are mandatory.** `[Source: X/@handle tweet, topic, date]` without a URL is a broken citation. Hundreds of brain pages end up with unreachable tweet references when the URL is omitted. Always: `[Source: X/@handle tweet, topic, date](https://x.com/handle/status/ID)`.
|
||||
3. **"User said it" isn't enough.** WHERE, ABOUT WHAT, WHEN. `[Source: User, direct message, 2026-04-07 12:33 PM PT]` -- not just `[Source: User]`.
|
||||
4. **Don't silently resolve conflicts.** When the user says one thing and an API says another, note the contradiction in compiled truth with both citations. Let the reader decide.
|
||||
@@ -71,5 +71,13 @@ SOURCE_PRIORITY = [
|
||||
4. Check timeline entries on 3 random pages. Each entry should have a source citation with date and context.
|
||||
5. Look for a page where the user stated something that contradicts an API result. Confirm the contradiction is noted, not silently resolved.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/_brain-filing-rules.md` — the canonical citation-format rules every
|
||||
brain write follows (this guide is the narrative walkthrough of the same
|
||||
contract).
|
||||
- `skills/citation-fixer/SKILL.md` — audits and repairs existing pages
|
||||
against that format.
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -2,19 +2,42 @@
|
||||
|
||||
## Goal
|
||||
|
||||
Route sub-agents to the cheapest model that can do the job, saving 10-40x on costs without sacrificing quality.
|
||||
Route sub-agents to the cheapest model that can do the job, saving large
|
||||
multiples on cost without sacrificing quality.
|
||||
|
||||
## What the User Gets
|
||||
|
||||
Without this: every sub-agent runs on Opus ($15/MTok). Entity detection on
|
||||
every message costs $3-5/day. Research tasks cost $10+ each.
|
||||
Without this: every sub-agent runs on your most expensive model. Entity
|
||||
detection fires on every message at top-tier rates; research tasks cost
|
||||
several dollars each.
|
||||
|
||||
With this: entity detection runs on Sonnet ($3/MTok, 5x cheaper). Research
|
||||
runs on DeepSeek ($0.50/MTok, 30x cheaper). Main session stays on Opus for
|
||||
quality. Total cost drops 70-80%.
|
||||
With this: entity detection runs on a cheap fast model, research execution
|
||||
runs on a budget model, and only planning/synthesis touch the expensive
|
||||
model. Total cost drops 70-80%.
|
||||
|
||||
(Illustrative input-token anchors from gbrain's canonical pricing table,
|
||||
`src/core/model-pricing.ts`: Opus-class $5/MTok, Sonnet-class $3/MTok,
|
||||
Haiku-class $1/MTok. Budget providers run well under $1/MTok. Prices
|
||||
drift — the pricing table is the source of truth, not this doc.)
|
||||
|
||||
## Implementation
|
||||
|
||||
### GBrain's native mechanism: model tiers
|
||||
|
||||
Before hardcoding vendors, use gbrain's tier routing. Every gbrain
|
||||
subagent/LLM call resolves through a named tier
|
||||
(`utility` / `reasoning` / `deep` / `subagent`), and you point each tier
|
||||
at whatever model you want once:
|
||||
|
||||
```bash
|
||||
gbrain config set models.tier.subagent anthropic:claude-haiku-4-5
|
||||
gbrain config set models.tier.deep anthropic:claude-opus-4-7
|
||||
```
|
||||
|
||||
Per-call override: `gbrain agent run --model <provider:model>`. The
|
||||
conventions file `skills/conventions/model-routing.md` is the canonical
|
||||
routing policy; this guide is the cost rationale behind it.
|
||||
|
||||
### Routing Table
|
||||
|
||||
| Task Type | Recommended Model | Why |
|
||||
@@ -44,14 +67,16 @@ on_every_message(text):
|
||||
3. FACTS: New info about existing entities -> update timeline
|
||||
4. CITATIONS: Every fact needs [Source: ...] attribution
|
||||
5. Sync changes to brain repo`,
|
||||
model: "sonnet-class", // fast + cheap
|
||||
model: "sonnet-class", // fast + cheap; haiku-class is cheaper still
|
||||
timeout: 120s
|
||||
})
|
||||
```
|
||||
|
||||
**Why Sonnet-class for detection:** Entity detection is pattern matching, not
|
||||
deep reasoning. Sonnet is 5-10x cheaper than Opus and fast enough for async
|
||||
detection. The main session continues on Opus while detection runs in parallel.
|
||||
**Why a cheaper class for detection:** Entity detection is pattern matching,
|
||||
not deep reasoning. Sonnet-class runs at a fraction of Opus-class cost, and
|
||||
Haiku-class at a fraction of that — both fast enough for async detection.
|
||||
The main session continues on your best model while detection runs in
|
||||
parallel.
|
||||
|
||||
### Research Pipeline Pattern
|
||||
|
||||
@@ -64,9 +89,9 @@ For research-heavy tasks, use a multi-model pipeline:
|
||||
```
|
||||
|
||||
**Why this works:** The planning and synthesis steps need taste and judgment
|
||||
(Opus). The execution step is mechanical data gathering (DeepSeek at 25-40x
|
||||
lower cost). You get Opus-quality output at DeepSeek-level cost for 80% of
|
||||
the work.
|
||||
(Opus-class). The execution step is mechanical data gathering (a budget
|
||||
model at a small fraction of the cost). You get top-tier output at
|
||||
budget-model cost for 80% of the work.
|
||||
|
||||
### When to Spawn Sub-Agents
|
||||
|
||||
@@ -82,15 +107,16 @@ the work.
|
||||
|
||||
The main session runs on your best model. Everything else runs on the
|
||||
cheapest model that can do the job. In practice, 60-70% of sub-agent
|
||||
work is entity detection (Sonnet) and research execution (DeepSeek),
|
||||
which are 10-40x cheaper than the main session model.
|
||||
work is entity detection and research execution, which run at a small
|
||||
fraction of the main session model's cost.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
1. **Sonnet, not Opus, for detection.** The most common mistake is running
|
||||
entity detection on Opus. Detection is pattern matching, not deep reasoning.
|
||||
Sonnet is 5-10x cheaper and fast enough. Reserve Opus for the main session
|
||||
where reasoning quality matters.
|
||||
1. **A cheap class, not Opus, for detection.** The most common mistake is
|
||||
running entity detection on Opus-class. Detection is pattern matching, not
|
||||
deep reasoning. Sonnet- or Haiku-class is several times cheaper and fast
|
||||
enough. Reserve Opus-class for the main session where reasoning quality
|
||||
matters.
|
||||
|
||||
2. **Don't block the main thread.** Sub-agents must run asynchronously. If the
|
||||
signal detector runs synchronously, the user waits 30-120 seconds for every
|
||||
@@ -98,10 +124,11 @@ which are 10-40x cheaper than the main session model.
|
||||
a response immediately.
|
||||
|
||||
3. **Cost optimization is multiplicative.** Entity detection runs on every
|
||||
single message. If you use Opus at $15/MTok for detection across 50
|
||||
messages/day, that's $3-5/day just for detection. Sonnet at $3/MTok brings
|
||||
that to $0.60-1.00/day. Over a month, the wrong model choice costs $100+
|
||||
more than necessary.
|
||||
single message, so the per-call price difference compounds across 50+
|
||||
messages/day. Routing detection from Opus-class ($5/MTok in) to
|
||||
Haiku-class ($1/MTok in) is a flat 5x cut on your highest-frequency LLM
|
||||
call — over a month, the wrong model choice for detection alone costs
|
||||
real money. (Current per-model rates: `src/core/model-pricing.ts`.)
|
||||
|
||||
## How to Verify
|
||||
|
||||
|
||||
@@ -16,11 +16,16 @@ benefit-focused bullets, waits for explicit permission, then runs the full
|
||||
upgrade flow including re-reading skills, running migrations, and syncing
|
||||
schema. The user gets new capabilities automatically.
|
||||
|
||||
## Self-upgrade modes (v0.42)
|
||||
## Self-upgrade modes
|
||||
|
||||
gbrain now stays current the way gstack does: it rides invocation frequency. A
|
||||
gbrain stays current the way gstack does: it rides invocation frequency. A
|
||||
throttled, cache-read-only check runs at the start of every `gbrain` invocation
|
||||
(CLI and MCP) and emits an `UPGRADE_AVAILABLE <old> <new>` marker on stderr. No
|
||||
(CLI and MCP) and emits an `UPGRADE_AVAILABLE <old> <new>` marker on stderr. The
|
||||
raw marker line is suppressed when stderr is an interactive TTY (a human sees
|
||||
only the plain `gbrain X -> Y available` sentence, not the machine token); set
|
||||
`GBRAIN_FORCE_UPGRADE_MARKER=1` if an agent harness parses the token but runs
|
||||
under a PTY. `<old>` is always the RUNNING binary's version, so a stale or
|
||||
foreign-written cache never nags about an upgrade this binary already has. No
|
||||
host cron required — every agent kind (Claude Code, Codex, OpenClaw, Hermes, the
|
||||
`gbrain serve` host behind a Perplexity thin client) converges to current by
|
||||
construction. The behavior is governed by one file-plane config key,
|
||||
@@ -44,6 +49,14 @@ execution. The trust model is TLS + GitHub (same as `gbrain upgrade`);
|
||||
signature verification is a tracked follow-up. Apply manually any time with
|
||||
`gbrain self-upgrade`.
|
||||
|
||||
The `auto` quiet-hours window is configured via the
|
||||
`self_upgrade.quiet_hours` config key
|
||||
(`gbrain config set self_upgrade.quiet_hours '{"start":23,"end":8,"tz":"US/Pacific"}'`).
|
||||
The quiet-hours *pattern* itself — gating any notification or background
|
||||
action on the user's local sleep window — is owned by
|
||||
[quiet-hours.md](quiet-hours.md); this doc only covers the self-upgrade
|
||||
hook into it.
|
||||
|
||||
## Implementation
|
||||
|
||||
### The Check (cron-initiated)
|
||||
@@ -163,15 +176,18 @@ Also persist in `~/.gbrain/update-state.json` so it survives agent context reset
|
||||
|
||||
If you loaded this SKILLPACK directly (copied or read from GitHub) without
|
||||
installing gbrain, you can still stay current. Both GBRAIN_SKILLPACK.md and
|
||||
GBRAIN_RECOMMENDED_SCHEMA.md have version markers:
|
||||
GBRAIN_RECOMMENDED_SCHEMA.md carry a `<!-- source: ... -->` header pointing
|
||||
at their canonical copies, and GBRAIN_RECOMMENDED_SCHEMA.md also carries a
|
||||
version marker:
|
||||
|
||||
```bash
|
||||
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md | head -1
|
||||
# Returns: <!-- skillpack-version: X.Y.Z -->
|
||||
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md | head -1
|
||||
# Returns: <!-- schema-version: X.Y.Z -->
|
||||
```
|
||||
|
||||
If the remote version is newer, fetch the full file and replace your local
|
||||
copy. Set up a weekly cron to check automatically.
|
||||
If the remote version is newer (or the remote SKILLPACK content differs from
|
||||
your local copy), fetch the full file and replace your local copy. Set up a
|
||||
weekly cron to check automatically.
|
||||
|
||||
## Tricky Spots
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
|
||||
|
||||
| Provider | env vars | default dims | cost ($/1M tokens) | local? | multimodal? |
|
||||
|---|---|---|---|---|---|
|
||||
| `zeroentropyai` | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no |
|
||||
| `zeroentropyai` (hosted API **shuts down 2026-09-04** — see note below) | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no |
|
||||
| `openai` | `OPENAI_API_KEY` | 1536 | 0.13 | no | no |
|
||||
| `openrouter` | `OPENROUTER_API_KEY` | 1536 | 0.02 | no | model-dependent |
|
||||
| `voyage` | `VOYAGE_API_KEY` | 1024 | 0.18 | no | yes (`voyage-multimodal-3`) |
|
||||
@@ -42,6 +42,8 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
|
||||
|
||||
**Note on local providers.** Ollama and llama-server have no required API key, so they don't show up in env-detection auto-pick. Pick them explicitly with `--embedding-model ollama:<model>` to avoid silently routing to a daemon that may not be running.
|
||||
|
||||
**Note on the ZeroEntropy hosted API.** ZeroEntropy announced (2026-07-24) that its hosted endpoints shut down on **2026-09-04**. A brain still embedding through the hosted API loses semantic retrieval entirely on that date — query embedding uses the same endpoint, so existing vectors become unqueryable, not just new content. Either self-host the Apache-2.0 zembed-1 weights via llama-server/Ollama (keeps every existing vector, no re-embed), or migrate with `gbrain migrate embeddings` — see [the migration guide](../guides/embedding-migration.md). `gbrain doctor` (check `provider_sunset`) flags affected brains and prints the paste-ready command with the brain's actual `--dim` filled in.
|
||||
|
||||
## If first import fails
|
||||
|
||||
If `gbrain import` fails with `expected N dimensions, not M`, run `gbrain doctor`. The output will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. **You should not need to delete `~/.gbrain`.** The bug-class that historically forced `rm -rf` recoveries is closed as of v0.37.
|
||||
|
||||
@@ -60,7 +60,15 @@ Isolation model:
|
||||
|
||||
The write fence is a **write** boundary within a source. It is not a privacy
|
||||
boundary, and it does not make every side effect prefix-clean. As of
|
||||
v0.42.72.0:
|
||||
v0.42.73.2:
|
||||
|
||||
- **The fence follows a delegated write.** When a client with `agent` scope
|
||||
hands work to a subagent via `submit_agent`, that subagent runs under its own
|
||||
slug confinement rather than the parent's OAuth binding. Both confinements are
|
||||
enforced, including on the path where deduplication redirects a write onto an
|
||||
existing page: the redirected target is checked against whichever confinement
|
||||
the calling context actually carries, so delegation does not widen what a
|
||||
client can write.
|
||||
|
||||
- **`add_link`/`remove_link` fence the `from` endpoint only.** A bound client
|
||||
can create an edge pointing AT a page it cannot write; the edge's `context`
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
|
||||
GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it
|
||||
accessible from other devices and AI clients, run `gbrain serve --http`
|
||||
(built-in HTTP transport with bearer auth, Postgres-only ... see
|
||||
[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options.
|
||||
(built-in HTTP transport with OAuth 2.1 + bearer auth, works on both PGLite
|
||||
and Postgres brains — see [DEPLOY.md](DEPLOY.md)) behind a public tunnel.
|
||||
Here are your tunnel options.
|
||||
|
||||
## ngrok (recommended)
|
||||
|
||||
@@ -58,10 +59,11 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits.
|
||||
| Works when laptop is off | No | No | Yes |
|
||||
| Cold start | None | None | None |
|
||||
| Timeout limits | None | None | None |
|
||||
| All 30 operations | Yes | Yes | Yes |
|
||||
| Full remote operation surface (100+ ops, minus `localOnly`) | Yes | Yes | Yes |
|
||||
| Setup time | 5 min | 10 min | 15 min |
|
||||
|
||||
**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth
|
||||
against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap,
|
||||
per-request audit log. Postgres-only by design (PGLite is local-only). See
|
||||
[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables.
|
||||
**Note:** `gbrain serve --http` is the built-in HTTP transport. OAuth 2.1 plus
|
||||
bearer auth against the `access_tokens` table, default-deny CORS, two-bucket rate
|
||||
limit, body cap, per-request audit log. Works on both PGLite and Postgres brains.
|
||||
See [DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and
|
||||
tunables.
|
||||
|
||||
+28
-34
@@ -1,35 +1,39 @@
|
||||
# Connect GBrain to ChatGPT
|
||||
|
||||
**Status (v0.26.0):** Unblocked. GBrain's `gbrain serve --http` ships OAuth 2.1
|
||||
with PKCE, which is the ChatGPT MCP connector's hard requirement. Before v1.0,
|
||||
this was a P0 TODO — the only major AI client that could not connect.
|
||||
ChatGPT's MCP connector requires OAuth 2.1 with PKCE — it does not support
|
||||
bearer-token MCP servers. GBrain's `gbrain serve --http` speaks exactly that,
|
||||
so ChatGPT connects natively.
|
||||
|
||||
ChatGPT does not support bearer-token MCP servers. You must use the OAuth 2.1
|
||||
HTTP server.
|
||||
This page covers only the ChatGPT-specific parts. The full server setup —
|
||||
starting `gbrain serve --http`, the admin bootstrap token, the `/admin`
|
||||
dashboard, tunnels, and `--bind` / `--public-url` — lives in
|
||||
[DEPLOY.md](DEPLOY.md). Do steps 1 (start the server) and 3 (expose it)
|
||||
from there, then come back for the ChatGPT client.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Start the HTTP server
|
||||
### 1. Start and expose the server (DEPLOY.md steps 1 + 3)
|
||||
|
||||
```bash
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
Save the admin bootstrap token printed on stderr. Open
|
||||
`http://localhost:3131/admin` and paste it to access the dashboard.
|
||||
Follow [DEPLOY.md — OAuth 2.1 Setup](DEPLOY.md#oauth-21-setup) to start
|
||||
`gbrain serve --http`, save the admin bootstrap token, and expose the server
|
||||
at a public HTTPS URL (e.g. `https://your-brain.ngrok.app`). ChatGPT's
|
||||
connector auto-discovers the spec-compliant endpoint at
|
||||
`/.well-known/oauth-authorization-server`.
|
||||
|
||||
### 2. Register a ChatGPT client
|
||||
|
||||
ChatGPT uses the authorization code flow with PKCE (browser-based OAuth).
|
||||
Register from the `/admin` dashboard:
|
||||
The ChatGPT-specific delta: ChatGPT uses the **authorization code flow with
|
||||
PKCE** (browser-based OAuth), so the client needs the `authorization_code`
|
||||
grant type and a redirect URI. Register from the `/admin` dashboard:
|
||||
|
||||
1. Click **Register client**.
|
||||
2. Name: `chatgpt`.
|
||||
3. Grant type: `authorization_code`.
|
||||
4. Scopes: `read`, `write` (leave `admin` unchecked for ChatGPT).
|
||||
5. Redirect URI: ChatGPT's OAuth redirect (copy it from the ChatGPT
|
||||
connector setup screen — something like
|
||||
`https://chat.openai.com/connector_platform_oauth_redirect`).
|
||||
5. Redirect URI: ChatGPT's OAuth redirect — **always copy the exact value
|
||||
from the ChatGPT connector setup screen** (it looks like
|
||||
`https://chatgpt.com/connector_platform_oauth_redirect`, but the domain
|
||||
has changed before; trust the setup screen, not this doc).
|
||||
6. Hit **Register**. The credential-reveal modal shows the `client_id` once
|
||||
with Copy and Download JSON buttons. There is no client secret for
|
||||
PKCE-based public clients.
|
||||
@@ -41,22 +45,11 @@ await oauthProvider.registerClientManual(
|
||||
'chatgpt',
|
||||
['authorization_code'],
|
||||
'read write',
|
||||
['https://chat.openai.com/connector_platform_oauth_redirect'],
|
||||
['<ChatGPT redirect URI from the connector setup screen>'],
|
||||
);
|
||||
```
|
||||
|
||||
### 3. Expose the server publicly
|
||||
|
||||
```bash
|
||||
brew install ngrok
|
||||
ngrok http 3131 --url your-brain.ngrok.app
|
||||
```
|
||||
|
||||
Your OAuth issuer URL becomes `https://your-brain.ngrok.app`. ChatGPT's
|
||||
connector auto-discovers the spec-compliant endpoint at
|
||||
`/.well-known/oauth-authorization-server`.
|
||||
|
||||
### 4. Add the connector in ChatGPT
|
||||
### 3. Add the connector in ChatGPT
|
||||
|
||||
1. Open ChatGPT > Settings > Connectors.
|
||||
2. Click **Add connector**.
|
||||
@@ -71,10 +64,11 @@ calls show up in the admin dashboard's live SSE feed in real time.
|
||||
## Scopes
|
||||
|
||||
ChatGPT clients can request any combination of `read`, `write`, `admin`. The
|
||||
scopes granted at consent time are enforced on every tool call. Four
|
||||
operations are `localOnly` and rejected over HTTP regardless of scope:
|
||||
`sync_brain`, `file_upload`, `file_list`, `file_url`. The HTTP server fails
|
||||
closed for any attempt to reach local filesystem surface area.
|
||||
scopes granted at consent time are enforced on every tool call. Operations
|
||||
flagged `localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain`
|
||||
and the `file_*` ops among them) are rejected over HTTP regardless of scope.
|
||||
The HTTP server fails closed for any attempt to reach local filesystem
|
||||
surface area.
|
||||
|
||||
Recommended ChatGPT scope: `read write`. Leave `admin` for your local CLI
|
||||
and the admin dashboard.
|
||||
|
||||
+36
-3
@@ -4,16 +4,28 @@
|
||||
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
|
||||
> end to end, plus the brain-first protocol that makes it worth it. This page is
|
||||
> the connection reference.
|
||||
>
|
||||
> Want the **full agent** — identity, per-turn context, schedules, and a private
|
||||
> repo as its durable body — not just a memory? That's `gbrain bootstrap`:
|
||||
> see the paste block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
|
||||
> Open a new empty folder (bootstrap creates the private repo for you), or make an
|
||||
> empty private repo under your own account and open the clone — bootstrap adopts it.
|
||||
|
||||
## Option 1: Local (recommended, zero server needed)
|
||||
|
||||
```bash
|
||||
claude mcp add gbrain -- gbrain serve
|
||||
claude mcp add gbrain -- gbrain serve --surface verbs
|
||||
```
|
||||
|
||||
That's it. Claude Code spawns `gbrain serve` as a stdio subprocess. No server, no
|
||||
tunnel, no token needed. Works with both PGLite and Supabase engines.
|
||||
|
||||
`--surface verbs` exposes the seven-verb memory protocol (`recall`, `remember`,
|
||||
`entity`, `synthesize`, `forget`, `context_pack`, `delta` — [MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)),
|
||||
the surface built for agents and quickstarts. Drop the flag for the full
|
||||
operation catalog (`get_page`, `put_page`, `search`, graph ops, …) — `full` is
|
||||
the default and what existing installs already run.
|
||||
|
||||
## Option 2: Remote, one command (fastest from a bearer token)
|
||||
|
||||
If GBrain is running somewhere as an HTTP server (`gbrain serve --http`, see the
|
||||
@@ -79,9 +91,30 @@ You should see results from your GBrain knowledge base.
|
||||
> **`list_skills` returns nothing?** Skill discovery is gated by `mcp.publish_skills`
|
||||
> on the host. New brains from `gbrain init` default it ON; brains upgraded from an
|
||||
> older release stay OFF until you opt in. Enable it on the host with
|
||||
> `gbrain config set mcp.publish_skills true`. The core tools (search, query,
|
||||
> get_page, put_page, think, find_experts) work regardless. Note: `capture` is a
|
||||
> `gbrain config set mcp.publish_skills true`. Skill discovery and the core tools
|
||||
> named here (search, query, get_page, put_page, think, find_experts) are
|
||||
> full-surface — on `--surface verbs` the agent sees only the seven memory verbs,
|
||||
> and `list_skills` isn't on the surface at all. Note: `capture` is a
|
||||
> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`.
|
||||
> Why brains differ on the default: [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
|
||||
|
||||
## Ambient recall at session boundaries (v0.45.7)
|
||||
|
||||
Two frozen verbs close the "no question fired" gap for long-lived sessions:
|
||||
`context_pack` (session-start warm-up + post-compaction rehydration) and
|
||||
`delta` ("what changed since my last wake" for heartbeats). Both are zero-LLM,
|
||||
sub-second, world-visibility by default, and available on `--surface verbs`.
|
||||
|
||||
- **Automatic (PGLite brains via `gbrain bootstrap`):** the bootstrap hook
|
||||
installer wires `SessionStart` (injects a warm pack; also fires on
|
||||
post-compaction re-entry, `source=compact`) and `PreCompact` (banks the
|
||||
window's standing entities so that rehydration pack is warm) into
|
||||
`.claude/settings.local.json`. Nothing to call; `GBRAIN_HOOKS=0` disables.
|
||||
- **Manual (any brain, incl. remote/Postgres):** call the verbs yourself at
|
||||
boundaries — `context_pack(entities, budget_tokens)` at session start /
|
||||
after compaction, `delta(session_id, budget_tokens)` on wakes. See
|
||||
[ambient recall](../guides/ambient-recall.md) for the placement frontier
|
||||
and the per-verb latency table.
|
||||
|
||||
## Remove
|
||||
|
||||
|
||||
@@ -27,6 +27,19 @@ Desktop bridges local MCP servers into Cowork via its SDK layer.
|
||||
This means: if `gbrain serve` is running and configured in Claude Desktop,
|
||||
you don't need a separate server for Cowork.
|
||||
|
||||
## Verify
|
||||
|
||||
In a Cowork session, try:
|
||||
|
||||
```
|
||||
Call get_brain_identity, then search my brain for [any topic]
|
||||
```
|
||||
|
||||
You should get pages from your brain back. If `list_skills` returns nothing,
|
||||
skill publishing is off on the host — enable it with
|
||||
`gbrain config set mcp.publish_skills true` (see
|
||||
[CLAUDE_CODE.md](CLAUDE_CODE.md) for the full gotcha).
|
||||
|
||||
## Which to use?
|
||||
|
||||
- **Remote server:** works even when your laptop is closed, available to all org members
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Connect GBrain to Claude Desktop
|
||||
|
||||
This page covers connecting Claude Desktop to a **remote** brain. For a brain
|
||||
on the same machine as Claude Desktop, a local stdio entry in
|
||||
`claude_desktop_config.json` with `"command": "gbrain", "args": ["serve"]`
|
||||
works too — but only against a full local install, never a thin-client one.
|
||||
|
||||
**Important:** Claude Desktop does NOT connect to remote MCP servers via
|
||||
`claude_desktop_config.json`. That file only works for local stdio servers.
|
||||
Remote HTTP servers must be added through the GUI.
|
||||
|
||||
+22
-9
@@ -4,10 +4,14 @@
|
||||
> tutorial walks both paths (local-from-nothing and connect-to-an-existing-brain)
|
||||
> end to end, plus the brain-first protocol that makes it worth it. This page is
|
||||
> the connection reference.
|
||||
>
|
||||
> Want the **full agent** — identity, memory, schedules, and a private repo as its
|
||||
> durable body — not just a connection? That's `gbrain bootstrap`: see the paste
|
||||
> block in the README and [docs/guides/bootstrap.md](../guides/bootstrap.md).
|
||||
|
||||
Codex CLI (`@openai/codex`, v0.130+) supports remote streamable-HTTP MCP servers
|
||||
with a bearer token read from an environment variable. The token lives in your
|
||||
shell env, not in Codex's config file.
|
||||
Recent versions of the Codex CLI (`@openai/codex`) support remote
|
||||
streamable-HTTP MCP servers with a bearer token read from an environment
|
||||
variable. The token lives in your shell env, not in Codex's config file.
|
||||
|
||||
## Fastest path: `gbrain connect`
|
||||
|
||||
@@ -51,11 +55,11 @@ Call get_brain_identity, then search my brain for [topic].
|
||||
`get_brain_identity` confirms whose brain you're connected to; `list_skills` shows
|
||||
everything it can do.
|
||||
|
||||
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host (default
|
||||
> ON for `gbrain init` brains, OFF for brains upgraded from older releases). Enable
|
||||
> it on the host: `gbrain config set mcp.publish_skills true`. The core tools
|
||||
> (search, query, get_page, put_page, think, find_experts) work regardless.
|
||||
> `capture` is CLI-only, not an MCP tool — write over MCP with `put_page`.
|
||||
> **`list_skills` empty?** It's gated by `mcp.publish_skills` on the host — enable
|
||||
> it with `gbrain config set mcp.publish_skills true`. The core tools (search,
|
||||
> query, get_page, put_page, think, find_experts) work regardless; `capture` is
|
||||
> CLI-only, so write over MCP with `put_page`. Why brains differ on the default:
|
||||
> [tutorial A1](../tutorials/connect-coding-agent.md#a1-on-the-host-serve-over-http).
|
||||
|
||||
## Remove
|
||||
|
||||
@@ -68,4 +72,13 @@ codex mcp remove gbrain
|
||||
- The token is a long-lived, full-access secret. Keep `GBRAIN_REMOTE_TOKEN` out of
|
||||
version control and prefer a scoped token if your host supports one.
|
||||
- Local stdio also works if you run the brain on the same machine:
|
||||
`codex mcp add gbrain -- gbrain serve`.
|
||||
`codex mcp add gbrain -- gbrain serve --surface verbs` — the memory-verb
|
||||
protocol ([MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)); drop the flag
|
||||
for the full operation catalog.
|
||||
- **Ambient recall (Codex has no lifecycle hooks — use the pull path).** At the
|
||||
start of a topical thread and after a compaction, call
|
||||
`context_pack(entities, budget_tokens)` to warm the standing entities; on a
|
||||
periodic wake call `delta(session_id, budget_tokens)` for "what changed since
|
||||
my last wake" (deduped per session). Both are zero-LLM, sub-second, world-only
|
||||
by default, and on `--surface verbs`. See
|
||||
[ambient recall](../guides/ambient-recall.md) for the placement frontier.
|
||||
|
||||
+79
-48
@@ -1,30 +1,35 @@
|
||||
# Deploy GBrain Remote MCP Server
|
||||
|
||||
> **v0.26.0+:** `gbrain serve --http` ships full OAuth 2.1 (client credentials,
|
||||
> auth code + PKCE, refresh rotation, optional DCR), an embedded React admin
|
||||
> dashboard at `/admin`, scoped operations, and a live SSE activity feed.
|
||||
> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back
|
||||
> to the `access_tokens` table and grandfathers tokens to `read+write+admin`.
|
||||
> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only);
|
||||
> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md)
|
||||
> for env vars and tunable defaults.
|
||||
> `gbrain serve --http` ships full OAuth 2.1 (client credentials, auth code +
|
||||
> PKCE, refresh rotation, optional DCR), an embedded React admin dashboard at
|
||||
> `/admin`, scoped operations, and a live SSE activity feed. Legacy bearer
|
||||
> tokens still work — `verifyAccessToken` falls back to the `access_tokens`
|
||||
> table and grandfathers tokens to `read+write+admin`. Both the legacy fallback
|
||||
> and the OAuth tables work on PGLite and Postgres (both engine schemas carry
|
||||
> `access_tokens`). See [SECURITY.md](../../SECURITY.md) for env vars and
|
||||
> tunable defaults.
|
||||
|
||||
Access your brain from any device, any AI client. GBrain ships two transports:
|
||||
`gbrain serve` (stdio) for local agents, and `gbrain serve --http` (v0.26.0+)
|
||||
for remote clients over OAuth 2.1.
|
||||
`gbrain serve` (stdio) for local agents, and `gbrain serve --http` for remote
|
||||
clients over OAuth 2.1.
|
||||
|
||||
## Three Paths
|
||||
|
||||
### Local stdio (zero setup)
|
||||
|
||||
```bash
|
||||
gbrain serve
|
||||
gbrain serve # full operation catalog (default)
|
||||
gbrain serve --surface verbs # just the 7 memory verbs (quickstart surface)
|
||||
```
|
||||
|
||||
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
|
||||
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
|
||||
`--surface verbs` exposes exactly the seven-verb memory protocol (`recall`,
|
||||
`remember`, `entity`, `synthesize`, `forget`, `context_pack`, `delta` —
|
||||
[MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)) instead of the full catalog;
|
||||
omit the flag (default `full`) for every operation.
|
||||
|
||||
### Remote over OAuth 2.1 (recommended, v0.26.0+)
|
||||
### Remote over OAuth 2.1 (recommended)
|
||||
|
||||
```bash
|
||||
gbrain serve --http --port 3131
|
||||
@@ -45,28 +50,27 @@ Supported clients:
|
||||
- **Perplexity** — OAuth 2.1 client credentials grant.
|
||||
- **Claude Code, Cursor, Windsurf** — can use OAuth or legacy bearer.
|
||||
|
||||
See the [OAuth 2.1 setup](#oauth-21-setup-v100) section below.
|
||||
See the [OAuth 2.1 setup](#oauth-21-setup) section below.
|
||||
|
||||
### Remote with legacy bearer tokens (pre-v0.26 deployments) — Postgres only
|
||||
### Remote with legacy bearer tokens (simplest)
|
||||
|
||||
```
|
||||
Your AI client (Claude Desktop, Perplexity, etc.)
|
||||
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
|
||||
→ gbrain serve --http (built-in transport with bearer auth)
|
||||
→ Postgres (pooler connection or self-hosted)
|
||||
→ Postgres or PGLite
|
||||
```
|
||||
|
||||
This requires:
|
||||
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
|
||||
running `gbrain serve --http` against a PGLite install fails fast at startup)
|
||||
2. A machine running `gbrain serve --http`
|
||||
3. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
4. A bearer token created via `gbrain auth create <name>`
|
||||
1. A machine running `gbrain serve --http` (works on both PGLite and Postgres
|
||||
brains)
|
||||
2. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
3. A bearer token created via `gbrain auth create <name>`
|
||||
|
||||
Pre-v1.0 tokens are grandfathered as `read+write+admin` scopes when you upgrade
|
||||
to the HTTP server, so no migration is required.
|
||||
Existing bearer tokens are grandfathered as `read+write+admin` scopes on the
|
||||
OAuth-capable HTTP server, so no migration is required.
|
||||
|
||||
## OAuth 2.1 Setup (v0.26.0+)
|
||||
## OAuth 2.1 Setup
|
||||
|
||||
### 1. Start the HTTP server
|
||||
|
||||
@@ -92,8 +96,8 @@ Save this token. Open `http://localhost:3131/admin` and paste it to access the
|
||||
dashboard. The dashboard shows live activity, registered clients, request logs,
|
||||
and per-client config export.
|
||||
|
||||
> **v0.26.9+:** `mcp_request_log.params` and the live SSE activity feed default
|
||||
> to a redacted summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`.
|
||||
> `mcp_request_log.params` and the live SSE activity feed default to a redacted
|
||||
> summary `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`.
|
||||
> Declared param keys are kept (intersected against the operation's spec); unknown
|
||||
> keys are counted but never named, and byte sizes round up to 1KB so size-probe
|
||||
> attacks can't binary-search secret content. Operators on a personal laptop who
|
||||
@@ -124,9 +128,9 @@ gbrain auth register-client perplexity \
|
||||
--scopes "read write"
|
||||
```
|
||||
|
||||
**v0.34 — source-scoped clients.** Multi-source brains can scope a client's
|
||||
write authority to one source and its read scope to a curated set with the
|
||||
new `--source` and `--federated-read` flags:
|
||||
**Source-scoped clients.** Multi-source brains can scope a client's write
|
||||
authority to one source and its read scope to a curated set with the
|
||||
`--source` and `--federated-read` flags:
|
||||
|
||||
```bash
|
||||
gbrain auth register-client dept-x-agent \
|
||||
@@ -138,9 +142,12 @@ gbrain auth register-client dept-x-agent \
|
||||
|
||||
`--source` controls the write authority — `put_page` / `add_link` / etc only
|
||||
land in `dept-x`. `--federated-read` controls the read axis independently;
|
||||
queries return rows from any of the listed sources. Omit both flags for the
|
||||
v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to
|
||||
`source_id='default'` on `gbrain upgrade`.
|
||||
queries return rows from any of the listed sources. Omit both flags for an
|
||||
unscoped super-client. Clients registered before source scoping existed are
|
||||
backfilled to `source_id='default'` on `gbrain upgrade`. Within a source,
|
||||
slug-level write fencing is also available: `--bound-slug-prefixes p1/,p2/`
|
||||
rejects slug-mutating writes outside the listed prefixes (update later with
|
||||
`gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`).
|
||||
|
||||
Host-repo wrappers can register programmatically:
|
||||
|
||||
@@ -156,9 +163,25 @@ await oauthProvider.registerClientManual(
|
||||
For self-service client registration (Dynamic Client Registration, RFC 7591),
|
||||
start the server with `--enable-dcr`. DCR is off by default.
|
||||
|
||||
DCR requests may include an optional `token_ttl_seconds` field (integer,
|
||||
seconds) to request a per-client access-token lifetime. The server clamps the
|
||||
request into an admin-configured window — never rejects over it — persists the
|
||||
effective value as the client's TTL override, and echoes it back as
|
||||
`token_ttl_seconds` in the registration response. Subsequent `/token` responses
|
||||
for that client carry the matching `expires_in`. Clients that omit the field
|
||||
keep the server default (`--token-ttl`). The window defaults fail-closed: min
|
||||
300 seconds, max bounded by your `--token-ttl` — a self-registering client
|
||||
cannot request a longer-lived token than the server default unless you
|
||||
explicitly widen the window:
|
||||
|
||||
```bash
|
||||
gbrain config set oauth.dcr_ttl_min_seconds 600
|
||||
gbrain config set oauth.dcr_ttl_max_seconds 86400
|
||||
```
|
||||
|
||||
### 3. Expose the server
|
||||
|
||||
**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
|
||||
**Bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
|
||||
To accept connections from the ngrok tunnel (or any non-loopback source),
|
||||
restart with `--bind`:
|
||||
|
||||
@@ -182,10 +205,10 @@ router exposes the spec-compliant discovery endpoint at
|
||||
|
||||
### 4. Scopes and localOnly
|
||||
|
||||
Every operation is tagged `read | write | admin`. Four operations are
|
||||
`localOnly` and rejected over HTTP regardless of scope: `sync_brain`,
|
||||
`file_upload`, `file_list`, `file_url`. Remote agents cannot reach local
|
||||
filesystem surface area.
|
||||
Every operation is tagged `read | write | admin`. Operations flagged
|
||||
`localOnly: true` in `src/core/operations.ts` (10 today — `sync_brain` and
|
||||
the `file_*` ops among them) are rejected over HTTP regardless of scope.
|
||||
Remote agents cannot reach local filesystem surface area.
|
||||
|
||||
| Scope | What it allows |
|
||||
|-------|---------------|
|
||||
@@ -193,10 +216,13 @@ filesystem surface area.
|
||||
| `write` | `put_page`, `delete_page`, `add_link`, `add_timeline_entry` |
|
||||
| `admin` | Client management, token revocation, sweep, local-only ops |
|
||||
|
||||
Write ops can additionally be fenced per client with `--bound-slug-prefixes`
|
||||
(see [Register OAuth clients](#2-register-oauth-clients) above).
|
||||
|
||||
## Legacy Bearer Token Setup
|
||||
|
||||
Keep using pre-v0.26 bearer tokens if you aren't ready to migrate. They
|
||||
grandfather to `read+write+admin` scopes on the HTTP server.
|
||||
Bearer tokens are the simple path when you don't need per-client scoping.
|
||||
They grandfather to `read+write+admin` scopes on the HTTP server.
|
||||
|
||||
### 1. Set up the tunnel
|
||||
|
||||
@@ -243,15 +269,20 @@ gbrain auth test \
|
||||
|
||||
## Operations
|
||||
|
||||
All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
`file_upload` (no timeout limits with self-hosted server).
|
||||
GBrain's full operation catalog (100+ operations in `src/core/operations.ts`)
|
||||
is available remotely, with no timeout limits on a self-hosted server. The
|
||||
only exceptions are the operations flagged `localOnly: true` — `sync_brain`
|
||||
and the `file_*` ops among them — which are rejected over HTTP regardless of
|
||||
scope (see [Scopes and localOnly](#4-scopes-and-localonly) above).
|
||||
|
||||
**Security note on `file_upload`:** remote MCP callers are confined to the working
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
**Security note on file access:** the `file_*` operations being localOnly is
|
||||
the first line of defense; as defense-in-depth, `file_upload` also confines
|
||||
any caller that isn't verifiably the trusted local CLI to the working
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and
|
||||
absolute paths outside cwd are rejected, and page slugs and filenames are
|
||||
allowlist-validated (alphanumeric + hyphens; no control chars, RTL overrides,
|
||||
or backslashes). Local CLI callers (`gbrain files upload ...`) keep
|
||||
unrestricted filesystem access since the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
@@ -321,8 +352,8 @@ Remote servers must be added via Settings > Integrations, NOT
|
||||
| put_page | 100-500ms | Write + trigger search_vector update |
|
||||
| get_stats | < 100ms | Aggregate query |
|
||||
|
||||
**Note:** `gbrain serve --http` shipped in v0.26.0 with OAuth 2.1 + admin
|
||||
dashboard baked into the binary. The custom HTTP wrapper pattern (see
|
||||
**Note:** `gbrain serve --http` has OAuth 2.1 + the admin dashboard baked
|
||||
into the binary. The custom HTTP wrapper pattern (see
|
||||
[voice recipe](../../recipes/twilio-voice-brain.md)) is still supported for
|
||||
teams that need bespoke middleware, but for most remote deployments the
|
||||
built-in server is the recommended path.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user