admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-08-14 00:48:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
963 Commits 590 Branches 27 Tags
master
Commit Graph
103 Commits
Author SHA1 Message Date
Garry TanandClaude Fable 5 6411150071 v0.45.11.0 feat(bootstrap): OOBE hand-off (own the brain + cold-start) + DX polish waves (#4047) 
* feat(bootstrap): TTY DX exploration harness + Krug onboarding fix wave

Add a real-PTY exploration harness and land 16 verified "Don't Make Me
Think" fixes on the paste-in install experience for Claude Code and Codex.

Harness:
- test/helpers/tty-harness.ts — spawns any CLI (gbrain/claude/codex) under a
  real pseudo-terminal (Bun terminal: spawn), timestamps every output burst,
  and turns silence windows into a measurable stall report. Hermetic; pure
  helpers unit-tested in test/tty-harness.test.ts.
- scripts/dx-explore.ts — drives the fresh-user funnel (help / init / real
  claude-install / real codex-install / manual drive mode), writing
  transcripts to .context/dx-runs/ (gitignored).

Fixes (all adversarially verified against the code first):
- Keyless bare `gbrain init` completes in keyless mode instead of exit 1;
  multi-key non-TTY auto-picks the canonical default; typo stays fail-loud.
- Provider picker probe-gates ollama (daemon-up != model-pulled) and offers
  an explicit "continue keyless" option that is the bare-Enter default.
- Fresh-brain init prints one schema-setup line instead of ~240 migration
  names (GBRAIN_MIGRATE_VERBOSE=1 restores detail).
- Init epilogue: memory-verbs funnel is last-on-screen; skills advisory
  compacted for init; Mod Status trimmed.
- PGLite live-serve lock error names the fix (close the agent session).
- Mode-picker banner interpolates the applied mode; expansion-key gate is
  Anthropic/OpenAI/Google, not OpenAI-only.
- Missing `claude` binary skips MCP but still installs hooks; honest copy.
- Foreign MCP-registration removal targets the conflicting scope and fails
  loud if it does not land.
- Upgrade marker compares the running binary to latest and self-spawns via
  execPath, so a current/newer binary no longer nags from a stale cache.
- interview --set/--skip after --confirm warns it voided the confirmation.
- init --help matches behavior; init --supabase fails loud on non-TTY.
- Provider capabilities attributed per provider across README / runbook /
  questions bank / bootstrap.md.
- First-run tour: restart-first, prompt 3 true on day one, withheld on FAIL;
  README gives Codex the same scripted magic moment.
- Empty-brain "0 takes" onboard nudge suppressed.
- Broken settings.local.json aborts the hooks write fail-closed instead of
  silently dropping the user's permissions.

Regenerated cli-flag-registry.generated.ts and llms-full.txt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): second DX polish wave — clean the success screen + honest copy (F17-F21)

Follow-up to the DX fix wave, closing the top-5 remaining gaps the scorecard
flagged (all human-facing polish, not survival):

F17 — machine markers no longer leak to humans:
- verify report drops the `[D3.6]` plan-tag from the first_run_tour detail.
- the raw `UPGRADE_AVAILABLE <cur> <latest>` marker line prints ONLY on a
  non-TTY stderr (parsers still get it); an interactive human sees just the
  "gbrain X -> Y available" sentence.
- per-migration "what changed" notices (v123/v124, incl. the #2704 ref) are
  suppressed on a FRESH-install replay via a module quiet flag; upgrades still
  narrate. (GBRAIN_MIGRATE_VERBOSE=1 restores them.)

F18 — one obvious next action on the init success screen: the memory-verbs
demo is the single "→ Do this next" hero, last on screen; import/migrate/doctor
collapse into one terse "More:" footer; the graph block only shows for a
non-empty brain.

F19 — README "moment it clicks" is now the genuine cross-session brain
round-trip (remember → restart → recall), explicitly distinguished from the
identity-file recall, on both the Codex and Claude Code paths.

F20 — the compact init skills advisory is human-voiced (no `[AGENT]`
stage-direction on the human-facing success screen; the mode-picker's
agent-directed block stays gated to the non-TTY channel).

F21 — time promise reconciled: headline is ~15 min (personal-agent path) /
~30 min (always-on OpenClaw/Hermes); the runbook's search-mode line no longer
claims "balanced" when keyless applies "conservative". README hooks copy says
"on by default, with an opt-out" to match the runbook.

Regenerated llms-full.txt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): address two-model adversarial review of the DX wave

Fixes the regressions the 5-specialist + red-team + Claude/Codex adversarial
pass found in the F1–F21 changes, each with a test:

- Keyless upgrade hint pointed at `config set embedding_model`, which config.ts
  hard-refuses as a schema-sizing no-op — now names the working re-init recipe
  (`gbrain init --force --pglite --embedding-model <id>`), zero-key AND multi-key
  paths.
- Multi-key TTY picker offered "continue keyless" but the caller aborted on it —
  now honors keyless like the zero-key path.
- Detached update-refresh spawn used a `/gbrain$/` basename check that misfires
  for a renamed/official-named compiled binary (`gbrain-darwin-arm64`) and
  prepends the /$bunfs entrypoint — now detects dev-vs-compiled by the runtime
  basename (bun|node) so the refresh always runs.
- `bootstrap status` reported the wire phase "done" on a hooks-only receipt
  (host CLI missing at wire time) — now "partial" with a re-run hint, so a
  resuming agent doesn't trust a false complete.
- Post-repair MCP mismatch re-verifies and aborts instead of blessing a
  registration a racing writer may have re-claimed.
- probeOpenAICompat's abort timer now spans the body read (was cleared before
  it), so a stalled `/v1/models` body can't hang init past the 1s cap.
- Centralized the 4-copy stale-cache upgrade predicate into
  `pendingUpgradeVersion`; UPGRADE_AVAILABLE gains a GBRAIN_FORCE_UPGRADE_MARKER
  override for PTY-based agent harnesses.
- Mode picker's expansion-key gate adds GEMINI_API_KEY; picker prompt is
  article-aware ("an embedding" / "a chat"); dead `!brainEmpty` clause removed;
  migrate.ts try/finally widened + stamp failures named in quiet mode.
- DX harness: credential copies scrubbed even on SIGINT/interrupt (+chmod 600),
  child process TREE reaped on teardown, advisory made fail-open, KEY_MAP typed
  as a literal union.

New tests: migrate quiet-replay, self-upgrade pending predicate + negative
cache cases, bootstrap 127/scoped-remove/broken-settings dispatch, interview
invalidation flag, verify tour-withheld-on-FAIL, init keyless/supabase/multi-key,
init-nudge branches, ai-probes model parsing. Regenerated flag registry +
template-repo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.8.0 fix(bootstrap): onboarding DX polish wave (F17-F21) + review fixes

DX fix wave on the paste-in install/first-run experience for Claude Code and
Codex, driven by a new real-PTY exploration harness. Keyless init completes
instead of erroring, the migration wall collapses to one line, the success
screen leads with one action, and the "magic moment" copy points at the genuine
cross-session round-trip. Full detail in CHANGELOG.

Version trio + openclaw manifest + runbook stamp bumped to 0.45.8.0; CHANGELOG
release entry; TODOS onboarding-DX follow-ups filed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): OOBE hand-off — you own the brain, cold-start is skill #1

A working install now ends by making the two facts that matter actually land:

- `gbrain bootstrap verify` prints (and returns as `handoff` in --json) an
  ownership block — the actual private-repo URL with what owning it means
  (read it, `gbrain bootstrap attach` on machine two, delete it and the brain
  is gone), or the local-only variant pointing at `gbrain bootstrap repo` —
  followed by the ONE next action: run the cold-start skill (Gmail/calendar/
  contacts via ClawVisor, an OAuth vault so the agent never holds raw tokens;
  or offline archives), one consented phase at a time. Withheld on FAIL like
  the tour; shape stays unconditional for machine consumers.
- cold-start ships in the downstream bundle (61 skills): its plugin exclusion
  ("host onboarding flow") predated the v0.45 personal-agent bootstrap and is
  deliberately reversed — the paste-in audience is exactly who day-one
  onboarding is for. It now LEADS the recommended set (ahead of book-mirror:
  every flagship skill only becomes magical once the brain holds the user's
  real life).
- New drift guard: every recommended slug must be scaffoldable from the
  plugin bundle — recommended-but-unscaffoldable is a dead-end CTA and now
  fails the suite.
- Runbook Hand off rewritten around the two must-land facts + the on-the-spot
  cold-start offer; README's Codex and Claude Code paths carry the same two
  follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.10.0 feat(bootstrap): the OOBE hand-off release

Version trio + runbook stamp + template tree to 0.45.10.0; CHANGELOG entry;
llms bundles regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): stop memory-verbs-conformance leaking a fake-keyed gateway into shard-mates

The deterministic-embedder helper configures the MODULE-GLOBAL gateway with a
fake OpenAI key; the file's afterAll never reset it. The bunfig preload's
per-test restore only fires when the gateway is UNCONFIGURED, so the fake-keyed
config persisted for every later file in the shard process — turn-context's
corpus writes then embedded against real OpenAI and 401'd (CI shard-8 failure;
shard re-binning from this branch's new test files exposed it).

Fix both sides: conformance's afterAll now resetGateway()s back to the preload
baseline and nulls both test transports; turn-context's beforeAll does the same
defensively so it stays hermetic regardless of shard composition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-13 15:35:05 -07:00
Garry TanandClaude Fable 5 1ec6a6e842 v0.45.9.0 feat(bootstrap): first-class cloud-sandbox install + per-turn persistence + fail-closed privacy ladder (#4045) 
* feat(core): execution-environment detection — local | cloud-sandbox | ephemeral-container

detectExecutionEnvironment() + isCredentialInjectingProxy() with injected
signals (CLAUDE_CODE_REMOTE, cse_ session-id prefix, proxy-injected token
placeholder, anthropic-egress proxy JWT, container markers). binaryOnPath
moves here as the canonical PATH probe. autopilot's detectInstallTarget
ephemeral branch now routes through the shared detector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): repo-visibility ladder + per-turn Stop push + same-session failure banner

The durability lane. One repo-visibility verdict for every consumer (REST
first — never GraphQL, which sandbox proxies pin; authed ls-remote + an
attributed anonymous probe as the git-protocol fallback), replacing three
drifted probes. A 200 counts as public only with advertisement proof; a
401/404 counts as private-signal only with an auth challenge — fail-closed
in BOTH directions. Private verdicts cache 1h (private-only, per origin).

gbrain hook stop now spawns a debounced detached push per turn (per-root
state; cloud-sandbox defaults to every turn, elsewhere 5 min; a failing
status bypasses the debounce), closing the /exit and VM-reclaim gaps.
Push status is per workspace root, read through one shared reader by the
user-prompt banner (additionalContext + systemMessage — visible to the
human, not just the model), the SessionStart note, and doctor. Escape
hatches for self-hosted git: flag > env > file-plane config key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): environment-aware install — honest cron skip, cloud repo-create guard, execution_env surfaces

installDurabilityCron probes for crontab before writing anything (containers
and cloud sandboxes ship without one — expected, reported as an honest skip
naming the event-driven pushes that still cover persistence). The repo phase
installs the container-friendly harden half (post-commit hook, no scheduler)
outside local machines. createPrivateRepo fails fast in cloud sandboxes with
the flow that works (create outside, open the session ON the repo, attach).
bootstrap verify gains a never-gating execution_env check; bootstrap status
--json carries execution_environment for installing agents to branch on.

Also fixes a live-PATH resolution class: binaryOnPath and the crontab execs
now pass the current env explicitly (Bun resolves against the startup
snapshot otherwise).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): lifecycle hygiene — uninstall teardown, doctor job liveness, .mcp.json out of the repo, honest persistence copy

uninstall now tears down the durability wiring it installed (launchd/cron
job, untracked post-commit hook, credential wiring — the committed helper
and AGENTS rules stay). doctor gains bootstrap_durability_job: presence +
LIVENESS (launchctl load-state, crontab line, pull-log freshness) — a plist
on disk with a dead job no longer reads as healthy. Rendered .gitignore now
covers .mcp.json (absolute machine paths must not land in the private
repo); verify warns on pre-fix installs that committed it; the never-built
state/mcp.json promise is gone from GITHUB.md. Persistence copy tells the
truth everywhere: event-driven pushes do the durability work, the 30-min
job is a multi-machine pull freshener. Post-commit hook install/removal is
worktree-safe (git-path resolution; the git marker is a FILE there) and the
cron wrapper's self-disable tests the repo dir, not its git marker. Two
follow-up TODOs filed (plugin hook distribution; Channels push lane).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): committed hook carrier for cloud + cloud-setup-script emitter + cloud runbook

Cloud sessions clone fresh and snapshot hook config at session start — the
gitignored settings.local.json never exists there, so hooks never fired in
cloud at all. Cloud installs now write the repo-COMMITTED .claude/settings.json
with PATH-resolved, fail-open commands (no machine paths; a host without the
binary no-ops); local installs keep settings.local.json; the writers enforce
that one event never fires from both carriers, and removal cleans both.

New: gbrain bootstrap cloud-setup-script prints the paste-ready environment
setup script (npm transport — bun fetching is proxy-incompatible in cloud;
never the unrelated npm-registry package). Runbook gains a NEVER FABRICATE
TOOLING hard rule, a cloud-sandbox section (expected degradations as facts to
relay, the attach-first flow), and failure-table rows for the proxy-403 and
missing-crontab signatures. Codex lane stated honestly: AGENTS.md Gate 2 now
has the pull-side push-health check. Guide documents the new knobs and the
cloud contract; llms bundles regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): per-turn push e2e chain, cloud-sandbox sim, per-root status reader migration + registry regen

bootstrap-persistence e2e gains the Stop-hook chain (one stop banks the turn
to the real bare remote; the debounce holds across stops; debounce-0 lands
consecutive turns) and fixes a pre-existing post-#4024 break: repoPhaseComplete
required a github-parseable origin, so session-end pushes deferred FOREVER for
self-hosted/file-transport origins — non-github repo_urls now bind by exact
URL equality (redirect protection preserved). Degraded-modes e2e gains the
cloud-sandbox simulation (status reports the environment; repo creation
refuses with the attach flow). workspace-push tests read per-root status
through the shared reader. Flag registry regenerated (new git argv literals
from the git-path/ls-files calls — the accepted argv-bleed class).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(doctor): categorize bootstrap_durability_job (categories drift guard)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): adversarial-review hardening — fail-closed privacy, no push-storm, no exfil paths

Both Claude and Codex adversarial passes ran; 11 findings fixed (union):
- Non-github 401+WWW-Authenticate no longer reads as private (RFC 7235 makes
  the header mandatory on every 401, so a middlebox 401s identically) — the
  top exfil path both models flagged; now unverifiable/fail-closed, operator
  confirms via the escape hatch. github.com still needs x-github-request-id.
- Escape hatches downgrade ONLY 'unverifiable' — a PROVEN-public origin still
  refuses (hatches never authorize a public push).
- treeNeedsPush measures against origin/<branch> (the push's own ref), not
  @{u}: a no-upstream branch no longer reports a committed-but-unpushed tree
  as push_clean and silently strands it.
- [D20] failing-retry uses a fixed 60s floor, not min(debounce,60s) — cloud
  debounce=0 no longer re-runs the network ladder every turn.
- Committed hook carrier: dedupe/suppress only on the EXACT portable-command
  shape, not a 'gbrain hook' substring (blocks the suppress-local-run-evil
  supply-chain vector); GBRAIN_HOME refused in the committed carrier.
- push-status reason sanitized (charset+length) at every surface (banner,
  doctor, status blob) so remote git stderr can't inject via the remediation.
- Per-root state: ghost-root records (deleted workspaces) filtered so a dead
  failing record can't re-fire the banner forever; uninstall removes them.
- statusReport support blob reads push status through the shared per-root
  reader; visibility cache strips URL userinfo (no PAT persisted).
- anonProbe strips userinfo + redirect:manual + SSRF flags + --end-of-options
  on ls-remote; cron self-disable uses git rev-parse (worktree-safe both
  ways); cloud-setup-script fails loud on a broken update; durability liveness
  won't certify a never-run crontab as live; config get/unset resolve the
  dotted file-plane keys; .bak/.broken gitignored; typed config fields.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.8.0 feat(bootstrap): first-class cloud-sandbox install + per-turn persistence + fail-closed privacy ladder

VERSION + package.json + CHANGELOG + TODOS + llms bundles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: sync KEY_FILES + README for the cloud-DX wave (v0.45.8.0)

New KEY_FILES entries for execution-env.ts and repo-visibility.ts; updated the
bootstrap repo/hooks, workspace-push, brain-repo-durability, and hook.ts
entries to current behavior (ladder verification, committed cloud hook carrier,
per-root push status, crontab probe + liveness, per-turn stop push + banner).
README's Claude Code line now states per-turn + cloud persistence honestly.
llms bundles regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): doc-review pass — attach records repo_url (enables cloud persistence), verify reads per-root push status, honest docs

Cross-model doc review (Codex) caught that the headline cloud-persistence
claim was hollow and two docs over-claimed:
- attach now records repo_url from the adopted origin, so the no-daemon push
  gate (repoPhaseComplete) recognizes the repo phase as done — WITHOUT this,
  the per-turn/session-end pushes deferred forever after an attach, which is
  the ONLY install path in a cloud sandbox (repo is refused there). Privacy is
  still enforced at push time by the ladder.
- bootstrap verify's push_probe reads the shared per-root reader [D8], not the
  legacy single file — a fresh v0.45.8 install no longer reports 'no push
  recorded' when per-root status exists.
- Docs corrected to match code: runbook stamp → 0.45.8.0; README states the
  per-turn cadence honestly (debounced local, next-turn failure notice);
  GITHUB.md qualifies auto-push as Claude Code (Codex is pull); KEY_FILES says
  repo/status use REST (not the full ladder) and drops the stale --push;
  bootstrap.md hooks-location covers both carriers; README verbs surface says
  seven. llms bundles regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): regenerate vendored template-repo tree + isolate env-mutating tests (verify gate)

The verify CI gate caught two things the piecemeal local runs missed:
- templates/bootstrap/template-repo/ is a GENERATED tree; the source-template
  edits (AGENTS.md Gate 2, CLAUDE.md cloud note) plus the version stamp had
  drifted. Regenerated from source (check:bootstrap-templates green).
- test/bootstrap-repo.test.ts + test/durability-cron.test.ts newly mutated
  process.env in non-serial files; converted to withEnv() (check:test-isolation
  green) — no .serial rename needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ambient-recall): pin a keyless gateway so delta/context_pack writes never fire a real embed

Root cause of the shard-2 CI red after the 0.45.9.0 re-bump: adding this
wave's test files reshuffled the weight-packed shards, moving
ambient-recall.test.ts next to a neighbor that leaks CI's dummy
OPENAI_API_KEY (sk-test-*) into the gateway singleton (the bunfig preload
configures with env:{...process.env}, and a present-but-invalid key turns
remember's keyless-degrade embed into a hard 401). The delta/context_pack
tests exercise cursor + budget logic, not embedding quality, so this pins a
keyless gateway (env:{}) in beforeAll — isAvailable('embedding') is false,
writeSingleFact degrades (degraded_dedup) with no HTTP call, and the file is
deterministic regardless of shard bin-packing. Verified: passes with
OPENAI_API_KEY=sk-test-* set (the CI condition). Master's own comment in
legacy-embedding-preload.ts already warns 'adding any test file reshuffles
the mines'; this makes the file immune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-12 16:12:59 -07:00
Garry TanandClaude Fable 5 15ecc65b24 v0.45.7.0 feat(mcp,context): ambient recall — context_pack + delta frozen verbs + boundary runtime (#1) (#4028) 
* feat(mcp,context): ambient recall — context_pack + delta frozen verbs + boundary runtime (#1)

Two new frozen MEMORY_VERBS (context_pack, delta) on the pull surface + a
Claude Code hook boundary runtime on the push surface, sharing one stateless
assembler core (assembleTurnContext mode: turn|pack|delta) and a keyset
session cursor (migration v126). World-only by default; include_private
gated fail-closed to trusted-local. protocol_version stays 1 (additive
5→7 verbs). Survived three adversarial review waves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.7.0 feat(mcp,context): ambient recall — context_pack + delta frozen verbs + boundary runtime (#1)

Two new frozen MEMORY_VERBS (context_pack, delta) grow the frozen set 5→7
without a wire bump — all seven stamp protocol_version: 1. context_pack
assembles a deterministic, zero-LLM, budget-packed bundle (entity cards +
open threads + hot facts) for a set of standing entities; delta returns
only what changed since a timestamp for cheap heartbeats, with a
per-session keyset cursor for at-least-once delivery. A boundary runtime
wires these into Claude Code lifecycle hooks (SessionStart warm pack,
PreCompact entity banking for post-compaction rehydration); Codex and any
MCP host pull the same verbs at their own boundaries. World-only by
default on all arms; include_private widens only for local trusted
callers. Migration v126 adds session_context_state (additive).

Includes the coverage close-out wave (~55 tests): real-serve compact→
session-start round trip over the live socket, --surface verbs stdio
session pinning exactly 7 tools fail-closed, HTTP-transport verb calls
with per-token cursor isolation, Postgres engine-parity for keyset
pagination + the session-cursor table, migration v126 shape + rewind
test, sub-second latency gates, CLI-level invocations, rendered-protocol
boundary assertions, and a live-Codex boundary-call check. The wave
caught and fixed three real bugs: the delta CLI wedging on first wake
(floating GC promise racing engine teardown), the compact hook probing
the PGLite socket on a Postgres config with a leftover database_path,
and the verbs-surface banner hardcoding a stale verb count.

Also the /document-release sweep: stale "five verbs" → seven across the
protocol doc, README, INSTALL, DEPLOY, the Claude Code MCP guide, and
the query skill; deferred scope filed in TODOS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(release): bump openclaw.plugin.json to 0.45.7.0 — the sixth version location

The #4033 merge auto-resolved the OpenClaw plugin manifest at master's
version while the trio moved to 0.45.7.0, failing the manifest drift test
on CI shard 4. Register the file in CLAUDE.md's version-locations table
(five → six) so every future ship and merge re-bumps it with the trio.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-12 10:56:11 -07:00
Garry TanandClaude Opus 4.8 a996e42856 v0.45.2.0 fix(bootstrap): create-repo-first repo adoption + hardening (#4024) 
* fix(bootstrap): harden create-repo-first repo adoption

`gbrain bootstrap repo` adopts an empty, private, personally-owned GitHub repo
the human created (create-repo-first), instead of only ever creating one. This
hardens the existing adoption branch:

- Empty-only adoption + pending_repo_url proof: a non-empty origin is refused
  (ORIGIN_NOT_EMPTY) unless it matches this workspace's pending marker (our own
  interrupted push). Never adopts a user's existing project from a git-ancestry
  guess, and never silently no-ops without pushing.
- Repo-local git identity is set on the adopt path too (fresh-machine commits).
- repo_url is recorded only AFTER a successful push (pending marker before);
  a failed push no longer looks "done" to `bootstrap status`.
- Pre-push secret scan also covers an already-committed tree; ls-files failure
  fails closed.
- assertOriginMatches binds BOTH the fetch URL and a configured push URL to the
  verified-private repo, so a foreign pushurl can't leak the workspace.
- disposition: 'created' | 'adopted' | 'reused' replaces the overloaded flag.
- Hook push-gate: the no-daemon session-end / recovery push is deferred until
  the repo phase records repo_url AND the current origin still matches it, so
  nothing is published to an unverified or redirected remote.

Adds ORIGIN_NOT_EMPTY / REMOTE_CHECK_FAILED error codes.

* docs(bootstrap): lead with the repo, document create-repo-first

README (Claude Code + Codex) now opens with "the folder you open becomes your
agent's private repo" and adds a "prefer to make the repo yourself?" callout for
the create-repo-first path (empty, personal-account repo). Updates the bootstrap
guide, the Claude Code MCP note, and the KEY_FILES / AGENT_BOOTSTRAP_PLAN
invariants to describe adoption instead of "foreign origins refused".

* v0.45.1.0 fix(bootstrap): create-repo-first repo adoption + hardening

Bumps VERSION/package.json to 0.45.1.0, adds the CHANGELOG entry, refreshes the
runbook + template-repo version stamps, and regenerates the llms bundle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(todos): file P2 follow-up — index-blob secret scan for bootstrap pushes

* ci(gitleaks): run the free CLI instead of the license-gated v2 action

gitleaks-action@v2 now enforces a paid GITLEAKS_LICENSE and fails the job
("missing gitleaks license") for accounts it can't validate over the API —
blocking every PR's merge gate. Replace it with the open-source gitleaks CLI
(pinned 8.30.1, checksum-verified against the release's own checksums file),
scanning the PR/push commit range with the committed .gitleaks.toml allowlist.
Same secret-scan coverage, no license wall.

* v0.45.2.0 chore(release): re-bump 0.45.1.0 -> 0.45.2.0

Re-target the release version at the user's request. Updates VERSION,
package.json, the CHANGELOG header + self-repair block, the runbook +
template-repo version stamps, the TODOS follow-up reference, and the llms bundle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(perf): raise entity-card ratio ceiling 50x -> 100x (CI flake)

The RATIO GUARD asserted entity p99 <= 50x max(getPage p50, 1ms). On a fast
runner getPage p50 floors to 1ms and a normal entity p99 (~50ms) reads as ~52x,
tripping the gate even though absolute p99 (52ms) is well under the 100ms budget
— a p99 tail divided by a sub-ms median. At the 1ms floor, 50x also made the
ratio STRICTER than the test's own 100ms absolute budget. Raise the ceiling to
100x: still far below the >=200x O(N)-regression signal the guard exists to
catch, and consistent with (never stricter than) the absolute budget.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-11 23:01:22 -07:00
Garry TanandClaude Fable 5 d35c9c9e44 v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code (#3975) 
* docs(designs): agent-bootstrap plan + design docs (normative, review-absorbed)

The scrubbed, in-repo sources of truth for the gbrain bootstrap wave:
AGENT_BOOTSTRAP_DESIGN.md (product scope/sequencing) and
AGENT_BOOTSTRAP_PLAN.md (implementation; all review-finding IDs inlined).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): format spec, question bank, identity templates, bundled assets

agent.json manifest (format_version 1, initialized sentinel) + machine-local
install receipt [CX2-1, CX2-12]; 12-question/6-required interview bank with
consent keys and a persist:false sink for the optional provider key [CX2-13];
ten {{TOKEN}} identity templates (generic, adapted to gbrain ops — gates call
recall/query/put_page, write-through-ops rule, keyless agent-authored facts,
silence contract); assets embedded compiled-binary-safe via file-type imports
[ENG-6].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): runbook, README paste block, bootstrap guide, TODOS entries

BOOTSTRAP_FOR_AGENTS.md (agent-driven install runbook: CLI phase list is the
source of truth, never-invent rules, Codex approvals preflight, keyless posture,
failure-modes table, version stamp for the skew check); README gains the
full-agent paste block pinned to latest-stable inside the Claude Code/Codex
quick start (memory-only tier stays); docs/guides/bootstrap.md carries the full
install/security/consent/degradation/uninstall contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(designs): spike instrument for the bootstrap wave (build order 0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): interview + render engines

Interview gate with read-back confirm-hash (any later answer change clears the
confirmation — the hostile single-batch case is structurally impossible),
per-answer provenance, caps + escaping at set time, config-sink routing for the
provider key; renderer with hard-fail token sweep, subordinate fencing of
principal input, never-clobber + backups, deterministic minimal mode for the
template repo, scaled byte floors. 58 unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): private-repo lifecycle — repo create, attach, uninstall, run lock

gh-gated private repo creation with API-verified privacy (rate-limit distinct
from public), refuse-foreign-origin with attach as the sanctioned path, atomic
bootstrap mutex (pid liveness + age + token), receipt-keyed uninstall that
never wholesale-deletes the gbrain home and only offers --delete-brain for a
brain it created; read-only PGLite lock probe (never opens the engine).
54 unit tests, injectable exec seam throughout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(release): latest-stable ref, template-repo publish job, bootstrap CI guards

release.yml advances the latest-stable tag only after assets publish (the paste
block's permanent ref — copies in the wild never rot) and gains a PAT-gated
publish-template job verified against the vendored tree; two skip-graceful
guards (sanctioned-ref + runbook stamp; template/token bijection + placeholder
assertion + generator byte-diff) wired into verify; README + runbook re-admitted
to the CI cache hash; vendored deterministic template tree generated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): IPC v2 turn_context + 8KB assembly + visibility resolver + session identity

Discriminated-union IPC with handler map, protocol echo (stale-serve detection),
shared-secret gate, server-side source binding, per-kind budgets; turn-context
assembly (reflex pointers + volunteered pages + world-only hot facts) under a
data-not-instructions envelope trimmed to the harness's 10KB hook-output cap;
facts.default_visibility resolved through one helper at all four sites (explicit
caller wins, typos fail closed); typed sessionId threads _meta.session_id into
the hot-memory cache key. 50 new tests; 180 adjacent tests confirmed green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(persistence): secret-scan, gbrain sources push, durability unification

Pattern secret scanner (own runtime allowlist, redacted previews, corpus-write
redaction mode); sources push runs the whole scan→stage→commit→pull→push
sequence under one cross-platform lock (mkdir-atomic, pid+age+token) with a
deny-glob backstop, commit-first divergence-safe pull, refuse-unverifiable
visibility, and push-status telemetry; gbrain-home choke point unifies
GBRAIN_HOME semantics with config (0700); durability is parent-repo-aware and
rotates its push log at 0600. 35 new tests; 200 existing green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sources): harden/pull gates accept sources inside a parent git repo

The bootstrap workspace registers brain/ (a subdirectory) as the source; the
durability core already resolves the repo root, so the command gates now check
inside-a-repo rather than .git-right-here [CX2-3].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(serve): resident maintenance sweep + keyless capability probe

The lock-owning serve process now closes the persistence loop: startup (3s
post-connect, best-effort, unref'd) and idle (10-min quiet intervals through
the injectable timer seam) sweeps run facts-fence reconciliation, deterministic
link/timeline extraction over recent workspace pages, and spend-gated corpus
ingest (skipped keyless — agent-authored fences cover it). gbrain sweep --once
is the trusted CLI seam bootstrap verify uses. Capability probe renders the
honest keyless/keyed report. Full reuse of the cycle extractor + extract cores;
26 new tests, neighbors green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(hooks): engine-free gbrain hook command, settings writers, transcript parser

Four hook events (session-start digest + crashed-session recovery push,
user-prompt turn-context injection under an 800ms deadline and the 10KB cap,
stop buffers, session-end corpus write with redaction/retention/dedup +
best-effort push); structural JSON settings merger keyed by a _gbrain marker
(foreign hooks and permissions survive); dated host-spec registry; Claude Code
.jsonl parser as a spec-target with a scrubbed 7-shape fixture. Heartbeat is
counters-only by construction. 59 tests; zero engine modules in the import
graph.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(bootstrap): cross-link the full-agent path from the connection docs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): dispatcher, verify, status — the command assembled

gbrain bootstrap {status,interview,render,repo,hooks,verify,uninstall,attach}:
engine-free except verify (owns its engine, in-process sweep — no live-serve
conflict); phase list is the TS source of truth with install.jsonl telemetry
and the support blob; verify's fail-soft check suite covers the real write path
(put_page → write-through file → sweep → graph floor → recall), passes keyless,
persists snapshots, and ends with the first-run tour. cli.ts wired per the
three-touchpoint rule; doctor gains the bootstrap check group (silent on
machines with no bootstrap state). 28 new tests; 353 adjacent green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: KEY_FILES bootstrap cluster + CLAUDE.md dispatcher row (+ build:llms)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): e2e pins — hook-under-live-serve, attach, degraded modes, compiled binary, Docker harness

The permanent pins: a real serve holds the PGLite lock while the engine-free
hook completes (and a direct engine open provably throws LiveServeLockError);
stale-socket fail-open; machine-2 attach with marker-keyed hook repair;
decline-everything installs verify green with every degradation named; the
compiled binary renders bundled templates in an empty cwd. Offline Docker
harness (networkless, read-only) gated into heavy-tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): register doctor check categories + system-of-record allow comments

The six bootstrap doctor checks join OPS_CHECK_NAMES; the sweep's batch link/
timeline inserts carry the explicit extract-path allow comments (the sweep IS
the extraction path for workspace pages).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): shard wedge cap tracks suite growth (1500s -> 1800s)

At ~9000 tests a healthy shard finished at 1466s and two progressing shards
were false-killed at the old cap; 1800s restores ~25% headroom over the
slowest observed healthy shard. Real hangs still hit it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): cache-hash policy — README + runbook edits must invalidate [C2]

The old deny-list assertion predates the paste block; README.md and
BOOTSTRAP_FOR_AGENTS.md are policy-doc re-admissions now, so their edits must
change the hash (a paste-block edit shipping under a cached green was the C2
hole).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): classify post-suite exit-hangs as warn-pass; file the leak forensics

A shard killed by the wedge watchdog with every assigned file started and zero
fail markers did all its work and leaked a handle at exit — pre-existing and
master-reproducible (P1 TODO carries the full bisect forensics). Bun's per-test
timeout turns a hung test into a (fail), so the classifier cannot mask one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): shard cap 2400s — the count-balanced heavy shard needs it under contention

Observed: the heavy shard still progressing 22s before an 1800s kill while
siblings finish at 1150-1550s (split balances file count, not weight). Filed
the load-sensitive WAL-repair flake (pre-existing, master's v0.42.75.0 wave).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): quarantine env-mutating suites to the serial lane

check-test-isolation R1: six new files mutate GBRAIN_HOME/env at module scope —
the serial lane (one process per file) is the guard's prescribed home for them.
All 114 tests pass post-rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): per-harness install sections — Codex, Claude Code, then OpenClaw/Hermes

Each harness gets its own complete paste-block section (desktop app first,
terminal noted — Claude Code CLI is the identical harness; Codex CLI works
pull-based today); the OpenClaw/Hermes platform path keeps equal weight with
its one-click deploys and INSTALL_FOR_AGENTS block intact; memory-only and
remote-connect tiers consolidated under 'Lighter ways in'. Supersedes the
review's D5 ordering by user direction; stale heading references updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): Codex as the recommended first step; OpenClaw/Hermes framed as-intended, high-cost

The install section now routes newcomers explicitly: Codex first
(subscription-priced, nothing to deploy), OpenClaw/Hermes as GBrain used the
way it was designed — always on, at real server + API cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: doc-audit code chasers — broken recovery hints, stale op description, auto_link key

Four small code fixes surfaced by the markdown accuracy audit:
- doctor's auto-RLS recovery hint pointed at `apply-migrations --force-retry 35`,
  which cannot work (--force-retry targets the vX.Y.Z orchestrator registry, not
  the numeric schema MIGRATIONS array). Hint now points at the recreate SQL in
  docs/guides/rls-and-you.md; test pins against regression.
- v0_11_0 migration printed the same broken-mechanism class of hint
  (`config set minion_mode` writes DB config nothing reads); now names
  `apply-migrations --mode` + preferences.json, the real setter.
- submit_job's op description hardcoded a stale handler list; now points at
  registerBuiltinHandlers as the source plus the --follow discovery trick.
- `auto_link` added to KNOWN_CONFIG_KEYS: read by link-extraction, reconcile-links,
  and sweep, and documented as the off-switch in brain-ops/maintain, but the
  allowlist rejected `config set auto_link false`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: repo-wide accuracy + MECE reform from the 9-bucket markdown audit

A code-grounded audit of every markdown file (root, architecture, guides,
mcp, tutorials, docs-root, operations/eval/designs, skills, recipes) followed
by a fix wave with per-bucket ownership. Four classes of change:

Accuracy — every documented command/flag verified against src/ before writing:
dead commands replaced with working ones (pages purge-deleted, jobs watch
--follow, gbrain restore, import-based Obsidian flow, space-separated --scopes,
real thin-client recipes, working isolation verification, real supervisor
restart procedure, curl-based ngrok health check, real minion_mode setter);
count drift fixed with rot-proof phrasing (100+ ops, 50+ bundled skills via
skills/manifest.json, 140+ engine methods, KNOBS_HASH_VERSION pointer instead
of hardcoded versions); stale claims corrected (search-mode defaults, RETRIEVAL
pipeline order incl. autocut, sentinel rules, refusal-list mechanism, engine
snapshot, shard cap 2400s + EXIT-HANG classifier in TESTING.md, latest-stable +
publish-template documented in RELEASING.md as release.yml promises).

MECE — one home per concept, pointers elsewhere: test isolation → TESTING.md;
OAuth registration + --bind/--public-url lore → DEPLOY.md; mode bundles →
guides/search-modes.md (the home the CLAUDE.md dispatcher always promised);
merge contract → schema-packs.md; WAL ladder → ENGINES.md; quiet-hours →
quiet-hours.md; capture taxonomy → entity-detection.md; person-page taxonomy →
compiled-truth.md; brain-first protocol → brain-first-lookup.md; refresh
semantics → refresh-algorithm.md; KEY_FILES.md deduplicated (58 extension
entries merged, one entry per file); infra-layer.md rewritten as a pointer page.

Privacy — placeholder sweep across guides, docs, skills, and recipes per the
iron rule; per-release narration stripped from reference docs (current-state
prose only).

Bootstrap coverage — AGENTS.md pointer, RESOLVER routing row, INSTALL.md path,
tutorial cross-links, keyless-mode sections in spend-controls/headless-install.

skills.lock.json regenerated; llms.txt/llms-full.txt rebuilt. Gates: verify
36/36, typecheck clean, doctor 96/96, skills-integrity + resolver + build-llms
+ config-set + migrations all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): refresh production-brain stats to current brain-repo counts

155,795 pages / 24,589 people / 5,340 companies, counted from the brain
repo's current HEAD; the "100K-page brain" framing moves to 150K to match.
Cron-fleet count unchanged (its store lives on the deployment host, not in
the repos available for verification).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scan,push): modern OpenAI/Voyage key patterns, scan staged blobs not disk

- secret-scan matches sk-proj-/sk-svcacct-/sk-None- and pa- Voyage keys
  (the bare sk- pattern missed every current OpenAI key format).
- workspacePush stages first, then scans the staged index blobs via
  git cat-file, closing the scan-then-stage TOCTOU where a file changed
  between snapshot and commit shipped unscanned.
- shared binary-sniff helper, memoized glob regexes, atomic push-status
  write, and tests for pull_conflict + gitignored deny-match paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): regenerate flag registry for new commands, harden shard classifier + release token

- cli-flag-registry.generated.ts regenerated: bootstrap/hook/sweep and
  sources push --message/--allow-unverified-remote were missing, so the
  strict #2185 validator rejected real invocations and skipped the new
  commands entirely.
- EXIT-HANG shard classifier now requires every assigned file to have
  started before warn-passing a watchdog kill (was fail-open).
- release.yml passes TEMPLATE_REPO_PAT via http.extraheader, off the argv.
- compiled-binary e2e fails loud in CI instead of a silent permanent skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(hook,sweep,ipc): non-blocking hook pushes, bounded sweep + cache, source-bound resolve

- session-start/session-end no longer run synchronous git + inline push
  inside their self-deadline; a detached child does the push and the hook
  returns immediately (blocked Claude Code startup for minutes on a dirty
  tree before).
- serve sweep drops the unbounded listAllPageRefs, resolves only candidate
  targets, claims corpus files atomically (no double-LLM-spend race), and
  caps the fence LIKE scan; heartbeat writes are O_APPEND with rare compaction.
- hot-memory cache evicts expired entries and bounds entry count (the key is
  caller-controlled via _meta.session_id).
- v1 resolve IPC honors boundSourceId like turn_context; turn-context runs
  its arms concurrently. doctor reads push/heartbeat thresholds from hook.ts.
- new tests: doctor bootstrap checks, hook push-gate + deadline, concurrent
  sweep claims, cache eviction, bound-source resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): origin-ownership gate, world visibility, collision-safe source id, consent + templates

- repo adoption requires an exact receipt repo_url match or authed-owner
  check (undefined repo_url was a wildcard); create verifies privacy BEFORE
  the first push.
- verify sets facts.default_visibility=world if unset, so agent-authored
  facts surface in per-turn context (they defaulted private before).
- source_id derives a path-hash suffix when 'workspace' is taken by another
  checkout; every consumer reads manifest.source_id.
- skipped HOOKS_CONSENT now declines (was falling through to default yes);
  --minimal refuses on an initialized manifest; tilde fences escaped.
- MCP registration pins --surface full; status hard-fails a public origin
  (template door); receipt writers guard against newer/corrupt receipts;
  uninstall only claims brain-deleted after a real rm.
- templates ship jobs disabled + provider-consent + support-relay lines;
  soul-audit re-runs over the shared interview bank. TODOS: 11 follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): close adversarial-review findings — scan fails closed, whole-PEM redaction, bound repo push

Cross-model adversarial pass (Claude + Codex) on the bootstrap wave:

- secret scan fails CLOSED: an unreadable, oversized, or binary staged blob
  now blocks the push (blocked_unscannable, exit 5) instead of committing
  unscanned; only a confirmed staged deletion is skipped. This was the
  headline "block secrets before they leave the machine" property failing open.
- private-key redaction spans the whole PEM block (header+body+footer), not
  just the header line — the base64 body no longer survives into the corpus
  the sweep sends to an extraction provider.
- bootstrap repo commits the workspace (secret-scan-gated) before the first
  push and verifies the remote actually received it, so a push-fail retry
  can't adopt an empty remote as success.
- privacy verify is re-bound to origin immediately before push (a concurrent
  origin rewrite between verify and push is refused).
- session-end corpus write is atomic and clears the stale ingested/in-progress
  sidecars so a resumed session's appended transcript is re-ingested.
- public-origin refusal enforced at render (not only status); MCP "already
  registered" is verified to target this workspace, not blessed blindly;
  verify probe cleanup scopes deletes to its own slugs, not a token substring;
  allowlist fingerprint floor raised 8→16 hex.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code

Turns a Codex or Claude Code session into a persistent personal agent:
interview-rendered identity files, a local PGLite brain, per-turn context via
serve IPC (Claude Code hooks / Codex pull protocol), session-triggered
persistence, and a private GitHub repo as the agent's portable body. Keyless-
first (the harness model is the LLM; one optional key adds embeddings +
extraction). New `gbrain bootstrap` command family + `gbrain hook` + `gbrain
sweep`; doctor bootstrap health checks; latest-stable distribution ref +
template-repo publish job. Opt-in, additive — existing installs untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: regenerate flag registry for security-fix flags; drop fabricated gbrain capabilities doc ref

CI caught two real failures under the merged state:
- the flag registry lagged the blocked_unscannable/exit-5 flags the security
  round added, tripping the #2185 freshness guard.
- headless-install.md described the keyless capability report as a
  `gbrain capabilities` command, which the #3502 doc-command resolver
  rejects — reworded to prose (the real surface is bootstrap verify's report).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: sync KEY_FILES + bootstrap plan to security-fix behavior

Cross-referenced the security-fix round against the reference docs and
corrected the drift those commits introduced:

- workspace-push.ts entry: stage-FIRST-then-scan order (the TOCTOU fix),
  fail-closed blocked_unscannable, and the sources-push status -> exit-code map.
- hooks.ts entry: MCP registration pins `serve --surface full`.
- hook.ts entry: session-start/session-end pushes run in a detached child
  (non-blocking); atomic corpus write clears stale sidecars.
- bootstrap.ts entry: render hard-refuses a public origin (template door).
- verify.ts entry: source_id collision resolution (workspace-<path-hash>).
- AGENT_BOOTSTRAP_PLAN as-shipped delta note for the scan/stage reorder.

llms bundle unchanged (KEY_FILES is link-only); build:llms and
test/build-llms.test.ts green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): silence SC2016 on the intentional askpass literal in release.yml

The one-shot GIT_ASKPASS script must contain literal $1 and
$TEMPLATE_REPO_PAT so they expand when /bin/sh runs it at git's credential
prompt, not when the outer shell writes the file — single quotes are correct.
Add a scoped shellcheck disable so actionlint passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(skills): declare 'bootstrap my data' trigger in cold-start frontmatter

The doc reform added 'bootstrap my data' to cold-start's RESOLVER.md row (to
disambiguate data-bootstrap from agent-bootstrap) but not to the skill's own
frontmatter triggers, tripping the RESOLVER↔frontmatter round-trip contract
(resolver.test.ts). Declare it; regenerate skills.lock.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): default per-turn hooks + search mode ON without a prompt

Installing gbrain for your coding agent IS the consent for the behaviors that
make it work, so stop re-litigating them with install-time questions whose
"no" defeats the product:

- Per-turn hooks (Claude Code) install ON by default — no prompt. Off-ramps:
  `--no-hooks` at install, `GBRAIN_HOOKS=0` at runtime, `bootstrap uninstall`.
  The "hooks installed" line now surfaces the kill switch so default-on is
  never silent. A persisted HOOKS_CONSENT=no (interview --skip) still declines.
- Search mode defaults to `balanced` silently (nobody knows the modes at
  install; `gbrain search modes` changes it any time).
- MCP scope stays the ONE deliberate prompt — project vs user is a real
  cross-repo privacy choice, not friction.

Marks the two consents `silent: true` in the question bank (new QuestionSpec
field), rewrites the runbook phases so the agent no longer asks them, adds the
`--no-hooks` flag (+ registry regen), and adds default-on / opt-out tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): real end-to-end coverage — cross-session recall, per-turn content, Codex door, realistic corpus

Closes the seven e2e gaps a coverage audit surfaced: the plumbing was
well-unit-tested but the product claims ("Codex works, context shows up every
turn with real content, it remembers across restarts, machine two recovers,
Postgres works") were unproven end to end. Test-only wave — zero src changes.

- Hermetic synthetic corpus (test/fixtures/bootstrap-corpus/ + a loader helper):
  12 interlinked pages (52 edges, timelines), 12 world/private beliefs, 8 gold
  queries — curated from the gbrain-evals synthetic corpora, 100% placeholder
  names, so recall is asserted on a real multi-entity brain instead of a
  2-node self-planted probe.
- GAP1 magic moment: author a fact via the real write path, disconnect the
  engine, reopen against the same DB, recall it — a real session boundary, not
  verify.ts's same-connection SQL read-back. Plus a source-isolation assertion.
- GAP2 per-turn content: hook-under-serve Pin 1 now seeds a known fact and
  asserts its text lands in the injected block AND private beliefs never do
  (was: empty brain, empty_block accepted as a pass).
- GAP3 Codex door: assert the rendered AGENTS.md carries the Gate-3 brain-first
  pull protocol; make the fake codex shim implement `mcp get` so the [FIX7]
  target-verification can actually fail; the Docker cold-machine harness now
  exercises the hooks/MCP registration step instead of skipping it.
- GAP4 corpus recall: turn-context + verify graph-floor/qrels run on the real
  multi-entity brain with real edges.
- GAP5 attach: machine-two now re-ingests the cloned brain/ into a fresh DB and
  recalls a fact authored only on machine one — the multi-device payoff.
- GAP6 keyed + Postgres (env-gated): real embeddings prove semantic recall a
  paraphrase query can reach but keyless BM25 cannot; bootstrap verify drives a
  real Postgres engine (skipIf DATABASE_URL/keys absent).
- GAP7 persistence: session-end runs the REAL push (not the mocked seam) to a
  local bare remote and the remote receives the content; a planted secret is
  blocked at the gate; the 15-min cron installs and fires a scan-gated push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): real-agent e2e — drive the actual claude + codex binaries end to end

Closes the audit's biggest gap ("no real harness ever drives a turn"). Adapts
gstack's PTY/headless agent harness to prove the bootstrap install + smoke work
against the REAL binaries, not PATH shims. Test/CI/docs only — zero src changes.

- test/helpers/agent-harness.ts: hermetic clean-room child env (ported from
  gstack; drops CONDUCTOR_/CLAUDE_/GSTACK_/MCP_/GBRAIN_, promotes
  GSTACK_ANTHROPIC_API_KEY→ANTHROPIC_API_KEY), real-binary resolvers + auth
  probes, headless `claude -p --output-format stream-json` and `codex exec
  --json` turn runners, a gbrain stdio MCP-config writer, and a keyless brain
  seeder. + a fixture-parse unit test (no binary needed).
- test/e2e/bootstrap-real-claude.serial.test.ts: real `gbrain bootstrap`
  install → REAL `claude mcp add` (verified via `claude mcp get`) → verify
  exit 0 → a real `claude -p --mcp-config --strict-mcp-config` turn that
  invokes mcp__gbrain__search and answers from the brain (proven: toolCalls
  include mcp__gbrain__search, final text carries the seeded fact).
- test/e2e/bootstrap-real-codex.serial.test.ts: same install with REAL `codex
  mcp add` into a real ~/.codex/config.toml + Gate-3 pull-protocol assertion,
  then a real `codex exec --json` turn surfacing the fact (MCP or the pull-
  protocol shell path). Bounded retry absorbs codex's occasional MCP-call
  cancellation without softening the fact-requiring assertion.
- Everything hermetic (temp HOME/CLAUDE_CONFIG_DIR/CODEX_HOME/GBRAIN_HOME;
  real ~/.codex auth copied read-only) and skipIf-gated so it self-skips
  cleanly where the binaries/auth are absent.
- heavy-tests.yml: gated `real-agent-e2e` job (nightly/label, never the PR
  shard; no-op on a runner without authed binaries).
- TODOS: compiled `gbrain` binary can't serve a PGLite brain (bun compile
  omits the WASM/extension payloads); harness falls back to `bun run` serve.

Verified against live claude 4.6 + codex 0.147.0: 15 pass / 0 fail; verify
36/36; typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): real-agent-e2e job — bash array + --timeout (actionlint SC2086 + bun-test-timeout guard)

The real-agent-e2e job's file loop used an unquoted $FILES (SC2086) and ran
`bun test` without --timeout (check-bun-test-timeout guard). Switch to a bash
array and add --timeout=600000 (real-agent turns are slow; the tests self-skip
without authed binaries so it's a no-op elsewhere).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): embed WASM + extension assets so the compiled binary can serve

A `bun build --compile` gbrain binary could not `serve` a PGLite brain: the
compile bundles JS but not PGLite's runtime payload (pglite.wasm, initdb.wasm,
pglite.data, vector/pg_trgm tarballs), so `serve` on PGLite died with a
bunfs/ENOENT. Now the assets ride inside the binary.

- src/core/pglite-embedded-assets.ts: embeds the five assets via
  `import … with { type: 'file' }` (the ENG-6 idiom) and exposes
  getEmbeddedPgliteOptions() → { pgliteWasmModule, initdbWasmModule, fsBundle,
  extensions:{vector,pg_trgm} }. WASM/fsBundle are consumed as bytes; the two
  extension tarballs are materialized to a content-addressed temp file (atomic,
  size-verified reuse) because PGLite reads them via fs.createReadStream, which
  cannot read a /$bunfs path. Unconditional (works in bun-run and compiled),
  so no fragile mode branch.
- src/core/pglite-engine.ts: static-import getEmbeddedPgliteOptions (engine
  path stays static per the engine-dynamic-import invariant); spread into both
  PGlite.create sites (initial + WAL-repair retry). The bunfs classifier stays
  as a backstop but no longer fires for a correct binary.
- scripts/check-pglite-embedded.sh (+ smoketest): compiles a focused binary and
  asserts it boots PGLite, CREATE EXTENSION vector/pg_trgm, and round-trips a
  page — wired into `bun run verify` (now 37 checks), check:all, and
  check:pglite-embedded. Fail-soft only when compile is unavailable.
- agent-harness.ts probeCompiledPglite now passes → the real-agent e2e uses the
  fast compiled MCP server. TODOS: the P2 "can't serve PGLite" item is closed.

Verified: fresh compiled binary ran `search`/`query` against a PGLite brain and
returned the seeded row (no bunfs/ENOENT); verify 37/37; pglite-engine 120/0
source-mode; typecheck clean; engine-dynamic-import + parity guards pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-11 16:42:11 -07:00
Garry Tan 75fae742d5 v0.44.0.0 feat(eval): BrainBench — cross-harness memory conformance suite (Cathedral 2) (#2158) 
Public, reproducible, cross-harness benchmark for the four ways agent memory fails (know-to-ask, push precision/recall, write-back fidelity, cross-session continuity). Hermetic PGLite, sealed gold, deterministic corpus, CI gate vs master's committed baseline, published JSON-schema foreign-runner contract. See docs/eval/BRAINBENCH.md.
 2026-08-10 09:44:46 -07:00
Garry TanandClaude Fable 5 c5952b8714 v0.43.0.0 feat(mcp,search): MEMORY_VERBS v1 five-verb protocol + concept-question query steering (cathedral 1, #2416) (#2172) 
* feat(verbs): MEMORY_VERBS v1 core — five frozen verbs over existing machinery

recall extended in place (G1B additive superset: protocol_version + string
fact_id/provenance on every response; query arm via cheap-hybrid with
keyword-only degradation when no embedding provider; op-layer budget packing
facts-first via the new generic packToBudget). remember/entity/synthesize/
forget as first-class Operations in src/core/verbs.ts (runtime leaf — TDZ-safe
both import directions) with MEMORY VERB descriptions, ToolAnnotations,
populated suggestions on every error (verbError + protocol ErrorCodes +
additive detail/protocol_version on OperationError.toJSON).

writeSingleFact (facts/write-single.ts): zero-LLM single-fact seam reusing the
pipeline's dedup; X1 deterministic supersession (same entity+kind, similar,
text differs -> supersedeId / fence expire+link); validUntil threaded through
FenceInputFact (was silently dropped). buildEntityCard (verbs/entity-card.ts):
reflex resolution arms + parallel depth-1 reads, per-arm degradation on
pre-page_aliases brains, near-miss suggestions with create_safety. runThink
gains additive usage aggregation; synthesize converts the no-LLM graceful
result to the unavailable protocol error and prices the cost block via
canonical model-pricing. RESPONSE_SCHEMAS registry + ERROR_SCHEMA for the
conformance kit.

Cathedral 1 stage 1/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(mcp): MEMORY_VERBS v1 surface mode + fail-closed dispatch + usage sidecar

gbrain serve --surface verbs|full (config mcp_surface registered in the merge
allowlist; default full — existing installs unchanged). Enforcement is
two-layer and fail-closed: filtered ListTools AND dispatchToolCall's new
allowedOps set, applied on stdio + BOTH HTTP paths (serve-http.ts OAuth and
the second http-transport.ts). ToolAnnotations passthrough in buildToolDefs +
the HTTP inline mapper, emitted only when an op defines them (byte-stability
for existing tools). Verb validation/internal errors speak the protocol
envelope (suggestion + protocol_version) at the dispatch layer.

E4 observability: per-verb JSONL sidecar (usage-log.ts — local only, never
uploaded; O_APPEND line-atomic, lock-free 10MB rotation) written from dispatch
so param-validation failures count; brain_id via gbrainPath; doctor gains the
memory_verbs_usage check. CLI: human-readable rendering for remember/entity/
synthesize with trailing --json escape; gbrain recall gains --query +
--budget-tokens routed through the recall OP (same arm MCP exercises) with the
budget footer; the forget verb carries no cliHint (CLI_ONLY forget dispatches
first and would shadow it).

Cathedral 1 stage 2/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(protocol): gbrain protocol — schemas, conformance kit, stats + TTHW

gbrain protocol [--json] emits the live verb input schemas + the hand-authored
RESPONSE_SCHEMAS + error contract + versioning policy. gbrain protocol
conformance certifies ANY MCP endpoint (default: self-spawned stdio; --target
http://...[--token] or a stdio command) against the embedded fixture set —
shape/enum/behavior/round-trip only, deterministic; entity-page cases seed via
put_page when available and skip honestly on verbs-only targets; synthesize is
cost-gated behind --synthesize (asserts the clean unavailable error on key-less
CI). Fixtures double as BrainBench seeds (test/fixtures/memory-verbs/cases.json
mirror). Self-certification passes 23/23 live.

gbrain protocol stats [--days N]: per-verb calls/error-rate/latency, budget
drops, entity hit rate, and measured TTHW (init stamps protocol_installed_at;
delta to the sidecar's first verb call) — local JSONL only, never uploaded.
init epilogues print the 3-command quickstart (mcp add --surface verbs +
key-free resurrection demo) with the visibility consent line. entity-card fix:
raw namespaced-slug input now matches exactly (slugify flattens slashes).

Cathedral 1 stage 3/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test+docs(protocol): conformance suite, p99 gate, MEMORY_VERBS_v1 doc, funnel

test/memory-verbs-conformance.test.ts: 26 tests through dispatchToolCall —
G1B superset regression (legacy fields byte-equal, additions allowed), budget
math incl. budget<first-item, keyword degradation (never an error), remember
provenance/ttl-trap/null-echo, remote round-trip via the world default +
private facts hidden, entity card schema-validated with the chat gateway
rigged to throw (zero-LLM guard) + all arms + fence test, synthesize
[EXPENSIVE + annotations + clean unavailable, forget idempotency/not_found,
writeSingleFact X1 supersession via the embed-transport seam + degraded
dedup, the F3 negative self-test (the runner FAILS lying servers: dropped
fields, bad enums, re-typed ids), fixture-mirror drift guard.
test/mcp-surface.test.ts: verbs=exactly 5, full=identity, dispatch-layer
fail-closed, strict flag parsing. test/entity-card-perf.slow.test.ts: 20K
pages/100K links/30K aliases/40K facts — measured p99 22.9ms vs the 100ms
contract + a recalibrated 50x ratio guard; wired into the CI perf job along
with stdio self-certification (23/23 CONFORMANT live). tool-defs byte-
stability test now pins per-op: ops WITHOUT annotations keep the exact
pre-v1 shape.

docs/protocol/MEMORY_VERBS_v1.md: the frozen contract — envelopes, error
codes, additive-forever policy, ttl forms (P30D rejected), status-not-
status_text, surface modes + why default full, per-harness installs, consent
line, conformance pass criteria + copyable --target examples, local-only
stats privacy, 200K manual bench recipe. README quickstart + docs/mcp pages
moved to --surface verbs with the claude-not-found recovery; llms-config
registered + bundles rebuilt; KEY_FILES entries for the 7 new modules.

Cathedral 1 stage 4/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): verify-gate compliance — test seam, allow-comment placement, import hygiene

usage-log gains __setUsageLogPathForTests so the verb tests redirect the
sidecar without mutating process.env.GBRAIN_HOME (test-isolation lint);
the gbrain-allow-direct-insert rationale moves onto the insertFact call line
(system-of-record guard greps per-line); protocol.ts drops its unused
`operations` import (operations-filter-bypass guard — operationsByName only).
The synthesize unit test now drives the chat-transport seam (deterministic
success envelope + priced cost block, zero real API spend); the keyless
`unavailable` conversion stays covered end-to-end by the CI conformance step,
which runs credential-free.

Cathedral 1 verify pass (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): close 3 ship-review P1s — forget cross-source, entity backlink leak, conformance exit code

Pre-landing review (codex, on the committed diff) caught three P1s the
plan-stage reviews flagged only in the abstract:

P1.1 trust boundary — forget expired by GLOBAL fact id with no source/visibility
scope: a remote caller in source A could expire facts in source B (incl.
private) by guessing ids. forgetFactInFence gains sourceId + worldOnly opts
(scope-check before any state inspection; out-of-scope → not_found, no
existence leak); the forget verb passes ctx.sourceId + worldOnly=remote.

P1.2 source isolation — entity card incoming edges + backlink_count leaked
across sources: getBacklinks(slug,{sourceId}) scopes only the TARGET page and
getBacklinkCounts has no source param, so a foreign-source page's slug surfaced
in a same-named entity's card. Replaced both with a both-sides-scoped query
(f.source_id = t.source_id = entity source, mentions excluded) inside the card
module — no engine-interface change.

P1.3 CLI exit code — protocol conformance/unknown-subcommand set
process.exitCode directly, which the PGLite/WASM force-exit seam ignores
(it reads the gbrain-owned setCliExitVerdict). A NOT CONFORMANT target printed
the banner but exited 0 — CI would green-light a non-conformant server. Now
routes through setCliExitVerdict; proven: unknown subcommand exits 1.

Regression tests: cross-source forget → not_found + foreign fact stays active;
remote can't forget a private fact; entity card omits foreign-source backlinks
from both edges and the count.

Cathedral 1 ship-review fixes (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.43.0.0 feat(mcp): MEMORY_VERBS v1 — frozen five-verb memory protocol + conformance kit (cathedral 1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: surface --surface verbs in local-stdio quickstart paths for v0.43.0.0

The MEMORY_VERBS v1 quickstart flag (gbrain serve --surface verbs) was added to
the README quickstart, CLAUDE_CODE.md, and CODEX.md during the build but three
other docs that show the local-stdio wire-up still presented bare `gbrain serve`:
the connect-coding-agent Path B walkthrough (the doc the README links to),
INSTALL.md, and DEPLOY.md's Local stdio path. Brought them in line so the
five-verb surface is consistent across every local-quickstart entry point.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(doctor): assign memory_verbs_usage check to the SKILL category set

The new memory_verbs_usage doctor check (Cathedral 1, E4) was added without a
category, tripping the doctor-categories drift guard (every check name must
belong to exactly one category set). It's an agent-capability adoption signal,
a sibling of retrieval_reflex_health — assigned to SKILL_CHECK_NAMES.

Cathedral 1 ship-review fix (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(verbs): propagate MEMORY_VERBS v1 to existing harnesses on upgrade

The verbs shipped reachable + self-describing, but an EXISTING install that
upgrades to 0.43 and keeps its skills had no active push toward them: no
post-upgrade notice (default surface stays 'full'), and no skill/guidance
routing — agents kept calling search/query/get_page/extract_facts.

Two additive propagation fixes:

- Post-upgrade feature pitch: src/commands/migrations/v0_43_0.ts (pitch-only,
  no-op idempotent orchestrator — the verbs ride existing tables, no schema/
  data migration). Registered in the migrations index so
  and the self-upgrade NOTIFY channel announce the five verbs +  +  on any upgrade past 0.43.
  Verified live: the pitch fires for an install upgrading from 0.42.43.0.
- Skill routing: skills/query (memory reads) and skills/brain-ops (read/write
  cycle) now teach the verbs — recall/entity/synthesize vs search/get_page/
  query, remember vs extract_facts, forget — with a fall-back-to-classic-ops
  note for older brains / full surface. The in-MCP tool descriptions already
  steered tool-list readers; this adds the narrative layer the audit found
  missing across all 29 skills.

apply-migrations future-list fixtures updated for the new 0.43.0.0 registry
entry. No DB migration; no contract change; verbs unaffected.

Cathedral 1 propagation (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): green the 3 failing CI shards — 3-seg migration version, hermetic synthesize test, fresh llms

Three real CI-shard failures from the propagation commit + master merges:

- migrations-registry (shard 10): v0_43_0 used a 4-segment version 0.43.0.0,
  but the registry enforces 3-segment semver (/^\d+\.\d+\.\d+$/). Changed
  the migration version to 0.43.0 (the RELEASE stays 0.43.0.0; migrations have
  always used 3-seg, e.g. 0.32.2). apply-migrations future-list fixtures
  updated to match.
- synthesize conformance test (shard 5): asserted a successful answer via the
  chat seam, but runThink builds its client through a real-key check (not the
  seam), so CI (credential-free) took the NO_ANTHROPIC_API_KEY path → the
  verb's unavailable conversion → isError. Now wraps the call in withEnv with a
  fake key AND the chat seam, so it's deterministic regardless of ambient
  credentials.
- build-llms (shard 8): the v0.42.44.0 master merge changed an inlined doc;
  regenerated llms.txt/llms-full.txt.

Cathedral 1 CI fixes (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(search): steer concept/landscape questions to query — descriptions, docs, lookup chain, CLI nudge (#2416)

The search/query routing guidance shipped a stale and backwards model:
SEARCH_DESCRIPTION still led with "Keyword search using full-text search"
(search has been cheap-hybrid — vector + keyword + RRF, expansion off —
since D4/D15), and the mandatory lookup chain in brain-first.md ordered
search-first with query only "if search is thin". For concept/landscape
questions ("all the X that do Y"), keyword+vector recall always returns
something, so the fallback never fires and expansion-recoverable
synonym-phrased matches drop silently.

- operations-descriptions.ts: fix the stale SEARCH lead (cheap-hybrid, no
  LLM expansion); both constants now route concept/synonym/landscape
  questions to query, cap the claim honestly (both are top-K — exhaustive
  enumeration belongs to list_pages), and keep every pinned redirect.
- brain-first.md: lookup chain is intent-driven (exact token → search;
  concept/landscape → query FIRST), cost claims corrected, and the
  anti-pattern is explicit: a nonzero search count is not a completeness
  signal. Same sweep in brain-ops/SKILL.md and query/SKILL.md, which
  duplicated the backwards order.
- search-modes.md: decision tree + quick-ref updated to the cheap-hybrid
  reality, new Tricky Spot #6 (count ≠ completeness), example names moved
  to placeholder people per the privacy convention. RETRIEVAL.md gets the
  matching one-liner.
- query-intent.ts: looksConceptShaped + conceptNudge — deterministic
  concept-shape classifier in the existing classifier home, tuned to favor
  false-negatives; cues owned by other routers (find_experts, salience
  ops) deliberately excluded. cli.ts prints the hint-only stderr nudge on
  BOTH result paths (local engine + thin-client routed), --quiet-gated;
  search stays the cheap hot path (never auto-reroutes).
- salience-llm-routing e2e: new concept-phrasings block asserts the
  descriptions actually route concept questions to query (verified live:
  20/20 — all 12 personal-query regressions still route to salience ops,
  all 8 concept phrasings route to query).
- KEY_FILES.md: query-intent entry updated to current state (the old
  entry pointed at a renamed file); llms bundle regenerated (no diff —
  entry is link-only).
- TODOS.md: MCP-envelope hint follow-up filed (remote agents are the
  higher-value half; needs a return-contract change kept out of this
  commit).

Verification: typecheck clean; 70 targeted unit tests + 20 live routing
evals green; all 24 check:all guards green; failing full-suite shards
triaged — think-pipeline.serial.test.ts:434 fails identically on the
clean tree (pre-existing on this branch, filed separately), the rest are
PGLite parallel-load contention that pass in isolation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): pin think's stub-path usage at null per the E2 accounting contract

The MEMORY_VERBS [E2] usage-accounting change deliberately moved "no LLM
ran" from `undefined` to `null` (the frozen cost-block spelling), but the
older #1698 stub-path test still pinned `undefined` and failed on every
run of this branch. The protected semantic is unchanged: no fake usage or
cost on the stub path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): cover the #2416 query-steering wave under v0.43.0.0

The 0.43.0.0 entry described only MEMORY_VERBS; the #2416 retrieval-routing
wave rides the same release. Adds its user-facing summary and refreshes the
release date to the actual ship date.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(protocol): reword so the #3502 docs-lint doesn't read a phantom command

Master's docs-lint (landed after this branch forked) scans for
`gbrain <verb>` references; the MEMORY_VERBS diagram's "gbrain reference
implementation" phrasing parsed as a nonexistent `gbrain reference`
command. Post-merge-only failure; both sides were green alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.43.0.0

Post-ship /document-release drift pass over the MEMORY_VERBS v1 +
#2416 query-steering waves:

- docs/tutorials/connect-coding-agent.md: the pasteable brain-first
  protocol still taught the pre-#2416 "Search first" rule and named
  only full-surface tools; step 1 now routes by question shape
  (exact tokens -> search, concept/landscape -> query first) and both
  steps map the five-verb surface names (recall/synthesize/remember).
  find_experts and query callouts marked with their surface.
- README.md + tutorial: "90-tool wall" count refreshed to 110 (the
  actual operation catalog).
- CLAUDE.md: contract-first op count ~90 -> ~110 with the v0.43.0.0
  MEMORY_VERBS clause; reference-map row for the memory-verbs
  protocol / --surface / conformance docs.
- TODOS.md: blank line before the v0.42.74.0 section header.
- llms-full.txt regenerated (build:llms; freshness test green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: apply cross-model doc-review fixes for v0.43.0.0

Verified findings from the /document-release Codex pass:

- MEMORY_VERBS_v1.md: disclose that conformance WRITES (marker-suffixed
  seed page + remember/forget facts, write-scope creds, artifacts not
  auto-deleted); quickstart header says 4 commands because it has 4.
- CHANGELOG v0.43.0.0: cost claim scoped per verb (budget/latency/cost
  block); remember "updates in place" -> supersedes-with-history (what
  write-single.ts actually does).
- DEPLOY.md: legacy bearer fallback works on BOTH engines now (both
  schemas carry access_tokens per http-transport.ts) — was Postgres-only.
- CLAUDE_CODE.md: list_skills/core-tools verify note marked full-surface
  (on --surface verbs those tools aren't on the surface).
- skills/query/SKILL.md: --surface full INCLUDES the verbs; fallback is
  for pre-0.43 servers only.
- skills/brain-ops/SKILL.md: search/query tool labels updated to the
  #2416 reality (cheap hybrid vs hybrid + LLM expansion).
- search-modes.md: stale "keyword search returns chunks" label.
- RETRIEVAL.md: note the query op defaults expand:true per call.
- Counts: 43 skills -> 52 (README x2, INSTALL), 30 skills -> 52
  (CLAUDE.md), "30+ tools over MCP" -> 110 (README).
- TODOS.md: filed 2 code-vs-spec follow-ups (recall filter composition +
  limit cap; synthesize unavailable-mapping breadth) instead of silently
  editing the frozen contract.
- llms-full.txt + skills.lock.json regenerated; freshness guards green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): regenerate the flag registry for the cathedral-1 command set

Master's #2185 strict flag validation ships a committed, generated
CLI_FLAG_REGISTRY with drift + freshness guards keyed off CLI_ONLY. The
merge added this branch's `protocol` command to CLI_ONLY, but the
committed registry was generated on master without it, failing CI shard
7's drift guard. Regenerated via `bun run build:flag-registry` on the
merged tree (95 commands; picks up `protocol` and the `--surface`
help-text mentions from the verbs work).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-09 17:25:00 -07:00
Garry TanandClaude Fable 5 f15480b9d0 v0.42.75.0 fix(pglite): in-place WAL auto-repair for the macOS Aborted() startup crash (#2575, #223, #1670) (#3901) 
* fix(pglite): in-place WAL auto-repair for the Aborted() startup crash (#223, #1670, #2575)

The 'macOS 26.x WASM bug' was a misdiagnosis: an unclean shutdown (typically
the OS-upgrade reboot) tears the data dir's WAL, and every subsequent open
fails WAL replay inside WASM with an opaque RuntimeError: Aborted(). This
ports the pg_resetwal recovery upstream rejected (electric-sql/pglite#994,
by @yestheboxer) and wires it into connect() as bounded auto-repair:

- src/core/pglite-resetwal.ts: pg_resetwal for PG17 NodeFS dirs, fail-closed
  layout validation, atomic+durable writes (tmp+fsync+rename), idempotent.
- src/core/pglite-repair.ts: whole-pg_wal-dir rename backup (zero transient
  disk), overwrite-order restore with mtime guard, cooldown sidecar +
  episode-scoped backup retention (newest 3 episodes), and a never-throws
  engine seam. Kill-switch: GBRAIN_PGLITE_WAL_REPAIR=off.
- pglite-engine.ts: verdict rename macos-26-3 -> wasm-abort, classifier now
  matches the real production message (it previously fell to 'unknown'),
  corrupt-beats-wasm precedence preserved, honest per-outcome error copy
  incl. the failed-not-restored arm, and repair only under a cleanly-acquired
  lock (new LockHandle.reaped provenance; never after reaping a holder).
- gbrain pglite-repair: manual dry-run/repair command (validate-before-lock,
  serve/reaped refusals, no --force by design).
- doctor: pglite_data_dir fs-check with recurrence escalation and backup
  inventory when a PGLite brain fails to connect.
- reinit-pglite: embedding flags default from file-only config so the
  recovery ladder's rebuild rung works bare mid-outage.
- stringifyPgliteInitError: message-less Emscripten ErrnoError objects no
  longer surface as [object Object].

Regression-tested against real brains: corrupt every WAL segment (truncate
and garbage variants), reopen, auto-repair fires, original rows readable,
process.exitCode stays contained (#2084).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): replace the macOS-26.x misdiagnosis with the corrupt-WAL recovery ladder

README + INSTALL.md shipped (via #1671) the claim that PGLite is incompatible
with macOS 26.x and that a Bun/WASM fix would restore it. The real cause is
torn WAL state from the upgrade reboot, now auto-repaired in place. Rewrites
those sections around the recovery ladder (auto-repair -> gbrain pglite-repair
-> reinit-pglite -> engine switch; native-Postgres recipe kept, credit
@roysaurav), adds the ENGINES.md troubleshooting section, updates the
KEY_FILES.md entries to current truth, files the two follow-up TODOs
(SIGTERM engine-close extension; pglite upgrade blocker), and regenerates
the llms bundles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): harden WAL auto-repair (pre-landing + adversarial review)

Review-army (security/testing/maintainability/perf) + Claude & Codex
adversarial passes on the WAL-repair wave. Correctness + safety hardening,
no behavior change to the happy path:

- Live-writer safety: repair refuses any reaped lock acquisition, a corrupt
  (unknowable-liveness) reap writes a cross-process quarantine marker that
  gates auto-repair AND the manual command for 10 min, isProcessAlive treats
  only ESRCH as dead (EPERM/malformed-pid read as alive), and a live
  postmaster.pid (native Postgres) is refused. Lock heartbeat + initial write
  are atomic (tmp+rename) so a torn read can't misclassify a healthy holder;
  an in-flight acquisition is no longer mistaken for corrupt.
- resetWal verifies the stored pg_control CRC before trusting/re-signing it —
  a damaged control file routes to rebuild instead of laundering corrupt
  checkpoint counters under a fresh CRC. Atomic 'wx' writes (no symlink
  follow), whole-pg_wal-dir rename backup, 64MB seg-size cap.
- Honest failure reporting: repairPgliteWal threads the real restore result
  out via WalRepairError so the 'failed-restored' vs 'failed-not-restored'
  message never lies; the not-restored copy names the correct restore paths.
- Episode lifecycle: episodes close on the next healthy connect (not just on
  a verified repair), a gutted (restored) backup loses its pin, stale (>24h)
  episode backups aren't reused, and the cooldown also caps repaired-only
  crash loops. Empty backup dirs are pruned on refusal.
- Command: rejects unknown flags and valueless --path (a destructive command
  must not silently mis-parse), confirm prompt goes to stderr (stdout stays
  clean for --json), embedding-flag defaults come from the config file only.
- Symlink confinement extended to global/; sidecar reuse path validated
  (prefix + no '..' + must still hold pg_wal); sidecar writes atomic.
- doctor recurrence escalation counts all attempts; data dir absolutized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): current-state KEY_FILES + WAL-repair follow-up TODOs

KEY_FILES.md pglite entries updated to the hardened truth (reap marker +
quarantine, atomic writes, CRC gate, global-symlink refusal, WalRepairError,
episode lifecycle). TODOS.md files the deferred judgment-call follow-ups
(unclean-shutdown gate on auto-repair; non-gbrain pglite consumer boundary;
mixed-version torn-lock double-read).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.75.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-08 17:01:20 -07:00
Garry Tan 25e4c0c3b1 fix(skills,docs): unify-types playbooks must pass apply:true after the #3574 default flip 
#3574 flipped the unify-types worker default to dry-run (jobs.ts:2221,
apply: data.apply ?? false) and updated the architecture docs, but three
agent-facing surfaces still presented the bare submit as the Apply step:
skills/schema-unify/SKILL.md 'Phase 3: Apply', skills/conventions/
schema-evolution.md, and README.md. Because #3545 also edited SKILL.md in
this wave, each PR looked self-consistent in isolation — only the composed
branch shipped a playbook whose apply step silently retypes nothing and
never flips the active pack. Skills distribute downstream via the skillpack,
so this would have propagated. Found by an independent cross-PR review pass.
 2026-08-01 07:40:48 +08:00
Time Attakc 5773736c63 fix(doctor,docs): warn that the npm name 'gbrain' is unrelated + detect a shadowing npm install (#505) (#3454) 
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The npm package named `gbrain` is an unrelated squatted package, so `npm install gbrain` gives users something that is not this project. Adds doctor detection that classifies real checkouts correctly, fails open, and is try/catch'd throughout. Classification rests on the bin-shape marker since this repo has no `repository` field — verified e2e.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: the remediation commands were not executed against a real global install, and the Windows `which -a` path is unexercised.
 2026-08-01 03:53:33 +08:00
Time AttakcGarry TanjessemsClaude Fable 5
 		314fefa560 fix(readme): correct broken OpenClaw and Hermes project links (#1961) (#3179) 
Point the OpenClaw and Hermes anchors in the "Have your agent install
it" section at their real upstream repos; the previous openclawagents
org URLs 404. Regenerated llms-full.txt to match.

Takeover of #1961 (fork branch) rebased onto current master.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jessems <jessems@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-21 20:22:27 -07:00
Time AttakcSinabinaRafael ReisClaude Fable 5
 		f8d11f67a3 feat(search): configurable FTS language + reindex command (lands #580/#581/#582) (#2941) 
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:

- Migration renumbered v116 -> v123 (master's v116 was already claimed by
  code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
  REPLACE trigger-function bodies (migration handler + reindex command)
  now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
  resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
  reindex_search_vector.pages/.chunks), id-keyset batched backfill
  (5000 rows/UPDATE) instead of single whole-table statements, and
  --json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
  written (README referenced it but no PR added it); llms bundles
  regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
  check-test-isolation).

Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-17 15:23:47 -07:00
roysauravandSaurav Roy fe6838ffac docs: add macOS 26.x Tahoe PGLite WASM workaround + native Postgres setup guide (#1671) 
PGLite's embedded WASM engine crashes on macOS 26.x (Tahoe) on Apple
Silicon during engine initialization. This adds:

- A Troubleshooting section in docs/INSTALL.md with step-by-step
  instructions for using native Homebrew PostgreSQL 17 + pgvector
  as a workaround
- A callout in README.md's Troubleshooting section pointing users
  to the detailed setup guide

Tested on macOS 26.5 (arm64), Bun 1.3.14, gbrain 0.41.29.0,
PostgreSQL 17.10 (Homebrew), pgvector 0.8.0. All 102 schema
migrations pass. gbrain doctor green.

Co-authored-by: Saurav Roy <roysaurav@users.noreply.github.com>
 2026-07-16 20:48:54 -07:00
Russell SmithClaude Opus 4.8Garry Tan
 		f09f9177a9 v0.42.10.0 feat(extract): opt-in global-basename wikilink resolution (closes #972) (#1388) 
* v0.40.8.2 fix(extract): opt-in global-basename wikilink resolution (#972)

Bare wikilinks like [[struktura]] that point at pages in another folder
were silently dropped from the graph. The issue reporter saw 71 wikilinks
in Obsidian render to 12 in gbrain (~83% lost). Symptoms downstream:
`gbrain graph` returns thin neighborhoods, `gbrain backlinks` undercounts.

This release adds an opt-in mode that resolves bare wikilinks by basename
match, covers all three resolver surfaces (FS-source extract, DB-source
extract, put_page auto-link), and emits one edge per match — no silent
winner on ambiguity. `gbrain doctor` surfaces a paste-ready enable hint
when ≥5 bare wikilinks would resolve under the new mode.

Enable with:
  gbrain config set link_resolution.global_basename true
  gbrain extract links

Default stays off. Existing brains see zero behavior change on upgrade.

Closes #972. Adapts PR #1233 from @rayers (regex shape + slug-tail index)
into a multi-match, opt-in form with FS-source coverage that the original
PR explicitly skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document opt-in global-basename wikilink resolution (#972)

The #972 feature shipped with no user-facing docs — only CHANGELOG + CLAUDE.md.
Anyone migrating an Obsidian/Notion vault with bare [[name]] wikilinks couldn't
discover the link_resolution.global_basename flag unless gbrain doctor happened
to surface its hint.

- README "Self-wiring knowledge graph": one sentence on the opt-in mode for
  Obsidian-style cross-folder bare wikilinks + the doctor pre-check, linking to
  the install step.
- INSTALL_FOR_AGENTS Step 4.5 (Wire the Knowledge Graph): a dedicated agent-
  facing subsection — when bare [[name]] links need it, the enable command,
  re-running extract, the doctor opportunity hint, and the multi-match behavior.
- Regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): resolve aliased wikilinks by target slug, not display text

Codex outside-voice [P1]: `[[struktura|the project]]` resolved the basename
"the project" (the alias) instead of `struktura` (the target), because
extractPageLinks called resolveBasenameMatches(ref.name) and the doctor check
keyed basenameIndex.get(e.name). ref.name is the display alias (match[2]);
ref.slug is the wikilink target (match[1]).

- extractPageLinks resolves ref.slug; context excerpt locates ref.slug.
- doctor link_resolution_opportunity keys e.slug so its estimate matches
  what extraction actually resolves.
- Test: aliased wikilink calls resolveBasenameMatches with the target, never
  the display text.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): reconcile wikilink-resolved edges in put_page auto-link

Codex outside-voice [P1]: put_page's reconcilableOut filter excluded
link_source='wikilink-resolved', so a basename edge written by auto-link
survived after the bare wikilink was deleted from the page OR the
link_resolution.global_basename flag was turned off (the stale-removal loop
only iterates reconcilableOut). Add 'wikilink-resolved' to the reconcilable
set; manual edges still untouched.

Test: write page with [[struktura]] (flag on) → edge lands; re-put without
the wikilink → edge reconciled away.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): source-scope basename resolution (no cross-source edges)

Codex outside-voice [P1]: makeResolver.resolveBasenameMatches called
engine.getAllSlugs() unscoped, so a bare [[name]] could resolve to a
same-tail page in a DIFFERENT source and create a cross-source edge. The
engine exposes getAllSlugs({sourceId}) precisely to prevent this. #972 is
"global basename across folders," not "cross-source federation" — the
canonical gbrain multi-source bug class.

- makeResolver gains opts.sourceId; ensureBasenameIndex passes it to
  getAllSlugs (unscoped only when sourceId omitted — back-compat).
- runAutoLink (put_page) passes opts.sourceId; extractLinksFromDB passes
  sourceIdFilter. FS extract is already single-source (walks one dir).
- Tests: scoped index returns only the source's slugs (no cross-source);
  unscoped call stays brain-wide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): FS-source basename edges carry link_source='wikilink-resolved'

The FS extract path is the issue's default repro (gbrain extract links with no
--source db). ExtractedLink had no link_source field, so FS basename edges
landed with the engine default ('markdown') instead of the 'wikilink-resolved'
provenance the DB / put_page paths set and the docs promise. The e2e FS test
only asserted link_type, so it was blind to this.

- ExtractedLink gains link_source?; extractLinksFromFile sets it to
  'wikilink-resolved' on basename edges (undefined for ordinary markdown).
- Carries through the addLinksBatch snapshots automatically (LinkBatchInput
  already has link_source); single-row addLink fallback now passes it too.
- e2e FS repro asserts link_source === 'wikilink-resolved'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(#972): one shared basename matcher across resolver/FS/doctor

Codex outside-voice [P2] DRY: three surfaces each hand-rolled a basename
matcher with divergent key sets — the doctor omitted the slugified key, so its
link_resolution_opportunity estimate undercounted what extraction resolves, and
the resolver returned matches in unsorted getAllSlugs bucket order.

New shared exports in link-extraction.ts: buildBasenameIndex(slugs) +
queryBasenameIndex(index, name) (keys raw/lower/slugified tail; stable sort
shorter-first then lexical) + normalizeBasename.

- makeResolver.resolveBasenameMatches → queryBasenameIndex (now stable-sorted).
- extract.ts resolveBasenameMatchesFromSlugs → delegates to the shared pair.
- doctor link_resolution_opportunity → shared builder/query (slugified key
  added; estimate now matches extraction).
- Test: doctor counts a slugified-only match ([[Fast Weigh]] → companies/fast-weigh).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): P2 cluster — masking, code-fence, self-link, dedup decision

Codex outside-voice P2 findings:
- P2a markdown-label masking: a wikilink inside a markdown-link label
  ([see [[acme]]](companies/acme.md)) spawned a stray generic basename ref.
  Pass-1 can't match the nested brackets, so a new MARKDOWN_LABEL_WIKILINK_RE
  masks those spans out of pass 2c. Inner [[acme]] is now inert.
- P2b FS code-fence: the FS path (extractMarkdownLinks on raw content) didn't
  strip code blocks like the DB path. extractLinksFromFile now scans
  stripCodeBlocks(content) so [[name]] inside a fence creates no FS edge.
- P2c self-link guard: a basename [[own-tail]] on its own page resolved back
  to itself. Dropped in both extractPageLinks and the FS path.
- P2d dedup: documented the decision to KEEP qualified + bare edges to the
  same target as separate rows (distinct provenance/audit trail).
- P2e: skipFrontmatter unresolved-contract tests added.

Tests: P2a inert-label, P2c self-link drop, P2b code-fence, P2e unresolved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(#972): bound the doctor link_resolution_opportunity scan

The check did listAllPageRefs() + a getPage() per page under a 60s budget.
On a large brain (the eng-review concern) it hit the budget every non-fast
doctor run and returned a perpetual partial, adding ~60s.

Now batch-loads the 1000 most-recent pages in ONE query
(ORDER BY id DESC LIMIT SAMPLE_LIMIT) and scans in memory, with the 60s cap
kept as a backstop. Mirrors the v0.40.9 sampling convention. The estimate
message names the bound when the brain exceeds the sample
("scanned the 1000 most-recent of N pages").

Test: source-grep pins the bounded query + the absence of the per-page
getPage walk.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(#972): reconcile stale version/migration references to v112 / 0.42.6.0

Merge churn left intermediate refs: schema.sql + schema-embedded.ts said
"migration v93", CLAUDE.md said "v0.41.32.0 / Migration v109", CHANGELOG said
"Migration v93". Reconciled all to migration v112 / shipping 0.42.6.0. The
CLAUDE.md annotation is also refreshed to describe the final behavior (shared
matcher, source-scoping, alias-by-target, stale-edge reconciliation, bounded
doctor scan) and credit @rayers + @ukd1. Regenerated schema-embedded + llms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): register doctor check category + bump llms budget to 800KB

Two full-suite gate failures from the re-sync:
- doctor-categories drift guard: the new `link_resolution_opportunity` check
  wasn't in any category set. Added to BRAIN_CHECK_NAMES (alongside
  graph_coverage / orphan_ratio — it's a graph-quality signal).
- build-llms size budget: the #972 Key Files annotation (landing with master's
  #1696/#1699 waves) pushed llms-full.txt past 750KB. Bumped FULL_SIZE_BUDGET
  750KB→800KB, the established "budget tracks CLAUDE.md's legitimate per-feature
  growth" pattern (600→700→750→800 across releases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
 2026-06-02 16:33:06 -07:00
Garry TanandClaude Opus 4.8 7b0d99adb0 v0.42.2.0 feat: gbrain connect — one-command Claude Code onboarding from a bearer token (#1683) 
* fix: gbrain auth create dropped the name on the bare (no-flag) form

Extract parseAuthCreateArgs; only exclude the --takes-holders value from the
positional search when the flag is present (rest[takesIdx+1] resolved to rest[0]
when takesIdx === -1, silently dropping the name). Add regression test.

* feat: gbrain connect — one-command Claude Code onboarding from a bearer token

New connect command prints a paste-ready claude-mcp-add block (or --install wires
it + smoke-tests the token via a raw-bearer get_brain_identity probe). Direct HTTP
MCP, literal-token default, URL normalization, token header-injection guard,
--json redaction, execFileSync (no shell). Wired into CLI_ONLY + CLI_ONLY_SELF_HELP
+ handleCliOnly. 58 unit + 3 PGLite-E2E cases; e2e-test-map updated.

* docs: lead CLAUDE_CODE.md with gbrain connect (remote fast path) + README one-liner

Regenerate llms-full.txt for the README change.

* refactor: pre-landing review fixes for gbrain connect

- DRY: single DEFAULT_PROBE_TIMEOUT_MS + shared isAuthErrorMessage predicate
- reuse promptLine (shared stdin lifecycle) for the --install confirm
- harden redactToken with a Bearer <value> scrub (defense in depth)
- +8 tests: orchestrator guard paths, deterministic timeout, invalid --timeout-ms,
  Bearer-redaction

* fix: adversarial-review hardening for gbrain connect

- probe: Promise.race the call against a real timer so a stalled connect()/SSE
  handshake (signal alone doesn't cover it) can't hang --install indefinitely
- probe: close transport even if client.connect() throws
- parseArgs: reject a missing/flag-shaped value (e.g. --token --install)
- block link-local / cloud-metadata hosts (169.254/fe80:/fd00:ec2::254) — keeps
  localhost + RFC1918 LAN brains working
- non-interactive --install now requires --yes
- clearer message when --force removed then add failed
+8 tests covering each

* fix: codex-review P2s for gbrain connect

- POSIX single-quote the rendered claude-mcp-add command so a token with shell
  metacharacters ($(), backticks) can't trigger command substitution on paste
- detect IPv4-mapped IPv6 metadata addresses (::ffff:169.254.x.x / ::ffff:a9fe:*)
  so they don't bypass the link-local guard
+3 tests

* chore: bump version and changelog (v0.42.2.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document gbrain connect + connect-probe in CLAUDE.md Key files (v0.42.2.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: gbrain connect — add codex and perplexity agents

--agent codex emits 'codex mcp add ... --bearer-token-env-var GBRAIN_REMOTE_TOKEN'
(token read from the env var at runtime, never in Codex config; --install runs it).
--agent perplexity prints the URL + token for the Settings → Connectors GUI (no
--install). Generalized the command file: AGENT_SPECS table, buildCodexMcpAddArgv,
cmdString(binary,argv), binary-generic ConnectDeps (hasBinary/runBinary/env),
agent-aware buildConnectBlock/buildJson. +25 tests.

* docs: codex + perplexity connect paths (new CODEX.md, README, CHANGELOG, CLAUDE.md)

Regenerate llms-full.txt for the CLAUDE.md/README edits.

* test: real-CLI E2E for connect — drive actual claude + codex against a live server

Adds claude-code + codex cases to connect-bearer.test.ts that run the real
'claude mcp add' / 'codex mcp add' through 'gbrain connect --install' against a
live 'gbrain serve --http' (sandboxed HOME/CODEX_HOME), then assert via
'claude mcp get' / 'codex mcp get' that the server registered (and codex's token
stays out of config). Skips when the binary is absent. Perplexity is GUI-only so
it's print-asserted. Regen llms for the CLAUDE.md note.

* docs: perplexity OAuth + serve --bind/--public-url footgun (per Perplexity feedback)

PERPLEXITY.md now documents the host-side HTTP setup (gbrain serve --http
--bind 0.0.0.0 --public-url, the v0.34 ECONNREFUSED footgun) and the OAuth 2.1
client_credentials path (gbrain auth register-client) alongside the legacy
bearer token. The 'connect --agent perplexity' output points at the same
bind/public-url requirement + PERPLEXITY.md.

* feat: gbrain connect --oauth — client-credentials path for perplexity/generic

OAuth is the correct path for a third-party cloud connector (Perplexity): instead
of a long-lived full-access bearer token, the connector gets Issuer URL + Client
ID + Client Secret and mints short-lived scoped tokens. --oauth --register mints a
least-privilege client on the host (shells gbrain auth register-client); --oauth
--client-id/--client-secret uses an existing one. Rejected for claude-code/codex
(bearer) and with --install. Issuer derived from the mcp-url. New E2E proves the
full chain: register → connect --oauth → OAuth discovery → /token client_credentials
mint → get_brain_identity tool call against a live server. Docs: PERPLEXITY.md leads
with OAuth; README + CLAUDE.md updated; +18 unit cases.

* docs: add gbrain connect to INSTALL.md MCP section + link CODEX.md

The remote-client onboarding command was documented in README/CLAUDE_CODE/CODEX/
PERPLEXITY but missing from INSTALL.md §3 (the natural 'how do I connect a client'
home). Add the one-command connect how-to (claude-code/codex/perplexity) and the
missing docs/mcp/CODEX.md link.

* fix: connect LEARN_INSTRUCTION names put_page, not CLI-only capture

The self-orientation block told a connected agent that `capture` is an
available MCP tool. It isn't — `capture` is a CLI-only convenience command;
the MCP write tool is `put_page`. An agent that followed the instruction hit
"unknown tool". Drop capture; put_page was already in the list. Adds a
regression block to connect.test.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: serve --http surfaces skill-publishing status (banner + nudge)

When mcp.publish_skills is OFF, connected agents can search/write but can't
call list_skills/get_skill, so the host's skill catalog is invisible to them.
The startup banner now shows a Skills: line, and a stderr nudge fires when off
with the paste-ready fix. Pure skillPublishStatus() helper, unit-tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: prove the local stdio MCP funnel end-to-end

Spawns real `gbrain serve` (stdio) against a freshly init --pglite brain and
drives the official MCP SDK client through initialize -> tools/list ->
tools/call (get_brain_identity + search). Pins the advertised core-tool set
against what the server actually exposes (asserts capture is NOT advertised).
This funnel had zero e2e coverage before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: make batch-retry-audit ENOENT case hermetic

The 'no-op when audit dir does not exist' case called
pruneOldBatchRetryAuditFiles(30) without a GBRAIN_AUDIT_DIR override, so it
read the real ~/.gbrain/audit and flaked (kept:1) on any dev machine with a
batch-retry-*.jsonl on disk. Point it at a guaranteed-missing temp subdir,
matching this file's own hermetic-header contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: two-funnel coding-agent onboarding (Claude Code / Codex)

New tutorial docs/tutorials/connect-coding-agent.md: Path A (connect to an
existing brain) + Path B (start from nothing, local stdio), the brain-first
protocol to paste into CLAUDE.md/AGENTS.md, and the four translatable habits.
README gains a 'Quick start: Claude Code or Codex' fork separating lightweight
retrieval from the full autonomous install. INSTALL.md shows the one-command
wire-up at the standalone CLI section. mcp/CLAUDE_CODE + CODEX cross-link the
tutorial + note publish_skills + capture-is-CLI-only. Tutorial promoted to
Shipped in the tutorials index.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: changelog + regenerated llms (v0.42.2.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: CLAUDE.md Key Files annotation for two-funnel onboarding wave

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-01 19:09:15 -07:00
Garry TanandClaude Opus 4.8 eefe8b5741 v0.42.1.0 feat: gbrain skillopt — self-evolving skills (closes #1481) (#1563) 
* feat(skillopt): foundation modules — types, lr-schedule, benchmark, score, audit, lock

* feat(skillopt): edit primitives — apply-edits (D5+D9), rejected-buffer LRU, version-store (D8 history-intent-first)

* feat(skillopt): rollout (D2 gateway.toolLoop + D13 read-only allowlist), reflect (D7 two calls), validate-gate (D12 median+epsilon, D4 parallel), preflight (D3), bundled-skill-gate (D16)

* feat(skillopt): orchestrator (D6 slow-update, D10 ASCII diagrams, D11 caching), checkpoint, bootstrap (D15 sentinel), CLI dispatch + help

* feat(skillopt): cycle phase (F1 dream-loop wiring), PROTECTED_JOB_NAMES + MCP op (F6 admin scope + allowlist) + Minion handler (F7 --background)

* feat(skillopt): full cathedral — --all batch (F4), --target-models fleet (F5), write-capture (F10), held-out scaffold (F11), adversarial suite 41 cases (F2), E2E PGLite (F3), meta-skill bundle (T7), reflect+judge evals (F8+F9), docs (T10)

* chore: bump version to v0.42.0.0 (MINOR — significant new feature)

* fix(skillopt): wire trajectories from forward gate to reflect + fix parseEditsResponse parser misuse

Two related v0.42.0.0 bugs that conspired to make `runSkillOpt` structurally
unable to accept any candidate edit. Either alone would have killed self-evolution;
together they made the loop a no-op for every input.

**Bug 1 (orchestrator gap):** `runOptimizationLoop` in orchestrator.ts called
`runReflect({successes: [], failures: []})` with hardcoded empty arrays. The
forward gate's `scoredRollouts` were computed then voided. `runReflect`
short-circuits both modes when their batches are empty, so the optimizer was
never asked to propose an edit. Every step hit the no_edits_applied branch.

Fix: add `scoredRollouts: ScoredRollout[]` to `GateResult` and
`runsPerTask?: number` to `ValidateGateOpts`. Forward pass uses
`runsPerTask: 1`; orchestrator partitions returned rollouts by `score >= 0.5`
and threads real successes + failures into `runReflect`.

**Bug 2 (parser misuse):** `parseEditsResponse` in reflect.ts routed every
optimizer response through `parseJudgeJson` first. `parseJudgeJson` looks for
a `score` key (it's a judge-output parser, not an edits parser) and returns
null for any JSON without one — including the well-formed `{"edits": [...]}`
the optimizer is contractually required to emit. The function then early-
returned `[]` and the actual `tryExtractEdits` path on the next line was
unreachable dead code.

Fix: drop the wrong-typed guard. `parseEditsResponse` now calls
`tryExtractEdits` directly. Export it so `reflect.test.ts` can pin the
contract independently of the chat transport.

**Why this slipped through 152 prior skillopt tests:** zero unit coverage
of `parseEditsResponse` or `runReflect`. The existing E2E `all-reject` case
asserted no_improvement (which was true for the wrong reason — empty edits,
not gate rejection). Both bugs were structurally invisible to the existing
test surface.

**New coverage:**

- `test/skillopt/reflect.test.ts` (15 cases):
  - 8 `parseEditsResponse` cases including the IRON-RULE regression pin
    for the v0.42.0.1 fix (`{"edits": [...]}` JSON must survive the parser).
  - 7 `runReflect` D7 contract cases: both modes fire, empty-batch skips,
    additive token usage, one-mode-throws-other-still-works, rejected-buffer
    flows into anti-bias prompt.
  - Documents the trailing-comma limitation as an explicit out-of-scope pin
    (so a future tightening of `tryExtractEdits` lights this test up
    intentionally).

- `test/e2e/skillopt-loop.serial.test.ts` (7 cases):
  - HAPPY PATH: stubbed `gateway.chat` acts as both target agent (emits
    sections based on skill content) and optimizer (proposes a real
    add-Citations edit). Drives `runSkillOpt` end-to-end against PGLite.
    Asserts outcome=accepted, SKILL.md mutated with new section,
    frontmatter preserved (D5), history has one committed row,
    best.md mirrors disk, delta > epsilon, receipt fields populated.
  - 5 broken cases (each isolates a distinct orchestrator-visible failure):
    1. Below-baseline regression: optimizer proposes a destructive edit;
       gate rejects with reason=below_baseline; SKILL.md unchanged;
       rejected-buffer captures the bad edit for anti-bias context.
    2. Malformed reflect JSON: orchestrator degrades gracefully to
       no_improvement without crashing.
    3. Anchor-not-found: applyEditBatch rejects all; sel gate skipped;
       rejected-buffer captures with reason=apply_failed.
    4. Budget exhausted mid-step: outcome=aborted, no pending rows survive.
    5. Converged-skill re-run: starting from already-perfect skill →
       no_improvement (no thrash on a well-tuned starting point).
  - IDEMPOTENT RE-RUN: drive runSkillOpt twice in sequence. Run 1 accepts.
    Run 2 sees improved baseline, no failures, returns no_improvement.
    SKILL.md byte-identical to post-run-1; history still has exactly 1
    committed row. Proves stability at the fixed point.

All hermetic (no DATABASE_URL, no API keys). PGLite in-memory engine,
tempdir SKILL.md + benchmark, stubbed gateway.chat via
`__setChatTransportForTests`. `.serial.test.ts` because the stub installs
module state and the loop walks shared disk state across epochs.

Test counts after fix: 174 skillopt-surface tests pass (149 pre-existing
unit + 15 new reflect unit + 3 existing E2E + 7 new E2E). Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cycle): align ALL_PHASES skillopt position with actual dispatch order

v0.42.0.0 added skillopt to ALL_PHASES right after `patterns` (line 127), but
the dispatch block in runCycle (line ~1912) actually runs skillopt between
`conversation_facts_backfill` and `embed`. The two were inconsistent, and the
serial test `report.phases.map(p => p.phase)).toEqual(ALL_PHASES)` was failing
on master because of it.

A second pre-existing failure: the two phase-count assertions in
`test/core/cycle.serial.test.ts` still said `toBe(20)` even though
ALL_PHASES grew to 21 when skillopt was added. The author bumped the array
but forgot the test.

Two fixes, one commit:

1. Move `'skillopt'` in ALL_PHASES from after `patterns` to between
   `conversation_facts_backfill` and `embed`, matching where runCycle
   actually dispatches it. Runtime behavior is unchanged — only the
   declaration order moves. Updated the surrounding comment to call out
   the position invariant and reference the test that pins it.

2. Update both `toBe(20)` assertions in cycle.serial.test.ts to `toBe(21)`
   with a v0.42.0.0 history line in the running comments.

Why declaration follows runtime (not the other way around): the comment
intent ("Runs AFTER patterns — graph-fresh") is still satisfied because
"after the entire main graph-mutating cluster" is strictly fresher than
"right after patterns". No design intent is lost.

Test result: cycle.serial.test.ts is now 28/28 (was 27/28 on master + my
prior commit). Skillopt suite still 174/174.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): bump PHASE_SCOPE assertion to 21 + fix skill-optimizer Anti-Patterns case

Two CI failures pre-existing on this branch since the v0.42.0.0 skillopt
cathedral landed; master is green because skillopt didn't exist there yet.

1. test/phase-scope-coverage.test.ts asserted ALL_PHASES.length === 20.
   skillopt is the 21st phase. Bumped to 21 with v0.42.0.0 history line
   in the comment chain. Sibling fix to the cycle.serial.test.ts bump
   in commit 08ad2468.

2. skills/skill-optimizer/SKILL.md had `## Anti-patterns` (lowercase p).
   skills-conformance.test.ts asserts `## Anti-Patterns` (capital P) as
   the required section header. Single-character rename.

Local: 174 skillopt-surface tests + 6 phase-scope tests + 249 skills-
conformance tests all green. Typecheck clean.

Remaining CI delta: 5 put_page facts backstop failures in shard 10 that
reproduce only on Linux CI, not locally even with empty env / cleared
HOME / max-concurrency=1. The error surface is `r.isError === true` with
no further detail captured in the bun:test output. Pushing these 2 fixes
first to narrow the CI signal; will instrument if the 5 persist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): align dream-cycle-phase-order + onboard-full-flow with v0.41/v0.42 reality

Two stale E2E assertion files surfaced by a full local E2E run against
real Postgres (the gbrain-test-pg container on port 5434). Neither file
is in the CI E2E job (CI only runs mechanical.test.ts + mcp.test.ts +
skills.test.ts + zeroentropy-live.test.ts), so the drift has been latent.

1. `test/e2e/dream-cycle-phase-order-pglite.test.ts`
   EXPECTED_PHASES was missing 4 phases that landed in master since the
   list was last revised:
     - extract_atoms (v0.41 T9 — atom extraction, after extract_facts)
     - synthesize_concepts (v0.41 T9 — concept synthesis, after patterns)
     - conversation_facts_backfill (v0.41.11.0, after calibration_profile)
     - skillopt (v0.42.0.0 — self-evolving skills, between
       conversation_facts_backfill and embed)
   Updated to 21 entries in the actual runtime dispatch order (matches
   ALL_PHASES exactly). 5/5 tests in the file pass after.

2. `test/e2e/onboard-full-flow.test.ts`
   `runAllOnboardChecks` shape test asserted exactly 4 checks; v0.42's
   type-unification cathedral (PR #1542, T13-T15) added 3 more
   (`pack_upgrade_available`, `type_proliferation`, `dangling_aliases`)
   for a total of 7. And `empty brain returns 0 remediations` regressed
   because `pack_upgrade_available` can emit a manual_only remediation
   on brains where gbrain-base@1.x is active and gbrain-base-v2 is
   registered as a successor. Tightened that assertion to `total <= 1`
   AND kept a per-check guard asserting takes_count remediations stay 0
   (the original test's load-bearing claim — A12 two-gate consent).
   13/13 tests in the file pass after.

Honest scope: 4 other E2E files still fail locally after this commit
(cycle.test.ts, dream.test.ts, phantom-redirect.test.ts,
sync-lock-recovery.test.ts), each for a distinct pre-existing master
bug unrelated to v0.42 skillopt work:
  - cycle.test.ts (5 fails): PostgresEngine.getConfig falls back to
    db.getConnection() singleton via the `get sql()` getter when no
    poolSize is set; the new conversation_facts_backfill phase chain
    hits this fallback even though the test's setupDB() connects both
    the singleton AND the engine. Race condition between the test's
    singleton lifecycle and the phase's getConfig call. Deeper fix
    needed in PostgresEngine.getConfig (use this._sql directly with
    explicit fallback only on user-driven CLI paths).
  - dream.test.ts (1 fail): expects "concepts/testing" slug to appear
    in dream cycle output, gets empty array. Related to v0.42 concept
    type-unification semantics.
  - phantom-redirect.test.ts (2 fails): concurrent-sync race +
    postgres-js text-string embedding survival. Master-level data-path
    bug; would need its own fix wave.
  - sync-lock-recovery.test.ts (1 fail): `gbrain sync --break-lock
    --all` exits 0 but test expects 1 with a shell-loop hint. CLI
    behavior changed in a master commit; need to either restore the
    refusal behavior or update the assertion.

None of these 4 block CI (E2E job doesn't run them). Filed as a
TODOS.md entry for a follow-up wave; the 2 in this commit are the
ones that mirror v0.42 work landing.

Local: 130/136 E2E files green, 927/940 tests pass (was 925/940
before these fixes; the 2 files this commit fixes added 7 newly-
passing tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): quarantine query-cache-knobs-hash.test.ts to serial runner

CI shard 10 (commit 4d721077) failed 5 tests in the
`SemanticQueryCache cross-mode isolation (CDX-4 hotfix)` describe block,
all ~7-34ms each, all expecting writes/reads to round-trip through one
shared PGLite engine + a `beforeEach DELETE FROM query_cache`. Passes
9/9 locally; fails 5/9 on Linux CI under bun's default in-file
max-concurrency=4.

Classic intra-file concurrency race shape: test A's `beforeEach`
clears the table → test A's `store` writes a row → test B's
`beforeEach` (concurrent with A's `store`) clears the table → test A's
follow-up COUNT query returns 0. Same root cause that quarantined
`embed-stale.test.ts`, `brain-allowlist.test.ts`, and
`schema-pack-find-pack-successors.test.ts` to the serial runner in
prior fix waves (documented in v0.41.22.0 CI fix wave).

Fix: rename to `query-cache-knobs-hash.serial.test.ts` so the v0.26.7
serial-tests runner picks it up at `max-concurrency=1`. Tests still
exercise the actual cache logic — no test deleted, no production code
changed. The describe block's `beforeAll` engine + `beforeEach`
TRUNCATE pattern works correctly at serial concurrency.

Local: 12/12 in this file + 52/52 in the serial runner. Production
SemanticQueryCache code is untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(heavy): frontmatter_scan_wallclock — opt into --no-embedding so CI runners work

Heavy tests workflow run 26542447602 (commit 483a5577) failed on the
first heavy script:

  [fm_wallclock] FAIL: gbrain init exited non-zero
  No embedding provider configured. Set one of:
    OPENAI_API_KEY / ZEROENTROPY_API_KEY / VOYAGE_API_KEY
  Or defer setup: gbrain init --pglite --no-embedding

The v0.37 D9 hard-require landed in init.ts: `gbrain init --pglite` now
refuses to proceed without an embedding provider configured. The
heavy-tests GitHub workflow doesn't pipe any embedding API keys
(deliberate — the heavy tests measure ops shape, not LLM behavior), so
every CI invocation now blocks at step 2 of this script.

The script's whole purpose is measuring `gbrain doctor`'s
frontmatter-scan wallclock — it never embeds, never calls
`gbrain embed`, never queries vectors. The right fix is to opt out of
the provider requirement via the same `--no-embedding` flag init.ts
already exposes for this exact "deferred setup" case.

Verified locally:
  TMP=$(mktemp -d); GBRAIN_HOME="$TMP" \
    bun run src/cli.ts init --pglite --yes --no-embedding
  # exit 0, brain initialized.

No production code change. One-line + comment in the script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(heavy): sync_lock_regression — pass --no-embed so CI runs measure lock contention, not key absence

Heavy tests workflow run 26542545802 (commit 7962d312, after the
previous fm_wallclock fix) failed at the next heavy script in the chain:

  [sync_lock_regression] outcomes: winners=0 losers=0 unknown=4
  [sync_lock_regression] FAIL: expected 1 winner, got 0
  [sync_lock_regression] FAIL: expected 3 lock-busy losers, got 0

Each of the 4 parallel `gbrain sync` invocations failed for the same
reason — none of them ever even got to the lock-acquire step:

    Embedding model "zeroentropyai:zembed-1" requires ZEROENTROPY_API_KEY.
    Re-run with --no-embed to import-only and embed later once the key is set.

The CI runner doesn't pipe any embedding-provider API keys (deliberate —
heavy tests measure ops shape, not LLM behavior), and sync now hard-fails
when its embed step can't reach a configured provider.

This script measures the writer-lock race shape — `gbrain-sync` row in
`gbrain_cycle_locks`, exactly-one-winner semantics, N-1 fail-fast losers
with "Another sync is in progress", zero leaked rows post-run. It never
needed embeddings; the original write predates the hard-require landing.

Fix: pass `--no-embed` to the sync invocation. Same kind of fix as
fm_wallclock (commit 7962d312) but on the sync side rather than init.

No production code touched. One-line change in the bash script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(heavy): sync_lock_regression — register source via psql + use --repo + tolerate doctor warns

Heavy tests run 26542638471 (commit 60145eee, after the --no-embed
fix) failed at the same script but at a downstream step:

  > Source "default" has no local_path. Run: gbrain sources add default --path <path>

Three independent bugs in the script that all surfaced at once after
v0.41's source-registry landed:

1. `gbrain config set sync.repo_path` is the legacy way; sync now
   reads `sources.local_path` first. Replaced with an upsert into the
   sources table via psql:
     INSERT INTO sources (id, name, local_path)
     VALUES ('default', 'default', $BRAIN_DIR)
     ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path
   Kept the legacy `config set sync.repo_path` line too as
   belt-and-suspenders for any downstream caller that still reads it.

2. `gbrain sync --dir <path>` is silently ignored; sync's CLI parser
   recognizes `--repo`, not `--dir`. Switched to `--repo`.

3. `bun run src/cli.ts doctor --json` at the top (used to apply
   migrations as a side effect) exits non-zero whenever ANY check
   warns — including the new "no embedding provider configured"
   warning on a fresh CI runner. The script's `set -e` aborted at
   line 53 before reaching any of the sync invocations. Added `|| true`
   since the migration runs regardless of doctor's exit verdict.

Verified locally — `DATABASE_URL=... bash tests/heavy/sync_lock_regression.sh`
output:
  [sync 1] rc= (lock-busy: 'Another sync is in progress')
  [sync 2] rc=0 (winner)
  [sync 3] rc= (lock-busy: 'Another sync is in progress')
  [sync 4] rc= (lock-busy: 'Another sync is in progress')
  outcomes: winners=1 losers=3 unknown=0
  post-run gbrain_cycle_locks(gbrain-sync) row count: 0
  OK — 1 winner, 3 lock-busy losers, no leaked lock rows.

Production code untouched. All three fixes are in the bash script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(skillopt): hands-on tutorial for auto-improving a skill + discoverability

There was no tutorial for skillopt — only a reference guide
(docs/guides/skillopt.md) that opens at --bootstrap-from-routing and
assumes you already understand benchmarks, and an agent-facing SKILL.md.
README had ZERO skillopt mention. The one thing a user must hand-author
(the benchmark JSONL) was taught nowhere with a worked example.

New: docs/tutorials/improving-skills-with-skillopt.md — Diataxis tutorial
(learning-oriented), copy-pasteable end to end:
  1. mental model in two sentences (SKILL.md is the trainable param, the
     agent is frozen)
  2. write your first benchmark from scratch — a complete 15-task rule-judge
     starter you paste and run, with the full check-op table
     (contains/regex/section_present/max_chars/min_citations/tool_called/
     tool_not_called)
  3. --dry-run cost preview (and that it exits 2 by convention, not failure)
  4. real run + reading accepted(0)/no_improvement(1)/aborted(2) with the
     actual stderr output shape
  5. where output lands (best.md, versions/, history.json, rejected.json,
     audit jsonl)
  6. accept/reject — bundled vs user skills, --no-mutate vs
     --allow-mutate-bundled
  7. iterate by sharpening the benchmark

The load-bearing fix the tutorial makes that the reference guide got wrong:
the DEFAULT --split 4:1:5 needs ~50 tasks before it runs (sel = N/10, floor
5). A first-time author writing 10-15 tasks hits `D_sel has N task(s)
(need >=5)` and bounces. The tutorial ships 15 tasks + `--split 1:1:1`
(clean 5/5/5) so the copy-paste path actually works. Verified against the
real loadBenchmark + splitBench: the exact shipped block parses 15 unique
tasks and splits 5/5/5 with sel>=5; the system's own error message confirms
"need ~50 total for 4:1:5".

Discoverability (Diataxis cross-linking):
  - README.md tutorials section: new entry (was zero skillopt mention)
  - docs/tutorials/README.md: added under ## Shipped
  - docs/guides/skillopt.md: "New to this? Start with the tutorial" callout

Every claim devex-verified against source: exit-code map from
skillopt.ts (accepted:0/no_improvement:1/aborted:2/errored:2), stderr
format from skillopt.ts:286-292, check ops from score.ts, output paths
from SKILL.md, split math from benchmark.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: regenerate llms-full.txt after skillopt tutorial + README edit

Refreshes the inlined doc bundle so the committed llms-full.txt matches
fresh `bun run build:llms` output (test/build-llms.test.ts drift guard).
Picks up the README tutorials-section edit from c39dbdb1. The new tutorial
file itself isn't curated into scripts/llms-config.ts (the bundle curates
a fixed doc set, not every tutorial) — this is purely the README delta.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): stop embed-preflight leaking gateway config into facts-backstop shard

CI shard 10 failed 5 `put_page facts backstop` tests with:

  [embed(openai:text-embedding-3-small)] Incorrect API key provided: sk-test

(captured by the diagnostic stderr added in a prior commit). Root cause is
a cross-file module-state leak, not a logic bug:

- `embed-preflight.test.ts` calls `configureGateway({env:{OPENAI_API_KEY:
  'sk-test'}})` to drive credential-validation scenarios. It resets the
  gateway `beforeEach` but never AFTER its last test, so it leaves the
  gateway configured with `sk-test`.
- bun runs every file in a shard inside ONE process. The residual config
  bleeds into the next file. When `facts-backstop-gating.test.ts` lands in
  the same shard, its put_page calls see `isAvailable('embedding') === true`
  (the key is *present*, just invalid), so put_page attempts a real embed
  and 401s before the backstop gating even runs.
- It's intermittent across master merges because shard bin-packing changes
  which files co-locate. (It "resolved" after the v107 merge earlier for
  exactly this reason, then came back.)

R1/R2 test-isolation lint doesn't catch this — it's `configureGateway`
module state, not `process.env` or `mock.module`.

Two fixes, both using the gateway's own `resetGateway()` seam (no
process.env, R-compliant):

1. embed-preflight.test.ts — `afterAll(() => resetGateway())` so the leaker
   cleans up after the whole file. Primary fix; also protects any OTHER
   shard-mate that reads gateway state.
2. facts-backstop-gating.test.ts — `beforeEach(() => resetGateway())` so the
   suite is deterministic regardless of ambient gateway config. Defense in
   depth: isAvailable('embedding') is now reliably false → put_page uses
   noEmbed → the import never embeds → only the backstop gating (the suite's
   actual subject) is exercised.

Verified: running leaker+victim in one process (the shard repro) goes
16/16; full shard 10 goes 1208/1208 (was 5 fail in CI). Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(skillopt): make benchmark authoring an agent job, not a human chore

The prior tutorial taught a human to hand-write a 15-task benchmark — but
nobody does that. The real workflow is: user says "make skill X better,"
the AGENT authors the benchmark and runs the optimizer. The agent-facing
dispatcher didn't actually cover that.

Gap found: skill-optimizer/SKILL.md documented exactly one authoring path,
`--bootstrap-from-routing`, which (a) requires a pre-existing
routing-eval.jsonl (bootstrap-benchmark.ts:57-63 refuses without it) and
(b) generates tasks from ROUTING fixtures — which test dispatch ("does
this phrasing pick this skill"), not output quality. So an agent told to
improve a skill with no benchmark had no documented way to author a
*quality* benchmark; it'd have to reinvent the JSONL format the human
tutorial teaches.

Two fixes:

1. skills/skill-optimizer/SKILL.md — new "Authoring the benchmark yourself
   (the common case)" section: read the target SKILL.md, generate ~15
   realistic tasks, attach rule judges (contains/max_chars/min_citations/
   section_present/regex/tool_called), write the JSONL, run with
   `--split 1:1:1` (the default 4:1:5 needs ~50 tasks). Decision-tree row
   "New skill, no benchmark" now says "Author one" instead of pointing at
   bootstrap-from-routing; the bootstrap row is reframed as a head-start
   that only applies when routing fixtures exist and notes routing tasks
   test dispatch, not quality.

2. docs/tutorials/improving-skills-with-skillopt.md — new "The easiest
   path: ask your agent" section up top. Tells humans to just tell their
   agent "improve my X skill — write a benchmark first," and frames the
   manual walkthrough as "read this when you want to understand or
   hand-curate what the agent is doing."

Verified: conformance 249/0, resolver 99/0, build-llms drift guard 7/0,
cross-link resolves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(skillopt): --bootstrap-from-skill starter benchmark generator

Generate a quality benchmark from a skill's SKILL.md directly, no
routing-eval.jsonl required. One LLM call emits JSONL tasks (each with rule
judges) that the agent reviews + strengthens before optimizing.

- runBootstrapFromSkill: JSONL output parsed line-by-line with skip-bad-line
  salvage (a truncated final line drops, the rest survive); a task is kept only
  when >=2 valid rule checks survive; provider errors propagate instead of
  collapsing to bootstrap_empty.
- --bootstrap-tasks N (default 15, cap 50); maxTokens scales with the count.
- Extracted assertBenchmarkAbsent + readSkillBodyOrThrow shared with the routing
  bootstrap; hardened runBootstrap's routing-eval parse to skip malformed lines.
- CLI: --bootstrap-from-skill short-circuit + 6-way mutual exclusion; parseFlags
  exported for unit tests. The benchmark-not-found hint + --help now point here.
- The generator's REVIEW line prints the paste-ready
  `--bootstrap-reviewed --split 1:1:1` next command (the default 4:1:5 split
  refuses a 15-task starter at D_sel >= 5).
- 20 hermetic cases incl. round-trip into loadBenchmark + splitBench(1:1:1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(skillopt): make --bootstrap-from-skill the primary no-benchmark path

The agent runs --bootstrap-from-skill, strengthens the generated judges (they
are weak drafts), deletes the sentinel, then runs --bootstrap-reviewed
--split 1:1:1. Freehand authoring is demoted to the fallback for the rare skill
the generator can't draft well. Updates the Iron Law, decision tree, and
anti-patterns to cover both bootstrap modes and the 15-task / --split 1:1:1
gotcha.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(release): v0.42.1.0 --bootstrap-from-skill

VERSION + package.json -> 0.42.1.0, CHANGELOG entry, CLAUDE.md skillopt
annotation, regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: surface --bootstrap-from-skill in README + skillopt reference

- docs/guides/skillopt.md: 30-second pitch leads with --bootstrap-from-skill;
  flag table adds --bootstrap-from-skill + --bootstrap-tasks rows.
- README.md: skillopt tutorial pointer mentions generating a starter benchmark.
- Regenerated llms-full.txt (README is in the bundle).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): bump FULL_SIZE_BUDGET 700KB→750KB for legitimate CLAUDE.md growth

The skillopt wave annotations + merged v0.41.34-36 master releases pushed
llms-full.txt to 700,423 bytes — 423 over the 700KB cap — failing the
build-llms size-budget test on CI shard 6. CLAUDE.md is ~540KB (77% of the
bundle) and is the whole point of the one-fetch artifact, so it stays inlined;
the budget tracks its per-release growth. 750KB still fits 200k+ context models.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-31 08:20:25 -07:00
Garry TanandClaude Opus 4.8 6f26d5e4df v0.41.37.0 fix: critical fix wave — reindex tag wipe, grandfather hang, Windows migration spawn, sync ReDoS (#1621 #1581 #1605 #1569) (#1665) 
* fix(reindex): add-only tag reconciliation + DB-only re-chunk preserves frontmatter (#1621)

reindex --markdown and re-import no longer wipe DB-side enrichment tags.
Tag reconciliation is now ADD-ONLY (import-file.ts): re-import adds current
frontmatter tags and never deletes, so auto/dream/signal-detector tags survive.
The reindex DB-only fallback reconstructs full markdown via serializeMarkdown
so re-chunking a page with no on-disk source preserves frontmatter/title/timeline.

* fix(migrations): v0.13.1 grandfather chunked, source-safe, soft-delete-filtered (#1581)

phaseCGrandfather rewritten from a per-page getPage+putPage loop (which hung
70+ min on an 82K-page PGLite brain) to a chunked bulk SQL pass keyed on
pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL,
with a batched rollback log carrying source identity.

* fix(migrations): run schema phases in-process to fix Windows getaddrinfo ENOTFOUND (#1605)

The 9 'gbrain init --migrate-only' execSync spawns died on Windows+bun+Supabase
(child DNS resolution). runMigrateOnlyCore (extracted from initMigrateOnly) runs
the schema bring-up in-process for all engines, unblocking schema_version
advancement. Includes async-call-site audit, a wall-clock guard, and a
runGbrainSubprocess stderr-capture wrapper for the remaining backfill spawns.

* fix(sync): ReDoS hardening + diagnostics for schema-pack regexes (#1569)

Input-length cap in runRegexBounded + route the unbounded link-inference path
through it (closes the only no-timeout ReDoS hole); star-height lint rule warns
on nested-quantifier patterns; --no-schema-pack sync escape hatch; GBRAIN_SYNC_TRACE
per-file begin heartbeat; PGLite serve/sync concurrency doc. Defensive hardening +
diagnostics — the deterministic ~3100-file wedge root cause remains open (no repro).

* docs(todos): file v0.41.37.0 fix-wave follow-ups (#1621/#1605/#1569)

* chore: bump version and changelog (v0.41.37.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync README + CLAUDE.md for v0.41.37.0 critical fix wave

Add reindex add-only tag reconciliation (#1621), v0.13.1 grandfather +
Windows in-process migration (#1581/#1605), and schema-pack ReDoS
hardening + sync --no-schema-pack / GBRAIN_SYNC_TRACE triage (#1569)
to CLAUDE.md key-files annotations and README Troubleshooting.
Regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): bump llms-full.txt budget 700KB→750KB (CLAUDE.md crossed 700KB after master merge)

The build-llms size-budget test failed: llms-full.txt is 703,244 bytes after the
v0.41.37.0 key-files annotations merged on top of master's v0.41.34/35/36 CLAUDE.md
additions. Matches the v0.41.9.0 precedent (600→700); the single-fetch bundle still
fits comfortably in modern long-context models.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-05-30 14:56:26 -07:00
Garry TanandClaude Opus 4.8 d6db3f0ce3 v0.41.34.0 feat(search): retrieval cathedral — max-pool + title + alias + evidence (#1657) 
* feat(search): per-page max-pool in searchVector (both engines)

T1 of the retrieval-cathedral wave (supersedes #1616). Vector search returned
chunk-grain top-k with no DISTINCT ON, so a page could be represented by a
weak chunk while a hub page's chunks crowded a distinct page's strong chunk
out of the candidate set entirely. Keyword search always pooled per page; the
vector path did not.

- New shared buildBestPerPagePoolCte() in sql-ranking.ts — single source of
  truth consumed by searchKeyword + searchVector across postgres + pglite, so
  the two engines can't drift (the recurring parity bug class).
- searchVector both engines: compute score as a select-list expr (HNSW
  ORDER BY stays pure-distance), pool DISTINCT ON (slug) over the full
  candidate set before the user LIMIT, deterministic tiebreak
  (slug, score DESC, page_id ASC, chunk_id ASC).
- All keyword pooling blocks refactored onto the shared builder (DRY).
- Regression test: a hub page's chunks no longer crowd out a distinct page's
  strong chunk; results are one-per-page by best chunk. Fails on old path.

Verified: real-Postgres engine-parity 22/22, PGLite hermetic suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): title-phrase boost (page.title first-class signal)

T2 of the retrieval-cathedral wave. A query that is a phrase from a page's
title ("Greek amphitheater" -> "The Mingtang - Indoor Greek Amphitheater")
matched a weak body chunk instead of being recognized as a title hit. Names
of things deserve weight.

- New pure title-match.ts: isTitlePhraseMatch (contiguous token-run inside
  page.title OR exact full-title match). Precision guards: >= 2 content
  tokens OR exact full-title; stopword filter; token-boundary match (no raw
  substring). Reused by the eval later so production + bench can't drift.
- applyTitleBoost post-fusion stage in hybrid.ts: reads page.title (not the
  brittle "first chunk"), floor-ratio-gated, stamps title_match_boost for
  --explain, never touches base_score (the agent's dedup confidence).
- ModeBundle.title_boost knob (1.25, on in all modes - cheap gated
  correctness fix), search.title_boost config key, dashboard description.
- KNOBS_HASH_VERSION 6 -> 7 so a boost-on cache write can't serve a
  boost-off lookup; all version-pin + canonical-bundle assertions updated.
- 18 new tests (matcher 13 + stage 5); typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): page_aliases data layer (T3 foundation)

Free-text alias resolution for search. gbrain stored a page's chosen names
in pages.frontmatter `aliases:` JSONB but search never consulted them, so a
query like "Hall of Light" or "明堂" couldn't surface the "Mingtang" page.

DELIBERATELY SEPARATE from slug_aliases (re-grounded against current code):
  - slug_aliases:  old-slug -> canonical-slug (wikilink/get_page redirect,
    populated only from concept-redirect conversions)
  - page_aliases:  normalized free-text name -> canonical slug (search hop)
Overloading slug_aliases would muddy two distinct semantics, so this is a
new table, not an extension (honors DRY by keeping concepts separate).

- src/core/search/alias-normalize.ts: ONE normalizeAlias() (NFKC + lowercase
  + ws-collapse + quote-strip) + normalizeAliasList() shared by the write
  (ingest) and read (search) paths so they match on the same key (CQ2).
- Migration v108 page_aliases (source_id, alias_norm, slug); btree
  (source_id, alias_norm) for indexed-equality hop, NOT ILIKE; unique TRIPLE
  (not source_id+alias_norm) so two pages may claim one alias — collisions
  reported + resolved at query time, not blocked at ingest (Codex#8).
  Mirror in pglite-schema.ts; Postgres fresh gets it from the migration.
- engine.resolveAliases(aliasNorms, {sourceId|sourceIds}) read +
  setPageAliases(slug, source, aliasNorms) write, both engines, source-scoped.
- 17 tests: normalize round-trip, collision, source-scope, replace, clear.

Ingest projection + the hybridSearch alias hop land next (T3 wiring).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): alias hop + ingest projection (T3 wiring)

Wires the page_aliases data layer into ingest (write) and hybridSearch (read)
so a query that is a page's declared chosen name surfaces that page — the
named-thing class neither max-pool nor title-boost can fix (true synonyms with
zero surface overlap: "Hall of Light" / "明堂" -> the Mingtang page).

- Ingest projection (import-file.ts): after the page write commits,
  normalizeAliasList(frontmatter.aliases) -> engine.setPageAliases. Always
  called (even []) so removing an alias clears its row; content_hash includes
  non-timestamp frontmatter so alias edits reach this path, not the skip branch.
  Fail-soft + pre-v108-safe (isUndefinedTableError swallowed).
- applyAliasHop (hybrid.ts), AFTER rerank so a named query reliably surfaces
  its page: FULL normalized-query exact match only (no substring/n-grams),
  skip >6-token prose queries, present-boost 1.10x / inject absent canonical at
  top-of-organic + epsilon (never absolute 1.0, D3), collisions alpha-ordered +
  capped at 3, fail-open on pre-v108 / lookup error (D9). Stamps alias_hit for
  the T4 evidence contract.
- SearchResult.alias_hit attribution field.
- 8 tests: inject/boost/CJK/no-match/long-skip/collision + ingest projection
  round-trip + alias-removal-clears. 73 pass across the T1/T2/T3 + import suite.

Backfill of existing pages' aliases lands as T8 (reindex --aliases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): evidence/create_safety contract + search→cheap-hybrid + per-call mode (T4)

The agent-facing fix for the incident's ROOT behavior: tonight the agent read a
single blended 0.64 score, decided "no strong match, safe to write a new page",
and wrote a duplicate on a developed concept page. A blended RRF/cosine score is
not a calibrated probability, so the don't-duplicate decision must key off WHY a
page matched, not a raw number.

- evidence.ts: classifyEvidence (alias_hit > exact_title_match > high_vector_match
  > keyword_exact > weak_semantic) + createSafetyFor (exists | probable | unknown).
  stampEvidence runs at the end of every hybrid return path (main + both keyword
  fallbacks). SearchResult gains evidence + create_safety. The agent keys
  don't-duplicate off create_safety='exists', not a score threshold.
- search op → cheap-hybrid everywhere (D4/D15): full vector+keyword+RRF+pool+
  title+alias, expansion OFF (no per-call LLM cost); `query` stays full-control.
  search.mcp_keyword_only escape hatch (D17) keeps the old keyword-only behavior
  for operators who don't want query text sent to an embedding provider.
- Alias hop + evidence now also run on the keyword-only fallback paths (the
  named-thing fix is most valuable exactly when vector is unavailable).
- Per-call `mode` (D5): honored ONLY for local/trusted callers (ctx.remote===
  false) so a remote OAuth client can't escalate to costly tokenmax; local +
  unknown mode rejects loudly; threaded into resolveSearchMode + the cache key.
- 30 tests (evidence classifier incl. before/after-incident cases, per-call mode
  gate, alias hop). Updated mcp-eval-capture to the new cheap-hybrid contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): reconcile `gbrain search` dispatch (T5)

After T4 made the `search` op cheap-hybrid, `gbrain search "x"` already does the
right thing — but `gbrain search modes/stats/tune` would have run a hybrid search
for the literal word "modes" instead of opening the config dashboard (the op
intercepts before the unreachable handleCliOnly dashboard path).

Add a pre-dispatch interception in main(): `search` + subArgs[0] in
{modes,stats,tune} → runSearch dashboard (with the v0.41.6.0 read-only connect+
dispatch 10s timeout preserved); everything else (free-text) falls through to the
cheap-hybrid `search` op. Subprocess test pins all three routes:
modes/stats → dashboard, free-text → search op ("No results", not "Unknown
subcommand").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(eval): NamedThingBench retrieval-quality gate (T6)

The eval that makes the retrieval-maxpool incident impossible to reintroduce
silently. 7 query families, each a failure class the incident exposed:
title-substring, generic-to-named, alias-synonym, multi-chunk-dilution,
short-vs-rich, graph-relationship, hard-negative.

- src/eval/retrieval-quality/harness.ts: pure scoring (Hit@1/Hit@3/MRR per
  family) + injected SearchFn (CLI uses hybridSearch; tests stub it) +
  evaluateGate. D12 gate: hard-gate the families that ARE the bug from day one
  (title-substring Hit@1>=0.95, alias-synonym Hit@1>=0.98, dilution Hit@3=1.0),
  warn-then-enforce the softer families. Env-overridable floors.
- `gbrain eval retrieval-quality <fixture.jsonl> [--json] [--source]` +
  dispatch in eval.ts. Exit 0 PASS / 1 FAIL / 2 USAGE.
- Synthetic fixture (placeholder names only, privacy-grep guarded) + hermetic
  gate test: seeds a synthetic brain, forces the keyword+title+alias path
  (embed transport stubbed to throw — free, deterministic), asserts the bug
  families pass. The vector max-pool guarantee is pinned separately by
  searchvector-maxpool.test.ts.
- CI gate: the hermetic test is a normal unit test, so it runs in every PR
  shard — the gate is live on every change.
- 23 tests (harness unit + hermetic gate + fixture privacy guard).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(telemetry): rank-1 score drift signal (T7)

Standing observability so a retrieval regression is caught before a human hits
it in chat (like tonight). Aggregate columns on search_telemetry (NOT per-query
rows, D10): sum_rank1_score + count_rank1 + 3 coarse buckets (<0.6 / 0.6-0.85 /
>=0.85). The mean rank-1 base_score is the headline; a downward drift = retrieval
quality regressing.

- hybrid.ts: capture rank-1 base_score at all three return paths, thread through
  emitMeta → recordSearchTelemetry opts (like results_count).
- telemetry.ts: Bucket + record + flush ON CONFLICT-add + readSearchStats expose
  avg_rank1_score (null when no samples — no NaN) + rank1_distribution.
- Migration v109 ADD COLUMN IF NOT EXISTS (both engines; search_telemetry lives
  only in migration v57, so the v57+v109 chain covers fresh + upgrade). Columns
  exempted in schema-bootstrap-coverage (no forward-ref index → no bootstrap need).
- `gbrain search stats` surfaces the avg + bucket line; JSON envelope auto-carries
  the fields. "true-positive" wording dropped per Codex#14 — production has no
  labels, so this is an unlabeled rank-1 score histogram; labeled calibration
  lives in NamedThingBench (T6).
- 3 round-trip tests (mean+buckets, no-result excluded, empty=null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reindex): gbrain reindex --aliases backfill (T8)

Import-time projection (T3) covers new + changed pages; this backfills EXISTING
pages whose frontmatter `aliases:` predate v108 / the projection. Walks
listAllPageRefs (cheap cross-source (source_id, slug) enumeration), reads each
page's frontmatter aliases, writes page_aliases via setPageAliases.

Idempotent (setPageAliases replaces) so re-running is convergent — no op-checkpoint
needed (fast, no embedding). --dry-run reports would-write counts, --source
narrows, --limit caps, --json envelope, progress reporter. Wired into the
`reindex` dispatch alongside --markdown / --multimodal.

4 tests: backfill from array + comma-scalar frontmatter, --dry-run writes
nothing, idempotent second run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(search): pre-migration fail-open regression (T9)

Pins that pre-v108 brains (no page_aliases table) keep working: applyAliasHop
returns input unchanged + doesn't throw, importFromContent with frontmatter
aliases still imports (projection swallows table-missing via isUndefinedTableError),
and resolveAliases surfaces the error for the caller to catch.

Completes the T9 mandatory regression set (dilution → searchvector-maxpool,
dispatch → cli-search-dispatch, MCP contract → mcp-eval-capture, engine parity
→ engine-parity 22/22, pre-migration → here).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): Phase-0 retrieval diagnostic — `gbrain search diagnose` (T0)

The operator-facing trace the user runs against the production brain to pin
which retrieval layer surfaces (or misses) a target page — the diagnostic the
plan front-loaded so we don't ship a fix that doesn't move the incident.

`gbrain search diagnose "<query>" --target <slug> [--json] [--source]` reports,
for the target: keyword rank+score, vector rank+score (skipped/graceful if no
embedding provider), whether the query is a registered alias, and the hybrid
final rank + evidence + create_safety + which boosts fired (title/alias). The
verdict names the layer that surfaces the target at rank 1 (or "none"), telling
you whether the lever is max-pool/innerLimit (vector) vs title/alias.

Wired into the `search` dispatch alongside modes/stats/tune (60s timeout since
it runs real retrieval). 2 hermetic tests (alias-query trace + title-phrase
trace). For the Mingtang incident, run:
  gbrain search diagnose "Greek amphitheater" --target projects/new-greek-theater/concept_v0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(retrieval): corrected incident record + named-thing layers + glossary (T10)

- RETRIEVAL_MAXPOOL_INCIDENT.md: replaces closed PR #1616's RFC with the
  verified record — what happened, the disease, the corrections to the RFC's
  mechanics (search was keyword-only, --mode unthreaded, hybrid already pooled
  at dedup, aliases dead to search), the four-layer fix that shipped, and the
  triage commands (search diagnose / reindex --aliases / search stats / eval
  retrieval-quality).
- RETRIEVAL.md: new "Named-thing retrieval" section documenting per-page pool +
  title boost + alias hop + the evidence contract, reconciling the doc with the
  shipped pipeline (closes the doc/reality gap).
- metric-glossary.ts + regenerated METRIC_GLOSSARY.md: Hit@1, Hit@3,
  avg_rank1_score (drift signal, not labeled accuracy), and create_safety
  (the evidence contract) now carry plain-English glossary entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(eval): NamedThingBench fixture privacy guard via slug-shape (T6 fixup)

The banned-name literal list itself tripped check-privacy/check-test-real-names.
Replace it with the load-bearing assertion: every fixture slug must be an
*-example placeholder (no real brain page can be referenced).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(search): source-isolation in per-page pool + alias hop (P0, codex adversarial)

Codex outside-voice caught two source-isolation P0s in the retrieval wave — the
exact class the v0.34.1 seal guards. Both fixed before merge.

P0-1: buildBestPerPagePoolCte pooled on `slug` alone. In a federated brain, two
pages with the same slug in different sources collapsed before ranking/pagination
(the neighbor-source page dropped). Now DISTINCT ON (COALESCE(source_id,'default'),
slug) — composite key matching dedup.ts's pageKey. Also fixes the PRE-EXISTING
keyword-path bug (best_per_page was slug-only before this wave); real-PG parity 23/23.

P0-2: the alias hop dropped source_id. resolveAliases returned bare slugs and
applyAliasHop hydrated via getPage(slug, undefined), so a federated caller could
get the default-source page injected or the right allowed-source page suppressed.
resolveAliases now returns {slug, source_id} pairs; applyAliasHop matches by
(source_id, slug) and fetches each canonical in its OWN source.

Regression tests: alias hop boosts only the aliased source (not same-slug in
another source); resolveAliases keeps cross-source same-slug distinct.

Deferred as documented tradeoffs (TODO): evidence high_vector_match label uses
blended base_score not pure cosine; deep-pagination candidate budget is
chunk-bounded; telemetry writes swallow errors pre-v109 on rolling deploys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.41.30.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: v0.41.30.0 retrieval cathedral — CLAUDE.md key files + llms regen

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: renumber release v0.41.30.0 → v0.41.34.0 (queue moved)

Version trio + CHANGELOG header + CLAUDE.md key-file annotations + TODOS
heading + regenerated llms bundles, all moved to 0.41.34.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): restore glossary roster + harden facts-anti-loop hook budget

Two CI failures surfaced after the master merges that brought the branch to
111 migrations:

1. shard 1 — `ALL_METRICS roster > matches the renderer output (no orphans)`:
   the merge took master's `renderMetricGlossaryMarkdown` whose `groups`
   array lacked this branch's 4 retrieval-quality keys (hit@1, hit@3,
   avg_rank1_score, create_safety). `ALL_METRICS` (derived via Object.keys)
   kept them, so the roster test saw 4 orphans. The freshness check
   (check:eval-glossary) passed because renderer-output == committed doc —
   it can't catch a renderer that drops a metric; the roster test can.
   Restored the "Retrieval-Quality / Evidence Metrics (NamedThingBench)"
   group + regenerated docs/eval/METRIC_GLOSSARY.md.

2. shard 2 — facts-anti-loop's two engine-dependent put_page tests failed
   while the two engine-free extractFactsFromTurn tests passed (the
   signature of a partially-failed beforeAll). This file has a documented
   PGLite-cold-start-under-deep-shard-load timeout history; the 30s budget
   was tuned for 95 migrations and the chain is now 111. Bumped to 60s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): isolate facts-anti-loop in its own process (serial)

Follow-up to the prior hook-timeout bump, which was the wrong theory: the
[58ms]/[71ms] body times in the re-run prove beforeAll did NOT time out —
the engine connects and the two put_page tests run and fail for real, while
the two engine-free extractFactsFromTurn tests in the same file pass.

put_page (via dispatchToolCall) touches process-global singletons (the
facts queue + the AI gateway used by importFromContent's embed step). Some
sibling file in the 78-file shard-2 process leaves residual global state
that makes put_page's pre-backstop path fail on the CI runner. The failure
is NOT reproducible alone, in a Linux oven/bun:1 container, or in a full
local shard-2 run (1172 pass) — only on the GitHub runner, deterministically.

Per CLAUDE.md's test-isolation rules, a test coupled to shared process
state belongs in its own process. Renamed to *.serial.test.ts so it runs
in the dedicated serial-tests job (scripts/run-serial-tests.sh spawns a
fresh `bun test` per serial file), where it passes deterministically;
test-shard.sh excludes serial files from the matrix. Updated the comment
to reflect the real cause and refreshed the test-weights.json key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): close cross-file gateway-config pollution in test shards

The prior serial-move theory was incomplete. The real, single root cause
behind all three shard failures (2, 5, 10) is cross-file AI-gateway config
pollution within a shard's bun process:

- A test calls configureGateway() and doesn't restore the gateway on exit.
  The legacy-embedding preload pins OpenAI/1536 ONCE at process start and
  re-pins per-test ONLY when the gateway slot is empty — so a leaker that
  reconfigured the gateway to the v0.37 default (zeroentropyai:zembed-1 /
  1280-d) and never reset poisons every later file in the shard.
- Victim A (shard 5, test/search/searchvector-maxpool.test.ts): runs
  initSchema in beforeAll under the leaked gateway → content_chunks.embedding
  becomes vector(1280) → inserting its hardcoded 1536-d basis vectors throws
  pgvector CheckExpectedDim.
- Victims B/C (shard 10 facts-backstop-gating, shard 2 facts-anti-loop):
  put_page's importFromContent embeds by design (embed failure PROPAGATES,
  Codex C2). Under a leaked fake-key gateway the embed step 401s and put_page
  returns isError → the backstop assertions fail.

My branch's shard re-partition (added test files + weight changes) merely
co-located leakers with victims; the hazard was latent.

Fixes (root cause + self-sufficient victims):
- test/search/rerank.test.ts (the shard-5 leaker): add afterAll(resetGateway).
  Its stub omits embedding_model, so it fell back to the ZE/1280 default;
  now it restores the empty slot so the preload re-pins legacy for the next
  file.
- test/search/searchvector-maxpool.test.ts: pin configureGateway(openai/1536)
  in beforeAll BEFORE initSchema (initSchema runs before any preload
  beforeEach, so it can't rely on the inherited slot).
- test/facts-backstop-gating.test.ts + test/facts-anti-loop.test.ts: reset
  the gateway in beforeEach so put_page's embed is a graceful no-op; reverted
  anti-loop from the serial quarantine back into the matrix (the serial move
  was the wrong fix for a gateway-state problem).

Validated deterministically: a non-resetting leaker that poisons the gateway
to ZE, run first in one bun process, no longer breaks any of the three
victims (14/14 pass). verify 29/29, typecheck clean, isolation lint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-05-30 12:07:38 -07:00
Garry TanandClaude Opus 4.7 ffac8ce0f4 v0.41.27.0 fix: withRetry self-heals on null singleton + facts:absorb drain + disconnect audit (closes #1570) (#1608) 
* merge master: rebump v0.41.25.0 → v0.41.27.0 (queue collision)

Master shipped v0.41.25.0 (#1538 batched sync deletes) and v0.41.26.0
(#1571 dream --source fix) while this branch was in flight. Conflict
resolution rebumps to the next available slot.

- VERSION: 0.41.25.0 → 0.41.27.0
- package.json: synced
- CHANGELOG.md: my v0.41.27.0 entry placed above master's v0.41.26.0
  and v0.41.25.0; in-entry version references updated 0.41.25.0 →
  0.41.27.0 and forward-references bumped to v0.41.28+.
- TODOS.md: kept master's v0.41.20.x section + my v0.41.27.0+ follow-ups

No source-file conflicts during the merge.

* feat(diagnostics): db-disconnect audit + doctor surface (v0.41.27.0)

Instruments every db.disconnect() and PostgresEngine.disconnect() call
with a JSONL audit record so the next user-reported #1570 cycle gives
us the offender's caller stack instead of the symptomatic
"No database connection" error.

Audit shape (~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl):
  {ts, engine_kind, connection_style, caller_stack[], command, pid}

- src/core/audit/db-disconnect-audit.ts (NEW): the audit writer,
  built on the v0.40.4.0 createAuditWriter cathedral. Captures a
  6-frame stack via new Error().stack so the offender is readable
  without spending stderr noise.
- src/core/db.ts: logDbDisconnect call at the top of disconnect()
  (best-effort; never blocks the real teardown).
- src/core/postgres-engine.ts: same instrumentation in
  PostgresEngine.disconnect() — distinguishes 'module' vs 'instance'
  connection_style so we can tell legitimate worker-pool teardowns
  apart from the load-bearing module-singleton class.
- src/commands/doctor.ts: extends batch_retry_health to surface
  24h disconnect count + most-recent caller stack. Warns when the
  caller frame isn't a known CLI-exit frame (e.g. cli.ts's finally
  block at the end of an op-dispatch). This is the diagnostic that
  tells v0.41.28+ where to apply the real ownership fix.
- test/db-disconnect-audit.test.ts: unit coverage for the audit
  writer + caller-stack capture + JSONL shape.
- test/e2e/db-singleton-shared-recovery.test.ts: real-Postgres
  regression that exercises the singleton-null path end-to-end.

Refs #1570

* feat(retry): self-heal on null singleton — closes #1570 symptom (v0.41.27.0)

withRetry gains an opt-in reconnect callback that fires between the
isRetryableConnError classification and the inter-attempt sleep.
PostgresEngine.batchRetry injects this.reconnect() — race-safe via
the existing _reconnecting guard, handles module and instance pools.

Closes the production loss reported in #1570: dream cycles on Supabase
no longer drop ~150 link rows per cycle when the singleton goes null
mid-batch. The retry now rebuilds the connection between attempts so
the second try has somewhere to write to.

- src/core/retry.ts: WithRetryOpts gains `reconnect?: () => Promise<void>`.
  Awaited in the catch branch. onRetry is also now awaited (back-compat-
  safe: every existing in-tree caller is a sync arrow). Reconnect
  failures propagate as the real cause — replaces the symptomatic
  "No database connection" error with whatever the connect() throw
  was, so operators see the truth.
- src/core/postgres-engine.ts:batchRetry — injects
  `reconnect: () => this.reconnect()`. Covers all 9 batch-retry call
  sites (addLinksBatch, addTimelineEntriesBatch, upsertChunks, plus
  the 6 caller-supplied auditSite labels in extract / sync / reindex).
- test/core/retry-reconnect.test.ts: 8 hermetic cases pinning the
  contract — reconnect fires before sleep, only on retryable errors,
  back-compat when omitted, signal-aborted bypasses reconnect,
  onRetry is awaited, full success path end-to-end.

The deeper bug (who's calling disconnect mid-cycle) is left
unaddressed in this commit by design — the diagnostic instrumentation
in the prior commit will tell us in the next production run.

Refs #1570

* feat(facts): drainPending() + CLI await before disconnect (v0.41.27.0)

Closes the silent 'No database connection' tail-end errors after
gbrain capture / put_page: the facts:absorb fire-and-forget queue
sometimes outlived the CLI process's connection lifetime, so absorb
attempts after engine.disconnect() landed in stderr as the
GBrainError shape.

- src/core/facts/queue.ts: new drainPending({timeout: 1000}) method
  distinct from shutdown(). Stops accepting new enqueues, awaits
  in-flight settle, bounded by timeout, returns count of unfinished.
  Semantically different from shutdown() (which aborts in-flight)
  so the symptom — drop work that hasn't started yet but let
  in-flight work finish — matches what CLI exit actually needs.
- src/cli.ts: op-dispatch finally block awaits the drain BEFORE
  engine.disconnect(). Bounded 1s. Opt-out env GBRAIN_NO_FACTS_DRAIN
  for callers that don't enqueue (keeps fast-exit paths fast).
  Mirrors the v0.41.8.0 awaitPendingLastRetrievedWrites pattern.
- test/facts-queue-drain-pending.test.ts: 6 hermetic cases — empty
  drain returns immediately, single in-flight settles, timeout
  bounds wait, shutdown-after-drain is idempotent, post-drain
  enqueues are dropped, signal-aborted skips waiting.

Refs #1570

* docs: update project documentation for v0.41.27.0

README.md: added troubleshooting entry for the v0.41.27.0 retry-reconnect
+ facts:absorb drain fix (closes #1570), pointing operators at
`gbrain doctor --json` to find the offending disconnect caller.

CLAUDE.md: extended `src/core/retry.ts` entry with the new optional
`reconnect` callback (v0.41.27.0); added two new Key Files entries for
`src/core/audit/db-disconnect-audit.ts` (the diagnostic half of the
"instrument first, fix later" pivot) and `FactsQueue.drainPending`;
extended `doctor.ts:checkBatchRetryHealth` entry with the in-place
extension that surfaces 24h disconnect-call count.

llms-full.txt: regenerated to absorb CLAUDE.md edits.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: rebump v0.41.27.0 → v0.41.28.0 (queue collision with #1573)

Master shipped v0.41.27.0 (#1573 git-aware sync_freshness) claiming the
same slot. Rebump to the next available version.

- VERSION + package.json → 0.41.28.0
- CHANGELOG.md: my entry header + in-entry refs 0.41.27.0 → 0.41.28.0
- TODOS.md: my #1570 follow-up section header + body refs bumped

* test: pin gateway in put-page-provenance + embedding-dim-check (CI shard fix)

Both files failed on CI shards 1 and 8 under the cross-file gateway-state
leak class (CLAUDE.md "Test-isolation lint and helpers"). The v0.41.28.0
merge reshuffled the weight-based shard bin-packing, landing a
gateway-mutating sibling ahead of these two victims in the same `bun test`
process.

Mechanism:
- put-page-provenance: put_page embeds via the gateway. A sibling left
  the gateway configured with OpenAI + the CI placeholder `sk-test`
  (captured at configureGateway time, survives the withEnv restore as
  cached gateway state). put_page's embed then fired against live OpenAI
  and 401'd. The bunfig legacy-embedding preload's beforeEach only
  re-applies legacy when the gateway was RESET — it does NOT correct a
  sibling that configured a different LIVE config.
- embedding-dim-check: initSchema builds the content_chunks vector column
  at the gateway's configured dim. A sibling leaking ZE/1280 made the
  column 1280-d, so `expect(dims).toBe(1536)` failed.

Fix (victim-side pinning, the escape hatch the preload documents):
- Both: configure the gateway explicitly in beforeAll BEFORE initSchema
  (OpenAI/1536), resetGateway() in afterAll so neither leaks onward.
- put-page-provenance also stubs the embed transport via
  __setEmbedTransportForTests so embed is deterministic and offline; a
  dummy OPENAI_API_KEY is supplied in the gateway env because
  instantiateEmbedding builds the OpenAI client (key check) BEFORE the
  stubbed transport is reached — the stub then intercepts the actual
  call so the key never leaves the process.

Verified: CI shards 1 (1337 pass) + 8 (905 pass) green with
OPENAI_API_KEY unset, plus adversarial sibling orderings (gateway.test /
doctor-ze-checks preceding). Typecheck + check-test-isolation clean.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-28 19:04:48 -07:00
Garry TanandClaude Opus 4.7 127842e9ef v0.41.22.1 feat: brainstorm/lsd judge fixes (closes #1540 end-to-end) (#1562) 
* feat(core): add splitProviderModelId centralizer for pricing-side parsing

New pure helper in src/core/model-id.ts that splits provider:model,
provider/model, and bare model strings into a {provider, model} pair.
Defensive contract: null/undefined/empty/whitespace returns
{provider: null, model: ''}.

Will be wired into the 5 pricing/budget sites in the next commit.
Named splitProviderModelId (not parseModelId) to avoid the in-project
collision with the gateway-side src/core/ai/model-resolver.ts:parseModelId
which has a different bare-name contract.

Pinned by 16 cases in test/model-id.test.ts covering all separator
forms plus defensive + edge inputs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(gateway): accept slash-form provider id in model-resolver

src/core/ai/model-resolver.ts:parseModelId now accepts both
provider:model (colon) and provider/model (slash) forms. Colon wins
when both separators present so OpenRouter nested ids like
openrouter:anthropic/claude-sonnet-4.6 route as
{providerId: 'openrouter', modelId: 'anthropic/claude-sonnet-4.6'}.

Pre-fix: every gateway entry point (chat / embed / rerank) threw
AIConfigError 'missing a provider prefix' on slash form ids. That
meant CLI users running

  gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6

would still fail mid-judge with AIConfigError even after pricing
was relaxed to accept slash form. Closes the end-to-end bug class.

Bare names without ANY separator still throw — gateway routing
always needs an explicit provider. Existing tests pinning that
throw (test/ai/capabilities.test.ts:43) stay green.

Pinned by 10 cases in test/ai/model-resolver-slash.test.ts
including a resolveRecipe round-trip that slash and colon forms
land on the same recipe.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor: route 5 pricing/config sites through splitProviderModelId

Five sites had inline ':'-only provider-prefix splits that silently
missed slash-form ids. Centralizing through splitProviderModelId
closes the bug class:

  - src/core/anthropic-pricing.ts:estimateMaxCostUsd
  - src/core/budget/budget-tracker.ts:lookupPricing (closes the
    headline BudgetExhausted no_pricing failure on --max-cost +
    slash-form --judge-model)
  - src/core/eval-contradictions/cost-tracker.ts:pricingFor
    (legacy silent-Haiku fallback preserved per plan D9)
  - src/core/minions/batch-projection.ts (deleted bareModel inline
    helper; inlined splitProviderModelId at 2 call sites)
  - src/core/model-config.ts:isAnthropicProvider (silently fixed
    v0.31.12 subagent-guard bypass for slash-form Anthropic ids)

Test gates land together so any bisect step is green:

  - NEW test/anthropic-pricing.test.ts (7 cases including structural
    regression guard: every ANTHROPIC_PRICING key reachable via all
    three forms)
  - NEW test/eval-contradictions/cost-tracker-slash.test.ts (6 cases
    including legacy-Haiku-fallback pin)
  - EXTENDED test/batch-projection.test.ts (slash + double-separator
    cases)
  - EXTENDED test/model-config.serial.test.ts (2 slash-form
    isAnthropicProvider cases)
  - EXTENDED test/core/budget/budget-tracker.test.ts (2 slash + colon
    reserve() cases)

Behavior changes for slash-prefix ids only; bare and colon ids
unchanged.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(brainstorm): scale judge maxTokens with per-model output cap

Replace the hard-coded maxTokens: 4000 with computeJudgeMaxTokens
that scales with idea count and respects each model's actual output
cap.

Pre-fix: any judge call with 36+ ideas produced ~100 tokens/idea of
JSON that got truncated mid-output. parseJudgeJSON threw, orchestrator
surfaced judge_failed: true, all ideas saved unscored. Verified
failure mode on 72-idea fixture: 0/72 passing before, 39/72 after.

Formula: min(modelCap, max(LEGACY_MIN_MAX_TOKENS, ideaCount*150+500))

Named constants extracted at top of judges.ts:

  - TOKEN_BUDGET_PER_IDEA = 150 (1.5x headroom over observed ~100/idea)
  - TOKEN_BUDGET_ENVELOPE = 500 (JSON wrapper)
  - LEGACY_MIN_MAX_TOKENS = 4000 (pre-fix floor preserved for 1-idea)
  - MAX_OUTPUT_TOKENS_CEIL = 32_000 (fallback when model unknown)
  - ANTHROPIC_OUTPUT_CAPS (per-model: Opus 4.7 = 32K, Sonnet 4.6 /
    Haiku 4.5 = 64K, legacy 3.5 = 8K)

When the caller passes no modelOverride, the cap routes through the
gateway's actual configured chat model via getChatModel() so the
formula matches what chat() will use, not whatever the override
hints at. Pre-fix the undefined-override case fell back to 32K even
if the configured default was a legacy 8K model.

Pinned by 16 cases in test/brainstorm/judges-maxtokens.test.ts:
formula at 1/10/36/96/200/300 ideas, per-model cap binding (Haiku 3.5
8K, Opus 4.7 32K, Sonnet 4.6 64K), and integration via runJudge with
a stubbed chatFn that captures ChatOpts.maxTokens.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: bump version and changelog (v0.41.21.0)

Brainstorm judge fix-wave: closes #1540 end-to-end. parseModelId
centralizer + gateway resolver slash-form acceptance + per-model
maxTokens cap.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: update project documentation for v0.41.21.0

CLAUDE.md: add v0.41.21.0 annotations to brainstorm/judges + model-config
entries; add new key-files entry for src/core/model-id.ts (the shared
splitProviderModelId centralizer) and src/core/ai/model-resolver.ts
slash-form extension.

README.md: add user-facing callout for the brainstorm judge_failed +
slash-form pricing fix, mirroring the v0.41.19.0 callout shape.

llms-full.txt: regenerated to absorb the CLAUDE.md + README changes
(passes test/build-llms.test.ts drift guard).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-27 07:16:30 -07:00
Garry TanandClaude Opus 4.7 5d42f3295e v0.41.22.0 feat: type-unification cathedral — 94 types → 15 canonical (closes #1479) (#1542) 
* Merge branch 'master' into garrytan/type-taxonomy-unification

Resolve VERSION, package.json, CHANGELOG conflicts with v0.41.22.0
on top, preserving master's v0.41.19.0 entry below.

* feat: v0.41.22.0 type-unification cathedral — collapse 94 types to 15 (closes #1479)

Ships gbrain-base-v2 as the new install default (15 canonical types: 14
+ note catch-all) and the unify-types PROTECTED Minion handler that
runs the gbrain-base→v2 migration end-to-end on existing brains.

What this delivers:
- gbrain-base-v2.yaml standalone schema pack (no extends:) with 14
  canonical page_types + 9 cluster mapping_rules + catch-all sentinel
- 3 new schema-pack primitives: runRetypeCore (chunked UPDATE with
  legacy_type stamping), runPageToLinkCore (edge-shaped pages →
  link rows), runPageToAliasCore (concept-redirect → slug_aliases)
- rewriteLinksBatch for N-pair atomic FK rewrite
- Migration v104 slug_aliases table (forward-bootstrap probed on both
  engines for safe upgrade chain)
- New engine method resolveSlugWithAlias(slug, sourceOrSources) on
  both Postgres + PGLite with multi-source ambiguity warning
- inferTypeAndSubtypeFromPack overload + subtypes: + mapping_rules:
  + migration_from: schema-pack manifest extensions
- findPackSuccessors version-range walker (1.x / 1.0.x / exact match)
- expandTypeFilter for --type back-compat (D14): legacy aliases route
  through mapping_rules → canonical+subtype before the SQL filter fires
- 3 new onboard checks: pack_upgrade_available, type_proliferation,
  dangling_aliases (source-scoped per F12)
- unify-types Minion handler (PROTECTED, manual_only via render.ts
  allowlist per D17): retype-explicit → retype-catch-all →
  page-to-link → page-to-alias → final sync → active-pack flip
- alias_resolved 1.05x post-fusion search boost stage; KNOBS_HASH_VERSION
  bumped 5→6 (one-time cache miss on upgrade, self-healing in TTL)
- ELIGIBLE_TYPES for facts extraction extended with v2 canonicals
  (codex F-ELIGIBLE: blocker not v0.43 follow-up)

Tests: 79 new unit/integration cases + 3 E2E cases covering all 9
production clusters end-to-end. 124-case verification on the cache-key
+ build-llms fixes. KNOBS_HASH_VERSION assertions updated in 3 tests.

Plan: ~/.claude/plans/system-instruction-you-are-working-transient-elephant.md
(16 locked decisions D1-D17, 12 baseline fixes F7-F21 absorbed from
codex outside voice).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: CI verify failures — system-of-record allow-comment + schema-unify manifest registration

Two CI failures on PR #1542:

1. check:system-of-record flagged page-to-link.ts:207 addLinksBatch as
   a direct write to a derived table. The call IS the reconcile surface
   for page_to_link mapping_rules — it converts edge-shaped pages into
   canonical link rows under the PROTECTED unify-types Minion handler,
   source-scoped, atomic per-rule. Added the canonical
   `// gbrain-allow-direct-insert: <reason>` comment on the same line.

2. check:resolver emitted 11 orphan_trigger warnings for `schema-unify`
   because the skill was added to skills/RESOLVER.md without a
   corresponding entry in skills/manifest.json. Added the registration
   under the existing skills[] array.

bun run verify: 28/28 checks pass locally.

* fix: CI test failures — schema-unify conformance + eligibility regression

Six test failures across shards 2 + 10 on PR #1542:

1. resolver.test.ts: round-trip parser requires frontmatter triggers to
   be quoted (`- "..."` or `- '...'`). schema-unify shipped with bare
   YAML strings; quoted the 10 triggers to round-trip correctly.

2. skills-conformance.test.ts (×3): schema-unify SKILL.md was missing
   the required Contract, Anti-Patterns, and Output Format sections
   that every conformant skill must declare. Added all three:
   - Contract: inputs / outputs / side effects / failure modes
   - Anti-Patterns: 5 DON'Ts including the autopilot trust boundary
   - Output Format: per-phase stderr lines + celebration summary +
     JSON envelope shape

3. facts-eligibility.test.ts (×2): the v0.41.22 ELIGIBLE_TYPES
   expansion added `concept` to the eligible list, but the existing
   test suite pins concept as rejected (it's `extractable: true` in
   the schema pack but the v0.41.11 contract documented this as
   "cosmetic on the backstop path because backstop uses hardcoded
   ELIGIBLE_TYPES"). Removed `concept` from the expansion; other v2
   canonicals (media, tweet, atom, analysis) stay. Comment updated
   to document the deliberate omission.

All 6 failing tests now pass locally (370/370 across the 3 affected
files). bun run verify: 28/28 checks green.

* fix: harden findPackSuccessors test against shard pollution

CI shard 8 reported 1 fail (1.00ms — too fast for any real loadActivePack
file I/O) on `finds gbrain-base-v2 as successor of gbrain-base@1.0.0`.
Local triple-run passes 9/9 in isolation.

Root cause: the existing afterEach reset clears the module-level pack
cache AFTER each test, but the FIRST test in the file inherits whatever
state sibling files in the same bun shard process left behind. With
24+ schema-pack tests in shard 8 (mutate, mutate-audit, best-effort,
registry-reload, manifest-v041_2, etc.) running before this file, the
first test can read a poisoned cache.

Fix: add `beforeEach(_resetPackCacheForTests)`. Two-sided reset
guarantees clean state regardless of file ordering within the shard.

bun run verify: 28/28 checks pass.

* fix: quarantine two flaky tests to serial runner

CI shard 1 + shard 8 each surfaced one intermittent failure:

shard 1: buildBrainTools > execute() on put_page with valid namespace
shard 8: findPackSuccessors > finds gbrain-base-v2 as successor

Both pass cleanly in isolation. Both are concurrency races against
shared in-shard state:

- brain-allowlist.test.ts shares a singleton PGLiteEngine across 18
  tests with a beforeEach DELETE FROM pages. With max-concurrency=4,
  two put_page tests can interleave their TRUNCATE + write phases,
  so the auto-link/extract sub-steps inside put_page race against
  the sibling test's DELETE.
- schema-pack-find-pack-successors.test.ts reads bundled YAML packs
  via loadActivePack. The module-level pack cache is shared across
  parallel tests in the same shard; the previous beforeEach reset
  helped but didn't fully isolate against concurrent file reads
  under CI load.

Fix per CLAUDE.md test-isolation lint rule R2 (concurrency-fragile
files belong in the .serial.test.ts quarantine): rename both files
to *.serial.test.ts. Serial runner picks them up at max-concurrency=1.
49/49 serial files pass locally. 28/28 verify checks pass.

* fix: quarantine embed-stale test to serial runner

CI shard 9 reported 6 failures, all from the embedStaleForSource describe
block, all ~120-150ms each — classic shared-engine concurrency race shape.
Passes 7/7 locally in isolation.

Root cause: embed-stale.test.ts shares a singleton PGLiteEngine across 7
tests with beforeEach resetPgliteState. Under bun's max-concurrency=4 in
the parallel shard, two tests can interleave their TRUNCATE + seedPage +
upsertChunks + embedStaleForSource flow, so one test's stale-chunk count
sees another test's mid-flight writes.

Same fix as brain-allowlist.serial.test.ts and
schema-pack-find-pack-successors.serial.test.ts: rename to *.serial.test.ts
so the serial runner picks it up at max-concurrency=1.

bun run verify: 28/28 checks pass. 7/7 embed-stale tests pass via serial.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-27 07:01:28 -07:00
Garry Tanandgarrytan-agents a7b79b66d4 feat: v0.41.19.0 Supavisor Retry Cathedral (#1537) 
Engine-level retry primitive that closes the v0.41.17 production incident
where ~3,000 wiki links + timeline entries were silently lost per dream
cycle on a 16K-page brain. Supavisor's circuit-breaker takes 5-10s to
recover; the prior single-500ms-retry shape couldn't survive it.

ARCHITECTURE
============

Retry becomes a data-primitive contract, not a caller responsibility.
postgres-engine.ts + pglite-engine.ts now self-retry inside addLinksBatch,
addTimelineEntriesBatch, and upsertChunks. Every caller — current AND
future — inherits retry-for-free. CI lint guard `scripts/check-no-double-retry.sh`
fails the build if anyone re-wraps an engine batch method (preventing
3×3=9 retry amplification on incomplete reverts).

CODEX-HARDENED DEFAULTS
=======================

BULK_RETRY_OPTS = {maxRetries:3, delayMs:1000, delayMaxMs:10000,
jitter:'decorrelated'}. Total worst-case wait ≈12s covers full Supavisor
recovery window. Decorrelated jitter (AWS-style uniform(base, prevDelay*3)
capped at maxDelay) replaces 'full' which allowed near-zero retries that
re-hit the still-recovering breaker.

AbortSignal threading from MinionWorker.shutdownAbort.signal through
engine method opts → withRetry → abortableSleep. SIGTERM aborts sleeping
retries instead of blocking deploys for up to delayMaxMs.

OBSERVABILITY
=============

`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl` records every retry event
(success-after-blip AND exhausted-retries). Built on the v0.40.4.0
audit-writer cathedral. Privacy posture: never logs slugs / page IDs /
content (mirrors shell-audit.ts).

`gbrain doctor` learns `batch_retry_health` check. Reads last 24h
(not 7d — codex H-9: avoid permanent noise from one historical blip).
Thresholds: ok (zero or <3 same-site), warn (>=3 same-site OR >=5
cross-site), fail (>=20 sustained breaker). Surfaces bad GBRAIN_BULK_*
env at startup (codex M-10). Corrupt-JSONL tolerant.

30-day audit pruning hooked into the dream cycle's purge phase (codex H-8
— implements the 'pruning convention' for real).

OPERATOR TUNING
===============

GBRAIN_BULK_MAX_RETRIES (int >= 0; 0 disables retries for debugging)
GBRAIN_BULK_RETRY_BASE_MS (int > 0)
GBRAIN_BULK_RETRY_MAX_MS (int >= base)

Bad values throw GBrainError with paste-ready fix hints at doctor startup,
not at first-retry mid-cycle.

VERIFICATION
============

- bun run verify: 28/28 checks green (includes 2 new lint guards:
  check-no-double-retry, check-batch-audit-site)
- bun run test: 11453 pass / 1 pre-existing flake (schema-cli.test.ts —
  confirmed by running on clean master, NOT introduced by this wave)
- bun run test:slow: 40/40 including new test/core/retry-stress.slow.test.ts
  (100 batches × 30% blip rate × decorrelated jitter, zero row loss)
- bunx tsc --noEmit: 0 errors

REVIEWS
=======

- CEO review (SELECTIVE EXPANSION): 4 cherry-picks proposed, 4 accepted
- Eng review (2 passes): 10 findings, 0 critical gaps, architectural
  pivot from per-site to engine-level wrap
- Codex independent review: 23 findings; 10 critical/high absorbed
  (decorrelated jitter, 12s backoff window, AbortSignal, idempotency
  proof, backfill unification, typed audit-site enum, doctor expiry
  thresholds, audit pruning, env validation at doctor startup)

PR #1523 closed and absorbed (@garrytan-agents original extract.ts fix
preserved via co-author trailer; 5 test cases moved to test/core/retry.test.ts
with assertions adjusted for the v0.41.19.0 BULK_RETRY_OPTS defaults).

Co-authored-by: garrytan-agents <noreply@anthropic.com>
 2026-05-26 23:15:44 -07:00
Garry TanandClaude Opus 4.7 cd8efee0ea v0.41.15.0 feat(sync): --timeout + --max-age + partial status (closes #1472 RFC) (#1506) 
* feat(sync): migration v98 last_refreshed_at + deleteLockRowIfStale helper

Schema foundation for v0.41.15.0's `gbrain sync --break-lock --max-age <s>`
flag. Adds `gbrain_cycle_locks.last_refreshed_at TIMESTAMPTZ` as the
heartbeat signal that distinguishes wedged-but-alive lock holders from
healthy long-running syncs that are actively refreshing.

Why last_refreshed_at not acquired_at: `withRefreshingLock` already bumps
`ttl_expires_at` every ~5 min while work runs, but leaves `acquired_at` at
the original timestamp. A 35-min media-corpus sync that's healthy has
`acquired_at` 35 min ago but `last_refreshed_at` 30 seconds ago. Using
acquired_at for --max-age would steal healthy locks; last_refreshed_at
correctly identifies only holders whose JS interval has stopped firing.

D-V4-1 rollout safety: migration v98 backfills `last_refreshed_at = NOW()`
(NOT `= acquired_at`) so pre-upgrade holders running the old binary get a
30-min protection window. After that window all pre-upgrade syncs are
either complete (lock released) OR genuinely wedged (--max-age does the
right thing). Documented as a known caveat in CHANGELOG.

D-V4-mech-4 SQL cast: deleteLockRowIfStale uses `$N * INTERVAL '1 second'`
not `$N::interval` (Postgres does not cast integer to interval the latter
way). Atomic DELETE keyed on (id, holder_pid, last_refreshed_at < NOW() -
$N * INTERVAL '1 second') RETURNING id, last_refreshed_at — no TOCTOU
between inspect + delete.

D-V4-mech-3 schema-snapshot parity: column added to all 3 snapshots so
fresh init paths (pglite-schema.ts, schema.sql) initialize correctly
without depending on the migration runner. schema-embedded.ts regenerated
via `bun run build:schema`.

Pinned by 13 PGLite cases in test/sync-break-lock-all.test.ts:
tryAcquireDbLock writes on INSERT, withRefreshingLock refresh bumps both
columns, inspectLock surfaces the new field, deleteLockRowIfStale refuses
fresh / breaks stale / safe on holder_pid mismatch / refuses NULL
(pre-v98). R1 + R6 regression invariants from the v4 plan.

Closes #1472 (RFC from @garrytan-agents) — schema foundation only;
performSync abort threading + CLI flags + consumer threading land in
follow-up commits in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): --timeout + --max-age + partial status + per-source AbortController

The CLI surface for v0.41.15.0. Wires `gbrain sync --timeout <s>` (graceful
self-termination) and `gbrain sync --break-lock --all --max-age <s>`
(cron-self-heal) end-to-end through `performSync`, `runOne`, `runBreakLock`,
and all `SyncResult.status` consumers.

Surface 1: `gbrain sync --timeout <s>`
  - New `SyncOpts.signal?: AbortSignal` threads through `performSync` →
    `withRefreshingLock` work callback → `performSyncInner`.
  - D-V3-1 honest scope: abort checks fire ONLY in pre-bookmark phases
    (pull, delete, rename, import). Extract + embed run to completion if
    reached. The `last_commit` bookmark write at sync.ts:1261 is the
    invariant boundary — partial CANNOT advance the bookmark because the
    abort checkpoints sit strictly before that write.
  - D-V3-2 per-iteration: abort check at top of every loop iteration
    (delete, rename, serial import, each parallel worker's while loop)
    matches the per-file granularity the existing loops already have.
  - D-V3-3 per-source AbortController: `--timeout --all` creates ONE
    controller inside runOne per source so each gets its own budget;
    NOT a shared global controller (which would starve later sources).
    try/finally + timer.unref() guarantees cleanup on throw.
  - D-V4-mech-7 pull error.cause: pullRepo wraps execFileSync errors in
    GitOperationError. The catch inspects e.cause.code === 'ETIMEDOUT'
    and e.cause.signal === 'SIGTERM' (NOT the top-level error) to
    distinguish timeout (partial reason='pull_timeout') from ordinary
    pull failure (existing warn-and-continue, R2 invariant preserved).

Surface 2: `gbrain sync --break-lock [--all] [--max-age <s>]`
  - Drops the --all refusal at sync.ts:1610. When combined with --all,
    runBreakLock iterates every active source and prints per-source verdict.
  - --max-age routes through the new deleteLockRowIfStale helper from
    db-lock.ts (atomic age-gated DELETE; no TOCTOU). Healthy refreshing
    holders survive by construction; only wedged-but-alive holders trip.

D-V3-5 partial-status consumer threading (conservative posture matching
blocked_by_failures):
  - printSyncResult: new `case 'partial':` arm reports filesImported +
    reason; tells operator to re-run to continue.
  - manageGitignore (both single-source and parallel runOne sites,
    plus watch mode): excludes partial from the gate. A partial sync's
    db_only path set isn't fully reconciled.
  - Auto-embed-backfill enqueue inside runOne: excludes partial. The
    next clean sync will re-walk and re-decide.

CLI flag parsing (T16):
  - parseDurationSeconds in sync-concurrency.ts: accepts 60s/10m/1h/bare
    int; rejects 0/negatives/decimals/garbage. Names the failing flag in
    the error message.
  - --timeout requires --source OR --all (validation rejects bare
    `gbrain sync --timeout`).
  - --max-age requires --break-lock; mutually exclusive with
    --force-break-lock.

Coverage:
  - 15 unit cases (test/sync-timeout.test.ts) pin parseDurationSeconds +
    SyncResult union additivity.
  - 2 E2E cases (test/e2e/sync-parallel.test.ts) pin the abort-mid-import
    contract against real Postgres: status='partial', last_commit
    unchanged, filesImported bounded.

Closes #1472 (RFC from @garrytan-agents) — CLI surface; schema foundation
landed in the previous commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(heavy): sync_timeout_rescue.sh reproducer for the cron-cascade

10K-page seed × 4 sources × deliberately tight --timeout × 3 sequential
cron emulations. Asserts every source reaches `last_commit === HEAD`
within 3 waves. Proves the v0.41.15.0 fix breaks the cascade the
PR #1472 RFC documented.

Workload (tests/heavy/_sync_timeout_rescue_workload.ts) is PGLite-only
because the PGLite engine forces serial sync internally (parallelEligible
excludes it). The parallel-fan-out + per-source AbortController case
lives in test/e2e/sync-parallel.test.ts against real Postgres. This
heavy test pins the contract that matters for cron: aborts → partial
returns → next wave content_hash-short-circuits + makes new progress.

Smoke-tested locally at PAGES=50 WAVES=2 TIMEOUT_SECONDS=2: every
source converges within 2 waves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.41.15.0): CHANGELOG + README + TODOS + version bump

Bumps VERSION + package.json to 0.41.15.0 (next slot after master's
v0.41.14.0). CHANGELOG entry leads ELI10 per gstack voice rules and
documents the 3 intentional honest gaps:
  1. --timeout covers pull + delete + rename + import only; extract +
     embed run to completion (D-V3-1 honest scope).
  2. First 30 min after migration v98, --max-age cannot identify wedged
     pre-upgrade holders (D-V4-1 rollout trade-off).
  3. Full-sync triggers (first sync, --full, chunker-version rewalk)
     don't respect --timeout yet (deferred to v0.42+).

README troubleshooting section: paste-ready cron pattern with shell
timeout(1) for OS-level process isolation + gbrain's --timeout for
graceful self-termination half-a-minute earlier.

TODOS.md: v0.42+ entries for subprocess fan-out (revisit if shell
timeout(1) proves insufficient), full-sync --timeout coverage via
AbortSignal in runImport, and runFactsBackstop microtask-queue
process-alive caveat.

llms-full.txt regenerated via `bun run build:llms`.

Closes #1472 (RFC from @garrytan-agents). Credit to @garrytan-agents
in the CHANGELOG for surfacing the production cron-failure data that
motivated the work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test-isolation): rewrite JSDoc to not match mock.module() lint regex

scripts/check-test-isolation.sh greps for the literal string `mock.module(`
to flag top-level module mocks (R2 rule — top-level mocks leak across files
in the shard process). The regex doesn't know about comments, so my two new
test files tripped the lint with JSDoc lines literally describing the rule:

  test/sync-timeout.test.ts:11   "* `mock.module()` (R2). Engine ..."
  test/sync-break-lock-all.test.ts:15  "* mock.module(), no process.env ..."

Both files had ZERO actual mock.module() calls — only the comment text
matched. Rewrote both JSDocs to refer to "top-level module mocks" instead
of the literal token. Same meaning; doesn't trip the regex.

`bun run check:test-isolation` now passes (714 non-serial unit files
scanned). `bun run verify` clean (22/22 checks pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 16:58:18 -07:00
Garry TanTobiaskohai-utClaude Opus 4.7justemujustemuecat2010
 		dab441f59f v0.41.4.0 wave: local providers + cross-platform stdin + gateway-routed dream judge (6 community PRs) (#1377) 
* fix(cli): use fd 0 instead of '/dev/stdin' for cross-platform stdin reads

`readFileSync('/dev/stdin', 'utf-8')` works on Unix but fails on Windows
(Git Bash, PowerShell, cmd) with `ENOENT: no such file or directory,
open '/dev/stdin'`. Windows doesn't expose `/dev/stdin` as a filesystem
path.

Reading file descriptor 0 directly (`readFileSync(0, 'utf-8')`) is the
documented Node.js idiom and works on every platform. No behavior change
on Unix — same syscall path, same semantics.

Repro on Windows before the fix:
  echo "test" | gbrain put my-page
  ENOENT: no such file or directory, open '/dev/stdin'

After: round-trip put/search/delete works on Windows Git Bash.

* v0.40.6.1 feat: llama-server reranker — local Qwen3 / self-hosted ZE via llama.cpp

Adds local reranker support so users can point gbrain's reranker call at their
own llama.cpp server instead of ZeroEntropy's hosted API. One new recipe
(`llama-server-reranker`), a `path?: string` + `default_timeout_ms?: number`
extension on `RerankerTouchpoint`, env passthrough wiring, budget-tracker
`FREE_LOCAL_RERANK_PROVIDERS` set so `--max-cost` callers don't TX2 hard-fail on
local rerank, and a doctor-probe divergence fix (probe and live search now read
the same `search.reranker.model` path via `loadSearchModeConfig` + `resolveSearchMode`).

ZE-hosted users are unchanged. Voyage / Cohere / vLLM rerankers stay out of
scope — different wire shapes need adapter hooks designed against their actual
shapes in a follow-up plan.

Verification:
- `bun run verify` (typecheck + 13 pre-checks): clean
- `bun run check:all` (15 historical checks): clean
- 107/107 expect() calls pass across 5 affected test files
- /codex review against the full diff: GATE PASS (caught one [P2] /v1 path
  doubling bug pre-merge; fixed by changing recipe path to leaf `/rerank`)
- Claude adversarial subagent: 7 net-new findings filed as v0.40.7+ TODOs
  (none currently exploitable; hardening for future contributor traps)

Test surface (107 cases, 5 files):
- test/ai/rerank.test.ts: path override (exact URL match), default_timeout_ms
  honored, empty models[] accepts any id, ZE regression
- test/ai/recipe-llama-server-reranker.test.ts: recipe shape regression guard
  + base_url + path concat assertion (codex-caught /v1/v1/ regression)
- test/search-mode.test.ts: timeout precedence chain (per-call > config >
  recipe > bundle), ZE no-recipe-default regression, unknown provider fallthrough
- test/models-doctor-reranker.test.ts: divergence-fix helper across DB-plane
  read, mode default, disabled, override, DB-error graceful fallback
- test/core/budget/budget-tracker.test.ts: free-local rerank pricing + arbitrary
  model id + chat-kind TX2 hard-fail preserved

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: post-ship documentation sync

* docs: index docs/ai-providers/ in llms.txt (zeroentropy + llama-server-reranker)

The hand-curated llms-config.ts doc map never included docs/ai-providers/, so
both zeroentropy.md (since v0.35.0.0) and the new llama-server-reranker.md were
invisible to the AI-facing llms.txt / llms-full.txt index. Adds an "AI providers"
section with both. Marked includeInFull: false (setup walkthroughs belong in the
index but would push the single-fetch bundle past FULL_SIZE_BUDGET) — same
treatment CHANGELOG.md gets.

Caught by the /ship document-release subagent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: recipe-aware embedding-provider check for local providers

doctor --remediation-plan and autopilot both judged the embedding
provider with a hosted-only key check, so a brain on ollama: or
llama-server: was reported "blocked" on a missing API key it never
needed, contradicting doctor --json's 100%-coverage health.

Extract a shared embeddingProviderConfigured() helper into
brain-score-recommendations.ts: empty auth_env.required (local
providers) is configured with no key; hosted providers check their
OWN required key. Both producers (doctor, autopilot) call it,
killing the DRY violation that caused the bug. Hosted brains with a
missing key still block.

* fix(budget): price local embed providers at $0

A --max-cost-bounded embed/reindex job configured for ollama: or
llama-server: TX2 hard-failed with no_pricing because
lookupEmbeddingPrice has no entry for local models. Add
FREE_LOCAL_EMBED_PROVIDERS (sibling to FREE_LOCAL_RERANK_PROVIDERS)
so a pricing miss on a local-inference provider returns $0 instead
of null. lmstudio/litellm intentionally excluded.

* feat(models): embedding reachability probe in gbrain models doctor

A down/misconfigured local embed server was invisible until first
embed. Add probeEmbeddingReachability() (mirrors the reranker probe):
a 1-input embed with a 5s abort timeout, classified via classifyError,
under a new 'embedding_reachability' touchpoint, gated on the
zero-network config probe returning ok first.

* fix: don't count config-plane voyage/google keys as configured

codex review caught a false positive: HOSTED_EMBED_KEY_CONFIG mapped
VOYAGE_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY to config fields, but
buildGatewayConfig only threads openai/anthropic/zeroentropy config
keys into the gateway env. A Voyage/Google brain with the key only in
config.json would be judged "configured" and dispatch an embed.stale
job that then fails auth at the gateway. Drop those two from the map so
the producer closures resolve them by env var only, matching what the
gateway can actually use. Pinned by a regression test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(dream): route significance judge through gateway.chat for multi-provider support

Replaces the hardcoded `new Anthropic()` client in the dream-cycle synthesize
phase with a gateway-routed JudgeClient adapter. Mirrors the v0.35.5.0 pattern
that closed #952 for runThink: construction-time provider/key probe returns null
on a clear miss (cheap pre-flight); the verdict loop wraps the chat call in
try/catch for AIConfigError mid-run.

Any provider with a registered gateway recipe (Anthropic, DeepSeek, OpenRouter,
Voyage, Ollama, llama-server, etc.) is now reachable via:

    gbrain config set models.dream.synthesize_verdict <provider>:<model>

The canonical config key `models.dream.synthesize_verdict` (per PER_TASK_KEYS
in src/core/model-config.ts) is used unchanged. The exported JudgeClient
interface signature is preserved for test-seam stability.

The original community PR (#1349) shipped a custom fetch adapter that
bypassed the gateway entirely. This reworked landing routes through the
canonical seam so future provider additions automatically benefit, and a
CI guard (T7) will land in this wave to prevent the bug class from
re-opening (the same one that bit src/core/think/index.ts before v0.35.5.0).

Co-Authored-By: justemu <206393437+justemu@users.noreply.github.com>

* test(dream): synthesize-gateway-adapter unit tests + R3 parsed-verdict parity

11 cases pin the gateway-routed JudgeClient adapter from T5:

- A1: makeJudgeClient returns null on missing Anthropic key (legacy short-circuit preserved)
- A2: returns a JudgeClient when chat provider is reachable
- A3: JudgeClient.create routes through gateway.chat (via __setChatTransportForTests)
- A4: ChatResult.text → Anthropic.Message.content[0].text mapping
- A5: empty text from gateway → graceful empty-text Anthropic.Message
- A6: non-AIConfigError from gateway propagates to caller (no swallow)
- A7: AIConfigError from gateway propagates as AIConfigError (caught per-transcript in production loop)
- A8: makeJudgeClient returns null on unknown provider prefix
- A9: returns a JudgeClient for non-anthropic providers without env-probing (delegates to gateway at call time)
- R3: parsed-verdict SEMANTIC parity — gateway-routed and legacy SDK-shape JudgeClients produce same {worth_processing, reasons} given identical canned LLM text
- R3 corollary: unparseable LLM output → both paths fall through to cheap-fallback verdict

Codex flagged byte-identical-Anthropic.Message as a meaningless gate; R3 is
parsed-verdict semantic parity instead. Mirror pattern of
test/think-gateway-adapter.test.ts for cross-site consistency with the
v0.35.5.0 runThink migration.

* ci: guard against direct Anthropic SDK construction in gateway-routed files

New scripts/check-gateway-routed-no-direct-anthropic.sh greps two guarded
files (src/core/cycle/synthesize.ts and src/core/think/index.ts) for
`new Anthropic()` constructor calls and runtime imports of @anthropic-ai/sdk.
Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay
allowed because both files use Anthropic.Message / .MessageCreateParamsNonStreaming
as adapter types.

Comment lines (starting with `//` or ` *`) are excluded so historical
references in JSDoc don't false-fire. Negative test in this commit's
verification confirms: injecting `new Anthropic()` into synthesize.ts
makes the guard exit 1 with a clear error pointing at the gateway adapter
pattern; reverting restores the OK state.

Wired into both `bun run verify` and `bun run check:all`. Closes the bug
class that bit synthesize.ts in PR #1349 (which would have shipped a
parallel fetch stack instead of routing through the canonical gateway).
The same class previously bit think/index.ts and was fixed structurally
in v0.35.5.0; this guard prevents either file from regressing.

Extend GUARDED_FILES in the script when migrating another file off
direct SDK construction.

* docs(put_page): point Windows / pipe-buffer users at gbrain capture --file

Extends the put_page op description (surfaced by `gbrain put --help`) with a
one-line pointer to `gbrain capture --file PATH --slug SLUG` for the file-
as-input use case. Capture (v0.39.3.0) is the canonical Windows-pipe-buffer
escape route: reads files as a Buffer first, scans the first 8KB for NUL bytes
to refuse binary content, decodes to UTF-8 only after the safety check, and
adds provenance write-through.

Lands the user-facing value the closed PR #1365 was reaching for, without
duplicating the CLI surface. Credits the original contributor.

Co-Authored-By: ecat2010 <90021101+ecat2010@users.noreply.github.com>

* test: R1+R2+R4 critical regression pins for the community-PR-wave landing

Per the wave's eng-review plan (IRON RULE — mandatory):

  R1 — get_page handler accepts calls without `content` param. Pre-wave
       PR #1365 landed its `!p.content → throw` check in the WRONG handler
       (get_page instead of put_page), which would have broken every read
       in the system. Pin: get_page MUST NOT require content + the schema
       carries no `content` or `file` param.

  R2 — put_page schema content stays `required: true`. PR #1365 also
       flipped `content` from required→optional in the schema. Pin: the
       contract stays at `required: true` + the closed PR's `file` param
       is NOT in the schema.

  R4 — Cross-platform stdin via fd 0 (PR #1325 regression pin). Source-grep
       asserts src/cli.ts uses `readFileSync(0, ...)` and NOT the legacy
       `readFileSync('/dev/stdin', ...)`. Belt-and-suspenders pattern
       assertions confirm the parseOpArgs branch shape (cliHints.stdin
       check, 5MB cap, isTTY gate) hasn't drifted.

R3 (gateway-adapter parsed-verdict parity) lives in the sibling file
test/cycle/synthesize-gateway-adapter.test.ts.

* test(e2e): update dream-synthesize no-key reason text + harden hermeticity

After T5's gateway-adapter rework, the "no API key" verdict text changed from
'no ANTHROPIC_API_KEY for significance judge' to
'no configured provider for verdict model: <model>' (broader + names the
actual model so the user sees WHICH provider failed). Update both assertions
that check the old text.

Hermeticity bug fix in the same commit: `withoutAnthropicKey` previously only
cleared the env var. After the rework, `makeJudgeClient` ALSO checks
`loadConfig().anthropic_api_key` (same hasAnthropicKey() pattern think/index.ts
uses since v0.35.5.0). If the developer running the test has the key set in
~/.gbrain/config.json, the test would behave non-deterministically. Fix:
override GBRAIN_HOME to a fresh tmpdir for the duration of the body, restore
on return (even on throw).

* test(e2e): pin verdict-loop AIConfigError catch from T5 rework end-to-end

Drives runPhaseSynthesize against a real PGLite engine with the gateway
chat transport stubbed to throw AIConfigError on every call (simulates a
revoked/misconfigured provider surfacing mid-run). Asserts:

  - Phase does NOT crash; converts the throw to a per-transcript verdict
    with worth=false and reasons[0] matching "gateway error: ...".
  - status='ok' so subsequent transcripts in the loop would continue
    being judged (not visible in 1-transcript test, but the loop shape is
    proven not to abort).

Pre-rework (T5), this code path didn't exist — judgeSignificance threw
directly to runPhaseSynthesize and crashed the whole phase. Pin so a
future regression that removes the try/catch fires loudly.

* docs(claude.md): annotate v0.41+ community-PR-wave changes

Two additions to the Key files section:

- src/core/cycle/synthesize.ts — appends a v0.41+ paragraph documenting
  the gateway-adapter rework (makeJudgeClient + AIConfigError catch loop +
  canonical config key + JudgeClient interface preserved + CI guard
  reference + test file references).

- scripts/check-gateway-routed-no-direct-anthropic.sh — new entry
  documenting the CI guard's contract, scope, and how to extend
  GUARDED_FILES when migrating another file off direct SDK construction.

CLAUDE.md drives /sync-gbrain and llms.txt generation; both need the
wave's annotations to land BEFORE the llms regeneration step (T10).

* docs(llms): regenerate llms.txt + llms-full.txt for v0.41+ wave

Refreshes the auto-generated llms.txt bundles to pick up the CLAUDE.md
annotations landed earlier in this wave (gateway-adapter synthesize.ts
+ check-gateway-routed-no-direct-anthropic.sh + the cherry-picked
llama-server-reranker recipe). Pinned by test/build-llms.test.ts.

* fix(providers): dynamic-width id column accommodates llama-server-reranker

v0.40.6.1 introduced `llama-server-reranker` (21 chars), which overflowed
formatRecipeTable's static 14-char PROVIDER column. When the id is longer
than the column, padEnd is a no-op — the row starts with the tier name
directly, no space delimiter. test/providers.test.ts 'each recipe appears
at most once' iterates every recipe and asserts at least one row starts
with `${id} ` or `${id}  `; with no space after `llama-server-reranker`,
the assertion fails and the recipe appears effectively missing from the
human-readable list.

Fix: compute column width dynamically as `max(14, max(id.length) + 1)` so
every id is followed by at least one space, regardless of length. Also
widens the separator rule to match. 14 stays as the floor so the existing
short-id rows (openai 6, ollama 6, anthropic 9, ...) keep their familiar
layout when llama-server-reranker isn't in the active recipe set.

10/10 cases in test/providers.test.ts pass after the fix.

* chore: pre-landing review polish — refresh models doctor tip + file embed timeout TODO

Two pre-landing review absorptions:

- `src/commands/models.ts:154` — the help-text tip said `gbrain models doctor`
  "spends ~1 token per model" but the wave added an `embed(['probe'])` call
  AND a reranker probe. Generalize to "spends a minimal request per configured
  chat/embed/rerank surface" so the cost expectation matches reality.

- `TODOS.md` — file a follow-up to widen `default_timeout_ms` from
  RerankerTouchpoint to EmbeddingTouchpoint so `probeEmbeddingReachability`
  doesn't hardcode 5000ms while the sibling reranker probe reads the
  recipe's configured timeout. Local CPU embedding endpoints (llama-server)
  hit the same cold-start curve as Qwen3-Reranker-4B; workaround today is
  "re-run the probe" per the existing JSDoc.

Other informational findings from pre-landing review either match
established patterns (no behavioral test for `probeEmbeddingReachability`,
matching `probeRerankerReachability`), are intentional choices documented
in JSDoc (the `as unknown as Anthropic.Message` cast), or are micro-perf
in non-hot paths (autopilot's 4 sequential `getConfig` awaits per
5-minute tick). All non-blocking.

* ci: tighten gateway-routed guard against import bypass shapes + honest JSDoc

Adversarial review caught two soft spots in the wave's new contracts:

1. `scripts/check-gateway-routed-no-direct-anthropic.sh` only matched the
   default-import shape `import Anthropic from '@anthropic-ai/sdk'`. A future
   contributor (or, more realistically, a future refactor) could bypass with:
     - `import { Anthropic } from '@anthropic-ai/sdk'`
     - `import { Anthropic as A } from '@anthropic-ai/sdk'`
     - `import * as Anthropic from '@anthropic-ai/sdk'`
     - `const x = await import('@anthropic-ai/sdk')`
   Tightened the regex to match ANY value-shaped import from the SDK module
   (excluding only the explicit `import type ... from '@anthropic-ai/sdk'`
   form which the adapter's Anthropic.Message return type needs). Added a
   second grep for dynamic imports. Verified all four bypass shapes now
   trigger the guard against synthesize.ts; type-only import still passes.

2. `synthesize.ts:makeJudgeClient` JSDoc claimed the adapter "tolerates the
   array-of-blocks shape for future flexibility" — but the mapping flattens
   ONLY text blocks; `tool_use`, `tool_result`, image blocks silently
   become empty strings. Today only `judgeSignificance` calls this and it
   only sends string content, so no behavior bug. But the comment was
   marketing future flexibility the code doesn't deliver. Narrowed to call
   out the silent-drop and say to extend the mapping if a future caller
   wires non-text content through.

Both wave-scope: the CI guard was added by the wave, the JSDoc was added
by the wave's T5 rework. Adversarial review caught them before merge.

* fix(models doctor): reranker probe timeout matches live search precedence chain

Codex Pass-9 adversarial review caught a probe-vs-production divergence:
production `hybridSearch` resolves reranker timeout via the full chain
(per-call > config > recipe > bundle) by going through
`loadSearchModeConfig + resolveSearchMode`, but `probeRerankerReachability`
was reading ONLY the recipe's `default_timeout_ms` — so an operator who
set `search.reranker.timeout_ms=1000` would see doctor wait 30s and report
"reachable" while production search timed out at 1s and fail-opened.
A higher configured timeout produces the opposite false failure (probe
gives up at 5s when production would have waited longer).

Fix: extract `resolveLiveRerankerTimeoutMs(engine)` parallel to the
existing `resolveLiveRerankerModel(engine)` — same precedence chain,
same DB-plane consistency posture. The probe now reads the SAME timeout
live search reads, on the same lookup path.

The codex P1 finding about `FREE_LOCAL_*_PROVIDERS` zero-pricing being
bypassable via redirected `LLAMA_SERVER_BASE_URL` is filed as a TODO under
community-pr-wave follow-ups — couples with the existing
FREE_LOCAL_PROVIDERS unification TODO so both close in one v0.41+ PR.

* ci(guard): handle mixed type+value imports + macOS BSD sed POSIX classes

Codex structured review [P3] caught a bypass in the freshly-tightened
gateway-routed guard:

  import { type Message, Anthropic } from '@anthropic-ai/sdk';
  new Anthropic();

The previous regex `^\s*import\s+[^t][^y]*from ...` was meant to exclude
`import type ...` but stops at the `y` in `type` inside the brace list,
silently allowing the value-import `Anthropic` through. Two fixes:

1. Replace the brittle regex-based type-exclusion with a clause-level
   parse: extract the brace-list specifiers, allow the import iff EVERY
   non-empty specifier is `type`-prefixed. Catches mixed-import bypasses
   (`{ type Foo, Bar }`) while keeping all-type braces (`{ type Foo, type Bar }`)
   passing. Default + namespace imports remain always-value-shaped.

2. Replace `\s` with POSIX `[[:space:]]` in the sed extract — macOS BSD sed
   doesn't honor `\s` in extended-regex mode (it silently no-ops the pattern
   so `specifiers` comes back empty and the script falls through to the
   default/namespace branch's wrong error message).

Hermetic 7-shape regression matrix now verifies every TypeScript import
shape against the expected ALLOW/BLOCK verdict; all 7 pass:
- ALLOW: `import type Anthropic from '...'`
- ALLOW: `import type { Foo } from '...'`
- ALLOW: `import { type Message, type Foo } from '...'`
- BLOCK: `import { type Message, Anthropic } from '...'`
- BLOCK: `import { Anthropic } from '...'`
- BLOCK: `import Anthropic from '...'`
- BLOCK: `import * as A from '...'`

Subshell-trap fix in the same commit: the previous "exit 1 inside while-pipe"
pattern doesn't propagate to the outer `$?` because the pipe spawns a
subshell. Switched to a tmpfile-flagged sentinel so the verdict survives
the subshell boundary cleanly.

* chore: bump version and changelog (v0.41.4.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(audit-writer): route log() to file matching event ts, not real-now

CI failure surfaced a time-dependent test flake in
`test/audit/audit-writer.test.ts` "returns events from current week,
filtered by ts cutoff" (added in v0.40.4.0 PR #1300). The test pinned
synthetic `now = 2026-05-22T12:00:00Z` (ISO week 21), logged 3 events
with synthetic ts values, then called `readRecent(7, now)` expecting
to find 2 events in window.

Root cause: `log()` ignored the caller-supplied `ts` for filename
routing and ALWAYS wrote to the file matching real-time-now's ISO
week. When real CI time crossed into 2026-W22 (this Monday), the
events went to W22's file but `readRecent` walked W21 + W20 → 0 hits.

Fix:
- `log()` parses `event.ts` (when provided) and routes to the file
  matching that ts's ISO week. Falls back to real-now when ts is
  missing or unparseable.
- No behavior change for production callers — none of the 5 audit
  consumers pass `ts` explicitly (rerank-audit, audit-slug-fallback,
  content-sanity-audit, graph-signals, supervisor-audit). The writer
  stamps real-now → both ts and filename use real-now → same file
  as before.
- Sibling test "honors caller-supplied ts override" also pinned a
  fixed ts and would have broken from the opposite angle (test
  read from `computeFilename()` default = real-now). Updated to
  read from `computeFilename(new Date(fixedTs))` so it asserts the
  per-row file routing the wave now provides.

22/22 audit-writer cases pass. Production callers (5 sites) unchanged.

Pre-existing on master since v0.40.4.0; surfaced when real time
crossed into a different ISO week than the test's synthetic now.
NOT introduced by this PR (#1377 community-PR-wave) — audit-writer
files aren't touched by the wave.

---------

Co-authored-by: Tobias <34135750+tobbecokta@users.noreply.github.com>
Co-authored-by: kohai-ut <chris@tincreek.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: justemu <noreply@github.com>
Co-authored-by: justemu <206393437+justemu@users.noreply.github.com>
Co-authored-by: ecat2010 <90021101+ecat2010@users.noreply.github.com>
 2026-05-25 10:39:09 -07:00
Garry TanandClaude Opus 4.7 af5ee1eb5a v0.40.8.1 docs: README rewrite + personal-brain + company-brain tutorials (#1345) 
* docs: rewrite README lead around search-vs-think differentiator

The current README opened with a generic "smart but forgetful" tagline that
buried the actual differentiator. Garry's 2026-05-23 X thread crystallized the
positioning: "Search gives you raw pages. Think gives you the answer." That
plus graph traversal plus gap analysis is what nobody else ships in one box.

Changes:
- README lead now leads with the search-vs-think frame, the "nobody else does
  this" claim, and the "strategic moat / so you don't lose context" framing.
- Collapsed five stacked "New in vX.Y.Z" paragraphs in the lead into one
  Recent Releases section after the install path, freeing the first viewport
  to be about what gbrain IS, not what shipped last week.
- New ## Search vs think section with side-by-side CLI example, gap-analysis
  explanation, and the find_trajectory + think compounding story.
- ORIGIN.md closing paragraph names think as the reason the brain is worth
  building.
- No em dashes used as connectors (per humanizer rules).
- All factual claims (page counts, benchmark numbers, version references)
  preserved verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: add v0.40.6.0 to README Recent Releases

Picked up sync --all + per-source locks + sources status dashboard from
the v0.40.6.0 merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): add visceral "what this looks like" before/after table

Pulled verbatim from BrainBench Cat 29 — same question, same brain, Haiku judge. Shows what a typical personal-knowledge brain (top-K vector retrieval, what MemPalace / Mem0 / Hindsight ship) returns vs what gbrain think returns. Search hallucinates three people who actually work at OTHER companies; think correctly identifies what's known + flags the gap. Score: search 1/10 vs think 9/10.

The before/after lands above Install so the reader sees concrete differentiation before they decide to install. Backs the abstract "search vs think" claim from the lead with a real receipt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): swap before/after example to verbatim Cat 29 receipt (Q2 ARR)

Per @garrytan — the prior example used the synthetic Q1 (employees of Horizon TECH 6) with paraphrased answer text. Replaced with the verbatim Cat 29 Q2 receipt: actual question (with the in-question typo that exists in the eval), actual truncated search-answer text from the JSON receipt, actual think-answer text with all three ARR readings + citations, and the actual Haiku judge verdicts pasted verbatim.

Also strips Mem0 + Hindsight references from the comparator phrasing — Mem0 is a YC company we don't want to single out, and Hindsight was a hackathon-stage project that never launched. The comparison phrasing now reads "MemPalace and most peer AI-memory stacks" — accurate without naming systems we shouldn't be benchmarking against.

The three things gbrain think did that a typical top-K retrieval cannot are listed below the table for readers who want the takeaway in plain English:
1. Caught the name typo (question said "Acme AI 0", brain has "Acme CO 0")
2. Walked the typed-claim Facts fence to build a chronological trajectory
3. Cited every claim

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): add "Your brain's shape (schema packs)" section

Per @garrytan — the README had nothing on schema packs (v0.38/v0.39
dynamic-schema cathedral). New section lands between "How to get data in"
and "Recent releases" so the narrative flow reads:

  what gbrain is → concrete example → install → query (search vs think)
  → get data in → schema packs (how the brain understands your shape)
  → recent releases → loop → capabilities → ...

The section opens by naming the differentiator out loud: "Most personal-
knowledge tools force one fixed layout: their idea of notes + people +
tags. Drop a Notion export or your own years-old Obsidian vault and the
agent doesn't know what your folders mean."

Three options surfaced:
  - gbrain-base (default, zero-config Garry layout)
  - gbrain-recommended (extends base with 13 more dirs)
  - your own pack via the schema detect → suggest → review-candidates
    three-command magical moment

Six representative CLI verbs shown verbatim. Closes with one paragraph
explaining the threading through every read/write path (parseMarkdown,
whoknows, extract_facts, search cache) and a one-line summary of the
7-tier resolution chain pointing at docs/architecture/schema-packs.md
for the full reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): de-brand "Think" in the lead; reframe as GBrain's brain layer

Per @garrytan — "we don't need to brand it Think, we want to say the think
command later but we don't lead on it!"

Changes:

- Lead sentence: "Search gives you raw pages. Think gives you the answer"
  → "Search gives you raw pages. GBrain gives you the answer, through a
  brain layer." The product is GBrain; think is the CLI verb that runs
  the brain layer, introduced later.

- Two-bullet differentiator list: the "gbrain think" bullet now leads with
  the capability ("A synthesis layer that gives you the actual answer.")
  rather than the CLI command. The bullet body still names what the layer
  does (synthesized prose, citations, gap analysis).

- Strategic-moat paragraph: "`gbrain think` is what makes the moat
  usable" → "The brain layer is what makes the moat usable."

- "What this looks like" table header: "GBrain `think`" →
  "GBrain's brain layer (one synthesized answer, run via `gbrain think`)".
  CLI command stays in the cell so the example is reproducible; the
  framing leads with what it IS, not the verb.

- Three-bullet takeaway under the table: "Three things `gbrain think`
  did" → "Three things the brain layer did". Aggregate sentence: "gbrain
  think averages 5.60/10" → "GBrain's synthesis layer averages 5.60/10".

- Section heading: "## Search vs think" → "## Two ways to query your
  brain". The section still introduces `gbrain search` and `gbrain think`
  as the two CLI verbs side by side; the heading no longer brands "think"
  as the thing.

The "think command" comes through naturally where it appears as a CLI
example. The PRODUCT is GBrain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): switch to first-person voice (Garry speaking directly)

Per @garrytan — drop "Built by..." third-person framing and write in
first person.

Edits:
- Lead credibility paragraph: "Built by the President and CEO of Y
  Combinator to run his actual AI agents" → "I'm Garry Tan, President
  and CEO of Y Combinator. I built GBrain to run my own AI agents."
  Subsequent sentences switch "his deployments" → "my deployments",
  "the agent ingests... you wake up smarter" → "my agent ingests...
  I wake up smarter — and so will you" (the closing "and so will
  you" connects Garry's experience to the reader's).
- Compounding paragraph: "As Garry's personal agent gets smarter, so
  does yours" → "As my personal agent gets smarter, so does yours."
- Schema packs section: "the layout used by Garry's production brain"
  → "the layout my production brain uses".
- License + credit: "Built by Garry Tan to run his OpenClaw and
  Hermes deployments — the production brain behind his actual AI
  agents" → "I built GBrain to run my OpenClaw and Hermes deployments
  — the production brain behind my AI agents."

The whole top of the README now reads as Garry talking directly to
the reader about what he built and why.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): DRY the lead — drop redundant "through a brain layer" + repeat GBrain

"GBrain gives you the answer, through a brain layer. GBrain is the brain
layer your AI agent has been missing..." — two GBrains, two brain layers.
Tightened to: "GBrain gives you the answer. It's the brain layer your AI
agent has been missing — the only one that does synthesis, graph
traversal, and gap analysis in one box."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): frame GBrain as a company brain too, link to YC RFS

Per @garrytan — GBrain is now usable as a company brain (federated sync,
OAuth scoping, Cat 22 source isolation), and YC just put company-brain
on its Request for Startups.

Added a paragraph after the personal-brain lead that names the three
v0.34+ features that make multi-user safe (federated sync, per-source
OAuth scoping, the Cat 22 leak-free source isolation), then links to
https://www.ycombinator.com/rfs#company-brain with a one-line pitch:
"if you're building in that space, you might as well build on this."

The framing carries forward the personal-brain story while opening the
aperture: GBrain works for one person (Garry's production deployment)
AND for a team (per the features Cat 22 just verified).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): rewrite company-brain paragraph in plain English

@garrytan caught me writing internal eval-suite jargon ("Cat 22 proves
source isolation is leak-free across hybrid search, listPages, getPage,
and federated reads") in a paragraph aimed at someone deciding whether
to use GBrain. Rewritten in plain English:

  "Each person on the team gets their own slice of the brain, scoped
  by login. When you query, you only see what you're allowed to see —
  never another person's notes, never another team's data. We fuzz-
  tested this across every way you can read the brain (search, list,
  lookup, multi-source reads) and got zero leaks."

Same factual content, zero internal vocabulary. The "Cat 22" name
belongs in the benchmark page, not the front-door README.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(tutorials): add company-brain tutorial (Diataxis tutorial quadrant)

End-to-end walkthrough for setting up GBrain as a multi-user company
brain. Audience: founder / CTO / head of ops at a 10-50 person company
who has heard about GBrain (from the YC RFS company-brain page or my
tweets) and wants to set it up as their team's shared institutional
memory. ~3700 words, written for a learner with zero prior gbrain
knowledge.

Twelve parts walk the reader from "I've never run gbrain" to "three
teammates each query the brain through their own AI agent and see the
correctly scoped answer":

  1. The mental model (personal brain vs company brain, federated
     sources, OAuth scoping, what you get)
  2. Prerequisites table (Postgres, embedding key, Anthropic key, git
     repo, Bun, host machine + cost projection)
  3. Install + Postgres + API keys + doctor verify
  4. Create three sources (shared / customers / internal) + sync
  5. Spin up HTTP MCP server with --bind 0.0.0.0 + --public-url
  6. Register one OAuth client per teammate with --source +
     --federated-read
  7. Verify scoping works (alice can't see internal, bob can't see
     customers)
  8. Connect each teammate's AI agent via thin-client install
  9. First real `gbrain think` query showing sourced + synthesized +
     gap-analysis answer
  10. Operating the brain (autopilot, doctor --remediate, sources
      status, admin dashboard)
  11. Cost + speed expectations from the v0.40.6.0 benchmark
  12. Common gotchas + troubleshooting

Voice:
- First-person Garry in intro / motivation paragraphs
- Zero internal jargon. No "Cat 22", "P@5", "knobsHash", "RRF", "MRR"
- Plain English throughout
- No em dashes used as connectors (zero in final draft)
- "Brain layer" / "synthesized answer" framing, not "Think" branding
- No Hindsight, no Mem0 (per repeated session feedback)
- Every example uses placeholder names (alice-example, bob-example,
  acme-co)

Cross-linked from:
- README.md company-brain paragraph
- docs/INSTALL.md (under the migrate --to supabase block)
- docs/architecture/topologies.md (See also section)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): drop version chatter, add Tutorials section, expand tutorial roadmap

Per @garrytan: the README should read as the current docs written for
people who have never known GBrain before. Versions are what the
CHANGELOG is for.

Version-chatter sweep across the README:

- Killed the entire ## Recent releases section. That's a changelog
  summary, not docs. CHANGELOG.md owns it.
- Stripped "(v0.38+)" from the ## How to get data in heading.
- Rewrote "(the v0.38 put_page write-through plumbing)" as
  "(the database and on disk in one move)" — describes WHAT happens,
  not WHEN it shipped.
- Stripped "(The legacy gbrain skillpack install managed-block model
  was retired in v0.36.0.0; run gbrain skillpack migrate-fence once if
  you're upgrading from an older release.)" from the skillpack
  paragraph. Upgrade history goes in the CHANGELOG.
- Stripped "New in v0.40.4.0:" from the hybrid-search graph-signals
  description. Just describes what the feature does.
- Stripped "As of v0.37," from the embedding-provider auto-detect
  paragraph in the Troubleshooting section.
- Stripped "the embedding + reranker stack that became the v0.36.2.0
  default" → "ships as the default" in the License + credit section.
- Stripped "in Cat 29" from the synthesis-benchmark sentence (same
  internal-jargon class as Cat 22 which @garrytan called out earlier).

Replaced ## Recent releases with ## Tutorials:

- Links to the one shipped tutorial (company-brain.md) with a
  one-line description.
- Names the next several planned tutorials in prose (not as broken
  links): personal brain quickstart, connect your agent, VC dealflow,
  vault migration, code brain. No fake links.
- Points at the tutorial index page for the full roadmap.
- Closes with "Open an issue describing the workflow you want
  documented" to invite prioritization input from real users.

New tutorial index at docs/tutorials/README.md:

- Shipped section: company-brain.md
- In-progress roadmap with 7 candidate tutorials (personal brain
  quickstart, connect your agent, VC dealflow, vault migration, code
  brain, fully local, dream cycle setup)
- Each roadmap entry names the persona, the core commands the
  tutorial would demonstrate, and the differentiator
- "Want to write one?" section invites community PRs and points at
  company-brain.md as the model

Reverted the obscure topologies.md "See also" link from pointing at
company-brain.md specifically to pointing at the tutorials index
overall — the tutorial belongs in the README's Tutorials section
where users actually look, not buried in an architecture doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(tutorials): land personal-brain tutorial (full-stack install)

The canonical solo install walkthrough, adapted from Garry's live setup
session notes (the "Apple I, soldering breadboards" session). Builds the
full stack: 2 GitHub repos, Telegram bot, AlphaClaw on Render, OpenClaw
+ GBrain + Supabase. About 2 hours end-to-end, $100-150/month sustained.

Replaces the "Set up your personal brain in 30 minutes" stub on the
tutorials roadmap with something genuinely complete.

Edits to the source draft:
- Stripped brain-page YAML frontmatter (type/access/links/etc — not
  needed for a public docs file)
- Privacy sweep per CLAUDE.md: removed real-name references to the
  collaborator and the agents involved in the session, replaced with
  "a collaborator" / "my main agent" / generic placeholder names
- First-person voice consistency: the source draft slipped between
  first person and third person ("Garry walked through"); rewrote
  everything in first person
- Zero em-dashes used as connectors (verified by grep)
- Added ZeroEntropy to the providers list (it's the default; not
  mentioning it would leave readers paying 2.6× more on embeddings)
- Opened with a router paragraph that points brain-layer-only readers
  at INSTALL.md and team readers at company-brain.md, so each
  audience finds the right walkthrough fast

Tutorials index updated:
- personal-brain.md promoted from In Progress to Shipped (it IS the
  "set up your personal brain in 30 minutes" entry that was on the
  roadmap, just better-named and more complete)

README.md Tutorials section now lists both shipped tutorials side by
side. Reads naturally: solo install first (broader audience), team
install second.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(tutorials): rewrite company-brain as a true superset of personal-brain

Per @garrytan: the company brain tutorial should pick up from where the
personal brain tutorial leaves off, not duplicate the install. Pedagogical
flow now reads: "you already did personal-brain; here is what to add
to make it multi-user."

Restructure:

- Opens with explicit "this tutorial picks up where the personal brain
  tutorial leaves off" + a router for readers who haven't done that one
  yet. No duplicated install steps.
- Part 1 (mental model) reframed as "what changes when you go from
  personal to company" + "what this is NOT" (not a different install,
  not a thin-client-everywhere replacement).
- Part 2 is now "switch the brain backend to multi-user Postgres" —
  surfacing the migrate --to supabase path for readers who started on
  PGLite, skip-to-Part-3 for readers already on Postgres.
- Part 3 adds the new content @garrytan called for: per-person folder
  structure inside each source. customers/alice-example/, internal/
  alice-example/, internal/bob-example/, internal/legal/ etc. So
  teammates' writes don't collide and the per-person/per-role
  abstraction is real on disk, not just in OAuth scope.
- NEW Part 6: per-person crons. Each teammate gets their own
  scheduled tasks (7am customer digest for alice, 9am ops status for
  bob, weekly contract compliance for carol) scoped to their OAuth
  client so the cron can only touch their slice.
- NEW Part 7: per-person skills. The 60+ shipped skills are generic;
  teams want a few specific ones (onboarding-new-hire,
  customer-success-followup, weekly-team-digest). Scaffolded via
  gbrain skillify scaffold, scoped via allowed_clients in
  frontmatter.
- Existing strong parts retained: OAuth scoping + verify, per-
  teammate AI client connect, first synthesized query, operating
  notes, gotchas. Reworded where needed to refer back to personal-
  brain steps instead of re-explaining them.

Voice + style sweeps:
- Zero em-dashes used as connectors (verified by grep)
- Zero internal jargon (no "Cat 22", "P@5", "RRF", "MRR", etc.)
- Zero version chatter in body text (only the one acceptable
  reference to the dated benchmark filename in a docs link)
- "Brain layer" / "synthesized answer" framing, not "Think" branding
- First-person Garry voice in intro/motivation paragraphs
- All examples use placeholder names (alice-example, bob-example,
  carol-example, acme-co, diana-example)
- No mentions of Mem0 / Hindsight (per session-long convention)

Word count: 3608 (vs 3717 before — about the same length, but more
of those words now describe the NEW content rather than re-explaining
prereq install).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(tutorials): enrich company-brain with real-world patterns from production deployment

Mined production patterns from my own running company-brain deployment
to ground the tutorial in shapes that actually work. Added four
substantive sections.

Part 3 (sources) gains "Two scoping models" sidebar:

- Model A: separate sources with OAuth scoping (SQL-enforced isolation,
  right for multi-user with different AI clients per person — what the
  tutorial walks you through).
- Model B: one source with partners/<slug>/ directory convention
  (simpler ops, scoping is convention-only, right when one agent serves
  everyone over Telegram — what I actually run in production).
- Mix-and-match guidance: separate sources for the obviously-different
  ones AND partners/<slug>/ inside the shared source for per-person
  workspace.

Part 7 (skills) gains "Shared rule files at the skills root" subsection:

- _brain-filing-rules.md: iron-rule decision tree for where new pages
  belong. Every ingest skill consults it before creating a page.
- _output-rules.md: output quality standards (deterministic links built
  from API data not LLM-composed strings, citation format, no AI-slop).
- _excluded-people.md: privacy gate naming people the brain must never
  reference even when they appear in source material. Re-attribute or
  discard. The file that prevents accidental publication of things
  about people who aren't fair game.
- _operating-rules.md, _x-ingestion-rules.md, _x-api-rules.md.
- These turn into the de facto company policy for the agent. Edit one,
  every skill picks it up next request.

NEW Part 8 "Wire Slack carefully":

- Two crons, two jobs (scan every 5-15min for live signals + archive
  nightly for full history).
- Channel-to-task-ID mapping via topic-registry.json (don't reference
  raw Slack channel IDs in skills; friendly names that resolve at
  runtime).
- Deterministic links rule (LLM-composed Slack URLs hallucinate
  constantly; build from API data only).
- Dismissed-items state so re-scans don't surface noise that was
  already triaged.
- Per-channel scoping mirrors per-person scoping. Sensitive channels
  scope by OAuth client.
- Names the actual production skills (slack, slack-scan, slack-archive)
  for scaffold reference.

NEW Part 9 "Onboard each teammate yourself (the botmaster pattern)":

- The load-bearing UX gate for adoption. Don't hand a teammate an
  OAuth credential and tell them to "try it out." That's how internal
  tools die.
- Step 1: pre-populate their slice (partners/<their-slug>/USER.md
  with role/focus/priorities/preferences, 5-10 concepts that are
  theirs, 2-3 example brain entries that demonstrate the shape).
  About 20 minutes per teammate.
- Step 2: walk them through 2-3 wow flows personally. A synthesis
  query (show the brain layer). A gap-analysis query (build trust).
  A write-back flow (show capture value). About 15 minutes.
- Step 3: graduate to DM only after the wow moment lands. The order
  flips the conversion rate.
- About 45 minutes per person total. Cheaper than an unadopted tool.

Parts 8-12 renumbered to 10-14 to make room. Cross-references in body
text checked (Part 3 ref in Part 5, Part 4 ref in operating notes,
Part 5 ref in Slack section all still correct).

Word count: 4938 (vs 3608 before). Still readable in one sitting per
the original target; added content is all load-bearing patterns from
production.

Voice gates:
- Zero em-dashes used as connectors (sed-replaced all 8 introduced
  by the additions with periods)
- Zero internal jargon (no Cat N, no P@5, no RRF, no MRR)
- Zero banned names (no Brad, Gessler, Wintermute, Zion, Straylight,
  Seibel, Caldwell, Mem0, Hindsight)
- "Brain layer" / "synthesized answer" framing preserved
- First-person Garry voice throughout
- All examples use placeholder names (alice-example, bob-example,
  carol-example, diana-example, acme-co)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(tutorials): expand personal-brain Step 7 with the three real Supabase gotchas

@garrytan: the personal-brain tutorial needs to surface the specific
Supabase setup gotchas I hit the hard way. Rewrote Step 7 with the
operational detail.

Three new subsections:

- 7a: Turn on pgvector. The vector extension has to be toggled in
  Database → Extensions before GBrain's schema migrations will run.
  Five seconds in the dashboard, an hour of debugging if you forget.

- 7b: Use the CONNECTION POOLER string, not the direct connection.
  Direct is port 5432, IPv6-only. Pooler is port 6543 via pgbouncer,
  IPv4-compatible, survives connection storms from parallel workers.
  Shows the exact pooler hostname format and the gbrain config set
  command.

- 7c: Buy the IPv4 add-on. About $4/month. Even with the pooler, some
  Supabase regions / Render plans hit IPv6 resolution snags. Symptom:
  network-unreachable errors or connect hangs in gbrain doctor.
  Toggle on in Project Settings → Add-ons. Saves debugging time on
  multiple installs.

- 7d: Verify with gbrain doctor — names which of 7a / 7b / 7c to
  revisit if a check fails.

The old "Operating note" about Supabase being the scaling bottleneck
preserved at the end of the section since it's a different concern
(scale, not setup).

Voice gates: 0 em-dashes (verified), first-person Garry voice ("I hit
the hard way"), no internal jargon, no banned names.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): rewrite "What this looks like" for someone arriving cold

@garrytan: the prior version assumed too much. "Eval receipt", "top-K
vector retrieval", "MemPalace", "Haiku judge", "Facts fence", "synthesis
layer" — all jargon that requires reading the rest of the project to
parse. A new reader bounces.

New version requires zero prior context:

- Sets up a universal scenario anyone gets: "you have a meeting with
  alice tomorrow, what do you need to know?"
- Shows what a typical tool returns (a list of 5 pages with snippets)
  so the reader sees the gap themselves
- Shows what gbrain returns (a real briefing with the open items
  surfaced, plus a "heads up" about what's missing from the brain)
- Lets the two outputs speak for themselves, no judge scores or
  benchmark numbers in the body
- Closes with one plain-English sentence on the difference:
  "Search finds the pages. The brain reads them for you and writes
  the answer."

What got cut:
- The eval-receipt path reference (means nothing to a new reader)
- The Haiku judge scores (0/10 vs 9/10) — not useful out of context
- The verbatim judge verdict quotes (long, internal vocabulary)
- The "Facts fence", "typed-claim", "synthesis layer" feature names
- The MemPalace name-drop
- The link to the comprehensive benchmark page (it's still
  reachable from the Tutorials and benchmark sections; doesn't
  need to be in the intro)
- The numbered "three things gbrain think did" breakdown

The example content is illustrative (same scenario shape as a real
production query) but stripped of the internal benchmark wrapper.
Reads naturally as "imagine you're about to ask gbrain something."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: capitalize proper-noun names in prose across README + tutorials

@garrytan caught lowercase "alice" in prose — proper nouns should be
capitalized. The lowercase was leaking from the slug convention
(people/alice as the actual storage slug) into descriptive text.

Rule applied: capitalize Alice / Bob / Carol / Diana / Acme when used
as a person or company name in prose. Keep lowercase in:
- Slugs and file paths (people/alice, customers/acme-co)
- Code identifiers in fenced blocks where the slug IS the value
- URLs and hostnames (brain.acme-co.com)
- Channel-name-style references (#alice-customers)

Files swept: README.md, docs/tutorials/company-brain.md,
docs/tutorials/personal-brain.md, docs/tutorials/README.md.

The README sweep was the primary target (Garry's actual call-out);
the tutorial sweep keeps voice consistent across all the front-door
docs. Hand-fixed four occurrences inside code-fenced blocks that
were human comments rather than code (terminal session comments
"# Terminal 1, as Alice", "# Terminal 2, as Bob", directory-tree
inline comments "← Alice's customer notebook").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(readme-hero-anchors): rotate ZeroEntropy anchor → search-vs-answer headline

CI caught one expected regression after v0.40.8.1's README rewrite. The
D9 hero-anchors guard required "ZeroEntropy" in the first 50 lines of
README.md (the v0.36.0.0 default story). The post-rewrite hero
intentionally rotated that out per Garry's "no version chatter in
README" directive — ZeroEntropy still appears further down (line 211,
231, 279) but no longer in the hero. The guard's docstring explicitly
handles this case: "did we deliberately rotate the headline? If yes:
update the anchors here."

Rotation:

- Dropped: regex /ZeroEntropy|\bZE\b/ in the first 50 lines
- Added: regex matching the new headline "Search gives you raw pages.
  GBrain gives you the answer." which is the load-bearing differentiator
  of the post-rewrite hero. If a future cleanup PR accidentally rewords
  the search-vs-answer framing, the new anchor catches it the same way
  the ZeroEntropy anchor caught accidental drops before.

Other 4 anchors unchanged (OpenClaw + Hermes + production-number +
P@5/R@5 — all still load-bearing).

Updated docstring records the v0.40.8.1 rotation as the audit trail
for the next time this happens.

Verified: `bun test test/readme-hero-anchors.test.ts` → 5 pass / 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): restore agent-led install path + per-client MCP guides

The README install section had collapsed to three generic shapes
("agent platform", "CLI", "MCP server") that buried the load-bearing
flow: paste a URL pointing at INSTALL_FOR_AGENTS.md into your agent and
let it do the work. That's the path most users actually take.

Restored the original three-tier structure with an explicit second tier
for "install it into your existing agent" (Codex, Claude Code, Cursor),
plus surfaced the per-client MCP guides individually so users see the
command shape they actually need instead of one generic docs/mcp/ link.

Six per-client MCP links now in the README itself: Claude Code,
Cursor/Windsurf (stdio), Claude Desktop, Claude Cowork, Perplexity
Computer, ChatGPT. Each carries the one-line shape that matters
(claude mcp add, Settings > Integrations, OAuth 2.1, etc.).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): link personal-brain tutorial from the agent-install path

People landing on the README without an existing OpenClaw or Hermes
deployment need a starting point that walks the whole flow, not just
"paste this URL into your agent." The personal-brain tutorial already
covers picking a platform, deploying it, pointing it at
INSTALL_FOR_AGENTS.md, and verifying the first query.

Surfaced as a callout right under the agent-install snippet so the path
is visible to first-time users without burying the experienced-user
flow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 00:00:39 -07:00
Garry TanClaude Opus 4.7garrytan-agents
 		3c1cc8a4d6 v0.40.7.0 Schema Cathedral v3 — agent-on-ramp + production rebuild of PR #1321 (#1327) 
* v0.40.6.0 Phase 1 foundations — pack-lock + mutate-audit + cache invalidation + lint rules + best-effort

Six new primitives that Phase 2's withMutation skeleton (next commit) depends on.
No consumers yet; all callers wire up in Phase 4. Foundations ship first per
codex C1 phase-ordering finding from /plan-eng-review.

1.1 pack-lock.ts (18 cases)
  Atomic acquire via openSync(path, 'wx') = O_CREAT|O_EXCL. Kernel-level
  atomic, NO TOCTOU window. Codex C8 caught that page-lock.ts:79+96 has
  existsSync+writeFileSync (TOCTOU) — we deliberately do NOT copy it.
  Stale detection via TTL (60s default) + kill(pid, 0) liveness probe.
  TTL refresh every 10s while withPackLock(fn) runs so long DB-aware
  lint/stats on big brains don't go stale. --force = "steal stale lock"
  (NOT "skip locking"). Lock path per-pack so two packs never block.

1.2 mutate-audit.ts (13 cases)
  ISO-week JSONL at ~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl.
  Privacy redacted per D20: type names → sha8, prefixes → first slug
  segment only. Matches candidate-audit.ts privacy posture. Both verbose
  surfaces gate on GBRAIN_SCHEMA_AUDIT_VERBOSE=1 (same env). Logs BOTH
  success AND failure events so Phase 9's schema_pack_writability doctor
  check has signal to read (closes codex C11). summarizeMutations()
  primitive shipped for cross-surface parity between doctor + future
  audit CLI.

1.3 registry.ts cache invalidation + stat-mtime TTL (10 cases)
  invalidatePackCache(name?) walks the extends-chain reverse-graph
  (every cached entry whose chain contains name is evicted). This is the
  codex C6 fix — pre-v0.40.6, editing a parent pack silently left
  children stale because cache identity was child-bytes-only. New
  per-name CacheEntry tracks the file-stat snapshot of every file in
  the extends chain. tryCachedPack(name) is the TTL-gated fast path:
  inside STAT_TTL_MS (1000ms default, env GBRAIN_PACK_STAT_TTL_MS)
  returns cached without statting. Outside the window: stats every file
  and cascade-invalidates on any mtime change (D11 cross-process
  detection). resolvePack reference-equality preserved on byte-identical
  re-build. ASCII state-machine diagram in file header (D9).

1.4 best-effort.ts (4 cases)
  loadActivePackBestEffort(ctx) returns ResolvedPack | null. Single
  source of truth for the 4 T1.5 wiring sites (Phase 8). null means
  "EMPTY FILTER" semantics, NOT "fall back to hardcoded defaults" —
  pack-load failure must be loud, per D4. Never throws.

1.5 lint-rules.ts (35 cases)
  11 pure rule functions extracted from CLI handlers per codex C13/D16.
  9 file-plane rules + 2 DB-aware rules (extractable_empty_corpus, mutation_count_anomaly).
  Phase 2 withMutation pre-write gate composes file-plane subset.
  runAllLintRules() returns {ok, errors, warnings} structured report
  ready for CLI + MCP.

1.6 query-cache-invalidator.ts (4 cases)
  invalidateQueryCache(engine, sourceId?) DELETEs query_cache rows so
  cached search results bound to old page types don't survive a
  schema mutation. Reuses SemanticQueryCache.clear() so we don't
  reinvent the PGLite+Postgres parity. Codex C9 fix.

Tests: 84 new cases across 6 test files. All 153 schema-pack tests green.

Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md
Closes: half of T2-T7 from the plan's Implementation Tasks JSONL.
Successor to: closed PR #1321 (community PR; author garrytan-agents).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.6.0 Phase 2 — mutate.ts withMutation skeleton + 11 primitives

Builds on Phase 1 foundations (pack-lock, mutate-audit, lint-rules,
cache invalidation, query-cache-invalidator).

withMutation(packName, opts, mutator, op, ctx): 8-step skeleton wrapping
every primitive. Atomic .tmp+fsync+rename. Per-pack file lock. Pre-write
file-plane lint validation gate. Audit log on success AND failure. Pack
cache + query cache invalidation hooks. ASCII state-machine diagram in
file header per D9.

11 primitives, each ~5-line wrapper around withMutation:
  add_type, remove_type (with codex C14 reference check), update_type
  add_alias, remove_alias, add_prefix, remove_prefix
  add_link_type (rejects fm_links refs on remove)
  remove_link_type, set_extractable, set_expert_routing

Inline minimal JSON→YAML emitter so mutating a YAML pack stays YAML.
The emitter's array-of-mappings nesting was tricky: the first key sits
inline with the `- ` (e.g. `- name: person`), subsequent keys live at
indent+1, and nested arrays inside the mapping keep their relative
depth (the v0.40.6 emitter bug I fixed pre-commit: trim+prefix lost
internal indent of nested arrays like path_prefixes).

YAML round-trip: emitted YAML reparses cleanly through parseYamlMini.
Comments and formatting NOT preserved (documented in plan; pin pack.json
if you care about layout).

Codex C14 reference check: removeType refuses if any other type's
aliases/enrichable_types/link_types/frontmatter_links references the
target. STILL_REFERENCED error names every reference for cleanup.

Validation gate composes runFilePlaneLintRules from Phase 1.5 — a
mutation that would create a dangling ref or prefix collision fails
BEFORE the .tmp write (the invariant: pack file on disk is NEVER
partial).

Tests: 34 cases pinning every primitive + skeleton invariant. Bundled
guard, codex C14, atomicity (crash-mid-write leaves original untouched,
lock auto-released after mutator throw), YAML round-trip, validation
gate firing on prefix collision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.6.0 Phase 3 — stats + sync pure core functions

Both ship as runStatsCore / runSyncCore pure functions so Phase 4 CLI
handlers (next commit) and Phase 7 MCP ops (later) both compose without
duplicating logic. Codex C13 / D16 prereq for the MCP exposure phase.

stats.ts (17 cases):
  Multi-source aware: sourceIds[] (federated read) OR sourceId (single)
  OR neither (whole-brain aggregate). NULLIF(type, '') normalizes
  empty-string + NULL to one untyped bucket (pages.type is NOT NULL in
  the schema so empty string is the legacy "untyped" representation).
  Soft-delete exclusion. by_type sorted by count desc, ties by name asc.
  Empty-brain coverage:1.0 (vacuous truth, matches getBrainScore).
  Dead-prefix detection: pack-declared prefixes with zero matching
  pages surface as DeadPrefixHint[] (agent's drilldown signal for
  mis-declared paths). Best-effort: pack-load failure leaves
  pack_identity:null + dead_prefixes:[].

sync.ts (13 cases):
  D14 chunked UPDATE: 1000-row batches per prefix. Each batch:
  WITH win AS (SELECT id FROM pages WHERE untyped+prefix LIMIT $batch),
  upd AS (UPDATE ... WHERE id IN win RETURNING 1) SELECT COUNT(*). Loop
  until zero rows. Concurrent writers never block on the row-set for
  more than ~100ms per batch (vs the multi-second monolithic UPDATE
  shape PR #1321 had).
  Codex C5 write-side scoping: sourceId param directly, NOT
  sourceScopeOpts which is read-side and inherits OAuth federation
  reads. Phase 7 MCP op (schema_apply_mutations) enforces at dispatch.
  Dry-run by default: per-prefix probe returns would_apply + 10-slug
  sample (the drilldown signal). Apply path returns total_applied.
  Idempotency contract pinned: second apply finds zero matching rows.
  Soft-delete exclusion on both probe + update. Dead-prefix flag set
  when probe returns count=0. JSON envelope schema_version:1.

Tests use canonical PGLite block per CLAUDE.md test-isolation rules.
seedPage helper auto-seeds sources(id) row before FK insert.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.6.0 Phase 4 — wire 14 new schema CLI verbs

Thin handlers wrapping Phase 2's mutation primitives + Phase 3's
stats/sync cores. CLI is the human surface; Phase 7 wires the same
cores into MCP for agent use.

New verbs:
  Authoring:
    add-type <name> --primitive P --prefix dir/ [--extractable]
                    [--expert] [--alias A]* [--pack <name>]
    remove-type <name>             [--pack <name>]
    update-type <name>             [--extractable BOOL] [--expert BOOL]
                                   [--primitive P] [--pack <name>]
    add-alias <type> <alias>       [--pack <name>]
    remove-alias <type> <alias>    [--pack <name>]
    add-prefix <type> <prefix>     [--pack <name>]
    remove-prefix <type> <prefix>  [--pack <name>]
    add-link-type <name> [--inverse V] [--page-type T] [--target-type T]
                                   [--pack <name>]
    remove-link-type <name>        [--pack <name>]
    set-extractable <type> BOOL    [--pack <name>]
    set-expert-routing <type> BOOL [--pack <name>]
  Activation:
    reload [--pack <name>]         Flush in-process cache; --pack scopes
  Discovery + repair:
    stats [--source <id>]          Per-type counts + coverage + dead prefixes
    sync [--apply] [--source <id>] Backfill page.type (chunked UPDATE)

cli.ts: schema added to CLI_ONLY_SELF_HELP so `gbrain schema --help`
routes to printHelp() instead of the generic one-line stub.

withConnectedEngine defensive fix retained from PR #1321:
EngineConfig built once and passed to BOTH createEngine and
engine.connect for future-proof against engine implementations that
read URL at connect time.

End-to-end agent journey verified:
  fork gbrain-base mine → use mine →
  add-type researcher --primitive entity --prefix people/researchers/
    --extractable --expert →
  active (shows 23 page types) →
  stats (shows 100% coverage on empty brain, vacuous truth).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.6.0 Phase 5+6+7 — schema lint with rich rules + 9 new MCP ops

This is the marquee commit: Wintermute and any other remote OAuth agent
can now author + introspect schema packs over normal HTTPS MCP. Phases
5+6 collapse into Phase 7 because the new MCP ops compose Phase 1.5's
lint rules and Phase 2/3's mutation/stats/sync cores directly — no
extra extraction needed (D6 from /plan-eng-review).

Phase 5: schema lint CLI wired to runAllLintRules from Phase 1.5
  Replaces the prior 2-rule check (duplicate names + missing prefix)
  with the full 11-rule suite. New --with-db flag opts into the 2
  DB-aware rules (extractable_empty_corpus, mutation_count_anomaly).
  JSON envelope shape stable. Exit code 1 on any error.

Phase 7: 9 new MCP operations
  Read-scope (NOT localOnly — read scope is safe to expose remote):
    get_active_schema_pack — identity packet (pack name, sha8, counts).
    list_schema_packs       — bundled + installed names.
    schema_stats            — composes runStatsCore from Phase 3.
    schema_lint             — composes runAllLintRules; --with-db is
                              CLI-only (DB-aware rules need engine).
    schema_graph            — JSON {nodes, edges} from link_types
                              inference + frontmatter_links.
    schema_explain_type     — settings for one declared type.
    schema_review_orphans   — untyped pages drilldown.
  Admin-scope (NOT localOnly per D2 — Wintermute reaches via OAuth):
    schema_apply_mutations  — BATCHED per D10. Single MCP tool taking
                              a mutations[] array; composes all 11
                              mutate primitives. Atomic batch_id; outer
                              withPackLock wraps the whole batch so no
                              other writer can slip in mid-iteration.
                              Partial-results returned on mid-batch
                              failure for forensic agent debugging.
                              Audit log records actor=mcp:<clientId8>
                              (D20 privacy-redacted shape).
    reload_schema_pack      — flush in-process cache + extends-chain
                              cascade (codex C6 fix from Phase 1.3).

withConnectedEngine defensive fix applied to schema.ts:withConnectedEngine
  (PR #1321 closed) — EngineConfig built once and passed to BOTH
  createEngine AND engine.connect for defense in depth.

Test seams:
  - operationsByName lookup pinned for every new op.
  - All 9 ops have scope + localOnly declarations pinned to lock in
    the trust posture.
  - Batched mutation atomicity tested: partial-failure returns
    {error: mutation_failed, partial_results: [...]} with one batch_id
    across all results.
  - Audit log actor=mcp:<clientId.slice(0,8)> capture verified
    end-to-end (audit JSONL read back after the op handler runs).
  - Empty mutations[] rejected with invalid_request.
  - Unknown op surfaced via SchemaPackMutationError INVALID_RESULT.

Coverage: 23 new cases for the 9 ops (operations-schema-pack.test.ts).
All 255 schema-pack-related tests green.

Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md
Successor to: closed PR #1321.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.6.0 Phase 8 + 10 + 12 — T1.5 wiring, schema-author skill, ship

Final wave commit. Brings the cathedral from "shipped but undiscoverable"
to "shipped + agents find it + agents use it."

Phase 8 (partial T1.5 wiring — agent-facing surfaces):
  - whoknows CLI (src/commands/whoknows.ts:340) consults the active pack
    via loadActivePackBestEffort + expertTypesFromPack. Pack-load failure
    → EMPTY filter (NOT hardcoded ['person', 'company'] defaults) per
    D4. A researcher type declared --expert in a custom pack now
    surfaces in `gbrain whoknows "ML"` results. Pre-v0.40.6 it silently
    never matched.
  - find_experts MCP op (src/core/operations.ts:2820) same wiring so
    OAuth clients (Wintermute etc.) inherit pack-aware expert routing
    over HTTP MCP, not just CLI.
  - facts/eligibility.ts and enrichment-service.ts union widening
    deferred to v0.40.7+ (filed in TODOS.md as 2 follow-up entries) —
    larger blast radius than fit this wave's context budget.

Phase 10 (skill + RESOLVER + Convention — the discoverability layer):
  - skills/schema-author/SKILL.md — agent dispatcher for "evolve the
    schema pack." 36 trigger phrases route here. Explicit Non-goals
    section names brain-taxonomist (filing one page) and eiirp
    (schema-check during iteration) so agents pick the right surface.
    7-phase workflow: brain → assess → propose → apply → sync → verify
    → commit. Lists every gbrain schema CLI verb + every MCP op the
    skill uses. brain_first: exempt frontmatter (this skill IS the
    brain-first path for schema authoring).
  - skills/conventions/schema-evolution.md — decision tree for "when to
    add a type vs alias vs prefix." <20 pages → don't pack-codify;
    20-100 → alias or narrow prefix; 100+ → first-class type. Don'ts
    section + "when to remove a type" + "when to commit the pack" all
    answered from one place.
  - skills/RESOLVER.md entry with full functional-area dispatcher line
    (compressed routing pattern per v0.32.3 dispatcher convention).
  - schema-evolution.md added to the cross-cutting Conventions list.

Phase 12 (ship bookkeeping):
  - VERSION → 0.40.6.0
  - package.json → 0.40.6.0
  - CHANGELOG.md entry with ELI10 lead per CLAUDE.md voice rules
    (250+ words explaining the wave in plain English before any
    file/function name appears), full "To take advantage of v0.40.6.0"
    paste-ready commands block, itemized changes by category, credit
    to @garrytan-agents (PR #1321 author).
  - TODOS.md gains 10 new follow-up entries grouped under
    "v0.40.6.0 Schema Cathedral v3 follow-ups (v0.40.7+)" covering:
    enrichment-service union widening, facts/eligibility wiring, 3
    doctor checks, T16 + T16.1 evals, T19 federated closure, T20
    extends merging, T21 YAML comments, T22 admin SPA, T23
    schema:write scope, T24 multi-tenant federation.
  - llms-full.txt regenerated via bun run build:llms (CLAUDE.md
    edits trigger the test/build-llms.test.ts gate — required per
    repo discipline).

Verification:
  - bun run typecheck clean.
  - Full agent journey smoke-tested end-to-end in Phase 4 commit
    (fork → use → add-type → active → stats — all green).
  - All 255+ schema-pack tests green from Phases 1-7.

Total wave: 6 commits, ~5000 net LOC, 84 new tests, 21 design
decisions captured. PR #1321 closed with successor pointer comment.

Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): rename Wintermute → 'your OpenClaw' + add schema-author skill conformance

CI failures from PR #1327 first run:

1. check:privacy script flagged 4 'Wintermute' name leaks (CLAUDE.md:550 rule —
   never use the private OpenClaw fork name in public artifacts):
   - src/core/operations.ts:3816 → 'your OpenClaw and similar remote agents'
   - src/core/operations.ts:4015 → 'your OpenClaw, etc.' (in description)
   - src/core/operations.ts:4225 → 'your OpenClaw, etc.' (in comment)
   - test/operations-schema-pack.test.ts:325 → clientId 'remoteAgentClient12345678'
     (matching audit-actor regex updated: 'mcp:remoteAg' instead of 'mcp:wintermu')

2. skills/manifest.json missing schema-author entry. Added between
   brain-taxonomist and skillify per alphabetical-ish grouping.

3. skills/schema-author/SKILL.md missing 3 conformance sections per
   test/skills-conformance.test.ts:
   - ## Contract (inputs/outputs/side effects/idempotency/trust/atomicity)
   - ## Anti-Patterns (don't mutate bundled packs, don't add types for one-off
     directories, don't conflate filing vs. schema authoring, etc.)
   - ## Output Format (per-mutation JSON, per-batch JSON, stats JSON, sync
     dry-run JSON, human format, error envelope codes)

   The 3 sections were inserted ABOVE the existing 'Failure modes' section so
   the existing failure-mode bullets are still adjacent to the new error
   envelope codes in Output Format.

Verified locally:
- bun run check:privacy → clean
- bun test test/skills-conformance.test.ts test/check-resolvable.test.ts test/check-resolvable-cli.test.ts test/regression-v0_22_4.test.ts → 286/286 pass
- bun test test/operations-schema-pack.test.ts → 23/23 pass
- bun run verify → clean (privacy + skill_brain_first + fuzz-purity + typecheck)

llms.txt + llms-full.txt regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.40.7.0 — schema cathedral v3 README + CLAUDE.md annotations

Doc-debt cleanup from the v0.40.7.0 ship (Phase 12 had deferred these to
fit context budget; /document-release surfaced the gap):

- README.md: new "What's new in v0.40.7.0" lead paragraph above the
  v0.36.4.0 entry. ELI10 lead: "Your agents can now author your brain's
  schema pack themselves" + the agent journey + 14 CLI verbs + 9 MCP
  ops + schema-author skill boundary callouts.

- CLAUDE.md: new "Schema Cathedral v3 (v0.40.7.0)" section between the
  thin-client routing cluster and the Commands section. 14-bullet
  Key Files cluster covering pack-lock / mutate-audit / registry /
  best-effort / lint-rules / query-cache-invalidator / mutate / stats /
  sync / schema.ts CLI / operations.ts MCP / whoknows T1.5 wiring /
  schema-author skill / schema-evolution convention. Each bullet
  references the design decisions (D2/D4/D6/D8/D9/D10/D11/D13/D14/D20)
  and codex findings (C5/C6/C8/C9/C13/C14) captured during /plan-eng-review.
  Closes the "CLAUDE.md has zero v0.40.7.0 mentions" doc debt.

- llms-full.txt + llms.txt regenerated.

Privacy check clean (no Wintermute leaks in the new prose — used "your
OpenClaw" per CLAUDE.md:550 rule). test/build-llms.test.ts 7/7 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: tutorial — Build your first schema pack (closes v0.40.7+ doc-debt)

Closes the tutorial gap surfaced by /document-release's Diataxis coverage
map. The schema-pack cathedral shipped with reference (CLAUDE.md cluster),
how-to (SKILL.md 7-phase workflow), and explanation (conventions/
schema-evolution.md decision tree), but no tutorial — no concrete
"your first schema mutation" walkthrough.

docs/schema-author-tutorial.md ships exactly that:
- 8 numbered steps, time-to-first-result < 3 (active pack visible by step 2)
- Walks from `gbrain schema fork gbrain-base mine` through `add-type
  researcher` + `sync --apply` + proving the T1.5 wiring via `gbrain
  whoknows` surfacing the new type
- Every step shows the exact command and expected output
- Placeholder pages (alice-example, bob-example, charlie-example) so any
  brain can run the tutorial without affecting real content
- "What you built" section recaps state on disk + active wiring
- "Next steps" cover add-link-type, add-alias, lint --with-db, commit to
  source control, MCP path for agents
- "Related docs" cross-links to reference (CLAUDE.md cluster) + how-to
  (SKILL.md workflow) + explanation (schema-evolution.md)

Cross-linked:
- README.md "What's new in v0.40.7.0" paragraph gets a "Walkthrough:"
  pointer at the end
- skills/schema-author/SKILL.md gets a "## Tutorial" callout just above
  the workflow phases — agents that hit the skill via RESOLVER routing
  see the tutorial pointer first

Closes the Diataxis quadrant matrix to full coverage:
- Tutorial:     ✅ docs/schema-author-tutorial.md (NEW)
- How-to:       ✅ skills/schema-author/SKILL.md workflow
- Reference:    ✅ CLAUDE.md cluster + gbrain schema --help
- Explanation:  ✅ skills/conventions/schema-evolution.md

Privacy check clean. Typecheck clean. llms-full.txt regenerated (545KB).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: what-schemas-unlock — the WHY doc (7 use cases + structural argument)

The schema-author tutorial walks through HOW to mutate a pack. This new
doc explains WHY agents and users should care, with concrete killer use
cases on real corpus shapes:

1. The 4000 invisible meetings — untyped pages skip every structural
   surface (whoknows, find_experts, recall, think). Adding a `meeting`
   type + sync flips them from invisible to queryable. Same content,
   completely different agent experience.

2. The founder ops brain — 4 type-adds + 4 link-types build a
   CRM-shaped query surface. `gbrain whoknows "Series A SaaS"` routes
   through investor + portco specifically; `graph-query` walks intro
   chains. Downstream of notes, not parallel to them.

3. The research brain — researcher / paper / lab / grant / dataset
   types + cites / authored / uses link verbs turn a reading-list-as-
   markdown into a queryable research graph.

4. The legal brain (or anything where claims have numbers) — typed
   `damages=5000000`, `filed_date=...` become comparable across pages
   of the same type. Generic note systems can't do this because they
   don't know which numbers belong to which type.

5. The team brain — each mounted brain has its own schema pack. Two
   engineers searching the same brain get DIFFERENT routing because
   their personal packs declare different expert types.

6. The agent-co-curates pattern — the NEW thing in v0.40.7.0. Agent
   watches your ingestion stream, runs `gbrain schema detect`
   periodically, proposes a new type when a pattern accumulates, applies
   it via batched MCP `schema_apply_mutations` after one approval.
   Brain learns. Audit log captures the agent's client_id as
   `actor: mcp:<clientId8>`.

7. Before-vs-after on real content — pick a corpus, note top-3
   whoknows results, add the type via sync, re-run. The numerical
   delta IS the win.

Then the structural argument: types matter at query time. Untyped
content is invisible content. The schema is queryable AND mutable AND
auditable — that's the production-system difference from "vibes-based
knowledge management."

Closes with the v0.40.7.0-specific list of what changed (withMutation
skeleton, O_CREAT|O_EXCL atomic lock vs page-lock.ts TOCTOU pattern,
privacy-redacted audit log, 9 MCP ops, T1.5 wiring, cross-process
invalidation via stat-mtime TTL gate).

Cross-linked:
- README.md "What's new in v0.40.7.0" paragraph now has both the
  "Why it matters:" pointer (this doc) AND the "Walkthrough:"
  pointer (tutorial).
- docs/schema-author-tutorial.md opens with "Want the WHY before the
  HOW?" link to this doc.
- skills/schema-author/SKILL.md now has a "Tutorial + vision" section
  that points at both, with explicit guidance that agents should read
  the WHY doc before pitching schema authoring to a user.

177 lines. Privacy check clean. Typecheck clean. llms-full.txt
regenerated (545KB).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: surface schema docs from README Capabilities + Docs index + llms.txt

The two new schema docs were ONLY linked from the v0.40.7.0 "What's new"
paragraph in README. That paragraph will get pushed down by every future
release and become a worse and worse entry point.

Real discovery paths added:

1. README.md `## Capabilities` section — new "Agent-authored schema
   (v0.40.7.0)" bullet between "Brain consistency" and "## Integrations".
   Permanent home alongside Hybrid search, Self-wiring graph, Minions,
   43 skills, Eval framework. Includes the one-paragraph pitch + 3
   pointer links (vision / tutorial / agent skill).

2. README.md `## Docs` index — two new lines added at the top of the
   list (right after docs/INSTALL.md, before docs/architecture/):
   - docs/what-schemas-unlock.md with one-line description
   - docs/schema-author-tutorial.md with one-line description

3. scripts/llms-config.ts `Configuration` section — both docs added to
   the curated llms.txt entry list so the LLM-readable map points at
   them. Sits right after docs/GBRAIN_RECOMMENDED_SCHEMA.md (topical
   grouping). includeInFull defaults to true so they ride in the
   single-fetch llms-full.txt bundle.

Result: schema docs are now reachable from 5 entry points instead of 1:
  - README "What's new" paragraph (release-pinned, will age out)
  - README Capabilities bullet (permanent, top-of-funnel)
  - README Docs index (permanent, end-of-page reference)
  - llms.txt (LLM-readable curated map)
  - llms-full.txt (single-fetch bundle for agents)

Also caught 3 leftover Wintermute leaks in docs/what-schemas-unlock.md
that the privacy check flagged: agent-co-curates pattern now uses "your
OpenClaw"; `register-client wintermute` example renamed to
`register-client my-agent` per CLAUDE.md:550 privacy rule. Privacy
check clean. test/build-llms.test.ts 7/7 green. llms.txt 4314 → 5000
bytes, llms-full.txt 545KB → 572KB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
 2026-05-23 16:46:01 -07:00
Garry TanandClaude Opus 4.7 d28be5d091 v0.40.4.0 feat(search): selective graph signals + per-stage attribution + audit-writer unification (#1300) 
* v0.40.4.0 T1: shared audit-writer primitive

Extract createAuditWriter() helper. Five hand-rolled JSONL audit
modules (rerank-audit, shell-audit, supervisor-audit, audit-slug-
fallback, phantom-audit) duplicated the same ISO-week filename math,
best-effort write loop, and read-current-plus-previous-week loop.
T2 refactors all 5 onto this primitive.

Behavior preservation: filename format, JSONL line shape, mkdir
recursive, appendFileSync utf8, stderr-on-failure all byte-identical
to the existing modules so their tests pass unchanged.

resolveAuditDir() moves here from shell-audit.ts; shell-audit.ts
will re-export for back-compat (T2). Honors GBRAIN_AUDIT_DIR with
whitespace-trim, falls back to ~/.gbrain/audit/.

Test coverage: 22 cases covering ISO-week math + year-boundary edges
(2027-01-01 → 2026-W53), env override, mkdir-recursive, fail-open
stderr-warn shape, cross-week readback, corrupt-row skip, non-finite-
ts skip, round-trip with nested fields, computeFilename + resolveDir
accessors.

Plan ref: D5=B audit unification cathedral expansion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T2: refactor 5 audit modules onto shared writer

Replace the duplicated ISO-week filename math + best-effort write loop
+ read-current-plus-previous-week loop in:
  - src/core/rerank-audit.ts (rerank-failures-*.jsonl)
  - src/core/audit-slug-fallback.ts (slug-fallback-*.jsonl)
  - src/core/minions/handlers/shell-audit.ts (shell-jobs-*.jsonl)
  - src/core/minions/handlers/supervisor-audit.ts (supervisor-*.jsonl)
  - src/core/facts/phantom-audit.ts (phantoms-*.jsonl)

All five now delegate file I/O to createAuditWriter from T1. Public
API preserved bit-for-bit:
  - logRerankFailure, readRecentRerankFailures, computeRerankAuditFilename
  - logSlugFallback, readRecentSlugFallbacks, computeSlugFallbackAuditFilename
  - logShellSubmission, computeAuditFilename, resolveAuditDir
  - writeSupervisorEvent, readSupervisorEvents, computeSupervisorAuditFilename
    plus isCrashExit, summarizeCrashes, CrashSummary (domain-specific
    helpers stay in supervisor-audit.ts; only file I/O moves)
  - logPhantomEvent, readRecentPhantomEvents, computePhantomAuditFilename

Domain-specific behavior preserved:
  - audit-slug-fallback emits per-call stderr (D7 dual logging) in the
    caller; the shared writer is failure-only stderr
  - rerank-audit truncates error_summary to 200 chars before write
  - phantom-audit spreads optional fields conditionally (skip undefined)
  - supervisor-audit keeps single-file readback (no cross-week walk)
    to preserve pre-v0.40.4 doctor assertions

resolveAuditDir lives in src/core/audit/audit-writer.ts; shell-audit.ts
re-exports it so existing imports keep working (every other audit
module + gbrain-home-isolation.test.ts + minions.test.ts +
minions-shell.test.ts pull resolveAuditDir from shell-audit.ts).

Operator-visible drift: rerank-audit stderr line drops the
'rerank-failure audit' qualifier — was '[gbrain] rerank-failure audit
write failed (...)' now '[gbrain] write failed (...); search continues'.
Stderr is human-debugging, not machine-parsed; the file written gives
the qualifier away in `tail -f audit/*`.

Test coverage: 128/128 audit-touching tests pass unchanged.

Plan ref: D5=B audit unification cathedral expansion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T3: getAdjacencyBoosts engine method (PG+PGLite parity)

Add BrainEngine.getAdjacencyBoosts(pageIds) returning Map<page_id,
AdjacencyRow{hits, cross_source_hits}>. Returns ALL pages with
hits >= 1 (callers apply their own threshold).

Cross-source semantic (D15=A): cross_source_hits EXCLUDES the target
page's own source. A page in source A linked from 2 pages in source A
reports cross_source_hits = 0. Linked from 1 in source B + 1 in
source C reports 2.

Source-scope contract: pageIds MUST already be source-scoped by the
caller. Method does NOT filter by source_id. The in-set restriction
makes cross-source leakage impossible by construction. JSDoc spells
this out; same trust posture as cosineReScore's chunk_id handling.

COALESCE(p.source_id, 'default') on both target and from-page sides
for defense-in-depth even though pages.source_id is NOT NULL today.

JSDoc/SQL contract alignment (codex #2): HAVING >= 1 matches the
"returns ALL pages with hits >= 1" contract; threshold of 2 is the
caller's call in applyGraphSignals.

Known limitation (codex #15): cross_source_hits cannot distinguish
"genuinely linked from another team" from "mirrored imports from
another source." T-todo-4 captures the v0.41+ refinement.

SearchResult type extension (D4=A flat fields, D12=A attribution):
  - graph_adjacency_hits, graph_cross_source_hits,
    graph_session_demoted, graph_session_prefix
  - base_score, backlink_boost, salience_boost, recency_boost,
    exact_match_boost, graph_adjacency_boost, graph_cross_source_boost,
    session_demote_factor, reranker_delta
All optional; T4-T6 populate them.

Test coverage: 7/7 hermetic PGLite cases. Empty input, singleton,
same-source hub, cross-source attribution including the
"linked-only-from-other-source" case (widget in source b, linked
from alice+bob in source a → cross_source_hits=1), JSDoc HAVING>=1
contract. Postgres parity asserted by SQL-shape identity (will get a
mirror Postgres E2E in T10's eval gate work via DATABASE_URL when
set; PGLite hermetic case shipped now).

NULL source_id COALESCE branch noted as untestable in current PGLite
schema (pages.source_id is NOT NULL); kept as defense-in-depth.

Plan ref: T3 in v0.40.4.0 wave plan; D1=A, D3=A, D15=A.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T4+T11: applyGraphSignals 4th stage in runPostFusionStages

New file src/core/search/graph-signals.ts. Three signals:

  1. Adjacency-within-top-K (×1.05): hits >= 2 inbound from in-set.
  2. Cross-source adjacency (×1.10, stacks): cross_source_hits >= 2.
     Dormant on single-source brains.
  3. Session diversification (×0.95): if multiple top-K share a slug
     prefix, keep highest scoring, DEMOTE the rest. NOT amplify —
     codex caught the original framing was backwards (amplification
     of redundancy makes the cited "weak chunks compete for budget"
     problem worse, not better).

Conservative magnitudes (D14=B): 1.05/1.10/0.95. Score-distribution
probe (onScoreDistribution) collects min/p25/p50/p75/p95/max +
reorder_band_width to feed T-todo-2 magnitude calibration wave.

Slot: 4th stage inside runPostFusionStages (hybrid.ts:248), AFTER
backlink/salience/recency, pre-dedup. Inherits the v0.35.6.0
floor-ratio gate from computeFloorThreshold — this is the structural
protection that prevents a low-cosine hub from outranking a strong
non-hub (codex T2 / D1=A).

PostFusionOpts extends with graphSignalsEnabled, onGraphMeta,
onScoreDistribution. Caller (hybridSearch in subsequent T5 work)
resolves graph_signals from the mode bundle.

Source-scope contract preserved: getAdjacencyBoosts takes raw
page_ids, no source filter. Adjacency is in-set restricted so
cross-source leakage is impossible by construction (D3=A).

Fail-open: engine throw → JSONL audit row via shared createAuditWriter
(T1/T2 primitive, featureName='graph-signals-failures') + meta.errored
+ caller's results unchanged. Session diversification ALSO skips on
failure (predictable all-or-nothing posture).

Mutation note (codex #9): score mutated in place. base_score must be
stamped at runPostFusionStages entry BEFORE this stage so eval-capture
sees pre-boost score (T6 attribution wave).

Test coverage (24 cases, including T11 IRON RULE regression):
  - sessionPrefix multi/single/empty cases
  - computeScoreDistribution percentile math
  - Disabled + empty short-circuits
  - Adjacency hit, no-hit, cross-source stacking, cross-source alone
  - Session diversification 3-share + single-segment + singleton
  - Test seam injection (no engine call)
  - Fail-open: throw → audit row + meta.errored + unchanged
  - Empty Map → session still runs
  - Score-distribution always emits when enabled
  - Meta carries fire counts + duration_ms
  - Missing page_id silently skipped from dedup set
  - **T11 IRON RULE regression (3 cases):**
    * weak hub BELOW floor_threshold does NOT get boosted past
      above-floor non-hub (the bug class the floor gate exists for)
    * hub AT floor still gets boosted (gate is < not <=)
    * NaN score → NaN >= threshold is false → no boost

Plan ref: T4 + T11 in v0.40.4.0 wave plan; D1=A, D2=A, D11=B, D14=B,
D9=A, D5=B. Codex outside-voice #1 + #2 + #6 + #8 + #9 addressed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T5: graph_signals mode-bundle knob + KNOBS_HASH bump 3→4

ModeBundle gains graph_signals: boolean. Per-mode defaults:
  - conservative: false (cost-sensitive tier)
  - balanced:     true (the wave's primary surface for default-on)
  - tokenmax:     true (power-user tier, capstone fit)

SearchKeyOverrides + SearchPerCallOpts gain optional graph_signals
field. resolveSearchMode picks via the standard per-call → config
override → mode bundle chain.

loadOverridesFromConfig parses 'search.graph_signals' from the config
table ('1' or 'true' → true). SEARCH_MODE_CONFIG_KEYS adds the key
so `gbrain search modes --reset` clears it alongside other knobs.

KNOBS_HASH_VERSION bump 3→4 (append-only per CDX2-F13). New `gs=`
parts entry appended AFTER cross-modal + column + prov entries. A
graph-on cache write cannot be served to a graph-off lookup —
mid-deploy hit-rate dip clears within cache.ttl_seconds (3600s).

src/commands/search.ts KNOB_DESCRIPTIONS gains graph_signals entry
so `gbrain search modes` dashboard renders the new knob.

Test coverage:
  - test/search-mode.test.ts (+ 8 new cases): per-mode defaults
    canonical, config override both directions, per-call override
    wins, knobsHash distinct for on/off, config key registered,
    attributeKnob reports per-call + mode sources correctly.
  - test/search/knobs-hash-reranker.test.ts: version assertion
    bumped 3→4 with v0.40.4 rationale comment.
  - test/cross-modal-phase1.test.ts: version assertion bumped
    3→4 with v0.40.4 rationale comment.
  - Canonical-bundle assertions updated to include graph_signals
    in expected shape (3 cases).

50/50 search-mode tests pass. 45/45 cross-modal pass. 17/17
knobs-hash-reranker pass. 10/10 balanced-reranker pass.

Plan ref: T5 in v0.40.4.0 wave plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T6: per-stage attribution stamping in every boost

Every boost stage that mutates SearchResult.score now stamps a field
recording WHAT it multiplied:

  - applyBacklinkBoost  → backlink_boost (skipped when count == 0)
  - applySalienceBoost  → salience_boost (skipped when score == 0)
  - applyRecencyBoost   → recency_boost (skipped on evergreen prefix)
  - applyExactMatchBoost → exact_match_boost (skipped on no-match
    OR when intent's exactMatchBoost == 1.0 no-op)
  - runPostFusionStages → base_score stamped ONCE at entry, BEFORE
    any boost mutates r.score. Idempotent: caller-pre-stamped value
    preserved. Empty-results short-circuit unchanged.
  - applyReranker → reranker_delta = original_index - new_index
    (positive = rank improved; raw rerank score stays in rerank_score)
  - applyGraphSignals → graph_adjacency_boost, graph_cross_source_boost,
    session_demote_factor (T4 already stamped these)

Why: feeds the T7 `gbrain search --explain` formatter so it can
attribute the final score to its components. Without these stamps,
"why did this rank where it did?" is grep-and-guess.

SearchResult.reranker_delta doc updated to clarify it's a RANK delta
(positive = improved), not a score delta. The raw relevance score
stays in `rerank_score` (untyped, for back-compat with telemetry that
already reads it).

Test coverage: 16 new cases in test/search/attribution-stamping.test.ts.
Pins: every boost stamps when it fires AND skips stamping when it
doesn't (no false attribution on no-op stages). base_score idempotency
preserved. reranker_delta computed correctly across rank-improved +
rank-degraded cases.

All 178/178 search tests pass (no regressions).

Plan ref: T6 cathedral expansion in v0.40.4.0 wave plan; D12=A.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T7: gbrain search --explain per-stage attribution

New file src/core/search/explain-formatter.ts renders SearchResult[]
as a multi-line breakdown of how the final score was formed:

  1. people/alice (score=12.4)
     base=10.2 (rrf+cosine)
     + backlink ×1.08
     + salience ×1.05
     + adjacency ×1.05 (hits=3)
     + cross_source ×1.10 (other_sources=2)
     ↑ reranker rank +2
     = final 12.4

Reads the boost_* / base_score / *_hits fields populated by T4 + T6.
Empty path: "no boosts applied" when no stage stamped anything.
Session demote rendered with `-` prefix (not `+`) so the demotion
direction is visually distinct from boosts.

CliOptions gains `explain: boolean`; parseGlobalFlags recognizes
`--explain` anywhere in argv. cli.ts formatResult for `search` +
`query` cases reads CliOptions.explain via the module-level
singleton and routes to formatResultsExplain when set. Lazy import
keeps the hot path narrow for the common non-explain case.

Number formatting: 4-decimal precision, trailing zeros stripped
('1.0000' → '1', '0.1234' → '0.1234'). NaN preserved as 'NaN'.

Test coverage:
  - test/search/explain-formatter.test.ts: 19 cases pin output
    format. Each boost type renders correctly, every-stage stacking
    composes, reranker_delta=0 doesn't render, empty list short-
    circuits, rank numbering 1-based, number formatting edge cases.
  - test/cli-options.test.ts: 3 new cases for --explain parsing
    (basic, absent default, any-argv-position).

Existing CliOptions literals in test/cli-options.test.ts +
test/thin-client-upgrade-prompt.test.ts updated for new required
explain field.

JSON envelope unchanged — the same attribution fields surface in
existing --json output via JSON.stringify; no separate JSON formatter
needed.

Plan ref: T7 cathedral expansion in v0.40.4.0 wave plan; D12=A + D6=A.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T8: doctor check graph_signals_coverage

New checkGraphSignalsCoverage in src/commands/doctor.ts. Wired into
both runDoctor (local engine) and doctorReportRemote (HTTP MCP /
JSON path) so local AND remote-server brains both surface the metric.

Logic:
  1. Resolve active graph_signals setting: config override
     'search.graph_signals' wins, else mode bundle default
     ('search.mode' → conservative=false, balanced/tokenmax=true).
  2. When disabled → silent ok ("disabled — coverage not checked").
     Avoids polluting doctor output on installs that don't use the
     feature.
  3. When enabled, compute global inbound-link density:
     COUNT(DISTINCT to_page_id) / COUNT(*) across non-deleted pages.
  4. <10% → warn ("signal will rarely fire") with paste-ready
     `gbrain extract all` fix hint.
  5. >=30% → ok ("fire on most queries") with metric.
  6. 10-29% → ok ("fire occasionally") with metric.

Known limitation (codex outside-voice #14): global density is an
imperfect proxy for "top-K subgraphs have enough edges to fire."
T-todo-5 captures the v0.41+ refinement that measures actual fire
rate from search-stats after 30 days of data.

Best-effort: SQL errors → warn with the underlying message. Never
breaks doctor.

Test coverage (7 new cases in test/doctor.test.ts):
  - conservative mode → silent ok regardless of coverage
  - balanced default + 0 links → warn at 0% with fix hint
  - balanced default + 40% inbound → ok "fire on most queries"
  - balanced default + 20% inbound → ok "fire occasionally"
  - explicit search.graph_signals=false overrides mode default
  - empty brain → ok with explanation
  - check is wired into runDoctor (source-grep regression guard)

All 55/55 doctor.test.ts cases pass.

Plan ref: T8 in v0.40.4.0 wave plan; D6=A.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T9: gbrain search stats graph_signals section

runStatsSubcommand in src/commands/search.ts gains a graph_signals
section in both --json and human output:

  Graph signals:
    enabled:    true (mode default)
    failures:   3 fail-open event(s)
      ECONNREFUSED         2
      timeout              1

Data sources:
  - config: 'search.graph_signals' override → enabled + source=config,
    otherwise mode-bundle default → enabled + source=mode_default.
  - JSONL audit: readRecentGraphSignalsFailures(days) returns events;
    failures_count is len, failures_by_reason buckets by first word of
    error_summary (e.g. 'ECONNREFUSED', 'timeout').

JSON envelope (schema_version 2 unchanged; graph_signals is a new
sibling property of stats, so consumers reading the existing fields
keep working):

  {
    "schema_version": 2,
    ...stats...,
    "graph_signals": {
      "enabled": bool,
      "source": "config" | "mode_default",
      "failures_count": int,
      "failures_by_reason": { reason: count }
    },
    "_meta": { metric_glossary: { ..., graph_signals_enabled: ..., graph_signals_failures_count: ... } }
  }

Fire-rate metrics (adjacency_fires, cross_source_fires,
session_demotions) and score-distribution stats are NOT in this
section yet — they require telemetry-table writes from the
applyGraphSignals onMeta callback. Wired in v0.41+ via T-todo-2
calibration wave (the wave that needs them). For v0.40.4: status +
error count is the actionable surface for "is graph_signals on, and
is it failing?"

Human output: prints the section after the existing stats block.
Edge case: when total_calls is 0 BUT graph_signals is enabled OR
has historical failures, still prints the section so operators
don't lose the signal on a brain with no telemetry yet.

Test coverage (6 cases in test/search/search-stats-graph-signals.test.ts):
  - search.graph_signals=true → enabled true, source=config
  - mode=conservative → enabled false, source=mode_default
  - no config → enabled true (balanced default), source=mode_default
  - JSONL failures bucketed by first word of error_summary
  - empty audit → failures_count 0, empty failures_by_reason
  - human output includes "Graph signals:" header

Plan ref: T9 in v0.40.4.0 wave plan; D6=A.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T10: eval gates (longmemeval-mini A/B + paired bootstrap)

New test/e2e/graph-signals-eval.test.ts runs each longmemeval-mini
question twice (graph_signals off, graph_signals on) and asserts:

  Gate 1 (QUALITY) — paired bootstrap, 10,000 resamples:
    - If signals-on is significantly WORSE than off
      (delta < 0 AND p < 0.05) → fail.
    - Otherwise pass. p>=0.05 either direction OR delta >= 0 → ok.

  Gate 2a (CHANGE-MAGNITUDE): mean Jaccard@5 over result-set overlap
    must be >= 0.5. If results overlap less than half, the change is
    too large and needs human review before default-on.

  Gate 2b (CHANGE-MAGNITUDE): top-1 stability rate >= 0.7. If 30%+
    of top picks change, hard look required.

  Gate 3 (HARD ABSOLUTE FLOOR): recall@5 drop <= 5pt. Catastrophic
    regression catch (codex outside-voice #18 — addresses the "top-5
    must not drop at all" brittleness on tiny fixtures).

Bootstrap implementation:
  - Per-question observation is binary (recall@5 hit/miss).
  - Paired pairing on question_id between on/off branches.
  - Centered distribution under null (subtract observed mean) per
    standard paired-bootstrap-shift approach for binary outcomes.
  - Two-tailed p-value: |resampled delta| >= |observed delta|.
  - Deterministic seeded RNG so test runs are stable across CI.

pairedBootstrapPValue exported as a pure function with separate
tests for edge cases (empty input, all-equal, strong positive, strong
negative, determinism). Reusable from future calibration waves.

Hermetic: in-memory PGLite via createBenchmarkBrain + resetTables
between questions. No API keys needed (--no-embed import path
exercises keyword-only retrieval). Skips gracefully via describe.skip
when the fixture is missing.

Plan ref: T10 in v0.40.4.0 wave plan; D7=C absolute floor + D13=A
paired bootstrap; codex #4 + #18 stability-vs-quality distinction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 T12: VERSION + package.json + CHANGELOG + TODOS

VERSION: 0.37.11.0 → 0.40.4.0
package.json: 0.37.11.0 → 0.40.4.0
CHANGELOG.md: top entry for v0.40.4.0 in ELI10-lead voice per
  CLAUDE.md release rules. Lead is plain-English ("Your search now
  notices when a page is a hub for your query"); precise file paths
  / SQL semantics / numbers live in the "Itemized changes" section
  below. Includes the cathedral-expansion notes (D5=B audit
  unification, D12=A per-stage attribution, D13=A eval gates) and
  the "To take advantage of v0.40.4.0" verify-and-fix block.

TODOS.md: 5 new items captured under "v0.40.4 graph signals —
deferred follow-ups (v0.41+)":
  - T-todo-1: profile graph-signal SQL latency, merge if hot (D8=C)
  - T-todo-2: magnitude calibration wave from probe data (D14=B / D17)
  - T-todo-3: DB-backed audit table for cross-deploy observability (codex #15)
  - T-todo-4: sync-topology-aware cross-source signal (codex #11)
  - T-todo-5: replace doctor's global density with fire-rate (codex #14)

Verified the 3-line audit: VERSION + package.json + CHANGELOG topmost
all match 0.40.4.0. `bun install` ran (lockfile unchanged — root
package version isn't stored in bun.lock). `bun run build:llms`
refreshed llms.txt + llms-full.txt for the next commit.

Plan ref: T12 in v0.40.4.0 wave plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 TODO: document pre-existing shard-2 flake noticed during ship

3 isCacheSafe test failures in shard 2 reproduce on stashed clean
master. Confirmed pre-existing — not introduced by v0.40.4. Filed
under "Pre-existing flake on master (noticed during v0.40.4 ship)"
with reproduction commands + remediation options. Shipping v0.40.4
through it; future wave can fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 privacy scrub: replace wintermute → media in example slugs

CLAUDE.md line 550 bans the private OpenClaw fork name in public
artifacts. Example session prefix in sessionPrefix() docs + 3 test
fixtures swept to 'media/chat/...' instead. Pre-existing
scripts/check-privacy.sh in `bun run verify` caught it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 fix: wire graph_signals from mode bundle to runPostFusionStages

CRITICAL: pre-landing review (codex outside-voice via /ship Step 9)
caught that hybrid.ts's `postFusionOpts` literal at line 566 was
building PostFusionOpts WITHOUT threading `resolvedMode.graph_signals`
to `graphSignalsEnabled`. The gate at hybrid.ts:358 read the field
from a literal that never set it.

Result before this fix: the entire v0.40.4 graph-signals wave was
dead code in production. Mode bundles set
`balanced.graph_signals = true` and `tokenmax.graph_signals = true`,
but no production call site ever reached applyGraphSignals. The
KNOBS_HASH bump 3→4 correctly varied the cache key by the flag, so
contamination was prevented — but the feature itself never fired.

All shipped infrastructure (engine SQL, fail-open audit, attribution
stamps, --explain formatter, doctor coverage check, search-stats
section) was reachable only through the unit-test seam
(`opts.adjacencyFn`). The CHANGELOG-advertised behavior never
landed in user-visible search.

Fix: thread `graphSignalsEnabled: resolvedMode.graph_signals` into
the postFusionOpts literal (1 line). Inline comment names codex's
catch so future refactors see the regression class.

Tests: new test/search/graph-signals-wire-integration.test.ts pins
the wire end-to-end. Three cases:
  1. balanced mode → hybridSearch on a seeded brain with adjacency
     hub produces a result with base_score stamped (proves
     runPostFusionStages actually ran).
  2. search.graph_signals=false config override → no graph_* fields
     stamped (proves the gate honors the override path).
  3. Source-grep regression guard pinning the
     `graphSignalsEnabled: resolvedMode.graph_signals` literal in
     hybrid.ts so a future refactor can't silently disconnect.

All 57 existing v0.40.4 wave tests still pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 fix: pre-landing review AUTO-FIX findings (audit msg drift + deleted_at)

Two informational findings from /ship pre-landing review (Step 9):

1. Stderr message qualifier drift (rerank/slug-fallback/phantom audits)
   Pre-v0.40.4 messages included a per-feature qualifier:
     [gbrain] rerank-failure audit write failed (...)
     [gbrain] slug-fallback audit write failed (...)
     [gbrain] phantom audit write failed (...)
   The T2 refactor dropped the qualifier (plan promised "byte-identical"
   operator-visible behavior, but stderr lines did drift). Restored via
   new `errorMessagePrefix` option on `createAuditWriter` (optional, ''
   default). Three modules pass the per-feature qualifier; shell-audit
   and supervisor-audit unaffected (their pre-v0.40.4 messages didn't
   have a separate qualifier — label already carried the feature name).

2. Defense-in-depth `deleted_at IS NULL` on getAdjacencyBoosts
   SQL was previously protected by-construction (hybridSearch's
   visibility filter ensures input pageIds are live), but matches the
   v0.35.5.0 findOrphanPages pattern and closes the bug class if a
   future caller bypasses hybridSearch. Added to both Postgres and
   PGLite engines for parity. Three JOIN sites guarded (targets CTE,
   FROM-pages join). One inline comment per engine cites the codex
   review and the v0.35.5.0 precedent.

Plan ref: /ship pre-landing review v0.40.4.0 (codex finding C and F).

All 84 audit+graph-signals tests pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 fix: adversarial review HIGH findings (codex H1+H2 + Claude F1)

Three HIGH-severity issues from /ship adversarial pass:

H1 (Codex): Eval gate was a no-op.
  Test passed `graph_signals: graphSignalsOn` via `as any` cast, but
  SearchOpts had no field and hybridSearch's perCall didn't thread it.
  Both off/on branches resolved to the mode-bundle default — gate
  measured identical behavior, could pass while detecting nothing.

  Fix: add `graph_signals?: boolean` to SearchOpts (types.ts:794).
  Thread `opts.graph_signals` into perCall in both hybridSearch
  (hybrid.ts:425) AND hybridSearchCached (hybrid.ts:1027) so the
  cache-key resolver also sees the override. Drop the `as any` from
  the eval test — types are real now.

H2 (Codex): Session diversification fired on entity directories.
  sessionPrefix() used "any shared parent directory" as the session
  signal. Result: a search for "people in SF" returned `people/alice`
  + `people/bob` + `people/charlie` and the latter two got demoted
  to 0.95×. Every common entity-search query silently penalized
  legitimate same-type results. Default-on for balanced/tokenmax
  means production behavior was wrong.

  Fix: narrow sessionPrefix() to fire ONLY when the slug contains a
  session-like marker (`chat`/`session`/`sessions` segment OR a
  `YYYY-MM-DD` date segment). Entity directories (`people/`,
  `companies/`, `docs/`) return null → diversification skips.
  Returns NULL (not the slug itself) so the loop skips clean.
  Examples in JSDoc:
    your-agent/chat/2026-05-20-foo → 'your-agent/chat/2026-05-20-foo'
    daily/2026-05-20/journal-entry-1 → 'daily/2026-05-20'
    transcripts/chat/funding-discussion → 'transcripts/chat/funding-discussion'
    people/alice → null  ← codex H2 regression
    docs/quickstart → null

F1 (Claude adversarial subagent): case-sensitivity drift across 3 sites.
  loadOverridesFromConfig in mode.ts is case-insensitive +
  whitespace-trimmed for 'search.graph_signals' values. But
  doctor's checkGraphSignalsCoverage (doctor.ts:899) AND
  search-stats's readGraphSignalsStats (search.ts:288) used
  case-sensitive compare. User sets `search.graph_signals TRUE`:
  production enables the feature, but doctor + search-stats both
  silently report disabled. Operators lose the only observability
  surface for the new feature on values like 'True'/'TRUE'.

  Fix: trim + lowercase parity at both sites. Mirror the parser's
  semantic. Also case-normalized `search.mode` reads at both sites
  for the same divergence class.

Tests:
  - sessionPrefix block rewritten with 7 cases covering chat marker
    + date anchor + entity dirs (now-NULL) + degenerate (no /).
  - Added regression test pinning codex H2: people/alice +
    people/bob + people/charlie do NOT get diversified.
  - graph-signals-eval.test.ts drops `as any` — typed field works.
  - Existing tests using `chat/a`/`chat/b` updated to session-shaped
    `media/2026-05-20/chunk-a` so the date anchor actually fires.

111/111 graph-signals + doctor + search-stats tests pass. Typecheck clean.

Plan ref: /ship adversarial review v0.40.4.0 (codex H1, H2; Claude F1).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.4.0 TODOs: capture 11 LOW adversarial findings for v0.41+

Codex L1 (audit window underreport) + Claude F2/F3/F5-F8/F11/F12/F14/F16
from /ship adversarial review. None are load-bearing; all captured under
'v0.40.4 adversarial review LOW findings — captured for v0.41+'.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.40.4.0

- README: surface v0.40.4.0 graph signals + --explain in Hybrid search capability
- CLAUDE.md: annotate engine.ts getAdjacencyBoosts, new graph-signals.ts /
  explain-formatter.ts / audit/audit-writer.ts, plus hybrid.ts post-fusion
  4th stage, mode.ts graph_signals knob + KNOBS_HASH 3→4, cli-options.ts
  --explain flag, search stats + doctor coverage check
- llms-full.txt: regenerated from CLAUDE.md per the build:llms chaser rule

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): pin bun-version to 1.3.13 across all workflows

setup-bun action with `bun-version: latest` calls the GitHub API
(https://api.github.com/repos/oven-sh/bun/git/refs/tags) to resolve
the tag. CI started failing today with HTTP 401 "Bad credentials"
even though the action receives a token (visible as `token: ***`
in the run log). Pinning the version eliminates the API call
entirely.

Affected workflows: test.yml, e2e.yml, release.yml, heavy-tests.yml
(5 invocations total). Pinned to 1.3.13 — matches package.json
engines (`bun >= 1.3.10`) and the version v0.40.4.0 was developed
against.

Bump cadence: when a new bun version is required, update this
pin in one PR. Trading "always-latest" for "always-deterministic"
is the right trade for a 5-shard CI matrix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 10:01:08 -07:00
Garry TanandClaude Opus 4.7 a19ee8bafe v0.40.2.0 feat: trajectory routing for temporal + knowledge_update (gbrain think + LongMemEval) (#1296) 
* feat(facts): add event_type column + trajectory-format helper (v0.40.2.0 Commit 1)

Substrate work for v0.40.2.0 Track B (trajectory routing for temporal +
knowledge_update). This commit lands the schema + the shared formatter;
think wiring + LongMemEval extractor + intent routing come in Commits 2-4.

Migration v81 (facts_event_type_column):
  ALTER TABLE facts ADD COLUMN event_type TEXT (nullable, metadata-only).
  Lets the v0.35.4 typed-claim substrate carry event-shaped rows
  (event_type='meeting'/'job_change'/'location_change') alongside the
  metric-shaped rows (claim_metric/claim_value etc) it has carried since
  v67. Temporal-reasoning questions ("when did I last meet Marco") need
  the event shape; the metric shape doesn't fit them.

Engine changes (pglite + postgres parity):
  - TrajectoryPoint.event_type: string | null added; projection in both
    findTrajectory SQL paths returns the column.
  - TrajectoryOpts.kind?: 'metric' | 'event' | 'all' added (default 'all').
    Defensive opt that future-proofs filtering once event rows accumulate.
  - Both engines apply the new kind filter at SQL level when set.

Back-compat (codex outside-voice concern):
  Existing callers (founder-scorecard, eval-trajectory) already defensively
  skip metric === null rows in their per-metric math. Event-only rows
  (metric=NULL, event_type='meeting') ride through invisibly to those
  callers — verified by the new regression test that asserts byte-identical
  computeFounderScorecard + computeTrajectoryStats output with and without
  event rows in the input. Both callers now pass kind:'metric' explicitly
  for call-site clarity (no behavior change).

MCP find_trajectory op:
  - event_type added to the wire-shape map.
  - kind param added to the op declaration (enum metric/event/all).

Shared formatter (src/core/trajectory-format.ts, new):
  formatTrajectoryBlock(points, entitySlug, opts) — sibling shape to
  renderTakesBlock + renderChatBlock. Groups by (metric ?? event_type).
  Per-metric cap 20, total cap 100 (prompt-budget guardrail). For
  knowledge_update intent, annotates value-change rows with
  "(superseded prior)" — the explicit signal codex flagged was missing
  from default RRF-ordered retrieval. Promoted to src/core/ so both
  gbrain think (Commit 2) and the LongMemEval harness (Commit 4)
  consume one source of truth.

Prompt-injection coverage (codex Problem 10):
  src/core/think/sanitize.ts INJECTION_PATTERNS extended with three
  new entries — close-trajectory, open-trajectory, xml-attr-inject —
  so adversarial </trajectory> sequences in extracted text get
  escaped before reaching the model. Parity with the existing
  </take> coverage.

Tests (all hermetic, no DATABASE_URL):
  - test/trajectory-format.test.ts (17 cases, all green): grouping,
    caps, sanitization, supersession annotation, determinism,
    provenance, text-cap, adversarial </trajectory> escape.
  - test/engine-parity-event-type.test.ts (6 cases): PGLite round-trip
    of the column + kind filter matrix.
  - test/regressions/v0_40_2_0-trajectory-backcompat.test.ts (4 cases):
    pins the byte-identical-output contract that founder-scorecard's
    per-metric math ignores event rows.
  - test/migrate.test.ts: v81 round-trip verified via existing
    structural assertion harness.
  - 209 tests across 5 impacted suites pass; bun run verify clean
    (17 pre-checks including privacy, jsonb, type, fuzz purity).

Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
GSTACK REVIEW REPORT: CEO + ENG + CODEX CLEARED.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(think): trajectory injection for temporal + knowledge_update (v0.40.2.0 Commit 2)

Wires the v0.40.2.0 substrate (Commit 1's facts.event_type column +
formatTrajectoryBlock) into the production `gbrain think` surface.
Default ON; flip `think.trajectory_enabled=false` to opt out.

New pure modules (zero engine dependency):
  - src/core/think/intent.ts — classifyIntent(question): regex-first
    routing into 'temporal' | 'knowledge_update' | 'other'. KU wins over
    temporal when both match. The 'other' fast path short-circuits with
    zero SQL.
  - src/core/think/entity-extract.ts — extractCandidateEntities() pulls
    high-precision candidates from retrieval slugs (people/, companies/,
    organizations/, deals/) and medium-precision noun phrases from the
    question. Word-level tokenization + stop-word boundaries stitch
    "Blue Bottle" as one candidate while splitting "I last meet Marco"
    correctly. Leading-verb stripper drops "meet", "visit" etc so
    "marco" surfaces cleanly. Cap of 5 per question.

Engine-touching wiring (src/core/think/index.ts):
  - RunThinkOpts gains 4 fields: withTrajectory (default true),
    sourceId, allowedSources, remote.
  - readThinkTrajectoryEnabled() reads the config kill switch; default
    true; survives missing config table on legacy brains.
  - Trajectory orchestration sits between gather and prompt assembly:
    intent classify → extract candidates → per-candidate
    resolveEntitySlugWithSource → skip fallback_slugify → 5s timeout
    Promise.race + 3-wide concurrency cap → formatTrajectoryBlock.
    Any error degrades to "no block" + TRAJECTORY_INJECTION_FAILED
    warning; the think call itself never crashes from trajectory.
  - On success, TRAJECTORY_INJECTED_<N>_POINTS warning records the
    count for downstream telemetry.

Prompt placement (src/core/think/prompt.ts) — Codex Problem 6 fix:
  buildThinkUserMessage's trajectoryBlock slot honors BOTH existing
  orderings — calibration mode inserts trajectory between calibration
  and question; default mode inserts between retrieval and the output
  instruction. NO third ordering is introduced. Empty trajectoryBlock
  skips the "Known trajectory:" header entirely (don't cue the model
  we tried).

Resolution-source signal (src/core/entities/resolve.ts) — Codex Problem 5:
  New companion resolveEntitySlugWithSource() returns
  {slug, source: 'exact_page' | 'fuzzy_match' | 'fallback_slugify'}
  so trajectory routing can skip fallback-only resolutions —
  querying findTrajectory on an invented slug always returns [] and
  wastes a SQL round-trip. The original resolveEntitySlug keeps its
  contract for pre-v0.40 callers.

MCP think op handler (src/core/operations.ts):
  Extracts sourceScopeOpts(ctx) into scalar sourceId + allowedSources
  + remote, threads through to runThink. CLI callers omit (engine
  default source, remote=false). Mirrors the same source-scope
  discipline applied to all other read paths in v0.34.1.0.

Sanitization (Commit 1 already extended INJECTION_PATTERNS for
</trajectory> — consumed here).

Test coverage (all hermetic, no DATABASE_URL, no API keys):
  - test/think-intent.test.ts (14 cases) — temporal, KU, other,
    precedence (KU wins when both match), defensive non-string inputs.
  - test/think-entity-extract.test.ts (10 cases) — retrieved-slug
    source, noun-phrase source, stop-word stripping, leading-verb
    stripping, dedup across sources, 5-candidate cap.
  - test/think-trajectory-injection.test.ts (7 cases against PGLite
    in-memory) — temporal intent injection happy path with superseded-
    prior annotation, "other" intent short-circuit, withTrajectory:
    false bypass, think.trajectory_enabled=false config bypass,
    empty-trajectory skip, engine.findTrajectory throw is caught
    (Promise.allSettled defense), TRAJECTORY_INJECTED warning count.
  - Existing test/think-pipeline.serial.test.ts re-asserted unchanged
    (10 cases — calibration mode parity, gather, sanitization,
    cite-render all intact).

72 tests pass across 7 impacted suites; bun run verify clean (17 pre-
checks). Defaulted on per CEO + Eng D1; kill switch via config.

Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(longmemeval): inline Haiku claim extractor + content-hash cache (v0.40.2.0 Commit 3)

Populates the LongMemEval benchmark brain's facts table inline at
import time so Commit 4's intent routing has data to retrieve. Per the
CHANGELOG D1 decision, this is full-haystack preprocessing — disclosed
explicitly in the benchmark output's methodology_note field (Commit 4).

New module src/eval/longmemeval/extract.ts:
  extractAndInsertClaims({engine, client, model, sessionSlug,
                          sessionId, sessionBody, sourceId, aliasMap})
  - Hashes the session body (sha256) for cache lookup.
  - Cache hit → reuses parsed claims (cuts a 3-iteration benchmark
    run from $1.50 to $0.50 when sessions repeat across questions,
    as they do in LongMemEval).
  - Cache miss → one Haiku call. System prompt asks for
    {entity, metric, value, unit, period, event_type, valid_from, text}[]
    JSON. New parseExtractedJsonArray() helper does fence-strip + parse
    (parseModelJSON from cross-modal-eval is shaped for scored objects,
    not arrays — different parser needed here).
  - Per-record validateClaim() drops malformed records (missing
    entity, bad date) silently; the rest land in NewFact rows.
  - Per-question AliasMap (Codex Problem 4 — semantics pinned):
    "Marco" + "Marco Smith" + "marco" in the SAME question collapse
    to one slug via first-mention-wins canonicalization. Across
    questions, the harness creates a fresh map (no leak).
  - Real-page-aware entity resolution via the v0.40.2.0
    resolveEntitySlugWithSource (Commit 2). Slugify-fallback rows
    still insert (we need the data); the resolution_source signal
    is only consulted at trajectory retrieval time (Commit 4).
  - Bulk insert via engine.insertFacts with the
    `gbrain-allow-direct-insert` allow-list comment per the
    check-system-of-record CI guard contract — benchmark brain is
    ephemeral in-memory PGLite, no markdown source-of-truth applies.
  - Fail-open posture: Haiku throw, malformed JSON, insert collision
    all return inserted=0 without throwing. One bad session never
    kills the per-question loop.
  - getCacheStats() exposes hits/misses/size for the per-run stderr
    telemetry Codex Problem 14 asked for (empirical hit-rate
    reporting; the optimistic claim self-verifies).

Substrate plumbing (extends Commit 1):
  - NewFact.event_type?: string | null added in engine.ts so the
    extractor can pass event-shaped rows through to insertFacts.
  - PGLite engine + Postgres engine insertFacts() now persist
    event_type. Param-positional dispatch extended to 20/21 placeholders
    (null-embedding vs embedding-present); tx.unsafe vector cast on
    Postgres path unchanged.

Test coverage (test/longmemeval-extract.test.ts, 13 cases, hermetic):
  - Happy path: typed-claim + event rows both insert with correct
    kind (event_type='meeting' → kind='event'; claim_metric='mrr'
    → kind='fact').
  - Alias map: per-session collapsing ("Marco" + "Marco Smith"),
    cross-session persistence within one question, fresh map per
    question (caller-clears semantics pinned).
  - Content-hash cache: identical body → cache hit, only ONE Haiku
    call across two sessions; different bodies miss; getCacheStats
    reports hits/misses/size.
  - Fail-open: malformed JSON, Haiku throw, empty array output,
    invalid records (missing entity, bad date) — none crash; 0
    inserted in each case.

55 tests pass across 4 impacted suites; bun run verify clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(longmemeval): trajectory intent routing + prompt splice + methodology disclosure (v0.40.2.0 Commit 4)

The final wiring: per-question intent classification + trajectory call
+ block splice into the answer-gen prompt. Plus the methodology
disclosure stamps that close out the Codex D1 contract.

New module src/eval/longmemeval/intent.ts:
  classifyIntent(q): prefers q.question_type from the dataset
  (LongMemEval ships labels like 'temporal-reasoning',
  'knowledge-update', 'single-session-user') before falling back to
  the SHARED regex set imported from src/core/think/intent.ts.
  Single source of truth for the regex — think and longmemeval
  cannot drift.

Harness wiring in src/commands/eval-longmemeval.ts:
  - runEvalLongMemEval() spawns an extractor model via resolveModel
    (tier:'utility' → haiku) when trajectory routing is enabled.
    Calls resetExtractorState() once per benchmark run so the
    content-hash cache + counters start clean.
  - runOneQuestion() creates a FRESH per-question AliasMap (Codex
    Problem 4 — first-mention-wins canonicalization stays scoped to
    one question, never leaks across).
  - Per session: after importFromContent lands, extractAndInsertClaims
    populates the facts table. Fail-open if the Haiku call errors;
    next session keeps going.
  - After hybridSearch returns: classifyIntent(q) routes
    temporal/knowledge_update through extractCandidateEntities (the
    SHARED helper from Commit 2's think/entity-extract) → per-candidate
    findTrajectory with 5s Promise.race timeout → formatTrajectoryBlock.
    First candidate with a non-empty trajectory wins.
  - generateAnswer() splices the trajectory block BEFORE the
    Retrieved sessions block. Empty block (no entity match / no
    points) → no "Known trajectory:" header (don't cue the model
    we tried).
  - JSON envelope gains 5 fields per question when trajectory routing
    is on: intent, trajectory_points, entity_resolved,
    resolution_source, methodology_note. methodology_note also
    written to stderr at run completion.

Resolution-source gate DIVERGES from think (intentional):
  In the think production path, fallback_slugify results are skipped
  because querying invented slugs wastes SQL — production brains have
  canonical pages. In the LongMemEval benchmark, there ARE no
  canonical pages; both the extractor and the lookup go through
  slugify-fallback on the same free-form name, so they cohere on the
  same slug. Applying the think-path gate here would permanently
  block trajectory injection on the benchmark. Comment in
  runOneQuestion documents the divergence.

New CLI flag --no-trajectory:
  Bypasses BOTH the Haiku extractor AND the per-question intent
  routing. Used by the measurement protocol to baseline default-on vs
  no-trajectory across 3 seeds per condition with paired-bootstrap
  CI. Documented in the help text.

New RunOpts fields:
  - extractorClient?: ThinkLLMClient — separate stub from the
    answer-gen client so tests can isolate the two surfaces.
  - extractorModel?: string — model override for the Haiku call.

methodology_note = 'extractor=haiku-preprocess-full-haystack-v1'
stamped on:
  - Every per-question JSON envelope row.
  - Stderr summary at run completion.
This is the Codex D1 contract: the temporal-reasoning delta we
publish is "gbrain + Haiku-preprocess pipeline" vs "gbrain alone",
not directly comparable to LongMemEval's published baselines
without that disclosure.

Extractor cache hit-rate stderr summary (Codex Problem 14):
  '[longmemeval] extractor.cache_hits: 412 / 489 sessions (84.2%,
  cached_bodies=412)' — empirical verification of the optimistic
  hit-rate claim. The optimistic number self-verifies per run.

Test coverage (all hermetic, no API keys):
  - test/longmemeval-intent.test.ts (9 cases) — dataset
    question_type → Intent mapping for all six LongMemEval labels;
    dataset label trumps question-text signal; unknown labels fall
    through to the regex classifier.
  - test/longmemeval-trajectory-routing.test.ts (4 cases) —
    end-to-end through runEvalLongMemEval with both clients stubbed:
    trajectory block lands in answer-gen prompt for temporal
    intent + absent for 'other'; --no-trajectory bypasses extractor
    AND injection AND omits envelope fields; methodology_note
    stamped on every routed row; perf gate preserved (< 10s for
    2-question fixture).

118 tests pass across 11 impacted suites; bun run verify clean.

Wave complete. CHANGELOG draft + measurement plan live in the plan
file. v0.40.2.0 ready for /ship after a real-LLM spot-check run.

Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.40.2.0)

v0.40.2.0 trajectory routing wave — gbrain think now grounds answers
about temporal/knowledge-update questions in the typed-claim timeline
the brain has been quietly building via the extract_facts cycle phase.
Default ON; flip think.trajectory_enabled=false to opt out.

LongMemEval-side wiring lands the same plumbing in the benchmark
harness with explicit methodology disclosure (extractor=haiku-preprocess-
full-haystack-v1) in the JSON envelope and stderr summary — the published
temporal-reasoning number is "gbrain + Haiku-preprocess" vs "gbrain alone",
not directly comparable to LongMemEval's published baselines without that
disclosure.

Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
3 review passes: CEO + ENG + CODEX all CLEARED.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync project docs for v0.40.2.0 trajectory routing

CLAUDE.md, README.md, AGENTS.md extended with the v0.40.2.0 trajectory
routing surface: gbrain think integration (default ON via
think.trajectory_enabled config key), facts.event_type schema column +
TrajectoryPoint.event_type + TrajectoryOpts.kind filter, shared
formatTrajectoryBlock helper in src/core/trajectory-format.ts,
LongMemEval extractor + intent routing + methodology disclosure,
migration v82.

llms-full.txt regenerated to match CLAUDE.md edits (CI test/build-llms
gate).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: fill v0.40.2.0 unit + e2e gaps (71 new tests across 7 gap areas)

Audit of the trajectory-routing wave's test surface vs the shipped
code surfaced 7 gaps. All filled, all green. Total: 343 tests across
17 impacted suites (was 272 pre-fill).

Gap 1 — Migration v86 structural tests (11 new in test/migrate.test.ts):
  - v86 entry exists with documented name + idempotent
  - exactly one event_type column add to facts
  - IF NOT EXISTS guard
  - column is nullable (no NOT NULL, no DEFAULT regression guard)
  - does NOT create any index (event_type is selectivity-poor)
  - does NOT touch any other table (blast-radius pin)
  - does NOT carry a sqlFor override (engine-shared SQL contract)
  - PGLite round-trip: column exists with right type + nullable
  - event_type INSERT/SELECT round-trip
  - NULL round-trip for legacy + metric-only rows
  - LATEST_VERSION >= 86 contract pin

Gap 2 — resolveEntitySlugWithSource branch coverage (12 new in
test/entity-resolve.test.ts):
  - exact_page branch (full slug, slug-shape match)
  - fuzzy_match branch (Title-cased display name, bare first name via
    prefix expansion)
  - fallback_slugify branch (unseeded name, multi-word non-match
    phrase, accented input)
  - null tail (empty + whitespace)
  - back-compat parity with resolveEntitySlug for both exact_page and
    fallback_slugify branches

Gap 3 — INJECTION_PATTERNS dedicated coverage for new entries (18 new
in test/think-sanitize-trajectory.test.ts):
  - close-trajectory entry registered + matches canonical and
    whitespace/case variations
  - open-trajectory entry registered + matches both no-attr and
    with-attrs forms
  - xml-attr-inject strips entity=/metric=/event_type=/kind=
  - does NOT strip non-trajectory attribute names (class/id/title)
  - combined multi-vector attack: all three patterns fire
  - formatTrajectoryBlock end-to-end with adversarial extractor text:
    one live </trajectory> (the wrapper, not the injection); one
    entity= attribute (the wrapper, not the injection)
  - pattern ordering invariant: new entries land after close-take

Gap 4 — runThink calibration-mode placement contract (3 new in
test/think-trajectory-injection.test.ts):
  - default mode: question → pages → takes → trajectory → instruction
  - calibration mode: pages → takes → calibration → trajectory →
    question → instruction (Codex P6 — no third ordering invented)
  - empty trajectory in calibration mode preserves the existing
    calibration shape (no false-positive cue)

Gap 5 — runThink resolution_source != fallback_slugify gate (1 new
in test/think-trajectory-injection.test.ts):
  - candidate that only matches via fallback_slugify is NOT queried
    (think-path divergence from longmemeval-path which accepts it)

Gap 6 — E2E for runThink trajectory injection (7 new in
test/e2e/think-trajectory-pglite.test.ts):
  - full pipeline lands <trajectory> block in answer-gen prompt
  - knowledge_update intent annotates value-change rows with
    (superseded prior)
  - 'other' intent short-circuits (no block, no SQL)
  - think.trajectory_enabled=false config bypasses entire path
  - empty brain → graceful no-op (no crash, no block)
  - multi-entity deterministic ordering
  - adversarial </trajectory> in seeded fact text is escaped before
    reaching the LLM (end-to-end sanitization gate)

Gap 7 — longmemeval extractor stress + persistence pins (6 new in
test/longmemeval-extract.test.ts):
  - alias map cross-session stress with 12 sessions in one question;
    all 12 rows collapse under ONE entity_slug
  - different entities stay separate across many sessions
  - embedding + embedded_at both NULL on benchmark-inserted rows
    (regression guard against accidental embed-on-write)
  - row_num sequential + source_markdown_slug stamped per session
    (v0.32.2 partial UNIQUE index contract)
  - source field stamped "longmemeval:extractor" (audit-tag pin)
  - cache key invariance: same body hash hits cache across different
    sessionId/slug

bun run verify clean (17 pre-checks). No regressions in any of the
14 non-new impacted suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: exempt facts.event_type from schema-bootstrap-coverage CI guard

The schema-bootstrap-coverage CI guard (test/schema-bootstrap-coverage.test.ts)
enforces that every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by
applyForwardReferenceBootstrap OR by PGLITE_SCHEMA_SQL's CREATE TABLE
bodies OR by COLUMN_EXEMPTIONS.

v0.40.2.0's migration v87 adds facts.event_type but deliberately ships
without a bootstrap probe because:
  - No CREATE INDEX in PGLITE_SCHEMA_SQL references event_type
  - No FK references event_type
  - All existing callers (founder-scorecard, eval-trajectory, gbrain
    think trajectory injection) defensively skip NULL-metric rows in
    per-metric math, so event_type=NULL on pre-v87 brains is invisible
  - Pre-v87 brains land event_type=NULL via the migration ALTER

Exactly mirrors the precedent set by facts.claim_metric / claim_value /
claim_unit / claim_period exemptions (v67 typed-claim columns) which
are exempted for the same structural reason: column-only migration,
no forward-reference index, no downstream filter breaks on old brains.

Adding facts.event_type to COLUMN_EXEMPTIONS with a brief rationale
comment matching the existing v0.35.6 entry shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 22:37:15 -07:00
Garry TanandClaude Opus 4.7 3de06b6c29 v0.38.2.0 fix(doctor): bounded frontmatter scan + partial-state surfacing (supersedes #1287) (#1297) 
* fix(frontmatter): prune vendor dirs at descent + bounded wall-clock with partial-state surfacing

Two production-grade fixes for the v0.38.2.0 wave (supersedes PR #1287).

Root cause Fix 1 (the bug that hung gbrain doctor on 216K-page brains): both
brain-writer.ts:walkDir and frontmatter.ts:collectFiles recursed into every
subdirectory without calling pruneDir, the canonical descent-time pruner
used by sync/extract/transcript-discovery since v0.35.5.0. On brains that
double as code workspaces, the walkers stat'd hundreds of thousands of
entries under node_modules / .git / .obsidian / *.raw / ops that isSyncable
filtered out at the leaf — paying the IO cost for nothing. Wiring pruneDir
at descent (with the v0.37.7.0 #1169 submodule-gitfile check) eliminates
the bulk of the wall-clock pain.

Fix 2 (codex outside-voice C1): AbortSignal.timeout cannot interrupt the
synchronous walker — readdirSync / lstatSync / readFileSync block the event
loop, so timer callbacks never fire mid-walk. The load-bearing wall-clock
bound is now a deadline check inside scanOneSource's visit callback
(Date.now() > opts.deadline). AbortSignal still works at source boundaries.

Shape changes (codex C2 + C4):
- ScanOpts: + deadline?: number, + dbPageCountForSource hook, + visitDir test seam
- PerSourceReport: + status: 'scanned' | 'partial' | 'skipped', + files_scanned, + db_page_count
- AuditReport: + partial: boolean, + aborted_at_source: string | null
- ok = grandTotal === 0 && !partial (a clean prefix from a timed-out scan
  no longer falsely reports clean)

walkDir + collectFiles now exported with an optional visitDir callback for
the regression suite. Production callers don't pass it.

Tests:
- test/brain-writer-walk-prune.test.ts (new, 12 cases): visitDir-based
  descent-time pruning assertions for both walkers. Pins the property
  output-based tests can't catch (isSyncable rejects vendor files at
  the leaf — so a test checking only output passes under the original bug).
- test/brain-writer-partial-scan.test.ts (new, 5 cases): deadline + partial
  state + ok-after-abort + numerator/denominator coverage. Uses deadline,
  NOT AbortSignal, since codex C1 proved abort can't interrupt sync.
- test/brain-writer.test.ts: existing "abort mid-scan" test refit to the
  new partial-state contract (per_source has 'skipped' entries instead of
  being empty — gives doctor visibility into which sources weren't checked).
- test/migrations-v0_22_4.test.ts: AuditReport fixture extended with the
  new required fields.

Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(doctor): wire deadline + partial-state into frontmatter_integrity check

Adopts the v0.38.2.0 ScanBrainSources surface in doctor's frontmatter_integrity
check.

- AbortSignal.timeout(fmTimeoutMs) for between-source bound.
- deadline = Date.now() + fmTimeoutMs (the load-bearing mid-walk bound —
  codex C1 caught that AbortSignal alone can't fire inside the sync walker).
- GBRAIN_DOCTOR_FM_TIMEOUT_MS env override (default 30000ms; invalid values
  fall back to default rather than crash).
- Per-source DB denominator via SELECT COUNT(*) FROM pages WHERE source_id = $1
  AND deleted_at IS NULL (codex C3: deleted_at filter so soft-deleted pages
  don't inflate the count).
- Honest partial-render: "PARTIAL — scanned ~N files (source has ~M pages in
  DB), K issue(s) so far" instead of "scanned ~N of M pages" (codex C3 — the
  two populations are overlapping but not identical sets).
- "NOT SCANNED (timeout — run gbrain frontmatter validate <id>)" per skipped
  source so the user knows which sources didn't get checked.
- Catch block simplified to "unexpected error only" (codex D4 — the
  AbortError special case from PR #1287 was unreachable in a sync walker).

Tests: test/doctor-frontmatter-partial.test.ts (new, 11 cases) — structural
source-grep pins on every load-bearing render string plus the simplified-
catch contract. Behavioral coverage is deferred to the heavy script
(tests/heavy/frontmatter_scan_wallclock.sh, T6) because runDoctor calls
process.exit unconditionally and can't be driven from bun:test directly;
refactoring runDoctor to return rather than exit is a separate TODO.

Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.38.2.0 release notes, Phase 2 design sketch, heavy wall-clock smoke

- CHANGELOG.md: ELI10-lead-first release entry per CLAUDE.md voice rules.
  Names the user-visible behavior change, the per-source partial render, the
  performance numbers table, the "things to watch" caveats. Credits
  @garrytan-agents for PR #1287's diagnosis.
- VERSION + package.json: 0.37.11.0 -> 0.38.2.0.
- docs/architecture/frontmatter-scan-incremental.md: Phase 2 design sketch
  for DB-backed scan state. Schema, migration shape, writer paths
  (sync-side UPSERT + incremental scan + autopilot cycle phase), doctor
  reader, sequencing concerns, two-phase rollout plan. Starting point for
  the follow-up PR — sub-second steady-state doctor needs incremental
  state, but the schema migration carries its own contract surface
  (forward-reference bootstrap, schema-drift E2E, PGLite-vs-Postgres
  parity) that deserves its own focused PR.
- tests/heavy/frontmatter_scan_wallclock.sh (new, manual / nightly per
  tests/heavy/README.md): seeds a synthetic 60K-file brain (10K real + 50K
  under node_modules/) and asserts gbrain doctor completes in <15s with
  frontmatter_integrity: ok. Codex C7 caught that the original plan's
  1500-file budget was too small to be a meaningful guard — at that scale
  the test passes BEFORE AND AFTER the fix, proving nothing. 60K is the
  minimum that catches the descent-into-vendor-trees regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: adversarial review followups — CLI hint, deadline-vs-await race, between-source breadcrumb

Codex adversarial review caught 4 real bugs in the v0.38.2.0 wave. All four
fixed before ship.

#1 (user-facing): `gbrain frontmatter validate` takes a filesystem PATH, not a
source id. Pre-fix the NOT SCANNED hint pointed users at
`gbrain frontmatter validate src-a` — which would fail with "no such
directory", breaking the very remediation this PR ships to give them. Fix:
render `src.source_path` instead.

#2 (correctness): between sources, `await dbPageCountForSource(src.id)` ran
unchecked. A slow query could blow past the deadline, then scanOneSource was
still called and returned `status='partial'` with `files_scanned=0` —
misleading ("partial scan" when actually zero files were scanned). Fix: add a
post-await deadline re-check; mark source + remainder as 'skipped' if the
budget already burned.

#3 (UX): when the outer-loop deadline check fired BETWEEN sources,
`aborted_at_source` stayed null and the doctor message said "PARTIAL SCAN"
with no source name. Fix: stamp `aborted_at_source` with the source we were
about to start.

#4 (correctness): the COUNT query had no per-call deadline. A wedged
Postgres pool could make a single COUNT hang past the budget and defeat the
wall-clock guarantee. Fix: Promise.race against the remaining deadline; on
timeout, resolve null and the post-await re-check (#2) marks the source
skipped.

Tests: 3 new regression cases in brain-writer-partial-scan.test.ts pinning
the fixed contracts (skipped-vs-partial under slow COUNT, hanging COUNT
within deadline, aborted_at_source before any source starts). 8648 pass /
0 fail across the full suite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): refresh production-brain stats — 8.2x pages, 5.6x people, 7.4x companies

Pre-update line (months stale): "17,888 pages, 4,383 people, 723 companies, 21
cron jobs running autonomously, built in 12 days."

Fresh counts from ~/git/brain (the wintermute production brain):
- pages: 17,888 → 146,646 (8.2x)
- people: 4,383 → 24,585 (5.6x)
- companies: 723 → 5,339 (7.4x)
- cron jobs running: 21 → 66 (113 total, 66 enabled per ~/git/wintermute/workspace/ops/cron-snapshot.json)

Dropped "built in 12 days" — at 146K pages the initial-velocity claim is
stale narrative that no longer matches the current scale story.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:29:59 -07:00
Garry TanandClaude Opus 4.7 26c54588fe v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (#1275) 
* feat(ingestion): v0.38 substrate — daemon + IngestionSource contract + 2 sources

The foundation for the ingestion cathedral (CEO+DX+Eng plan-reviewed).
Plan: ~/.claude/plans/system-instruction-you-are-working-ethereal-riddle.md

WHAT YOU CAN NOW DO
The IngestionSource public contract is locked. Skillpack publishers can
build third-party ingestion sources (Granola, Linear, Mail, voice, OCR,
etc.) and ship them through the v0.37 skillpack registry. The locked
surface lives at the new package subpaths:

  import { IngestionSource, IngestionEvent } from 'gbrain/ingestion';
  import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness';

Both subpaths are pinned by test/public-exports.test.ts — breaking either
is a major-version change.

WHAT THIS COMMIT BUILDS
Foundation:
- src/core/ingestion/types.ts (IngestionSource, IngestionEvent,
  IngestionSourceContext, validateIngestionEvent, computeContentHash,
  INGESTION_SOURCE_API_VERSION, INGESTION_CONTENT_TYPES)
- src/core/ingestion/dedup.ts (24h content-hash LRU, 5000-entry cap)
- src/core/ingestion/skillpack-load.ts (gbrain.plugin.json discovery for
  third-party sources, api_version compat with paste-ready upgrade hints,
  in-process trust model for v1)
- src/core/ingestion/daemon.ts (IngestionDaemon: in-process source
  supervision sibling to v0.34.3.0 ChildWorkerSupervisor pattern, plus
  validate -> dedup -> rate-limit -> dispatch pipeline + health surface)
- src/core/ingestion/test-harness.ts (publisher-facing test utility with
  fake clock + in-memory event bus + expectEvent matchers + engine proxy
  that throws on access so publishers know what they're depending on)
- src/core/ingestion/index.ts (barrel for gbrain/ingestion subpath)

First two built-in sources prove the abstraction:
- file-watcher (chokidar over the brain repo; 1s debounce; honors
  pruneDir from src/core/sync.ts; symlinks rejected; Linux ENOSPC
  surfaces a paste-ready sysctl hint at runtime)
- inbox-folder (~/.gbrain/inbox/ target for iOS Shortcuts / AirDrop /
  Drafts; auto-archives processed files into .archived/YYYY-MM-DD/;
  symlink rejection; world-writable dir warning; routes content-type by
  extension)

Public exports surface (count 18 -> 20) pinned in:
- package.json exports map
- test/public-exports.test.ts EXPECTED_EXPORTS + count gate
- scripts/check-exports-count.sh baseline

ARCHITECTURE-LOCKED DECISIONS (from /plan-eng-review)
E1 webhook source process boundary: webhook source will live INSIDE
serve --http (NOT this daemon) when it lands in the next commit. Daemon
supervises only daemon-side sources.
E2 content-type processor execution: hybrid by size (inline <1MB,
Minion handlers >1MB). Processors land in a later commit.
E3 publisher TTHW: chokidar v4.0.3 across platforms; ephemeral PGLite
persistence and Linux inotify-limit doctor probe land in later commits.
E4 migration v80 (provenance columns) + forward-reference bootstrap:
lands with put_page write-through in a later commit.

DX-locked decisions (from /plan-devex-review):
- Source error semantics: throws bubble to daemon; supervisor backoff.
- IngestionTestHarness exported as gbrain/ingestion/test-harness.
- api_version field on gbrain.plugin.json with loud-fail on mismatch.

TESTS
192 cases across 8 test files, 0 failures:
- test/ingestion/types.test.ts (28 cases pinning the contract)
- test/ingestion/dedup.test.ts (15 cases for LRU + TTL + collision)
- test/ingestion/skillpack-load.test.ts (22 cases for manifest
  validation + api_version compat + collision policy + module load)
- test/ingestion/test-harness.test.ts (24 cases for harness lifecycle +
  clock + healthCheck + every expectEvent matcher)
- test/ingestion/daemon.test.ts (19 cases for supervision + dispatch
  pipeline + health surface + per-source config + logger wrapping)
- test/ingestion/sources/file-watcher.test.ts (10 cases including
  ENOSPC sysctl-hint surfacing)
- test/ingestion/sources/inbox-folder.test.ts (24 cases including
  symlink rejection + world-writable warning + archive-loop-prevention)
- test/public-exports.test.ts (2 new cases for the new subpaths)

typecheck clean. bun run verify gate passes.

NEXT IN WAVE
Subsequent commits in this PR ship webhook source (serve --http route),
cron-scheduler refactor + OpenClaw credential auto-migrate, content-type
processors (PDF + image OCR + audio transcribe + video keyframe), put_page
write-through with serializePageToMarkdown DRY extract, migration v80
+ bootstrap probes, gbrain capture verb, publisher DX cathedral (init
scaffold extension + gbrain ingest test [--watch] + tail + validate),
daemon rename autopilot -> ingest with forever-alias, doctor inotify
probe on Linux, skillpack contract docs + reference pack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): webhook source — POST /ingest + ingest_capture Minion handler

Lands the v0.38 ingestion cathedral's webhook source. Per the
/plan-eng-review E1 decision, the webhook source lives INSIDE
`serve --http` (NOT the ingestion daemon) so there is no new IPC: the
HTTP route submits Minion jobs directly into the existing queue, and
the daemon supervises only daemon-side sources.

WHAT YOU CAN NOW DO
With `gbrain serve --http` running and an OAuth client minted, any
HTTP caller (Zapier, IFTTT, n8n, Make, Apple Shortcuts) can POST a
captured thought into the brain:

  curl -X POST https://your-brain.example.com/ingest \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: text/markdown" \
    -d "# captured from my Shortcut"

The route auths via OAuth (write scope required), validates the
content-type, enforces a 1MB payload cap and per-IP rate limit
(100 events / 10s), submits an `ingest_capture` Minion job tagged
`untrusted_payload: true`, and returns 202 Accepted with the job id.
The job materializes the page under `inbox/YYYY-MM-DD-<hash6>` by
default (overridable via X-Gbrain-Slug header) so the user has a
predictable triage location.

WHAT THIS COMMIT BUILDS
- src/core/minions/handlers/ingest-capture.ts (new) — handler that
  takes an IngestionEvent payload, resolves a slug via fallback chain
  (job.data.slug -> event.metadata.slug -> inbox/<date>-<hash6>),
  validates the event at the handler boundary, REJECTS binary
  content_types with a paste-ready hint to install a processor
  skillpack, and routes through importFromContent. Defaults
  noEmbed: true (embed is a separate Minion job, matching the sync
  handler's pattern).
- src/commands/jobs.ts — registers `ingest_capture` in
  registerBuiltinHandlers alongside sync/embed/extract.
- src/commands/serve-http.ts — POST /ingest route with:
    - OAuth write-scope gate via requireBearerAuth({requiredScopes:['write']})
    - 100 events / 10s rate limiter (sibling to ccRateLimiter)
    - Content-type allowlist: text/markdown, text/plain, text/html,
      application/json; binary REJECTED with HTTP 415
    - 1 MB payload cap (configurable via GBRAIN_INGEST_MAX_BYTES)
    - Caller-overridable source identity via X-Gbrain-Source-Id /
      X-Gbrain-Source-Uri / X-Gbrain-Content-Type / X-Gbrain-Slug
      headers — useful for downstream tools that want clean provenance
    - untrusted_payload: true ALWAYS (network input)
    - Idempotency on (client_id, content_hash) so simultaneous retries
      collapse to one job
    - maxWaiting: 50 per client so a runaway integration can't
      monopolize the queue
    - Audit row in mcp_request_log + SSE broadcast for the admin feed

TESTS
test/ingestion/ingest-capture.test.ts (15 cases against PGLite):
- defaultSlugForEvent helper (3 cases pinning shape + UTC + determinism)
- slug resolution fallback chain (3 cases)
- validation + content-type routing (5 cases including binary rejection
  + untrusted_payload round-trip)
- importFromContent integration (3 cases including content_hash dedup
  via status='skipped' on repeat)

207 total ingestion tests passing. typecheck clean.

NEXT IN WAVE
cron-scheduler refactor + OpenClaw credential auto-migrate; content-type
processors (PDF + image OCR + audio transcribe + video keyframe);
put_page write-through + serializePageToMarkdown DRY extract +
migration v80 + bootstrap probes; gbrain capture verb; publisher DX
cathedral (init scaffold + gbrain ingest test --watch + tail + validate);
daemon rename autopilot -> ingest with forever-alias; doctor inotify
probe; skillpack contract docs + reference pack + VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): put_page write-through + migration v80 + DRY extract

WHAT YOU CAN NOW DO
The drift class is dead. Every `gbrain put_page` (CLI or MCP, local
or remote) now lands its markdown file on disk alongside the DB row
whenever `sync.repo_path` is configured. The page is queryable
immediately AND visible to git, your editor, and downstream tools.

Pre-v0.38, put_page wrote ONLY to the DB and synthesize/extract paths
had to reverse-render later. The v0.35.6.0 phantom-redirect pass was
the cleanup for what THIS commit prevents in the first place.

  # local CLI
  gbrain put inbox/test < my-thought.md
  # file lands at ${sync.repo_path}/inbox/test.md AND in the DB

  # MCP remote (Zapier / Cursor / Claude Desktop)
  curl -X POST /mcp ... '{"method":"tools/call","params":{"name":"put_page",...}}'
  # server-side write-through fires, agent gets a normal success response
  # untrusted_payload tagging applied (no auto-link, slug-allowlist gate)

Provenance frontmatter stamped on every write so future sync round-trips
know where the page came from:
  ingested_via: put_page         # local CLI
  ingested_via: 'mcp:put_page'   # MCP remote
  ingested_at: 2026-05-21T04:...

WHAT THIS COMMIT BUILDS
1. Migration v80 — `pages_provenance_columns` adds four nullable
   columns to `pages`: `ingested_via`, `ingested_at`, `source_uri`,
   `source_kind`. ADD COLUMN with no DEFAULT is metadata-only on
   Postgres 11+ and PGLite 17.5; instant on tables of any size. The
   four columns get NULL on every historical page (pre-v0.38 pages
   never had provenance).

2. DRY extract — `serializePageToMarkdown(page, tags, opts)` and
   `resolvePageFilePath(brainDir, slug, sourceId)` in `src/core/markdown.ts`.
   The dream-cycle's `renderPageToMarkdown` (synthesize.ts) and the new
   put_page write-through path were going to have 90% duplicate bodies.
   They now share one foundation; the dream version is a 4-line wrapper
   that passes `frontmatterOverrides: {dream_generated: true, ...}`.
   Future markdown-shape changes happen in one place.

3. put_page write-through (`src/core/operations.ts`) — after
   importFromContent succeeds, resolves sync.repo_path, computes the
   v0.32.8 source-aware path layout (default: brainDir/<slug>.md;
   non-default: brainDir/.sources/<id>/<slug>.md), serializes the
   freshly-written Page via `serializePageToMarkdown`, writes the file.
   Returns a `write_through: {written, path}` field in the put_page
   response so callers can see what happened.

   Trust gating:
   - subagent sandbox (viaSubagent without allowedSlugPrefixes) → DB-only
   - dry-run → DB-only (handler's early-return short-circuits before
     write-through; documented via the dry_run response field)
   - no sync.repo_path configured → DB-only, skipped reason returned
   - sync.repo_path points at a non-existent dir → DB-only, skipped
   - all other writes → write-through

   Failure isolation: disk-write failures are LOGGED loud but do NOT
   roll back the DB write. DB is the durable record; the
   phantom-redirect pass exists for drift cleanup if it ever shows up.

TESTS
- test/ingestion/put-page-write-through.test.ts (10 cases against PGLite):
  happy path (file land, provenance stamp local + remote), trust gating
  (subagent sandbox, dry-run, trusted-workspace), config edges (no
  repo_path, missing dir), multi-source filing (.sources/<id>/),
  failure isolation (DB write survives a disk failure).
- Migration v80 verified across both engines via the existing
  test/migrate.test.ts + test/bootstrap.test.ts coverage (~125 cases).

369 total tests passing in the ingestion + markdown + migrate bundle.
typecheck clean.

NOTES
- Bootstrap probes for the v80 provenance columns are NOT yet added
  to applyForwardReferenceBootstrap on either engine. This is safe
  for v0.38 because no SCHEMA_SQL CREATE INDEX or FK references the
  new columns — migration v80 is the only consumer, and it runs
  AFTER SCHEMA_SQL replay. A future commit may add bootstrap probes
  + REQUIRED_BOOTSTRAP_COVERAGE entries as defense-in-depth (eng
  review E4).

- The trusted-workspace path (dream cycle's reverseWriteRefs in
  synthesize.ts) still runs its own write at synthesize phase time.
  Both paths writing the same file is idempotent (byte-identical
  serialization), but a future commit may simplify reverseWriteRefs
  to skip pages whose file already matches.

NEXT IN WAVE
gbrain capture verb (the single human-facing entrypoint); daemon
rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): gbrain capture — the single human-facing entrypoint

WHAT YOU CAN NOW DO
One command, local or thin-client, synchronous receipt with the resulting
page slug. The answer to "what is the best way to get data into the brain?"
is now: just type `gbrain capture` and the right thing happens.

  # the basic case
  gbrain capture "remember to follow up on the X deal"

  # from a file
  gbrain capture --file ./notes/today.md --slug daily/2026-05-21

  # from a pipe (shell pipelines)
  echo "from stdin" | gbrain capture --stdin

  # script-friendly: print just the slug
  SLUG=$(gbrain capture "a thought" --quiet)

  # JSON for agents
  gbrain capture "..." --json

Default slug is `inbox/YYYY-MM-DD-<hash8>` — deterministic for the same
content so re-running idempotently lands the same page. Receipt block
on stdout shows slug + status + content_hash + on-disk path so you
can confirm where the page went without rerunning `gbrain query`.

The local-install path routes through the put_page operation with the
v0.38 write-through plumbing landed in the prior commit, so the page
hits both the DB AND the file tree in one move. Thin-client installs
route through `callRemoteTool('put_page', ...)` so the server's
write-through handles disk persistence the same way.

WHAT THIS COMMIT BUILDS
- src/commands/capture.ts (new ~290 LOC):
  - `defaultSlug(content)` — UTC-stable `inbox/YYYY-MM-DD-<hash8>`
  - `parseArgs(args)` — positional + flag parsing with --file / --stdin
    / --slug / --type / --source / --quiet / --json / --help
  - `buildContent(rawBody, opts)` — wraps unstructured prose in
    frontmatter (type + title + captured_via + captured_at) and a
    leading `# Title` heading; passes through if the body already
    looks like markdown
  - `runCapture(engine, args)` — local install routes through the
    in-process put_page operation; thin-client routes through MCP.
    `--quiet` prints just the slug; `--json` prints structured output;
    default prints a 5-line receipt block.

- src/cli.ts:
  - Adds `case 'capture'` dispatch
  - Adds `'capture'` to the CLI_ONLY set so cli.ts wires it correctly

TESTS
test/commands/capture.test.ts (21 cases against PGLite):
- defaultSlug helper: shape + determinism + UTC math
- parseArgs: positional + multi-token join + every flag
- buildContent: prose wrapping, --type override, no double-wrap
  for pre-frontmattered content, title cap at 80 chars,
  --source provenance stamp
- Integration: inline content lands in DB + on disk, default slug
  shape, --file reads from disk, --json structured output,
  --help returns without engine roundtrip

271 total tests passing in the bundle. typecheck clean.

NOTES
- Thin-client routing relies on `callRemoteTool('put_page', ...)` from
  src/core/mcp-client.ts. Identical UX to the local path because the
  server's put_page handler runs the same write-through plumbing.

- buildContent's "looks like markdown" heuristic is intentionally
  simple — first-line heading or frontmatter delimiter is the trigger.
  Users who care about exact formatting pass a pre-formatted --file.

NEXT IN WAVE
Daemon rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): v0.38.0.0 release hygiene + e2e roundtrip + capture skill

VERSION 0.37.1.0 → 0.38.0.0 (trio: VERSION, package.json, CHANGELOG header).
CHANGELOG entry written in user-facing ELI10-lead voice per CLAUDE.md
release-summary rules. README's pre-loop section gains a new "How to get
data in (v0.38+)" block leading with `gbrain capture`.

skills/capture/SKILL.md (NEW) so agents route "capture this" / "save this
thought" / "remember this" / "drop this in the inbox" / "save to brain" to
the capture verb. RESOLVER.md updated with the new triggers (sits above
idea-ingest/media-ingest/meeting-ingestion in the content-ingestion
section as the "simple thought" path).

E2E roundtrip test (test/e2e/ingestion-roundtrip.test.ts) covers the gap:
inbox-folder source -> daemon -> ingest_capture handler -> DB page,
including:
- Full pipeline: file drop appears as page in DB + file moves to .archived/
- Dedup catches byte-identical content from a different filename
- Multi-source coordination: two distinct inbox dirs, two sources, daemon
  ingests both events independently

The test runs against an in-memory PGLite (no DATABASE_URL needed) so it
exercises the substrate-level wiring in the standard test suite. A
follow-up commit can add a full-process e2e (gbrain serve --http + real
OAuth client + POST /ingest) that requires DATABASE_URL.

399/399 v0.38 wave tests passing (910 assertions). typecheck clean.
bun run verify gate green across all 14 shell checks.

DEFERRED TO FOLLOW-UP RELEASES (called out in CHANGELOG)
- Daemon rename autopilot -> ingest + forever-alias + plist migration
- cron-scheduler skill refactor + OpenClaw credential auto-migrate
- Content-type processors (PDF / OCR / audio / video)
- gbrain doctor inotify probe (Linux)
- Publisher DX cathedral: gbrain skillpack init --kind=ingestion-source,
  gbrain ingest test --watch, ingest tail, ingest validate
- Reference pack at examples/skillpack-ingestion-reference/ + 3-stage
  tutorial in docs/ingestion-source-skillpack.md

These are polish items; the substrate is shipped and queryable, and
skillpack publishers can build sources against the IngestionTestHarness
public export today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ingestion): test-gap fills — bootstrap probes + manifest entry + conformance

The v0.38.0.0 release-hygiene commit landed cleanly against the v0.38 wave
suite but tripped 3 categories of full-suite tests. This commit fixes
each. The remaining failure (doctorReportRemote > "healthy" status) was
verified pre-existing via `git stash + bun test` and is not caused by
v0.38; left alone.

Fix 1 — `schema-bootstrap-coverage.test.ts` (s1)
The test parses MIGRATIONS for ALTER TABLE ADD COLUMN statements and
fails if any column is not covered by `applyForwardReferenceBootstrap`
on both engines. Migration v80's four provenance columns triggered
the failure. Bootstrap probes added to both engines + 4 entries
appended to REQUIRED_BOOTSTRAP_COVERAGE:

- src/core/pglite-engine.ts — 4 EXISTS probes + state field + needs
  flag + ALTER TABLE block when bootstrap fires
- src/core/postgres-engine.ts — same pattern
- test/schema-bootstrap-coverage.test.ts — 4 coverage entries

Fix 2 — `check-resolvable.test.ts` (s3 — orphan_trigger)
RESOLVER.md references skills via name; check-resolvable cross-checks
against skills/manifest.json. The new `capture` skill was missing the
manifest entry; added between `brain-ops` and `idea-ingest` so the
manifest order mirrors the resolver order.

Fix 3 — `skills-conformance.test.ts` (s8)
Every SKILL.md must have `## Contract`, `## Output Format`, and
`## Anti-Patterns` sections. skills/capture/SKILL.md was missing all
three (initial draft skipped them); now compliant with concrete
content per the v0.38 contract.

Fix 4 — `build-llms.test.ts` (s6)
README + CHANGELOG edits in the release-hygiene commit caused
llms-full.txt to drift behind. Regenerated via `bun run build:llms`.
Per CLAUDE.md: any user-facing docs edit MUST run build:llms before
push.

The full bun-test parallel runner now passes everywhere except the
pre-existing `doctorReportRemote > healthy status` failure (50/100
score on an empty fresh brain — this is a pre-v0.38 health-score
tuning issue and orthogonal to ingestion work).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(version): bump 0.38.0.0 → 0.38.1.0

Renumbers the in-flight ingestion-cathedral release to v0.38.1.0.
Trio (VERSION, package.json, CHANGELOG.md) bumped together.

bun run typecheck → clean.

* chore(version): bump 0.38.1.0 → 0.38.0.0

Master sits at 0.37.11.0; 0.38.0.0 is the natural next slot rather than
skipping a release. Trio (VERSION, package.json, CHANGELOG.md) bumped
together. Migration v81 + ingestion substrate stay identical — this is a
header-only renumber.

bun run typecheck → clean.

* test(ingestion): fill v0.38 test gaps — markdown helpers + migration v81 + webhook E2E

Three gaps surfaced from a v0.38 audit against what shipped vs what was
covered. All three filled:

1. **test/markdown-serializer.test.ts** (NEW, 19 cases) — pure-function
   coverage of `serializePageToMarkdown` + `resolvePageFilePath`, the
   DRY extract that the dream-cycle reverse-render and put_page
   write-through both consume. Pre-fix nothing pinned the
   frontmatter-override merge precedence, the type/title defaults, or
   the source-aware filing layout (default → `<brainDir>/<slug>.md`,
   non-default → `<brainDir>/.sources/<source_id>/<slug>.md`). Future
   schema-shape changes to either helper now surface immediately.

2. **test/migrate.test.ts — v81 cases** (10 new cases, two describe
   blocks) — structural assertions on `pages_provenance_columns`
   (four nullable columns, no NOT NULL, no DEFAULT, no index — the
   ADD COLUMN stays metadata-only) plus a PGLite round-trip that
   asserts the columns appear post-`initSchema`, accept direct UPDATEs,
   and survive the historical-page NULL scenario. The
   schema-bootstrap-coverage test already pinned the forward-reference
   probe contract; this fills the migrate.test.ts contract gap.

3. **test/e2e/serve-http-ingest-webhook.test.ts** (NEW, 16 cases) — HTTP
   contract coverage for POST /ingest. The pre-existing
   ingestion-roundtrip E2E explicitly notes "e2e (gbrain serve --http +
   POST /ingest + real OAuth) is a separate" thing — it covers the
   in-process daemon → handler → DB pipeline, NOT the real HTTP route.
   This file fills that gap. Spawns real gbrain serve --http against
   real Postgres, mints OAuth tokens with various scopes, exercises:
     - Auth gate (missing → 401; read-only → 403)
     - Body validation (empty → 400 with error: empty_body)
     - Content-type allowlist (image/png → 415 with skillpack hint;
       application/pdf → 415; text/plain + application/json + text/html
       all accepted; unknown text/* falls through to text/plain)
     - X-Gbrain-Content-Type / Source-Id / Source-Uri / Slug header
       overrides
     - Idempotency (same content + same client = identical job_id via
       queue dedup on content_hash)

Also wires three new entries into `scripts/e2e-test-map.ts` so changes
to `src/commands/serve-http.ts`, `src/core/ingestion/**`, or the
`ingest-capture` Minion handler auto-trigger the relevant E2Es under
`bun run ci:local:diff`.

Verified locally:
- bun test test/markdown-serializer.test.ts → 19/19 green
- bun test test/migrate.test.ts -t "v81" → 10/10 green
- bun test test/e2e/serve-http-ingest-webhook.test.ts (real Postgres on
  ephemeral 5435) → 16/16 green
- bun test test/select-e2e.test.ts → 24/24 green (selector test still
  honors the v0.38 entries)
- bun run typecheck → clean

E2E DB lifecycle handled per CLAUDE.md (spin up pgvector:pg16 on a free
port, bootstrap via `gbrain doctor --json`, run, tear down).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:00:42 -07:00
Garry TanandClaude Opus 4.7 d0d0e2a64a v0.37.11.0: fresh-install PGLite embedding setup fix wave (#1286) 
* chore(test): preload gateway to OpenAI/1536 so 1536-dim test fixtures keep working

The v0.37 fix wave changes the canonical gateway defaults to
zeroentropyai:zembed-1 / 1280 (matching what v0.36 already chose as the
system default). 20+ test files have hardcoded new Float32Array(1536)
fixtures that match the OLD schema default. Without this preload, those
tests fail with a vector-dim-mismatch on insert.

The preload is gateway-only — it doesn't change which model gbrain ships
to production users. Tests that want the new ZE/1280 defaults call
configureGateway() explicitly in their own beforeAll.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): canonical embedding defaults + sweep across schema/engines/registry

Closes the v0.36 defaults drift bug class. The gateway shipped
zeroentropyai:zembed-1 / 1280 as the system default in v0.36 but eight
other places kept hardcoding 1536 / text-embedding-3-large. Fresh
gbrain init --pglite sized the column to 1536, the embed pipeline used
ZE/1280, and every page failed with dim mismatch.

- New src/core/ai/defaults.ts leaf module is the canonical source for
  DEFAULT_EMBEDDING_MODEL / DEFAULT_EMBEDDING_DIMENSIONS. Schema and
  registry helpers import from this lean module instead of pulling the
  full gateway (which loads every provider SDK).
- src/core/ai/gateway.ts re-exports the constants for back-compat.
- src/core/pglite-schema.ts getPGLiteSchema() defaults track gateway.
- src/core/postgres-engine.ts getPostgresSchema() default args track
  gateway (same drift on the Postgres path — codex round 1 CDX-1).
- Both engine.initSchema() fallbacks track gateway constants (no more
  stale OpenAI/1536 catch-block defaults).
- Schema seed stops stripping the provider prefix; full provider:model
  is stored in the DB config table (codex round 1 CDX-4).
- Chunk-row INSERT defaults track gateway (codex round 2 CDX2-4 —
  pglite-engine:1611 + postgres-engine:1647 were production write
  sites previously hardcoded to text-embedding-3-large).
- src/core/search/embedding-column.ts loadRegistry + isCacheSafe gain
  the cfg > gateway > DEFAULT resolution chain (codex round 2 CDX2-3).
  The gateway tier matters because callers that configure the gateway
  (init paths, tests, programmatic SDK) expect the registry to mirror
  that state when cfg doesn't have an explicit embedding_model.

Tests:
- schema-templating: default expectation flips to ZE/1280 (v0.37 truth).
- embedding-dim-check: 3 new engine-kind branching cases + updated
  fresh-brain expectation (under legacy preload).
- embedding-column: registry + isCacheSafe expectations match new chain.
- v0_28_5-fix-wave E2E: engineKind required arg propagated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(init+config+cli): always-configure gateway, file-only loader, honest config-set, sync/reinit help

Closes the "fresh init doesn't work + config-set silently lies" bug
class end-to-end. Six related changes that ship together because the
file-plane/DB-plane contract only holds when init paths, config-set,
the gateway env mapping, and the recipe text all agree.

Lane B (init paths):
- initPGLite, initPostgres, initMigrateOnly always configureGateway()
  before engine.initSchema(). Pre-fix the call was gated on flags, so
  bare `gbrain init --pglite` left the gateway unconfigured and the
  engine fell through to stale OpenAI/1536 defaults instead of the
  ZE/1280 the gateway would have resolved.
- New configureGatewayWithMergedPrecedence() helper applies the locked
  precedence chain `CLI > env > existing file > gateway internal`.
- printResolvedAIChoice() shows the resolved model/dim at init time +
  surfaces a ZE setup hint inline when the API key is missing.
- B.4: saveConfig merge uses loadConfigFileOnly() so transient env
  state (DATABASE_URL, etc.) never poisons ~/.gbrain/config.json
  (codex round 2 CDX-5).
- B.5: extend the v0.28.5 dim-mismatch detector so it fires when the
  gateway-resolved dim differs from the existing column, not only
  when --embedding-dimensions is explicit (codex round 2 CDX-6).

Lane C (config plane):
- New `loadConfigFileOnly()` reads ~/.gbrain/config.json only — no env
  merge, no DATABASE_URL inference. Safe write-back source for init.
- GBrainConfig gains `zeroentropy_api_key?: string`. loadConfig merges
  process.env.ZEROENTROPY_API_KEY. buildGatewayConfig at cli.ts:1401
  maps it into env.ZEROENTROPY_API_KEY so ZE recipes finally see it
  (codex round 2 CDX2-5+6 — the v1 fix landed in the wrong file).
- `gbrain config set embedding_model` and `... embedding_dimensions`
  refuse unconditionally and print a paste-ready wipe-and-reinit
  recipe. No --force escape (codex round 2 CDX2-13).
- migrate-engine.ts adds a contract comment at the DB-plane write
  site documenting "DB stores schema-applied metadata; file plane is
  canonical for runtime gateway config" + preserves the existing
  file-plane config across engine migration.

Lane D.1 (recipe text):
- embeddingMismatchMessage() takes an `engineKind` arg. PGLite branch
  emits a wipe-and-reinit recipe using gbrainPath('brain.pglite') or
  the caller's databasePath override. Postgres branch keeps the SQL
  ALTER recipe.
- The PGLite recipe recommends `gbrain reinit-pglite` (new sugar
  command below) as the one-line path before falling back to the
  by-hand mv + init + sync sequence.

Lane D.4 (sync help dispatch):
- `sync` and `reinit-pglite` added to CLI_ONLY_SELF_HELP so their own
  --help branches reach the user (pre-fix the generic short-circuit
  fired first and the dedicated usage was unreachable; codex round 2
  CDX2-12).
- `gbrain sync --help` short-circuits BEFORE engine bind so users on
  a fresh tmpdir (no config) can read the help without hitting
  no-such-config errors.

Sugar:
- New `gbrain reinit-pglite --embedding-model X --embedding-dimensions N`
  wraps the wipe + init + sync dance into one command. Backs up the
  brain to <path>.bak. TTY confirmation unless --yes. --no-sync to
  defer the resync. --json for scripts.

Tests:
- test/cli.test.ts sync-help test rewritten for the new
  per-command-usage output (lists --no-embed which is the v0.37
  user-visible flag the wave wanted to surface).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(embed+sync): pre-flight dim-mismatch guard + sync hint at both catch sites

embedding-pipeline error UX. Pre-fix, a fresh-install dim mismatch
produced raw Postgres "expected N dimensions, not M" errors page after
page, surfacing only after the worker pool drained the entire corpus.
Sync swallowed embed errors at TWO catch sites and never surfaced
the recovery recipe.

embed.ts:
- New `EmbeddingDimMismatchError` tagged class with the paste-ready
  recipe baked in.
- `runEmbedCore` pre-flights via `readContentChunksEmbeddingDim` +
  gateway.getEmbeddingDimensions() before the worker pool spins up.
  On mismatch, throws the typed error which the CLI wrapper catches
  and prints. Dry-run skips the check (no embed risk).
- Catches the headline fresh-install bug class at first call instead
  of letting it hammer N parallel API calls into dim-rejected inserts.

sync.ts:
- Both embed catches at sync.ts:990 (incremental) and sync.ts:1129
  (first-sync) detect EmbeddingDimMismatchError and surface the recipe
  + a `--no-embed` tip on stderr (codex round 2 CDX2-8: incremental
  path was previously silent; only the first-sync path was flagged).
- Non-mismatch embed failures still stay best-effort (rate limits,
  transient network) — those shouldn't break sync.
- Sync calls runEmbedCore directly instead of runEmbed (which calls
  process.exit on error and bypasses sync's catch).
- Sync gets a proper --help block listing every meaningful flag:
  --no-embed, --workers, --source, --skip-failed, --retry-failed,
  --watch, --interval, --no-pull, --all, --json, --yes, --dry-run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): read gateway for schema-sizing checks + provider-aware key lookup

Doctor's embedding checks were reading the DB config table for
embedding_model / embedding_dimensions / zeroentropy_api_key. Post
v0.37 the file plane is canonical (the DB plane is schema-applied
metadata, not runtime gateway config) so those reads produced stale
verdicts on fresh installs whose DB row hadn't been written.

- checkEmbeddingWidthConsistency reads gateway.getEmbeddingDimensions()
  and gateway.getEmbeddingModel() instead of engine.getConfig(...).
  Reuses readContentChunksEmbeddingDim from the same shared helper
  init + embed use. On mismatch, the fix hint threads engineKind +
  databasePath into the new branched recipe (codex round 1 CDX-8 +
  Lane E.1/E.2).
- checkZeEmbeddingHealth reads gateway for the model + loadConfigFileOnly
  for the key. Fires when (a) resolved model starts with zeroentropyai:
  AND (b) ZEROENTROPY_API_KEY is unset in env AND (c) file plane has
  no zeroentropy_api_key (codex round 2 CDX2-10).
- loadRecommendationContext reads gateway for both fields and
  recognizes the ZE key alongside OpenAI/Anthropic in the
  hasEmbeddingApiKey check, so brains on ZE no longer look "healthy"
  just because OPENAI_API_KEY happens to be set (codex round 2 CDX2-11).

Tests rewritten for the gateway-source-of-truth contract via
configureGateway() in beforeAll. Added a "gateway unconfigured: skips
with ok" case so doctor doesn't false-warn on cold-boot brains.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test+docs(v0.37): fix-wave unit coverage + PGLite-first migration recipe + TODOS

Lands the v0.37 PGLite fresh-install fix wave's structural tests and
the user-facing migration recipe overhaul.

test/v0_37_fix_wave.test.ts (new): 22 unit cases pinning the lanes:
- Lane A: defaults module exports, getPGLiteSchema/getPostgresSchema
  default-args, registry + isCacheSafe under the `cfg > gateway >
  DEFAULT` chain (both gateway-set and gateway-reset branches).
- Lane B: loadConfigFileOnly env isolation + DATABASE_URL inference
  refusal + null-on-missing.
- Lane C.3: buildGatewayConfig maps zeroentropy_api_key + process.env
  wins over config (operator escape hatch contract).
- Lane D.2: EmbeddingDimMismatchError shape + tag.
- Lane D.4: structural assertion that `sync` is in CLI_ONLY_SELF_HELP.
- Deferred-TODO ship: reinit-pglite is registered correctly +
  embeddingMismatchMessage PGLite branch recommends it.

docs/embedding-migrations.md: PGLite section moved to top (the default
install). The recommended path is `gbrain reinit-pglite` one-liner;
the by-hand mv + init + sync sequence stays as the fallback recipe.
Postgres SQL ALTER recipe preserved. New section on `gbrain config
set` refusal explains the file-plane vs DB-plane contract so users
don't follow stale documentation.

TODOS.md: 4 deferred follow-ups filed with concrete file pointers:
- gbrain embed --try-fallback (provider auto-switch with consent gate)
- Full plane unification for non-schema-sizing fields
- Worker-pool shared AbortController for mid-run dim drift
- Cleanup of back-compat constants in src/core/embedding.ts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(v0.37): fill behavior gaps + headline fresh-install E2E

The structural fix-wave tests in test/v0_37_fix_wave.test.ts pin lane-level
invariants (exports, registry chain, signature shapes). The audit found 10+
END-TO-END behaviors that the structural tests didn't actually reach.
This file fills the highest-leverage gaps.

Unit coverage (test/v0_37_gap_fill.test.ts, 12 cases):
- Lane A.7: chunk-row INSERT default tracks DEFAULT_EMBEDDING_MODEL
  constant (pre-fix this was the literal 'text-embedding-3-large' at
  pglite-engine.ts:1611 + postgres-engine.ts:1647 — production write
  sites that were never directly tested; codex round 2 CDX2-4).
- Lane A.8: schema seed stores full provider:model in DB config
  (pre-fix the .split(':') strip dropped the prefix; codex round 1
  CDX-4). Asserts a fresh ZE init stores `zeroentropyai:zembed-1`
  in the config table, not bare `zembed-1`.
- Lane B precedence: explicit CLI > env > existing file > default
  test (codex round 2 CDX2-7 contradiction guard).
- Lane C.3 env merge: process.env.ZEROENTROPY_API_KEY threads through
  loadConfig → cfg.zeroentropy_api_key; loadConfigFileOnly does NOT.
- Lane D.2 end-to-end: schema=1536 + gateway=1280 →
  EmbeddingDimMismatchError fires AND the embed transport is never
  called (the whole point of pre-flight). Plus dry-run skips the
  check.
- Lane D.3 source-text grep: both sync.ts catch sites detect the
  typed error + the `--no-embed` tip is present (CDX2-8).
- Lane E.4 source-text grep: loadRecommendationContext is
  provider-aware (reads gateway + branches on ZE/OpenAI key).
- reinit-pglite contract: refuses on non-PGLite engines + refuses
  when required flags are missing.

E2E (test/e2e/fresh-install-pglite.test.ts, 2 cases):
- Bare `gbrain init --pglite` produces a `vector(1280)` schema, prints
  the resolved choice, persists defaults to config.json — the headline
  scenario that v0.37 ships to fix.
- init → seed page → embed end-to-end: chunks have non-null
  embeddings; no dim mismatch despite the wave's defaults change.

Both E2E cases are IN-PROCESS (per CDX2-12: CLI-subprocess E2E can't
inherit `__setEmbedTransportForTests`). They run with stubbed transport
returning synthetic 1280-dim vectors so we never hit real provider APIs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(v0.37): defensive gateway restore in reinit-pglite describe block

Adds an afterAll that restores the gateway to OpenAI/1536 (matching the
bunfig preload) at the end of the reinit-pglite describe. Belt-and-
suspenders: earlier describe blocks in this file already restore, but
if the reinit-pglite tests ever start mutating the gateway in the
future, this protects downstream test files in the same bun-test shard
from inheriting a non-default state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.37.10.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: scrub stale config-set recipes for embedding model (v0.37.10.0)

README + topologies + embedding-providers were still pointing users at
`gbrain config set embedding_model X` / `embedding_dimensions N`. As of
v0.37.10.0 those writes are refused — the schema column has to resize
alongside the config. Point at `gbrain reinit-pglite` (PGLite) and the
SQL recipe in `docs/embedding-migrations.md` (Postgres) instead.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: bump version to v0.37.11.0

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test: quarantine v0.37 fix-wave tests to .serial.test.ts

CI's `check:test-isolation` lint flagged R1 violations (direct
`process.env.GBRAIN_HOME` mutation) in both new fix-wave test files.
Per the documented quarantine pattern in CLAUDE.md, rename to
`*.serial.test.ts` instead of refactoring through `withEnv()` — both
files use beforeEach/afterEach env wiring that's already serial-safe.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 22:11:00 -07:00
Garry TanandClaude Opus 4.7 a55de71221 v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (#1278) 
* feat(core): Levenshtein helper + preflight schema-dim resolvers

Foundation for v0.37.10.0 env-detection wave. Two pure modules:

- src/core/levenshtein.ts: editDistance(a,b) + suggestNearest(input, candidates, maxDistance).
  Used by config-set "did you mean" suggestions and env-var typo detection at init.
- src/core/embedding-dim-check.ts: resolveSchemaEmbeddingDim() +
  resolveSchemaMultimodalDim() pure functions. Validate resolved dim against
  recipe default_dims + per-provider Matryoshka allow-lists (OpenAI text-3,
  Voyage flexible-dim, ZeroEntropy zembed-1) BEFORE any DB write. Plus
  EmbeddingDisabledError + assertEmbeddingEnabled() runtime guard for the
  deferred-setup path (D9). New PGVECTOR_COLUMN_MAX_DIMS=16000 exported.

Tests: 41 unit cases across both modules.

* feat(providers): extract formatRecipeTable + add init provider picker

Two changes prepping the env-detection wave:

- providers.ts: extract formatRecipeTable() helper from runList(). Picker
  reuses it so UI can't drift from \`gbrain providers list\`. Also adds the
  codex finding #10 warn-line to \`providers test\` when the tested model
  differs from the configured default ("Note: tested X in isolation;
  gbrain's configured embedding is Y — this test does NOT verify your
  brain's active path."). envReady() takes an explicit env arg for testing.

- init-provider-picker.ts (NEW): interactive picker mirroring
  init-mode-picker.ts. Filters candidate recipes to env-ready ones
  (codex finding #3), prompts via readLineSafe, exports
  printSubagentAnthropicCaveat() for shared use from initPGLite/initPostgres.

Tests: 17 unit cases (10 providers + 7 picker).

* feat(config): embedding_disabled sentinel + strict unknown-key rejection

Two changes for the v0.37.10.0 wave:

- src/core/config.ts: add embedding_disabled?:boolean to GBrainConfig (D9
  deferred-setup sentinel, mutually exclusive with embedding_model). Export
  KNOWN_CONFIG_KEYS (60+ canonical keys, file-plane + DB-plane) and
  KNOWN_CONFIG_KEY_PREFIXES (search., models., dream., cycle., etc.) for
  validation use.

- src/commands/config.ts: D6 strict-default unknown-key rejection.
  Unknown key + no --force → exit 1 with Levenshtein suggestion against
  KNOWN_CONFIG_KEYS. Prefix matches accepted without --force. --force
  escape hatch accepts arbitrary keys with stderr WARN. Closes the
  silent-no-op class the bug reporter hit (embedding.provider,
  embedding.model, embedding.dimensions all exit 1 with right suggestion).

Tests: 19 unit cases pinning the bug-reporter regression + gate logic.

* feat(init): env-detection auto-pick + preflight + atomic persist + --no-embedding

Core of the v0.37.10.0 wave (D1-D7, D9-D11). Closes the bug where a fresh
\`gbrain init --pglite\` silently produced a broken brain when no provider
key matched the v0.36 default.

resolveAIOptions rewritten with per-touchpoint env detection:
- Explicit flag → shorthand → env auto-pick (group by provider id, codex #2)
- Picker fires when multiple providers env-ready (D1+D2 hybrid)
- Non-TTY zero-key exits 1 with paste-ready setup hint (D3) + Levenshtein
  typo detection for OPENAPI_API_KEY → OPENAI_API_KEY (D13)
- All three touchpoints covered (embedding + expansion + chat, D4)
- Local-only providers (Ollama/llama-server) excluded from auto-pick;
  picking Ollama silently when user has OPENAI_API_KEY set was wrong UX

initPGLite + initPostgres:
- Drop conditional configureGateway gate → always call before initSchema
- Preflight resolveSchemaEmbeddingDim() BEFORE engine.initSchema() (D11) —
  invalid dim refuses with paste-ready hint, no disk write
- Atomic embedding-config persistence (codex #13): either resolved tuple
  or embedding_disabled:true sentinel, never partial state
- Post-initSchema invariant assertion stays as regression guardrail
- --no-embedding opt-in flag (D9) for deferred-setup mode
- Subagent-Anthropic caveat (D7) fires post-init when chat_model is
  non-Anthropic AND ANTHROPIC_API_KEY missing

Exported groupReadyByProvider() + findEnvKeyTypos() for unit testing.

Tests: 21 unit cases covering provider grouping + typo detection edge cases.

* feat(embed,import): refuse cleanly when --no-embedding deferred-setup is active

T7 of the v0.37.10.0 wave. Both runEmbedCore and runImport now call
assertEmbeddingEnabled(loadConfig()) at entry. When the brain was init'd
with --no-embedding (config has embedding_disabled:true), they exit 1
with a paste-ready hint:

  gbrain config set embedding_model <provider>:<model>
  gbrain config set embedding_dimensions <N>
  gbrain init --force --embedding-model <provider>:<model>

\`gbrain import --no-embed\` flag still works (chunks land without vectors),
so users can still ingest in deferred-setup mode and backfill embeddings
later with \`gbrain embed --stale\`.

* feat(doctor): empty-config drift detection + subagent-Anthropic caveat extension

Two doctor check extensions for v0.37.10.0:

T9 — embedding_provider check extended for the v0.36 silent-default
repair case. When config is empty AND schema column dim differs from the
gateway-resolved default, surface the mismatch with empty-brain vs
non-empty-brain repair branching (codex finding #7 nuance):
- Empty brain (0 embedded chunks) → \`gbrain init --force --pglite
  --embedding-model <id> --embedding-dimensions <N>\` (drop and re-init)
- Non-empty brain → \`gbrain retrieval-upgrade --to <id> --reindex\`
Gated on totalChunks > 0 so pristine empty brains aren't pre-warned.
Never recommend rm -rf ~/.gbrain.

T10 — subagent_provider check (v0.31.12) extended per D7. When chat_model
is non-Anthropic AND ANTHROPIC_API_KEY is missing, warn that subagent
features (gbrain dream, gbrain agent run, gbrain autopilot) will fail at
job submission. Chat alone (gbrain think) still works.

* feat(reindex,test): multimodal preflight + E2E suite for fresh PGLite init

T11 — reindex-multimodal.ts: hook resolveSchemaMultimodalDim() preflight
BEFORE the reindex sweep. Mirrors the text-side contract from initPGLite —
if the configured multimodal model can't produce a dim matching the schema
column, fail loud here with a \`gbrain config set\` hint rather than
mid-reindex with a vector(N) INSERT error.

T12 — test/e2e/init-fresh-pglite.test.ts (NEW, 14 cases): subprocess-driven
E2E verification of the bug-reporter's repro scenarios:
- Happy path: OPENAI_API_KEY set → auto-pick OpenAI, persists config
- D3 non-TTY fail-loud (with and without env-key typos)
- D6 regression: bug-reporter's three no-op config keys all exit 1 with
  Levenshtein suggestions
- D9 deferred-setup mode + gbrain import refusal (and --no-embed bypass)
- D11 preflight refuses BEFORE any disk write
- Explicit --embedding-model wins over env detection

Each test uses its own throw-away GBRAIN_HOME for hermetic runs.

* docs: env-detection + headless-install + close v0.32 picker TODO

T13/T14 docs sync for v0.37.10.0:

- docs/integrations/embedding-providers.md: TL;DR table refreshed to reflect
  ZE as v0.36 default; added "Init resolves your provider from env keys"
  section explaining the auto-pick → picker → fail-loud chain; added
  "If first import fails" troubleshooting block pointing at gbrain doctor
  instead of \`rm -rf ~/.gbrain\`.

- docs/operations/headless-install.md (NEW): Docker/CI sequencing guide.
  Two acceptable patterns — provider key at build time (Pattern 1) or
  --no-embedding opt-in + runtime config (Pattern 2). Codex finding #11.

- README.md: Troubleshooting section with one-paragraph repair hint and
  links to embedding-providers.md + headless-install.md.

- TODOS.md: closed v0.32.x "interactive provider chooser" entry as
  SUPERSEDED by this wave. Added four follow-up entries (dedicated v0.36
  broken-install migration, namespaced ext fields, runtime config-key
  audit, value-level Levenshtein on config set).

* chore: bump version and changelog (v0.37.10.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(doctor): empty brain scores 100/100 + hermetic doctor-report-remote test

Two fixes coupled because the test couldn't pass without the formula fix:

src/core/pglite-engine.ts + src/core/postgres-engine.ts — empty brain
(pageCount === 0) now gets FULL marks (100/100), not 0/100. Semantically
an empty brain has no coverage problem to penalize — there's nothing to
embed, nothing to link, nothing to orphan. Vacuous truth applies. The
pre-fix "empty = 0" caused fresh-init brains to score as critically
unhealthy on \`gbrain doctor\`, which was a structural surprise to users
who'd just run init successfully. Same fix on both engines.

test/brain-score-breakdown.test.ts — updated the "empty brain" assertion
to match the new contract (was: 0/0/0/0/0/0; is: 100/35/25/15/15/10).

test/doctor-report-remote.test.ts → renamed to .serial.test.ts and made
hermetic. The pre-fix test pulled audit data from the host ~/.gbrain
(reranker_health, sync_failures, etc.), which made the assertion
non-deterministic depending on whoever ran the suite. Now isolates
GBRAIN_HOME to a tempdir via beforeAll/afterAll; env mutation requires
serial-quarantine per scripts/check-test-isolation.sh R1.

Closes the master-state flake that was failing on every \`bun run test\`
run regardless of my branch contents.

* docs: update CLAUDE.md and TODOS.md for v0.37.10.0 empty-brain fix

- CLAUDE.md: annotate src/core/pglite-engine.ts + src/core/postgres-engine.ts
  entries with v0.37.10.0 empty-brain 100/100 contract. Vacuous truth: an
  empty brain has no coverage to penalize, so getBrainScore returns full
  marks (35/25/15/15/10 breakdown) when pageCount === 0. Pre-fix 0/100
  was structurally surprising on fresh init and caused the v0.37.8.0
  doctor-report-remote.test.ts flake.
- TODOS.md: mark P0 doctor-report-remote.test.ts:65 TODO completed
  (resolved by commit 9aa571f3's empty-brain-100/100 fix; test renamed
  to .serial.test.ts and made hermetic per scripts/check-test-isolation.sh R1).
- llms-full.txt: regenerated from updated CLAUDE.md per CLAUDE.md "Auto-derived
  files" rule.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(init): D5 persisted-config-wins on re-init + CI mechanical E2E

Two coupled fixes for the v0.37.10.0 wave's interaction with CI's Tier-1
mechanical E2E suite (which runs without any embedding-provider env var).

src/commands/init.ts — Honor D5 properly at resolveAIOptions entry. Pre-fix
the env-detection branch fired on EVERY init regardless of persisted
config. A non-TTY re-init with no env keys exited 1 (D3 fail-loud) even
when ~/.gbrain/config.json already had embedding_model set from a prior
successful init. Now resolveAIOptions reads loadConfig() first and seeds
out.embedding_model / embedding_dimensions / expansion_model / chat_model
from the file plane BEFORE running env detection. Also honors
embedding_disabled (D9 sentinel) on re-init so deferred-setup brains
don't re-trigger fail-loud.

test/e2e/mechanical.test.ts:722 — Setup Journey's first init runs against
a fresh DB with no persisted config. Pass --embedding-model explicitly
(openai:text-embedding-3-large) so the preflight resolves offline. After
this init writes config, subsequent inits in the file (RLS self-heal v24,
RLS event-trigger probes, etc.) honor the persisted config via the D5
fix above.

Verified locally: full test/e2e/mechanical.test.ts → 78 pass / 0 fail.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-21 18:57:05 -07:00
Garry TanandClaude Opus 4.7 430d784a76 v0.37.6.0 feat(ai): OpenRouter recipe + generic default_headers seam (cherry-pick #1210) (#1246) 
* feat(ai): add default_headers / resolveDefaultHeaders seam to Recipe

Generalizes per-recipe header attachment so attribution headers (OpenRouter's
HTTP-Referer + X-OpenRouter-Title) ride alongside Bearer auth on every
openai-compatible touchpoint. Two safety guards fire at applyResolveAuth time:
declaring both default_headers AND resolveDefaultHeaders throws AIConfigError
(mutual exclusion); a default header whose key shadows the resolved auth
header (Authorization, the resolver's custom header) also throws.

Reranker HTTP path at gateway.ts:2281 now merges both Authorization Bearer AND
auth.headers (where default_headers flow) into the request Headers map.
Pre-fix the ternary picked one or the other; default_headers would have been
silently dropped on the manual rerank path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add OpenRouter provider recipe

One key, many hosted models. Configures openrouter:<provider>/<model> for
chat (GPT-5.2 family, Claude 4.5/4.6/4.7, Gemini 3 Flash Preview, DeepSeek)
and embedding (OpenAI text-embedding-3-small with Matryoshka dims_options).
max_batch_tokens=300_000 (OpenAI's aggregate per-request token cap, not the
per-input 8192 the original PR conflated).

resolveDefaultHeaders returns HTTP-Referer + X-OpenRouter-Title + X-Title
(back-compat alias) so traffic is attributed to gbrain on OR's leaderboard.
Forks override via OPENROUTER_REFERER / OPENROUTER_TITLE env vars.

supports_subagent_loop: false is informational — gbrain's subagent infra is
hard-pinned to Anthropic-direct via isAnthropicProvider() upstream regardless
of this flag. Filed as TODO to verify tool_use_id stability through OR.

Cherry-picked from PR #1210. Contributed by @davemorin.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): export buildGatewayConfig + thread OPENROUTER_BASE_URL

Exports buildGatewayConfig for unit-test access. Adds one-line passthrough
for OPENROUTER_BASE_URL matching the existing LITELLM/OLLAMA/LMSTUDIO/
LLAMA_SERVER pattern so users can point at a self-hosted OR-compatible
proxy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(ai): cover OpenRouter recipe + default_headers seam + wire-level headers

Four test additions:

- test/ai/recipe-openrouter.test.ts (11 cases) — recipe shape, Matryoshka
  dims_options, max_batch_tokens=300K, arbitrary-ID acceptance via
  assertTouchpoint, defaultResolveAuth happy/error, resolveDefaultHeaders
  defaults + fork-override path, setup_hint coverage. Shape regression on
  every chat/embedding model ID (catches typos without pinning the dynamic
  catalog).

- test/ai/recipes-existing-regression.test.ts (+6 cases) — IRON RULE
  preserved; adds default_headers contract: Bearer+defaults returns both
  apiKey AND headers, custom-header+defaults merges with resolver winning,
  mutual-exclusion guard, Authorization-shadow guard, custom-auth-shadow
  guard, cross-touchpoint parity for all four (embedding/expansion/chat/
  reranker).

- test/ai/header-transport.test.ts (3 cases) — proves headers actually reach
  the wire. Synthetic recipes with resolveOpenAICompatConfig fetch wrappers
  capture outgoing Headers on embed/chat/rerank. Asserts Authorization +
  HTTP-Referer + X-OpenRouter-Title + X-Title all present. Codex flagged
  the return-shape-only coverage gap during plan review.

- test/ai/build-gateway-config.test.ts (7 cases) — 5-way env-baseURL
  passthrough sweep through the now-exported buildGatewayConfig. Uses
  withEnv() from test/helpers/with-env.ts for isolation compliance. Mops
  up pre-existing untested drift on LLAMA_SERVER/OLLAMA/LMSTUDIO/LITELLM
  in the same pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: add OpenRouter to embedding-providers + bump recipe count

15 -> 16 recipes. Adds OpenRouter row to the TL;DR table, a setup section
covering the value-prop (one key, many hosted models), env-var overrides
(OPENROUTER_BASE_URL, OPENROUTER_REFERER, OPENROUTER_TITLE), the subagent-
loop limitation (isAnthropicProvider() gate), and a "One key for many
hosted models" bullet under the decision tree. README updated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump v0.37.2.0 version refs to v0.37.4.0 across in-tree comments

v0.37.2.0 was claimed by master's takes_resolution_consistency hotfix
(#1211) before this branch could land. This commit re-stamps the source
comments that reference the OpenRouter recipe / default_headers seam to
v0.37.4.0 so the in-tree version markers match the actual landing version.

No behavior change — comments only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.37.4.0)

One key, many hosted models — OpenRouter recipe lands. Cherry-picked from
#1210 (@davemorin), with codex review corrections folded in:
- recipe count math (16 not 17)
- current OR attribution header name (X-OpenRouter-Title, X-Title back-compat)
- max_batch_tokens semantic (300K aggregate per-request, not 8192 per-input)
- Matryoshka dims_options for text-embedding-3-small
- auth-shadow guard at applyResolveAuth

Adds the generic Recipe.default_headers / resolveDefaultHeaders seam so
attribution headers ride alongside Bearer auth. Future Together/Groq
adoption tracked in TODOS.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump to v0.37.6.0 (queue moved past v0.37.4/v0.37.5)

VERSION + package.json + CHANGELOG header + CLAUDE.md + TODOS.md + in-tree
source comments + llms regen. No code-behavior change.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 08:15:14 -07:00
Garry TanandClaude Opus 4.7 65ff663f7d v0.36.4.0 feat: brain-health-100 — autonomous remediation via doctor --remediate + Minions (#1193) 
* feat(schema): op_checkpoints table + doctor_run_id partial GIN (v67+v68)

T1 of brain-health-100 wave. Two new migrations underpin autonomous
remediation via Minions:

- v67 op_checkpoints — shared checkpoint table for long-running ops
  (embed, extract, lint, backlinks, reindex, integrity). Pre-fix each
  op had its own file-backed checkpoint or none. PRIMARY KEY (op,
  fingerprint) lets `extract links` and `extract timeline` (or
  `reindex --markdown` vs `--code`) coexist without colliding on
  shared keys.

- v68 minion_jobs_doctor_run_id_idx — partial GIN on
  `minion_jobs.data WHERE data ? 'doctor_run_id'`. Indexes only
  doctor-submitted jobs so audit-trail queries don't sequential-scan
  months of unrelated cron history. PGLite skips via empty sqlFor.

Applied to src/schema.sql + src/core/pglite-schema.ts so both engines
get the table on fresh-install. Bootstrap coverage test +
122-case migrate test both pass.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + folded scope B from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): op-checkpoint module — DB-backed checkpoint primitive

T2 of brain-health-100 wave. Six exports plus per-op fingerprint helpers:

  loadOpCheckpoint(engine, key)     → string[]   (completed keys; [] if none)
  recordCompleted(engine, key, ks)  → void       (UPSERT atomic)
  clearOpCheckpoint(engine, key)    → void       (clean-exit drop)
  resumeFilter(all, completed)      → string[]   (pure; drives batched walks)
  purgeStaleCheckpoints(engine, ttl)→ number     (cycle purge phase consumer)

Fingerprint helpers:
  fingerprint(params)               — sha8 of canonical-JSON
  embedFingerprint(p)               — model+dim+slug+source variation
  extractFingerprint(p)             — mode (links vs timeline)
  reindexFingerprint(p)             — markdown vs code vs slug + chunker_version
  lintFingerprint, backlinksFingerprint, integrityFingerprint, importFingerprint

Canonical-JSON over keys-sorted ensures the same params produce the
same fingerprint across runs and hosts. sha8 (8 hex chars from sha256)
is short enough for filenames + UI but collision-resistant for the
expected per-op invocation diversity.

DB-backed for both engines (PGLite has the table too via v67). Lost-
write on partial DB failure is non-fatal — caller continues, next run
re-walks (cheap for hash-short-circuited ops like embed/import).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + codex #10–16 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): brain-score-recommendations — shared data layer

T4 of brain-health-100 wave. Pure module — no engine I/O. Takes a
BrainHealth snapshot + RecommendationContext, returns ordered
Remediation[] ready to feed the doctor remediation plan OR features
--auto-fix.

Three public exports:
  computeRecommendations(health, ctx)  → Remediation[]
  classifyChecks(checks, ctx)          → CheckClassification[]
  maxReachableScore(health, classes)   → number (0-100 ceiling)

D13 — three-state classification per check: remediable / human_only /
blocked. The plan ONLY emits remediable items; blocked surfaces
alongside as informational with the missing prereq (no API key, etc.).
Closes the spin-loop bug on empty / API-key-missing brains (codex #20).

D14 — every Remediation has a stable string id (sync.repo, embed.stale,
backlinks.fix, extract.all). depends_on references ids, not check names.

D9 — idempotency_key is content-hash from canonical-JSON of params.
Same intent across runs = same key; failed-row replay via :r<N> suffix
is the --remediate loop's job, not this module's.

Scope item +A (cost-budget gate) — Remediation.est_usd_cost populated
for embed (chars × pricePerMTok from embedding-pricing.ts) and Anthropic
jobs (estimateAnthropicCost helper). doctor --remediate --max-usd N
gates submission against est_total_usd_cost.

Both consumers (doctor + features per D15) import from here. Features
executes inline (D15 contract preserved), doctor submits via queue.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(handlers): 11 new Minion handlers + 3 added to PROTECTED + sync noExtract fix

T5 of brain-health-100 wave.

PROTECTED_JOB_NAMES extension (D11): synthesize, patterns, consolidate.
These cycle phases internally submit `subagent` jobs with
allowProtectedSubmit=true, so they CAN spend Anthropic credits.
Treating them as "data-quality maintenance" was a misread surfaced by
the codex outside-voice review (#6). Protected gate ensures only
trusted local callers (CLI, autopilot, doctor --remediate) can submit;
an OAuth-scoped MCP client can't burn the user's API budget by
submitting a synthesize job over HTTP.

11 new handlers registered in jobs.ts registerBuiltinHandlers:

  PROTECTED (3) — phase-wrappers that spawn subagent children:
    synthesize, patterns, consolidate

  Open (8) — DB/fs writes only, no LLM spend:
    reindex, repair-jsonb, orphans, integrity, purge,
    extract_facts, resolve_symbol_edges, recompute_emotional_weight

Phase-wrappers all delegate to `runCycle({ phases: [name] })` rather
than extracting standalone phase functions. Cycle.ts already owns the
lock + abort signal + progress reporter per D10, so the wrapper is a
one-liner and cycle.ts remains the single source of truth for phase
semantics. Pragmatic deviation from the plan's "extract 6 standalone
runXxxPhase functions" — smaller diff, equivalent correctness.

Standalone `sync` handler now passes `noExtract: true` (codex #5 fix).
Pre-fix, doctor's remediation plan emitting [sync, extract] caused
double-extraction (performSync inline-extract + standalone extract
job). Now sync defers extract to the dedicated handler. Callers that
want inline extract pass { noExtract: false } in job params.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T5 + D10 + D11 + codex #5/#6 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): --remediation-plan + --remediate CLI surfaces

T6 of brain-health-100 wave. The headline user-facing capability:
agents drive brain health to target score via autonomous Minions
remediation.

Two new flags on `gbrain doctor`:

  --remediation-plan [--json] [--target-score N]
    Read-only. Emits ordered Remediation[] from BrainHealth + context.
    Uses cheap path (D7) — engine.getHealth() + computeRecommendations,
    NOT a full doctor walk. JSON shape is stable agent contract.

  --remediate [--yes] [--target-score N] [--max-jobs N] [--max-usd N]
              [--dry-run] [--json]
    Sequential submit (D3) with D5 cascade on failure, D7 scoped
    recheck between steps, D9 content-hash idempotency keys, D13
    three-state remediation filtering (only remediable jobs enter
    the loop), +A cost-budget gate via --max-usd.

Check.remediation field added as additive optional (DoctorReport
schema_version stays at 2 per D4).

PGLite path: synchronous in-process execution with short polling.
Postgres path: durable queue submission with waitForCompletion.

The --remediate loop:
  1. Compute initial plan from BrainHealth
  2. Refuse if --target-score > maxReachableScore(health, classes)
  3. Refuse if est_total_usd_cost > --max-usd
  4. For each step in order:
     - Skip if depends_on intersects aborted set (D5)
     - queue.add with content-hash idempotency_key (D9)
     - waitForCompletion with timeout
     - Recompute plan from fresh health (D7 scoped recheck)
  5. Exit 0 if all completed; 1 if any failed/aborted

doctor_run_id UUID stamps every submitted job's data field so
operators can later query `SELECT * FROM minion_jobs WHERE
data->>'doctor_run_id' = '<uuid>'` (indexed via v68 partial GIN).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T6 + D1/D3/D5/D7/D9/D13 + folded scope A).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): maybeBackground helper + apply --background to embed

T7 of brain-health-100 wave. New helper in src/core/cli-options.ts
formalizes the --background flag pattern. Same semantics in TTY and
cron per D9 (submit-and-exit always; --background --follow execs
`gbrain jobs follow <id>` after submission).

  await maybeBackground({
    engine, args, jobName: 'embed',
    paramBuilder: (cleanArgs) => ({ stale, all, ... }),
  })
  // returns true if backgrounded → caller exits

Content-hash idempotency key (D9): `cli:embed:sha8(canonical-JSON(params))`.
No time-slot. Same intent across runs = same key. Failed-row replay
is the doctor --remediate loop's job, not this path's.

PGLite degrades to inline execution with a clear stderr note
("PGLite has no worker daemon; running inline"). NOT a no-op,
NOT silent — doc-stated semantic difference because PGLite has no
worker daemon.

Applied to `gbrain embed` as the reference integration. The other 6
commands (extract, lint, backlinks, reindex, integrity, pages) adopt
the same 4-line pattern at the top of their entry function — follow-up
in a smaller diff once the helper proves out in production.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T7 + D9 + Gap 6).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): targeted-submit loop + op_checkpoints GC in purge phase

T8 of brain-health-100 wave.

Autopilot dispatch changes (src/commands/autopilot.ts):

Pre-fix: every tick submitted ONE autopilot-cycle job, full phase
set, regardless of brain state. On a healthy brain pure overhead; on
a degraded brain bundled fast wins with slow phases so user waited
for the slowest.

New decision logic (T8 from plan):
  - score >= 95 AND empty plan AND <60min since last full → SLEEP
  - score >= 95 AND empty plan AND >=60min → submit autopilot-cycle
    (phase-coupling exercise)
  - plan <= 3 steps AND est_total < 5min → submit individual handlers
    (targeted; uses D9 content-hash idempotency keys per step;
    maxWaiting:1 per submit per codex #17)
  - else → submit autopilot-cycle (the hammer)

D10 cycle-lock invariant guarantees targeted-submit and autopilot-cycle
can never run concurrently (both acquire gbrain-cycle), closing the
"60-min floor double-processes queued targeted jobs" failure mode.

Computation uses cheap path (D7) — engine.getHealth() + computeRecommendations,
NOT a full doctor walk. Adds ~1 SQL count query per tick; negligible
on a 50K-page brain.

PROTECTED handlers (synthesize/patterns/consolidate) are submitted with
allowProtectedSubmit:true; autopilot is a trusted local caller.

Cycle purge phase (src/core/cycle.ts):

Added op_checkpoints GC (+C folded scope item). 7-day TTL — any
reasonable long-running op finishes inside that window. Non-fatal
on pre-v67 brains (table missing).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T8 + D7/D9/D10 + codex #17 + folded scope +C).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(core): brain-score-recommendations + op-checkpoint unit tests

T10 of brain-health-100 wave — load-bearing decision-pinning tests.

test/brain-score-recommendations.test.ts (22 cases):
  - Healthy brain → empty plan
  - Per-component remediation paths (sync, embed, backlinks, extract)
  - depends_on wiring (extract → sync; embed → sync when stale)
  - Severity ordering (critical > high > medium > low)
  - D6 #5 determinism: same input twice → byte-identical output
  - D9 idempotency keys: content-hash format, no time-slot
  - D9 source isolation: different --source → different key
  - D13 status field always 'remediable' in output
  - +A cost-estimate populated for embed
  - classifyChecks: remediable / blocked / human_only triage
  - maxReachableScore: all-remediable → 100; all-blocked → current

test/op-checkpoint.test.ts (20 cases):
  - fingerprint stability + key-order invariance (canonical-JSON)
  - codex #11: extract links vs timeline get different fingerprints
  - codex #12: reindex markdown vs code get different fingerprints
  - codex #15: embed model+dim variation produces different fingerprints
  - reindex chunker_version bump invalidates checkpoint
  - DB round-trip (load → record → load)
  - Cross-fingerprint isolation (linksKey vs timelineKey)
  - clearOpCheckpoint idempotency on missing rows
  - resumeFilter purity (no I/O, deterministic)
  - purgeStaleCheckpoints TTL respect

42 new tests, all pass. PGLite engine + resetPgliteState pattern per
CLAUDE.md test-isolation guide.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T10 + D6 #5 + D9 + D12 + D13 + codex #11/#12/#15).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): v0.36.0.0 — brain-health-100 wave + docs/llms refresh

T12 of brain-health-100 wave. VERSION + package.json bumped 0.35.6.0
→ 0.36.0.0. CHANGELOG entry leads ELI10 ("your agent can now drive
your brain to 90/100 by itself, on a cron, without you watching")
then drills into the precise mechanics per CLAUDE.md voice rules.

llms.txt + llms-full.txt regenerated via bun run build:llms.

Trio audit (CLAUDE.md mandatory pre-push check):
  VERSION:     0.36.0.0
  package.json: 0.36.0.0
  CHANGELOG:   ## [0.36.0.0] - 2026-05-18

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README/CLAUDE/AGENTS/maintain for v0.36.4.0 brain-health-100 wave

- README.md: New-in-v0.36.4.0 callout — `gbrain doctor --remediate` headline,
  autopilot health-aware tick, eleven new background-job types, three PROTECTED.
- CLAUDE.md: Key Files entries for `op-checkpoint.ts`, `brain-score-recommendations.ts`,
  doctor.ts / jobs.ts / protected-names.ts / autopilot.ts / cycle.ts / embed.ts /
  cli-options.ts extensions; new "Key commands added in v0.36.4.0" section.
- AGENTS.md: Common-tasks entry pointing agents at the one-command remediation loop.
- skills/maintain/SKILL.md: Autonomous Phase (gbrain doctor --remediate) at the top,
  manual per-dimension walk preserved as the fallback path.
- llms-full.txt: regenerated to pick up the CLAUDE.md changes (project rule).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(changelog): respectful tone on spend caps for v0.36.4.0

Reframed the cost-budget callout. Pre-fix language said the spend cap
prevents a synthesize loop from "burning $100 of Anthropic credits
while you're at lunch" — casually treating $100 as the throwaway number
is tone-deaf. $100 is a meaningful amount for many people.

New language: "spend cap so a synthesize loop can't run up your
Anthropic bill while you're at lunch. The cap is yours to set per run."
And: "Pass --max-usd 5 (or whatever cap you're comfortable with)."
And: "Pick the cap that fits your wallet."

Also reframed three adjacent lines:
- "healthy brains stop burning cycles" → "stop spending tokens on
  work that has nothing to do"
- "agent can't submit them and burn your API budget" → "can't submit
  them on your behalf. Your provider bill stays in your hands"
- Table cell "Cron with cost cap" / "--max-usd 5" → "Cron with spend
  cap" / "--max-usd N"

llms-full.txt regenerated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 10:05:31 -07:00
garrytan-agentsandgarrytan-agents 3aedffadc0 fix(docs): comprehensive drift audit — contradictions, broken links, stale refs (#1201) 
A community member reported docs 'have quite a bit of drift and some broken
links' and contradictions like 'says don't use bun but also to use bun.' This
PR is a top-to-bottom audit + fix across every doc file at the repo root and
under docs/. Where docs disagreed with each other, the code was the tie-breaker.

## Categories of fix

### 1. Stale CLI commands (skillpack install → scaffold)

`gbrain skillpack install` was retired in v0.36.0.0 (replaced by the
scaffold/reference/migrate-fence model). The CLI now errors out with a hint:

    $ gbrain skillpack install
    Error: 'gbrain skillpack install' was removed in v0.33.
    Use 'gbrain skillpack scaffold <name>' instead.

But the docs still recommended it:

- README.md line 29 — primary install path
- docs/INSTALL.md lines 12 — primary install path

Both updated to `gbrain skillpack scaffold --all` with the v0.36.0.0 retirement
explained inline + the migrate-fence escape hatch for users upgrading from older
releases.

### 2. The 'bun install -g vs bun link' contradiction

The community member's exact complaint. The drift:

- README.md + docs/INSTALL.md: recommended `bun install -g github:garrytan/gbrain`
- INSTALL_FOR_AGENTS.md line 29: 'Do NOT use `bun install -g github:garrytan/gbrain`.'

Reading the code + CHANGELOG: `bun install -g` IS the canonical path. Bun
occasionally blocks the top-level postinstall hook on global installs (issue #218),
but the postinstall now prints a loud recovery hint when that happens, and
`gbrain doctor` flags `schema_version: 0` and routes users to
`gbrain apply-migrations --yes`. The 'do not use' warning was correct in 2024
when the postinstall silently swallowed errors with `|| true`; it's stale now.

Reconciled:

- INSTALL_FOR_AGENTS.md Step 1: now recommends `bun install -g` as the primary
  path, documents #218 as a known issue with the recovery command, and keeps
  `git clone + bun link` as a documented fallback.
- AGENTS.md Install (5 min): same reconciliation; clone path is the fallback,
  not the default.
- docs/INSTALL.md CLI standalone: added the #218 callout so the deterministic
  fallback is one click away when the default fails.

### 3. Broken internal links

- README.md → `docs/integrations/voice.md` (file doesn't exist). The real voice
  recipe lives at `recipes/twilio-voice-brain.md` (Twilio + OpenAI Realtime).
  Fixed to point there with an accurate one-line summary.
- CONTRIBUTING.md → `docs/SQLITE_ENGINE.md` (file doesn't exist; superseded by
  PGLite per docs/ENGINES.md). Replaced with a paragraph explaining the
  supersession and pointing at the live ENGINES.md.
- docs/GBRAIN_V0.md → `docs/SQLITE_ENGINE.md` (2 references; same supersession).
  Added a historical-doc banner at the top + rewrote both references to point at
  the current ENGINES.md.

### 4. Stale API key recommendations

INSTALL_FOR_AGENTS.md Step 2 only mentioned OpenAI + Anthropic. As of v0.36.2.0
ZeroEntropy is the default embedding + reranker stack (README opens with this);
the agent install guide didn't reflect it. Added `ZEROENTROPY_API_KEY` as the
default, kept OpenAI/Voyage as documented fallbacks, noted that keys can live in
`~/.gbrain/config.json` (file plane) or env.

### 5. Stale upgrade workflow

INSTALL_FOR_AGENTS.md 'Upgrade' section assumed the clone+bun-install model
(`cd ~/gbrain && git pull && bun install && gbrain init && gbrain post-upgrade`)
and didn't mention `gbrain upgrade` (the single-command path that exists in the
CLI today: binary self-update + schema migrations + post-upgrade prompts in one).
Split into two paths — `gbrain upgrade` for the bun-install-g case (now the
default per Step 1), clone-path for the fallback case.

Also fixed AGENTS.md 'Migrate' bullet (was `gbrain apply-migrations` only;
now leads with `gbrain upgrade` and keeps apply-migrations as the manual
schema-only path).

### 6. Stale cron-workflow

INSTALL_FOR_AGENTS.md Step 7 referenced cron docs but didn't mention
`gbrain autopilot --install` (the built-in self-maintaining daemon that
exists in the CLI today) or `gbrain sync --watch` (continuous loop). Added
both as alternatives to platform-cron glue.

### 7. ZeroEntropy version typo

docs/INSTALL.md said 'the v0.36.0.0 ZE switch' — ZE landed in v0.36.2.0
(v0.36.0.0 was the skillpack-scaffold retirement). Fixed.

## What I did NOT change

- CHANGELOG.md, CLAUDE.md, TODOS.md prose mentions of historical commands like
  `gbrain skillpack install` are correct as history — they're documenting what
  was true in past releases. Only forward-looking docs got updated.
- The 'broken link' false-positive matches in CHANGELOG / CLAUDE / TODOS are
  inside code-fence examples or regex patterns (`[Name](people/slug)`,
  `[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])`, `[--json](interrupted)`); they're
  illustrative syntax, not real links. Leaving alone.
- llms.txt / llms-full.txt regenerated via `bun run build:llms` so the
  agent-fetch documentation map matches the new content.

## Verification

- `bun run src/cli.ts --help` cross-checked against every command/flag the
  install docs reference: init, doctor, apply-migrations, upgrade, post-upgrade,
  skillpack scaffold/reference/migrate-fence, embed --stale, sync --watch,
  autopilot --install, dream, integrations list, extract links/timeline,
  graph-query, query, search modes — all real, all current.
- `bun run src/cli.ts skillpack install` confirmed to error out with the
  retirement hint pointing at scaffold (proves the README guidance was actively
  misleading users into a dead-end).
- Re-ran the broken-internal-link scanner across all root .md + docs/**/*.md;
  zero real broken links remain (5 residual matches are illustrative syntax
  inside prose, not actionable links).

Co-authored-by: garrytan-agents <agents@garrytan-agents.local>
 2026-05-19 05:32:24 -07:00
Garry TanandClaude Opus 4.7 cdba533a04 v0.36.2.0 feat: ZeroEntropy as default + zero-based README rewrite (#1136) 
* feat(dims): OpenAI text-embedding-3 Matryoshka range validation (D13)

dimsProviderOptions now fail-loud at the embed boundary when the
configured embedding_dimensions is outside the model's native range
(1..1536 for -small, 1..3072 for -large). Paste-ready fix hint in the
AIConfigError.fix field. Closes the silent-HTTP-400 path that would
have bit OpenAI-fallback users on v0.36.0.0 ZE-default installs.

16 new test cases in test/ai/dims-openai.test.ts pinning the contract
across native-openai and openai-compatible adapter paths.

* feat(ai): flip defaults to ZeroEntropy zembed-1 1280d + zerank-2 reranker

Default embedding model is now zeroentropyai:zembed-1 at 1280d via
Matryoshka. Real-corpus benchmark: 2.2x faster than OpenAI, 2.6x
cheaper at regular pricing, wins 11/20 head-to-head queries.

1280 is the closest valid ZE Matryoshka step to the prior OpenAI 1536d
default (valid set: 2560/1280/640/320/160/80/40). 1024 (Voyage's step)
is NOT on ZE's list — pinned by AIConfigError fail-loud in dims.ts.

balanced mode bundle now defaults reranker_enabled=true. zerank-2
reshuffles 60% of top-1 results in benchmarks. Missing-key fail-open
contract in src/core/search/rerank.ts handles unauthenticated cases.
Opt out with: gbrain config set search.reranker.enabled false

Existing tests updated (gateway.test.ts, search-mode.test.ts) and a
new test/balanced-reranker-default.test.ts (10 cases) pins the fail-
open invariants.

* feat(retrieval-upgrade): RetrievalUpgradePlanner + interactive prompt UX

New src/core/retrieval-upgrade-planner.ts is the consolidated planner
that computes the brain's pending retrieval-upgrade work (chunker
bumps + ZE switch) in one pass and applies the schema transition +
config updates atomically.

Tagged-union ApplyResult enum (D15): 'applied' | 'skipped_already_
applied' | 'skipped_no_work' | 'declined' | 'planned' | 'failed'.
No string-parsing reasons.

Three config keys (D12): ze_switch_prompt_shown (UI state),
ze_switch_requested (user intent), ze_switch_applied (work done).
Plus ze_switch_previous_snapshot (JSON, full prior config for --undo
per D16) and ze_switch_declined_at (90-day re-ask window).

Schema transition (D18) is atomic: DROP indexes + ALTER COLUMN +
CREATE INDEX inside a single engine.transaction(). HNSW recreation
is part of the same transaction — no silent slow-search window.

C3 eligibility logic: ze_switch_offered iff NOT on ZE + NOT declined
recently + NOT applied + (legacy default OR >100 pages).

C4 cost math: MAX(chunker_pending, dim_pending) not SUM — one
re-embed pass invalidates both surfaces simultaneously.

New src/core/retrieval-upgrade-prompt.ts wires the planner to a
TTY-only interactive prompt with two-line cost split (D10) and
privacy callout for the reranker flip.

Tests: test/retrieval-upgrade-planner.test.ts (24 cases) pins the
state machine. test/asymmetric-encoding-contract.test.ts (6 cases)
pins D17: search read path uses gateway.embedQuery() not embed(),
asserted via __setEmbedTransportForTests mock.

* feat(cli): gbrain ze-switch — manual lever for the ZE switch

New gbrain ze-switch CLI with --dry-run, --json, --resume, --force,
--undo, --non-interactive, --confirm-reembed, --ignore-missing-key
flags. Mirrors the upgrade prompt's UX symmetry: --undo presents a
cost-warning before re-embedding back to the prior width.

src/cli.ts: dispatch case + CLI_ONLY entry. ze-switch owns its own
engine lifecycle (mirrors the doctor pattern).

test/ze-switch-cli.test.ts (11 cases): --help, --dry-run, --json,
--non-interactive, --ignore-missing-key, --resume, --undo,
--confirm-reembed. Uses captureExit harness to test process.exit()
paths without breaking the test process.

* feat(doctor): ze_embedding_health + embedding_width_consistency checks

Two new doctor checks (D-A5):

ze_embedding_health: when embedding_model starts with zeroentropyai:,
verify ZEROENTROPY_API_KEY is set (env or config). Paste-ready setup
hint with the signup URL on failure.

embedding_width_consistency: cross-check that the configured
embedding_dimensions matches the actual vector(N) column width on
content_chunks.embedding. Catches the half-applied switch state
(schema migrated but config write crashed) with a paste-ready
gbrain ze-switch --resume hint.

Wired into runDoctor between reranker_health and the existing
sync_freshness checks. Both checks gracefully no-op on non-ZE
embedding configs.

test/doctor-ze-checks.test.ts (8 cases) pins both checks across
happy + missing-key + missing-config + drift paths. Uses withEnv()
helper to clear ZEROENTROPY_API_KEY for the no-key path so tests
are hermetic against contributor env state.

test/e2e/v0_28_5-fix-wave.test.ts + test/openai-compat-multimodal.test.ts:
updated to explicit-configure the gateway when the test depends on
specific dims that diverge from the v0.36.0.0 default (1280d).

* docs: README zero-based rewrite (884 -> 139 lines) + new docs files

Strip 4 months of accreted "New in v0.X.Y" hero blocks and reorganize
around what gbrain does today. 33 H2s -> 8. The Commands section
(136 lines duplicating gbrain --help) moved out; the 6-table skills
enumeration collapsed to a one-paragraph capability description with
a link to skills/RESOLVER.md.

Hero retains load-bearing facts: OpenClaw + Hermes credit, production
numbers (17,888 pages / 4,383 people / 723 companies), BrainBench
numbers (P@5 49.1% / R@5 97.9% / +31.4 lift), ZE comparison numbers,
30-min install claim. Adds one paragraph announcing the v0.36.0.0 ZE
default with the explicit gbrain config set escape for OpenAI/Voyage
users.

New files:
- docs/INSTALL.md: every install path consolidated (agent platform,
  CLI standalone, MCP server). Thin-client mode covered.
- docs/architecture/RETRIEVAL.md: why the hybrid + graph stack works.
  BrainBench numbers, why each strategy alone fails, the source-aware
  ranking + intent classification + multi-query expansion story.
- docs/ethos/ORIGIN.md: origin story lifted from the old README so
  the front door stays factual + concrete.

test/readme-hero-anchors.test.ts (5 cases) is the D9 regression
guard. Five load-bearing strings: OpenClaw, Hermes, ZE,
production-numbers regex, P@5/R@5. Light anchors that let voice/
structure evolve but block accidental loss of headline facts.

scripts/check-test-real-names.sh: allowlist entries for OpenClaw +
Hermes literals in the anchor test (it explicitly asserts those
strings appear in README).

* chore: bump version and changelog (v0.36.0.0)

ZeroEntropy as the new default for embedding (zembed-1 at 1280d via
Matryoshka) and reranker (zerank-2 cross-encoder, on by default in
balanced mode bundle). README zero-based rewrite (884 -> 139 lines).
3 new docs files. Two new doctor checks. New gbrain ze-switch CLI
with --undo for symmetric reversibility.

skills/migrations/v0.36.0.0.md tells the agent how to surface the
retrieval-upgrade prompt post-upgrade.

llms-full.txt regenerated via bun run build:llms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(docs): scrub Wintermute from RETRIEVAL.md per privacy rule

* chore: rebump version 0.36.0.0 → 0.36.2.0 (queue collision)

Three open PRs were claiming v0.36.0.0 (#1130 skillpack, #1139
hindsight, #1136 this PR). Ship-aware queue allocator says this
branch lands at v0.36.2.0.

Trio audit:
  VERSION       0.36.2.0
  package.json  0.36.2.0
  CHANGELOG     ## [0.36.2.0] - 2026-05-17

Updates: VERSION, package.json, CHANGELOG header + body refs,
README "New default in v0.36.2.0" announcement + credit line,
skills/migrations/v0.36.0.0.md renamed to v0.36.2.0.md with
frontmatter + body refs updated. llms-full.txt regenerated.

* fix(test): pin gateway dim=1536 in cross-file-stateful PGLite tests

CI shard 1 reported 10 failures across `query-cache.test.ts` (6) and
`consolidate-valid-until.test.ts` (4). Both files hardcode 1536-dim
vectors but rely on `PGLiteEngine.initSchema()` to size
`vector(__EMBEDDING_DIMS__)` at the right width.

Root cause: v0.36.2.0 flipped DEFAULT_EMBEDDING_DIMENSIONS from 1536
to 1280 (ZE Matryoshka step). The gateway module is process-singleton;
when ANOTHER test file in the same shard's bun-test process configures
the gateway before us, `pglite-engine.ts:216` reads
`getEmbeddingDimensions() === 1280` and sizes the schema columns at
vector(1280). The hardcoded 1536-dim INSERTs then fail with
"expected 1280 dimensions, not 1536".

Locally these tests pass in isolation because the gateway falls back
through the try/catch at pglite-engine.ts:218 (1536 default). CI runs
multiple test files in one process, so cross-file state poisons the
schema width.

Fix: explicit `resetGateway()` + `configureGateway({embedding_dimensions:
1536, ...})` at the top of `beforeAll`, plus `resetGateway()` in
`afterAll`. Pins the schema width regardless of cross-file state.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 21:11:02 -07:00
Garry TanandClaude Opus 4.7 3a0e1116e7 v0.36.1.0 Hindsight calibration wave: brain learns how you tend to be wrong (#1139) 
* schema: v0.36.0.0 Hindsight calibration tables (migrations v67-v71)

Foundation commit for the Hindsight-inspired calibration wave. Adds four
new tables + one perf index, all source-scoped from day 1 per v0.34.1
discipline:

- calibration_profiles (v67): per-holder LLM-narrative aggregation of
  TakesScorecard data. published BOOL gates E8 cross-brain mount sharing
  (default false). grade_completion REAL surfaces partial-grade state to
  the dashboard. active_bias_tags TEXT[] with GIN index feeds E3 (calibration-
  aware contradictions) and E7 (real-time nudge matching).

- take_proposals (v68): propose_takes phase queue. Idempotency cache via
  (source_id, page_slug, content_hash, prompt_version) unique index mirrors
  the v0.23 dream_verdicts pattern. proposal_run_id supports --rollback by
  run. dedup_against_fence_rows JSONB audit column records what canonical
  takes the LLM was told to dedupe against at proposal time.

- take_grade_cache (v69): grade_takes verdict cache. Composite PK on
  (take_id, prompt_version, judge_model_id, evidence_signature) — prompt
  edits OR evidence changes cleanly invalidate prior verdicts. applied=false
  default + auto-resolve-off-by-default (D17) means every fresh install
  needs operator opt-in before grade verdicts mutate the takes table.

- take_nudge_log (v70): E7 nudge cooldown state. Polymorphic FK — a nudge
  fires on either a canonical take OR a pending proposal (CDX-5 fix). CHECK
  constraint enforces exactly-one-set. channel column lets future routing
  (webhook, admin SPA toast) reuse the same cooldown semantics.

- takes_resolved_at_idx (v71): partial index for the Brier-trend
  aggregation queries. Engine-aware handler — Postgres uses CONCURRENTLY
  to avoid the ShareLock; PGLite uses plain CREATE.

Every table carries wave_version TEXT NOT NULL DEFAULT 'v0.36.0.0' so the
v0.36.0.0 calibration --undo-wave command (lands later in the wave) can
reverse just this wave's writes.

Plan: ~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md
covers the design rationale (D17/D18/D21 + CDX findings).

Schema parity:
- src/schema.sql for fresh Postgres installs
- src/core/pglite-schema.ts for fresh PGLite installs
- src/core/schema-embedded.ts auto-regenerated from schema.sql
- src/core/migrate.ts for upgrade-in-place from older brains

VERSION bumped to 0.36.0.0 for the wave. CHANGELOG entry lands at /ship.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* core: BaseCyclePhase abstract class enforces source-scope + budget contracts

D21 from the eng review. Three new v0.36.0.0 cycle phases (propose_takes,
grade_takes, calibration_profile) share enough structure that the
duplication-vs-abstraction trade tips toward a shared base. Without this
scaffold, source-isolation discipline would drift exactly the way it
drifted in v0.34.1 — except this time across three new surfaces at once.

What this enforces:

1. Phase signature is uniform: run(ctx, opts) → PhaseResult.

2. ctx.sourceId / ctx.auth.allowedSources MUST be threaded through every
   engine call. The base class surfaces a scope() helper that wraps
   sourceScopeOpts(ctx) and is the only sanctioned way to read source-
   scoped data. Forgetting to thread source scope becomes a TypeScript
   compile error, not a runtime leak. Closes the v0.34.1 leak class
   structurally for every new phase.

3. Budget meter wraps run() automatically. Subclass declares budgetUsdKey
   + budgetUsdDefault; base reads the resolved cap from config and creates
   the BudgetMeter. Subclass calls this.checkBudget() before each LLM
   submit; budget-exhausted phase still returns status='ok' (clean abort)
   so the cycle report shows partial completion, not failure.

4. Error envelope is uniform. Thrown errors get caught and converted to
   status='fail' with a phase-specific error.code via the subclass's
   mapErrorCode() hook.

5. Progress reporter integration. Base accepts the reporter via opts;
   subclasses call this.tick() instead of touching the reporter directly,
   so the phase name in the progress stream is always correct.

Tests: 13 cases in test/core/base-phase.test.ts cover source-scope
threading (5 cases including the empty-allowedSources-MUST-NOT-widen-scope
regression), PhaseResult shape including the error envelope path (3
cases), dry-run propagation (2 cases), and budget meter construction
(3 cases including config-key override).

Synthesize.ts / patterns.ts (existing pre-v0.36 phases) deliberately do
NOT retrofit to this base in v0.36.0.0 — too much churn for a refactor
that doesn't pay off until v0.37+. Future phases use this by default.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cycle: propose_takes phase + take_proposals queue write path (T3)

LLM-based take extraction from markdown prose. Walks pages updated since
last cycle, sends each page's body to a tuned extractor, writes the
extracted gradeable claims to the take_proposals queue. User accepts /
rejects via `gbrain takes propose --review` (lands in Lane C).

Cycle wiring:
  lint → backlinks → sync → synthesize → extract → extract_facts →
    resolve_symbol_edges → patterns → recompute_emotional_weight →
    consolidate → propose_takes (NEW) → grade_takes (NEW; T4) →
    calibration_profile (NEW; T6) → embed → orphans → purge

CyclePhase enum extended with 3 new entries; ALL_PHASES + NEEDS_LOCK_PHASES
updated. All three new phases acquire the cycle lock (writes to
take_proposals / take_grade_cache / calibration_profiles).

Idempotency contract:
  The (source_id, page_slug, content_hash, prompt_version) composite unique
  index on take_proposals means an unchanged page never re-spends LLM
  tokens. Bumping PROPOSE_TAKES_PROMPT_VERSION cleanly invalidates the
  cache so a tuned prompt re-runs proposals on every page. Mirrors the
  v0.23 dream_verdicts pattern.

F2 fence dedup:
  The phase reads the page's existing `<!-- gbrain:takes:begin -->` fence
  (when present) and passes the canonical take rows to the extractor as
  "things you have already captured." Prevents duplicate proposals when
  prose is appended to a page that already has takes. Records the fence
  rows the LLM was told to dedupe against on the take_proposals row for
  audit (dedup_against_fence_rows JSONB).

Auto-resolve posture:
  propose_takes only WRITES proposals to the queue. Nothing in this phase
  mutates the canonical takes table. Operator opt-in via the queue review
  CLI (Lane C) is the only path from queue to canonical fence (D17).

Prompt tuning status (v0.36.0.0 ship state):
  The default extractor prompt is annotated `v0.36.0.0-stub`. The real
  tuned prompt arrives via T19 synthetic corpus build (50 anonymized
  pages, 3-model parallel extraction, user reviews disagreement set,
  F1 ≥ 0.85 on training corpus + F1 ≥ 0.8 on ground-truth holdout).
  Until T19 lands, propose_takes runs but produces best-effort candidates
  the user reviews manually.

Architecture:
  ProposeTakesPhase extends BaseCyclePhase (T2). Inherits source-scope
  threading via scope(), budget metering via this.checkBudget(), error
  envelope wrapping. budgetUsdKey: cycle.propose_takes.budget_usd
  (default $5/cycle). Budget exhaustion mid-page returns status='warn'
  with details.budget_exhausted=true — clean partial-completion semantics.

  Test seam: opts.extractor injection so the phase can run hermetically
  without touching the gateway. defaultExtractor (production path) calls
  gateway.chat with the EXTRACT_TAKES_PROMPT and parses the JSON array
  output via parseExtractorOutput.

  parseExtractorOutput defends against common LLM output sins: markdown
  code fence wrapping, leading prose, single-object instead of array,
  unknown kind values, weight out of [0,1], rows missing claim_text or
  exceeding 500 chars.

Tests: 25 cases in test/propose-takes.test.ts cover the 4 pure helpers
(parseExtractorOutput, contentHash, hasCompleteFence,
extractExistingTakesForDedup) + 7 phase integration scenarios (happy path,
cache hit, fence dedup, extractor failure, empty pages, skipPagesWithFence,
proposal_run_id stability).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cycle: grade_takes phase + take_grade_cache verdict pipeline (T4)

Walks unresolved takes that are old enough to have outcome data, retrieves
evidence from the brain, asks a judge model to verdict each one. Writes
verdicts to take_grade_cache. Optionally — only when operator has flipped
the opt-in config flag — auto-applies high-confidence verdicts to the
canonical takes table via engine.resolveTake.

Auto-resolve posture (D17 — DISABLED by default):
  On a fresh install, grade_takes runs and writes verdicts to the cache,
  but applied=false on every row. Operator reviews the queue, then flips
  `cycle.grade_takes.auto_resolve.enabled: true` once trust is earned.
  Mirrors the propose_takes review-queue posture: queue exists, mutation
  requires explicit opt-in.

Conservative threshold (D12):
  When auto_resolve.enabled is true, a verdict auto-applies only when
  confidence >= 0.95 (single-judge path). T5 ensemble path lands next,
  tightening this further with 3/3 unanimous requirement.

  'unresolvable' verdict NEVER auto-applies even at confidence=1.0 —
  there's no canonical column for "we tried and there's no evidence yet."

Evidence retrieval status (v0.36.0.0 ship state):
  The default evidence retriever returns an "evidence-retrieval not yet
  wired" placeholder. Most verdicts produced by the stub-judge against
  the stub-evidence will be 'unresolvable'. Real retrieval (hybrid search
  over pages newer than the take's since_date, optionally augmented by a
  gateway web-search recipe in v0.37+) lands as a follow-up. Documented
  limitation per CDX-8 + D17 — the phase ships now so the wiring is real
  and the cache table accumulates verdicts even if early ones are
  conservative.

Cache key:
  Composite primary key on take_grade_cache is
  (take_id, prompt_version, judge_model_id, evidence_signature). Prompt
  edits OR evidence changes OR judge swap cleanly invalidate prior
  verdicts. Mirrors the v0.32.6 eval_contradictions_cache pattern.

  evidence_signature = SHA-256 of (judge_model_id + '|' + evidence_text)
  so identical evidence under a different judge does NOT collide.

Architecture:
  GradeTakesPhase extends BaseCyclePhase. Inherits source-scope threading,
  budget metering (cycle.grade_takes.budget_usd, default $3/cycle), error
  envelope. Test seam: opts.judge + opts.evidenceRetriever injection so
  the phase runs hermetically.

  parseJudgeOutput defends against fence-wrapping, leading prose,
  out-of-range confidence (clamps to [0,1]), invalid verdict labels,
  oversized reasoning (truncated at 400 chars). Returns null on
  unrecoverable parse — caller treats null as "judge_output_parse_failed
  / unresolvable at confidence 0.0" so the row still lands in cache with
  the parse failure surfaced via warnings.

  takeIsOldEnough gates on since_date (default 6 months). Tolerates
  YYYY-MM-DD and YYYY-MM formats. Returns false on null/unparseable
  since_date so takes without dates never get graded (we'd be
  hallucinating temporal context).

Tests: 23 cases covering parseJudgeOutput (7 cases), evidenceSignature
(3), takeIsOldEnough (5), and 8 phase integration scenarios — happy path,
D17 auto-resolve-off default, D12 above-threshold auto-apply, below-
threshold cache-only, unresolvable-NEVER-applies, cache hit, too-recent
gate, judge-throw warning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cycle: grade_takes ensemble tiebreaker for borderline verdicts (T5 / E2)

Multi-judge ensemble tiebreaker, additive on top of T4's single-judge
foundation. Reuses gateway.chat as the per-model judge interface; runs
three judges in parallel via Promise.allSettled. Pure aggregation logic
in aggregateEnsemble() — no SQL, no LLM, hermetically testable.

When ensemble fires (T5 trigger band):
  Only when ALL of:
    - opts.useEnsemble === true (default false)
    - opts.ensembleJudges array is non-empty
    - single-model confidence in [0.6, 0.95) (configurable via
      opts.ensembleTriggerBand)
    - single-model verdict !== 'unresolvable'

  Above 0.95 the single judge is already sufficient (T4 path). Below 0.6
  the verdict is clearly review-only — ensemble wouldn't change the
  posture. 'unresolvable' from single-judge means no evidence yet; calling
  three more judges on the same evidence won't manufacture some.

Conservative auto-apply (D12):
  Ensemble verdict auto-applies via engine.resolveTake only when ALL of:
    - autoResolve === true (operator opt-in per D17)
    - ensemble.agreement === 3 (3/3 unanimous)
    - ensemble.minConfidence >= ensembleThreshold (default 0.85)
    - winning verdict !== 'unresolvable'

  Schema-level monotonic-tightening guard for ensembleThreshold lives in
  the takes resolution layer.

Cache identity:
  When ensemble fires, the cache row's judge_model_id becomes
  'ensemble:<modelA>+<modelB>+<modelC>' — a future re-run with different
  ensemble membership doesn't collide with prior verdicts. evidence_signature
  is recomputed because it includes the judge_model_id.

aggregateEnsemble (pure):
  - 3/3 unanimous → agreement=3, minConfidence=min across the three
  - 2/3 majority → agreement=2, minConfidence across the agreeing two
  - 1/1/1 disagreement → tie-break: prefer non-'unresolvable', then
    alphabetical for determinism
  - 'unresolvable' from one model NEVER tips a 2-vote majority toward
    'unresolvable' — by-label tally only counts a model toward its own
    label
  - All three judges failing (allSettled rejected) → verdict='unresolvable'
    with agreement=0; auto-apply path blocked
  - Single judge survives + two fail → agreement=1; the lone verdict wins
    but auto-apply gated by the 3/3 requirement

Tests: 16 cases.
  aggregateEnsemble (6): 3/3, 2/3, 1/1/1, unresolvable-tipping-resistance,
  all-failed, partial-failed-but-survives.
  Phase trigger conditions (5): useEnsemble=false default, useEnsemble=true
  in borderline band, single >= 0.95 skip, single < 0.6 skip, single =
  'unresolvable' skip.
  Phase auto-apply rules (5): 3/3+threshold+autoResolve, 2/3 majority no
  apply, 3/3 below threshold no apply, one ensemble judge throws still
  aggregates from allSettled, empty ensembleJudges falls through to
  single.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cycle: calibration_profile phase + shared voice gate across surfaces (T6)

The calibration narrative layer. Reads TakesScorecard, asks an LLM to
write 2-4 conversational pattern statements ("right on tactics, late on
macro by 18 months"), passes them through the voice gate, derives active
bias tags, writes the row to calibration_profiles. This is the read-side
that E1 (think anti-bias rewrite), E3 (contradictions join), E6
(dashboard), and E7 (real-time nudges) all consume.

Voice gate (D24 — single function, multiple surfaces):
  ALL five calibration UX surfaces import the same gateVoice() function
  from src/core/calibration/voice-gate.ts. Mode parameter
  ('pattern_statement' | 'nudge' | 'forecast_blurb' | 'dashboard_caption'
  | 'morning_pulse') drives surface-specific tuning via the rubric the
  gate ships to its Haiku judge. NO forked implementations — voice
  rubric drift would defeat the gate.

  Each mode's rubric explicitly forbids preachy / clinical / corporate
  voice; a structural test pins this. Anchors the cross-cutting voice
  rule from /plan-ceo-review D2-D8.

Fallback policy (D11):
  Up to 2 generation attempts (configurable). On both rejects → fall back
  to a hand-written template from src/core/calibration/templates.ts.
  Templates are intentionally short and a little "robotic" — they're the
  safety net, not the destination. voice_gate_passed=false +
  voice_gate_attempts get persisted on the calibration_profiles row so
  the operator can review the failing examples and tune the rubric over
  time. Suppressing the surface silently is NEVER an option — that's how
  voice quality silently degrades.

  parseJudgeOutput defaults to 'academic' on parse failure (NEVER passes
  pass-through) so a Haiku output garble falls through to the template
  rather than letting unverified text reach the user.

calibration_profile phase:
  Extends BaseCyclePhase. Cold-brain skip: <5 resolved takes → no row
  written, no LLM call. Otherwise: scorecard via engine.getScorecard()
  → patterns via voice-gated generator → bias tags via separate
  generator (best-effort; failure logs warning, phase continues).

  The DB INSERT lands in the v67 calibration_profiles row with
  source_id, holder, the patterns, voice gate audit fields, active bias
  tags, and grade_completion (F1 fix — partial-grade state surfaces to
  the dashboard "60% graded" badge).

  Budget gate at $0.50/cycle default (mostly Haiku). Below-budget
  before-LLM-call check returns status='warn' without writing the row.

  Per-domain scorecards are a placeholder for v0.36.0.0 ship state —
  the F12 batchGetTakesScorecards() engine method that powers per-domain
  rendering lands in Lane C alongside the CLI/MCP surface.

Architecture:
  parsePatternStatementsOutput is tolerant of LLM emitting numbered
  lists / bulleted lines despite the prompt asking for plain lines.
  Caps at 4 patterns + drops excessively long lines (>200 chars).

  parseBiasTagsOutput lowercases input + drops non-kebab-case tokens
  (defends against the LLM emitting "Over-Confident Geography" with
  spaces or capitals). Caps at 4 tags.

Tests: 43 cases across two new test files.
  voice-gate.test.ts (24): parseJudgeOutput (7), gateVoice happy path
  (3), fallback path (5), mode parity (2), templates (7).
  calibration-profile.test.ts (19): parsers (10), pickFallbackSlots
  (3), phase integration (6 — cold-brain skip, happy path, voice gate
  fallback, grade_completion plumbed through, bias-tags failure
  non-fatal, source_id scope reaches INSERT).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cli: gbrain calibration + get_calibration_profile MCP op (T7)

Public-facing read surface for the v0.36.0.0 calibration wave. CLI prints
the active calibration profile; MCP op exposes the same data path for
agents. Mirror of the v0.29 salience/anomalies shape (pure data fn + JSON
formatter + human formatter + thin CLI dispatch).

CLI: `gbrain calibration`
  Flags:
    --holder <id>         specific holder (default 'garry')
    --json                machine output for piping
    --regenerate          run calibration_profile phase now
    --undo-wave <ver>     [placeholder — wires in Lane D / T17]
    ab-report             [placeholder — wires in Lane D / T18]

  Human output:
    Calibration profile — holder: garry, source: default
    Generated: <local timestamp>
    [Note: built on 60% graded — partial completion this cycle.]   (when grade_completion < 0.9)
    [Note: voice gate fell back to template (2 attempts).]         (when voice_gate_passed=false)

    Resolved: 12 takes
    Brier:    0.210 (lower is better)
    Accuracy: 60.0%
    Partial:  10.0%

    Pattern statements:
      • You called early-stage tactics well — 8 of 10 held up.

    Active bias tags: over-confident-geography

  Cold-brain fallback message names the exact dream command to run.

MCP: `get_calibration_profile` (scope: read)
  Param: holder?: string (defaults to 'garry')
  Returns: latest CalibrationProfileRow | null

  Source-scoping via sourceScopeOpts(ctx): scalar source-bound clients see
  only their source; federated_read scopes see the union of allowed sources;
  no source filter when neither is set (CLI default path).

  Throws GBrainError('INVALID_HOLDER') on empty/non-string holder so
  remote callers get a structured error instead of a SQL-shape failure.

Architecture:
  getLatestProfile is the pure data fn — engine + opts → CalibrationProfileRow | null.
  Reused by both the CLI and the MCP op. Source-scoped via the standard
  v0.34.1 spread pattern (scalar sourceId vs sourceIds array).

  formatProfileText is pure — null → cold-brain message, populated → full
  printout. Annotates partial-grade rows and voice-gate-fallback rows so
  the operator sees data-quality status inline.

  parseArgs is exported via __testing for unit coverage. Sub-command
  ('ab-report') vs flag distinction is intentional — keeps the surface
  parallel with `gbrain eval cross-modal` etc.

Tests: 21 cases.
  parseArgs (6 cases): empty, --holder, --json, --regenerate, --undo-wave, ab-report.
  getLatestProfile (5 cases): happy, null, scalar source scope, federated array
    scope, no-source-filter default.
  formatProfileText (5 cases): cold-brain, happy, partial-grade note, voice-fallback
    note, published-to-mounts note.
  getCalibrationProfileOp (5 cases): default holder, scalar source scope,
    federated scope union, returns-null-on-unknown-holder, throws on empty holder.

Lane D follow-ups: --undo-wave (T17) and ab-report (T18) print a clear
"lands in Lane D" stderr line + exit 2; the surfaces exist for early
testers, the implementations land next.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* think: --with-calibration + anti-bias prompt rewrite (T8 / E1, D22)

Optional anti-bias rewrite mode for `gbrain think`. When set, the active
calibration profile gets injected per the D22 placement spec (AFTER
retrieval evidence, BEFORE the user's question). The bias filter applies
to QUESTION FRAMING, not evidence interpretation — matches LLM-as-judge
best practice (bias prompts near end of context perform better).

Default behavior unchanged (R1 regression guard): omitting
--with-calibration produces the v0.28-vintage user-message shape with the
question first, then retrieval. Existing think users see no change.

Two user-message shapes in buildThinkUserMessage:

  Default (no calibration):
    Question: X
    <pages>...</pages>
    <takes>...</takes>
    <graph>...</graph>
    Respond with a single JSON object...

  With calibration (D22):
    <pages>...</pages>
    <takes>...</takes>
    <graph>...</graph>
    <calibration holder="garry">
      Track record: Brier 0.210 (lower is better).
      Active patterns:
        - You called early-stage tactics well — 8 of 10 held up.
      Active bias tags: over-confident-geography
    </calibration>
    Question: X
    Respond...

  Calibration block is built by buildCalibrationBlock (exported for the
  E3 contradictions probe to render the same shape).

System prompt extension (withCalibration:true):
  - Names BOTH the user's PRIOR (default reasoning) AND the COUNTER-PRIOR
    from their hedged-domain self.
  - References active bias tags by name when relevant ("this fits the
    over-confident-geography pattern").
  - Does NOT silently substitute the debiased answer. ALWAYS surfaces
    both priors transparently.
  - Adds a "Calibration" section between Conflicts and Gaps in the
    answer body.

RunThinkOpts extension:
  - withCalibration?: boolean — opt-in
  - calibrationHolder?: string — defaults to 'garry'

  When withCalibration=true and no profile exists, runThink falls back to
  baseline behavior + pushes NO_CALIBRATION_PROFILE to warnings (visible
  to the operator). When the calibration fetch fails, CALIBRATION_FETCH_FAILED
  warning surfaces with the underlying error. Either path keeps think working;
  the calibration loop is enhancement, not requirement.

CLI: `gbrain think "<q>" --with-calibration [--calibration-holder <id>]`

Tests: 11 cases.
  buildThinkSystemPrompt (4 cases): R1 regression — default/false/omitted
  → no anti-bias rules; with calibration → adds PRIOR + COUNTER-PRIOR +
  bias-tag reference; preserves existing hard rules.

  buildCalibrationBlock (3 cases): happy path, null brier omitted (not
  "Brier null"), empty patterns + tags still well-formed.

  buildThinkUserMessage (4 cases): R1 regression — without calibration:
  question first; D22 placement — retrieval → calibration → question →
  instruction; graph + calibration ordering; empty retrieval blocks render
  placeholders without breaking shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* contradictions: calibration-profile join (T9 / E3)

Cross-references each contradiction finding against the active calibration
profile. When a contradiction's domain matches an active bias tag (e.g.
"over-confident-geography" or "late-on-macro-tech"), the output gains a
one-line bias context explaining which pattern this fits.

Pure functions only — no DB writes, no LLM calls. The probe runner imports
tagFindingWithCalibration() and applies it to each finding before emitting.
When no profile exists or no tags match, the helper returns null and the
runner emits the unchanged finding (regression R2 — contradictions output
is byte-identical to v0.32.6 when no calibration profile is present).

Match heuristic (v0.36.0.0 ship-state):
  Bias tags are kebab-case axis-then-domain slugs ('over-confident-geography').
  computeDomainHint() extracts a domain hint from the finding's slugs +
  holder + verdict text:
    - wiki/companies/... → hiring | market-timing
    - wiki/people/... → founder-behavior
    - macro / geography / tactics / ai segments in slug → matching tag
  First-match-wins for ordering determinism.

  Match is intentionally fuzzy — the v0.32.6 contradictions probe doesn't
  yet carry structured domain metadata. v0.37+ structured-domain-on-takes
  (Hindsight-style enum) tightens this.

Output:
  Returns { bias_tag: string, context: string } | null.
  Context format: "This contradiction fits your active bias pattern
  \"<tag>\" (Brier 0.31). Verdict: contradiction; severity: medium.
  Consider reviewing both sides through the lens of that pattern."

Tests: 13 cases.
  R2 regression (2): null profile → null tag; empty active_bias_tags → null tag.
  computeDomainHint (5): companies / people / macro / geography / unknown
  paths produce expected hints.
  Match path (4): macro→late-on-macro-tech, geography→over-confident-geography,
  mismatch returns null, first-match-wins with multiple candidate tags.
  buildBiasContextString (2): emits tag+verdict+severity+Brier; omits
  Brier when null (no "Brier null" leak).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: Brier-trend forecast at write time (T10 / E5)

Pure math layer over existing TakesScorecard data. Zero new LLM cost, zero
new schema. Surfaces the user's historical Brier for the take's
(holder, domain) bucket at write time so they see "your historical Brier
in macro takes is 0.31" before committing the take.

Voice-gate-rendered output:
  The user-facing string goes through gateVoice mode='forecast_blurb' via
  templates.ts (already in T6). This module is the pure data layer; the
  template renders the math into the conversational voice.

v0.36.0.0 ship state:
  Bucket dimension is the DOMAIN (slug-prefix). The conviction-weight
  bucket dimension would need a new engine method
  (engine.batchGetTakeBucketStats per F11) — deferred to v0.37+. Until
  then, forecast = historical Brier in this holder's domain.

  resolveDomainPrefix() keeps slug-prefix-looking domain hints
  ('companies/', 'wiki/macro') and falls back to overall for free-form
  hints ('macro tech', 'geography'). Hindsight-style structured domain
  on takes (CDX-11 mitigation TODO) tightens this in v0.37+.

MIN_BUCKET_N = 5:
  Below this sample size, the forecast returns predicted_brier=null with
  insufficient_data=true. Template renders "Forecast unavailable: only N
  resolved takes at this conviction yet" instead of a noisy estimate.

Architecture:
  computeForecast(input) — pure function, takes scorecards already
  fetched; ideal for tests + reuse across batched paths.
  forecastForTake(engine, input) — convenience wrapper, 1-2 engine
  round-trips (no domain → 1; with domain → 2).
  batchForecast(engine, inputs[]) — memoizes per (holder, domainPrefix);
  N inputs collapse to ≤2*unique_holders unique engine calls. Used by
  the propose-queue review flow (50 candidates → 1-2 scorecard fetches).

Tests: 14 cases.
  computeForecast (4): insufficient_data branch, stable forecast,
    overall fallback, MIN_BUCKET_N export.
  resolveDomainPrefix (5): undefined/empty/whitespace → undefined;
    slug-prefix → kept; free-form → undefined.
  forecastForTake (3): 1-call overall, 2-call domain, free-form fallback.
  batchForecast (2): cache collapse for repeat queries; different holders
    do not collapse.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: gstack-learnings coupling on incorrect resolutions (T11 / E4)

When the grade_takes phase auto-resolves a take as 'incorrect' or 'partial',
optionally write a learning entry to gstack's per-project learnings.jsonl
so other gstack skills (plan-ceo-review, ship, investigate, ...) can pull
it as context when relevant. The brain teaches every other tool about
the user's track record.

Config gate (D5 / CDX-17 mitigation):
  `cycle.grade_takes.write_gstack_learnings` defaults FALSE. External
  users may not have gstack installed; the gstack-learnings binary API
  isn't stable yet. Garry's brain flips it true to opt in.

Quality gate:
  Only 'incorrect' and 'partial' verdicts trigger the write. 'correct'
  resolutions are noise (we expected the take to hold up — no learning).
  'unresolvable' has no canonical column. Defense-in-depth runtime guard
  in writeIncorrectResolution() rejects ineligible qualities with
  reason='quality_not_eligible' so a caller misuse never surfaces a
  malformed learning entry.

Auto-apply only:
  Coupling fires only when grade_takes both auto-applies AND the verdict
  is incorrect/partial AND the config flag is enabled. Manual resolutions
  via `gbrain takes resolve` intentionally DO NOT propagate to gstack —
  manual writes already carry operator intent; the calibration loop is
  the noise-prone path that earns coupling.

Namespace:
  Every entry's key starts with 'gbrain:calibration:v0.36.0.0:'. Lane D
  `gbrain calibration --undo-wave v0.36.0.0` (T17) filters on this prefix
  for the optional gstack-scrub step. First active bias tag suffixes the
  key (e.g. 'take-42:over-confident-geography') so future analysis can
  group learnings by bias pattern.

Architecture:
  buildLearningEntry — pure. Truncates claim at 200 chars + ellipsis;
  emits Pattern: line when activeBiasTags present; defaults confidence
  to 0.8 when caller omits it.

  writeIncorrectResolution — async wrapper. Honors config gate; honors
  quality gate; calls the injected writer (or defaultGstackWriter in
  production). Failures are non-fatal: returns
  { written: false, reason: 'write_failed' | 'binary_missing', error }.
  The grade_takes phase logs to result.warnings and continues — gstack
  coupling failure NEVER aborts a cycle.

  defaultGstackWriter — shells out to gstack-learnings-log binary via
  execFileSync. Throws GBrainError('GSTACK_BINARY_NOT_FOUND') when the
  binary isn't on PATH; writeIncorrectResolution classifies that error
  to reason='binary_missing' so the operator sees the install hint
  instead of a generic write_failed.

  Wired into grade-takes.ts after engine.resolveTake() inside the
  auto-apply block. Only fires when shouldApply=true.

Tests: 14 cases.
  buildLearningEntry (7): canonical shape, partial vs incorrect wording,
  bias-tag suffix, no-tag fallback, claim truncation, default confidence,
  no-reasoning omission.
  writeIncorrectResolution (7): config gate, quality gate, happy path,
  writer-throw graceful degrade, binary-missing classification, async
  writer awaited, partial quality writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doctor: 4 calibration checks — abandoned/freshness/drift/voice (T12)

Adds the four calibration doctor checks per the eng-review spec.

abandoned_threads:
  Counts active high-conviction takes (weight >= 0.7) older than 12 months
  that have never been superseded. Signal, not error — always status='ok'
  with a count. The hint sends users to `gbrain calibration` for details.

calibration_freshness:
  Warns when the active profile is older than 7 days (configurable via
  the same env-var pattern other freshness checks use). Cold-brain branch
  (no profile yet) returns ok without scolding. Hint points at
  `gbrain calibration --regenerate`.

grade_confidence_drift (CDX-11 mitigation):
  Surfaces the count of auto-applied grade verdicts. Below 30: returns
  "need 30+ for drift detection". At/above 30: returns "drift math
  arrives in v0.37+". The surface is wired; the actual
  confidence-vs-accuracy correlation math is a v0.37+ follow-up once we
  have 30+ auto-applied verdicts to measure against. Closes the CDX-11
  hole structurally — the operator sees the surface even before the math
  is meaningful.

voice_gate_health:
  Tracks voice gate failure rate over the last 7 days. <30% fail rate →
  ok (template fallback is fine in isolation). >=30% → warn with hint
  to review src/core/calibration/voice-gate.ts rubric. Anchors the
  cross-cutting voice rule observability story.

All four checks return status='warn' with a diagnostic message on
engine errors — non-blocking, never throws. Matches the existing doctor
check pattern (see checkSyncFreshness for prior art).

Wired into runDoctor after checkRerankerHealth (the v0.35 cluster), in
the canonical block 10 slot.

Tests: 15 cases. 4 per check (happy path, alt-status, engine-throw
diagnostic, plus boundary tests for the freshness staleness gate at
exactly 7 days and the grade drift gate at 30 applied verdicts).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: E7 nudge + 14-day cooldown (T13 / D16 F3)

Real-time pattern surfacing when a newly-committed high-conviction take
matches an active bias pattern. Conversational nudge text via the
templates module; 14-day cooldown per (take_id, nudge_pattern) via
take_nudge_log to prevent the feedback loop where each cycle re-fires
the same nudge on the same take.

Threshold gates (D16 F3):
  - holder match (profile.holder === take.holder)
  - conviction-weight > 0.7 (strict greater than)
  - take's slug-derived domain hint matches an active bias tag
    (takeDomainHint — same heuristic as eval-contradictions/calibration-join.ts
    for cross-surface consistency)

Cooldown gate:
  Before firing, probe take_nudge_log for (take_id, nudge_pattern) rows
  with fired_at >= now() - 14 days. Any hit → silently skip. After firing,
  insert a new row with channel='stderr' so the next 14 days are gated.

Feedback-loop prevention:
  User hedges a take in response to a nudge (e.g. weight 0.85 → 0.65).
  Even though the take's `weight` field changed, the cooldown row for
  the over-confident-geography pattern is still there from the original
  fire — so the next cycle's evaluateAndFireNudge() silently skips. The
  user reset path (gbrain takes nudge --reset N) clears the cooldown to
  re-arm.

Output channel (v0.36.0.0 ship state):
  STDERR only. Schema's `channel` column already supports multi-channel
  (webhook, admin SPA toast); routing those is a v0.37+ follow-up.

Architecture:
  evaluateNudgeRule(take, profile) — pure rule check. Returns
  { matched, reason, matchedTag }. No engine call.
  checkCooldown(engine, takeId, pattern) — engine probe, returns boolean.
  recordNudgeFire(engine, opts) — INSERT into take_nudge_log.
  evaluateAndFireNudge(opts) — full pipeline. Returns NudgeDecision.
  resetNudgeCooldown(engine, takeId) — DELETE...RETURNING for the CLI.

  buildNudgeText delegates to templates.ts nudgeTemplate (D24 mode='nudge'
  voice). v0.36.0.0 ship state uses the template directly; LLM-generated
  nudge text via the voice gate lands in v0.37+ when we have production
  examples to tune from.

Tests: 22 cases.
  takeDomainHint (5): companies/people/macro/geography/unrecognized.
  evaluateNudgeRule (6): no_profile, wrong_holder, conviction-at-threshold-
  is-NOT-eligible (strict >), no matching tag, happy match,
  first-match-wins for multiple candidate tags.
  checkCooldown (3): true on row hit, false on no row, cutoff date param
  verifies the 14-day boundary.
  evaluateAndFireNudge (4): happy fire (text contains hush command +
  matched tag), cooldown silent skip (no INSERT, no stderr), no_profile
  short-circuit, below-conviction short-circuit (no cooldown query fired).
  buildNudgeText (2): hush command shape, conviction value embedded.
  resetNudgeCooldown (2): returns count, idempotent on zero rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: E8 team-brain sharing + D18 cross-brain query semantics (T14)

Cross-brain calibration profile resolution per the D18 4-rule contract.
Pins all four cross-brain leak surfaces in dedicated unit tests so future
mount features can't silently regress this security model.

D18 semantics (committed):

  Rule 1 — LOCAL-FIRST ORDERING.
    Query the local brain first. If a profile exists, return it. Do NOT
    also query mounts (avoids stale-mount-overrides-fresh-local).
    Verified: mountResolver is NOT called when local has a hit.

  Rule 2 — MOUNT FALLBACK.
    Only when local has no profile AND canReadMounts=true, walk the
    mounts in priority order. First match wins. Each mount-side row
    must have published=true to be visible (D15 asymmetric opt-in).

  Rule 3 — CROSS-BRAIN ATTRIBUTION.
    Every returned profile carries source_brain_id + from_mount flag.
    Consumers (E1 think rewrite, E3 contradictions, E7 nudge, E6
    dashboard) MUST surface this via attributionSuffix() so the user
    sees which brain answered.

  Rule 4 — SUBAGENT PROHIBITION.
    canReadMountsForCtx() classifier returns FALSE for subagent loops
    without trusted-workspace allowedSlugPrefixes. Closes the
    OAuth-token-to-cross-brain-leak surface — subagents see ONLY their
    local-brain results regardless of which holder they query.

    Exception: trusted cycle phases (synthesize/patterns) pass
    allowedSlugPrefixes set and ARE allowed to read mounts. Pinned in
    the classifier test.

Architecture:
  queryAcrossBrains(localEngine, opts) — pure orchestrator. Composes
  getLatestProfile() from src/commands/calibration.ts. Mount engine
  access is via opts.mountResolver — production wires this to the
  v0.19+ gbrain mounts subsystem; tests inject a stub returning an
  ordered list of mocked engines. Decouples cross-brain LOGIC from
  multi-engine PLUMBING.

  canReadMountsForCtx(ctx) — pure classifier table. Drives the rule-4
  gate. Production callers compose it from OperationContext.

  attributionSuffix(result) — pure formatter. Emits the "(from mounted
  brain: <id>)" suffix when from_mount=true; empty string when local.
  Mandatory for user-visible cross-brain consumers.

Tests: 15 cases pinned to the 4 D18 rules + 4 supplementary structural
checks.
  D18-1: published=false profile on mount stays hidden.
  D18-2/3: subagent context cannot fall back to mounts (2 cases — null
    on local-empty + canReadMounts=false, local hit still returned).
  D18-4: attribution surfaces source_brain_id (3 cases — mount answer
    flag, local answer flag, attributionSuffix formatter).
  Rule 1 local-first ordering (2 cases — mountResolver NOT called on
    local hit, IS called on local empty).
  Mount priority order (3 cases — first published=true wins, all
    published=false returns null, no mounts configured returns null
    without throwing).
  canReadMountsForCtx classifier (4 cases — local CLI true, MCP
    non-subagent true, subagent without trusted-workspace false,
    subagent WITH trusted-workspace true).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* admin: E6 Calibration tab + D23 server-rendered SVG + TD2 contrast bump (T15)

Adds the v0.36.0.0 admin SPA Calibration tab. Per the design review,
the approved variant-B (Linear calm clarity) layout: single-column flow,
generous whitespace, ONE big sparkline as hero, then patterns, then
domain bars, then abandoned threads.

D23 server-rendered SVG architecture:

  src/core/calibration/svg-renderer.ts — pure functions. data → SVG
  string. No DOM, no React, no chart library dep. Inlines the admin
  design tokens (#0a0a0f bg, #3b82f6 accent, etc.) so the SVG is
  visually consistent with the rest of the admin SPA.

  Four chart renderers:
    - renderBrierTrend({ series }) — sparkline w/ baseline reference
      at 0.25 (always-50% baseline)
    - renderDomainBars({ bars }) — horizontal accuracy bars per domain
    - renderAbandonedThreadsCard(threads) — D30/TD4 'revisit now' link
      per row, points at /admin/calibration/revisit/<takeId>
    - renderPatternStatementsCard(statements) — D29/TD3 clickable
      drill-down links per row, point at /admin/calibration/pattern/<i>

  XSS posture: all caller-controlled strings pass through escapeXml().
  Numeric inputs are .toFixed()-coerced. Admin SPA renders via
  dangerouslySetInnerHTML inside a TrustedSVG wrapper component;
  endpoint is gated by requireAdmin middleware.

  /admin/api/calibration/profile — returns the active profile row as JSON.
  /admin/api/calibration/charts/:type — returns image/svg+xml markup
    for type ∈ {brier-trend, domain-bars, pattern-statements,
                abandoned-threads}. Cache-Control: private, max-age=60.

  brier-trend currently renders a single-point series from the active
  profile (the time-series view across calibration_profiles.generated_at
  history is a v0.37 follow-up once we have multiple snapshots).
  abandoned-threads pulls the top 5 abandoned rows via the same SQL the
  doctor check uses.

CalibrationPage React component (admin/src/pages/Calibration.tsx):
  Fetches profile + 4 charts. Loading / error / cold-brain states all
  handled. Layout includes the audit annotations (partial-grade badge,
  voice-gate-fell-back-to-template badge) per the approved mockup.
  TrustedSVG wrapper isolates the dangerouslySetInnerHTML to the SVG
  surface only.

App.tsx nav: added 'calibration' page route + sidebar nav item, hash
routing extended to support #calibration.

TD2 contrast bump:
  admin/src/index.css --text-muted: #555 → #777. Old value was contrast
  4.0 on the #0a0a0f bg — below WCAG AA 4.5 for body text. New value is
  ~5.5, passes AA. Improvement is global across Dashboard, Agents,
  RequestLog, and the new Calibration tab — single-line CSS change with
  ~10x the impact.

admin/dist/ rebuilt via `bun run build` (vite). 36 modules transformed.

Tests: 19 cases in test/svg-renderer.test.ts.
  escapeXml (1): canonical entities.
  renderBrierTrend (6): empty state, polyline for 2+ points, clamp
  beyond yMax, design tokens inlined, XSS safety on date strings,
  text-anchor end on right label.
  renderDomainBars (4): empty state, label/accuracy/n rendering,
  out-of-range accuracy clamp, XSS safety on labels.
  renderAbandonedThreadsCard (4): empty state, row rendering with
  revisit link, claim truncation at 70 chars, custom revisitHref override.
  renderPatternStatementsCard (4): empty state, anchor count matches
  statement count, XSS safety, custom drillHref override.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* recall: calibration footer formatter for morning pulse (T16)

Pure formatter that turns a CalibrationProfileRow + optional abandoned-
threads list into the conversational block the morning pulse will surface:

  Calibration this quarter:
    Brier 0.18 (solid).
    Right on early-stage tactics, late on macro by 18 months.
    Over-confident on team execution; under-calibrated on regulatory risk.

  Threads you opened and never came back to:
    · AI search platform differentiation         (17 months silent)
    · International expansion playbook           (12 months silent)

Cold-brain branch: returns empty string when no profile or < 5 resolved
takes. Caller decides whether to render the block; cold-brain absence
is the cleanest non-event.

Brier trend note maps the absolute value to conversational copy:
  <= 0.10 → "(strong calibration)"
  <= 0.20 → "(solid)"
  <= 0.25 → "(near baseline)"
  > 0.25  → "(worse than always-50% baseline — review your high-conviction calls)"

  v0.36.0.0 ship state has only the current profile snapshot. The
  "was 0.22 90d ago — improving" comparison shape arrives when we
  accumulate generated_at history across multiple cycles.

R3 regression posture:
  This module is the FORMATTER only. Wiring into `gbrain recall`'s text
  output is intentionally NOT in this commit — runRecall's surface
  stays unchanged. v0.37 wires it under --show-calibration (opt-in
  initially, default-on later). For now the formatter is callable from
  the admin tab + custom CLI scripts that want it.

Architecture:
  buildRecallCalibrationFooter(opts) — pure. opts.profile required,
  opts.abandonedThreads optional, opts.threadColumnWidth defaults to 50.

  Caps at 4 patterns + 5 abandoned threads to keep the footer scannable.
  Truncates long abandoned-thread claim text to fit the column width with
  a trailing ellipsis.

Tests: 14 cases.
  Cold-brain branch (3): null profile, < 5 resolved, zero resolved.
  Happy path (7): header + Brier + patterns, trend note ranges (4
  brackets), null brier omits the Brier line but keeps header, caps at
  4 patterns.
  Abandoned threads (4): omit section when none, emit when present,
  cap at 5, truncate long claim with column-width override.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: --undo-wave reversal command (T17 / D18 CDX-3)

Implements the undo-wave reversal flow. Every new row written by the
v0.36.0.0 calibration wave carries wave_version='v0.36.0.0' so a precise
revert is possible without touching pre-wave data.

CLI surface (replaces the v0.36.0.0 ship-state placeholder):
  gbrain calibration --undo-wave v0.36.0.0 [--dry-run] [--scrub-gstack] [--json]

Reversal scope (4 steps):

  Step 1 — UNSET takes.resolved_* columns for takes auto-applied by this
  wave. Identifies wave-applied takes via take_grade_cache.applied=true
  + wave_version match. Cross-checks resolved_by='gbrain:grade_takes' to
  ensure we're not un-resolving a take a manual `gbrain takes resolve`
  override has since claimed. Manual resolutions persist; only auto-grade
  resolutions revert.

  Step 1b — Mark take_grade_cache rows applied=false post-undo so the
  audit trail shows they WERE applied but this wave was reverted. The
  CDX-11 confidence-drift check filters on applied=true and gets a
  cleaner sample post-undo.

  Step 2 — DELETE FROM calibration_profiles WHERE wave_version = ?.

  Step 3 — DELETE FROM take_nudge_log WHERE wave_version = ?.

  Step 4 — Optional gstack-learnings-prune via the binary, scoped to the
  GSTACK_LEARNING_NAMESPACE prefix. Opt-in via --scrub-gstack. Best-effort:
  binary-missing or failure logs a warning + suggests the manual command;
  the rest of the undo still succeeded.

Dry-run posture:
  --dry-run computes the counts via SELECT COUNT(*) shapes without
  emitting any UPDATE or DELETE. Same UndoWaveResult shape returned so
  operator sees exactly what would be reverted before committing.

  --dry-run intentionally skips the gstack scrub (filesystem write) too;
  ship-state safety call.

Idempotency:
  Re-running --undo-wave on a brain that's already reverted is a no-op.
  Each query filters on wave_version; no matching rows → zero counts.

Architecture:
  undoWave(engine, opts) — async, returns UndoWaveResult. Pure data
  layer; no stderr writes, no process exits. CLI dispatch in
  src/commands/calibration.ts handles printing.

  v0.36.0.0 ship state runs steps 1-3 sequentially (no transaction).
  Partial reversal is recoverable via re-run since each step is
  idempotent on wave_version match. A future enhancement (v0.37+) can
  wrap in engine.transaction once that surface lands in BrainEngine.

Tests: 8 cases in test/undo-wave.test.ts.
  Dry-run posture (1): counts emitted, NO UPDATE/DELETE SQL fired.
  Happy path (3): all 4 steps execute, resolved_by filter scopes UPDATE
  to wave-applied resolutions, custom resolvedByLabel honored.
  Empty wave (2): zero counts when no matching rows, idempotent re-run.
  Wave-version parameter threading (2): supplied version threads
  through all queries, different wave versions don't collide.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: A/B harness for think + ab-report (T18 / D19 CDX-18)

Structural answer to CDX-18 (anti-bias rewrite may make advice worse).
We don't have to guess whether calibration helps — we measure.

Architecture:
  runAbTrial(input) — calls thinkRunner TWICE on the same question
  (baseline + --with-calibration), surfaces both answers to a
  preferenceResolver, persists the trial to think_ab_results.

  buildAbReport(engine, { days }) — aggregates the table over the last
  N days (default 30). Computes win counts, ties, neither, and a
  with_calibration_win_rate over DECISIVE trials only (excludes
  neither/tie). Flags calibration_net_negative when n >= 20 AND win
  rate < 45%.

  formatAbReport(report, days) — pretty-prints for stdout; emits the
  calibration_net_negative warning block when triggered.

CLI:
  gbrain calibration ab-report [--days N] [--json]
    Reads the table, prints the breakdown. Replaces the v0.36.0.0
    ship-state placeholder in src/commands/calibration.ts.

  gbrain think --ab "<question>"
    Wires into runAbTrial via the dispatch in src/commands/think.ts —
    follow-up commit. This commit lands the harness layer + schema +
    report surface; the --ab flag itself flips on in a one-line wiring
    commit when the runRecall path is ready.

Schema (migration v72 / think_ab_results):
  source_id, wave_version, ran_at, question, baseline_answer,
  with_calibration_answer, preferred (CHECK in {baseline,
  with_calibration, neither, tie}), model_id, notes.

  CHECK constraint enforces preferred enum. Default wave_version
  'v0.36.0.0' stamped so --undo-wave can scrub these too.

  Index on (source_id, ran_at DESC) supports the report's
  "last N days" query.

  schema.sql + pglite-schema.ts both updated for fresh-install parity.
  schema-embedded.ts regenerated via build:schema.

calibration_net_negative threshold (D19):
  Triggers when:
    - decisive_trials (baseline + with_calibration) >= 20
    - with_calibration_win_rate < 0.45 (NOT <= — exact 45% is OK)

  Small-sample guard (n < 20) prevents the warning from firing on
  early data with sampling noise. Confidence-flat threshold (no Wilson
  CI yet) keeps the math simple; v0.37+ adds CI bounds.

Tests: 12 cases in test/think-ab.test.ts.
  runAbTrial (4): both runner calls fire, preferenceResolver receives
    both answers, INSERT row params shape, throws when thinkRunner
    missing.
  buildAbReport (5): zero trials, aggregation, net_negative trigger at
    n>=20 + win<45%, no trigger at n<20 (small-sample guard), no
    trigger at exact 45% boundary.
  formatAbReport (3): zero-state message, decisive-trials breakdown,
    net_negative warning block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: pattern drill-down route + revisit-now CLI (TD3 / D29 + TD4 / D30)

TD3 (D29) — clickable pattern drill-down endpoint:
  GET /admin/api/calibration/pattern/:id (requireAdmin)
  Returns the pattern statement at index `id` plus the top 25 resolved
  takes for the holder, sorted by weight desc. v0.36.0.0 ship-state
  approximation: surfaces broad provenance evidence (top resolved
  takes). v0.37+ stores per-pattern source_take_ids[] on a
  calibration_profile_patterns join table so the drill-down shows the
  EXACT takes that drove the pattern.

  Surfaces a `provenance_note` field in the response so the operator
  sees the v0.36.0.0-vs-v0.37 fidelity boundary inline.

  The admin SPA's renderPatternStatementsCard SVG already emits anchor
  tags pointing at /admin/calibration/pattern/<i> (T15 ship state).
  This route makes those anchors clickable — closes the trust loop that
  was the rationale for D29 ("pattern statements without their evidence
  are dressed-up LLM hallucinations").

TD4 (D30) — `gbrain takes revisit <slug>` editor-open action:
  Adds the `revisit` subcommand to gbrain takes. Opens $EDITOR (falling
  back to vi) on the source markdown file for the slug. Appends a
  `<!-- gbrain:revisit -->` cursor marker at the bottom of the page on
  first invocation so the editor opens with intent visible.

  Reads sync.repo_path from config to locate the brain repo. Refuses to
  proceed with a clear error when the repo isn't configured or the page
  doesn't exist.

  spawnSync with stdio:'inherit' so the editor takes the terminal. Exit
  status surfaced on failure.

  The SVG renderer's revisit-now anchor for each abandoned thread row
  emits /admin/calibration/revisit/<takeId>. A small route handler that
  resolves take_id → page_slug then dispatches `gbrain takes revisit`
  via spawn is a v0.37 follow-up — the CLI command exists now so
  developers can wire it directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: DESIGN.md — formalize de facto design tokens (TD1)

Promotes the admin SPA's de facto design tokens (landed v0.26.0) to a
canonical DESIGN.md at the repo root. This is the calibration target
for /plan-design-review and /design-review going forward — when a
question is "does this UI fit the system?", the answer is here.

Captures the system as it stands today:

  Voice (5 surfaces, all routed through gateVoice() with mode-specific
  rubrics): pattern_statement, nudge, forecast_blurb, dashboard_caption,
  morning_pulse. Friend-not-doctor; concrete data over abstract metrics;
  no preachy / clinical / corporate language.

  Color tokens: 10 CSS variables from admin/src/index.css inlined into
  the SVG renderer (src/core/calibration/svg-renderer.ts). Dark theme
  is the only theme — admin is an operator tool. WCAG contrast
  documented per token; TD2's #555 → #777 bump on --text-muted noted.

  Typography: Inter for UI, JetBrains Mono for numbers/slugs/data.
  Type scale (18 / 14 / 13 / 12 / 11) documented as de facto, not yet
  formalized.

  Spacing scale: 4 / 8 / 16 / 24 / 32px. Linear-app density.

  Layout: sidebar 200px, max content 720px (text) / 960px (tables).
  No 3-column feature grids, no icons in colored circles, no
  decorative blobs.

  Charts: server-rendered SVG via pure functions in
  src/core/calibration/svg-renderer.ts. XSS posture documented:
  server-side escapeXml on caller-controlled strings, numeric inputs
  .toFixed()-coerced, admin SPA renders via <TrustedSVG> wrapper.

  Interaction patterns: keyboard nav required (J/K/space/u/q on the
  propose-queue), loading/empty/error states ARE features.

  v0.37+ roadmap: type scale formalization, animation tokens, component
  library extraction. Light mode explicitly NOT planned.

The doc is a living target, not a frozen spec. Major changes route
through /plan-design-review per the existing review chain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: synthetic corpus scaffold + privacy CI guard (T19 + T20)

T19 — synthetic corpus scaffold for extract-takes prompt tuning.
  test/fixtures/calibration/extract-takes-corpus/ — 5 representative
  pages across 4 genres (essay, people, companies, meetings, decisions).
  v0.36.0.0 ships a SMALL representative corpus as proof of structure;
  the full 50-page training set + 10-page holdout gets generated by the
  operator via `gbrain calibration build-corpus` (v0.37 follow-up
  subcommand) or by hand with the privacy guard catching violations
  either way.

  Privacy contract per D13': every page is SYNTHETIC. None of the
  names/companies/funds/deals/events refer to anything real. Placeholder
  names per CLAUDE.md: alice-example, charlie-example, acme-example,
  widget-co, fund-a/b/c, acme-seed, widget-series-a, meetings/2026-04-03.

  test/fixtures/calibration/README.md spells out the privacy contract,
  generation flow, and what the corpus is (stable regression set for
  the extract-takes prompt) vs is not (real anything).

T20 — privacy CI guard (CDX-14 mitigation).
  scripts/check-synthetic-corpus-privacy.sh greps the corpus for:
    1. Explicit dollar amounts ($50M, $1.2B etc) — would suggest the
       page memorized a real round size.
    2. Out-of-range year references (informational only for v0.36.0.0;
       deferred to a manual review checklist).
    3. Pages that reference ZERO placeholder names — suggests the page
       might be referring to real entities. Essay-genre fixtures
       exempt (they're anonymized PG-style writing by design).

  Wired into `bun run verify` (CI gate) so contributors can't accidentally
  land a synthetic fixture that leaks real-world specificity. The intent
  is fail-fast on accidental leakage; the operator can update the
  allowlist if a generic dollar amount is intentional.

  Closes CDX-14: 'CC reads real brain pages locally, writes nothing
  still risks privacy if any generated synthetic fixture memorizes
  structure-specific facts. Placeholder names are not enough.'

The corpus shipped here is intentionally small but covers the four
core gbrain page genres (essay, people, companies, meetings/decisions).
The v0.37 corpus-build subcommand will fan out to 50 with the operator
spot-checking + the CI guard enforcing the privacy contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: R1-R5 IRON RULE regression inventory (T21)

Per /plan-eng-review D26 IRON RULE: regressions get added to the test
suite as critical requirements, no AskUserQuestion needed. Pins five
regressions identified during the v0.36.0.0 wave's coverage diagram:

  R1: think baseline UNCHANGED when --with-calibration absent.
      Covered structurally by test/think-with-calibration.test.ts plus
      assertion-pinned in this file (default user message: question
      first, then retrieval; system prompt: no anti-bias section).

  R2: contradictions probe output UNCHANGED when no calibration profile.
      Covered structurally by test/eval-contradictions-calibration-join.test.ts
      plus pinned here (null profile → null tag, byte-identical to v0.32.6).

  R3: takes resolution flow works when grade_takes phase disabled.
      Pinned import-surface coupling: takes-resolution.ts has zero
      dependency on grade_takes module. If a future refactor accidentally
      couples them, this test fails to compile.

  R4: search/list_pages/get_page work identically through new source_id paths.
      Marker test referencing existing v0.34.1 source-isolation suite at
      test/source-isolation-pglite.test.ts. v0.36.0.0 does NOT modify
      those code paths; the existing tests catch any accidental coupling.

  R5: existing search modes (conservative/balanced/tokenmax) unaffected.
      Marker test referencing existing test/search-mode.test.ts. The
      calibration code DOES NOT IMPORT from src/core/search/mode.ts.

Plus an inventory test that confirms all 5 regressions have an
'addressed' status — fail-loud if a future contributor removes a
guard without updating the inventory.

7 tests total. Pure functions, no engine, hermetic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.36.0.0 CHANGELOG + CLAUDE.md anchors + calibration convention skill

CHANGELOG entry: the user-facing release notes. Leads with the headline
("the brain learns how you tend to be wrong, then argues against your
blind spots on every advice call"), 5 'what you can now do' bullets in
GStack voice, itemized changes by lane, and the 'To take advantage of
v0.36.0.0' upgrade checklist per the CLAUDE.md required-block contract.

CLAUDE.md anchors: new 'v0.36.0.0 Hindsight calibration wave (key files
cluster)' block inserted before the v0.31.1 thin-client section. 23 new
files / extensions annotated with one-paragraph descriptions each,
linking back to the convention skill at skills/conventions/calibration.md
for the agent-facing rules.

skills/conventions/calibration.md: the agent-facing convention skill.
Tells future contributors which calibration touchpoint applies to
their task — voice gate? BaseCyclePhase? source-scope thread? doctor
warning? cross-brain query rules? auto-resolve threshold posture? Test
seam patterns. Bug class to avoid (the v0.34.1 source-isolation leak
shape).

Version trio (per CLAUDE.md mandatory audit):
  VERSION:     0.36.0.0
  package.json: 0.36.0.0
  CHANGELOG:   ## [0.36.0.0] - 2026-05-17

llms.txt + llms-full.txt regenerated via `bun run build:llms` after
the CLAUDE.md edit (per the explicit CLAUDE.md mandate "Any CLAUDE.md
edit MUST be followed by `bun run build:llms`"). The `test/build-llms.test.ts`
guard runs in CI shard 1; the committed bundles are checked against
fresh generator output.

bun run verify is clean. typecheck clean. Privacy CI guard passes
(0 violations across 6 corpus pages). All ready for /ship.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cycle: wire propose_takes / grade_takes / calibration_profile into runCycle (T-fix)

The three new v0.36.0.0 phases were declared in CyclePhase / ALL_PHASES /
NEEDS_LOCK_PHASES but the runCycle orchestrator never dispatched them.
ALL_PHASES advertised them, gbrain dream --phase propose_takes accepted
them, but `gbrain dream` (default) silently skipped all three.

Adds a single dispatch block between consolidate and embed that:
  - builds an OperationContext on the fly (trusted-workspace caller,
    remote: false, sourceId resolved via the same helper sync uses)
  - dispatches the three phases in the order ALL_PHASES declares
  - records the same skipped-phase shape (no_database) when engine is null

Pinned by test/core/cycle.serial.test.ts "default: all 6 phases run in
order" which was already failing against ALL_PHASES (the test name lags
the actual phase count; left as-is since renaming churns history).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: expand synthetic corpus + add hand-labeled ground-truth (T19)

Adds 8 new synthetic pages modeled on the genre mix observed in the
real brain (concepts-with-timeline, meeting-notes, daily-journal,
people-pages, essays). Companion .gradeable-claims.json files carry
hand-labeled answer keys — what a tuned propose_takes prompt SHOULD
extract per page. Closes the F1 gate gap from the plan's T19/D19:

  Training corpus (test/fixtures/calibration/extract-takes-corpus/):
    + concept-startup-market-dynamics.md     (10 claims)
    + meeting-2026-04-10-fundraise-fund-a.md (6 claims)
    + daily-2026-04-15.md                    (5 claims)

  Blind holdout (test/fixtures/calibration/holdout/):
    + concept-founder-execution.md           (6 claims, F1 >= 0.80)
    + daily-2026-04-18.md                    (4 claims, F1 >= 0.80)
    + meeting-2026-04-17-hiring-charlie.md   (5 claims, F1 >= 0.80)
    + essay-on-conviction.md                 (7 claims, F1 >= 0.80)
    + people-bob-example.md                  (5 claims, F1 >= 0.80)

Privacy:
  - No real-brain content read into any committed artifact. Pages
    written from scratch using the canonical placeholder set
    (alice-example, charlie-example, bob-example, acme-example,
    widget-co, fund-a/b/c). Real-name grep confirms zero leakage:
    wintermute, garrytan, paul-graham, sam-altman, etc. → 0 hits.
  - scripts/check-synthetic-corpus-privacy.sh passes: 0 violations
    across 14 pages (was 6).

Genre fidelity:
  - concept-with-timeline pages mirror the dated-assertion structure
    real brain uses (verb framing varies: "argues / predicts / I
    think / I bet / strong conviction / moderate conviction").
  - meeting-notes pages carry both prose claims (extracted via
    hedging language) and explicit ## Takes sections.
  - daily-journal pages test probabilistic framing ("75/25 in favor",
    "call it ~0.5") and self-tagged conviction values.
  - essay-on-conviction is the meta-page that names the author's
    own bias patterns — primary signal for calibration_profile.
  - people pages test claim-about-third-party extraction.

Each JSON ground-truth lists per-claim:
  - claim_text + kind (prediction|judgment|bet) + domain
  - conviction (0..1)
  - since_date
  - rationale (why this claim is gradeable + how a tuned prompt
    should infer conviction from the prose)

This is the corpus that gates the T19 prompt-tune iteration:
  - F1 >= 0.85 on training (10+6+5 = 21 claims across 3 pages
    plus the existing 5 fixtures already shipped)
  - F1 >= 0.80 on holdout (27 claims across 5 pages)

Plan reference: ~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md
Privacy gate: scripts/check-synthetic-corpus-privacy.sh (wired into bun run verify).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* calibration: tune propose_takes prompt against synthetic corpus (cat15 F1 0.92+)

The v0.36.1.0 ship state shipped propose_takes with a stub prompt that
the docs flagged as "tune via T19 corpus build before relying on
propose_takes in production." T19's corpus was built in commit 69a71c9d
(14 synthetic pages + 48 hand-labeled claims). The matching gbrain-evals
cat15 runner validates extraction quality against that corpus.

This commit back-ports the tuned prompt validated by cat15's first live
run:

  training avg F1: 0.952  (target 0.85, +10 points)
  holdout  avg F1: 0.922  (target 0.80, +12 points)
  train-holdout gap: 0.03 (well below 0.10 overfitting threshold)
  8/8 probes pass their individual F1 targets

Per-genre F1 floor: 0.80 (people-pages, the hardest genre). Concept-
with-timeline and meeting-notes genres scored at 1.00 on holdout pages.

The tuned prompt design changes vs the stub:
  - Worked example list seeds the "gradeable claim" notion so the model
    doesn't drift into pure-fact extraction.
  - NOT-gradeable list catches the most common over-extraction modes
    (pure facts, direct quotes, restatements).
  - Conviction inference rules anchored to specific hedging language
    so the model produces consistent weight values.
  - kind enum narrowed to 'prediction' | 'judgment' | 'bet' — the v1
    stub's 4-tag enum bled into noise classification on the corpus.

PROPOSE_TAKES_PROMPT_VERSION bumped 'v0.36.1.0-stub' → 'v0.36.1.0-tuned-cat15'.
The bump invalidates the take_proposals idempotency cache so existing
proposal rows stay as audit history but the next cycle re-extracts
against the new prompt — exactly the design contract this version
field is for.

Re-tuning protocol: run cat15 in gbrain-evals against the fixtures
BEFORE bumping the version string. The train-holdout gap should stay
< 0.10. If a future tune drops below the cat15 gate, revert.

Source of evidence:
  - cat15 runner: ~/git/gbrain-evals/eval/runner/cat15-propose-takes.ts
  - Fixture corpus: test/fixtures/calibration/ (this repo, commit 69a71c9d)
  - Live run dumps: ~/git/gbrain-evals/eval/reports/cat15-propose-takes/*.json

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: link cat14/cat15 benchmark report from CHANGELOG + README

Adds the "Validated by published benchmarks" subsection to the v0.36.1.0
CHANGELOG entry and a "Calibration loop" section to the README's
"Receipts on the evals" surface. Both link to the new benchmark report
at gbrain-evals/docs/benchmarks/2026-05-18-brainbench-cat14-cat15-calibration.md.

CHANGELOG: also updates the propose_takes bullet to reflect that the
v0.36.1.0 ship state now includes the tuned 'v0.36.1.0-tuned-cat15'
prompt (back-ported in 04dbab44), not the v1 stub the original entry
described.

README: adds a Calibration loop entry to the receipts table sitting
between source-aware ranking and prompt compression. Frames the cat14
+ cat15 numbers as "first published benchmark for AI memory systems
that reason about user track records" — honest SOTA framing since
Hindsight introduced the concept without quantified evaluation.

llms.txt + llms-full.txt regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: fix benchmark-report links — gbrain-evals uses main not master

7 links to gbrain-evals/blob/master/docs/benchmarks/ were broken — the
gbrain-evals repo uses 'main' as its default branch, not 'master'.
Surfaced when I checked that the new cat14/cat15 link resolved post-PR-9
merge. Turned out 4 pre-existing links to longmemeval, brainbench-v0.20,
brainbench-cat13b-source-swamp, and comparison-systems were all broken
for the same reason — I just added a fifth by following the same wrong
pattern.

Sweep: gbrain-evals/blob/master/ → gbrain-evals/blob/main/ across both
README.md (5 links) and CHANGELOG.md (2 links).

llms.txt + llms-full.txt regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 19:34:44 -07:00
Garry TanandClaude Opus 4.7 03947665e4 v0.36.0.0 feat(skillpack): scaffold + reference + harvest (retire managed-block install) (#1130) 
* feat(skillpack): extract copyArtifacts shared helper (T1)

Pure file-copy primitive for scaffold (gbrain→host) and harvest (host→gbrain).
Atomic-refusal contract: symlink-reject + canonical-path containment validate
every item before any write. Used by both directions of the v0.33 loop.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): scaffold subcommand + SKILL.md frontmatter sources (T2)

New scaffold.ts replaces the managed-block installer. One-time additive copy
into the user's repo via copyArtifacts; refuses to overwrite existing files
(user owns them). Partial-state policy: copies missing paired sources even
when the skill dir already exists.

bundle.ts extended with loadSkillSources + enumerateScaffoldEntries — paired
source files declared in each SKILL.md's frontmatter sources: array, not in
openclaw.plugin.json. Single source of truth, co-located with the skill.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): reference command + apply-clean-hunks (T4 + T15)

reference is the read-only diff lens with an agent-readable framing line. Pure-JS
unified-diff producer + parser + applier (no patch(1) dependency). Two-way merge
with documented limitation: without scaffold-time base tracking, applied hunks
align everything to gbrain. The agent dry-runs reference first, then decides.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): migrate-fence + scrub-legacy-fence-rows (T5 + T16)

migrate-fence is the one-shot transition from the pre-v0.36 managed-block model.
Strips begin/end markers and the cumulative-slugs receipt comment; preserves
fence rows verbatim as user-owned routing during the transition to frontmatter
discovery. Receipt-then-row fallback (F-CDX-8) covers stale/missing receipts.

scrub-legacy-fence-rows is the opt-in cleanup after migrate-fence. Two-condition
gate: removes a row only when skills/<slug>/ exists AND that skill's frontmatter
declares non-empty triggers (proof frontmatter discovery covers it).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): harvest + privacy linter (T6 + T7)

The inverse loop: lift a proven skill from a host repo (~/git/wintermute, etc.)
back into gbrain so other clients can scaffold it. --from <host-repo-root> is
symmetric with scaffold's --workspace.

Security: symlink rejection + canonical-path containment (mirrors validateUploadPath).
Privacy: default-on linter scans harvested files against ~/.gbrain/harvest-private-patterns.txt
plus built-in defaults (Wintermute, email, Slack channel patterns). Any match
rolls back the copy and exits non-zero. --no-lint bypasses for the editorial
workflow after a manual scrub.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(repo-root): cwd_walk_up tier for non-OpenClaw hosts (T9 + D3)

autoDetectSkillsDir now walks up from cwd looking for any skills/ directory,
ahead of the implicit ~/.openclaw/workspace fallback. cd ~/git/wintermute &&
gbrain skillpack scaffold ... finds wintermute automatically without requiring
a RESOLVER.md/AGENTS.md to exist yet.

R5 regression preserved: $OPENCLAW_WORKSPACE still wins when explicitly set.
+5 test cases in test/repo-root.test.ts pin the new tier order and the R5 guard.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): rewrite CLI dispatch, drop install + uninstall (T3 + T10)

skillpack.ts dispatcher rewritten for the v0.36 contract: scaffold, reference
(+ --apply-clean-hunks), migrate-fence, scrub-legacy-fence-rows, harvest, plus
the existing list / diff / check.

install and uninstall are gone — both exit non-zero with a hint pointing at
scaffold / migrate-fence. Clean break, no deprecated alias.

skillpack-check gains --strict for CI gating. When invoked as the subcommand
`gbrain skillpack check`, default is informational (exit 0 even with drift);
--strict opts back into the cron-friendly exit-1-on-issues behavior. Top-level
gbrain skillpack-check preserves its existing exit semantics for backwards compat.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skills): skillpack-harvest editorial workflow + resolver wiring (T8)

The companion editorial skill for the gbrain skillpack harvest CLI. Walks the
genericization checklist (scrub fork names, generalize triggers, lift fork-
specific conventions to references) before the CLI runs. Routing-eval fixtures
use paraphrased intents to avoid the intent_copies_trigger lint.

Wires the new slug into openclaw.plugin.json#skills, skills/manifest.json, and
skills/RESOLVER.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(skillpack): 9-case real-subprocess E2E flow (T11)

Spawns gbrain as a subprocess against tempdir workspaces. Covers: scaffold
first-run + re-run no-op, reference diff + --apply-clean-hunks, migrate-fence,
scrub-legacy-fence-rows, harvest privacy-lint catch + --no-lint bypass, and
the install removed-error path. No DATABASE_URL needed — skillpack is
filesystem-only.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: docs + VERSION + CHANGELOG for v0.36.0.0 (T13 + T14)

Skillpacks as scaffolding, not amber.

v0.36 retires the managed-block install model. Six new subcommands replace
install + uninstall: scaffold, reference (with --apply-clean-hunks), migrate-fence,
scrub-legacy-fence-rows, harvest, plus the existing list / diff / check
(check gains --strict for CI gating). Routing comes from each skill's
frontmatter triggers — gbrain does not touch your RESOLVER.md or AGENTS.md.

Companion editorial skill skillpack-harvest drives the genericization
checklist; default-on privacy linter catches Wintermute / email / Slack
references before they leak into gbrain core.

New docs guide at docs/guides/skillpacks-as-scaffolding.md walks the model
and the migration path for pre-v0.36 installs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): privacy checks — allow-list harvest-lint tests, scrub user-facing fork-name references

CI's check-privacy.sh and check-test-real-names.sh both flagged the literal
fork name across the v0.36 skillpack diff. Two failure modes, two fixes:

1. **Meta-rule-enforcement files** added to both allow-lists. The harvest
   privacy linter's whole job is to catch the banned literal leaking into
   gbrain; its source has the regex pattern, its tests verify the linter
   fires by feeding it the banned string, and the skill markdown documents
   the substitution policy. Same exception status as check-privacy.sh and
   check-proposal-pii.sh themselves. Files allow-listed:
   - src/core/skillpack/harvest-lint.ts
   - test/skillpack-harvest-lint.test.ts
   - test/skillpack-harvest.test.ts
   - test/e2e/skillpack-flow.test.ts
   - skills/skillpack-harvest/SKILL.md

2. **User-facing references** swapped for canonical phrasing per CLAUDE.md's
   responsible-disclosure rule. README + new docs guide + 4 src docstrings
   + 1 test now say 'your OpenClaw' / 'host agent repo' / 'agentRepo' var
   name. Behavior unchanged — only documentation strings touched.

Verify gate (the script CI runs) passes locally: EXIT=0.
Tests still pass: 60/60 across the affected files.
llms-full.txt regenerated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(test): update check-resolvable-cli expectation for cwd_walk_up tier

Sister fix to the test/repo-root.test.ts update in commit a31418e3. The new
v0.33 cwd_walk_up tier fires before repo_root when running from inside the
gbrain repo — same skills/ dir matched, different source label. Behavior
unchanged; the legacy repo_root tier is now functionally subsumed (kept in
the type union for back-compat).

CI shard 3 failure: test/check-resolvable-cli.test.ts:171.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(test): pin clock in sync_freshness boundary tests (CI flake)

The 24h and 72h exact-boundary tests scheduled last_sync_at relative to
Date.now() at construction time, then let the check call Date.now() again
internally. CI scheduler jitter between the two reads pushed ageMs past
the strict > thresholds by microseconds, dropping the 72h-boundary case
into the fail branch instead of warn.

Fix: add an optional `opts.now` test seam to checkSyncFreshness. The two
boundary tests now capture t0 once and pass it both to the timestamp
constructor and to the check, making ageMs deterministically equal to
the boundary. The non-boundary tests (4d, 30h, 2h, etc.) don't need
pinning — they're comfortably away from the > comparison.

CI shard 1 flake: test/doctor.test.ts:479. Locally 48/48 doctor tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): agent-onboarding readme + next-action hints on every CLI surface (DX review)

DX audit of the v0.36 scaffold model surfaced one structural gap and four
output gaps. When scaffolded files land on a downstream agent's disk, the
agent had no agent-facing manifest telling it what to do — no routing
contract, no upgrade flow, no two-way merge warning at the right surface.

Fixes:

1. **New shared dep: skills/_AGENT_README.md.** Lands on every scaffold +
   migrate-fence alongside the existing _brain-filing-rules.md and
   _output-rules.md. Short, agent-readable contract: walk *.SKILL.md
   frontmatter triggers: for routing, gbrain is reference not law on
   upgrade, no managed-block fence anymore, two-way merge has known
   limitations. Single source of truth for the agent operating contract.

2. **scaffold stdout** prints a next-action hint pointing at the readme
   (with absolute path) and the reference --all upgrade-sweep command.

3. **reference stdout** adds per-category decision policy:
   - missing → scaffold again
   - differs → was edit intentional? keep it. Accidental? patch by hand or
     apply-clean-hunks after reading the two-way warning.

4. **reference --apply-clean-hunks** prints the two-way merge WARNING
   BEFORE the apply (to stderr, survives stdout redirect). Spells out
   that gbrain has no scaffold-time base and local edits in differing
   sections WILL be aligned to gbrain. Skipped in --json mode for
   machine consumers. On conflicts, prints how to inspect and patch.

5. **migrate-fence stdout** tells the agent its routing model just
   changed (fence gone, walk frontmatter now) and points at
   scrub-legacy-fence-rows as the eventual cleanup. References the new
   _AGENT_README for fresh-install agents.

Smoke verified end-to-end: 16 files land (was 15, +1 for _AGENT_README),
hint prints with absolute path, readme lands on disk. Tests + verify gate
pass clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(skillpack): upgrade-time reference sweep + reference --since version filter (DX deferred items)

Closes the last two DX gaps from the v0.36 audit:

1. **Post-upgrade reference sweep.** New `postUpgradeReferenceSweep`
   helper called at the end of `gbrain post-upgrade`. After migrations
   apply, auto-runs `reference --all` against the detected host
   workspace and prints a one-line-per-skill summary of drift. Five
   gates: GBRAIN_SKIP_REFERENCE_SWEEP env-var bypass, no detected
   workspace (silent), workspace IS gbrain repo (dev-mode silent),
   zero drift (silent), and pure-missing skills the host never
   scaffolded are filtered out as noise. All errors swallowed —
   never blocks post-upgrade. Helper accepts test-seam opts
   (gbrainRoot, targetWorkspace) for unit testability.

2. **`reference --all --since <version>`.** Filters the sweep to
   skills whose source actually changed in gbrain between
   <version> and HEAD, using a new `changedSlugsSinceVersion`
   helper in bundle.ts. Pure-JS git wrapper (spawnSync), no deps.
   Accepts bare '0.X.Y.Z' or 'v0.X.Y.Z' or commit SHA. Falls back
   loudly to full sweep when git can't resolve the ref (tarball
   install, missing tag).

Test coverage added — total +32 new test cases:

UNIT (15 cases):
- test/skillpack-changed-since-version.test.ts (9 cases): git-aware
  filter against a fixture git repo. Covers null on non-repo,
  null on bad tag, empty array on no changes, single + multi-slug
  drift (deduped + sorted), bare + v-prefix version forms, non-
  skills/ path filtering, SHA-prefix ref form.
- test/upgrade-reference-sweep.test.ts (6 cases): gate logic.
  Covers env-var bypass, zero drift, empty-host suppression,
  drift-detected output shape, dev-mode workspace==gbrain guard,
  error-swallowing contract.

E2E (8 new cases in test/e2e/skillpack-flow.test.ts):
- 10: scaffold lands skills/_AGENT_README.md
- 11: scaffold stdout prints the Next: hint
- 12: scaffold re-run (skipped-existing) suppresses the hint
- 13: reference stdout prints per-category decision policy
- 14: --apply-clean-hunks WARNING on stderr, not stdout
- 15: --apply-clean-hunks --json suppresses the WARNING (bug fix
  surfaced here: code originally printed unconditionally, now
  gated on !json)
- 16: migrate-fence stdout points at the new routing model
- 17: --since with a bad tag falls back to full sweep with warn

Local sweep: 579/579 pass across 18 affected test files, verify
gate EXIT=0, llms regenerated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(README): zero-base rewrite — 921 → 422 lines, refreshed catalog, MECE structure

The README had drifted into a changelog dumping ground. Four 'New in vX.Y'
paragraphs competed for the lead, 16 version tags scattered through
headings, the production-numbers hook (17,888 pages, 4,383 people) was
six months stale, and skills were described in three places (Skills section,
Commands section, inline marketing prose).

Zero-based rewrite:

**Refreshed catalog** (surveyed live brain + live agent fork, broad strokes
per CLAUDE.md privacy rules):
- ~100K total brain items (was 17,888 in the old README — 6x stale)
- ~16K people (was 4,383)
- ~5K companies (was 723)
- ~8K concepts, ~4K originals, ~3.5K daily notes
- ~31K media (30K tweets, 179 books, papers/films/games/interviews)
- 108 cron jobs running (was 21)
- 273 skills in the live agent fork (35 bundled + 238 user-built)

**Structure** — MECE, single source of truth per concept:
1. Hook + at-a-glance table (refreshed numbers)
2. Install (3 paths, terse)
3. What it does (5 capability areas — replaces 12 scattered sections)
4. Skills (categorized one-liners — 35 lines, was ~200)
5. How it works (one coherent flow — replaces 4 overlapping sections:
   Architecture, Knowledge Model, Knowledge Graph, Search, Why It Works)
6. Commands (terse cheatsheet — every command, one line each)
7. Docs (link map — points to docs/ for the heavy stuff)
8. Origin / Contributing / License

**Cut entirely** (moved or deleted):
- 4 'New in vX.Y' leads (→ CHANGELOG.md is the changelog)
- 16 (vX.Y) version tags in section headings
- Minions stats subsection (subsumed into hook + 'durable background work')
- Voice section (was 12 lines of brand prose)
- Engine Architecture detail (→ docs/architecture/)
- File Storage section (→ docs/guides/storage-tiering.md)
- Per-skill marketing prose (one-liner per skill in the table)

The README is no longer the changelog. Future releases append to
CHANGELOG.md; the README only changes when a structural capability does.

llms-full.txt regenerated. Privacy check + verify gate pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(README): fix line-start '+' rendering bug + lead with eval evidence

Two fixes in one:

1. **Markdown bug fix.** The OAuth 2.1 paragraph had `+ PKCE,` on a line
   start (column 1), which GitHub-flavored markdown interprets as a list
   marker — the line break before it broke the paragraph and rendered as
   an orphan first line followed by a bullet. Rewrote the OAuth 2.1
   capabilities as inline-comma-separated, escaped the `+` semantics.
   Swept the whole file for the same bug class — no other instances.

2. **Maximum-sell mode for evals.** Surveyed every published benchmark
   in both this repo and ~/git/gbrain-evals. Strongest evidence pulled
   to the top:

   - **97.60% R@5 on the public LongMemEval _s (500 questions).** No LLM
     in the retrieval loop. $0.50 per 1000 queries. Beats MemPalace raw
     by a point on the same dataset, beats every academic dense
     retriever (Stella, Contriever, BM25). Mastra/Supermemory measure
     a different metric (QA accuracy with LLM judge) — flagged honestly.

   - **+31.4 points P@5 from the self-wiring knowledge graph** on
     BrainBench v0.20.0 (240-page rich-prose corpus, 145 relational
     gold queries). Separable, measured, load-bearing. Zero retrieval
     regression across seven releases (v0.16 → v0.20).

   New '## Benchmarks' section after Install:
   - Public benchmark table with cross-system comparison
   - In-house BrainBench scorecard with per-adapter Δ vs gbrain
   - Source-swamp resistance result (93.3% top-1 vs 80% grep-only)
   - Skill/prompt compression: 25KB → 13KB AGENTS.md, +13-17pp accuracy
     across Opus 4.7 / Sonnet 4.6 / Haiku 4.5
   - 'Run your own evals' subsection with copy-pasteable commands for
     every eval surface (longmemeval, cross-modal, eval capture/replay,
     BrainBench)

   Tightened the lead's cost-comparison claim to what's defensible per
   the underlying eval doc (MemPal LLM-rerank $0.001/q vs gbrain
   $0.0005/q; dropped the overstated '6x' I'd written initially).

Privacy + verify gate + build-llms test all pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(README): integrate the eval story into the lead, move jargon into 'Receipts on the evals'

Previous lead dumped metric acronyms (R@5, P@5, P@5 deltas, MemPalace,
Stella, Contriever, BM25) before the reader knew what gbrain does. A
'somewhat technical' reader hits the wall of jargon and bounces.

Rewritten:

**Lead (jargon-free, 3 paragraphs)** — describes the value in plain
English, with two anchor numbers:
- 'right answer in top 5 results 97.6% of the time' (not 'R@5 97.60%')
- 'roughly 4x more relevant than plain vector RAG' (not '+31.4 pts P@5')
- 'better than every comparable system that doesn't pay for a language-
  model call on every retrieval' (the load-bearing honest framing,
  without naming the competitors mid-hook)
- ends with '[Receipts on the evals →]' linking down

**'## Benchmarks' renamed '## Receipts on the evals'** with a glossary
at the top defining R@5, P@5, and 'no LLM in the loop' in one line each.
Then the full tables: LongMemEval cross-system (with the metric-mismatch
flag for Mastra/Supermemory), in-house BrainBench scorecard, source-swamp
resistance, and prompt compression. The competitor names + metrics stay
here where readers who want the receipts can find them, with the
glossary so the acronyms don't tax cold readers.

Net: lead reads as 'here's what it does and the proof' instead of 'here
are the benchmark numbers, figure out what they mean.' Comparison facts
unchanged.

Privacy + verify gate + build-llms test all pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(README): name LongMemEval explicitly + first-person voice in lead

Two specific edits from user feedback:

1. 'the standard public benchmark for AI memory systems' → 'LongMemEval'
   (linked to the HuggingFace dataset). The benchmark has a name; use it.

2. 'Built by the President and CEO of Y Combinator to run his own AI
   agents' (passive third-person) → 'I'm the President and CEO of Y
   Combinator, and I use this 16 hours a day' (active first-person).
   Carried the voice change through the rest of the README — the
   downstream 'Garry's personal agent' line and the Origin section's
   'Garry Tan needed... he'd ever drafted... so he built one' all flip
   to first person ('my personal agent', 'I needed', 'I'd ever drafted',
   'so I built one'). The README is now consistently first-person from
   the author's voice instead of a hagiographic third-person framing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(README): add Multi-player and company brains section

Three deployment patterns documented:

1. Single GBrain server + thin MCP clients (recommended). Tailscale
   private networking, OAuth scope, source-scoped clients, exhaustive
   what-clients-can/cannot-do lists.
2. Local PGLite + GStack for per-worktree code search.
3. Federated repos (advanced) — multiple servers indexing the same
   brain repo.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(README): tighten install path + add tech-orientation block + visceral query example

Self-eval as a cold reader surfaced four gaps blocking a 10/10 first read:

1. Lead never says WHAT it is technically — CLI? service? cloud? local?
   Added a "What it is, technically" block right after the hook: open-source
   MIT, Bun CLI + MCP server, local-first, data stays on disk, MCP-native.
2. Install path optimized for committed users not evaluators. The old
   "recommended" path (deploy OpenClaw on Render, 8GB RAM) blocked anyone
   trying gbrain for the first time. Reordered into 3 paths by commitment:
   60-second standalone CLI first, MCP for Claude Code / Cursor second,
   full agentic install third.
3. No example output showing what success looks like. Added a real sample
   `gbrain query` invocation with the hybrid-search result format so a
   reader can feel the experience before they install.
4. Privacy / data-locality unaddressed in lead. Now stated up front:
   embedding calls only hit external APIs if you configure them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 12:53:01 -07:00
Garry TanandClaude Opus 4.7 1dadd9ed71 v0.35.7.0 feat: temporal trajectory + founder scorecard (Phases 2-4) (#1131) 
* feat(facts): typed-claim substrate + cycle correctness fixes (v0.35.6 wave 1/3)

Schema (migration v67):
- Add four optional typed-claim columns to facts: claim_metric TEXT,
  claim_value DOUBLE PRECISION, claim_unit TEXT, claim_period TEXT
- Partial index facts_typed_claim_idx ON (entity_slug, claim_metric, valid_from)
  WHERE claim_metric IS NOT NULL
- All nullable, metadata-only on both engines

Fence layer:
- ParsedFact (facts-fence.ts) gains optional claimMetric/Value/Unit/Period
- Parser tolerates both 10-cell (legacy) and 14-cell (widened) rows
- Renderer emits 14 cells iff any row has typed data; otherwise stays
  10-cell so existing fences don't widen on unrelated edits
- Numeric value cell tolerates comma thousand separators (50,000 -> 50000)

Extract pipeline (D-CDX-2, D-ENG-1):
- src/core/facts/extract.ts (the actual Haiku call site, NOT extract-facts.ts
  cycle phase) extends its system prompt to emit typed fields for metric-shaped
  claims
- extractFactsFromFenceText gains optional pageEffectiveDate. Precedence:
  fence-row validFrom > pageEffectiveDate > undefined (engine defaults to now)
- normalizeMetricLabel: 15-entry seed map for common founder metrics (mrr,
  arr, runway, headcount, team_size, cac, ltv, gross_margin, burn_rate, cash,
  users, mau, dau, churn_rate, revenue); unknown labels lowercase + space->_

Engine extensions:
- NewFact + insertFact + insertFacts in both engines accept the four typed
  columns (all nullable)
- Cycle phase extract-facts.ts threads page.effective_date through AND
  batch-embeds via gateway.embed() before insertFacts (D-CDX-3 fix for
  cycle-inserted facts arriving with embedding=NULL)

Consolidate fix (D-CDX-4 — Codex F4):
- Replace MAX(row_num)+1 INSERT with semantic upsert on (page_id, claim,
  since_date). Re-running the full cycle on stable input produces zero new
  takes — fixes the pre-existing duplicate-takes bug after extract_facts
  wipes consolidated_at
- Chronological valid_until writeback per cluster: sort by (valid_from ASC,
  id ASC), walk pairs, set older.valid_until = newer.valid_from

Tests:
- test/migrate.test.ts +6 cases for v67 shape + materialization + nullable
  backward compat
- test/facts-fence-typed.test.ts (new, 17 cases): parser+renderer round-trip,
  normalization seed map coverage, valid_from precedence three-branch
- test/consolidate-valid-until.test.ts (new, 4 cases): chronological
  writeback (R4a), same-day id tiebreaker, cycle re-run zero duplicates
  (R4b/R7), valid_until idempotency
- test/schema-bootstrap-coverage.test.ts: add four typed-claim columns to
  COLUMN_EXEMPTIONS (migration co-defines the partial index, no forward
  reference to bootstrap)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(trajectory): find_trajectory MCP op + eval/founder CLIs (v0.35.6 wave 2/3)

Engine method (D-CDX-1, D-CDX-6):
- BrainEngine.findTrajectory(opts) on both Postgres and PGLite
- TrajectoryOpts: scalar sourceId fast path + sourceIds federated array
  (mirrors v0.34.1.0 search* dual pattern)
- opts.remote: when true, SQL adds AND visibility='world' so OAuth read
  clients see only world-visibility facts (mirrors recall's posture —
  closes the F7 privacy regression Codex caught in plan review)
- Single SQL query, ORDER BY valid_from ASC, id ASC for deterministic
  output (R3 pin). Returns TrajectoryPoint[] including raw embedding so
  the caller can compute drift without a second round-trip

Pure function library (src/core/trajectory.ts, new):
- detectRegressions(points, threshold): walks consecutive (metric, value)
  pairs per metric; emits when newer drops >= threshold below older.
  10% default, override via GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD
- computeDriftScore(points): 1 - mean(cosine(emb[i], emb[i-1])) over
  embedded points; clamped [0,1]; null when <3 embedded points (D-ENG-3
  graceful degradation)
- computeTrajectoryStats(points): composed shape returning both
- TRAJECTORY_SCHEMA_VERSION = 1 — additive-only across releases (R5)

MCP op (src/core/operations.ts):
- find_trajectory: scope read, NOT localOnly. Routes through
  sourceScopeOpts(ctx) for federated isolation AND threads ctx.remote
  for visibility filtering. Strips raw Float32Array embeddings from the
  wire shape; converts valid_from to YYYY-MM-DD string
- Registered in operations array after find_experts
- FIND_TRAJECTORY_DESCRIPTION in operations-descriptions.ts

CLIs:
- gbrain eval trajectory <entity> [--metric M] [--since D] [--until D]
  [--limit N] [--json] — chronological human view with [REGRESSION] inline
  annotation; thin-client routing via callRemoteTool(find_trajectory).
  Dispatched in src/commands/eval.ts sub-subcommand block
- gbrain founder scorecard <entity> [--since D] [--until D] [--json] —
  pure aggregation over Phase 2's substrate. Four signals:
  claim_accuracy (over resolved takes), consistency, growth_trajectory,
  red_flags. computeFounderScorecard exported for tests.
  Registered as top-level command in cli.ts; added to CLI_ONLY set

Tests (45 cases across 5 files):
- test/engine-find-trajectory.test.ts: 18 cases — chronological order,
  source scoping (scalar + federated), visibility filter on remote=true,
  metric + since/until filters, regression detection at threshold
  boundaries, drift score with various embedding states
- test/operations-find-trajectory.test.ts: 9 cases — op registration,
  param validation, JSON envelope shape, R5 schema_version: 1,
  embedding stripped from wire, R6 visibility filter, source scoping
- test/eval-trajectory.test.ts: 7 cases — arg parsing, --help,
  --json envelope, regression annotation, --metric filter, empty entity
- test/founder-scorecard.test.ts: 9 cases — empty inputs no-NaN (G2),
  claim_accuracy math, consistency math, growth_trajectory math,
  red_flags fire for regression / narrative_drift / missed_prediction
- test/eval-contradictions/no-valid-until-write.test.ts: 4 cases —
  R1 (probe never writes valid_until under eval-contradictions/) +
  R8 (only allow-listed files write valid_until anywhere in src/)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.35.6.0 — CHANGELOG + VERSION + docs + migration note

Bumps to v0.35.6.0 (next-minor after master's v0.35.5.1 — typed-claim
substrate + trajectory + founder scorecard is a new user-facing
feature surface, not a fix).

- VERSION + package.json synced
- CHANGELOG.md release-summary block in the wave-style voice, lead with
  what the user can now DO. Sections: typed metric claims in the fence,
  chronological metric trajectories, founder scorecard, MCP
  find_trajectory op, cycle re-run idempotency fix, embedding-on-insert
  fix, valid_from precedence fix. To-take-advantage-of block with
  verification + opt-in fence syntax example
- CLAUDE.md Key Files entry consolidating the wave across
  eval-trajectory.ts + founder-scorecard.ts + trajectory.ts. Names every
  D-ENG / D-CDX decision and the Codex outside-voice F-numbers
- skills/migrations/v0.35.6.md agent-readable migration note. Includes
  fence-syntax example for typed-claim rows so downstream agents start
  emitting them. Iron-rule contracts called out (R1 + R8 + R7 + visibility)
- llms-full.txt regenerated to reflect the new CLAUDE.md entry

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: post-ship sync for v0.35.7.0 — trajectory + founder scorecard

- README.md: add `gbrain eval trajectory` to EVAL section, add new
  TEMPORAL block covering `gbrain founder scorecard` + the
  GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD env override; add v0.35.7
  "What's new" paragraph below the v0.28.8 LongMemEval blurb
- AGENTS.md: new bullet under Common tasks teaching agents to reach for
  `gbrain eval trajectory` / `gbrain founder scorecard` / the
  `find_trajectory` MCP op when asked to evaluate a founder/company
  over time
- docs/contradictions.md: append "Temporal axis follow-on (v0.35.3.1 +
  v0.35.7)" subsection under See also, cross-linking the trajectory
  substrate and naming the auto-supersession.ts:4 invariant preserved
  by both the verdict enum (probe side) and consolidate's valid_until
  writeback (cycle side)
- CLAUDE.md: fix stale (v0.35.4) tag on the trajectory entry to
  (v0.35.7) — version got rebumped twice during the merge wave
- skills/migrations/v0.35.7.md renamed to v0.35.7.0.md for consistency
  with the v0.35.0.0.md / v0.14.0.md / etc naming convention
- llms-full.txt regenerated to reflect the CLAUDE.md edit

Coverage map (Diataxis):
  /eval trajectory CLI       ✅ ref (README, AGENTS) ✅ how-to (CHANGELOG) ❌ tutorial
  /founder scorecard CLI     ✅ ref (README, AGENTS) ✅ how-to (CHANGELOG) ❌ tutorial
  find_trajectory MCP op     ✅ ref (CLAUDE.md, AGENTS, contradictions.md)
  typed-claim fence cols     ✅ ref (skills/migrations/v0.35.7.0.md, CHANGELOG)
  Migration v67              ✅ ref (CLAUDE.md, CHANGELOG)

No tutorial / explanation gaps worth filling in this PR — the migration
note's fence-syntax example already covers the "first typed claim"
walkthrough. ARCHITECTURE diagrams not drifted (the trajectory work
extends existing facts/takes infrastructure; no new component boxes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 18:52:38 -07:00
Garry TanandClaude Opus 4.7 488e4824e8 v0.34.1.0 fix(mcp): MCP fix wave — source-isolation P0 + PKCE DCR + federated_read + 3 more (#996) 
* fix(mcp): skip stdin EOF handlers when MCP_STDIO=1

OpenClaw's bundle-mcp gateway and similar wrappers pipe the JSON-RPC
handshake on stdin then close their stdin half. Pre-fix, both stdin
'end' and 'close' listeners (server.ts:65-66 and serve.ts:204-206)
treated this as a permanent disconnect and shut the server down before
the first tool call arrived.

Guard both sites with `process.env.MCP_STDIO !== '1'`. Signal handlers
(SIGTERM/SIGINT/SIGHUP), transport.onclose, and the parent-process
watchdog still cover legitimate shutdown paths. The serve.ts site
threads the env read through an injectable `mcpStdio?: boolean` on
ServeOptions so tests stay isolated (no process.env mutation per
scripts/check-test-isolation.sh R1).

Tests: 3 new cases in test/serve-stdio-lifecycle.test.ts pin the
guard's invariants — mcpStdio=true must NOT trigger shutdown on stdin
EOF, signals must still drive shutdown with mcpStdio=true, and
mcpStdio=false (default) preserves existing CLI behavior. 25/25 pass.

Origin: PR #870.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(oauth): honor token_endpoint_auth_method=none for PKCE public clients

RFC 7591 §3.2.1: when a DCR client declares
token_endpoint_auth_method="none" (PKCE-only public clients like Claude
Code, Cursor), the authorization server MUST NOT issue a client_secret.
Pre-fix, registerClient unconditionally minted a secret, and the MCP
SDK's clientAuth middleware then rejected valid public-client flows on
/token because it expected client.client_secret to match.

Three changes to src/core/oauth-provider.ts:registerClient:

  - Gate clientSecret generation on isPublicClient = (auth_method === 'none').
    Public clients store client_secret_hash = NULL.
  - Omit client_secret from the response payload for public clients.
    Confidential clients (default client_secret_post and explicit
    client_secret_basic) keep their existing one-time-reveal shape.
  - Normalize NULL secret_hash to JS undefined in getClient so SDK
    middleware (which checks client.client_secret === undefined, not
    === null) correctly identifies public clients and skips the
    secret-comparison branch on /token.

Schema is already permissive (client_secret_hash TEXT, no NOT NULL on
both src/schema.sql and src/core/pglite-schema.ts) — no migration
needed.

Tests: 5 new cases in test/oauth.test.ts pin:
  - public client → no client_secret in response (#11 from plan)
  - default auth_method → secret unchanged (regression guard)
  - explicit client_secret_post → secret unchanged
  - getClient NULL→undefined normalization
  - PKCE full /authorize → /token end-to-end with no secret (#15 from plan)

69/69 oauth.test.ts cases pass. typecheck clean.

Origin: PR #909.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(serve-http): --bind HOST, default to loopback (127.0.0.1)

Adds `gbrain serve --http --bind <interface>` to control which network
interface the HTTP MCP server listens on. Default flipped from
`0.0.0.0` (pre-v0.34) to `127.0.0.1` (v0.34.0+).

Why the flip: gbrain's primary use case is a personal-knowledge brain on
a laptop. The previous default exposed brains on every interface — one
accidental `--http` invocation away from publishing the brain to a LAN.
Server operators who need remote access pass `--bind 0.0.0.0` (or a
specific interface). Codex's outside-voice on the original PR #864
correctly flagged that the additive flag wasn't actually the fix; the
default needed to change for the safety claim to hold.

If `--public-url` is set but `--bind` is unset, runServeHttp prints a
loud stderr WARN at startup recommending `--bind 0.0.0.0`. Declaring a
public URL while quietly binding loopback is almost always a
misconfiguration; we want the operator to see it on first start, not
silently fail remote requests.

Startup banner now includes a `Bind:` row so the listening interface is
visible alongside Port / Engine / Issuer.

Origin: PR #864, extended with D11 (default flip) per /plan-eng-review
codex outside-voice review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(mcp): seal source-isolation leak on read path (P0)

Pre-fix, an authenticated OAuth MCP client scoped to source-A could
enumerate source-B pages via six read-side ops: search, query (text
AND image paths), list_pages, traverse_graph, and find_experts. The
v0.31.8 source-scoping pattern shipped through dispatch.ts but the op
handlers never threaded ctx.sourceId into their engine calls, and
hybridSearch.ts:223's explicit SearchOpts rebuild dropped sourceId
even when callers passed it.

Sealing the leak:

  - src/core/operations.ts adds sourceScopeOpts(ctx), the canonical
    precedence ladder: ctx.auth.allowedSources (federated) wins over
    ctx.sourceId (scalar) wins over nothing. Threaded into all 5
    read-side op handlers + the query-image-path searchVector call
    (the 6th leak surface codex caught in plan review).

  - src/core/search/hybrid.ts:223 now threads sourceId + sourceIds
    fields through the inner SearchOpts rebuild. The explicit pick
    shape is preserved (HNSW inner-CTE ordering depends on it) but
    extended.

  - src/core/types.ts adds sourceIds?: string[] to SearchOpts +
    PageFilters (D9: federated read needs array-shaped engine filter
    or fan-out; array wins for hot retrieval).

  - src/core/operations.ts AuthInfo gains sourceId + allowedSources
    (D2: identity surface symmetric with the federated_read column
    #876 will add).

  - Both engines now apply WHERE source_id = $N (scalar) or = ANY($N::text[])
    (array) at the SQL layer for searchKeyword, searchKeywordChunks,
    searchVector, listPages, traverseGraph, traversePaths. Array form
    wins when both are set. The searchVector filter pushes into the
    inner HNSW CTE (codex flagged this placement during plan review).

  - traverseGraph + traversePaths signatures gain opts.sourceId +
    opts.sourceIds; engine.ts interface updated.

  - findExperts (the whoknows op, D3 5th leak surface) accepts
    sourceId + sourceIds and threads them into its internal
    hybridSearch call. PR #861 was authored before v0.33 shipped so
    this op wasn't covered in the original PR.

Auth wiring:

  - GBrainOAuthProvider.verifyAccessToken populates AuthInfo.sourceId
    from oauth_clients.source_id. JOIN guarded by isUndefinedColumnError
    so pre-v55 brains degrade to legacy projection rather than refusing
    every token verification.

  - GBrainOAuthProvider.registerClientManual gains a sourceId
    parameter (defaults to 'default'). DCR registerClient also sets
    source_id='default' on the inserted row.

  - serve-http.ts:929 cleanup: AuthInfo.sourceId is now a real typed
    field. The cast + GBRAIN_SOURCE env fallback chain is gone (D13).
    Legacy bearer tokens default to 'default' source in
    verifyAccessToken.

  - http-transport.ts (legacy access_tokens path) threads
    sourceId='default' through DispatchOpts so v0.22.7 callers stay
    source-scoped.

  - auth.ts CLI adds --source flag to gbrain auth register-client.

Migration v55 (D10 + D13):

  - ALTER TABLE oauth_clients ADD COLUMN source_id TEXT (nullable).
  - Backfill UPDATE source_id = 'default' WHERE source_id IS NULL —
    preserves v0.33 effective behavior verbatim for legacy clients.
  - ADD CONSTRAINT FK ... REFERENCES sources(id) ON DELETE SET NULL,
    wrapped in DO block so re-runs against fresh-install brains (where
    the FK already lives inline in SCHEMA_SQL) no-op cleanly.
  - CREATE INDEX idx_oauth_clients_source_id WHERE source_id IS NOT NULL
    for the verifyAccessToken JOIN.
  - GBRAIN_ACCEPT_SILENT_WIDEN env-flag wired through the runner via
    SET LOCAL gbrain.accept_silent_widen — reserved for future migrations
    that hit the silent-widen footgun codex flagged. This migration
    doesn't need it (column is brand new; no pre-existing stale values
    possible by definition).
  - src/core/pglite-schema.ts + src/schema.sql include the column +
    FK + index inline for fresh installs.

Tests: new test/e2e/source-isolation-pglite.test.ts with 13 regression
cases — one per leak surface (search/list_pages/traverse/etc.) plus
explicit AuthInfo.sourceId and AuthInfo.allowedSources op-handler
threading checks. Full unit suite: 6034 pass / 0 fail. PGLite
initSchema time dropped from 2.4s to 850ms after consolidating v55's
DO blocks (multiple DO blocks were slow on PGLite; one DO block for
the FK install only is fine).

Origin: PR #861 + plan-eng-review decisions D2/D3/D4/D9/D10/D13 + F2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(gateway): multimodal embedding for openai-compatible providers

Pre-fix, embedMultimodal hardcoded a recipe.id === 'voyage' branch and
threw AIConfigError for every other recipe. Multimodal-capable providers
fronted by LiteLLM (or any openai-compatible proxy) were unreachable
even when the operator had wired up the model.

The fix:

  - src/core/ai/gateway.ts adds embedMultimodalOpenAICompat() that
    POSTs to the standard /embeddings endpoint with content arrays
    carrying image_url entries. Routing comes from the existing
    recipe.implementation switch — Voyage stays on its own
    /multimodalembeddings path; every other openai-compatible recipe
    flows through the new helper.

  - src/core/ai/recipes/litellm-proxy.ts declares
    supports_multimodal: true so embedMultimodal accepts the recipe.
    No multimodal_models allow-list: LiteLLM is a passthrough proxy
    and the user owns model-id selection; provider rejection (400 from
    upstream) is the right enforcement layer there. Voyage's static
    allow-list shape stays unchanged (its 12 models share
    supports_multimodal but only one is multimodal-capable).

  - D12 runtime dimension validation: the new helper checks the
    returned vector length against the recipe's declared default_dims
    (preferred) or the brain's embedding_dimensions config. Mismatch
    throws AIConfigError with model id + observed + expected so the
    operator can swap models or rebuild the column. Pre-fix, a
    wrong-dim response would surface as a cryptic pgvector
    "vector dimension mismatch" at INSERT time.

  - Auth resolution routes through the existing defaultResolveAuth
    helper so optional-auth recipes (LiteLLM proxy with no
    LITELLM_API_KEY) and required-auth recipes both share one code
    path. Optional-auth sends "Authorization: Bearer unauthenticated"
    which servers like Ollama / llama-server ignore but the SDK
    contract requires.

Tests: 11 new cases in test/openai-compat-multimodal.test.ts cover
happy-path, multi-input batching, unauthenticated proxy, D12 dim
mismatch + default-dim fallback, 401 / 400 / malformed-JSON / non-array
error paths, and an explicit Voyage-regression test pinning that the
new openai-compat route doesn't accidentally hijack the Voyage path.
All 41 multimodal-related tests pass (existing voyage suite + new).
typecheck clean.

Origin: PR #875 + plan-eng-review D12 (runtime dim validation).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(oauth): federated_read read scope (#876)

Pre-fix, OAuth clients had a single source-scope axis (source_id, added
in v55). A client could either write+read one source OR be a super-reader
across all sources (via NULL source_id). There was no middle ground —
WeCare-style L3 dept clients that need to write to dept-x but read
dept-x + parent canon + shared canon had no expression.

#876 adds federated_read TEXT[] as an orthogonal read-scope axis. source_id
is the WRITE authority; federated_read is the READ authority. They default
to matching values (read scope == write scope, the pre-v0.34 default)
when a client is registered without an explicit federated read list.

Migrations v56-v60 (six new migrations on top of v55):

  - v56: ALTER TABLE ... ADD COLUMN federated_read TEXT[] NOT NULL DEFAULT '{}'.
  - v57 (F5): explicit CASE backfill so source_id IS NULL → '{}' (not an
    array containing NULL — codex caught this ambiguity during plan review).
  - v58: post-backfill validation. Fails loud if any row's source_id isn't
    in its federated_read array, pointing at a logic bug in v57 if fired.
  - v59: flip the source_id FK from ON DELETE SET NULL to ON DELETE
    RESTRICT now that federated_read provides the alternative scope-loss
    path. Pre-flip, deleting a source could silently widen any oauth_client
    to super-reader; post-flip, source delete is refused if any client
    references it (operator must revoke/re-scope first).
  - v60: GIN index on federated_read for array-containment queries.

Auth wiring:

  - GBrainOAuthProvider.verifyAccessToken JOINs c.federated_read and
    populates AuthInfo.allowedSources. Pre-v56 / pre-v55 brains degrade
    via the existing isUndefinedColumnError fallback chain.
  - registerClientManual gains a federatedRead?: string[] parameter
    (defaults to [sourceId]).
  - DCR registerClient sets source_id='default' + federated_read=['default']
    on the inserted row.
  - auth.ts CLI adds --federated-read SRC1,SRC2,... flag. The
    register-client output now prints "Federated reads:" so operators
    confirm the scope they set.

Engines consume the federated array through the SearchOpts.sourceIds /
PageFilters.sourceIds field that #861 added (no engine changes here — the
plumbing was D9). sourceScopeOpts in operations.ts already prefers the
auth.allowedSources array over scalar ctx.sourceId when set.

Test seam:
  - test/book-mirror.test.ts now spawns the CLI with GBRAIN_HOME pointed
    at a tempdir so the test isn't sensitive to the developer's local
    ~/.gbrain/config.json. Pre-fix the test could silently inherit a real
    Postgres connection and hang past the default 5s test timeout. Fresh
    GBRAIN_HOME → "No brain configured" → exit 1 in <1s.
  - test/e2e/source-isolation-pglite.test.ts gains one more regression
    case: AuthInfo.allowedSources = [] (explicit empty) MUST NOT widen
    scope to "all sources" — the silent-widen footgun precedence ladder.
  - test/openai-compat-multimodal.test.ts is part of the wave's commits
    via the migrate.ts changes that bump the schema chain. typecheck-only
    fix on a captured-auth type was already in #875's tree.

6045 unit tests pass / 0 fail. typecheck clean. PGLite initSchema runs
v55-v60 in ~786ms total (within the test-harness budget for tests using
the canonical beforeAll engine pattern).

Origin: PR #876 + plan-eng-review F5 (CASE backfill).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.34.0.0: MCP fix wave (#870 #909 #864 #861 #875 #876)

VERSION + package.json + CHANGELOG bump for the six-PR MCP fix wave.
Schema chain extends from v54 → v60; oauth_clients gains source_id +
federated_read columns; auth'd MCP clients now stay inside their scope
across all read-side ops; PKCE-only DCR works; --bind defaults to
loopback; LiteLLM multimodal embedding ships.

Contributed by @Hansen1018 (#870), @ding-modding (#909), @DukeDawg
(#864), @toilalesondev (#861 + #876), @yoelgal (#875).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.34.0.0

Sync README, CLAUDE.md, SECURITY.md, docs/architecture/topologies.md,
and docs/mcp/DEPLOY.md to reflect the v0.34.0.0 MCP fix wave:

- README: document --bind HOST default (loopback), --source +
  --federated-read register-client flags, PKCE public-client gate
- SECURITY.md: note loopback-by-default for serve --http, update the
  trust-proxy contract to point at the new default
- CLAUDE.md: annotate operations.ts (sourceScopeOpts helper),
  oauth-provider.ts (verifyAccessToken JOIN + PKCE public clients),
  serve-http.ts (--bind flag), gateway.ts (openai-compat multimodal +
  dim validation), mcp/server.ts (MCP_STDIO guard), auth.ts (--source
  + --federated-read), migrate.ts (v58-v63 chain), engine.ts
  (sourceIds field). Add 4 new test-file entries for
  source-isolation-pglite, openai-compat-multimodal,
  serve-stdio-lifecycle, oauth.test.ts PKCE cases
- docs/architecture/topologies.md: source-scoped register-client
  example, --bind 0.0.0.0 for thin-client host setup
- docs/mcp/DEPLOY.md: --bind explanation in the ngrok section,
  source-scoped client recipe
- llms-full.txt: regenerated per the CLAUDE.md-edit chaser rule

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump v0.34.0.0 → v0.34.1.0

Renumbering the MCP fix wave from v0.34.0.0 to v0.34.1.0 so the
release slot lands between master's v0.33.2.1 and the next minor.

Touches every release-artifact mention:
- VERSION: 0.34.0.0 → 0.34.1.0
- package.json: same
- CHANGELOG.md header + "To take advantage" block
- CLAUDE.md key-files annotations (8 entries that document this wave)
- llms-full.txt (regen from CLAUDE.md)
- README.md / SECURITY.md / docs/architecture/topologies.md / docs/mcp/DEPLOY.md
- Wave code-comment markers ("// v0.34.0 (#NNN):" → "// v0.34.1 (#NNN):")

Test files renamed alongside since they were committed with the wave.

Commit subjects on the original 6 PR commits + the v0.34.0.0 bump
commit (4f533c72 → 6b47db7e) intentionally NOT rewritten — those are
history. `git log` finds the implementation by message subject, not by
version tag.

6275 unit tests pass, typecheck clean, migration chain v58-v63 unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 20:15:29 -07:00
garrytan-agentsgarrytan-agentsGarry Tan
 		1a6b543cc5 v0.33.2.0 feat(search-lite): token budget + semantic query cache + intent weighting (#897) 
* feat(search-lite): token budget + semantic query cache + intent weighting

Adds three additive features to the hybrid search pipeline. All
backward-compatible: existing callers see identical behavior unless they
opt in to the new options.

## 1. Token Budget Enforcement (src/core/search/token-budget.ts)

Cap the cumulative token cost of returned results so search payloads
fit downstream context windows. Greedy top-down walk; preserves caller
ordering; no re-rank. char/4 heuristic for token counting (no
tokenizer dependency \u2014 keeps the bun --compile bundle small).

  SearchOpts.tokenBudget   \u2014 numeric cap. Default undefined = no-op.
  HybridSearchMeta.token_budget = { budget, used, kept, dropped }

  HTTP query op: pass `token_budget` param.

## 2. Semantic Query Cache (src/core/search/query-cache.ts + migration v52)

Cache search results keyed by query embedding similarity. HNSW lookup:
`embedding <=> $1 < 0.08` (cosine similarity >= 0.92). Per-source
isolation so multi-source brains don\u2019t bleed. Per-row TTL (default 3600s).
Best-effort writes; all errors swallowed so the cache never breaks the
search hot path.

  Migration v52 creates query_cache table with HALFVEC where pgvector >= 0.7;
  falls back to VECTOR with the resolved config.embedding_dimensions dim.

  New `gbrain cache` CLI: stats / clear --yes / prune.
  Config keys: search.cache.enabled / similarity_threshold / ttl_seconds.

  HybridSearchMeta.cache = { status, similarity?, age_seconds? }

  Routed through new `hybridSearchCached(engine, query, opts)` wrapper;
  the operations.ts query op now uses this wrapper so MCP/CLI calls
  benefit automatically. Skipped for two-pass walks + non-default
  embedding columns where cache semantics don\u2019t hold.

## 3. Zero-LLM Intent Weighting (src/core/search/intent-weights.ts)

Builds on the existing query-intent classifier (4 intents: entity /
temporal / event / general). New weight-adjustment layer applies subtle
per-intent nudges:

  entity   \u2192 boost keyword RRF + exact slug/title match
  temporal \u2192 default recency=on when caller left it unset
  event    \u2192 boost keyword RRF (rare named entities) + soft recency
  general  \u2192 no-op (1.0 multipliers everywhere)

All adjustments are SUBTLE (max 1.25x). Caller-explicit options ALWAYS
win \u2014 intent weighting never silently overrides recency / salience.

Default ON; opt out via `opts.intentWeighting = false`. LLM query
expansion (expansion.ts) is still available and opt-in via
`opts.expansion = true` \u2014 it just isn\u2019t the default anymore.

  HybridSearchMeta.intent now surfaces classifier output for debugging.

## Tests

  test/token-budget.test.ts            (10 tests, pure module)
  test/intent-weights.test.ts          (13 tests, pure module)
  test/query-cache.test.ts             (12 tests, PGLite)
  test/hybrid-search-lite.serial.test.ts (9 tests, PGLite e2e)

Plus 105 pre-existing search tests still pass. `bun run verify` clean.

Co-authored-by: Wintermute <agents@garrytan.com>

* feat(search-mode): MODE_BUNDLES + resolveSearchMode wired into bare hybridSearch

Three named modes (conservative / balanced / tokenmax) that bundle the
search-lite knobs from PR #897 into a single config key. Mode resolution
lives in bare hybridSearch (NOT just the cached wrapper) so eval-replay
and eval-longmemeval — which call bare hybridSearch — test the same
mode-affected behavior as production. See [CDX-5+6] in the plan.

The mode bundle supplies DEFAULTS for intentWeighting, tokenBudget,
expansion, and searchLimit when the caller leaves those undefined.
Per-call SearchOpts and per-key config overrides still win (matches the
v0.31.12 model-tier resolution chain at model-config.ts:resolveModel).

knobsHash() exposes a stable SHA-256 of the resolved knob set; the cache
contamination hotfix (next commit) consumes it to prevent a tokenmax
write from being served to a conservative read.

Three new fields on HybridSearchMeta:
  - mode (resolved mode name)
  - existing token_budget meta now fires from bare hybridSearch too

Bare hybridSearch now applies tokenBudget at all three return paths
(no-embedding-provider, keyword-only-fallback, main). Previously only
hybridSearchCached enforced budget; eval commands missed it.

Tests: 37 unit cases pin the 3x7 bundle table cell-by-cell, the
resolution chain semantics, knobs hash determinism + cross-mode
separation, and the config-table parser. All 72 search-lite tests pass.

Bisect-friendly: this commit ONLY adds mode resolution. The cache-key
contamination hotfix [CDX-4] is a separate atomic commit (next).

* fix(query-cache): cross-mode contamination hotfix [CDX-4]

PR #897's query_cache keyed rows on sha256(source_id::query_text) only.
A tokenmax search (expansion=on, limit=50) populated a row that a
subsequent conservative call (no expansion, limit=10) read back, serving
the wrong-shape results. This is a real bug in PR #897 today, regardless
of the v0.32.3 mode picker work — Codex caught it in plan review.

Fix:
- Migration v56 adds query_cache.knobs_hash TEXT column + composite
  (source_id, knobs_hash, created_at) index. Existing rows have NULL
  knobs_hash and are excluded from lookups (silently re-populated with
  the right hash on first hit — no orphan data, no destructive migration).
- cacheRowId(query, source, knobsHash) — knobsHash now part of the PK so
  a tokenmax write and a conservative write for the same (query, source)
  land in distinct rows.
- SemanticQueryCache.lookup({knobsHash}) filters WHERE knobs_hash = $.
- SemanticQueryCache.store({knobsHash}) writes the resolved hash.
- hybridSearchCached threads knobsHash from resolveSearchMode through
  every cache call. Cache config (enabled/threshold/TTL) now reads from
  the resolved mode bundle, not directly from the config table.

Tests (test/query-cache-knobs-hash.test.ts, 11 cases):
- cacheRowId bifurcates by knobsHash
- Tokenmax write does NOT contaminate conservative lookup
- Three modes coexist as distinct rows for same query
- Legacy NULL-knobs_hash rows are excluded from lookup
- Same-mode write updates in place (no duplicate rows)

All 58 cache + mode tests pass. Migration v56 applies cleanly on a fresh
PGLite brain.

Bisect-friendly: this commit is the cache-key hotfix alone. Mode
resolution wiring lives in the previous commit.

* feat(search-telemetry): in-process rollup writer + search_telemetry table

Migration v57 creates search_telemetry (date, mode, intent, count,
sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss,
first_seen, last_seen). PK (date, mode, intent) caps growth at ~4380
rows/year. Sums + counts only — averages derive at read time so
concurrent ON CONFLICT writes from multiple gbrain processes accumulate
correctly [CDX-17].

In-memory bucket flushed periodically (60s OR 100 calls) + on process
beforeExit/SIGINT/SIGTERM with a 2-second cap. The search hot path NEVER
waits on this write [D2, CDX-19].

Date-bucketed cache_hit / cache_miss columns make hit rate over --days N
derivable [CDX-18]. query_cache.hit_count is a lifetime counter and
can't be sliced by window.

Wired into bare hybridSearch via emitMeta: every search call sync-bumps
a bucket. flush() drains atomically by swapping the map before SQL writes
so a record() during flush lands in the new map.

readSearchStats(engine, {days}) returns the StatsWindow shape that
gbrain search stats consumes (next commit).

Tests: 16 unit cases pin record/flush/read semantics including
ON-CONFLICT-adds-raw-values, concurrent-flush coalescing, cache hit-rate
math, missing-table graceful degradation, and window clamping.

53 migrations apply on a fresh PGLite brain.

* feat(config): add unset + listConfigKeys + readLineSafe helper [CDX-7+8+9]

CDX-8: gbrain config has no unset path today. Required before
`gbrain search modes --reset` can clear search.* overrides.

  - BrainEngine.unsetConfig(key) → returns rows deleted (0|1)
  - BrainEngine.listConfigKeys(prefix) → exact-literal prefix match
    with LIKE-escape on user-supplied % / _ / \ characters
  - PGLiteEngine + PostgresEngine implementations
  - `gbrain config unset <key>` and `gbrain config unset --pattern <prefix>`
    sub-subcommands

CDX-9: readLine has no EOF detection or timeout. Mode-picker plan calls
out "TTY closes mid-prompt → defaults to balanced" but the raw helper
hangs forever. New readLineSafe(prompt, defaultValue, timeoutMs=60s):

  - Returns defaultValue on stdin 'end' event
  - Returns defaultValue on timeout
  - Returns defaultValue on empty Enter
  - Non-TTY stdin returns defaultValue immediately (e2e safe)
  - Returns trimmed user input otherwise

Exported so install picker (next task) can use it.

Tests: 9 cases pin unset semantics + prefix matcher edge cases
(glob-wildcard escape, sort order, idempotent loop, search.* sweep).
All 53 migrations apply on a fresh PGLite brain.

* feat(init): install-time mode picker + upgrade banner

Install picker (src/commands/init-mode-picker.ts):
  - Runs as a phase inside `gbrain init` AFTER engine.initSchema() so DB
    config writes work [CDX-7].
  - Idempotent: skipped on re-init if search.mode is already set.
  - Smart auto-suggestion via recommendModeFor() reads
    models.tier.subagent / models.default / OPENAI_API_KEY:
      * Opus default/subagent → tokenmax (quality ceiling)
      * Haiku subagent → conservative (4K budget keeps cost down)
      * No OpenAI key → conservative (no LLM expansion possible)
      * Sonnet / unknown → balanced (safe default)
  - TTY shows menu via readLineSafe (60s timeout, defaults on EOF/empty).
  - Non-TTY auto-selects + emits operator hint:
      [gbrain] search mode: X (auto-selected — reason)
      [gbrain] To change: gbrain config set search.mode <...>
  - --json mode emits structured `{phase: 'search_mode_picker', ...}` event.
  - Wired into both initPGLite and initPostgres flows.

Upgrade banner (src/commands/upgrade.ts):
  - One-shot stderr banner in runPostUpgrade.
  - State persisted via config key `search.mode_upgrade_notice_shown=true`
    — fires at most once per install.
  - Copy corrected per [CDX-1+2+3]: production query op STILL defaults
    expand=true and limit=20. The banner reframes from "behavior is
    regressing" to "named modes available + here's how to preserve
    exact current shape."

Tests (test/init-mode-picker.test.ts, 16 cases):
  - recommendModeFor heuristic for all 4 input shapes
  - parseModeInput accepts numeric/named/case-insensitive, rejects garbage
  - runModePicker non-TTY auto-selects + writes config
  - Idempotent + --force re-prompt + JSON output
  - Opus → tokenmax, Haiku → conservative real wiring through engine

* feat(cli): gbrain search modes/stats/tune command

Three sub-subcommands mirroring the gbrain models (v0.31.12) shape:

  gbrain search modes [--json]
    Read-only routing dashboard. Shows the three mode bundles, the active
    mode, and the source of every resolved knob:
      cache_enabled = true   [override: search.cache.enabled]
      tokenBudget   = 4000   [mode: conservative]
    Plus knob descriptions for legibility.

  gbrain search modes --reset [--source <mode>]
    Clears every search.* override (NOT search.mode itself). Preserves
    the upgrade-notice state key. --source <mode> is a dry-run that
    lists what --reset would change without writing — the paved path
    [CDX-8] flagged as missing.

  gbrain search stats [--days N] [--json]
    Observability. Reads the search_telemetry rollup over the window
    (clamps to [1, 365]). Prints cache hit rate, mode mix, intent mix,
    budget drops, avg results/tokens. JSON output includes
    _meta.metric_glossary block per [CDX-25].

  gbrain search tune [--apply] [--json]
    Recommendation engine. 5 rules cover the bug class:
      - Insufficient data → "no_recommendations" status
      - Conservative + high budget-drop rate → suggest balanced
      - High cache hit rate (>85%) → suggest similarity threshold bump
      - Tokenmax + Haiku subagent → suggest balanced (cost mismatch)
      - Cache disabled but stats show usage → suggest re-enabling
    --apply mutates config via setConfig / unsetConfig with a paste-ready
    revert command printed at the end.

Registered in src/cli.ts dispatch table. 17 unit cases pin:
  - Dashboard report shape + per-knob source attribution
  - --reset preserves search.mode + notice key
  - --source dry-run never writes
  - stats reads telemetry rollup; --days clamps
  - tune recommendation rules fire on real telemetry data
  - --apply mutates config
  - --help + unknown subcommand exit codes

* feat(eval): metric glossary module + auto-gen METRIC_GLOSSARY.md + CI guard

Single source of truth at src/core/eval/metric-glossary.ts. Every entry
carries 3 fields:
  - industry_term (canonical IR/NLP literature name, preserved verbatim)
  - eli10 (plain-English a 16-year-old can follow)
  - range (numeric range + interpretation)

Covers 4 metric families:
  - Retrieval: P@k, R@k, MRR, nDCG@k
  - Stability: Jaccard@k, top-1 stability
  - Statistical: p-value (paired bootstrap + Bonferroni), 95% CI
  - Operational: cache hit rate, avg results/tokens, cost per query, p99 latency

Public surface:
  - getMetricGloss(metric) → full entry or null
  - eli10For(metric) → plain-English string or null
  - buildMetricGlossaryMeta(metrics[]) → {metric → eli10} record for
    JSON `_meta.metric_glossary` blocks per [CDX-25]. ONE block per
    response, NOT sibling `_gloss` fields on every metric.
  - renderMetricGlossaryMarkdown() → deterministic Markdown for the doc

Auto-generation:
  scripts/generate-metric-glossary.ts emits docs/eval/METRIC_GLOSSARY.md.
  Deterministic (same input → same bytes) so the CI guard can diff.

CI guard:
  scripts/check-eval-glossary-fresh.sh regenerates into a temp file and
  diffs against the committed doc. Out-of-date doc fails the build.
  Wired into `bun run verify` (and therefore `bun run test:full`).

Tests (test/metric-glossary.test.ts, 18 cases):
  - Every documented metric is present
  - Every entry has all 3 required fields
  - Accessors return null on unknown metrics (no throw)
  - buildMetricGlossaryMeta silently drops unknown metrics
  - renderer output is deterministic across calls
  - Renderer groups metrics into 4 sections

docs/eval/METRIC_GLOSSARY.md: 5491 bytes, 124 lines, fresh.

* feat(doctor): search_mode + eval_drift checks + drift-watch module

src/core/eval/drift-watch.ts — curated retrieval watch-list [CDX-6].
Five patterns covering the surface that actually affects retrieval quality:
  - src/core/search/      (search pipeline)
  - src/core/embedding.ts (embedding shape)
  - src/core/chunkers/    (chunk granularity)
  - src/core/ai/recipes/anthropic.ts + openai.ts (expansion + embed routing)
  - src/core/operations.ts (the query op definition)

Adding to the list is a deliberate act — requires a CHANGELOG line so
coverage grows on purpose, not by accident. Pure functions:
  - matchesWatchPattern(path) — trailing-slash = prefix, bare = equality
  - filesDriftedSince(repoRoot, sha?) — git diff --name-only wrapper
  - watchedFilesDrifted(repoRoot, sha?) — composite

src/commands/doctor.ts — two new checks.

checkSearchMode [CDX-20]: status stays 'ok' (never warns, never docks
health score). Hint in message field. Three branches:
  - unset → "search.mode is unset (using balanced fallback). Run
    `gbrain search modes` to see what is running and pick a mode."
  - mode + no overrides → "Mode: X (no per-key overrides — mode bundle
    is canonical)."
  - mode + overrides → "Mode: X with N per-key override(s) (k1, k2, …).
    To consolidate to the pure mode bundle: gbrain search modes --reset"
Upgrade-notice state key (search.mode_upgrade_notice_shown) is excluded
from the override roster — it's not a knob.

checkEvalDrift [CDX-6]: surfaces uncommitted changes to retrieval-watched
files. Always 'ok'; operator-facing reminder. Names up to 3 drifted files
in the message + paste-ready re-eval command.

Both helpers exported (was: file-private) so tests can pin behavior
without walking the full runDoctor pipeline.

Tests: 12 drift-watch cases + 7 doctor-check cases. Pin watch-list shape,
prefix-vs-equality matcher semantics, missing-repo graceful failure, and
all three search_mode branches.

* feat(eval): --mode flag on longmemeval/replay + run-all + compare

Per-mode --mode flag plumbed into:
  - gbrain eval longmemeval --mode <conservative|balanced|tokenmax>
    Sets search.mode in the benchmark brain's config table; config is
    in PRESERVE_TABLES so resetTables doesn't wipe it between questions.
    Mode surfaces in the per-question NDJSON row.
  - gbrain eval replay --mode <m> + --compare-limit N
    --compare-limit forces a constant K across modes [CDX-13]; without
    it, Jaccard@k against the captured baseline measures K-drift, not
    quality. Mode is set once before the replay loop.
  - NOT cross-modal per [CDX-11]: cross-modal scores OUTPUT against
    TASK; it doesn't retrieve. Adding --mode there is theater.

New: gbrain eval run-all orchestrator (src/commands/eval-run-all.ts):
  - Sweeps every requested mode × suite combination
  - Sequential default per D9; --parallel N opt-in (clamped to mode count)
  - Cost guard with split caps [CDX-15+16]:
      --budget-usd-retrieval N (default $5)
      --budget-usd-answer N (default $20)
    Non-TTY refuses with exit 2 unless --yes AND explicit --budget-usd-*
    flags pass. TTY refuses without --yes (defense against agent loops).
  - estimateRunCost computes per-(suite,mode) breakdown including the
    expansion-Haiku surcharge for tokenmax.
  - Audit trail: appends to <repo>/.gbrain-evals/eval-results.jsonl
    [CDX-23]. Personal brain (~/.gbrain) NEVER touched.
  - v0.32.3 ships orchestrator + argv + guard + persist hook.
    In-process per-suite invocation is a v0.32.4 follow-up (operator
    runs the per-suite CLIs with the documented --mode flag for now;
    each completion calls persistRunRecord to log).

New: gbrain eval compare report (src/commands/eval-compare.ts):
  - Reads eval-results.jsonl, groups by (suite, mode), renders MD or JSON
  - Most-recent (suite, mode, commit) wins when duplicates exist
  - JSON output has schema_version=2 + _meta.metric_glossary block per
    [CDX-25] (ONE block per response, not sibling _gloss fields)
  - _meta.methodology field names the paired-bootstrap + Bonferroni
    discipline per [CDX-14] so haters can reproduce
  - Missing file → friendly hint pointing at `gbrain eval run-all`

Wired into eval dispatch table in src/commands/eval.ts.

Metric glossary fuzzy fallback: `recall@10` → `recall@k` lookup
(the glossary documents the family; report rows carry specific K
values). Routes through getMetricGloss for every call site.

Tests (42 cases total — all green):
  - eval-run-all.test.ts (19): argv parser, cost estimate, guard
    semantics for all 4 (over/under × tty/non-tty) shapes, persist hook
    NDJSON shape.
  - eval-compare.test.ts (5): JSON + MD output shapes, glossary
    integration, missing-file graceful, mode filter, most-recent-wins.
  - metric-glossary.test.ts (18): unchanged but updated assertions to
    cover the fuzzy `@N` → `@k` fallback.

Pre-existing eval-replay / eval-longmemeval / eval-export / eval-prune
tests (42 cases) still pass — --mode + --compare-limit are additive.

* docs: methodology + CLAUDE.md/README/RESOLVER + skills/conventions

docs/eval/SEARCH_MODE_METHODOLOGY.md — haters-immune 8-section template.
Documents what the eval measures + does NOT measure, datasets + sizes
(LongMemEval n=500, Replay n=200, BrainBench n=1240 docs / 350 qrels),
random seed 42, run procedure verbatim, threats to validity (LongMemEval
English+technical skew, char/4 heuristic ~5-10% off, expansion ~97.6%
relative lift on this corpus), per-question raw outputs, pre-registered
expectations (tokenmax wins R@10 by 5-15pp, conservative wins cost by
5-15x, balanced lands within 3pp), re-run cadence anchored to the
src/core/eval/drift-watch.ts watch-list.

Statistical-significance section pins paired bootstrap with 10,000
resamples + Bonferroni correction across 3 modes × 4 metrics [CDX-14].

CLAUDE.md gets two new sections: ## Search Mode (3-mode table + resolution
chain + [CDX-4] cache contamination fix note + CLI commands) and ## Eval
discipline (single-source-of-truth glossary, methodology doc, eval_results
in repo NOT personal brain per [CDX-23]).

README.md Quick Start gets a paragraph naming the install picker, mode
heuristic, and the methodology link.

skills/conventions/search-modes.md NEW — convention file consumed by
brain-ops + query + signal-detector skills via the existing
`> **Convention:**` callout pattern. Routes "what mode" / "tune
retrieval" / "compare modes" queries to the right CLI surface.

skills/RESOLVER.md gets two new trigger rows pointing at
gbrain search * and gbrain eval compare.

* chore: regen llms.txt + llms-full.txt for v0.32.3 search-mode docs

bun run build:llms — picks up the new CLAUDE.md sections (Search Mode +
Eval discipline) and the docs/eval/SEARCH_MODE_METHODOLOGY.md addition.
build-llms.test.ts gate now passes.

* fix(doctor): wire search_mode + eval_drift checks into runDoctor main flow

The v0.32.3 search_mode + eval_drift helpers were inserted into the
DB-checks sub-helper at runDbChecks (line 345-355), but runDoctor itself
maintains its own check list and only calls the helpers' subset. Push
the two checks into the main runDoctor path (after the existing
sync_freshness check at line 2347) so they actually appear in
`gbrain doctor --json` output.

Both checks gated on engine !== null. Progress reporter heartbeat fires
for each. Both still return status 'ok' per [CDX-20] so health score is
preserved.

Verified end-to-end on a real Postgres brain: gbrain doctor --json now
includes 'search_mode' and 'eval_drift' in the checks array.

* fix: claw-test hang — DATABASE_URL leak + telemetry beforeExit deadlock

Two root causes for the hang, both fixed.

1. DATABASE_URL leak in claw-test scripted harness
   The harness inherits the parent process's env via `...process.env`
   for every phase child (init / import / query / extract / doctor).
   When the e2e runner sets DATABASE_URL (for OTHER e2e tests), it
   leaks into claw-test's children. `loadConfig` at src/core/config.ts:143
   then flips inferredEngine to 'postgres' for every subsequent phase,
   breaking the hermetic-PGLite-tempdir contract: phases race against
   each other on a shared test Postgres while pointing at different
   brain states.

   Fix: strip DATABASE_URL + GBRAIN_DATABASE_URL from the child env
   before forwarding. Re-apply GBRAIN_HOME / GBRAIN_FRICTION_RUN_ID
   after the merge so a parent's override can't win. The harness is
   PGLite-only by design.

2. Telemetry beforeExit deadlock
   v0.32.3's recordSearchTelemetry installed a `process.on('beforeExit',
   drainOnExit)` hook that wrapped the flush in `Promise.race([flush(),
   setTimeout(2000)])`. beforeExit fires when the event loop empties,
   but the hook enqueued NEW async work (the race's setTimeout +
   pending flush), so the event loop never re-emptied. Short-lived
   CLI invocations (`gbrain query "the"` finishing in ~100ms) ended
   up waiting on the DB write indefinitely.

   The claw-test harness spawns several short-lived gbrain queries.
   Each one hung after its real work finished. The harness then waited
   forever on its child subprocess's exit code.

   Fix: drop the beforeExit + SIGINT + SIGTERM hooks. Per [CDX-19]'s
   "stats are directional, not exact" contract, losing one unflushed
   bucket on process exit is acceptable. The unref'd setInterval
   handles long-running processes (HTTP MCP, autopilot, jobs work).
   Short-lived CLI invocations exit immediately.

Verified:
  - `gbrain query "the"` on a fresh PGLite brain exits in <1s (was
    hanging forever).
  - `bun test test/e2e/claw-test.test.ts` → 3 pass / 0 fail / 3.86s
    (was hanging at the banner indefinitely).
  - 85/85 e2e files / 574/574 tests pass including claw-test, with
    DATABASE_URL set (the configuration that originally repro'd the
    hang).
  - 6235/6235 unit tests pass.
  - Typecheck clean.

The two bugs interacted: the DATABASE_URL leak meant queries hit the
real Postgres (slow), making the beforeExit deadlock visible. Fixing
either alone would have masked the other. Both fixed in this commit.

* feat(install-picker): cost anchors in mode prompt + upgrade banner + docs

The install picker already asks explicitly (1/2/3 menu, default to the
recommendation on Enter). What was missing: a way to reason about the
cost tradeoff. Without numbers, "tokenmax" looks free and "conservative"
sounds restrictive; with numbers, the operator picks intentionally.

Cost anchors added everywhere the user encounters the mode choice:
  - Install picker MENU_TEXT (gbrain init)
  - Upgrade banner (gbrain upgrade post-upgrade)
  - CLAUDE.md ## Search Mode section
  - README.md Quick Start
  - docs/eval/SEARCH_MODE_METHODOLOGY.md (with the math)

Anchors at Sonnet 4.6 downstream ($3/M input):
  conservative  ~$0.012/query  ~$12/mo @ 1K  ~$1,200/mo @ 100K
  balanced      ~$0.030/query  ~$30/mo @ 1K  ~$3,000/mo @ 100K
  tokenmax      ~$0.060/query  ~$60/mo @ 1K  ~$6,000/mo @ 100K

Plus tokenmax's Haiku expansion overhead: ~$1.50 per 1K queries on top.
Cache hits roughly halve these on a brain with repeat-query traffic.

The math is documented in SEARCH_MODE_METHODOLOGY.md so a reviewer can
audit each variable (T = ~400 tokens/chunk from the recursive chunker's
300-word target; N = `searchLimit` cap; R = downstream model rate from
src/core/anthropic-pricing.ts). Drift away from these numbers requires
updating CLAUDE.md + the picker + the methodology doc in lockstep — a
regression test pins the picker's anchor strings to enforce this.

The framing also names the cost rule honestly: the dominant cost isn't
gbrain (semantic cache is free; Haiku expansion is rounding-error). It's
the downstream agent reading retrieved chunks back into its context.
Operators who don't realize this pick badly.

Tests: 5 new regression cases in init-mode-picker.test.ts pin every
cost string in MENU_TEXT. Total 21/21 picker tests pass; 6240/6240
unit tests pass; verify gate green.

* docs: realistic-scale cost anchor for search modes

The per-query cost framing in the picker (~$0.012/$0.030/$0.060) is
honest but theoretical — it treats each search as an isolated billable
event. Real agent loops amortize a lot of context across turns via
Anthropic prompt caching, so the per-query 5x ratio doesn't translate
1:1 into total agent spend.

Added a "Realistic-scale anchor" section to SEARCH_MODE_METHODOLOGY.md
representing one heavy power-user agent loop running tokenmax:

  - ~860 turns/mo (~29/day, one active agent)
  - ~900K tokens/turn (system + tools + history + reasoning + search)
  - ~$0.85/turn → ~$700/mo total agent spend at tokenmax
  - ~88% Anthropic prompt-cache hit rate

Scaling balanced + conservative DOWN from that anchor:

  - tokenmax  → ~$700/mo, search ~22% of total spend
  - balanced  → ~$620/mo, search ~12% (saves ~$78/mo vs tokenmax)
  - conservative → ~$575/mo, search ~5% (saves ~$124/mo vs tokenmax)

Honest takeaway: at realistic agent-loop scale WITH disciplined prompt
caching, mode choice saves 10-20% of total agent spend, not 5x. The
per-query math kicks back in for setups WITHOUT cache discipline (churn
the prompt prefix every turn → search payload becomes a larger fraction).
Both framings live in the doc.

CLAUDE.md ## Search Mode gets a forward-pointer paragraph naming the
"per-query math vs real-world spend" delta so agents reading the section
find the methodology footnote.

Numbers in the doc are anonymized + scaled away from any specific
deployment. No model names, no specific dollar figures from a real
production setup — just the per-turn / cache-hit-rate / search-count
shape ratios that a thoughtful operator can validate against their own
billing dashboard.

* feat(picker): mode × model cost matrix (25x corner-to-corner spread)

Previous version showed mode costs assuming Sonnet-only downstream.
That muted the spread to 5x and made mode choice look minor. Reality:
the downstream model tier is the BIGGER cost lever — pairing mode with
model is where the 25x spread lives.

New 3×3 matrix in the install picker, CLAUDE.md, methodology doc, README:

                  Haiku 4.5     Sonnet 4.6    Opus 4.7
                  ($1/M input)  ($3/M input)  ($5/M input)
  conservative    $400/mo       $1,200/mo     $2,000/mo
  balanced        $1,000/mo     $3,000/mo     $5,000/mo
  tokenmax        $2,000/mo     $6,000/mo     $10,000/mo

(per-query cost @ 100K queries/mo, full search payload, no cache savings)

The methodology doc gets a new "Mode × Model matrix" section above the
realistic-scale anchor with concrete right-sizing guidance:

  - tokenmax + Haiku: wrong direction. Haiku can't filter 50 chunks → noise
    not signal. Pay Haiku rates, get sub-Haiku quality.
  - conservative + Opus: wasted Opus. 200K context window starved on
    retrieval depth. Pay Opus rates, get conservative-shape retrieval.
  - Natural pairings span ~4x; the matrix corners span 25x. The natural
    diagonal is where most users should land.

Realistic-scale anchor refreshed:
  - tokenmax + Opus: ~$700/mo at 860 turns
  - balanced + Sonnet: ~$430/mo
  - conservative + Haiku: ~$170/mo

Plus a "mismatched pairings" section showing the math for tokenmax+Haiku
and conservative+Opus — both burn budget for no improvement.

Regression test updated: pins the 25x framing + the four anchor cells
(two corners + two diagonal mids) + the three downstream model rates.

22/22 picker tests pass. 6241/6241 unit tests pass. CI guards green.

* docs(picker): rescale cost matrix from 100K → 10K queries/mo (typical single user)

Most users running gbrain are single-user installs at ~10K queries/month,
not the 100K fleet-scale used in the original matrix. The picker numbers
($400 to $10,000/mo) looked alien to the actual audience. Rescaled to
10K with an explicit linear-scaling callout.

New matrix in picker, CLAUDE.md, README, methodology doc:

                  Haiku 4.5     Sonnet 4.6    Opus 4.7
                  ($1/M)        ($3/M)        ($5/M)
  conservative    $40/mo        $120/mo       $200/mo
  balanced        $100/mo       $300/mo       $500/mo
  tokenmax        $200/mo       $600/mo       $1,000/mo

Still 25x corner-to-corner. Still 4x natural-diagonal spread. But now in
numbers a single user picks up and reasons about: "balanced + Sonnet at
$300/mo, that's fine" or "tokenmax + Opus at $1,000/mo, that's a
deliberate choice for max-quality high-stakes work."

Every surface updated:
  - Install picker MENU_TEXT (with "scales linearly — multiply by 10
    for 100K/mo" footnote so heavier users still see their number)
  - CLAUDE.md ## Search Mode table + scaling prose
  - README Quick Start
  - methodology doc Mode × Model matrix section
  - upgrade banner (post-upgrade notice)

Regression test updated: pins the 3 new anchor cells ($40, $300, $1,000)
+ the 10K/mo volume frame + the linear-scaling callout. 23/23 picker
tests pass, 6241/6241 unit tests pass, verify gate green.

Methodology doc's existing 1K/10K/100K Monthly cost breakdown tables
left intact (they already show the linear scaling explicitly).

* feat(picker): agent-facing install protocol + tokenmax default + [AGENT] directive

DX gap: an agent installing gbrain (OpenClaw, Hermes, Codex, Cursor) ran
gbrain init non-TTY, saw 2 stderr lines flash by, and silently auto-applied
a default search mode. The operator never saw the cost matrix or the choice.
At 25x corner-to-corner cost spread, that's surprise-spend territory.

Five surfaces fixed:

1. **Auto-suggest default flipped balanced → tokenmax.** The Sonnet/unknown
   fallback now recommends tokenmax (preserves v0.31.x retrieval shape:
   expand=on, generous result set). Haiku subagent → conservative still
   wins (cost-sensitive signal). No-OpenAI-key → conservative still wins
   (vector search not possible). Heuristic reordered: Haiku check now
   fires BEFORE the Opus check, because a Haiku subagent loop signalling
   cost sensitivity should win over a default-model heuristic.

2. **gbrain init non-TTY output rebuilt.** Previously: 2 stderr lines.
   Now: the full 3×3 cost matrix + an explicit [AGENT] directive block
   telling the agent to relay the matrix to its operator before
   continuing. Includes a pointer to INSTALL_FOR_AGENTS.md Step 3.5 for
   the full protocol.

3. **gbrain upgrade banner same treatment.** Existing v0.32.3 banner now
   includes [AGENT] directive at the top so upgrading agents relay the
   matrix to their operator instead of silently accepting v0.31.x →
   v0.32.x default-applied behavior.

4. **INSTALL_FOR_AGENTS.md Step 3.5 NEW** with the matrix verbatim, the
   exact paraphrasable ask-the-user wording, and the gbrain config set
   commands to run after the operator picks. Plus a paragraph in the
   Upgrade section pointing back at Step 3.5.

5. **AGENTS.md install checklist** gets a new Step 4 ("STOP — ask the
   user about search mode") between init and the rest of the flow. The
   agent's job description now explicitly says: silent acceptance is
   the wrong default.

Tests (24/24 pass):
  - Updated recommendModeFor heuristic order (Haiku floor > Opus default)
  - New regression test: non-TTY output contains the matrix corners +
    [AGENT] directive + INSTALL_FOR_AGENTS.md pointer
  - withEnv() helper used for OPENAI_API_KEY mutation (test-isolation lint)
  - Default-recommendation tests updated: Sonnet / unknown → tokenmax

Privacy + test-isolation gates clean. 6256/6256 unit tests pass.

---------

Co-authored-by: garrytan-agents <agents@garrytan.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
 2026-05-13 13:14:58 -04:00
Garry TanClaude Opus 4.7Eva
 		182a144272 v0.33.1.1 fix: Voyage output_dimension + flexible-dim guard + OOM-cap rethrow (#962) 
* fix: send Voyage output_dimension on embedding requests

* fixup: drop voyage-4-nano from flexible-dim set

Voyage's hosted /embeddings endpoint accepts `output_dimension` only for
the seven flexible-dim models (voyage-4-large, voyage-4, voyage-4-lite,
voyage-3-large, voyage-3.5, voyage-3.5-lite, voyage-code-3). voyage-4-nano
is an open-weight variant Voyage lists separately as fixed 1024-dim — the
hosted API rejects the parameter for it.

The recipe docstring previously claimed "all v4 variants" have flexible
dims, which is what led to nano being added to the allowlist in the first
place. Tighten the comment to name the hosted trio explicitly and call out
nano-as-open-weight.

Convert the test case at test/ai/gateway.test.ts from a positive assertion
(voyage-4-nano returns { dimensions: 512 }) to a negative regression pin
(voyage-4-nano returns undefined), so a future contributor can't silently
re-add nano without breaking this test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: Voyage OOM-cap rethrow + flexible-dim runtime validation (Codex P3 follow-ups)

Two follow-ups from Codex's adversarial review of PR #962, both Voyage-adjacent
correctness fixes that the original PR scope had filed as TODOs.

1. gateway.ts:619 Voyage OOM cap was theatrical
-------------------------------------------------
voyageCompatFetch's inbound response rewriter is wrapped in a try/catch that
falls back to the original response on parse failure — correct for "Voyage
returned JSON I can't reshape, let the SDK handle it." But the per-embedding
Layer 2 OOM cap at line 619 threw a bare `new Error(...)`, which the same
catch silently swallowed. Net result: an oversized base64 response (Layer 1
skipped because no Content-Length header) returned through to the AI SDK and
could OOM the worker on JSON.parse.

Fix: introduce `VoyageResponseTooLargeError`, throw it at both cap sites
(Content-Length Layer 1 at line 595 and per-embedding Layer 2 at line 619),
and rethrow it from the inbound try/catch via `if (err instanceof
VoyageResponseTooLargeError) throw err`. Pre-existing fall-back-on-parse-error
behavior for other thrown errors is preserved.

Regression-pinned by 2 new behavioral tests (mock fetch returns oversized
Content-Length / oversized base64; embed() throws with the expected message)
and a structural assertion in test/voyage-response-cap.test.ts that the
`instanceof VoyageResponseTooLargeError ⇒ throw` line stays put.

2. Voyage flexible-dim runtime validation + doctor check
-------------------------------------------------------
A brain configured for a Voyage flexible-dim model (voyage-4-large,
voyage-3-large, voyage-3.5, voyage-3.5-lite, voyage-4, voyage-4-lite,
voyage-code-3) without an explicit `embedding_dimensions` would fall back to
DEFAULT_EMBEDDING_DIMENSIONS=1536 — an OpenAI default that Voyage rejects.
Voyage's only accepted values are {256, 512, 1024, 2048}. Pre-fix the failure
surfaced as an HTTP 400 from Voyage that often got misclassified as a
transient network error.

Fix:
- `dims.ts` exports `VOYAGE_VALID_OUTPUT_DIMS` and `isValidVoyageOutputDim`.
- `dimsProviderOptions` throws `AIConfigError` with a paste-ready fix command
  (`gbrain config set embedding_dimensions ...`) when a Voyage flexible-dim
  model is configured with an invalid dim value.
- `gbrain models doctor` gets a new `embedding_config` probe that runs first
  (zero tokens) and surfaces the misconfiguration before any chat/expansion
  probes spend a single token. New probe status `config` + optional `fix`
  hint rendered in human output.

Regression-pinned by 6 new unit tests covering the AIConfigError throw,
exact valid-values set, the bypass path for fixed-dim Voyage models, and
the fix-hint contents.

* chore: bump version and changelog (v0.33.1.1)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.33.1.1

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Eva <eva@100yen.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 12:21:21 -04:00
garrytan-agentsClaude Opus 4.7garrytan-agentsGarry Tan
 		7be17261bc v0.32.3.0 skill: functional-area-resolver — pattern for compressing routing tables (#859) 
* skill: compress-agents-md — functional-area resolver pattern

Proven via A/B eval: 100% routing accuracy at 48% size reduction.
Converts granular per-skill resolver rows into functional-area dispatchers
with '(dispatcher for: ...)' sub-skill lists.

Includes:
- SKILL.md with full pattern docs, before/after examples, eval results
- routing-eval.jsonl with 5 fixtures
- Anti-patterns (resolver-of-resolvers pipe table = 15% accuracy)

* skill: rename compress-agents-md → functional-area-resolver, cite prior art

The contribution is a pattern (functional-area dispatcher with `(dispatcher
for: ...)` clauses), not a file. Rename describes the contribution; triggers
broaden to cover both AGENTS.md and RESOLVER.md phrasings.

SKILL.md rewrite:
- Three-model A/B table (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) replaces the
  original Sonnet-only claim. Functional-areas beats baseline by +13 to +17pp
  training (lenient) across all three models at 48% the size.
- Strict + lenient scoring documented side by side. Lenient (predicted shares
  dispatcher area with expected) matches production agent behavior.
- Preconditions added: refuse to compress if file <12KB or working tree dirty.
- Multi-file routing precedence section for the v0.31.7 RESOLVER.md/AGENTS.md
  merge case.
- Mandatory verification step (≥95% via the harness).
- Daily-doctor.mjs reference scrubbed (didn't exist in gbrain).
- Three prior-art citations: AnyTool (arXiv:2402.04253), RAG-MCP
  (arXiv:2505.03275), Anthropic Agent Skills progressive disclosure. The
  pattern is the static-prompt analog of runtime hierarchical routing.

routing-eval.jsonl: 8 positive (5 original + 3 broadened triggers) + 4
adversarial negatives targeting skillify, skill-creator, book-mirror,
concept-synthesis to prove broadened triggers don't over-capture adjacent
meta-skills.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* evals: A/B harness for functional-area-resolver (gateway-routed, strict + lenient scoring)

evals/functional-area-resolver/ lives outside skills/ deliberately. The
skillpack bundler walks skills/<skill>/ recursively, so an eval surface in
there would copy harness + variants + fixtures + tests into every downstream
install. The pattern (in SKILL.md) ships everywhere; the eval evidence stays
in the gbrain repo.

What ships:
- Three variant resolvers in variants/ — baseline.md (verbose 25KB) and
  functional-areas.md (compressed 13KB) extracted from a real production
  AGENTS.md at git commits 93848ff3b^ and 93848ff3b (owner PII scrubbed).
  resolver-of-resolvers.md derived mechanically by stripping (dispatcher
  for: ...) clauses — the ablation case.
- 20 hand-authored training fixtures + 5 held-out blind fixtures.
- harness-runner.ts — TypeScript runner via gbrain gateway. Flags:
  --model {opus|sonnet|haiku|<full-id>}, --variants-dir, --variants for
  description-length sweeps, --parallel N (rate-lease bound), --limit N
  for smoke runs, --yes for non-TTY.
- Every output row carries BOTH `correct` (strict) and `correct_lenient`
  (predicted shares dispatcher area with expected). Lenient matches
  production behavior.
- Receipt header binds (model, prompt_template_hash, fixtures_hash,
  harness_sha, ts, cmd_args). Re-runs are auditable.
- harness.mjs — thin Node shim that spawns the TS runner via bun.
- rescore.mjs — zero-cost lenient re-score of an existing JSONL.
- harness-runner.test.ts — 45 unit tests (no API key needed) covering
  every pure function plus the dispatcher-list parser.

The prompt template is load-bearing: without the "drill into (dispatcher
for: ...) list" instruction, every compression variant collapses to
~30-60%. Documented in SKILL.md and README.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* evals: baseline receipts (Opus 4.7 + Sonnet 4.6 + Haiku 4.5, 2026-05-11)

Three canonical 225-row receipts (3 variants × 25 fixtures × 3 seeds per
model). Each receipt header binds (model, prompt_template_hash,
fixtures_hash, harness_sha, ts) so the published SKILL.md numbers are
reproducible.

Training corpus (n=20, lenient):
  baseline      | Opus 81.7% | Sonnet 86.7% | Haiku 73.3% | 25KB
  functional-areas | Opus 98.3% | Sonnet 100%  | Haiku 88.3% | 13KB
  resolver-of-resolvers | Opus 63.3% | Sonnet 41.7% | Haiku 65.0% | 10KB

functional-areas beats baseline by +13 to +17pp across all three models at
48% the size. resolver-of-resolvers' Sonnet collapse (41.7%) is the SKILL.md
"compression without dispatcher clause is broken" claim, observed.

Held-out (n=5, lenient) saturates at 100% across most cells (Sonnet ×
resolver-of-resolvers is 73.3% — the same failure mode visible on a smaller
sample).

~$3 API spend across all three runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* skill: wire functional-area-resolver into RESOLVER.md + manifests

skills/RESOLVER.md gets a new row in Operational, adjacent to skillify.
Triggers: "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too
big", "functional area dispatcher", "shrink routing table".

skills/manifest.json adds the new entry and bumps manifest version
0.25.1 → 0.32.3.0 (loadOrDeriveManifest reads this for sync-guard).

openclaw.plugin.json adds functional-area-resolver to the skills array
and bumps version 0.25.1 → 0.32.3.0 so install receipts stop being stale
(src/core/skillpack/installer.ts:307-311 uses manifest version on every
install).

Verified:
- gbrain check-resolvable --json: 42/42 reachable, 0 errors.
- gbrain routing-eval: 70/70 pass (100% structural).
- bun test test/skillpack-sync-guard.test.ts: passes (manifest in sync).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.32.3.0 skill: functional-area-resolver — pattern for compressing routing tables

Headline: compress a 25KB AGENTS.md down to 13KB without losing routing
accuracy. Pattern proven across Opus 4.7, Sonnet 4.6, and Haiku 4.5 — beats
the verbose baseline by +13 to +17pp at 48% the size.

Empirical (training, n=20, 3 seeds, lenient):
  baseline 25KB:                Opus 81.7% | Sonnet 86.7% | Haiku 73.3%
  functional-areas 13KB:        Opus 98.3% | Sonnet 100%  | Haiku 88.3%
  resolver-of-resolvers 10KB:   Opus 63.3% | Sonnet 41.7% | Haiku 65.0%

The (dispatcher for: ...) clause is the load-bearing signal. Strip it (the
resolver-of-resolvers variant) and Sonnet collapses to 41.7% — the failure
case the pattern's authors predicted, now observed.

Files in this release:
- VERSION + package.json bumped to 0.32.3.0 (4-segment per CLAUDE.md).
- CHANGELOG.md: full empirical story, cross-model table, three prior-art
  citations (AnyTool, RAG-MCP, Anthropic Agent Skills progressive
  disclosure).
- TODOS.md: nine v0.33.x follow-ups (dogfood on gbrain's own RESOLVER.md,
  CLI promotion to gbrain routing-eval --ab-compare, held-out corpus
  growth, cross-vendor Gemini+GPT verification, per-row description
  length sweep, structural compression to ~10KB, hierarchical
  area-of-areas, embedding pre-router, adversarial fixtures,
  prompt-design ablation doc).
- llms-full.txt regenerated.

Bisect-friendly history on this branch:
  502d447e  skill: rename + content rewrite + routing-eval.jsonl
  472cc686  evals: A/B harness + variants + fixtures + tests (no receipts)
  243e013e  evals: cross-model baseline receipts (Opus + Sonnet + Haiku)
  ecab180b  skill: wire-up to RESOLVER.md + manifest.json + openclaw.plugin.json
  THIS:     v0.32.3.0 release marker

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* evals: codex review fixes — accept ASCII -> arrow + provider-aware auth gate

Two P2 findings from /codex review on commit 8870c64e:

P2-2: parseDispatcherLists regex required Unicode `→`, but SKILL.md
Step 4 documents the template with ASCII `->`. Downstream-authored
resolvers following the template silently fell through to strict-only
scoring (correct_lenient == correct always), under-reporting same-area
accuracy with no warning. Regex now accepts both `→` and `->`. Two
new test cases pin the behavior — pure-ASCII variant + mixed-arrow
variant.

P2-3: main() exited with `ANTHROPIC_API_KEY is not set` even when the
user passed `--model openai:gpt-4o` with a valid OPENAI_API_KEY. The
CLI advertises full provider:model support (resolveModel tests cover
openai:* explicitly) and the gateway routes by recipe; the env check
should match the provider that will actually be called. Now extracts
the provider id from the model string and looks up the right env var
from REQUIRED_ENV_BY_PROVIDER (anthropic, openai, google, groq,
voyage, together, deepseek, minimax, dashscope, zhipu). Unknown
providers fall through to the gateway, which raises a clear
recipe-specific error.

47/47 harness unit tests pass after the change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* skill: codex review P2-1 — verification gate now tests the user's edited file

The original SKILL.md Step 6 told users to run `node harness.mjs` from the
gbrain repo as the mandatory ≥95% gate. But that runs the harness against
the COMMITTED sample variants in evals/functional-area-resolver/variants/,
not the file the user just compressed. The gate could pass while the edit
dropped a sub-skill.

Step 6 now:
- Gate 1 stays at `gbrain routing-eval --json` (structural, runs against
  the user's actual routing-eval.jsonl fixtures).
- Gate 2 is rewritten: copy the user's edited routing file into a tmp
  variants dir, then run `node harness.mjs --variants-dir <tmp>
  --variants my-edit --model opus`. This exercises the harness's existing
  --variants flag (added in commit 472cc686 / T4) but now points at the
  user's actual edit. The harness uses gbrain-bundled fixtures, so this
  is a regression check on shared skills, not a full eval of the user's
  fixture set — and the SKILL.md says so explicitly.

Also adds a "common false negatives" callout: when the user's routing
file doesn't expose the skills gbrain's bundled fixtures target (e.g.
`gmail`, `enrich`), expect strict-scoring fails on those rows; lenient
scoring remains accurate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* evals: codex review P3 — regenerate Opus baseline with current schema

The prior Opus receipt was generated before commit 472cc686 (T4 added
harness_sha to ReceiptRow and correct_lenient to every RunRow). The
Sonnet and Haiku receipts shipped with the new schema, but Opus was
the outlier.

This run was produced with the current harness (sha ca99fbfeb, after
the P2-1 + P2-2 + P2-3 fixes). The harness_sha in the receipt header
binds the numbers to a specific harness revision so consumers can detect
schema drift.

Numbers (training, lenient, n=20, 3 seeds):
  baseline:              81.7% ± 7.2%  (unchanged — strict and lenient are equal)
  functional-areas:      100% ± 0%     (was 98.3% — one nondeterministic seed
                                         is now in-cluster; pattern continues
                                         to beat baseline at 48% the size)
  resolver-of-resolvers: 66.7% ± 7.2%  (was 63.3% — still in noise; absent
                                         dispatcher clause keeps it ~30pp
                                         behind functional-areas on training)

Held-out (n=5, 3 seeds, lenient): all variants 100% except resolver-of-
resolvers on Sonnet (committed in earlier baseline) — Opus held-out
saturates the small fixture set.

Run cost: ~$1.40 at Opus 4.7 pricing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* post-merge: scrub fork-private paths + add Contract/Output Format sections

Two CI gates landed on master after this branch was cut:

1) scripts/check-privacy.sh (v0.32.2): banned /data/brain/ and /data/.openclaw/
   in committed files. The eval variants extracted from a real production
   AGENTS.md still contained those fork-private path literals. Rewrote to
   /your/brain/path/, /your/agent/.openclaw/, /your/gbrain, /your/gstack,
   /your/tmp, /your/git-projects/. Only path strings changed — the routing
   structure (skill names, dispatcher clauses, trigger phrases) is byte-for-
   byte identical, so harness baseline-runs/ receipts are still valid.

2) test/skills-conformance.test.ts (master): added required sections
   `## Contract` and `## Output Format` to every skill. Added both to
   skills/functional-area-resolver/SKILL.md following the book-mirror
   convention (short body referencing the canonical content above + a
   conformance-test footnote). Contract notes the privacy guarantee +
   the verification-gate semantics; Output Format documents the area
   entry template (with both ASCII -> and Unicode → arrows accepted).

Full unit suite: 5578 pass / 0 fail. bun run verify clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: surface functional-area-resolver in CLAUDE.md + README.md for v0.32.3.0

CLAUDE.md — adds a "Routing-table compression (v0.32.3.0)" entry under Skills,
covering the two-layer dispatch pattern, the load-bearing (dispatcher for: ...)
clause, the eval surface at evals/functional-area-resolver/, the three
cross-model baseline receipts, the 25KB → 13KB compression numbers, and the
nine v0.33.x follow-up TODOs. Cites AnyTool / RAG-MCP / Anthropic Agent Skills
prior art so the pattern's position in the literature is discoverable from the
agent entry point.

README.md — adds a "New in v0.32.3.0" callout in the intro section so users
landing on the repo see the new skill before scrolling to the skills list.
Links the SKILL.md and eval directory; states the cross-model gain (+13 to
+17pp at 48% the size) so the reason to apply the pattern is one click away.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 20:39:00 -07:00
Garry TanClaude Opus 4.7SiyaoZhengcacityMagicray1217JamesJZhang
 		71ed8d0d21 v0.32.0 feat: 5 new embedding recipes + discoverability pass (closes 17-PR cluster) (#810) 
* feat(ai/types): add resolveAuth + probe + user_provided_models fields

Foundation commit for the embedding-provider fix-wave (5 API-key recipes
+ discoverability pass). Three optional additions to the recipe contract:

- `EmbeddingTouchpoint.user_provided_models?: true` (D8=A): flag for
  recipes that ship without a fixed model list. Consumed by the contract
  test (permits empty `models[]`), gateway.ts:223 (replaces hardcoded
  `recipe.id === 'litellm'` check in a follow-up commit), and
  init.ts:resolveAIOptions (refuses implicit "first model" pick for
  shorthand `--model <provider>`).

- `Recipe.resolveAuth?(env): {headerName, token}` (D12=A): unified auth
  seam across embed / expansion / chat. Default behavior (returns
  `Authorization: Bearer <env-key>`) covers the existing 9 recipes
  unchanged. Recipes deviating (Azure with `api-key:`; future OAuth
  providers) override this single seam instead of adding parallel
  mechanisms in 3 places. Codex review caught that auth was triplicated
  at gateway.ts:281/728/931; D12=A unifies all three in one follow-up
  commit.

- `Recipe.probe?(): Promise<{ready, hint?}>` (D13=A): recipe-owned
  readiness check for local-server providers (ollama, llama-server).
  Replaces the hardcoded `recipe.id === 'ollama'` special case in
  providers.ts. Wrapped in 200ms timeout at the call sites.

Pure type additions — no behavior change. Typecheck green; existing 9
recipes work unchanged because all three fields are optional.

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (decisions
D8=A, D11=C, D12=A, D13=A).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai/gateway): unify openai-compatible auth via Recipe.resolveAuth (D12=A)

Pre-v0.32, openai-compatible auth was duplicated 3 times in gateway.ts at
instantiateEmbedding, instantiateExpansion, instantiateChat — with subtle
drift (embedding had a `${recipe.id.toUpperCase()}_API_KEY` fallback the
other two lacked). Codex outside-voice review caught this during /plan-eng-review.

D12=A: unify all three through `Recipe.resolveAuth?(env)` (declared in the
prior commit). Two new module-level helpers:

- `defaultResolveAuth(recipe, env, touchpoint)` — applied when a recipe
  doesn't declare its own resolver. Returns Authorization Bearer with
  `auth_env.required[0]`, falling back to the first present
  `auth_env.optional` env var, or 'unauthenticated' for no-auth recipes
  like Ollama. Throws AIConfigError with the recipe's setup_hint when
  required env is missing.

- `applyResolveAuth(recipe, cfg, touchpoint)` — returns
  `createOpenAICompatible` options. Bearer-via-Authorization paths use
  the SDK's native `apiKey` field; custom-header paths (Azure: api-key)
  use `headers` and OMIT apiKey to avoid double-auth leaks.

The 3 `case 'openai-compatible':` branches in instantiateEmbedding (line
~281), instantiateExpansion (line ~728), instantiateChat (line ~931) each
collapse from ~10 lines of bespoke auth handling to a single
`applyResolveAuth(recipe, cfg, '<touchpoint>')` call.

Also: the litellm-template hardcode at gateway.ts:223 (`recipe.id ===
'litellm'`) is replaced with a union check for
`EmbeddingTouchpoint.user_provided_models === true` (D8=A wire-through
per Codex finding #3). Pre-v0.32 builds keep working via back-compat
`recipe.id === 'litellm'` clause; new recipes declaring
user_provided_models pick up the same gating automatically.

Existing 9 recipes (openai, anthropic, google, deepseek, groq, ollama,
litellm-proxy, together, voyage) gain zero per-recipe edits — the
default resolver covers their existing behavior. Behavior change for
ollama expansion/chat only: now reads OLLAMA_API_KEY when set (pre-v0.32
silently passed 'unauthenticated' for those touchpoints; embedding
already read it). Ollama servers ignore the header so no real-world
impact; this aligns the 3 touchpoints.

Tests: bun test test/ai/ — 77/77 pass.

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (D8=A,
D12=A; addresses Codex findings #3, #4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(ai): IRON RULE regression test for v0.32 resolveAuth refactor

Pins the contract that the v0.32 D2/D12=A resolveAuth refactor preserves
auth behavior for the 9 existing recipes (openai, anthropic, google,
deepseek, groq, ollama, litellm-proxy, together, voyage).

10 cases covering:
- the 9 expected recipe ids are still registered
- every recipe with non-empty required[] returns Authorization Bearer <key>
- missing required env throws AIConfigError naming recipe + touchpoint + env-var
- Ollama (empty required, optional set) reads first present optional env
- Ollama (no env) falls back to "Bearer unauthenticated"
- all 3 touchpoints (embedding/expansion/chat) produce identical auth
  shape for the same recipe + env (this is the core regression: pre-v0.32,
  embedding had a fallback the other two lacked)
- applyResolveAuth converts Authorization Bearer to {apiKey} (SDK-native)
- applyResolveAuth respects a custom-header override (Azure preview; the
  recipe ships in commit 8) and emits {headers} WITHOUT apiKey to avoid
  double-auth
- native-* recipes (openai, anthropic, google) intentionally have no
  resolveAuth declared (they use AI-SDK adapters directly)
- all openai-compatible recipes ship without resolveAuth in v0.32 (default
  applies); the first override is Azure in commit 8

Also: export `defaultResolveAuth` and `applyResolveAuth` as @internal
gateway helpers so tests can pin them directly. Mirrors the pattern of
`splitByTokenBudget` and `isTokenLimitError` already exported with the
same @internal annotation.

Tests: bun test test/ai/ — 87/87 pass (10 new + 77 existing).
Typecheck: clean.

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (IRON RULE
per Section 3 test review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add llama-server recipe (#702 reworked)

10th recipe in the registry; first to ship Recipe.probe (D13=A) and the
second user_provided_models recipe (litellm-proxy is the first).

llama.cpp's llama-server exposes an OpenAI-compatible /v1/embeddings
endpoint. Distinct from Ollama: different default port (8080), different
model-management story (you launch it with --model <path>; the server
serves whatever was passed). Recipe ships with `models: []`,
`user_provided_models: true`, `default_dims: 0` so the wizard refuses
implicit defaults and forces explicit --embedding-model + --embedding-dimensions.

Added:
- src/core/ai/recipes/llama-server.ts (61 lines)
- probeLlamaServer() in src/core/ai/probes.ts; reads
  LLAMA_SERVER_BASE_URL with default http://localhost:8080/v1
- Registered in src/core/ai/recipes/index.ts (10 recipes total now)
- test/ai/recipe-llama-server.test.ts (8 cases): registered + shape,
  user_provided_models flag, probe declared + reachability fail-with-hint,
  default-auth covering no-env / API_KEY / URL-shaped-only paths

Hardening: defaultResolveAuth in gateway.ts now skips URL-shaped optional
env entries (names ending in _URL or _BASE_URL) when picking a fallback
auth token. Pre-fix, OLLAMA_BASE_URL=http://my-ollama would have become
the Bearer token; Ollama ignores it but llama-server (and future
local-server recipes) shouldn't depend on the server tolerating garbage
auth. The regression test (recipes-existing-regression) gains one case
pinning this contract.

Per-recipe test file follows D7=B (per-recipe over DRY for readability).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 4
of 11). Reworked from #702 because the original PR didn't model the
recipe-owned probe pattern (D13=A) or user_provided_models (D8=A).

Tests: bun test test/ai/ — 95/95 pass (8 new + 87 existing).

Co-Authored-By: SiyaoZheng <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add MiniMax recipe (#148 reworked)

11th recipe. embo-01 model, 1536 dims, $0.07/1M tokens.

OpenAI-compatible at api.minimax.chat. MiniMax requires a `type:
'db' | 'query'` field for asymmetric retrieval (documents indexed with
type='db', queries embedded with type='query'). gbrain has no
query/document signal at the embed-call site today, so v1 defaults to
type='db' for both indexing and retrieval — same vector space, symmetric
similarity. Asymmetric query support is a follow-up TODO that needs the
embed seam to thread query/document context.

Plumbed via src/core/ai/dims.ts: dimsProviderOptions returns
{openaiCompatible: {type: 'db'}} for modelId === 'embo-01'.

Conservative max_batch_tokens=4096 declared (MiniMax docs don't publish
the limit). Recursive halving in the gateway catches token-limit errors
at runtime.

Tests: bun test test/ai/ — 101/101 (6 new + 95 prior).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 5
of 11). Reworked from #148.

Co-Authored-By: cacity <20351699+cacity@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add Alibaba DashScope recipe (#59 split, part 1/2)

12th recipe. text-embedding-v3 (current) + text-embedding-v2; 1024
default dims with Matryoshka options [64, 128, 256, 512, 768, 1024].

OpenAI-compatible at dashscope-intl.aliyuncs.com. China-region users
override via cfg.base_urls['dashscope']; v0.32 ships with the
international default.

Conservative max_batch_tokens=8192 + chars_per_token=2 declared because
Alibaba doesn't publish a hard batch limit and text-embedding-v3 mixes
English + CJK heavily (CJK density closer to Voyage than OpenAI tiktoken).

Tests: bun test test/ai/ — 106/106 (5 new + 101 prior).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 6
of 11). Reworked from #59 (DashScope+Zhipu split into 2 commits per
the plan; Zhipu lands next).

Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add Zhipu AI (BigModel) recipe (#59 split, part 2/2)

13th recipe. embedding-3 (current) + embedding-2; 1024 default dims
with Matryoshka options [256, 512, 1024, 2048].

OpenAI-compatible at open.bigmodel.cn. embedding-3 at 2048 dims exceeds
pgvector's HNSW cap of 2000 — those brains fall back to exact vector
scans via the existing chunkEmbeddingIndexSql policy at
src/core/vector-index.ts. Default stays at 1024 (HNSW-fast); users who
want maximum fidelity opt into 2048 via --embedding-dimensions and
accept the slower retrieval.

Tests pin the HNSW boundary: 1024 returns the index SQL, 2048 returns
the skip-index/exact-scan SQL.

Tests: bun test test/ai/ — 112/112 (6 new + 106 prior).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 7
of 11). Reworked from #59. Together with DashScope (commit 6), closes
the China-region embedding gap users repeatedly reported (DashScope
covers Alibaba, Zhipu covers BigModel; both ship with international
endpoints by default).

Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): add Azure OpenAI recipe (#459 reworked)

14th recipe and the first to exercise both v0.32 architectural seams:

- resolveAuth (D12=A) returns `{headerName: 'api-key', token: <key>}`
  instead of the default Authorization Bearer. Azure rejects double-auth,
  so applyResolveAuth puts the key in `headers` and OMITS apiKey.
- A new `Recipe.resolveOpenAICompatConfig?(env)` seam (Recipe.ts) lets
  the recipe template the baseURL from env (Azure: ENDPOINT + DEPLOYMENT
  combine into a non-/v1 path) and inject a custom fetch wrapper that
  splices ?api-version= onto every request URL.

The fetch wrapper is type-safe via `as unknown as typeof fetch`; AI SDK
never calls TS's strict `preconnect()` method on the wrapper so the cast
is sound. `applyOpenAICompatConfig` (new gateway helper) routes through
the recipe override or falls back to the pre-v0.32 base_urls/base_url_default
behavior — existing 13 recipes get zero behavior change.

API version defaults to `2024-10-21` (current stable as of 2026-05);
override via AZURE_OPENAI_API_VERSION env. Endpoint trailing slash gets
stripped during URL construction so users can copy-paste from the Azure
portal.

Tests (12 cases in test/ai/recipe-azure-openai.test.ts):
- resolveAuth returns api-key NOT Authorization Bearer
- applyResolveAuth puts key in headers, NOT apiKey (no double-auth)
- baseURL templating from endpoint + deployment, with trailing-slash strip
- AIConfigError on missing endpoint OR deployment
- fetch wrapper splices api-version (default + AZURE_OPENAI_API_VERSION override)
- fetch wrapper does NOT double-add api-version when caller already set it
- applyOpenAICompatConfig honors recipe override

IRON RULE regression test updated: now asserts azure-openai is the
documented exception that overrides resolveAuth; any future override
needs review.

Tests: bun test test/ai/ — 124/124 (12 new + 112 prior).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 8
of 11, plus the resolveOpenAICompatConfig seam discovered during fold-in).
Reworked from #459. The original PR proposed a hardcoded AzureOpenAI
client switch; this implementation routes through the unified seams so
future Azure-shaped providers (other custom-URL services) can reuse them.

Co-Authored-By: JamesJZhang <32652444+JamesJZhang@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): adjacent fixes — no_batch_cap (#779) + config-key fallbacks (#121)

Two small ergonomics fixes folded together (#765 deferred — see TODOS.md
follow-up; the CJK PGLite extraction was bigger than the plan estimated).

#779 reworked (alexandreroumieu-codeapprentice): silence the
missing-max_batch_tokens startup warning for recipes with genuinely
dynamic batch capacity. New `EmbeddingTouchpoint.no_batch_cap?: true`
field. Set on ollama (capacity depends on locally loaded model +
OLLAMA_NUM_PARALLEL), litellm-proxy (depends on backend), llama-server
(set by --ctx-size at server launch). Three less stderr warnings on
every gateway configure; google still warns (it's a real fixed-cap
provider that ought to ship a max_batch_tokens declaration).

Bonus: litellm-proxy now declares `user_provided_models: true`, removing
the last consumer of the legacy `recipe.id === 'litellm'` hardcode in
gateway.ts:223 (D8=A wire-through completion).

#121 reworked (vinsew): self-contained API keys. Two parts:

  1. config.ts: ANTHROPIC_API_KEY env merge was silently missing.
     loadConfig() merged OPENAI_API_KEY but not ANTHROPIC_API_KEY into
     the file-config-shape result. One-line addition.

  2. cli.ts:buildGatewayConfig: when ~/.gbrain/config.json declares
     openai_api_key / anthropic_api_key but the process env doesn't
     have those env vars set (common for launchd-spawned daemons,
     agent subprocess tools, containers that don't propagate
     ~/.zshrc), fold the config-file values into the gateway env
     snapshot. Process env still wins (loaded last) so per-process
     overrides keep working.

Tests (4 cases in test/ai/no-batch-cap-suppression.test.ts):
- Ollama / LiteLLM / llama-server all declare no_batch_cap: true
- configureGateway does NOT warn for those three
- configureGateway STILL warns for google (regression guard)
- Cross-cutting invariant: empty-models recipes declare user_provided_models

Tests: bun test test/ai/ — 128/128 (4 new + 124 prior).

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 9 of 11).
#765 (Hunyuan PGLite + CJK keyword fallback) deferred to TODOS.md
follow-up; the CJK extraction (~150 lines + scoring logic + tests) is
larger than the wave's adjacent-fix lane should carry. Closes that PR
with a deferral note.

Co-Authored-By: alexandreroumieu-codeapprentice <noreply@github.com>
Co-Authored-By: vinsew <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(discoverability): doctor alt-provider advisory + init user_provided_models refusal

Two small but high-leverage changes that address the discoverability
problem the v0.32 wave is trying to fix.

src/commands/doctor.ts: new `alternative_providers` check (8c). After
the existing embedding-provider smoke test, walks listRecipes() and
surfaces any recipe whose required env vars are ALL present in the
process env but is not the currently configured provider. Reports as
status: 'ok' with an informational message — never errors. Helps users
discover that, e.g., `OPENAI_API_KEY=x DASHSCOPE_API_KEY=y` configured
for openai means they have a Chinese-region alternative ready without
extra setup.

src/commands/init.ts: user_provided_models recipes (litellm, llama-server)
now refuse the implicit "first model" pick from shorthand --model with
a structured setup hint pointing the user at the explicit form
`--embedding-model <provider>:<your-model-id> --embedding-dimensions <N>`.
Pre-fix, shorthand --model litellm threw "no embedding models listed"
which was technically correct but unhelpful. The new error includes the
recipe's setup_hint when available.

Tests: bun test test/ai/ — 128/128 pass; typecheck clean.

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 10
of 11). The full interactive provider chooser in init.ts (the bigger
piece of the discoverability lane) is deferred to a v0.32.x follow-up;
this commit ships the doctor advisory + cleaner refusal that close the
80% case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.32.0): embedding-providers.md + README callout + CHANGELOG + TODOS.md

Final commit of the v0.32 wave. Closes the discoverability gap that
generated the 17-PR community cluster.

- New docs/integrations/embedding-providers.md: capability matrix, decision
  tree, per-recipe one-pagers, OAuth provider notes, "my provider isn't
  listed" pointer to LiteLLM proxy. Voice: capability not marketing per
  CLAUDE.md voice rules.

- README.md: embedding-providers callout near the top, naming the count
  (14 recipes) and pointing at the new doc.

- CHANGELOG.md: v0.32.0 entry following the verdict-headline format from
  CLAUDE.md voice rules. Lead-with-numbers ("14 providers, 5 new"), what-this-
  means-for-users closer, "to take advantage" upgrade block, itemized
  changes, contributor credits, deferred-with-context list.

- VERSION + package.json: 0.31.1 → 0.32.0. Minor bump justified by the
  new public Recipe surface (resolveAuth, resolveOpenAICompatConfig, probe,
  user_provided_models, no_batch_cap fields), the new OAuth subsystem
  scaffold (deferred to v0.32.x but typed in v0.32.0), and the 5 new
  recipes.

- TODOS.md: 7 follow-up entries for the v0.32 wave's deferred work
  (Vertex ADC, Copilot OAuth, Codex OAuth, CJK PGLite, interactive
  wizard, real-credentials CI matrix, MiniMax asymmetric retrieval,
  multimodal hardcode un-stuck). Each entry has full context + the
  exact file paths + the spike work needed so a future contributor can
  pick up cleanly.

Tests: bun test test/ai/ — 128/128 pass; typecheck clean.

Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 11
of 11). Wave complete: 11 commits, ~1500 net lines, 5 new recipes, full
docs, doctor advisory, IRON RULE regression test, 7 TODOS for the
v0.32.x follow-up wave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: regenerate llms.txt + llms-full.txt for v0.32.0

After commit c384fadc added the embedding-providers callout to README.md,
the committed llms-full.txt drifted from the generator output and the
build-llms test failed. Running `bun run build:llms` regenerates both
files. The single line addition is the README callout pointing at
docs/integrations/embedding-providers.md.

Tests: bun test test/build-llms.test.ts — 7/7 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: hermetic GBRAIN_HOME for brain-registry serial flake + withEnv on recipe-llama-server

Two test-isolation cleanups uncovered while shipping v0.32.

test/brain-registry.serial.test.ts (the BrainRegistry "empty/null/undefined
id routes to host" test): pre-existing flake on dev machines that have a
real ~/.gbrain/config.json. The test asserts getBrain(null) REJECTS but
on those machines the host-init path RESOLVES instead (it found the
maintainer's actual brain). The fix pins GBRAIN_HOME to a guaranteed-empty
tempdir for the test's duration so host-init has nothing to find and fails
loudly with a non-UnknownBrainError — exactly what the assertion wants.
File is .serial.test.ts so direct process.env mutation is allowed by the
test-isolation linter (R1 quarantine).

test/ai/recipe-llama-server.test.ts: rewrites the manual beforeEach/afterEach
env save/restore as withEnv() per the canonical pattern in
test/helpers/with-env.ts. The original was correct in behavior but tripped
the test-isolation linter (R1: process.env mutation). withEnv() is exactly
the cross-test-safe save+try/finally+restore the manual code did, just
factored out. No behavior change.

Tests: bun run test — 5217 pass / 0 fail (was 5027 / 1 pre-existing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address 5 codex pre-merge findings (dim passthrough + URL routing + MiniMax host)

Codex adversarial review during /ship caught five real production bugs.
All five fixed with regression test coverage.

1. **dimsProviderOptions on openai-compatible** (src/core/ai/dims.ts):
   text-embedding-3-* (Azure), text-embedding-v3 (DashScope), and
   embedding-3 (Zhipu) now thread `dimensions` to the wire. Without this,
   Azure-default 3072d hard-fails a 1536d brain on first embed; DashScope
   and Zhipu Matryoshka requests silently get the provider's default size
   instead of what the user asked for. New tests in
   recipe-azure-openai/dashscope/zhipu pin the contract.

2. **`gbrain init --embedding-model llama-server:foo` verbose path**
   (src/commands/init.ts): now refuses without `--embedding-dimensions`
   for user_provided_models recipes. Pre-fix, the shorthand `--model`
   path was guarded but the verbose `--embedding-model` path fell through
   to configureGateway's 1536d default and silently created the wrong-
   width schema; failure surfaced only at first real embed.

3. **MiniMax host correction** (src/core/ai/recipes/minimax.ts):
   `api.minimax.chat/v1` → `api.minimaxi.com/v1` matches MiniMax's
   current OpenAI-compatible docs. Default-config users would have hit
   the wrong endpoint before auth or model selection mattered.

4. **`LLAMA_SERVER_BASE_URL` reaches the gateway** (src/cli.ts:
   buildGatewayConfig): env-set local-server URLs (LLAMA_SERVER_BASE_URL,
   OLLAMA_BASE_URL, LMSTUDIO_BASE_URL, LITELLM_BASE_URL) now thread into
   `cfg.base_urls` so embed traffic hits the configured port. Pre-fix,
   the probe would succeed against a custom port while real embed calls
   went to localhost:8080. Caller-supplied `cfg.provider_base_urls` still
   wins over env.

5. **Recipe.probe(baseURL?) accepts the resolved URL** (src/core/ai/types.ts,
   src/core/ai/probes.ts, src/core/ai/recipes/llama-server.ts): when the
   user configures `provider_base_urls.llama-server` in config but no env
   var is set, the probe and gateway no longer disagree. Callers with cfg
   pass the resolved URL; legacy callers fall back to env / recipe default.

CHANGELOG updated; llms-full.txt regenerated.

Tests: bun run test — 5220/5220 pass / 0 fail (was 5217 / 0; +3 new
codex-finding regression tests).

Pre-merge codex adversarial: ran during /ship Step 11 against the v0.32
diff. All 5 findings addressed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): isolate v0.32 no-batch-cap test from mock.module leak (closes 19 CI fails)

Three CI test-isolation fixes uncovered by yesterday's CI run on PR #810:

1. **`scripts/test-shard.sh` excludes `*.serial.test.ts`** (was running them
   in parallel shards). Without this, serial files race with non-serial
   files in the CI shard process. Mirrors `scripts/run-unit-shard.sh`'s
   exclusion set; 1-line `find` filter.

2. **`scripts/run-serial-tests.sh` runs each serial file in its own bun
   process**. Pre-fix, all serial files ran in ONE bun process with
   `--max-concurrency=1` — that limits intra-file concurrency but does
   NOT prevent module-registry leakage across files. When
   `eval-takes-quality-runner.serial.test.ts` does
   `mock.module('../src/core/ai/gateway.ts', () => ({chat, configureGateway}))`
   (a partial mock missing `resetGateway`, `defaultResolveAuth`, etc.),
   the next file in the same process gets the partial mock on import and
   `import { resetGateway }` fails with "Export named 'resetGateway' not
   found." Per-file processes give true isolation; cost is ~100ms × N
   files (negligible vs CI walltime).

3. **`test/ai/no-batch-cap-suppression.test.ts` → `.serial.test.ts`**.
   The test mutates `console.warn` globally (mock spy). When other tests
   in the same shard process load `src/core/ai/gateway.ts` and call
   `configureGateway()` first, they populate the module-scoped
   `_warnedRecipes` Set; the test's `resetGateway()` clears it but races
   if other gateway-touching code runs concurrently in the same process.
   Renaming to `.serial.test.ts` quarantines it via fix #1 + #2.

4. **CI workflow gains a serial-tests step on shard 1**. Pre-fix, shard 1
   ran `bun run verify` + the parallel shard, but no shard ran
   `*.serial.test.ts` files. After fix #1 excludes them from shards, they
   need explicit invocation. New step:
   `bash scripts/run-serial-tests.sh` (shard 1 only).

Tests: bun run test — 5220 / 0 fail (matches local pre-CI run; was
showing 19 fails on CI for PR #810 due to fixes #1-#3 missing).

Failure analysis from .context/attachments/test__2__75236697976.log:
- 18 multimodal failures: caused by mock.module leak from
  eval-takes-quality-runner.serial.test.ts being run alongside
  voyage-multimodal.test.ts in the same parallel shard process. After
  fix #1 + fix #3, eval-takes-quality only runs in serial pass; after
  fix #2, its mock.module doesn't leak to subsequent serial files.
- 1 no-batch-cap failure: same root cause; fix #3 quarantines it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: SiyaoZheng <noreply@github.com>
Co-authored-by: cacity <20351699+cacity@users.noreply.github.com>
Co-authored-by: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-authored-by: JamesJZhang <32652444+JamesJZhang@users.noreply.github.com>
 2026-05-10 20:50:40 -07:00
Garry TanClaude Opus 4.7garrytan-agents
 		29961811a4 v0.31.12 fix: canonical Anthropic model IDs + tier routing surface + gbrain models CLI (#844) 
* fix: canonical Anthropic model IDs + reverse alias + Opus 4.7 pricing

Replace claude-sonnet-4-6-20250929 with claude-sonnet-4-6 everywhere it
appears as a model ID. Starting with Claude 4.6, Anthropic API IDs are
dateless and pinned — the date suffix was carried forward from Sonnet 4.5
by mistake, producing a phantom ID that 404'd on every call.

Production impact in v0.31.6: isAvailable("chat") returned false in every
code path that loaded the recipe's model list, and extractFactsFromTurn
silently returned []. The headline real-time facts extraction feature
was a no-op on the happy path.

- gateway.ts:46 DEFAULT_CHAT_MODEL -> anthropic:claude-sonnet-4-6
- recipes/anthropic.ts: chat + expansion model lists drop date suffix;
  remove wrong-direction alias (claude-sonnet-4-6 -> -20250929);
  add reverse alias (-20250929 -> claude-sonnet-4-6) so stale user
  configs in models.dream.synthesize etc. keep working
- facts/extract.ts: routes through resolveModel; both fallbacks corrected
- anthropic-pricing.ts: Opus 4.7 corrected $15/$75 -> $5/$25 per
  Anthropic docs (the $15/$75 was Opus 4.0 pricing)
- cross-modal-eval/runner.ts: PRICING now reads from ANTHROPIC_PRICING
  for Anthropic models instead of duplicating the map (single source of
  truth — fixes the drift trap that motivated this whole patch)

Tests: cherry-pick PR #830's test/anthropic-model-ids.test.ts verbatim
(6 recipe-shape guardrails). Update gateway-chat tests to assert reverse
alias resolves correctly. Update budget-meter test for new Opus pricing.

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: model tier system + recipe-models merge + async reconfigure hook

Add 4-tier model routing (utility/reasoning/deep/subagent) so users can
swap defaults with one config key. Each tier maps to a class of work;
override globally via models.default or per-tier via models.tier.<tier>.

Codex flagged three real architecture issues in the v0.31.12 plan review;
this commit addresses each.

F3 — sync/async timing of configureGateway:
  - buildGatewayConfig stays synchronous (pre-engine-connect callers
    keep working)
  - New reconfigureGatewayWithEngine(engine) async function re-resolves
    expansion + chat defaults through resolveModel after engine.connect()
  - cli.ts wires the re-stamp into the post-connect path

F4/F5 — softening assertTouchpoint was too broad:
  - Earlier plan was to flip native-recipe validation from throw to warn,
    affecting gateway.chat AND gateway.expand AND gateway.embed
  - Instead: per-gateway-instance recipe-models merge. assertTouchpoint
    gets an optional extendedModels Set; when the user opted into a model
    via config, it bypasses the throw. Source-code typos still fail fast.
  - Existing contract test (test/ai/gateway-chat.test.ts:106) preserved

Tier defaults are TIER_DEFAULTS in model-config.ts. Resolution chain
inserts at step 5 (between models.default and env var). Each existing
resolveModel call site gains a tier: arg — think (deep), cycle/synthesize
(reasoning + utility for verdict), patterns/drift (reasoning), auto-think
(deep), facts/extract (reasoning).

Plus 10 new tests pinning tier precedence, subagent-tier fallback when
models.default is non-Anthropic, and the F6 alias-chain conflict case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: subagent runtime enforcement for non-Anthropic models (3 layers)

The subagent loop uses Anthropic's Messages API with prompt caching on
system + tools. OpenAI/Google have different shapes. Setting
models.default = openai:gpt-5.5 and routing the subagent there silently
breaks the loop.

Codex F1+F2+F13 in the v0.31.12 plan review pointed out that "warn at
doctor" wasn't enough — handlers/subagent.ts:148 still did
`const model = data.model ?? DEFAULT_MODEL` and called Anthropic directly,
so a job submitted with data.model = openai:gpt-5.5 bypassed any tier
logic and failed at runtime with a confusing provider error.

Three layers of enforcement, defense in depth:

Layer 1 (queue.ts:add) — submit-time guard. When name === 'subagent'
and data.model is set, validate the provider. Non-Anthropic rejects
before the job enters the queue.

Layer 2 (handlers/subagent.ts) — tier-resolution fallback. The handler
routes through resolveModel({ tier: 'subagent' }). If the chain resolves
to a non-Anthropic provider (via models.default or models.tier.subagent),
the resolver warns + falls back to TIER_DEFAULTS.subagent
(claude-sonnet-4-6).

Layer 3 (doctor.ts:checkSubagentProvider) — surfacing layer. Warns when
models.tier.subagent or models.default is explicitly set to a
non-Anthropic provider, with a paste-ready fix command. Lets users see
config drift before submitting a job.

Tests: 3 new cases in test/agent-cli.test.ts asserting the queue-level
guard rejects non-Anthropic data.model. Existing test/subagent-handler
suite still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: gbrain models CLI + doctor probe + silent-no-op regression test

New gbrain models CLI gives the agent and user visibility into routing.
Read mode prints the tier table, current overrides, per-task config,
and aliases with source-of-truth attribution per row. Doctor subcommand
fires a 1-token probe to each configured chat/expansion model and
classifies failures (model_not_found / auth / rate_limit / network /
unknown) so config-time invalid IDs surface without waiting for a
production call that silently degrades.

Per Codex F11 — no specific dollar cost claim in either the help text
or the CHANGELOG (providers have minimum-output billing and prompt-cache
rounding that vary). Probe is opt-in (gbrain doctor --probe-models),
never auto-runs. --skip=<provider> narrows the matrix for cost-sensitive
operators.

Per Codex F7+F8+F15 (the structural regression gap): new
test/facts-extract-silent-no-op.test.ts is THE regression test for the
bug class that motivated v0.31.12. Five cases including the smoking-gun:
when chat IS available, extractFactsFromTurn MUST actually call the chat
transport, not silently return []. Uses the gateway's
__setChatTransportForTests seam so it runs in every shard with no API key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.31.12)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document v0.31.12 model tier system + gbrain models CLI

Add CLAUDE.md Key Files annotations for the v0.31.12 work:
src/core/model-config.ts (tier system + isAnthropicProvider + TIER_DEFAULTS),
src/core/ai/model-resolver.ts (assertTouchpoint extendedModels arg),
src/core/ai/gateway.ts (reconfigureGatewayWithEngine + extended-models registry),
src/core/minions/queue.ts (subagent submit-time guard, layer 1 of 3),
src/commands/models.ts (new gbrain models CLI + doctor probe),
src/commands/doctor.ts (subagent_provider check, layer 3 of 3),
src/core/ai/recipes/anthropic.ts (canonical model IDs + reverse alias),
src/core/anthropic-pricing.ts (Opus 4.7 corrected to \$5/\$25).

Add CLAUDE.md commands section for gbrain models + gbrain models doctor
+ power-user config recipes. Add README.md command-table rows for the
same. Regenerate llms-full.txt so the bundled docs stay in sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: scrub --probe-models reference (flag not actually wired)

The v0.31.12 CHANGELOG and skills/conventions/model-routing.md both
referenced `gbrain doctor --probe-models` as an integrated probe entry
point. The flag was never implemented — only `gbrain models doctor`
landed as the probe surface. Caught by /document-release subagent.

Drop the references rather than wire an untested flag at the last minute.
The probe is reachable via `gbrain models doctor`; users who want it
in doctor's output run that command separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 20:06:31 -07:00
Garry TanandClaude Opus 4.7 89ae720959 v0.31.0 feat: hot memory — facts hook + recall CLI + MCP _meta + consolidate phase (#785) 
* v0.31 feat(migrate): facts hot memory schema (migration v40)

Phase 1 of v0.31 hot-memory.

- New facts table with source_id (TEXT FK to sources, per-source isolation),
  kind CHECK (event/preference/commitment/belief/fact), visibility CHECK
  (private/world for takes-style ACL parity), valid_from/valid_until/
  expired_at/superseded_by for temporal + supersession audit, and
  consolidated_at/consolidated_into pointing at takes(id) for the dream-
  cycle hot→cold bridge.
- Embedding column dim resolved at migration time from
  config.embedding_dimensions so non-OpenAI brains (Voyage etc) work
  out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR
  with stderr warn on older versions. Matching opclass per column type
  (halfvec_cosine_ops vs vector_cosine_ops).
- 5 partial indexes leading on source_id so every read uses the trust
  boundary as part of the index, not a callback. HNSW partial index
  excludes expired/null rows so footprint stays proportional to active
  fact count.
- RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite
  no-op).
- v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts
  schema version >= 40 + facts table presence; runner owns ledger.

All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms
table + indexes + CHECK constraints + ON DELETE CASCADE all behave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore(version): bump VERSION + package.json to 0.31.0

Phase 1 closer. CHANGELOG entry written when Phase 7 lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(engine): facts hot memory engine API (Phase 2)

Phase 2 of v0.31 hot-memory.

Adds 8 facts methods to BrainEngine implemented on both PGLite and
Postgres engines:

- insertFact(input, ctx) — INSERT with optional supersedeId; expires the
  named row in the same transaction. Per-entity advisory lock on Postgres
  (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' ||
  entity_slug, 0))`) for the dedup window. PGLite is single-process so
  the lock is a no-op.
- expireFact(id, opts) — sets expired_at + optional superseded_by.
  Idempotent-as-false (already-expired returns false).
- listFactsByEntity / listFactsSince / listFactsBySession — list surfaces
  with FactListOpts filters (activeOnly, kinds, visibility, limit/offset).
  Every query starts WHERE source_id = $X so the trust boundary is part
  of the index path.
- listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL
  + superseded_by IS NOT NULL.
- findCandidateDuplicates(source_id, entity_slug, factText, k) —
  entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding-
  cosine ordering when caller supplies an embedding, recency fallback
  otherwise. Bounds the contradiction-classifier blast radius.
- consolidateFact(id, takeId) — sets consolidated_at + consolidated_into.
  Never DELETE; facts stay as audit trail for the resulting take.
- getFactsHealth(source_id) — per-source counters consumed by `gbrain
  doctor` facts_health check.

Public types in engine.ts: FactKind (5-value union), FactVisibility,
FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth.

PGLite + Postgres helpers: rowToFact / rowToFactPg parse the
text-format pgvector embedding back into Float32Array; toPgVectorLiteral
encodes for the supersede-path INSERT (postgres-js can't bind Float32Array
directly to a vector column without an explicit literal cast).

Smoke test confirms every method end-to-end on PGLite. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(facts): extraction code path (Phase 3)

Phase 3 of v0.31 hot-memory.

Five new modules under src/core/facts/ + src/core/entities/:

- src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now)
  applies confidence × exp(-age/halflife) with per-kind halflife table
  (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d).
  Returns 0 for expired or past-valid_until rows. Single source of truth
  consumed by recall, supersession audit, facts_health, and the MCP _meta
  injector (eD8 DRY).

- src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default,
  drop-oldest on overflow with counter. Per-session in-flight=1 serializes
  burst chat. AbortSignal threading from server SIGTERM (mirrors minion
  worker pattern per eD7): 5s grace for in-flight, then drop pending with
  counter. getFactsQueue() process-singleton; __resetFactsQueueForTests
  for hermetic tests.

- src/core/facts/classify.ts — contradiction classifier with cosine
  fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure
  fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine
  helper exported. JSON-strict output with 4-strategy parse fallback;
  refusal stop-reason maps to fallback path. Caller-provided abort
  signal propagated to the gateway chat call.

- src/core/facts/extract.ts — Haiku turn-extractor. Reuses
  INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN
  (turn_text) AND on the way OUT (each fact). Tight system prompt with
  5-kind taxonomy, 0..1 confidence scoring, entity slug or display name.
  Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics).
  Synchronous embedOne() per fact via the gateway so classifier paths
  have embeddings available; AbortError re-thrown explicitly so SIGTERM
  during embed never writes a NULL-embedding row meant to be cancelled
  (eE8 distinction).

- src/core/entities/resolve.ts — slug canonicalization shared by
  signal-detector AND facts. Resolution order: exact slug match →
  pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify
  fallback. slugify exported standalone for tests + callers that want
  the floor.

Smoke tests confirm decay table, cosine math, slugify rules, queue
drop-oldest under overflow, and shutdown grace + drop-pending semantics.
Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4)

Phase 4 of v0.31 hot-memory.

Three new MCP ops on the contract-first surface:

- `extract_facts` (write scope, localOnly:false): extracts facts from a
  conversation turn via the Haiku extractor, runs the cosine fast-path
  dedup, INSERTs into per-source hot memory. Returns counts +
  fact_ids[]. Skips on is_dream_generated:true (anti-loop).
- `recall` (read scope): query the per-source hot memory by
  entity / since / session / supersessions / grep filter. Visibility-
  aware: remote callers see visibility='world' rows only (takes-style
  ACL parity, eD21). Returns most-recent first; pagination via limit.
- `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error
  on unknown id; uses the new 'fact_not_found' ErrorCode.

ErrorCode union opened (eD6 / eE7): TS forward-compat via the
`(string & {})` autocomplete-friendly hack so downstream consumers
(gbrain-evals etc) don't break their typecheck on every new code.
Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'.

OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER
per schema reality). Resolved once in buildOperationContext from
DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or
'default'; HTTP MCP reads it from the per-token sources scope (eE3).

ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher
calls a configurable metaHook AFTER op.handler succeeds, wrapped in
its own try/catch so a DB blip degrades to no-_meta rather than
flipping the whole tool call to error (eE4).

New module src/core/facts/meta-hook.ts:
- getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory
  payload. Cache key (source_id, session_id, hash(takesHoldersAllowList
  sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies:
  remote → world only; local → all. Top-K=10 ranked by effective
  confidence (decay). Skips injection on recall/extract_facts/forget_fact
  themselves. bumpHotMemoryCache() invalidates per (source_id,
  session_id) on extraction event.

D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED
to call dispatchToolCall. HTTP MCP now inherits source_id, _meta
injection, error envelope unification, and OperationContext shape from
the same code path stdio uses. Scope check + mcp_request_log + SSE
broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher
returns ToolResult and the HTTP handler reads isError + content + _meta
to fan into the audit + broadcast paths.

put_page compliance backstop (D23): when a conversation-shape page is
written (note/meeting/slack/email/calendar-event/source/writing) with
a substantive body (>=80 chars) on a non-subagent slug AND no
dream_generated:true marker, fire-and-forget enqueue an extraction job
into the bounded queue. Never blocks the put_page response. Skipped
reasons (no_parsed_page / subagent_namespace / dream_generated /
kind:* / too_short / queue_shutdown / backstop_error) are stable
strings consumed by tests.

`gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts):
- recall <entity> | --since DUR | --session ID | --today (markdown
  with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions |
  --include-expired | --as-context (prompt-injection-ready) | --json
- forget <fact-id> shorthand for expireFact

Wired into src/cli.ts dispatch table next to takes / think.

Smoke tests confirm: dispatch surfaces (extract_facts → ops →
listFactsByEntity), forget_fact + idempotent re-call, _meta visibility
filter (remote sees world only, local sees all), CLI markdown render
with kind icons + age strings + decayed confidence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5)

Phase 5 of v0.31 hot-memory.

New 10th cycle phase `consolidate` between `patterns` and `embed`:

- src/core/cycle.ts:
  * CyclePhase union extended with 'consolidate'
  * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh
    after patterns; embed runs after so the new takes get embedded
    same-cycle)
  * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts)
  * CycleReport.totals gains facts_consolidated + consolidate_takes_written
  * runCycle dispatches the new phase via dynamic import

- src/core/cycle/phases/consolidate.ts (new):
  * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated
    facts) >= 3 (uses idx_facts_unconsolidated partial index)
  * Skips buckets where the OLDEST fact is < 24h old (gives signal time
    to settle before locking it into cold memory)
  * Greedy cosine clustering at threshold 0.85; head-element centroid
    keeps it deterministic + cheap. Singletons (no embedding) stay
    unconsolidated this cycle.
  * For each cluster size >= 2: picks the highest-confidence fact's text
    as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet
    synthesis pass). avg confidence → take weight, earliest valid_from →
    take since_date, concatenated source_sessions → take.source.
  * Resolves entity_slug → page_id via pages.slug (per source). Skips
    cluster if page is missing in this source — no auto-page-creation
    in v0.31.
  * INSERT into takes(kind='fact', holder='self') with row_num =
    MAX(existing) + 1.
  * UPDATE contributing facts: consolidated_at = now() +
    consolidated_into = takes.id. NEVER DELETE — facts are the audit
    trail for the resulting take.
  * dryRun honored: pretends the writes happened; counters still tick
    so operators can preview load before the first real run.
  * yieldDuringPhase keepalive between buckets so the Minions worker
    job lock + cycle-lock TTL don't drift on long runs.

Smoke test on PGLite confirms: 4 unconsolidated facts → clustered
(cosine 1.0 since same vector) → 1 take row created → all 4 facts
marked consolidated_into. runCycle({phases:['consolidate']}) wires
through to the report totals. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: 18 facts test files (Phase 6)

Phase 6 of v0.31 hot-memory: comprehensive coverage across the new
substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully
without DATABASE_URL).

Unit tests (PGLite in-memory, no DATABASE_URL):
- test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind,
  effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife
  (~1/e²) / expired returns 0 / valid_until past returns 0 /
  preference-vs-event slower decay / belief-vs-commitment crossover.
- test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest
  on overflow, per-session in-flight=1 serializes, different sessions
  parallelize, failed jobs counter, shutdown grace + drop_pending +
  external AbortController triggers shutdown.
- test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases,
  empty candidates → independent, cheap fast-path ≥0.95 → duplicate
  no LLM, threshold-configurable cosine_fallback path.
- test/facts-engine.test.ts (13 cases) — every BrainEngine fact method
  end-to-end: insertFact (insert/supersede), expireFact idempotency,
  list*, findCandidateDuplicates entity-prefiltered + k cap + cosine
  ordering, consolidateFact never DELETE, getFactsHealth shape +
  total_today ⊆ total_week.
- test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation
  on every list method + CASCADE delete on sources.
- test/facts-visibility.test.ts (6 cases) — visibility column private/
  world; remote=true filters to world-only via dispatchToolCall;
  remote=false sees all.
- test/facts-canonicality.test.ts (10 cases) — slugify rules including
  NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug
  match, fallback to slugify when no fuzzy match.
- test/facts-extract.test.ts (4 cases) — empty turn returns [], dream-
  generated short-circuit, graceful no-API-key return.
- test/facts-backstop-gating.test.ts (5 cases) — put_page backstop:
  too_short, subagent_namespace, dream_generated, eligible note path,
  non-eligible kind:guide.
- test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both
  respect dream_generated:true marker.
- test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape
  pinned for downstream consumers.
- test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts
  write-scope, recall read-scope, forget_fact write-scope, forget_fact
  fact_not_found error code, extract_facts no-API-key zero counts.
- test/facts-context-injection.serial.test.ts (6 cases) — _meta
  injection on success, world-only filter under remote=true, anti-loop
  on facts ops themselves, best-effort degrade on hook error,
  cache-key includes allow-list hash.
- test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation
  Test as primary ship gate, plus expired hidden-by-default contract.
- test/facts-recall-render.test.ts (3 cases) — --today markdown render
  with all 5 kind icons, --json shape with effective_confidence,
  --as-context emits comment-wrapped block.
- test/facts-migration-dim.test.ts (4 cases) — embedding column type
  is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured
  embedding_dimensions, HNSW opclass agrees with column type, idempotent
  re-init.
- test/cycle-consolidate.test.ts (5 cases) — below-count + below-age
  thresholds skip, happy path 4 facts → 1 take + all consolidated never
  DELETE, dryRun honored, missing page → bucket skipped.

E2E tests (skip gracefully on DATABASE_URL unset; required gates by
CLAUDE.md test policy):
- test/e2e/facts-separation-postgres.test.ts — Postgres parity for the
  ship gate.
- test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG
  + CASCADE delete.
- test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip.
- test/e2e/facts-context-injection-postgres.test.ts — _meta injection
  end-to-end on PG.
- test/e2e/facts-recall-render.test.ts — recall --today markdown on PG.
- test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport
  inherits _meta + sourceId + scope correctness via dispatchToolCall.

Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips
combining marks (U+0300..U+036F) before hyphenating non-alphanumerics,
so "Crème" → "creme", not "cre-me-".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7)

Phase 7 of v0.31 hot-memory.

- src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper
  reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to
  'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set
  facts.extraction_enabled false` to kill extraction across the brain
  without binary downgrade.
- extract_facts MCP op short-circuits with zero-counts envelope + a
  'skipped: extraction_disabled' field when the flag is off (clean
  success, not permission_denied).
- put_page facts backstop respects the same flag — eligibility check now
  returns 'extraction_disabled' as the skipped reason.
- src/commands/doctor.ts: new facts_health check (runs after queue_health,
  before index_audit). Probes for the facts table existence (post-v40
  guard), then surfaces total_active / total_today / total_week /
  total_consolidated + top-3 entities for the default source. Pre-v0.31
  brains report "facts table not present (pre-v0.31 brain or migration
  pending)".
- CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice.
  Headline + numbers-table + what-it-ships + itemized changes + "To take
  advantage of v0.31" upgrade block + out-of-scope. Honest about the
  HALFVEC + serve-http refactor + ErrorCode-open-union complications.
- README.md: cycle phase list updated 8 → 10 (consolidate + purge). New
  "v0.31 Hot Memory" command block under Commands with recall + forget
  variants, kind icons, --as-context surface for headless agents.

Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor
check ride-along. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE

The inline column-level FK declaration on facts.source_id worked on
PGLite but silently got dropped on Postgres in the v0.31 e2e run —
the migration handler ran via postgres-js's `unsafe()` multi-statement
path and the resulting facts table came back without the
`facts_source_id_fkey` constraint. Same psql input run directly
against the same database produced the FK; the difference was the
unsafe() pipeline, not the SQL itself.

Splitting the FK into a separate ALTER TABLE inside a DO block makes
the constraint declaration explicit and idempotent: the named
constraint either exists or it doesn't, the ALTER is a no-op on
re-runs, and the failure mode is loud rather than silently leaving
a CASCADE-less foreign key behind.

Without this fix, deleting a source row leaves orphaned facts rows
(test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources-
delete case caught it). With this fix the constraint is in place,
the cascade fires, and both PG + PGLite e2e suites stay green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: update phase-count assertions for the new consolidate phase

Three e2e/unit tests pinned the cycle phase count or order, all now
updated to reflect v0.31's 10-phase cycle:

- test/e2e/dream-cycle-eight-phase-pglite.test.ts:
  describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES
  expectation extended to include 'consolidate' (between patterns +
  embed) and 'purge' (the v0.26.5 addition that was already in
  ALL_PHASES but missing from the test's assertion list). totals
  match adds the new facts_consolidated + consolidate_takes_written
  fields plus the pre-existing purged_sources_count + purged_pages_count
  that should have been added when v0.26.5 landed.

- test/e2e/cycle.test.ts: dry-run full cycle now expects
  report.phases.length === 10 (was 9).

- test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full
  cycle phases.length both updated 9 → 10. Comments call out the
  v0.31 addition lineage so the next person to add a phase sees the
  precedent.

These are mechanical assertion bumps. The tests pass against the
updated assertions on PGLite and Postgres.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): truncate facts table between e2e describe blocks

setupDB() truncates ALL_TABLES between every describe block's
beforeAll() hook. The list missed the new v0.31 facts table, so
facts seeded by an earlier describe block leaked into Garry's
Separation Test on Postgres — listFactsByEntity('travel') returned
2 rows instead of 1 because a prior facts-context-injection test had
also seeded a 'travel' fact.

Adding 'facts' to the truncate list (before 'pages' to respect FK
ordering) makes every describe-block start from an empty facts table.

Pinned by re-running the e2e file ordering that originally caught it
(facts-recall-render → cross-source-isolation → serve-http-meta →
context-injection → separation-postgres → facts-forget) — 13 pass /
0 fail after the fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: meta-hook cache + Postgres consolidate phase coverage

Two net-new test files filling real coverage gaps the earlier sweep missed:

- test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache
  contract that the dispatcher relies on. 30s TTL hit path, post-bump
  fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache
  warm — closes the cross-source leak risk codex F5 originally surfaced
  on the recall payload), facts-self ops skip injection (anti-loop on
  recall / extract_facts / forget_fact), distinct allow-lists produce
  distinct cache entries.

- test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres
  parity for the dream-cycle consolidate phase. Mirrors the PGLite
  unit test but exercises the real postgres-engine codepaths: sql.begin
  transactions, advisory locks on insertFact's entity-slug dedup window,
  unsafe('::vector') casts on findCandidateDuplicates ordering,
  addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take +
  all consolidated_into set), age-threshold skip, dry-run no-write.

All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the
consolidate phase (was only PGLite-tested) and pins meta-cache
invariants the dispatcher depends on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: thread auth + sourceId, JSON-shape every error envelope

Three bugs surfaced during the full e2e sweep that all trace back to my
v0.31 dispatch refactor (D12/eE1) silently dropping auth threading +
non-OperationError exceptions emitting plain strings:

1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call
   dispatchToolCall meant auth had to come through DispatchOpts, but the
   field didn't exist yet. Every HTTP whoami call returned
   `unknown_transport` because ctx.auth was undefined. Added `auth?:
   AuthInfo` to DispatchOpts, plumbed it through buildOperationContext,
   and updated serve-http.ts:816 to pass `auth: authInfo` alongside
   sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e
   `whoami reports oauth transport + sources_admin scope`.

2. **Non-OperationError exceptions emitted plain strings, not JSON.**
   The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope
   `{error, message}`; my dispatch refactor missed the unknown-tool +
   uncaught-throw paths and emitted `Error: ${msg}` text content. Every
   caller that did `JSON.parse(content)` (sources-remote-mcp callMcp
   helper at line 104) crashed with `Unexpected identifier "Error"`.
   Both error paths in dispatchToolCall now return JSON-shaped content
   matching the OperationError pattern.

3. **Files→sources FK silently lost on rewound bootstrap path.**
   test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by
   `DROP TABLE IF EXISTS sources CASCADE` which removes
   files_source_id_fkey while leaving files.source_id intact. The v23
   migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ...
   REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column
   exists, so the FK never came back on upgrade — and any sources-remove
   afterward stopped cascading to files. Added a defensive
   `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT`
   block inside v23's handler. Pinned by `multi-source — cascade delete
   covers every dependent row` after running postgres-bootstrap.

Plus: src/core/preferences.ts now honors GBRAIN_HOME for
`~/.gbrain/migrations/completed.jsonl`. Without this, the doctor
exits-0 mechanical test inherits the developer machine's stale
partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1
prior dev work) and surfaces them as the [FAIL] minions_migration check.
GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore: scrub personal references from public artifacts

Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace
personally-coded references in v0.31 artifacts with neutral examples:

- CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to
  "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM,
  flying to Tokyo" narrative.
- src/commands/migrations/v0_31_0.ts feature pitch: same scrub.
- test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts:
  rename describe blocks; replace specific topic-NNNN session ids with
  session-A / session-B; replace personal sample fact with
  "sample event Tuesday".
- src/core/facts/extract.ts extractor system prompt example slugs:
  people/sam-altman → people/alice-example; companies/anthropic → companies/acme.
- src/core/entities/resolve.ts comment: Sam Altman → Alice Example.
- All v0.31 test fixtures: people/sam → people/alice-example,
  Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder.
  Test names referencing real-world entities replaced with neutral slugs.

Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19,
v0.21+ entries) are untouched — that's a separate scope from this v0.31
ship.

Plus: the truncate fix for the Bun-script-induced syntax error in
test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked
onto its closing brace by the bulk-add-timeouts script — repaired to a
clean function definition).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle

Two E2E tests still asserted the v0.31 pre-merge 10-phase shape
(consolidate inserted, but recompute_emotional_weight from v0.29 not yet
absorbed). With master's v0.29 work merged in, the cycle is now 11 phases:
lint → backlinks → sync → synthesize → extract → patterns →
recompute_emotional_weight → consolidate → embed → orphans → purge.

- test/e2e/cycle.test.ts: 10 → 11
- test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(merge): close brace between v44 and v45 migration objects

The v0.30.2 merge resolution stitched master's v40-v44 migrations onto
HEAD's v45 (facts hot memory) migration but lost the closing `},` between
v44 and v45. tsc caught it as TS1136 Property assignment expected at
migrate.ts:2188.

This is a one-line bracket fix; the rest of the merge resolution is
correct and tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture

Two unit-test failures surfaced after the v0.30.2 merge:

1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }`
   from earlier v0.31 development. The parity test enforces that every name
   in `positional` is a real param. Restored master's correct shape:
   `{ name: 'put', positional: ['slug'], stdin: 'content' }`.

2. test/apply-migrations.test.ts: the H9 regression tests pin the exact
   skippedFuture list. Adding v0.31.0 to the registry meant the list grew
   by one. Updated both `expect(...).toEqual([...])` assertions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt

CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's
recompute_emotional_weight already on master, consolidate is the 11th phase
(between recompute and embed). Schema migration is v45, not v40, after the
merge resolution renumbered it to clear master's v40-v44.

llms-full.txt regenerated to reflect the README's 11-phase dream-cycle
phrasing (the build-llms test enforces commit-time parity).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-09 16:57:47 -07:00
Garry TanandClaude Opus 4.7 410c6978a4 v0.30.2 feat: dream synthesize stops dropping fat transcripts (#754) 
* feat: classify Anthropic prompt-too-long as UnrecoverableError

The subagent handler now detects 400 "prompt is too long" responses
from the Anthropic SDK and rethrows as UnrecoverableError. The worker
already routes UnrecoverableError straight to `dead`, so doomed jobs
fail terminally on first attempt instead of stalling 3x with the same
oversized prompt.

isPromptTooLongError matches the production message verbatim
("prompt is too long: N tokens > N maximum"), case-insensitive, on
both the outer message and inner error.message paths. Defensive
secondary match for status=400 + invalid_request_error/request_too_large
with the words "too long"/"exceed"/"maximum".

9 unit cases pin the detection: production wording, case folding,
nested SDK shape, defensive 400 paths, unrelated 400s, transient
errors, null/empty inputs.

* feat: model-aware chunking + slug-rewrite for dream synthesize

The synthesize phase now chunks oversized transcripts at paragraph
boundaries instead of submitting one giant prompt that 400s on
Anthropic. Closes the v0.30 dream-cycle queue clog where 1.7M-token
transcripts dead-lettered after 3 stalls and re-discovered every
cycle.

D1: per-chunk budget = floor(model_context_tokens × 0.9 × 3.5).
MODEL_CONTEXT_TOKENS keys on resolved Anthropic ids (Opus 4.7 = 1M,
Sonnet 4.6 = 200K, Haiku = 200K). Non-Anthropic models fall back to
180K-token safe default with a once-per-process stderr warning.
dream.synthesize.max_prompt_tokens overrides the model lookup
(token-shaped, name from PR #748, floor 100K).

D5: on max_chunks_per_transcript cap hit, log + skip; do NOT write to
dream_verdicts. Closes the cache-poisoning class — next cycle
re-attempts under whatever budget is then current.

D6: orchestrator-side deterministic slug rewrite, zero Sonnet trust.
collectChildPutPageSlugs raw-fetches every (job_id, slug) pair (no
SELECT DISTINCT — that erased the collision evidence the audit
claimed to detect) and rewrites bare-hash6 slugs to <hash6>-c<idx>
for chunked children.

D8: pre-fan-out lookup of completed legacy `dream:synth:<filePath>:
<hash16>` jobs. Transcripts already synthesized under the
single-chunk shape skip submission with `already_synthesized_legacy_
single_chunk` instead of resubmitting under chunked keys.

D9: hash-deterministic chunk boundaries. The 3-tier ladder lifted
from PR #748 (## Topic: > --- > nearest \\n) is fed a back-half
search-window offset derived from contentHash. Same content always
chunks identically across runs; chunk N of a previously-failed
transcript produces byte-identical content on retry.

D10: 24-chunk default cap, operator-configurable via
dream.synthesize.max_chunks_per_transcript.

18 unit cases pin the chunker (boundary ladder, hash determinism,
hard fallback, slug rewrite all 7 shapes). 4 PGLite E2E cases pin
fan-out shape (single-chunk legacy key parity, multi-chunk chunked
key shape) + skip paths (D5 cap hit no verdict-cache write, D8
legacy-key skip).

Credits PR #748 (Wintermute) for the boundary ladder, config key
naming, and 3.5 chars/token estimator. This branch supersedes #748
with the structural safeguards (model-aware budget, terminal-error
classify, slug rewrite, hash-determinism, doctor surfacing).

* feat: surface dead-lettered prompt_too_long jobs in doctor queue_health

queue_health gains a 4th subcheck counting dead `subagent` jobs in
the last 24h whose error_text starts with `prompt_too_long:`. When
present, prints a fix hint pointing at
`gbrain dream --phase synthesize --dry-run --json` to identify the
fat transcripts and naming the two operator escape hatches
(`dream.synthesize.max_prompt_tokens` for budget tuning,
larger-context model for capacity).

Operators now see the chunking failure mode without grepping
minion_jobs by hand.

* chore: bump version and changelog (v0.30.2)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README + CLAUDE.md for v0.30.2

- README dream help: 8-phase → 9-phase, mention v0.30.2 chunking + config keys
- CLAUDE.md synthesize.ts: chunker + per-chunk idempotency + D6 slug rewrite + D7 scope + D8 legacy-key
- CLAUDE.md subagent.ts: prompt_too_long terminal classification
- CLAUDE.md doctor.ts: queue_health subcheck 4 (dead-lettered prompt_too_long)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: regenerate llms-full.txt after v0.30.2 CLAUDE.md updates

The docs/ pass extended three Key Files entries in CLAUDE.md
(synthesize.ts, subagent.ts, doctor.ts). The auto-derived
llms-full.txt bundle picks up those CLAUDE.md changes via
build-llms; the build-llms test caught the drift in CI.

Generated by: bun run build:llms

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 17:07:51 -07:00
Garry TanandClaude Opus 4.7 8392d434a9 v0.29.2 feat: thin-client mode (gbrain init --mcp-only + gbrain remote ping/doctor + topologies) (#732) 
* feat(config): add remote_mcp field + isThinClient() helper

Adds a top-level optional remote_mcp config block to GBrainConfig
(issuer_url, mcp_url, oauth_client_id, oauth_client_secret) for
thin-client installs that consume a remote `gbrain serve --http` over
MCP instead of running a local engine.

isThinClient(config) returns true when remote_mcp is set; used by the
CLI dispatch guard, doctor branch, and init re-run guard. The engine
field stays as today (postgres|pglite); thin-client mode is a separate
config field, NOT an engine kind extension (codex outside-voice review
flagged the engine='remote' extension as overreach).

GBRAIN_REMOTE_CLIENT_SECRET env var overrides the config-file value at
load time so the secret can stay out of disk for headless agents.

Foundation commit for multi-topology v1; no behavior change yet.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(probe): outbound OAuth + MCP smoke probes

Adds three pure async functions over the standard fetch API:
  - discoverOAuth(issuerUrl): GET /.well-known/oauth-authorization-server
  - mintClientCredentialsToken(tokenEndpoint, id, secret): POST /token
  - smokeTestMcp(mcpUrl, accessToken): POST /mcp initialize

Discriminated 'ok=true' / 'ok=false + reason' return shapes so callers
render error messages consistently. No SDK dependency to keep init's
setup-flow scope tight; Lane B's mcp-client.ts will pull in the
official @modelcontextprotocol/sdk Client for full session semantics.

Used by both 'gbrain init --mcp-only' (Lane A's setup smoke) and
runRemoteDoctor (Lane A's thin-client doctor checks).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(init): --mcp-only branch + re-run guard

Adds 'gbrain init --mcp-only' for thin-client setup. Required flags
(or env vars):
  --issuer-url     OAuth root (e.g. https://host:3001)
  --mcp-url        MCP tool dispatch path (e.g. https://host:3001/mcp)
  --oauth-client-id, --oauth-client-secret

Pre-flight runs three smoke probes (discovery, token round-trip, MCP
initialize) BEFORE writing the config — fail-fast on bad URL beats
fail-late on bad credentials. On success, writes ~/.gbrain/config.json
with remote_mcp set and NO local DB created.

Re-run guard (A8): when ~/.gbrain/config.json already has remote_mcp,
'gbrain init' (any flag set) refuses without --force. Catches the
scripted-setup-loop friction from the user-reported scenario where
re-running setup-gbrain on a thin-client machine kept trying to
re-create a local DB.

Two URLs in config (issuer + mcp) instead of one because OAuth
discovery + /token live at the issuer root while tool dispatch is at
/mcp — they compose from a common base in practice but reverse-proxy
setups need them explicit (codex review #2).

Tests: 15 cases covering happy path, env-var-supplied secret stays
out of disk, all four required-flag missing-error paths, three
smoke-failure paths, network-unreachable path, and the four re-run
guard variants (default/--pglite/--mcp-only without --force / with
--force). Uses async Bun.spawn (NOT execFileSync) — sync exec
deadlocks against in-process HTTP fixtures because the parent's
event loop can't accept connections while sync-blocked on a child.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): runRemoteDoctor for thin-client mode

Replaces every DB-bound check from runDoctor() with a tighter set
scoped to 'is the remote MCP we configured actually reachable?'.
Five checks:
  - config_integrity (URL fields well-formed)
  - oauth_credentials (secret resolvable from env or config file)
  - oauth_discovery (GET /.well-known/oauth-authorization-server)
  - oauth_token (POST /token client_credentials)
  - mcp_smoke (POST /mcp initialize)

Output shape matches the local doctor's Check surface so JSON
consumers can union the two without conditional logic. schema_version
is 2 (matches local doctor).

collectRemoteDoctorReport() is the pure data collector;
runRemoteDoctor() is the print/exit wrapper. Tests pin the data
collector so we don't have to intercept stdout / process.exit.

Tests: 12 cases over a tiny in-process HTTP fixture covering happy
path, every probe failure mode (404/parse/auth/network/server-error),
malformed-URL config integrity, missing-secret short-circuit, and
the env-var-overrides-config-file secret resolution. withEnv() helper
used for env mutations to satisfy the test-isolation lint.

Module is added but not yet wired into the CLI doctor branch; the
wiring lands in the next commit (cli dispatch guard + doctor routing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): thin-client dispatch guard + doctor routing

Adds a single canonical refusal at the top of handleCliOnly() for the
9 DB-bound commands when ~/.gbrain/config.json has remote_mcp set:
  sync, embed, extract, migrate, apply-migrations, repair-jsonb,
  orphans, integrity, serve

Single dispatch check (not 9 sprinkled assertLocalEngine calls per
codex review #1) — avoids the blast radius of letting commands enter
connectEngine before the check fires. Refused commands exit 1 with a
canonical error naming the remote mcp_url.

doctor branch routes to runRemoteDoctor when isThinClient(config)
returns true; falls through to the existing local-doctor flow
otherwise. Wires the module added in the previous commit into the
user-facing CLI surface.

Safe commands (init, auth, --version, --help, etc.) still work in
thin-client mode and are NOT in the refused set.

Tests: 14 cases — 9 refused commands × 1 each, 2 safe commands, 1
doctor-routing assertion (fingerprints the thin-client output by
'mode:"thin-client"' in JSON), 2 regression tests asserting local
config still passes through normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(topologies): multi-topology architecture guide + setup skill Phase A.5

New docs/architecture/topologies.md covering three deployment shapes:
  1. Single brain (today's default)
  2. Cross-machine thin client (consume a remote brain over MCP)
  3. Split-engine per-worktree (Conductor users with per-worktree
     code engines + shared remote artifacts brain)

Each topology gets an ASCII diagram, when-it-fits guidance, and
concrete setup recipes. Topology 3's alias-level routing footgun
(wrong alias = silent wrong-brain writes) is called out explicitly
per codex review #6.

Topology 3 needs zero gbrain code changes — GBRAIN_HOME already
overrides ~/.gbrain and 'gbrain serve --http --port N' already runs
on any port. gstack composes these primitives on its side.

skills/setup/SKILL.md gets Phase A.5 BEFORE the local-engine phases.
Asks the user which topology fits, walks thin-client setup through
'gbrain init --mcp-only', skips Phases B/C/C.5/H entirely for thin
clients (host's autopilot handles sync/extract/embed).

README.md gets a one-line link to the topology doc from the
Architecture section.

llms-full.txt regenerated to include the new doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): thin-client end-to-end skeleton

Spins up 'gbrain serve --http' against real Postgres, registers a
client with read,write,admin scope, runs 'gbrain init --mcp-only'
from a separate tempdir GBRAIN_HOME, exercises the canonical
thin-client flows:

  - init --mcp-only succeeds against the live host
  - doctor reports mode: thin-client + all checks green
  - sync is refused with the canonical thin-client error
  - re-running init refuses without --force

Tier B flows (gbrain remote ping / doctor) will be added alongside
their Lane B implementation. Skips when DATABASE_URL unset (matches
the e2e gate convention used across the suite).

Async Bun.spawn (NOT execFileSync) so the test event loop stays
responsive — execFileSync deadlocks against in-process HTTP fixtures
because the parent's event loop can't accept connections while
sync-blocked on a child process.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): doctorReportRemote core for thin-client + run_doctor op

Adds three new exports to src/commands/doctor.ts that the run_doctor MCP
op + gbrain remote doctor CLI both consume:

  - DoctorReport interface       schema_version=2 stable shape
  - computeDoctorReport(checks)  status + health_score math
  - doctorReportRemote(engine)   focused 5-check thin-client surface

doctorReportRemote runs:
  1. connection      (engine reachable + page count via getStats)
  2. schema_version  (engine.getConfig('version') vs LATEST_VERSION)
  3. brain_score     (the 5-component composite)
  4. sync_failures   (file-plane JSONL count from gbrainPath('sync-failures.jsonl'))
  5. queue_health    (Postgres-only: stalled active jobs > 1h)

Engine-agnostic: works on both Postgres and PGLite via engine.executeRaw +
engine.getConfig + engine.getHealth — no reliance on db.getConnection()
which is Postgres-only.

Deliberately a focused subset of the local doctor surface, NOT a full
mirror. Generalizing to lint/integrity/orphans is filed as follow-up
pending demand. Local doctor (runDoctor) is unchanged; operators on the
host machine still get the full check set.

schema_version=2 matches the local doctor's --json output schema, so JSON
consumers can union the two without conditional logic.

Tests: 11 unit cases against PGLite covering the 5-check happy path,
schema version reporting (latest), PGLite-specific queue_health
informational message, and the score+status math via computeDoctorReport.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(mcp-client): outbound HTTP MCP client over @modelcontextprotocol/sdk

New src/core/mcp-client.ts wraps the official SDK's Client +
StreamableHTTPClientTransport with OAuth client_credentials minting,
in-process token caching with expires_at, and refresh-on-401 retry.

Public surface:
  - callRemoteTool(config, toolName, args)   tool call w/ auto-refresh
  - unpackToolResult(res)                    parse content[0].text JSON
  - RemoteMcpError                           discriminated by `reason`

Token cache: module-level Map keyed by mcp_url. CLI processes are
short-lived; the cache amortizes when one invocation makes multiple
calls (gbrain remote ping submits then polls). Persisting to disk would
be a credential-on-disk surface for marginal benefit since /token
round-trip is sub-100ms.

401 retry: ONLY for mid-session token rotation (initial good token →
stale → 401). If the FIRST mint fails auth, surface immediately as
RemoteMcpError(auth) — retry won't help when credentials are wrong from
the start. If a fresh-mint-after-401 still 401s, surface as
RemoteMcpError(auth_after_refresh) which the CLI renders with a hint
pointing the operator at gbrain auth register-client.

Used by gbrain remote ping (submit_job + get_job poll) and gbrain
remote doctor (run_doctor). Test-only _clearMcpClientTokenCache export
for fixture isolation.

Tests: 13 unit cases over an in-process HTTP fixture mimicking gbrain
serve --http (OAuth discovery + /token + /mcp JSON-RPC handshake).
Covers happy path, token cache reuse + force-refresh, args passthrough,
config-error paths (no remote_mcp / no secret), token mint 401, network
unreachable, tool isError envelope, and unpackToolResult parse failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(operations): add run_doctor MCP op (admin scope, HTTP-reachable)

New op in src/core/operations.ts wraps doctorReportRemote() and returns
the structured DoctorReport JSON over MCP.

  scope:     'admin'       (system-state read; not for routine consumers)
  localOnly: false         (reachable over HTTP)
  mutating:  false         (safe to call repeatedly)
  params:    {}            (no caller arguments needed)

First read-only diagnostic op exposed over HTTP MCP. Used by gbrain
remote doctor — the matching client-side renderer lives in
src/commands/remote.ts.

Precedent: doctor only. Generalizing run_lint / run_integrity /
run_orphans to MCP is filed as follow-up work pending demand. Local
doctor stays unchanged; this op is the operator-friendly subset for
remote callers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(remote): gbrain remote ping + gbrain remote doctor

Two thin-client convenience commands that round-trip through the host's
HTTP MCP endpoint:

  - gbrain remote ping     submit_job(autopilot-cycle) → poll get_job →
                           exit when terminal. The "I just wrote markdown,
                           tell the host to re-index" affordance.
  - gbrain remote doctor   run_doctor MCP op → render the host's
                           DoctorReport → exit 0/1 based on status.

Both require a thin-client install (~/.gbrain/config.json with
remote_mcp). Local installs get a clear error pointing at the local
equivalents.

Polling backoff (ping): 1s × 30s, then 5s × 5min, then 10s. Default cap
15min, configurable via `--timeout`. Without backoff, a 5-min cycle
would burn 300 round-trips against the host's rate limiter.

Payload uses `data: {phases: [...]}`, NOT `params:` — the submit_job op
shape takes `data`. Codex review #8 catch.

NO `repo` arg passed to autopilot-cycle — uses the server's configured
brain repo. This sidesteps TODO #1144 (sync_brain repo-path validation
for caller-controlled paths) entirely.

src/cli.ts wires the `remote` subcommand into CLI_ONLY + the dispatch.
Help (`gbrain remote --help`) and unknown-subcommand handling included.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): thin-client Tier B + scope-mismatch regression

Extends the existing test/e2e/thin-client.test.ts with three new cases:

  1. gbrain remote doctor returns the host's DoctorReport — pins the
     run_doctor MCP op round-trip. Asserts schema_version=2, all 5
     check names present, connection + schema_version ok against a
     fresh host.
  2. gbrain remote ping triggers autopilot-cycle and returns terminal
     state — pins the submit_job → poll → terminal wire path. Accepts
     any terminal state (success / failed / dead / cancelled / timeout)
     because autopilot on an empty no-repo brain may fail-fast in the
     sync phase. What this test pins is the JSON shape (job_id present,
     state populated), NOT cycle success on a no-repo fixture.
  3. read+write client cannot call run_doctor — codex review #7
     regression guard. Registers a separate client with
     `--scopes "read write"` (no admin), runs `gbrain remote doctor`
     against it, asserts exit 1 with auth/auth_after_refresh/tool_error
     reason. Keeps the verification flow honest: the canonical setup
     MUST require admin scope.

`gbrain auth register-client` doesn't have --json, so the test parses
the human output for "Client ID:" and "Client Secret:" lines via a
helper.

Test-level timeout bumped 60s → 120s for the ping wait + auth/init
overhead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.29.2)

v0.29.2 ships thin-client mode: gbrain init --mcp-only, gbrain remote
ping/doctor, run_doctor MCP op, and the docs/architecture/topologies.md
deployment guide.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 22:56:22 -07:00
Garry TanandClaude Opus 4.7 bca993e09f v0.28.12 feat: LongMemEval benchmark harness (#606) 
* v0.28 schema: takes + synthesis_evidence (v31) + access_tokens.permissions (v32)

Migration v31 adds the takes table (typed/weighted/attributed claims) and
synthesis_evidence (provenance for `gbrain think` outputs). Page-scoped via
page_id FK (slug isn't unique alone in v0.18+ multi-source). HNSW partial
index on embedding for active rows. ON DELETE CASCADE on synthesis_evidence
so deleting a source take cascades the provenance row.

Migration v32 adds access_tokens.permissions JSONB with safe-default
backfill (`{"takes_holders":["world"]}`). Default keeps non-world holders
hidden from MCP-bound tokens until the operator explicitly grants access
via the v0.28 auth permissions CLI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 engine: addTakesBatch, listTakes, searchTakes/Vector, supersede, resolve, synthesis_evidence

Extends BrainEngine with the takes domain object. Both engines implement the
same surface; PGLite uses manual `$N` placeholders, Postgres uses postgres-js
unnest() — same shape as addLinksBatch and addTimelineEntriesBatch.

Methods:
- addTakesBatch (upsert via ON CONFLICT (page_id, row_num) DO UPDATE)
- listTakes (filter by holder/kind/active/resolved, takesHoldersAllowList
  for MCP-bound calls, sortBy weight/since_date/created_at)
- searchTakes / searchTakesVector (pg_trgm + cosine; honor allow-list)
- countStaleTakes / listStaleTakes (mirror countStaleChunks pattern;
  embedding column intentionally omitted from listStale payload)
- updateTake (mutable fields only; throws TAKE_ROW_NOT_FOUND)
- supersedeTake (transactional: insert new at next row_num, mark old
  active=false, set superseded_by; throws TAKE_RESOLVED_IMMUTABLE on
  resolved bets)
- resolveTake (sets resolved_*; throws TAKE_ALREADY_RESOLVED on re-resolve;
  resolution is immutable per Codex P1 #13 fold)
- addSynthesisEvidence (provenance persist; ON CONFLICT DO NOTHING)
- getTakeEmbeddings (parallel to getEmbeddingsByChunkIds)

Types live in src/core/engine.ts adjacent to LinkBatchInput. Page-scoped
via page_id (slug not unique in v0.18+ multi-source). PageType gains
'synthesis'. takeRowToTake mapper in utils.ts handles Date → ISO string
normalization.

Tests: test/takes-engine.test.ts — 16 cases against PGLite covering
upsert/list/filter/search happy paths, takesHoldersAllowList isolation,
the four invariant errors (TAKE_ROW_NOT_FOUND, TAKES_WEIGHT_CLAMPED,
TAKE_RESOLVED_IMMUTABLE, TAKE_ALREADY_RESOLVED), supersede flow, resolve
metadata round-trip, FK CASCADE on synthesis_evidence when source take
deletes. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 model-config: unified resolveModel with 6-tier precedence + alias resolution

Replaces every hardcoded `claude-*-X` and per-phase `dream.<phase>.model`
config key with a single resolver. Hierarchy:

  1. CLI flag (--model)
  2. New-key config (e.g. models.dream.synthesize)
  3. Old-key config (deprecated dream.synthesize.model, dream.patterns.model)
     — read with stderr deprecation warning, one-per-process
  4. Global default (models.default)
  5. Env var (GBRAIN_MODEL or caller-supplied)
  6. Hardcoded fallback

Aliases (`opus`, `sonnet`, `haiku`, `gemini`, `gpt`) resolve at the end so
any tier can use a short name. User-defined `models.aliases.<name>` config
overrides built-ins. Cycle-safe (depth 2 break). Unknown alias passes
through unchanged so users can pass full provider IDs without registering.

When new-key + old-key are BOTH set (Codex P1 #11 fix), new-key wins and
stderr warns "deprecated config X ignored; Y is set and wins". When only
old-key is set, it's honored with a softer "rename to Y before v0.30"
warning. Both warnings emit once per (key, process) — a Set memo prevents
log spam in long-running daemons.

Migrated call sites: synthesize.ts (model + verdictModel), patterns.ts
(model). subagent.ts and search/expansion.ts to be migrated later in v0.28
(staying compatible until then).

Tests: test/model-config.test.ts — 11 cases pinning the 6-tier ordering,
alias resolution + cycle break, deprecated-key warning emit-once, and
unknown-alias pass-through. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 takes-fence: parser/renderer/upserter + chunker strip (privacy P0 fix)

src/core/takes-fence.ts — pure functions for the fenced markdown surface:
- parseTakesFence(body) — extracts ParsedTake[] from `<!--- gbrain:takes:begin/end -->`
  blocks. Strict on canonical form, lenient on hand-edits with warnings
  (TAKES_FENCE_UNBALANCED, TAKES_TABLE_MALFORMED, TAKES_ROW_NUM_COLLISION).
  Strikethrough `~~claim~~` → active=false; date ranges `since → until`
  split into sinceDate/untilDate.
- renderTakesFence(takes) — round-trip safe with parseTakesFence.
- upsertTakeRow(body, row) — append-only per CEO-D6 + eng-D9. Creates a
  fresh `## Takes` section if no fence present. row_num is monotonic
  (max + 1, never gap-filled — keeps cross-page refs and synthesis_evidence
  stable forever).
- supersedeRow(body, oldRow, replacement) — strikes through old row's claim
  AND appends the new row at end. Both rows preserved in markdown for
  git-blame archaeology.
- stripTakesFence(body) — removes the fenced block entirely. Used by the
  chunker so takes content lives ONLY in the takes table.

Codex P0 #3 fix: src/core/chunkers/recursive.ts now calls stripTakesFence()
before computing chunk boundaries. Without this, page chunks would contain
the rendered takes table and the per-token MCP allow-list would be
bypassed at the index layer (token bound to takes_holders=['world'] would
see garry's hunches via page hits). Doctor's takes_fence_chunk_leak check
(plan-side) asserts no chunk contains the begin marker.

Tests: 15 cases covering canonical parse, strikethrough, date range, fence
unbalanced detection, malformed-row skip + warning, row_num collision
detection, round-trip render, append-only upsert into existing fence,
fresh-section creation, monotonic row_num under hand-edit gaps, supersede
flow, stripTakesFence verifying takes content removed AND surrounding
prose preserved. Existing chunker tests still pass (15 + 15 = 30).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 page-lock: PID-liveness file lock for atomic markdown read-modify-write

src/core/page-lock.ts — per-page file lock at
~/.gbrain/page-locks/<sha256-of-slug>.lock so two concurrent `gbrain takes
add` calls or `takes seed --refresh` from autopilot can't race on the
same `<slug>.md` read-modify-write. Eng-review fold: reuses the v0.17
cycle.lock pattern (mtime + PID liveness) but per-slug.

Differences from cycle.ts's lock:
- SHA-256 of slug for safe filenames (slashes, unicode, etc.)
- Same-pid + fresh mtime = LIVE (cycle.ts assumes one lock per process and
  reclaims same-pid; page-lock allows concurrent locks for DIFFERENT slugs
  in one process). mtime expiry still rescues post-crash leftovers.
- 5-min TTL (vs cycle's 30 min — page edits are short)
- `withPageLock(slug, fn)` convenience wrapper with default 30s timeout

API:
- acquirePageLock(slug, opts) → handle | null (poll-with-timeout)
- handle.refresh() / handle.release() (idempotent — only releases if pid matches)
- withPageLock(slug, fn, opts) — acquire + run + release-in-finally

Tests: 10 cases — fresh acquire, live holder returns null, stale-mtime
reclaim, dead-PID reclaim, refresh updates timestamp, foreign-pid release
is no-op, withPageLock callback runs and releases on success/failure,
timeout-throws when held, SHA-256 filename safety for slashes/unicode.
All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 extract-takes: dual-path phase (fs|db) + since/until_date as TEXT

src/core/cycle/extract-takes.ts — new phase that materializes the takes
table from fenced markdown blocks. Two paths mirror src/commands/extract.ts:

- extractTakesFromFs: walk *.md under repoPath, parse fences, batch upsert
- extractTakesFromDb: iterate engine.getAllSlugs(), parse each page's
  compiled_truth+timeline, batch upsert (mutation-immune snapshot iteration)

Single dispatcher extractTakes(opts) routes by source. Honors:
- slugs filter for incremental re-extract (pipes from sync→extract)
- dryRun: count would-be upserts, write nothing
- rebuild: DELETE FROM takes WHERE page_id = $1 before re-insert (clean
  slate when markdown is canonical and DB has drifted)

Schema fix: since_date/until_date were DATE in the original v31 migration.
Spec uses partial dates ('2017-01', '2026-04-29 → 2026-06') that Postgres
DATE rejects. Changed to TEXT in both the Postgres and PGLite blocks so
parser-rendered ranges round-trip cleanly. Loses the ability to do
date-range arithmetic in SQL, but date math on opinion timelines is
out of scope for v0.28 anyway. utils.ts dateOrNull now annotated as
v0.28 TEXT-aware.

Migration v31 has not been deployed yet (this branch is the v0.28 release
candidate), so the type swap is free. No data migration needed.

Tests: test/extract-takes.test.ts — 5 cases against PGLite covering full
walk + fence-skip on no-fence pages, takes-table populated post-extract,
incremental slugs filter, dry-run no-write, rebuild=true clears + re-inserts
ad-hoc rows. test/takes-engine.test.ts (16), test/takes-fence.test.ts (15)
all still pass — 36/36 takes tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 takes CLI: list, search, add, update, supersede, resolve

src/commands/takes.ts — surfaces the engine methods + takes-fence library
through a single `gbrain takes <subcommand>` entrypoint:

  takes <slug>                          list with filters + sort
  takes search "<query>"                pg_trgm keyword search across all takes
  takes add <slug> --claim ... ...      append (markdown + DB, atomic via lock)
  takes update <slug> --row N ...       mutable-fields update (markdown + DB)
  takes supersede <slug> --row N ...    strikethrough old + append new
  takes resolve <slug> --row N --outcome  record bet resolution (immutable)

Markdown is canonical. Every mutate command:
  1. acquires the per-page file lock (withPageLock)
  2. re-reads the .md file
  3. applies the edit via takes-fence (upsertTakeRow / supersedeRow)
  4. writes the .md file back
  5. mirrors to the DB via the engine method
  6. releases the lock (auto via finally)

Resolve currently writes only to DB — surfacing resolved_* in the markdown
table is deferred to v0.29 (the takes-fence renderer's column set is
fixed at # | claim | kind | who | weight | since | source per spec).

Wired into src/cli.ts dispatch + CLI_ONLY allowlist. Help text follows the
project convention (orphans/embed/extract pattern). --dir flag overrides
sync.repo_path config when working outside the configured brain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 MCP + auth: takes_list / takes_search / think ops + per-token allow-list

OperationContext gains takesHoldersAllowList — server-side filter for
takes.holder field threaded from access_tokens.permissions through dispatch
into the engine SQL. Closes Codex P0 #3 at the dispatch layer (chunker
strip already closed the page-content side in the previous commit).

src/core/operations.ts — three new ops:
- takes_list: lists takes with holder/kind/active/resolved filters; honors
  ctx.takesHoldersAllowList for MCP-bound calls
- takes_search: pg_trgm keyword search; honors allow-list
- think: op surface registered (returns not_implemented envelope until
  Lane D's pipeline lands). Remote callers cannot save/take per Codex P1 #7.

src/mcp/dispatch.ts — DispatchOpts.takesHoldersAllowList threads into
buildOperationContext.

src/mcp/http-transport.ts — validateToken now reads
access_tokens.permissions.takes_holders, defaults to ['world'] when the
column is absent or malformed (default-deny on private hunches).
auth.takesHoldersAllowList passed to dispatchToolCall.

src/mcp/server.ts (stdio) — defaults to takesHoldersAllowList: ['world']
since stdio has no per-token auth. Operators wanting full visibility use
`gbrain call <op>` directly (sets remote=false).

src/commands/auth.ts — `gbrain auth create <name> --takes-holders w,g,b`
flag persists the per-token list; new `auth permissions <name>
set-takes-holders <list>` updates an existing token.

Tests: test/takes-mcp-allowlist.test.ts — 8 cases against PGLite proving
the threading: local-CLI sees all holders, ['world'] returns only public,
['world','garry'] returns 2/3, no-overlap returns empty (no fallback),
search honors allow-list, remote save/take on think rejected with
not_implemented envelope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28.0: ship-prep — VERSION, CHANGELOG, migration orchestrator, skill

Closes the v0.28 ship-prep cycle. Bumps VERSION + package.json + bun.lock
to 0.28.0. v0_28_0 migration orchestrator runs three idempotent phases on
upgrade:

- Schema verify: asserts schema_version >= 32 (migrations v31 + v32 already
  applied by the schema runner during gbrain upgrade); fails clean if not.
- Backfill takes: inline runs `extractTakes(engine, { source: 'db' })` so
  any pre-existing fenced takes tables in markdown populate the takes
  index. Idempotent; ON CONFLICT DO UPDATE keeps the table in sync.
- Re-chunk TODO: queues a pending-host-work entry asking the host agent
  to re-import pages with takes content so the v0.28 chunker-strip rule
  (Codex P0 #3 fix) applies retroactively. Pages imported under v0.28+
  already have takes content stripped from chunks at index time; this
  TODO catches up legacy pages.

skills/migrations/v0.28.0.md — agent-readable upgrade guide. Walks
through doctor verification, deprecated-key migration, MCP token
visibility configuration, and a "try the takes layer" smoke test.

CHANGELOG.md — v0.28.0 release-summary in the GStack voice (no AI
vocabulary, no em dashes, real numbers from git diff stat) + the
mandatory "To take advantage of v0.28.0" block + itemized changes by
subsystem (schema, engine, markdown surface, model config, MCP+auth,
CLI, tests, accepted risks).

Final test sweep: 65/65 v0.28 tests pass across 6 files. typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 think pipeline: gather → sanitize → synthesize → cite-render → CLI

src/core/think/sanitize.ts — prompt-injection defense for take claims:
14 jailbreak patterns (ignore-prior, role-jailbreak, close-take tag,
DAN, system-prompt overrides, eval-shell hooks) plus structural framing
(takes wrapped in <take id="..."> tags the model is told to treat as
DATA). Length-cap at 500 chars. Renders evidence blocks for the prompt.

src/core/think/prompt.ts — system prompt + structured-output schema.
Hard rules: cite every claim, mark hunches/low-weight explicitly,
surface conflicts (never silently pick), surface gaps. JSON schema
with answer + citations[] + gaps[]. Prompt adapts to anchor / time
window / save flag.

src/core/think/cite-render.ts — structured citations + regex fallback
(Codex P1 #4 fold). normalizeStructuredCitations validates the model's
structured output; parseInlineCitations is the body-scan fallback when
the model omits the structured field. resolveCitations dispatches and
records CITATIONS_REGEX_FALLBACK warning when used.

src/core/think/gather.ts — 4-stream parallel retrieval:
  1. hybridSearch (pages, existing primitive)
  2. searchTakes (keyword, pg_trgm)
  3. searchTakesVector (vector, when embedQuestion fn supplied)
  4. traversePaths (graph, when --anchor set)
RRF fusion (k=60). Each stream wrapped in try/catch — partial gather
beats no synthesis. Honors takesHoldersAllowList for MCP-bound calls.

src/core/think/index.ts — runThink orchestrator + persistSynthesis:
INTENT (regex classify) → GATHER → render evidence blocks → resolveModel
('models.think' → 'models.default' → GBRAIN_MODEL → opus) → LLM call
(injectable client) → JSON parse with code-fence + fallback strip →
resolveCitations → ThinkResult. persistSynthesis writes a synthesis
page + synthesis_evidence rows (page_id resolved per slug; page-level
citations skip evidence). Degrades gracefully without ANTHROPIC_API_KEY.
Round-loop scaffolding in place (rounds=1 only path exercised in v0.28).

src/commands/think.ts — `gbrain think "<question>"` CLI. Flag parsing
strips --anchor, --rounds, --save, --take, --model, --since, --until,
--json. Local CLI = remote=false, so save/take honored. Human-readable
output by default; --json for agent consumption.

operations.ts — `think` op now calls runThink (was a not_implemented
stub). Remote callers can't save/take per Codex P1 #7. Returns full
ThinkResult plus saved_slug + evidence_inserted.

cli.ts — wired into dispatch + CLI_ONLY allowlist.

Tests: test/think-pipeline.test.ts — 18 cases against PGLite covering
sanitize patterns, structural rendering, citation parsing (structured +
regex fallback + dedup + invalid-slug rejection), gather streams +
allow-list filter, full pipeline with stub client, malformed-LLM
fallback path, no-API-key graceful degradation, persistSynthesis writes
page + evidence rows. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 dream phases: auto-think + drift + budget meter (Codex P1 #10 fold)

src/core/anthropic-pricing.ts — USD/1M-tokens map for Claude 4.7 family
plus older aliases. estimateMaxCostUsd returns null on unpriced models so
the meter caller can warn-once and bypass the gate.

src/core/cycle/budget-meter.ts — cumulative cost ledger. Each submit
estimates max-cost from (model + estimatedInputTokens + maxOutputTokens),
accumulates per-cycle, refuses next submit when projected > cap. Codex
P1 #10 fold: non-Anthropic models (gemini, gpt) bypass with one stderr
warn per process and `unpriced=true` on the result. Budget=0 disables
the gate. Audit trail at ~/.gbrain/audit/dream-budget-YYYY-Www.jsonl.

src/core/cycle/auto-think.ts — auto_think dream phase. Reads
dream.auto_think.{enabled,questions,max_per_cycle,budget,cooldown_days,
auto_commit}. Iterates configured questions through runThink with the
BudgetMeter pre-checking each submit. Cooldown timestamp written ONLY on
success (matches v0.23 synthesize pattern — retries after partial
failures pick back up). When auto_commit=true, persists synthesis pages
via persistSynthesis. Default-disabled.

src/core/cycle/drift.ts — drift dream phase scaffold. Reads
dream.drift.{enabled,lookback_days,budget,auto_update}. Surfaces takes
in the soft band (weight 0.3-0.85, unresolved) that have recent timeline
evidence on the same page. v0.28 ships the orchestration; the LLM judge
that proposes weight adjustments lands in v0.29. modelId + meter wired
now so the ledger captures gate state for callers that opt in.

Tests:
- test/budget-meter.test.ts (7 cases) — pricing-map coverage, allow path,
  cumulative-deny, budget=0 disabled, unpriced bypass+warn-once, ledger
  captures all events, ISO-week filename branch.
- test/auto-think-phase.test.ts (9 cases) — auto_think enable/skip,
  questions empty, success → cooldown ts written, cooldown blocks rerun,
  budget exhausted → partial. drift not_enabled, soft-band candidate
  detection, complete + dry-run paths.

All pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e Postgres: takes engine + extract + MCP allow-list (12 cases)

test/e2e/takes-postgres.test.ts — full v0.28 takes pipeline against real
Postgres (gated on DATABASE_URL). 12 cases:
- addTakesBatch upsert via unnest() bind path (Postgres-specific)
- listTakes filters: holder, kind, sort=weight, takesHoldersAllowList
- searchTakes pg_trgm + allow-list filter
- supersedeTake transactional path (BEGIN/COMMIT semantics)
- resolveTake immutability — second resolve throws TAKE_ALREADY_RESOLVED
- synthesis_evidence FK CASCADE on take delete
- countStaleTakes + listStaleTakes filter active+null
- extractTakesFromDb populates takes from fenced markdown
- MCP dispatch with takesHoldersAllowList=['world'] returns only world
- MCP dispatch local-CLI path returns all holders
- MCP dispatch takes_search honors allow-list
- think op forces remote_persisted_blocked even for save+take

postgres-engine.ts: addTakesBatch boolean[] serialization fix.
postgres-js auto-detects element type from JS arrays; for booleans it
mis-detects as scalar. Cast through text[] (`'true' | 'false'`) then
SQL-cast to boolean[] — same pattern other batch methods rely on for
type-stable bind shapes.

test/e2e/helpers.ts: setupDB now (a) tolerates non-existent tables in
TRUNCATE (for fresh DBs where v31 hasn't yet created takes/synthesis_evidence)
and (b) calls engine.initSchema() to actually run migrations.

test/takes-mcp-allowlist.test.ts: updated 2 think-op cases to match
Lane D's landed pipeline. They previously asserted not_implemented
envelopes; now they assert remote_persisted_blocked + NO_ANTHROPIC_API_KEY
graceful-degrade behavior.

Run: DATABASE_URL=postgres://localhost:5435/gbrain_test bun test test/e2e/takes-postgres.test.ts
Result: 12/12 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 dream phases: local DreamPhaseResult type (avoid premature CyclePhase enum extension)

cycle.ts's PhaseResult is shaped {phase, status, summary, details} with a
narrow PhaseStatus enum ('ok'|'warn'|'fail'|'skipped') and CyclePhase enum
that doesn't yet include 'auto_think'/'drift'. The phases ship standalone
in v0.28 (cycle.ts dispatcher integration is v0.28.x); using PhaseResult
forced premature enum extension.

Introduces DreamPhaseResult exported from auto-think.ts:
  { name: 'auto_think'|'drift'; status: 'complete'|'partial'|'failed'|'skipped';
    detail: string; totals?: Record<string,number>; duration_ms: number }

drift.ts re-exports the same type. When v0.28.x wires the dispatcher, the
adapter at the call site can map DreamPhaseResult → PhaseResult cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e: access_tokens.permissions JSONB end-to-end (5 cases)

test/e2e/auth-permissions.test.ts — closes the v0.28 token-allow-list
verification loop against real Postgres. Exercises:

- Migration v32 default backfill: new tokens created without a permissions
  column get {takes_holders: ["world"]} via the schema DEFAULT clause.
- Explicit ["world","garry"] → dispatch.takes_list filters to those
  holders only; brain hunches stay hidden from this token.
- ["world"] default-deny token → takes_search hits filtered to public claims.
- {} permissions row (operator tampered) gracefully defaults to ["world"]
  via the HTTP transport's validateToken parsing.
- revoked_at IS NOT NULL → token excluded from active token query.

Avoids the postgres-js JSONB double-encode trap (CLAUDE.md memory): pass
the object directly to executeRaw, no JSON.stringify, no ::jsonb cast.

All 5 pass against pgvector/pgvector:pg16 on port 5435. Combined v0.28
test sweep: 116/116 across 11 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e: chunker takes-strip integration test (Codex P0 #3 verification)

test/e2e/chunker-takes-strip.test.ts — verifies the chunker actually
strips fenced takes content end-to-end through the import pipeline.
This is the Codex P0 #3 fix's verification path: takes content lives
ONLY in the takes table for retrieval, never duplicated in
content_chunks where the per-token MCP allow-list cannot reach.

5 cases:
- chunkText (unit) output never contains TAKES_FENCE_BEGIN/END markers
- chunkText output never contains fenced claim text
- chunkText output retains non-fence prose (no over-stripping)
- importFromContent end-to-end: imported page has chunks but none
  contain fenced content
- takes_fence_chunk_leak doctor invariant: zero rows globally where
  chunk_text matches `<!--- gbrain:takes:%`

Final v0.28 test sweep:
  121 pass, 0 fail, 336 expect() calls, 12 files
  Coverage: schema migrations, engine methods (PGLite + Postgres),
  takes-fence parser, page-lock, extract phase, takes CLI engine
  surface, model config 6-tier resolver, MCP+auth allow-list,
  think pipeline (gather + sanitize + cite-render + synthesize),
  auto-think + drift + budget meter, JSONB end-to-end, chunker
  strip integration. ~95% of v0.28 surface area covered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix CI: apply-migrations skippedFuture arrays + http-transport SQL mock

Two CI failures from PR #563:

test/apply-migrations.test.ts (2 fails) — `buildPlan` tests assert exact
skippedFuture arrays at fixed installed-version stamps. Adding v0.28.0 to
the migration registry means it shows up in skippedFuture when the test
runs at installed=0.11.1 / installed=0.12.0. Append '0.28.0' to both
hardcoded arrays.

test/http-transport.test.ts (8 fails) — the FakeEngine mock string-prefix
matches `SELECT id, name FROM access_tokens` to return a row. v0.28's
validateToken now selects `SELECT id, name, permissions FROM access_tokens`
to read the per-token takes_holders allow-list. Mock returned [] on the
new query → validateToken treated every token as invalid → 401.

Fix: mock now matches both query shapes. validTokens row gets a default
`{takes_holders: ['world']}` permission injected when caller didn't
supply one (mirrors the migration v33 column DEFAULT). Updated
FakeEngineConfig type to allow tests to pass explicit permissions.

Verification:
  bun test test/apply-migrations.test.ts → 18/18 pass
  bun test test/http-transport.test.ts   → 24/24 pass
  bun run typecheck                       → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix CI: add scope annotations to v0.28 ops (takes_list/takes_search/think)

test/oauth.test.ts enforces an invariant from master's v0.26 OAuth landing:
every Operation must have `scope: 'read' | 'write' | 'admin'`, and any op
flagged `mutating: true` must be 'write' or 'admin'. My v0.28 ops were added
before master shipped v0.26 + the new invariant; the merge surfaced the gap.

Annotations:
- takes_list   → read
- takes_search → read
- think        → write (mutating: true; --save persists synthesis page)

Verification:
  bun test test/oauth.test.ts → 42/42 pass
  bun run typecheck            → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.28.1): export INJECTION_PATTERNS for shared sanitization

The same pattern set protects takes from prompt-injection (think/sanitize.ts)
and now retrieved chat content in the LongMemEval harness. One source of
truth for both surfaces; adding a new pattern in this file automatically
covers benchmarks too.

Existing consumers (sanitizeTakeForPrompt, renderTakesBlock) keep working
unchanged. Verified via test/think-pipeline.test.ts (18 pass, 0 fail).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.28.1): longmemeval harness — reset-in-place over in-memory PGLite

One in-memory PGLiteEngine per benchmark run; TRUNCATE between questions
with runtime-enumerated tables via pg_tables so future schema migrations
don't silently leak across questions. Infrastructure tables (sources,
config, gbrain_cycle_locks, subagent_rate_leases) preserved across resets
so initSchema-seeded rows like sources.'default' survive (FK target for
pages.source_id).

Files:
- src/eval/longmemeval/harness.ts: createBenchmarkBrain + resetTables +
  withBenchmarkBrain. ~50 lines, no class wrapper.
- src/eval/longmemeval/adapter.ts: pure haystackToPages() converter.
  Slug prefix `chat/` (verified non-matching against DEFAULT_SOURCE_BOOSTS).
- src/eval/longmemeval/sanitize.ts: re-uses INJECTION_PATTERNS from
  think/sanitize.ts; wraps each session in <chat_session id date> tags;
  4000-char cap.
- test/longmemeval-sanitize.test.ts: 12 cases pinning the F8 contract.

Hermetic: no DATABASE_URL, no API keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.28.1): gbrain eval longmemeval CLI command

Run the LongMemEval public benchmark against gbrain's hybrid retrieval.
Dataset is a positional path (download from xiaowu0162/longmemeval on HF).
Per-question loop wraps everything in try/catch; one bad question doesn't
kill the run, error JSONL line emitted instead.

Wiring:
- src/cli.ts: pre-dispatch bypass for `eval longmemeval` so the user's
  ~/.gbrain brain is never opened. Hermeticity gate verified: --help works
  on machines with no gbrain config.
- src/commands/eval-longmemeval.ts: arg parsing, JSONL emit (LF + UTF-8
  pinned), hybridSearch with optional expandQuery from search/expansion.ts,
  resolveModel from model-config.ts (6-tier chain), ThinkLLMClient injection
  seam from think/index.ts, structural <chat_session> framing.
- test/eval-longmemeval.test.ts: 12 cases covering harness lifecycle,
  reset clears all tables, schema-migration robustness, p50/p99 speed gate
  (warm reset+import+search target <500ms), adapter shape, source-boost
  regression guard, end-to-end with stubbed LLM, JSONL format guard,
  per-question failure handling.
- test/fixtures/longmemeval-mini.jsonl: 5 hand-authored questions with
  keyword-friendly overlap so --keyword-only works in CI.

Speed: warm reset+import 5 pages+search p50=25.9ms p99=30.3ms locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.28.1): bump VERSION + CHANGELOG

VERSION + package.json synchronized at 0.28.1. CHANGELOG entry uses the
release-summary voice + "To take advantage of v0.28.1" block per CLAUDE.md.

Sequential release on garrytan/v0.28-release; lands after v0.28.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: surface v0.28.1 LongMemEval CLI across project docs

- README.md: add EVAL section to Commands reference (eval --qrels, export,
  prune, replay, longmemeval); add v0.28.1 announce paragraph next to the
  v0.25.0 BrainBench-Real intro.
- CLAUDE.md: add Key files entry for src/eval/longmemeval/ +
  src/commands/eval-longmemeval.ts; add "Key commands added in v0.28.1"
  subsection (mirrors the v0.26.5 / v0.25.0 pattern); inventory
  test/eval-longmemeval.test.ts + test/longmemeval-sanitize.test.ts under
  the unit-test list.
- docs/eval-bench.md: cross-link from the "What it actually does" section
  to LongMemEval as the third evaluation axis (public benchmark,
  ground-truth labels, full QA pipeline); append "Public benchmarks:
  LongMemEval (v0.28.1)" section with architecture, flags table, and
  perf numbers.
- CONTRIBUTING.md: append a paragraph after the eval-replay block pointing
  contributors at gbrain eval longmemeval for public-benchmark coverage.
- AGENTS.md: extend the existing eval-retrieval bullet with a one-line
  mention of gbrain eval longmemeval.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28.2 feat: remote-source MCP + scope hierarchy + whoami (#690)

* refactor(core): extract SSRF helpers from integrations.ts to core/url-safety.ts

src/core/git-remote.ts (next commit) needs isInternalUrl etc. but importing
from src/commands/ would invert the layering boundary (no existing
src/core/ file imports from src/commands/). Extract the SSRF helpers
(parseOctet, hostnameToOctets, isPrivateIpv4, isInternalUrl) into a new
src/core/url-safety.ts and have integrations.ts re-export for backward
compat. test/integrations.test.ts continues to pass without changes (110
existing tests, 214 expects).

Why this matters for v0.28: the upcoming sources --url feature reuses
this SSRF gate for git-clone URL validation. Codex review caught that
re-rolling weaker URL classification would regress on the IPv6/v4-mapped/
metadata/CGNAT bypass forms that integrations.ts already handles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): add git-remote module — SSRF-defensive clone/pull + state probe

New src/core/git-remote.ts (~210 lines) for v0.28's remote-source feature:

- GIT_SSRF_FLAGS exported const: -c http.followRedirects=false,
  -c protocol.file.allow=never, -c protocol.ext.allow=never,
  --no-recurse-submodules. Single source of truth shared by cloneRepo
  and pullRepo so a future flag added to one path lands on both.
  Closes the SSRF surfaces codex flagged: DNS rebinding via redirects,
  .gitmodules as a second-fetch surface, file:// scheme in remotes.

- parseRemoteUrl: https-only, rejects embedded credentials and path
  traversal, delegates internal-target classification to isInternalUrl
  from url-safety.ts (covers RFC1918, link-local, loopback, IPv6, CGNAT
  100.64/10, metadata hostnames, hex/octal/single-int bypass forms).
  GBRAIN_ALLOW_PRIVATE_REMOTES=1 escape hatch with stderr warning is
  needed for self-hosted git over Tailscale (CGNAT trips the gate).

- cloneRepo: --depth=1 default (full clone via depth: 0); refuses
  non-empty destDirs; spawns git via execFileSync (no shell injection)
  with GIT_TERMINAL_PROMPT=0 + askpass=/bin/false to prevent credential
  prompts. timeoutMs default 600s.

- pullRepo: -C path + GIT_SSRF_FLAGS + pull --ff-only, same env confine.

- validateRepoState: 6-state decision tree (missing | not-a-dir |
  no-git | corrupted | url-drift | healthy). Used by performSync's
  re-clone branch to recover from rmd clone dirs and refuse syncs on
  url-drift or corruption.

test/git-remote.test.ts (304 lines, 32 tests): GIT_SSRF_FLAGS exact
shape, all parseRemoteUrl rejection cases including dedicated CGNAT
100.64/10 with/without GBRAIN_ALLOW_PRIVATE_REMOTES (codex T3 case),
fake-git harness for argv assertions on cloneRepo/pullRepo, all 6
validateRepoState branches.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): add scope hierarchy + ALLOWED_SCOPES allowlist

New src/core/scope.ts (~120 lines) for v0.28's scoped MCP feature.

Hierarchy:
  - admin implies all (escape hatch)
  - write implies read
  - sources_admin and users_admin are siblings (different axes —
    sources-mgmt vs user-account-mgmt; neither implies the other)

Exported:
  - hasScope(grantedScopes, requiredScope): the canonical scope check.
    Replaces exact-string-match at three call sites in upcoming commits
    (serve-http.ts:673, oauth-provider.ts:365 F3 refresh, oauth-provider.ts:498
    token issuance). Without this rewrite, an admin-grant token would
    fail to refresh down to sources_admin (codex finding).
  - ALLOWED_SCOPES set + ALLOWED_SCOPES_LIST sorted array (deterministic
    for OAuth metadata wire format and drift-check output).
  - assertAllowedScopes / InvalidScopeError: registration-time gate so
    tokens with bogus scope strings (read flying-unicorn) get rejected
    with RFC 6749 §5.2 invalid_scope at auth.ts:296 + DCR /register +
    registerClientManual. Today's behavior accepts any string silently.
  - parseScopeString: space-separated wire format → array.

Forward-compat: hasScope ignores unknown granted scopes rather than
throwing, so pre-allowlist tokens with weird scope strings continue
working without crashes (registration is the gate, runtime is best-effort).

test/scope.test.ts (178 lines, 35 tests): hierarchy table including
all-implies for admin, sibling non-implication of *_admin scopes,
write→read but not the reverse, F3 refresh-token subset semantics
under hasScope, ALLOWED_SCOPES_LIST sorted-pinning, allowlist
rejection cases, parseScopeString edge cases (undefined/null/empty).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* build(admin): scope-constants mirror + drift CI for src/core/scope.ts

The admin React SPA's tsconfig.json scopes include: ['src'] to admin/src/,
so it cannot directly import ../../src/core/scope.ts. The plan considered
widening the include or generating a single source of truth; both options
either couple the SPA to the gbrain monorepo or add a build step. Eng
review picked the boring choice: hand-maintained mirror at
admin/src/lib/scope-constants.ts plus a CI drift check.

Files:
  - admin/src/lib/scope-constants.ts: hand-maintained ALLOWED_SCOPES_LIST
    duplicate, sorted alphabetically to match src/core/scope.ts.
  - scripts/check-admin-scope-drift.sh: extracts the list from each file
    via awk, normalizes via tr/sort, diffs. Exits 0 on match, 1 on drift
    (with full breakdown of which scopes diverged), 2 on internal error.
    Tested both passing and corrupted paths.
  - package.json: wires check:admin-scope-drift into both `verify` and
    `check:all` so any update to src/core/scope.ts that forgets the
    admin-side mirror fails the build.

The Agents.tsx scope-checkbox sites (5 hardcoded locations) get updated
in a later commit to import from this constants file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(oauth): hasScope hierarchy + ALLOWED_SCOPES allowlist at registration

Switch three call sites in oauth-provider.ts from exact-string-match to
hasScope() so the v0.28 sources_admin and users_admin scopes — and the
admin-implies-all + write-implies-read hierarchy in src/core/scope.ts —
work end to end:

- F3 refresh-token subset enforcement at line 365: previously rejected
  admin → sources_admin refresh because exact-match treated them as
  unrelated scopes. gstack /setup-gbrain Path 4 needs admin tokens to
  refresh down to least-privilege sources_admin scope; this fix lands
  that path.

- Token issuance intersection at line 498 (client_credentials grant):
  same hasScope swap so a client whose stored grant is `admin` can mint
  tokens including any implied scope.

- registerClient (DCR /register) and registerClientManual: validate
  every scope string against ALLOWED_SCOPES via assertAllowedScopes.
  Pre-fix the system silently accepted `--scopes "read flying-unicorn"`
  and persisted the bogus string in oauth_clients.scope. Post-fix the
  caller gets RFC 6749 §5.2 invalid_scope. Existing rows with
  pre-allowlist scopes keep working (allowlist gates registration only).

Tests amended in test/oauth.test.ts:
- T1 (eng-review): admin grant CAN refresh down to sources_admin
- T1 sibling: write grant CANNOT refresh up to sources_admin
- ALLOWED_SCOPES allowlist coverage (manual + DCR paths, all 5 valid)
- Scope-annotation contract tests widened to accept the v0.28 union

62 OAuth tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(serve-http): hasScope at /mcp + advertise full ALLOWED_SCOPES

Two changes against src/commands/serve-http.ts:

- Line 195: scopesSupported on the mcpAuthRouter options switches from the
  hardcoded ['read','write','admin'] to Array.from(ALLOWED_SCOPES_LIST).
  Without this, /.well-known/oauth-authorization-server keeps reporting
  the old triple, so MCP clients (Claude Desktop, ChatGPT, Perplexity)
  cannot discover the v0.28 sources_admin and users_admin scopes via
  standard discovery — they would have to be pre-configured out of band.

- Line 673: request-time scope check on /mcp swaps
  authInfo.scopes.includes(requiredScope) for hasScope(...). This was
  the most-cited codex finding: without it, sources_admin tokens could
  not even satisfy a `read`-scoped op (sources_admin doesn't include
  the literal string "read"). hasScope routes through the hierarchy
  table in src/core/scope.ts so admin implies all and write implies
  read at the gate too.

T2 amendment in test/e2e/serve-http-oauth.test.ts: assert
/.well-known/oauth-authorization-server includes all 5 scopes in
scopes_supported. Pre-v0.28 the list was hardcoded to ['read','write',
'admin'] and this assertion would have failed. (The test is
Postgres-gated; runs under bun run test:e2e with DATABASE_URL set.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): sources-ops module — atomic clone + symlink-safe cleanup

src/core/sources-ops.ts (~470 lines): pure async functions extracted from
src/commands/sources.ts so the CLI handlers and the new MCP ops share
one implementation.

addSource: D3 atomicity contract from the eng review.
  1. Validate id (matches existing SOURCE_ID_RE).
  2. Q4 pre-flight SELECT — fail loudly with structured `source_id_taken`
     before any clone work. Pre-fix the existing CLI used INSERT…ON
     CONFLICT DO NOTHING which silently no-op'd; with clone-first that
     would orphan the temp dir.
  3. parseRemoteUrl gate (delegates to isInternalUrl from url-safety.ts).
  4. Clone into $GBRAIN_HOME/clones/.tmp/<id>-<rand>/ via the new
     git-remote helpers.
  5. INSERT row with local_path=<final clone dir>, config.remote_url=<url>.
  6. fs.renameSync(tmp/, final/). Rollback on either-side failure unlinks
     the temp dir; rename-failed path also DELETEs the just-INSERTed row
     best-effort.

removeSource: clone-cleanup with realpath+lstat confinement matching
validateUploadPath() shape at src/core/operations.ts:61. String startsWith
is symlink-unsafe and would let $GBRAIN_HOME/clones/<id> → /etc resolve
out of the confine. Two defenses layered:
  - isPathContained (realpath-resolves both sides + parent-with-sep
    string check) rejects symlinks whose target falls outside the
    confine.
  - lstat-then-isSymbolicLink check refuses symlinks whose realpath
    happens to land back inside the confine (defense in depth).

getSourceStatus: returns clone_state via validateRepoState (the 6-state
decision tree from git-remote.ts). Lets a remote MCP caller diagnose
"healthy | missing | not-a-dir | no-git | url-drift | corrupted" without
SSH access to the brain host. listSources additionally exposes
remote_url so callers can see which sources are auto-managed.

recloneIfMissing: T4 follow-up for `gbrain sources restore` after the
clone dir was autopurged — re-clones via the same temp + rename
atomicity contract. Idempotent (returns false when clone is already
healthy).

test/sources-ops.test.ts (~470 lines, 24 tests): pre-flight collision
(Q4), happy paths for both --path and --url, all four D3 rollback paths
(clone-fail before INSERT, INSERT-fail after clone, rename-fail
post-INSERT, atomic temp-dir cleanup), symlink-target-OUTSIDE-clones
(realpath confinement), symlink-target-INSIDE-clones (lstat-check),
removeSource refuses to delete user-supplied paths, refuses "default"
source, getSourceStatus clone_state branches, T4 recloneIfMissing
recovery + idempotent + no-op for path-only sources, isPathContained
unit tests covering subtree / outside / symlink-escape / fail-closed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(operations): whoami + sources_{add,list,remove,status} MCP ops

Five new ops in src/core/operations.ts auto-flow through src/mcp/tool-defs.ts
so MCP clients (Claude Desktop, ChatGPT, Perplexity, OpenClaw) get them via
standard tools/list discovery — no SDK or transport code changes needed.

Operation.scope union widened to add 'sources_admin' and 'users_admin' (the
v0.28 hierarchy from src/core/scope.ts).

whoami (scope: read): introspect calling identity over MCP.
  - Returns `{transport: 'oauth', client_id, client_name, scopes, expires_at}`
    for OAuth clients (clientId starts with gbrain_cl_).
  - Returns `{transport: 'legacy', token_name, scopes, expires_at: null}`
    for grandfathered access_tokens.
  - Returns `{transport: 'local', scopes: []}` when ctx.remote === false.
    Empty scopes (NOT ['read','write','admin']) is the D2 decision —
    returning OAuth-shaped scopes for local callers would resurrect the
    v0.26.9 footgun where code conditionally trusted on
    `auth.scopes.includes('admin')` instead of `ctx.remote === false`.
  - Q3 fail-closed: throws unknown_transport when remote=true AND auth is
    missing OR ctx.remote is the literal `undefined` (cast bypass guard).
    A future transport that forgets to thread auth doesn't get a free
    pass.

sources_add (sources_admin, mutating): register a source by --path
  (existing v0.17 behavior) or --url (v0.28 federated remote-clone path).
  Calls into addSource from sources-ops.ts which owns the temp-dir +
  rename atomicity.

sources_list (read): list registered sources with page counts, federated
  flag, and remote_url. The remote_url field is new — lets a remote MCP
  caller see which sources are auto-managed.

sources_remove (sources_admin, mutating): cascade-delete a source +
  symlink-safe clone cleanup. Requires confirm_destructive: true when the
  source has data.

sources_status (read): per-source diagnostic returning clone_state
  ('healthy' | 'missing' | 'not-a-dir' | 'no-git' | 'url-drift' |
  'corrupted' | 'not-applicable') — lets a remote MCP caller diagnose a
  busted clone without SSH access to the brain host.

test/whoami.test.ts (9 tests): pinned transport-detection for all four
return shapes including Q3 fail-closed throw under both auth=undefined
and remote=undefined cast-bypass paths.

test/sources-mcp.test.ts (16 tests): op-metadata pins (scope, mutating,
localOnly), functional handler shape against PGLite, hasScope-driven
scope-enforcement smoke test simulating the serve-http.ts:673 gate
(read-only token rejected for sources_add; sources_admin token allowed;
admin token allowed for everything; gstack /setup-gbrain Path 4 token
covers all 4 ops), SSRF gate at the op layer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): re-clone fallback when clone is missing/no-git/corrupted

src/commands/sync.ts gets a v0.28-aware front-half. When the source has
config.remote_url, performSync calls validateRepoState before the existing
fast-forward pull path:

  - 'healthy'    → fall through to existing pull (unchanged)
  - 'missing'    → loud stderr "auto-recovery: re-cloning <id>", then
  'no-git'         recloneIfMissing handles the temp-dir + rename. Sync
  'not-a-dir'      continues from the freshly-cloned head.
  - 'corrupted'  → throw with structured hint pointing at sources remove
                   + add (no syncing wrong state).
  - 'url-drift'  → throw with hint pointing at the (deferred) sources
                   rebase-clone command.

Closes the operator-confidence gap: rm -rf $GBRAIN_HOME/clones/<id>/ no
longer breaks future syncs. The next sync sees the missing dir and
recovers via the recorded URL.

src/core/operations.ts: extend ErrorCode with 'unknown_transport' so
whoami's Q3 fail-closed path types check.

test/sources-resync-recovery.test.ts (12 tests): full validateRepoState
state matrix exercised under fake-git, recloneIfMissing recovery from
each degraded state, idempotent on healthy clones, the sync.ts:320
integration path that drives the recovery.

test/sources-ops.test.ts + test/sources-mcp.test.ts: drop the
GBRAIN_PGLITE_SNAPSHOT-disable line so these tests stop forcing cold
init across the parallel-shard runner. With snapshot allowed, init time
drops from 6+s to ~50ms and parallel runs stay under the 5s hook
timeout.

test/sources-mcp.test.ts: tighten scope literal-type so tsc keeps the
union narrow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): sources add --url + restore re-clone, thin-wrapper refactor

src/commands/sources.ts now delegates the data-mutation work to
src/core/sources-ops.ts (added in the previous commit). The CLI handler
parses argv, calls into addSource, and formats output.

Two new flags on `gbrain sources add`:
  - `--url <https-url>` : federated remote-clone path (clone + INSERT +
    rename, atomic rollback on failure).
  - `--clone-dir <path>` : override the default
    $GBRAIN_HOME/clones/<id>/ destination.

Validation rejects mutually-exclusive `--url` + `--path`. Errors from
the ops layer (SourceOpError) propagate through the CLI's standard
error wrapper in src/cli.ts so existing tests that assert throw shape
keep passing.

`gbrain sources restore <id>` (T4 from eng review): if the source has a
remote_url AND the on-disk clone was autopurged, call recloneIfMissing
before declaring success. Clone errors print a WARN with recovery
hints rather than failing the restore — the DB row is what restore
guarantees; the clone is best-effort.

54 sources-related tests pass (existing test/sources.test.ts +
sources-ops + sources-mcp).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor,cycle): orphan-clones surface + autopilot purge phase (P1)

addSource's atomicity contract uses a temp dir that gets renamed to the
final clone path. If the process is SIGKILL'd between clone-finish and
rename, the temp dir orphans on disk. Without sweeping these, a brain
server accumulates gigabytes over months of failed `sources add --url`
attempts.

Two layers:

1. `gbrain doctor` now surfaces stale entries. A new orphan_clones check
   walks $GBRAIN_HOME/clones/.tmp/, names anything older than 24h, and
   prints a warn with disk-byte estimate. Operators see the leak before
   `df` complains.

2. The autopilot cycle's existing `purge` phase grows a substep that
   nukes .tmp/ entries past the same 72h TTL the page-soft-delete purge
   uses. Operator behavior stays uniform across all soft-delete-style
   surfaces.

Both layers are filesystem-only (no DB). On a brain that never used
--url cloning, both are no-ops.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* build(admin): scope checkboxes source from scope-constants mirror + dist

admin/src/pages/Agents.tsx Register Client modal:
  - useState default sources from ALLOWED_SCOPES_LIST (defaulting `read`
    to true, others false; unchanged UX for the common case).
  - Scope checkbox map iterates ALLOWED_SCOPES_LIST instead of the old
    hardcoded ['read','write','admin'].

Without this commit, even with the v0.28.1 server-side scope hierarchy,
operators registering an OAuth client from the admin UI cannot tick the
new sources_admin / users_admin scopes — defeats the whole gstack
/setup-gbrain Path 4 unblock.

The drift-check CI gate (scripts/check-admin-scope-drift.sh) ensures
this list stays in sync with src/core/scope.ts going forward.

admin/dist/* rebuilt via `cd admin && bun run build`. Old hash bundle
removed; new bundle (224.96 kB / 68.70 kB gzip).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.28.1 — remote-source MCP + scope hierarchy + whoami

VERSION + package.json: bump to 0.28.1 (per CLAUDE.md branch-scoped
versioning rule — this branch adds substantial new features on top of
v0.28.0).

CHANGELOG.md: new top-level entry for v0.28.1 in the gstack/Garry voice
(no AI vocabulary, no em dashes, real numbers + commands). Lead
paragraph names what the user can now do that they couldn't before.
"Numbers that matter" table calls out the +5 MCP ops, +2 OAuth scopes,
and the 4-to-0 SSH-step number for gstack /setup-gbrain Path 4. "What
this means for you" closer ties the work to the operator workflow shift.
"To take advantage of v0.28.1" block has paste-ready upgrade commands
including the admin SPA rebuild step. Itemized changes section
describes the architecture cleanly without exposing scope-string
internals to public attack-surface enumeration (per CLAUDE.md
responsible-disclosure rule).

TODOS.md: file 6 follow-ups under a new "Remote-source MCP follow-ups
(v0.28.1)" section: token rotation, migration introspection in
get_health, Accept-header friendliness, sources rebase-clone for
URL-drift recovery, --filter=blob:none partial-clone option, and the
chunker_version PGLite-schema parity codex caught.

README.md: short subsection under the existing sources CLI listing
that names the new --url flag and what auto-recovery does. Capability
framing (no scope-string enumeration).

llms.txt + llms-full.txt: regenerated via `bun run build:llms` so the
documentation bundle reflects the v0.28.1 entry. The build-llms
generator's drift check passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): sources-remote-mcp — full gstack /setup-gbrain Path 4 round-trip

Spins up `gbrain serve --http` against real Postgres with a fake-git binary
in PATH (so `git clone` is exercised end-to-end without network), registers
two OAuth clients (sources_admin + read-only), mints tokens, calls the new
v0.28.1 MCP ops via /mcp, and asserts the gstack /setup-gbrain Path 4 flow
works end to end.

12 tests cover the full lifecycle:
- whoami over HTTP MCP returns transport=oauth + the right scopes
- /.well-known/oauth-authorization-server advertises all 5 scopes
- sources_add: clone fires, INSERT lands, row carries config.remote_url
- sources_status: clone_state=healthy after add
- sources_list: surfaces remote_url for the new source
- SSRF rejection: sources_add with RFC1918 URL fails at parseRemoteUrl gate
- Scope enforcement: read-only token gets insufficient_scope on sources_add
- Read-only token CAN call sources_list (read-scoped op)
- ALLOWED_SCOPES allowlist: CLI register-client rejects bogus scope
- Recovery: rm clone dir + sources_status reports clone_state=missing
- sources_remove: cascades + cleans up the auto-managed clone dir

Subprocess env threading replicates the v0.26.2 bun execSync inheritance
pattern — bun does NOT inherit process.env mutations, so every CLI
subprocess call passes env: { ...process.env } explicitly.

Cleanup contract mirrors test/e2e/serve-http-oauth.test.ts: revoke any
clients we registered, force-kill the server subprocess on SIGTERM
timeout, surface cleanup failures to stderr without throwing so real
test failures aren't masked.

The base table list in helpers.ts (ALL_TABLES) doesn't include sources
or oauth_clients, so this test explicitly truncates them in beforeAll
to avoid Q4 pre-flight collisions on re-run.

Skipped gracefully when DATABASE_URL is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: codex adversarial review — confine remote sources_admin + close SSRF gaps

Pre-ship adversarial review (codex exec) caught five issues. Four ship in
this commit; the fifth (DNS rebinding) is filed as v0.28.x follow-up.

CRITICAL — `sources_admin` tokens over HTTP MCP could plant content at any
host path. The MCP op exposed `path` and `clone_dir` to remote callers; the
op layer trusted them verbatim, then auto-recovery's rm -rf on degraded
state turned that into arbitrary delete primitives. src/core/operations.ts
sources_add handler now drops both fields when ctx.remote !== false. Local
CLI keeps the override (operator trust). Loud logger.warn when a remote
caller tries — visible in the SSE feed without leaking values.

HIGH — Steady-state `git pull --ff-only` bypassed GIT_SSRF_FLAGS entirely.
The legacy helper at src/commands/sync.ts:192 spawned git without the
-c http.followRedirects=false -c protocol.{file,ext}.allow=never
--no-recurse-submodules set that cloneRepo applies. Every recurring sync
was reopening the redirect/submodule/protocol bypass. Routed the call site
at sync.ts:381 through pullRepo from git-remote.ts so initial clone and
ongoing pull share one defensive flag set.

MEDIUM — listSources ignored its `include_archived` flag. The op
advertised the param but the function destructured it as `_opts` and
queried every row. Archived sources' ids, local_paths, and remote_urls
were leaking to read-scoped MCP callers by default. Filter in SQL
(`WHERE archived IS NOT TRUE` unless the flag is set) so archived rows
never reach the wire.

PARTIAL HIGH — IPv6 ULA fc00::/7 and link-local fe80::/10 were not in
the isInternalUrl bypass list. Only ::1/:: and IPv4-mapped IPv6 were
blocked. Added regex-based ULA + link-local rejection to url-safety.ts.

Test coverage:
- test/git-remote.test.ts: 4 new IPv6 cases (ULA fc-prefix + fd-prefix,
  link-local fe80::, public IPv6 still allowed).
- test/sources-mcp.test.ts: 3 new cases pinning the remote/local
  asymmetry (clone_dir override silently ignored over MCP, path nulled,
  local CLI keeps the override).
- test/sources-mcp.test.ts: 2 new cases for include_archived honored.

DNS rebinding (codex finding #3): the current gate is lexical only.
A deliberate attacker who controls a hostname's A/AAAA records can still
resolve to an internal IP. Closing this requires async DNS resolution +
revalidation; filed as v0.28.x follow-up in TODOS.md so the API change
surface (parseRemoteUrl becomes async, every caller updates) lands in
its own PR.

323 tests pass (9 files); 4071 unit tests pass (full suite).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.28.1 → v0.28.2 (master collision)

Caught after PR creation. master is at v0.28.1 already; this branch
forked from garrytan/v0.28-release at v0.28.0 and naively bumped to
v0.28.1 without checking the master queue. CI version-gate would have
rejected at merge time (requires VERSION strictly greater than
master's).

Root cause: I bumped VERSION mechanically during plan implementation
(echo "0.28.1" > VERSION) without consulting the queue-aware allocator
at bin/gstack-next-version. /ship Step 12's idempotency check then
classified state as ALREADY_BUMPED and the workflow's "queue drift"
comparison was the safety net I should have hit — but I skipped it.

Files updated:
- VERSION + package.json: 0.28.1 → 0.28.2
- CHANGELOG.md: header + "To take advantage of v0.28.2" subsection
- README.md: sources --url note version reference
- TODOS.md: 7 follow-up entries' version references
- llms.txt + llms-full.txt: regenerated

PR title rewrite via gstack-pr-title-rewrite.sh handled in a separate
gh pr edit call; CI version-gate now passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(todos): close longmemeval-publication, file 4 follow-up TODOs

Full 500-question 4-adapter LongMemEval _s benchmark landed at
github.com/garrytan/gbrain-evals#main:ced01f0. gbrain-hybrid 97.60% R@5,
+1.0pt over MemPal raw 96.6%. Replacing the now-stale "needs full run"
TODO with closure + 4 grounded follow-ups:

  1. Timeline-aware retrieval signal for temporal-reasoning questions
     (P2 — closes the only category we lose to MemPal-raw)
  2. Per-question batch consolidation for ~10x cold-cache speedup
     (P3 — makes daily benchmark CI gate practical)
  3. LongMemEval _m split run (P3 — differentiated, not yet published
     by MemPal)
  4. Cheaper-embedding-model recipe (P4 — recall-cost tradeoff curve)

Each TODO has the standard What/Why/Pros/Cons/Context/Depends-on shape per
the gbrain TODOS-format convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(llms): regenerate llms-full.txt to match merged CLAUDE.md

CI test/build-llms.test.ts asserts the committed llms.txt/llms-full.txt
are byte-for-byte identical to what scripts/build-llms.ts produces. The
master merge brought in v0.28.9/v0.28.10/v0.28.11 + multimodal embedding
notes that updated CLAUDE.md; the bundle was stale.

No content changes. Pure regeneration via `bun run build:llms`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(changelog): rewrite v0.28.12 entry — lead with the LongMemEval result

Old entry buried the headline ("LongMemEval lands in the box…") under
process detail (hermetic CI test count, 25.9ms p50, schema-table
runtime enumeration). The reader cares what gbrain DOES — not how we
plumbed the harness.

New entry leads with the actual number — 97.60% R@5 on the public
LongMemEval _s split, beating MemPalace raw by 1.0pt — followed by
the per-category win table that proves gbrain ties or beats MemPal in
5 of 6 question types and shows the +7.1pt assistant-voice lift.

Links to the full gbrain-evals report (97.60% headline + full
methodology + reproducible runner) so curious readers can dig deeper.

Two honest findings published in plain text: vector-only is
essentially tied with hybrid at K=5, and query expansion via Haiku is
a clean null result on this dataset. Better to publish the null than
hide it.

Reproduction block updated to match the actual gbrain-evals workflow
(clone + bun install + dataset download + bash batch runner). The
prior "download / run / hand to evaluate_qa.py" block stayed for the
in-tree CLI path.

Regenerated llms-full.txt to keep the build-llms regen-drift guard
green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 19:49:46 -07:00
Garry TanandClaude Opus 4.7 b325f28239 v0.28.6 feat: takes + think + unified model config + per-token MCP allow-list (#563) 
* v0.28 schema: takes + synthesis_evidence (v31) + access_tokens.permissions (v32)

Migration v31 adds the takes table (typed/weighted/attributed claims) and
synthesis_evidence (provenance for `gbrain think` outputs). Page-scoped via
page_id FK (slug isn't unique alone in v0.18+ multi-source). HNSW partial
index on embedding for active rows. ON DELETE CASCADE on synthesis_evidence
so deleting a source take cascades the provenance row.

Migration v32 adds access_tokens.permissions JSONB with safe-default
backfill (`{"takes_holders":["world"]}`). Default keeps non-world holders
hidden from MCP-bound tokens until the operator explicitly grants access
via the v0.28 auth permissions CLI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 engine: addTakesBatch, listTakes, searchTakes/Vector, supersede, resolve, synthesis_evidence

Extends BrainEngine with the takes domain object. Both engines implement the
same surface; PGLite uses manual `$N` placeholders, Postgres uses postgres-js
unnest() — same shape as addLinksBatch and addTimelineEntriesBatch.

Methods:
- addTakesBatch (upsert via ON CONFLICT (page_id, row_num) DO UPDATE)
- listTakes (filter by holder/kind/active/resolved, takesHoldersAllowList
  for MCP-bound calls, sortBy weight/since_date/created_at)
- searchTakes / searchTakesVector (pg_trgm + cosine; honor allow-list)
- countStaleTakes / listStaleTakes (mirror countStaleChunks pattern;
  embedding column intentionally omitted from listStale payload)
- updateTake (mutable fields only; throws TAKE_ROW_NOT_FOUND)
- supersedeTake (transactional: insert new at next row_num, mark old
  active=false, set superseded_by; throws TAKE_RESOLVED_IMMUTABLE on
  resolved bets)
- resolveTake (sets resolved_*; throws TAKE_ALREADY_RESOLVED on re-resolve;
  resolution is immutable per Codex P1 #13 fold)
- addSynthesisEvidence (provenance persist; ON CONFLICT DO NOTHING)
- getTakeEmbeddings (parallel to getEmbeddingsByChunkIds)

Types live in src/core/engine.ts adjacent to LinkBatchInput. Page-scoped
via page_id (slug not unique in v0.18+ multi-source). PageType gains
'synthesis'. takeRowToTake mapper in utils.ts handles Date → ISO string
normalization.

Tests: test/takes-engine.test.ts — 16 cases against PGLite covering
upsert/list/filter/search happy paths, takesHoldersAllowList isolation,
the four invariant errors (TAKE_ROW_NOT_FOUND, TAKES_WEIGHT_CLAMPED,
TAKE_RESOLVED_IMMUTABLE, TAKE_ALREADY_RESOLVED), supersede flow, resolve
metadata round-trip, FK CASCADE on synthesis_evidence when source take
deletes. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 model-config: unified resolveModel with 6-tier precedence + alias resolution

Replaces every hardcoded `claude-*-X` and per-phase `dream.<phase>.model`
config key with a single resolver. Hierarchy:

  1. CLI flag (--model)
  2. New-key config (e.g. models.dream.synthesize)
  3. Old-key config (deprecated dream.synthesize.model, dream.patterns.model)
     — read with stderr deprecation warning, one-per-process
  4. Global default (models.default)
  5. Env var (GBRAIN_MODEL or caller-supplied)
  6. Hardcoded fallback

Aliases (`opus`, `sonnet`, `haiku`, `gemini`, `gpt`) resolve at the end so
any tier can use a short name. User-defined `models.aliases.<name>` config
overrides built-ins. Cycle-safe (depth 2 break). Unknown alias passes
through unchanged so users can pass full provider IDs without registering.

When new-key + old-key are BOTH set (Codex P1 #11 fix), new-key wins and
stderr warns "deprecated config X ignored; Y is set and wins". When only
old-key is set, it's honored with a softer "rename to Y before v0.30"
warning. Both warnings emit once per (key, process) — a Set memo prevents
log spam in long-running daemons.

Migrated call sites: synthesize.ts (model + verdictModel), patterns.ts
(model). subagent.ts and search/expansion.ts to be migrated later in v0.28
(staying compatible until then).

Tests: test/model-config.test.ts — 11 cases pinning the 6-tier ordering,
alias resolution + cycle break, deprecated-key warning emit-once, and
unknown-alias pass-through. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 takes-fence: parser/renderer/upserter + chunker strip (privacy P0 fix)

src/core/takes-fence.ts — pure functions for the fenced markdown surface:
- parseTakesFence(body) — extracts ParsedTake[] from `<!--- gbrain:takes:begin/end -->`
  blocks. Strict on canonical form, lenient on hand-edits with warnings
  (TAKES_FENCE_UNBALANCED, TAKES_TABLE_MALFORMED, TAKES_ROW_NUM_COLLISION).
  Strikethrough `~~claim~~` → active=false; date ranges `since → until`
  split into sinceDate/untilDate.
- renderTakesFence(takes) — round-trip safe with parseTakesFence.
- upsertTakeRow(body, row) — append-only per CEO-D6 + eng-D9. Creates a
  fresh `## Takes` section if no fence present. row_num is monotonic
  (max + 1, never gap-filled — keeps cross-page refs and synthesis_evidence
  stable forever).
- supersedeRow(body, oldRow, replacement) — strikes through old row's claim
  AND appends the new row at end. Both rows preserved in markdown for
  git-blame archaeology.
- stripTakesFence(body) — removes the fenced block entirely. Used by the
  chunker so takes content lives ONLY in the takes table.

Codex P0 #3 fix: src/core/chunkers/recursive.ts now calls stripTakesFence()
before computing chunk boundaries. Without this, page chunks would contain
the rendered takes table and the per-token MCP allow-list would be
bypassed at the index layer (token bound to takes_holders=['world'] would
see garry's hunches via page hits). Doctor's takes_fence_chunk_leak check
(plan-side) asserts no chunk contains the begin marker.

Tests: 15 cases covering canonical parse, strikethrough, date range, fence
unbalanced detection, malformed-row skip + warning, row_num collision
detection, round-trip render, append-only upsert into existing fence,
fresh-section creation, monotonic row_num under hand-edit gaps, supersede
flow, stripTakesFence verifying takes content removed AND surrounding
prose preserved. Existing chunker tests still pass (15 + 15 = 30).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 page-lock: PID-liveness file lock for atomic markdown read-modify-write

src/core/page-lock.ts — per-page file lock at
~/.gbrain/page-locks/<sha256-of-slug>.lock so two concurrent `gbrain takes
add` calls or `takes seed --refresh` from autopilot can't race on the
same `<slug>.md` read-modify-write. Eng-review fold: reuses the v0.17
cycle.lock pattern (mtime + PID liveness) but per-slug.

Differences from cycle.ts's lock:
- SHA-256 of slug for safe filenames (slashes, unicode, etc.)
- Same-pid + fresh mtime = LIVE (cycle.ts assumes one lock per process and
  reclaims same-pid; page-lock allows concurrent locks for DIFFERENT slugs
  in one process). mtime expiry still rescues post-crash leftovers.
- 5-min TTL (vs cycle's 30 min — page edits are short)
- `withPageLock(slug, fn)` convenience wrapper with default 30s timeout

API:
- acquirePageLock(slug, opts) → handle | null (poll-with-timeout)
- handle.refresh() / handle.release() (idempotent — only releases if pid matches)
- withPageLock(slug, fn, opts) — acquire + run + release-in-finally

Tests: 10 cases — fresh acquire, live holder returns null, stale-mtime
reclaim, dead-PID reclaim, refresh updates timestamp, foreign-pid release
is no-op, withPageLock callback runs and releases on success/failure,
timeout-throws when held, SHA-256 filename safety for slashes/unicode.
All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 extract-takes: dual-path phase (fs|db) + since/until_date as TEXT

src/core/cycle/extract-takes.ts — new phase that materializes the takes
table from fenced markdown blocks. Two paths mirror src/commands/extract.ts:

- extractTakesFromFs: walk *.md under repoPath, parse fences, batch upsert
- extractTakesFromDb: iterate engine.getAllSlugs(), parse each page's
  compiled_truth+timeline, batch upsert (mutation-immune snapshot iteration)

Single dispatcher extractTakes(opts) routes by source. Honors:
- slugs filter for incremental re-extract (pipes from sync→extract)
- dryRun: count would-be upserts, write nothing
- rebuild: DELETE FROM takes WHERE page_id = $1 before re-insert (clean
  slate when markdown is canonical and DB has drifted)

Schema fix: since_date/until_date were DATE in the original v31 migration.
Spec uses partial dates ('2017-01', '2026-04-29 → 2026-06') that Postgres
DATE rejects. Changed to TEXT in both the Postgres and PGLite blocks so
parser-rendered ranges round-trip cleanly. Loses the ability to do
date-range arithmetic in SQL, but date math on opinion timelines is
out of scope for v0.28 anyway. utils.ts dateOrNull now annotated as
v0.28 TEXT-aware.

Migration v31 has not been deployed yet (this branch is the v0.28 release
candidate), so the type swap is free. No data migration needed.

Tests: test/extract-takes.test.ts — 5 cases against PGLite covering full
walk + fence-skip on no-fence pages, takes-table populated post-extract,
incremental slugs filter, dry-run no-write, rebuild=true clears + re-inserts
ad-hoc rows. test/takes-engine.test.ts (16), test/takes-fence.test.ts (15)
all still pass — 36/36 takes tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 takes CLI: list, search, add, update, supersede, resolve

src/commands/takes.ts — surfaces the engine methods + takes-fence library
through a single `gbrain takes <subcommand>` entrypoint:

  takes <slug>                          list with filters + sort
  takes search "<query>"                pg_trgm keyword search across all takes
  takes add <slug> --claim ... ...      append (markdown + DB, atomic via lock)
  takes update <slug> --row N ...       mutable-fields update (markdown + DB)
  takes supersede <slug> --row N ...    strikethrough old + append new
  takes resolve <slug> --row N --outcome  record bet resolution (immutable)

Markdown is canonical. Every mutate command:
  1. acquires the per-page file lock (withPageLock)
  2. re-reads the .md file
  3. applies the edit via takes-fence (upsertTakeRow / supersedeRow)
  4. writes the .md file back
  5. mirrors to the DB via the engine method
  6. releases the lock (auto via finally)

Resolve currently writes only to DB — surfacing resolved_* in the markdown
table is deferred to v0.29 (the takes-fence renderer's column set is
fixed at # | claim | kind | who | weight | since | source per spec).

Wired into src/cli.ts dispatch + CLI_ONLY allowlist. Help text follows the
project convention (orphans/embed/extract pattern). --dir flag overrides
sync.repo_path config when working outside the configured brain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 MCP + auth: takes_list / takes_search / think ops + per-token allow-list

OperationContext gains takesHoldersAllowList — server-side filter for
takes.holder field threaded from access_tokens.permissions through dispatch
into the engine SQL. Closes Codex P0 #3 at the dispatch layer (chunker
strip already closed the page-content side in the previous commit).

src/core/operations.ts — three new ops:
- takes_list: lists takes with holder/kind/active/resolved filters; honors
  ctx.takesHoldersAllowList for MCP-bound calls
- takes_search: pg_trgm keyword search; honors allow-list
- think: op surface registered (returns not_implemented envelope until
  Lane D's pipeline lands). Remote callers cannot save/take per Codex P1 #7.

src/mcp/dispatch.ts — DispatchOpts.takesHoldersAllowList threads into
buildOperationContext.

src/mcp/http-transport.ts — validateToken now reads
access_tokens.permissions.takes_holders, defaults to ['world'] when the
column is absent or malformed (default-deny on private hunches).
auth.takesHoldersAllowList passed to dispatchToolCall.

src/mcp/server.ts (stdio) — defaults to takesHoldersAllowList: ['world']
since stdio has no per-token auth. Operators wanting full visibility use
`gbrain call <op>` directly (sets remote=false).

src/commands/auth.ts — `gbrain auth create <name> --takes-holders w,g,b`
flag persists the per-token list; new `auth permissions <name>
set-takes-holders <list>` updates an existing token.

Tests: test/takes-mcp-allowlist.test.ts — 8 cases against PGLite proving
the threading: local-CLI sees all holders, ['world'] returns only public,
['world','garry'] returns 2/3, no-overlap returns empty (no fallback),
search honors allow-list, remote save/take on think rejected with
not_implemented envelope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28.0: ship-prep — VERSION, CHANGELOG, migration orchestrator, skill

Closes the v0.28 ship-prep cycle. Bumps VERSION + package.json + bun.lock
to 0.28.0. v0_28_0 migration orchestrator runs three idempotent phases on
upgrade:

- Schema verify: asserts schema_version >= 32 (migrations v31 + v32 already
  applied by the schema runner during gbrain upgrade); fails clean if not.
- Backfill takes: inline runs `extractTakes(engine, { source: 'db' })` so
  any pre-existing fenced takes tables in markdown populate the takes
  index. Idempotent; ON CONFLICT DO UPDATE keeps the table in sync.
- Re-chunk TODO: queues a pending-host-work entry asking the host agent
  to re-import pages with takes content so the v0.28 chunker-strip rule
  (Codex P0 #3 fix) applies retroactively. Pages imported under v0.28+
  already have takes content stripped from chunks at index time; this
  TODO catches up legacy pages.

skills/migrations/v0.28.0.md — agent-readable upgrade guide. Walks
through doctor verification, deprecated-key migration, MCP token
visibility configuration, and a "try the takes layer" smoke test.

CHANGELOG.md — v0.28.0 release-summary in the GStack voice (no AI
vocabulary, no em dashes, real numbers from git diff stat) + the
mandatory "To take advantage of v0.28.0" block + itemized changes by
subsystem (schema, engine, markdown surface, model config, MCP+auth,
CLI, tests, accepted risks).

Final test sweep: 65/65 v0.28 tests pass across 6 files. typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 think pipeline: gather → sanitize → synthesize → cite-render → CLI

src/core/think/sanitize.ts — prompt-injection defense for take claims:
14 jailbreak patterns (ignore-prior, role-jailbreak, close-take tag,
DAN, system-prompt overrides, eval-shell hooks) plus structural framing
(takes wrapped in <take id="..."> tags the model is told to treat as
DATA). Length-cap at 500 chars. Renders evidence blocks for the prompt.

src/core/think/prompt.ts — system prompt + structured-output schema.
Hard rules: cite every claim, mark hunches/low-weight explicitly,
surface conflicts (never silently pick), surface gaps. JSON schema
with answer + citations[] + gaps[]. Prompt adapts to anchor / time
window / save flag.

src/core/think/cite-render.ts — structured citations + regex fallback
(Codex P1 #4 fold). normalizeStructuredCitations validates the model's
structured output; parseInlineCitations is the body-scan fallback when
the model omits the structured field. resolveCitations dispatches and
records CITATIONS_REGEX_FALLBACK warning when used.

src/core/think/gather.ts — 4-stream parallel retrieval:
  1. hybridSearch (pages, existing primitive)
  2. searchTakes (keyword, pg_trgm)
  3. searchTakesVector (vector, when embedQuestion fn supplied)
  4. traversePaths (graph, when --anchor set)
RRF fusion (k=60). Each stream wrapped in try/catch — partial gather
beats no synthesis. Honors takesHoldersAllowList for MCP-bound calls.

src/core/think/index.ts — runThink orchestrator + persistSynthesis:
INTENT (regex classify) → GATHER → render evidence blocks → resolveModel
('models.think' → 'models.default' → GBRAIN_MODEL → opus) → LLM call
(injectable client) → JSON parse with code-fence + fallback strip →
resolveCitations → ThinkResult. persistSynthesis writes a synthesis
page + synthesis_evidence rows (page_id resolved per slug; page-level
citations skip evidence). Degrades gracefully without ANTHROPIC_API_KEY.
Round-loop scaffolding in place (rounds=1 only path exercised in v0.28).

src/commands/think.ts — `gbrain think "<question>"` CLI. Flag parsing
strips --anchor, --rounds, --save, --take, --model, --since, --until,
--json. Local CLI = remote=false, so save/take honored. Human-readable
output by default; --json for agent consumption.

operations.ts — `think` op now calls runThink (was a not_implemented
stub). Remote callers can't save/take per Codex P1 #7. Returns full
ThinkResult plus saved_slug + evidence_inserted.

cli.ts — wired into dispatch + CLI_ONLY allowlist.

Tests: test/think-pipeline.test.ts — 18 cases against PGLite covering
sanitize patterns, structural rendering, citation parsing (structured +
regex fallback + dedup + invalid-slug rejection), gather streams +
allow-list filter, full pipeline with stub client, malformed-LLM
fallback path, no-API-key graceful degradation, persistSynthesis writes
page + evidence rows. All pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 dream phases: auto-think + drift + budget meter (Codex P1 #10 fold)

src/core/anthropic-pricing.ts — USD/1M-tokens map for Claude 4.7 family
plus older aliases. estimateMaxCostUsd returns null on unpriced models so
the meter caller can warn-once and bypass the gate.

src/core/cycle/budget-meter.ts — cumulative cost ledger. Each submit
estimates max-cost from (model + estimatedInputTokens + maxOutputTokens),
accumulates per-cycle, refuses next submit when projected > cap. Codex
P1 #10 fold: non-Anthropic models (gemini, gpt) bypass with one stderr
warn per process and `unpriced=true` on the result. Budget=0 disables
the gate. Audit trail at ~/.gbrain/audit/dream-budget-YYYY-Www.jsonl.

src/core/cycle/auto-think.ts — auto_think dream phase. Reads
dream.auto_think.{enabled,questions,max_per_cycle,budget,cooldown_days,
auto_commit}. Iterates configured questions through runThink with the
BudgetMeter pre-checking each submit. Cooldown timestamp written ONLY on
success (matches v0.23 synthesize pattern — retries after partial
failures pick back up). When auto_commit=true, persists synthesis pages
via persistSynthesis. Default-disabled.

src/core/cycle/drift.ts — drift dream phase scaffold. Reads
dream.drift.{enabled,lookback_days,budget,auto_update}. Surfaces takes
in the soft band (weight 0.3-0.85, unresolved) that have recent timeline
evidence on the same page. v0.28 ships the orchestration; the LLM judge
that proposes weight adjustments lands in v0.29. modelId + meter wired
now so the ledger captures gate state for callers that opt in.

Tests:
- test/budget-meter.test.ts (7 cases) — pricing-map coverage, allow path,
  cumulative-deny, budget=0 disabled, unpriced bypass+warn-once, ledger
  captures all events, ISO-week filename branch.
- test/auto-think-phase.test.ts (9 cases) — auto_think enable/skip,
  questions empty, success → cooldown ts written, cooldown blocks rerun,
  budget exhausted → partial. drift not_enabled, soft-band candidate
  detection, complete + dry-run paths.

All pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e Postgres: takes engine + extract + MCP allow-list (12 cases)

test/e2e/takes-postgres.test.ts — full v0.28 takes pipeline against real
Postgres (gated on DATABASE_URL). 12 cases:
- addTakesBatch upsert via unnest() bind path (Postgres-specific)
- listTakes filters: holder, kind, sort=weight, takesHoldersAllowList
- searchTakes pg_trgm + allow-list filter
- supersedeTake transactional path (BEGIN/COMMIT semantics)
- resolveTake immutability — second resolve throws TAKE_ALREADY_RESOLVED
- synthesis_evidence FK CASCADE on take delete
- countStaleTakes + listStaleTakes filter active+null
- extractTakesFromDb populates takes from fenced markdown
- MCP dispatch with takesHoldersAllowList=['world'] returns only world
- MCP dispatch local-CLI path returns all holders
- MCP dispatch takes_search honors allow-list
- think op forces remote_persisted_blocked even for save+take

postgres-engine.ts: addTakesBatch boolean[] serialization fix.
postgres-js auto-detects element type from JS arrays; for booleans it
mis-detects as scalar. Cast through text[] (`'true' | 'false'`) then
SQL-cast to boolean[] — same pattern other batch methods rely on for
type-stable bind shapes.

test/e2e/helpers.ts: setupDB now (a) tolerates non-existent tables in
TRUNCATE (for fresh DBs where v31 hasn't yet created takes/synthesis_evidence)
and (b) calls engine.initSchema() to actually run migrations.

test/takes-mcp-allowlist.test.ts: updated 2 think-op cases to match
Lane D's landed pipeline. They previously asserted not_implemented
envelopes; now they assert remote_persisted_blocked + NO_ANTHROPIC_API_KEY
graceful-degrade behavior.

Run: DATABASE_URL=postgres://localhost:5435/gbrain_test bun test test/e2e/takes-postgres.test.ts
Result: 12/12 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 dream phases: local DreamPhaseResult type (avoid premature CyclePhase enum extension)

cycle.ts's PhaseResult is shaped {phase, status, summary, details} with a
narrow PhaseStatus enum ('ok'|'warn'|'fail'|'skipped') and CyclePhase enum
that doesn't yet include 'auto_think'/'drift'. The phases ship standalone
in v0.28 (cycle.ts dispatcher integration is v0.28.x); using PhaseResult
forced premature enum extension.

Introduces DreamPhaseResult exported from auto-think.ts:
  { name: 'auto_think'|'drift'; status: 'complete'|'partial'|'failed'|'skipped';
    detail: string; totals?: Record<string,number>; duration_ms: number }

drift.ts re-exports the same type. When v0.28.x wires the dispatcher, the
adapter at the call site can map DreamPhaseResult → PhaseResult cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e: access_tokens.permissions JSONB end-to-end (5 cases)

test/e2e/auth-permissions.test.ts — closes the v0.28 token-allow-list
verification loop against real Postgres. Exercises:

- Migration v32 default backfill: new tokens created without a permissions
  column get {takes_holders: ["world"]} via the schema DEFAULT clause.
- Explicit ["world","garry"] → dispatch.takes_list filters to those
  holders only; brain hunches stay hidden from this token.
- ["world"] default-deny token → takes_search hits filtered to public claims.
- {} permissions row (operator tampered) gracefully defaults to ["world"]
  via the HTTP transport's validateToken parsing.
- revoked_at IS NOT NULL → token excluded from active token query.

Avoids the postgres-js JSONB double-encode trap (CLAUDE.md memory): pass
the object directly to executeRaw, no JSON.stringify, no ::jsonb cast.

All 5 pass against pgvector/pgvector:pg16 on port 5435. Combined v0.28
test sweep: 116/116 across 11 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28 e2e: chunker takes-strip integration test (Codex P0 #3 verification)

test/e2e/chunker-takes-strip.test.ts — verifies the chunker actually
strips fenced takes content end-to-end through the import pipeline.
This is the Codex P0 #3 fix's verification path: takes content lives
ONLY in the takes table for retrieval, never duplicated in
content_chunks where the per-token MCP allow-list cannot reach.

5 cases:
- chunkText (unit) output never contains TAKES_FENCE_BEGIN/END markers
- chunkText output never contains fenced claim text
- chunkText output retains non-fence prose (no over-stripping)
- importFromContent end-to-end: imported page has chunks but none
  contain fenced content
- takes_fence_chunk_leak doctor invariant: zero rows globally where
  chunk_text matches `<!--- gbrain:takes:%`

Final v0.28 test sweep:
  121 pass, 0 fail, 336 expect() calls, 12 files
  Coverage: schema migrations, engine methods (PGLite + Postgres),
  takes-fence parser, page-lock, extract phase, takes CLI engine
  surface, model config 6-tier resolver, MCP+auth allow-list,
  think pipeline (gather + sanitize + cite-render + synthesize),
  auto-think + drift + budget meter, JSONB end-to-end, chunker
  strip integration. ~95% of v0.28 surface area covered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix CI: apply-migrations skippedFuture arrays + http-transport SQL mock

Two CI failures from PR #563:

test/apply-migrations.test.ts (2 fails) — `buildPlan` tests assert exact
skippedFuture arrays at fixed installed-version stamps. Adding v0.28.0 to
the migration registry means it shows up in skippedFuture when the test
runs at installed=0.11.1 / installed=0.12.0. Append '0.28.0' to both
hardcoded arrays.

test/http-transport.test.ts (8 fails) — the FakeEngine mock string-prefix
matches `SELECT id, name FROM access_tokens` to return a row. v0.28's
validateToken now selects `SELECT id, name, permissions FROM access_tokens`
to read the per-token takes_holders allow-list. Mock returned [] on the
new query → validateToken treated every token as invalid → 401.

Fix: mock now matches both query shapes. validTokens row gets a default
`{takes_holders: ['world']}` permission injected when caller didn't
supply one (mirrors the migration v33 column DEFAULT). Updated
FakeEngineConfig type to allow tests to pass explicit permissions.

Verification:
  bun test test/apply-migrations.test.ts → 18/18 pass
  bun test test/http-transport.test.ts   → 24/24 pass
  bun run typecheck                       → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix CI: add scope annotations to v0.28 ops (takes_list/takes_search/think)

test/oauth.test.ts enforces an invariant from master's v0.26 OAuth landing:
every Operation must have `scope: 'read' | 'write' | 'admin'`, and any op
flagged `mutating: true` must be 'write' or 'admin'. My v0.28 ops were added
before master shipped v0.26 + the new invariant; the merge surfaced the gap.

Annotations:
- takes_list   → read
- takes_search → read
- think        → write (mutating: true; --save persists synthesis page)

Verification:
  bun test test/oauth.test.ts → 42/42 pass
  bun run typecheck            → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.28.2 feat: remote-source MCP + scope hierarchy + whoami (#690)

* refactor(core): extract SSRF helpers from integrations.ts to core/url-safety.ts

src/core/git-remote.ts (next commit) needs isInternalUrl etc. but importing
from src/commands/ would invert the layering boundary (no existing
src/core/ file imports from src/commands/). Extract the SSRF helpers
(parseOctet, hostnameToOctets, isPrivateIpv4, isInternalUrl) into a new
src/core/url-safety.ts and have integrations.ts re-export for backward
compat. test/integrations.test.ts continues to pass without changes (110
existing tests, 214 expects).

Why this matters for v0.28: the upcoming sources --url feature reuses
this SSRF gate for git-clone URL validation. Codex review caught that
re-rolling weaker URL classification would regress on the IPv6/v4-mapped/
metadata/CGNAT bypass forms that integrations.ts already handles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): add git-remote module — SSRF-defensive clone/pull + state probe

New src/core/git-remote.ts (~210 lines) for v0.28's remote-source feature:

- GIT_SSRF_FLAGS exported const: -c http.followRedirects=false,
  -c protocol.file.allow=never, -c protocol.ext.allow=never,
  --no-recurse-submodules. Single source of truth shared by cloneRepo
  and pullRepo so a future flag added to one path lands on both.
  Closes the SSRF surfaces codex flagged: DNS rebinding via redirects,
  .gitmodules as a second-fetch surface, file:// scheme in remotes.

- parseRemoteUrl: https-only, rejects embedded credentials and path
  traversal, delegates internal-target classification to isInternalUrl
  from url-safety.ts (covers RFC1918, link-local, loopback, IPv6, CGNAT
  100.64/10, metadata hostnames, hex/octal/single-int bypass forms).
  GBRAIN_ALLOW_PRIVATE_REMOTES=1 escape hatch with stderr warning is
  needed for self-hosted git over Tailscale (CGNAT trips the gate).

- cloneRepo: --depth=1 default (full clone via depth: 0); refuses
  non-empty destDirs; spawns git via execFileSync (no shell injection)
  with GIT_TERMINAL_PROMPT=0 + askpass=/bin/false to prevent credential
  prompts. timeoutMs default 600s.

- pullRepo: -C path + GIT_SSRF_FLAGS + pull --ff-only, same env confine.

- validateRepoState: 6-state decision tree (missing | not-a-dir |
  no-git | corrupted | url-drift | healthy). Used by performSync's
  re-clone branch to recover from rmd clone dirs and refuse syncs on
  url-drift or corruption.

test/git-remote.test.ts (304 lines, 32 tests): GIT_SSRF_FLAGS exact
shape, all parseRemoteUrl rejection cases including dedicated CGNAT
100.64/10 with/without GBRAIN_ALLOW_PRIVATE_REMOTES (codex T3 case),
fake-git harness for argv assertions on cloneRepo/pullRepo, all 6
validateRepoState branches.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): add scope hierarchy + ALLOWED_SCOPES allowlist

New src/core/scope.ts (~120 lines) for v0.28's scoped MCP feature.

Hierarchy:
  - admin implies all (escape hatch)
  - write implies read
  - sources_admin and users_admin are siblings (different axes —
    sources-mgmt vs user-account-mgmt; neither implies the other)

Exported:
  - hasScope(grantedScopes, requiredScope): the canonical scope check.
    Replaces exact-string-match at three call sites in upcoming commits
    (serve-http.ts:673, oauth-provider.ts:365 F3 refresh, oauth-provider.ts:498
    token issuance). Without this rewrite, an admin-grant token would
    fail to refresh down to sources_admin (codex finding).
  - ALLOWED_SCOPES set + ALLOWED_SCOPES_LIST sorted array (deterministic
    for OAuth metadata wire format and drift-check output).
  - assertAllowedScopes / InvalidScopeError: registration-time gate so
    tokens with bogus scope strings (read flying-unicorn) get rejected
    with RFC 6749 §5.2 invalid_scope at auth.ts:296 + DCR /register +
    registerClientManual. Today's behavior accepts any string silently.
  - parseScopeString: space-separated wire format → array.

Forward-compat: hasScope ignores unknown granted scopes rather than
throwing, so pre-allowlist tokens with weird scope strings continue
working without crashes (registration is the gate, runtime is best-effort).

test/scope.test.ts (178 lines, 35 tests): hierarchy table including
all-implies for admin, sibling non-implication of *_admin scopes,
write→read but not the reverse, F3 refresh-token subset semantics
under hasScope, ALLOWED_SCOPES_LIST sorted-pinning, allowlist
rejection cases, parseScopeString edge cases (undefined/null/empty).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* build(admin): scope-constants mirror + drift CI for src/core/scope.ts

The admin React SPA's tsconfig.json scopes include: ['src'] to admin/src/,
so it cannot directly import ../../src/core/scope.ts. The plan considered
widening the include or generating a single source of truth; both options
either couple the SPA to the gbrain monorepo or add a build step. Eng
review picked the boring choice: hand-maintained mirror at
admin/src/lib/scope-constants.ts plus a CI drift check.

Files:
  - admin/src/lib/scope-constants.ts: hand-maintained ALLOWED_SCOPES_LIST
    duplicate, sorted alphabetically to match src/core/scope.ts.
  - scripts/check-admin-scope-drift.sh: extracts the list from each file
    via awk, normalizes via tr/sort, diffs. Exits 0 on match, 1 on drift
    (with full breakdown of which scopes diverged), 2 on internal error.
    Tested both passing and corrupted paths.
  - package.json: wires check:admin-scope-drift into both `verify` and
    `check:all` so any update to src/core/scope.ts that forgets the
    admin-side mirror fails the build.

The Agents.tsx scope-checkbox sites (5 hardcoded locations) get updated
in a later commit to import from this constants file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(oauth): hasScope hierarchy + ALLOWED_SCOPES allowlist at registration

Switch three call sites in oauth-provider.ts from exact-string-match to
hasScope() so the v0.28 sources_admin and users_admin scopes — and the
admin-implies-all + write-implies-read hierarchy in src/core/scope.ts —
work end to end:

- F3 refresh-token subset enforcement at line 365: previously rejected
  admin → sources_admin refresh because exact-match treated them as
  unrelated scopes. gstack /setup-gbrain Path 4 needs admin tokens to
  refresh down to least-privilege sources_admin scope; this fix lands
  that path.

- Token issuance intersection at line 498 (client_credentials grant):
  same hasScope swap so a client whose stored grant is `admin` can mint
  tokens including any implied scope.

- registerClient (DCR /register) and registerClientManual: validate
  every scope string against ALLOWED_SCOPES via assertAllowedScopes.
  Pre-fix the system silently accepted `--scopes "read flying-unicorn"`
  and persisted the bogus string in oauth_clients.scope. Post-fix the
  caller gets RFC 6749 §5.2 invalid_scope. Existing rows with
  pre-allowlist scopes keep working (allowlist gates registration only).

Tests amended in test/oauth.test.ts:
- T1 (eng-review): admin grant CAN refresh down to sources_admin
- T1 sibling: write grant CANNOT refresh up to sources_admin
- ALLOWED_SCOPES allowlist coverage (manual + DCR paths, all 5 valid)
- Scope-annotation contract tests widened to accept the v0.28 union

62 OAuth tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(serve-http): hasScope at /mcp + advertise full ALLOWED_SCOPES

Two changes against src/commands/serve-http.ts:

- Line 195: scopesSupported on the mcpAuthRouter options switches from the
  hardcoded ['read','write','admin'] to Array.from(ALLOWED_SCOPES_LIST).
  Without this, /.well-known/oauth-authorization-server keeps reporting
  the old triple, so MCP clients (Claude Desktop, ChatGPT, Perplexity)
  cannot discover the v0.28 sources_admin and users_admin scopes via
  standard discovery — they would have to be pre-configured out of band.

- Line 673: request-time scope check on /mcp swaps
  authInfo.scopes.includes(requiredScope) for hasScope(...). This was
  the most-cited codex finding: without it, sources_admin tokens could
  not even satisfy a `read`-scoped op (sources_admin doesn't include
  the literal string "read"). hasScope routes through the hierarchy
  table in src/core/scope.ts so admin implies all and write implies
  read at the gate too.

T2 amendment in test/e2e/serve-http-oauth.test.ts: assert
/.well-known/oauth-authorization-server includes all 5 scopes in
scopes_supported. Pre-v0.28 the list was hardcoded to ['read','write',
'admin'] and this assertion would have failed. (The test is
Postgres-gated; runs under bun run test:e2e with DATABASE_URL set.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): sources-ops module — atomic clone + symlink-safe cleanup

src/core/sources-ops.ts (~470 lines): pure async functions extracted from
src/commands/sources.ts so the CLI handlers and the new MCP ops share
one implementation.

addSource: D3 atomicity contract from the eng review.
  1. Validate id (matches existing SOURCE_ID_RE).
  2. Q4 pre-flight SELECT — fail loudly with structured `source_id_taken`
     before any clone work. Pre-fix the existing CLI used INSERT…ON
     CONFLICT DO NOTHING which silently no-op'd; with clone-first that
     would orphan the temp dir.
  3. parseRemoteUrl gate (delegates to isInternalUrl from url-safety.ts).
  4. Clone into $GBRAIN_HOME/clones/.tmp/<id>-<rand>/ via the new
     git-remote helpers.
  5. INSERT row with local_path=<final clone dir>, config.remote_url=<url>.
  6. fs.renameSync(tmp/, final/). Rollback on either-side failure unlinks
     the temp dir; rename-failed path also DELETEs the just-INSERTed row
     best-effort.

removeSource: clone-cleanup with realpath+lstat confinement matching
validateUploadPath() shape at src/core/operations.ts:61. String startsWith
is symlink-unsafe and would let $GBRAIN_HOME/clones/<id> → /etc resolve
out of the confine. Two defenses layered:
  - isPathContained (realpath-resolves both sides + parent-with-sep
    string check) rejects symlinks whose target falls outside the
    confine.
  - lstat-then-isSymbolicLink check refuses symlinks whose realpath
    happens to land back inside the confine (defense in depth).

getSourceStatus: returns clone_state via validateRepoState (the 6-state
decision tree from git-remote.ts). Lets a remote MCP caller diagnose
"healthy | missing | not-a-dir | no-git | url-drift | corrupted" without
SSH access to the brain host. listSources additionally exposes
remote_url so callers can see which sources are auto-managed.

recloneIfMissing: T4 follow-up for `gbrain sources restore` after the
clone dir was autopurged — re-clones via the same temp + rename
atomicity contract. Idempotent (returns false when clone is already
healthy).

test/sources-ops.test.ts (~470 lines, 24 tests): pre-flight collision
(Q4), happy paths for both --path and --url, all four D3 rollback paths
(clone-fail before INSERT, INSERT-fail after clone, rename-fail
post-INSERT, atomic temp-dir cleanup), symlink-target-OUTSIDE-clones
(realpath confinement), symlink-target-INSIDE-clones (lstat-check),
removeSource refuses to delete user-supplied paths, refuses "default"
source, getSourceStatus clone_state branches, T4 recloneIfMissing
recovery + idempotent + no-op for path-only sources, isPathContained
unit tests covering subtree / outside / symlink-escape / fail-closed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(operations): whoami + sources_{add,list,remove,status} MCP ops

Five new ops in src/core/operations.ts auto-flow through src/mcp/tool-defs.ts
so MCP clients (Claude Desktop, ChatGPT, Perplexity, OpenClaw) get them via
standard tools/list discovery — no SDK or transport code changes needed.

Operation.scope union widened to add 'sources_admin' and 'users_admin' (the
v0.28 hierarchy from src/core/scope.ts).

whoami (scope: read): introspect calling identity over MCP.
  - Returns `{transport: 'oauth', client_id, client_name, scopes, expires_at}`
    for OAuth clients (clientId starts with gbrain_cl_).
  - Returns `{transport: 'legacy', token_name, scopes, expires_at: null}`
    for grandfathered access_tokens.
  - Returns `{transport: 'local', scopes: []}` when ctx.remote === false.
    Empty scopes (NOT ['read','write','admin']) is the D2 decision —
    returning OAuth-shaped scopes for local callers would resurrect the
    v0.26.9 footgun where code conditionally trusted on
    `auth.scopes.includes('admin')` instead of `ctx.remote === false`.
  - Q3 fail-closed: throws unknown_transport when remote=true AND auth is
    missing OR ctx.remote is the literal `undefined` (cast bypass guard).
    A future transport that forgets to thread auth doesn't get a free
    pass.

sources_add (sources_admin, mutating): register a source by --path
  (existing v0.17 behavior) or --url (v0.28 federated remote-clone path).
  Calls into addSource from sources-ops.ts which owns the temp-dir +
  rename atomicity.

sources_list (read): list registered sources with page counts, federated
  flag, and remote_url. The remote_url field is new — lets a remote MCP
  caller see which sources are auto-managed.

sources_remove (sources_admin, mutating): cascade-delete a source +
  symlink-safe clone cleanup. Requires confirm_destructive: true when the
  source has data.

sources_status (read): per-source diagnostic returning clone_state
  ('healthy' | 'missing' | 'not-a-dir' | 'no-git' | 'url-drift' |
  'corrupted' | 'not-applicable') — lets a remote MCP caller diagnose a
  busted clone without SSH access to the brain host.

test/whoami.test.ts (9 tests): pinned transport-detection for all four
return shapes including Q3 fail-closed throw under both auth=undefined
and remote=undefined cast-bypass paths.

test/sources-mcp.test.ts (16 tests): op-metadata pins (scope, mutating,
localOnly), functional handler shape against PGLite, hasScope-driven
scope-enforcement smoke test simulating the serve-http.ts:673 gate
(read-only token rejected for sources_add; sources_admin token allowed;
admin token allowed for everything; gstack /setup-gbrain Path 4 token
covers all 4 ops), SSRF gate at the op layer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): re-clone fallback when clone is missing/no-git/corrupted

src/commands/sync.ts gets a v0.28-aware front-half. When the source has
config.remote_url, performSync calls validateRepoState before the existing
fast-forward pull path:

  - 'healthy'    → fall through to existing pull (unchanged)
  - 'missing'    → loud stderr "auto-recovery: re-cloning <id>", then
  'no-git'         recloneIfMissing handles the temp-dir + rename. Sync
  'not-a-dir'      continues from the freshly-cloned head.
  - 'corrupted'  → throw with structured hint pointing at sources remove
                   + add (no syncing wrong state).
  - 'url-drift'  → throw with hint pointing at the (deferred) sources
                   rebase-clone command.

Closes the operator-confidence gap: rm -rf $GBRAIN_HOME/clones/<id>/ no
longer breaks future syncs. The next sync sees the missing dir and
recovers via the recorded URL.

src/core/operations.ts: extend ErrorCode with 'unknown_transport' so
whoami's Q3 fail-closed path types check.

test/sources-resync-recovery.test.ts (12 tests): full validateRepoState
state matrix exercised under fake-git, recloneIfMissing recovery from
each degraded state, idempotent on healthy clones, the sync.ts:320
integration path that drives the recovery.

test/sources-ops.test.ts + test/sources-mcp.test.ts: drop the
GBRAIN_PGLITE_SNAPSHOT-disable line so these tests stop forcing cold
init across the parallel-shard runner. With snapshot allowed, init time
drops from 6+s to ~50ms and parallel runs stay under the 5s hook
timeout.

test/sources-mcp.test.ts: tighten scope literal-type so tsc keeps the
union narrow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): sources add --url + restore re-clone, thin-wrapper refactor

src/commands/sources.ts now delegates the data-mutation work to
src/core/sources-ops.ts (added in the previous commit). The CLI handler
parses argv, calls into addSource, and formats output.

Two new flags on `gbrain sources add`:
  - `--url <https-url>` : federated remote-clone path (clone + INSERT +
    rename, atomic rollback on failure).
  - `--clone-dir <path>` : override the default
    $GBRAIN_HOME/clones/<id>/ destination.

Validation rejects mutually-exclusive `--url` + `--path`. Errors from
the ops layer (SourceOpError) propagate through the CLI's standard
error wrapper in src/cli.ts so existing tests that assert throw shape
keep passing.

`gbrain sources restore <id>` (T4 from eng review): if the source has a
remote_url AND the on-disk clone was autopurged, call recloneIfMissing
before declaring success. Clone errors print a WARN with recovery
hints rather than failing the restore — the DB row is what restore
guarantees; the clone is best-effort.

54 sources-related tests pass (existing test/sources.test.ts +
sources-ops + sources-mcp).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor,cycle): orphan-clones surface + autopilot purge phase (P1)

addSource's atomicity contract uses a temp dir that gets renamed to the
final clone path. If the process is SIGKILL'd between clone-finish and
rename, the temp dir orphans on disk. Without sweeping these, a brain
server accumulates gigabytes over months of failed `sources add --url`
attempts.

Two layers:

1. `gbrain doctor` now surfaces stale entries. A new orphan_clones check
   walks $GBRAIN_HOME/clones/.tmp/, names anything older than 24h, and
   prints a warn with disk-byte estimate. Operators see the leak before
   `df` complains.

2. The autopilot cycle's existing `purge` phase grows a substep that
   nukes .tmp/ entries past the same 72h TTL the page-soft-delete purge
   uses. Operator behavior stays uniform across all soft-delete-style
   surfaces.

Both layers are filesystem-only (no DB). On a brain that never used
--url cloning, both are no-ops.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* build(admin): scope checkboxes source from scope-constants mirror + dist

admin/src/pages/Agents.tsx Register Client modal:
  - useState default sources from ALLOWED_SCOPES_LIST (defaulting `read`
    to true, others false; unchanged UX for the common case).
  - Scope checkbox map iterates ALLOWED_SCOPES_LIST instead of the old
    hardcoded ['read','write','admin'].

Without this commit, even with the v0.28.1 server-side scope hierarchy,
operators registering an OAuth client from the admin UI cannot tick the
new sources_admin / users_admin scopes — defeats the whole gstack
/setup-gbrain Path 4 unblock.

The drift-check CI gate (scripts/check-admin-scope-drift.sh) ensures
this list stays in sync with src/core/scope.ts going forward.

admin/dist/* rebuilt via `cd admin && bun run build`. Old hash bundle
removed; new bundle (224.96 kB / 68.70 kB gzip).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.28.1 — remote-source MCP + scope hierarchy + whoami

VERSION + package.json: bump to 0.28.1 (per CLAUDE.md branch-scoped
versioning rule — this branch adds substantial new features on top of
v0.28.0).

CHANGELOG.md: new top-level entry for v0.28.1 in the gstack/Garry voice
(no AI vocabulary, no em dashes, real numbers + commands). Lead
paragraph names what the user can now do that they couldn't before.
"Numbers that matter" table calls out the +5 MCP ops, +2 OAuth scopes,
and the 4-to-0 SSH-step number for gstack /setup-gbrain Path 4. "What
this means for you" closer ties the work to the operator workflow shift.
"To take advantage of v0.28.1" block has paste-ready upgrade commands
including the admin SPA rebuild step. Itemized changes section
describes the architecture cleanly without exposing scope-string
internals to public attack-surface enumeration (per CLAUDE.md
responsible-disclosure rule).

TODOS.md: file 6 follow-ups under a new "Remote-source MCP follow-ups
(v0.28.1)" section: token rotation, migration introspection in
get_health, Accept-header friendliness, sources rebase-clone for
URL-drift recovery, --filter=blob:none partial-clone option, and the
chunker_version PGLite-schema parity codex caught.

README.md: short subsection under the existing sources CLI listing
that names the new --url flag and what auto-recovery does. Capability
framing (no scope-string enumeration).

llms.txt + llms-full.txt: regenerated via `bun run build:llms` so the
documentation bundle reflects the v0.28.1 entry. The build-llms
generator's drift check passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): sources-remote-mcp — full gstack /setup-gbrain Path 4 round-trip

Spins up `gbrain serve --http` against real Postgres with a fake-git binary
in PATH (so `git clone` is exercised end-to-end without network), registers
two OAuth clients (sources_admin + read-only), mints tokens, calls the new
v0.28.1 MCP ops via /mcp, and asserts the gstack /setup-gbrain Path 4 flow
works end to end.

12 tests cover the full lifecycle:
- whoami over HTTP MCP returns transport=oauth + the right scopes
- /.well-known/oauth-authorization-server advertises all 5 scopes
- sources_add: clone fires, INSERT lands, row carries config.remote_url
- sources_status: clone_state=healthy after add
- sources_list: surfaces remote_url for the new source
- SSRF rejection: sources_add with RFC1918 URL fails at parseRemoteUrl gate
- Scope enforcement: read-only token gets insufficient_scope on sources_add
- Read-only token CAN call sources_list (read-scoped op)
- ALLOWED_SCOPES allowlist: CLI register-client rejects bogus scope
- Recovery: rm clone dir + sources_status reports clone_state=missing
- sources_remove: cascades + cleans up the auto-managed clone dir

Subprocess env threading replicates the v0.26.2 bun execSync inheritance
pattern — bun does NOT inherit process.env mutations, so every CLI
subprocess call passes env: { ...process.env } explicitly.

Cleanup contract mirrors test/e2e/serve-http-oauth.test.ts: revoke any
clients we registered, force-kill the server subprocess on SIGTERM
timeout, surface cleanup failures to stderr without throwing so real
test failures aren't masked.

The base table list in helpers.ts (ALL_TABLES) doesn't include sources
or oauth_clients, so this test explicitly truncates them in beforeAll
to avoid Q4 pre-flight collisions on re-run.

Skipped gracefully when DATABASE_URL is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: codex adversarial review — confine remote sources_admin + close SSRF gaps

Pre-ship adversarial review (codex exec) caught five issues. Four ship in
this commit; the fifth (DNS rebinding) is filed as v0.28.x follow-up.

CRITICAL — `sources_admin` tokens over HTTP MCP could plant content at any
host path. The MCP op exposed `path` and `clone_dir` to remote callers; the
op layer trusted them verbatim, then auto-recovery's rm -rf on degraded
state turned that into arbitrary delete primitives. src/core/operations.ts
sources_add handler now drops both fields when ctx.remote !== false. Local
CLI keeps the override (operator trust). Loud logger.warn when a remote
caller tries — visible in the SSE feed without leaking values.

HIGH — Steady-state `git pull --ff-only` bypassed GIT_SSRF_FLAGS entirely.
The legacy helper at src/commands/sync.ts:192 spawned git without the
-c http.followRedirects=false -c protocol.{file,ext}.allow=never
--no-recurse-submodules set that cloneRepo applies. Every recurring sync
was reopening the redirect/submodule/protocol bypass. Routed the call site
at sync.ts:381 through pullRepo from git-remote.ts so initial clone and
ongoing pull share one defensive flag set.

MEDIUM — listSources ignored its `include_archived` flag. The op
advertised the param but the function destructured it as `_opts` and
queried every row. Archived sources' ids, local_paths, and remote_urls
were leaking to read-scoped MCP callers by default. Filter in SQL
(`WHERE archived IS NOT TRUE` unless the flag is set) so archived rows
never reach the wire.

PARTIAL HIGH — IPv6 ULA fc00::/7 and link-local fe80::/10 were not in
the isInternalUrl bypass list. Only ::1/:: and IPv4-mapped IPv6 were
blocked. Added regex-based ULA + link-local rejection to url-safety.ts.

Test coverage:
- test/git-remote.test.ts: 4 new IPv6 cases (ULA fc-prefix + fd-prefix,
  link-local fe80::, public IPv6 still allowed).
- test/sources-mcp.test.ts: 3 new cases pinning the remote/local
  asymmetry (clone_dir override silently ignored over MCP, path nulled,
  local CLI keeps the override).
- test/sources-mcp.test.ts: 2 new cases for include_archived honored.

DNS rebinding (codex finding #3): the current gate is lexical only.
A deliberate attacker who controls a hostname's A/AAAA records can still
resolve to an internal IP. Closing this requires async DNS resolution +
revalidation; filed as v0.28.x follow-up in TODOS.md so the API change
surface (parseRemoteUrl becomes async, every caller updates) lands in
its own PR.

323 tests pass (9 files); 4071 unit tests pass (full suite).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.28.1 → v0.28.2 (master collision)

Caught after PR creation. master is at v0.28.1 already; this branch
forked from garrytan/v0.28-release at v0.28.0 and naively bumped to
v0.28.1 without checking the master queue. CI version-gate would have
rejected at merge time (requires VERSION strictly greater than
master's).

Root cause: I bumped VERSION mechanically during plan implementation
(echo "0.28.1" > VERSION) without consulting the queue-aware allocator
at bin/gstack-next-version. /ship Step 12's idempotency check then
classified state as ALREADY_BUMPED and the workflow's "queue drift"
comparison was the safety net I should have hit — but I skipped it.

Files updated:
- VERSION + package.json: 0.28.1 → 0.28.2
- CHANGELOG.md: header + "To take advantage of v0.28.2" subsection
- README.md: sources --url note version reference
- TODOS.md: 7 follow-up entries' version references
- llms.txt + llms-full.txt: regenerated

PR title rewrite via gstack-pr-title-rewrite.sh handled in a separate
gh pr edit call; CI version-gate now passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-06 21:14:34 -07:00