docs(eval): BrainBench methodology + reference entries

Seam disclosure table, metric formulas, pre-registered expectations,
determinism posture, gate governance, gold methodology (incl. the blind
double-label receipt), interop (foreign runners, memory-verbs conformance
kit conversion path, naming note). eval-bench.md links the two gate layers;
KEY_FILES.md carries the per-file entries.

docs/architecture/KEY_FILES.md

@@ -68,6 +68,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
 - `src/commands/reindex-aliases.ts` — `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source <id>]`: backfills the free-text alias layer for EXISTING pages whose frontmatter `aliases:` predate the alias table (the import-time projection covers new + changed pages). Reads each page's frontmatter `aliases:`, writes via `engine.setPageAliases`. Idempotent + convergent (setPageAliases replaces a page's alias set) so no op-checkpoint needed; walks `listAllPageRefs` (cheap cross-source enumeration), `--source` narrows. Pinned by `test/search/reindex-aliases.test.ts`.
 - `src/eval/retrieval-quality/harness.ts` + `src/commands/eval-retrieval-quality.ts` + `test/fixtures/retrieval-quality/namedthing.jsonl` — NamedThingBench, the retrieval-quality eval that makes the named-thing-miss incident impossible to reintroduce silently. Seven query families, each a distinct failure class: `title-substring` (the direct regression), `generic-to-named` (tourist label → named thing), `alias-synonym` (declared alias / romanization → canonical), `multi-chunk-dilution` (one strong chunk among many weak — stresses max-pool), `short-vs-rich`, `graph-relationship` (guardrail), `hard-negative` (precision guard, must NOT return a page). `gbrain eval retrieval-quality <fixture.jsonl>` runs it with hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects a `SearchFn` (CLI uses `hybridSearch`, tests stub) so it's engine-agnostic. Metric glossary entries (`hit@1`/`hit@3`) added to `src/core/eval/metric-glossary.ts`. Pinned by `test/eval-retrieval-quality.test.ts` + `test/retrieval-quality-harness.test.ts`.
 - `docs/architecture/RETRIEVAL.md` + `docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md` — retrieval-pipeline architecture reference + the named-thing-miss incident write-up (root cause, the five-layer fix, the eval that pins it).
+- `src/eval/brainbench/` + `src/commands/eval-brainbench.ts` — BrainBench, the cross-harness memory conformance suite (`gbrain eval brainbench`; methodology in `docs/eval/BRAINBENCH.md`). `types.ts` carries the PUBLISHED interchange shapes (fixture/gold/result/baseline — mirrored as JSON Schemas in `evals/brainbench/schema/`; breaking changes bump the schema versions). `fixtures.ts`: strict loader/validator + corpus `fixtures_hash` (covers fixture AND gold files); a `gold` key inside a fixture turn is a validation error — gold is SEALED in the gold dir and adapters only ever see sanitized `PublicTurn`s. `seed.ts`: fail-fast hermetic seeding (`importFromContent` noEmbed + NULL-embedding `insertFact`; any non-`imported` status ⇒ `SeedError` ⇒ fixture `seed_failed` ⇒ run exit 2). `adapters/shared.ts`: ONE `runReflexPipeline` all three adapters drive with declarative config (pointer budget, suppression mode) — cross-harness comparability is structural; `openclaw.ts` (seam `production`, the shipped pipeline), `claude-code.ts` (seam `contract`; exports the UserPromptSubmit hook wire types a future integration implements; no conversation memory by contract), `codex.ts` (seam `contract`; static entity-index preamble whose slugs deliberately don't count as injections + ≤1 per-turn fragment). `metrics/`: know-to-ask (+false-fire anti-gaming companion), push (micro-averaged P/R), write-back (drives the PRODUCTION conversation→facts pipeline via the injectable-extractor seam; gold extractor in CI, real extractor under `--llm`), continuity (writer→reader pairs on a shared brain through DIFFERENT adapters; pointer-injection OR stored-fact keyword probe). `harness.ts`: ONE in-memory PGLite per run + `resetTables` between fixtures (longmemeval engine-sharing pattern); read-only suites share one seeding across all adapters; emits per-(harness×suite) cells + re-scoreable turn rows; `source_isolation_violations` counted per turn and gated at zero. `scoreboard.ts`: markdown render, canonical diff-stable committed baseline (4-decimal rounding, sorted keys, receipts excluded), `compareBaselines` with main-baseline governance — same-hash count-aware gate vs corpus-bless mode (the committed baseline must byte-match the run; regressions vs main require a `justification`). The CLI brings its own PGLite (cli.ts routes before connectEngine), writes `--out` as the canonical CI artifact, and terminates via an explicit grace-tick `process.exit(verdict)` (0 pass / 1 regression / 2 error) because PGLite stomps `process.exitCode` and Bun discards queued stdout on exit. `runBrainBenchCore()` is the in-process entry `eval run-all` uses (one record per sweep, `EvalRunRecord` schema_version 3, `mode: 'n/a'`). Pinned by `test/brainbench-*.test.ts` + `test/eval-brainbench-e2e.test.ts`.
+- `evals/brainbench/` — the committed BrainBench corpus: 141 fixtures (135 generated + 6 hand-authored spike) / 241 gold-annotated turns across 7 categories (kta-pos/kta-neg/push/write-back/continuity/multi-source/adversarial), ~15% holdout (excluded from the CI gate, scored in published `--include-holdout` runs). `generator/gen.ts` rebuilds the corpus byte-identically (Mulberry32, seed 42; whole-cloth fictional universe from curated synthetic name pools so scenario privacy is structural; prose is template-synthesized with PRNG-selected variants — deliberately no LLM pass, difficulty stays controlled; several know-to-ask variants intentionally exercise documented v1 reflex limits so the baseline measures the roadmap). `gold/` is sealed; `schema/` is the foreign-runner contract (gbrain-evals drives the suite as a subprocess via `--fixtures DIR --gold DIR --json --out FILE`); `baselines/main.json` is the committed gate baseline; `_ledger.json` records counts/seed/rebuild command. CI: the `.github/workflows/test.yml` `brainbench` job + `scripts/ci-brainbench-gate.sh` (fetches MAIN's baseline via `git show origin/master:…` — a PR cannot rewrite what it's compared against; first-landing path runs ungated) + `scripts/render-brainbench-delta.ts` (compact step-summary/PR-body delta block from the `--out` artifact). Privacy: `scripts/check-synthetic-corpus-privacy.sh` scans `evals/brainbench/{fixtures,gold}` in `bun run verify`.
 - `src/core/types.ts` extension + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` — the wiring layer for the retrieval cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op uses a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query), surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`.
 - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. Sub-subcommand dispatch on `args[0]` routes `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. `gbrain eval cross-modal` is in the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
 - `src/commands/eval-cross-modal.ts` — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (partial cost guardrail) via the shared `resolveCycleDefault(explicit, isTty)` in `src/core/eval/cycle-default.ts`; the cost-estimate banner appends `cycleDefaultSuffix(...)` (`for 1 cycle(s) (non-interactive default; --cycles N for more)`) when the value is the silent non-TTY fallback, so the 1-vs-3 difference isn't hidden. Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit<T>(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`.

docs/eval-bench.md

@@ -8,6 +8,15 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
 [`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
 that lives on top of that format.
 
+If you're touching **memory behavior** rather than retrieval ranking — the
+Retrieval Reflex push path, conversation→facts write-back, cross-session
+continuity, source isolation — the gate for that layer is **BrainBench**
+(`gbrain eval brainbench`): see [`eval/BRAINBENCH.md`](./eval/BRAINBENCH.md).
+The two stack: this doc's capture→baseline→replay loop gates query-level
+result sets; BrainBench gates the memory behaviors above them, with its own
+committed baseline (`evals/brainbench/baselines/main.json`) compared against
+MAIN's copy in CI so a PR can't self-approve a regression.
+
 ## v0.41 update — the LOOP is now real
 
 Before v0.41, you could capture eval rows and replay them but nothing

docs/eval/BRAINBENCH.md

@@ -0,0 +1,147 @@
+# BrainBench — cross-harness memory conformance methodology
+
+BrainBench generalizes gbrain's internal eval surface into a reproducible,
+cross-harness benchmark for agent memory. It operationalizes the four failure
+modes of the agent-memory thesis: **know-to-ask** (nobody has a push path),
+**push precision/recall** (the intrusion budget must be enforced),
+**write-back fidelity** (memory write is even less solved than read), and
+**cross-session continuity** (continuity that survives the harness hop).
+Every subsequent memory PR must move — or hold, with a recorded justification —
+a BrainBench number to merge.
+
+Operator quickstart, corpus layout, and fixture-authoring rules live in
+[`evals/brainbench/README.md`](../../evals/brainbench/README.md). This document
+is the methodology: what the numbers mean, what they deliberately do not mean,
+and how the gate governs change.
+
+## Seam disclosure (read this before comparing rows)
+
+Every scoreboard row carries a `seam` column:
+
+| Harness | Seam | What the row actually measures |
+|---|---|---|
+| `openclaw` | **production** | The shipped OpenClaw context-engine pipeline, byte-for-byte (`extractCandidates` → `resolveEntitiesToPointers`, 3-pointer budget, prior-context suppression, markdown pointer block). |
+| `claude-code` | **contract** | gbrain's memory primitives driven through the UserPromptSubmit hook wire contract (`{prompt, session_id, cwd}` in → `{hookSpecificOutput.additionalContext}` out, exported from `src/eval/brainbench/adapters/claude-code.ts`). 2-pointer budget; NO conversation memory — a hook sees only the current prompt, so suppression is off and the re-injection cost is visible as `false_fire_rate`. |
+| `codex` | **contract** | The fragments model: a static entity-index preamble (computed once, slugs not counted as injections) + at most ONE per-turn fragment. Measures how much push quality degrades when injection is mostly static. |
+
+**Contract rows do NOT measure third-party harness behavior.** They measure
+gbrain's primitives under each harness's injection-shape constraints. The rows
+are comparable because fixtures, brain, and gold are identical — only the seam
+contract varies. When a real integration lands (the hooks/fragments PR), its
+adapter swaps transport (exec the real hook) and flips to `production` with
+continuous numbers. Also not graded, by design: the production orchestrator's
+config gate, integration heartbeat, and 1500 ms timeout wrapper.
+
+All three adapters drive ONE shared pipeline (`adapters/shared.ts`) with
+declarative configs — comparability is structural, not disciplined.
+
+## Metrics (formulas)
+
+All micro-averaged per (harness × suite) cell; registered in
+`src/core/eval/metric-glossary.ts` (plain-English in
+[`METRIC_GLOSSARY.md`](METRIC_GLOSSARY.md)); JSON output carries one
+`_meta.metric_glossary` block.
+
+- `know_to_ask_failure_rate` = |should-retrieve turns where injected ∩ (gold ∪ acceptable) = ∅| / |should-retrieve turns|. Lower better.
+- `false_fire_rate` = |stay-silent turns with any injection| / |stay-silent turns|. Lower better. Anti-gaming companion: "always inject" cannot win both.
+- `push_precision` = Σ|injected ∩ (gold ∪ acceptable)| / Σ|injected| over turns with injection. `acceptable_slugs` count for precision, not recall.
+- `push_recall` = Σ|injected ∩ gold| / Σ|gold| over should-retrieve turns. Pointer budgets cap this by design.
+- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
+- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
+- `continuity_rate` = |decision probes recalled by the reader| / |probes|, over writer→reader pairs replayed through DIFFERENT harness adapters on a shared brain. A probe succeeds via pointer injection or stored-fact keyword lookup. Headline = mean over off-diagonal (writer ≠ reader) pairs.
+- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant.
+- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
+- `extraction_recall` / `extraction_precision` — `--llm` runs only: the real extractor's output vs gold keyword probes.
+
+### What know-to-ask deliberately means in v1
+
+It grades the **deterministic injection decision** — the Reflex pipeline that
+ships at the seam. The agent never "knows to ask"; the reflex pushes. An
+agent-LLM-in-the-loop replay (did the *model* issue a retrieval call when the
+reflex stayed silent?) is **pre-registered as the `--live` extension**:
+fixture-compatible, seeded, N-repeat methodology — and unimplemented. No LLM
+grading is faked in v1.
+
+### Difficulty is stratified on purpose
+
+Several know-to-ask variants exercise documented v1 reflex limits (lowercase
+mentions, surname-only references — `src/core/context/entity-salience.ts`).
+Gold records what SHOULD happen; the committed baseline records what the
+current system does (`know_to_ask_failure_rate` ≈ 0.15 at v1). The gap is the
+measured roadmap, not a bug in the bench.
+
+## Pre-registered expectations (v1, recorded before the first published run)
+
+1. The production seam (openclaw) leads `push_recall` strictly: 3-pointer > 2-pointer > 1-fragment budgets. *(Observed at landing: 0.81 / 0.65 / 0.45.)*
+2. The no-suppression contract (claude-code) is the only seam with `false_fire_rate` > 0. *(Observed: 0.02–0.03.)*
+3. `write_back_fidelity` = 1.0 and `provenance_accuracy` = 1.0 in deterministic mode — the production pipeline must not lose or mis-attribute gold facts it was handed. Anything below 1.0 is a pipeline bug, not benchmark noise.
+4. `source_isolation_violations` = 0 everywhere.
+5. `push_precision` = 1.0 at v1 (exact-match resolution arms cannot inject an irrelevant page on this corpus); expected to dip below 1.0 when fuzzy/semantic resolution lands — that dip is the precision/recall trade made visible.
+
+## Determinism & statistical posture
+
+The harness is deterministic end-to-end: regex extraction + SQL resolution
+(zero LLM, zero embeddings — facts seed with NULL embeddings; keyword/alias
+arms carry retrieval), seeded PRNG corpus, one in-memory PGLite reset between
+fixtures. Two runs produce identical metrics, so N-repeat error bars are
+meaningless here (stddev = 0 by construction, the gbrain-evals "deterministic
+adapters" convention) and the gate can be exact: **any flipped gold item is a
+real behavior change.** Bootstrap/CI discipline applies to the future `--live`
+and `--llm` published runs, which are model-stochastic.
+
+## Gate governance (decision 4 — why a PR can't self-approve)
+
+CI (`.github/workflows/test.yml` `brainbench` job, local parity
+`scripts/ci-brainbench-gate.sh`) fetches the baseline **from main**
+(`git show origin/master:evals/brainbench/baselines/main.json`) and compares
+HEAD's fresh run against it:
+
+- **Same `fixtures_hash`** → count-aware gate: any newly-failed gold item, any
+  adverse gated-metric move, or any isolation violation fails (exit 1).
+- **Different hash** (the PR changed fixtures) → **corpus-bless mode**: the
+  PR's committed baseline must EXACTLY match HEAD's actual run (the file
+  cannot lie; exit 2 until `--update-baseline` is re-run), and any adverse
+  move vs main's baseline requires a `justification` string in the committed
+  baseline — visible in the PR diff, judged by the reviewer.
+- `--allow-regression "reason"` is the local one-off escape hatch; the reason
+  is recorded in the run output. It is not available to CI.
+
+The committed baseline is diff-stable by construction (metrics rounded to 4
+decimals, keys sorted, receipts excluded) so its diff IS the PR's visible
+BrainBench delta. Holdout fixtures (~15%) are excluded from the gate and
+scored only in published runs (`--include-holdout`).
+
+## Gold methodology
+
+Gold derives from the corpus generator (the same PRNG step that authors a turn
+authors its annotation, so gold-vs-text drift is structurally impossible for
+generated fixtures), plus hand-authored spike fixtures that froze the schema.
+A 10% double-label validation pass (independent agent review of fixture text vs
+gold, blind to the generator's intent) is run at corpus-change time; its
+receipt is recorded in the corpus `_ledger.json` and any disagreement is a
+fixture bug to fix, not a tolerance to average over.
+
+## Interop
+
+- **Foreign runners (gbrain-evals):** the subprocess contract is
+  `gbrain eval brainbench --fixtures DIR --gold DIR --json --out FILE`;
+  schemas in `evals/brainbench/schema/`. The sibling gbrain-evals repo wires
+  this as `eval/runner/brainbench-memory.ts` with a published scorecard.
+- **Memory-verbs conformance kit (Cathedral 1):** conformance scenarios
+  convert to BrainBench fixtures via the published fixture schema
+  (`schema_version` 1) once that wave lands — the conversion path is the
+  schema itself; no bespoke importer is required.
+- **Naming note:** "BrainBench" historically also names the in-house
+  retrieval corpus in the sibling gbrain-evals repo (the 145-query relational
+  suite, Cat taxonomy) and `test/cathedral-ii-brainbench.test.ts` (v0.20.0
+  code-graph recall pins). This suite — the cross-harness memory conformance
+  bench — is the generalization the name now primarily refers to; the older
+  references stand unchanged.
+
+## Extends docs/eval-bench.md
+
+The capture → baseline → replay loop in [`eval-bench.md`](../eval-bench.md)
+gates *retrieval result sets* at the query level. BrainBench gates the
+*memory behaviors* above them. The two share the receipts discipline and the
+.gbrain-evals run ledger (`EvalRunRecord` v3; brainbench records once per
+sweep under `mode: 'n/a'`).