Merge origin/master into garrytan/skillopt-eval-explainer

Resolve version collision: master shipped v0.42.10.0 (wikilink
global-basename) while the skillopt-eval-explainer work was in-branch as
v0.42.9.0. Bump the wave to v0.42.11.0 (strictly greater than master) per
the version-locations IRON RULE; keep both CHANGELOG entries.

CLAUDE.md: keep the branch's restructured thin orientation; master's only
change was a per-file-index annotation for the wikilink feature, which this
branch deliberately moved out of CLAUDE.md. Carried that documentation into
docs/architecture/KEY_FILES.md as a current-state link-extraction.ts entry
so nothing is lost.

Regenerated llms.txt / llms-full.txt via build:llms. Version trio audited
(VERSION = package.json = CHANGELOG = 0.42.11.0); typecheck + current-state
guard green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -32,8 +32,12 @@ start here.
 ## Read this order
 
 1. `./AGENTS.md` (this file) — install + operating protocol.
-2. [`./CLAUDE.md`](./CLAUDE.md) — architecture reference, key files, trust boundaries,
-   test layout.
+2. [`./CLAUDE.md`](./CLAUDE.md) — orientation + resolver: architecture, cross-cutting
+   invariants, the reference map, inline ship rules. It routes to on-demand detail docs:
+   [`./docs/architecture/KEY_FILES.md`](./docs/architecture/KEY_FILES.md) (per-file index —
+   read a file's entry before editing it), [`./docs/TESTING.md`](./docs/TESTING.md) (test
+   tiers + isolation lint + E2E lifecycle), and
+   [`./docs/architecture/thin-client.md`](./docs/architecture/thin-client.md) (remote-MCP seam).
 3. [`./docs/architecture/brains-and-sources.md`](./docs/architecture/brains-and-sources.md)
    — the two-axis mental model (brain = which DB, source = which repo in the DB). Every
    query routes on both axes. Read before writing anything that touches brain ops.
@@ -108,7 +112,9 @@ diff-aware subset during fast iteration on a focused branch. Requires Docker
 Manual path: `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin
 up the test Postgres container, run `bun run test:e2e`, tear it down).
 
-Ship via the `/ship` skill, not by hand.
+Ship via the `/ship` skill, not by hand. The full release + contributor process
+(CHANGELOG voice, version-locations sync, PR conventions, community-PR-wave) lives in
+[`./docs/RELEASING.md`](./docs/RELEASING.md); read it before shipping.
 
 ## Privacy
 

CHANGELOG.md

@@ -2,6 +2,132 @@
 
 All notable changes to GBrain will be documented in this file.
 
+## [0.42.11.0] - 2026-06-03
+
+**Self-improving skills can no longer cheat. When you run `gbrain skillopt` to let
+a skill rewrite itself, it now has to prove the change actually helps on a set of
+tasks it wasn't optimized against — and for the skills gbrain ships, it won't
+overwrite them in place unless you hand it that independent check.**
+
+Here is the problem. `gbrain skillopt` treats a skill's SKILL.md as something it
+can edit and re-score against a benchmark, keeping edits that score higher. The
+trap: an edit can score higher on its own benchmark while quietly getting worse at
+the real job (classic "teaching to the test"). Until now the safety net for that —
+a held-out check — was documented but never actually wired in, the run's report
+showed a fake baseline score of 0, and a "final test" score was never computed. So
+you couldn't tell from the receipt whether a skill genuinely improved.
+
+This release makes the loop honest. Pass `--held-out <file.jsonl>` (a set of tasks
+with different IDs than your benchmark) and a candidate that climbs the benchmark
+but slips on the held-out set is refused. The run report now records the real
+baseline score and a real test-set score, so "did this skill get better" is a
+number you can read. `--no-mutate` finally writes the proposed rewrite to disk for
+review (it was a stub), and `--max-runtime-min` is actually enforced.
+
+For the ~47 skills gbrain ships, the bar is higher: mutating one in place now
+*requires* `--held-out` with at least 5 independent tasks. Without it you get a
+`proposed.md` to review instead of a silent overwrite. The held-out file must use
+task IDs disjoint from the benchmark — point it at a copy of the benchmark and the
+run refuses, because an overlapping check can't catch overfitting.
+
+The `run_skillopt` MCP tool got a security tighten in the same pass: it validates
+the skill name and confines benchmark/held-out paths to the skills directory for
+remote callers, so an admin token can't read arbitrary host files through it.
+
+## To take advantage of v0.42.11.0
+
+`gbrain upgrade` is all you need — these are behavior changes to an existing
+command, no migration.
+
+1. **Optimize a user skill with the new safety net:**
+   ```bash
+   gbrain skillopt my-skill --held-out skills/my-skill/held-out.jsonl
+   ```
+   The held-out file is the same JSONL shape as the benchmark, with task IDs that
+   do NOT appear in the benchmark.
+2. **Optimize a bundled (shipped) skill in place** — now requires the held-out
+   check; otherwise it writes `proposed.md` for review:
+   ```bash
+   gbrain skillopt brain-ops --allow-mutate-bundled --held-out skills/brain-ops/held-out.jsonl
+   ```
+3. **Read the honest receipt:** `gbrain skillopt ... --json` now reports
+   `baseline_sel_score`, `best_sel_score`, `baseline_test_score`, and `test_score`.
+
+### Itemized changes
+
+#### Added
+- **Held-out validation gate (F11) is now wired into the optimizer loop.** `--held-out <path>`
+  (CLI), `held_out_path` (background job + `run_skillopt` MCP op), and `heldOutPath`
+  (batch/fleet) load an independent task set; the gate runs at checkpoint acceptance
+  and blocks any candidate whose held-out score regresses below baseline. Previously
+  `runHeldOutGate` existed but nothing called it.
+- **Final-test eval.** After optimization, the best skill and the baseline are scored
+  on the held-out test split; receipts now carry `test_score` + `baseline_test_score`.
+- **Shared `scoreSkillOnTasks` primitive** (`validate-gate.ts`) used by the baseline
+  eval, final-test, held-out gate, and external eval harnesses so they can't drift.
+
+#### Changed
+- **Bundled-skill mutation requires a non-empty held-out set (>=5 tasks).** Enforced in
+  core mutation policy (`assertBundledMutationHeldOut`), so it fires for every entry
+  point — CLI, batch, fleet, background job, and the `run_skillopt` MCP op. Without it,
+  the run hard-refuses (exit 2) and points you at `proposed.md`.
+- **Held-out must be independent of the benchmark.** A held-out file sharing task IDs
+  with the benchmark is rejected (an overlapping check can't catch overfitting).
+- **Honest receipts.** `baseline_sel_score` is the real measured baseline (was hardcoded
+  to 0).
+- **`run_skillopt` MCP op hardening:** validates `skill_name` is kebab-case and confines
+  caller-supplied benchmark/held-out paths to the skills directory for remote callers.
+
+#### Fixed
+- **Multi-turn tool loops work again.** Any agent loop that calls a tool and feeds the
+  result back (`gbrain skillopt` rollouts AND production background subagent jobs) was
+  crashing the moment the model called a tool, with "messages do not match the
+  ModelMessage[] schema". The shipped AI SDK had tightened its message + tool-schema
+  validation; the gateway now wraps tool schemas correctly and converts tool results into
+  the structured shape the SDK expects, so the loop round-trips. Surfaced end-to-end by the
+  SkillOpt real-LLM eval — the kind of bug only running the feature against a live model
+  catches.
+- **Budget-capped Haiku runs no longer score a silent zero.** Claude Haiku 4.5's
+  canonical (dateless) model id was missing from the pricing table, so any cost-capped run
+  on Haiku (`gbrain skillopt --max-cost`, eval harnesses) hit "no pricing entry" on the
+  first model call of every rollout, which the validation gate then swallowed as a `0`
+  score — a pricing crash that looked exactly like a real "0 out of N" measurement. The
+  pricing entry is added, and the gate now re-throws budget/pricing errors loudly instead
+  of recording a hollow zero. Surfaced by the SkillOpt real-LLM eval.
+- **The optimizer now knows what the scorer rewards.** `gbrain skillopt`'s reflect step
+  was only shown a pass/fail score and the agent's transcript, never the benchmark's
+  success criteria — so on a skill judged by structure (e.g. "must include a Confidence:
+  line") it proposed plausible-but-off edits that never satisfied the check, every
+  candidate scored 0, the validation gate rejected them all, and the skill never changed.
+  The reflect prompt now includes a plain-English description of exactly how the output is
+  scored, with an instruction to satisfy it through genuine content, not empty keywords.
+  In an end-to-end run this took a deficient skill from 0.00 to 1.00 on a held-out set it
+  never trained on. Reward-hacking is still defended by the independent held-out gate.
+  Surfaced by the SkillOpt real-LLM eval.
+- **`--no-mutate` now writes `proposed.md`** with the winning rewrite (was a stub that
+  wrote nothing).
+- **`--max-runtime-min` is enforced** via a wall-clock deadline between optimization steps.
+
+### For contributors
+- Eval-internal ablation knobs on `runSkillOpt` (not exposed on the CLI): `reflectMode`
+  (`'both'`/`'failure-only'`), `disableValidationGate`, and `optimizerMode`
+  (`'reflect'`/`'one-shot-rewrite'`), recorded in the receipt + audit for replayability.
+  These drive the SkillOpt benchmark suite in the sibling `gbrain-evals` repo.
+- New tests: `test/skillopt/rollout.test.ts`, held-out + one-shot-rewrite unit cases, and
+  e2e coverage for the held-out gate (block/allow), bundled enforcement, no-mutate write,
+  runtime deadline, receipt honesty, and no-DB-pollution.
+- **CLAUDE.md restructured into a thin orientation + resolver (592KB → 39KB).** The per-file
+  index, command surface, test discipline, thin-client routing, and the verbose release
+  process moved to on-demand docs (`docs/architecture/KEY_FILES.md`, `docs/TESTING.md`,
+  `docs/architecture/thin-client.md`, `docs/RELEASING.md`); CLAUDE.md keeps the North Star,
+  architecture + cross-cutting invariants, the IRON RULES, and a reference map that routes to
+  the detail. Per-file entries are now current-state only — release history lives in
+  CHANGELOG + git. `scripts/check-key-files-current-state.sh` (wired into `bun run verify`)
+  fails the build if append-only version narration returns to the reference docs or CLAUDE.md
+  grows past its cap, so the bloat cannot recur. The llms bundle drops from ~740KB to ~204KB.
+  `scripts/ci-cache-hash.sh` now keeps the relocated policy docs test-affecting so a change to
+  them still invalidates the CI cache.
+
 ## [0.42.10.0] - 2026-06-02
 
 **Wikilinks like `[[struktura]]` that point at pages in another folder finally connect.** Until now, if you wrote `[[struktura]]` in `concepts/knowledge-graph.md` and the actual page lived at `projects/struktura.md`, GBrain silently dropped the link from its graph. Obsidian users saw a dense web of connections in their vault and a thin, broken graph inside GBrain. The issue reporter had 71 wikilinks across 20 pages — GBrain captured 12.
@@ -66,6 +192,7 @@ Closes https://github.com/garrytan/gbrain/issues/972.
 - `KNOWN_CONFIG_KEYS` (in `src/core/config.ts`) adds `'link_resolution'` and `'link_resolution.global_basename'` so `gbrain config set ...` accepts the new key without `--force`.
 - Tests: 38 new cases pinning the contract. `test/link-extraction.test.ts` adds 17 cases covering `WIKILINK_GENERIC_RE` shape (anchor / display / strip / escape paths), the `extractEntityRefs` pass-2c no-double-emit invariant, `resolveBasenameMatches` multi-match + index-built-once + missing-`getAllSlugs` degradation, and the `extractPageLinks` opt routing under both flag states. `test/extract-fs.test.ts` adds 11 cases for the pure-function helpers (`resolveBasenameMatchesFromSlugs`, `resolveSlugAll`) plus 3 round-trip tests of the issue's exact repro inside a PGLite brain. `test/doctor.test.ts` adds 7 cases for the new doctor check (skip / ok / warn paths + the cross-surface wiring source-grep). `test/e2e/global-basename-pglite.test.ts` adds 7 end-to-end cases against an in-memory PGLite brain covering FS-source, DB-source, and put_page auto-link paths under both flag states.
 - PR #1233 from @rayers contributed the kernel of the resolver-side approach (the generic wikilink regex + slug-tail index pattern). This PR keeps that mechanism, makes it opt-in via the new config flag, replaces the first-write-wins lookup with multi-match return, and extends the coverage to the FS-source path that the issue's repro actually hits.
+
 ## [0.42.8.0] - 2026-06-01
 
 **Scraped junk stops landing in your brain as if it were real content, and when something looks off, your agent gets told instead of being left to guess.**

TODOS.md

@@ -1,5 +1,34 @@
 # TODOS
 
+## v0.42.9.0 SkillOpt eval-readiness follow-ups (v0.42+)
+
+Deferred from the v0.42.9.0 wave (held-out gate wiring + ENFORCE + ablation opts).
+Adversarial-review findings that are real but not blockers — the shipped fixes are
+complete and tested; these are hardening/cleanup.
+
+- [ ] **P2 — Extract `promoteCandidate` helper (DRY).** The candidate-promotion
+  sequence (optional `runHeldOutGate` → branch on `mutateDecision.mutate` →
+  `acceptCandidate` else `writeProposed` → set outcome/finalText) is duplicated between
+  the one-shot-rewrite block and the main loop accept branch in
+  `src/core/skillopt/orchestrator.ts`. A future change to the held-out gate or promotion
+  policy must be applied in two places. Extract a shared `promoteCandidate({...})`. Deferred
+  this wave to avoid a >20-line refactor of freshly-tested accept-path code.
+- [ ] **P2 — Harden bundled-skill detection.** `getBundledSkillContext`
+  (`src/core/skillopt/bundled-skill-gate.ts`) only sets `isBundled` when the skills dir was
+  resolved via the `install_path` tier. If the same bundled `skills/` is found via
+  `cwd_walk_up` / `repo_root` / `$GBRAIN_SKILLS_DIR`, `isBundled=false` and the D16 ENFORCE
+  never fires (same weakness governs `--allow-mutate-bundled` itself — pre-existing, not a
+  v0.42.9.0 regression). Fix: compare realpaths against the canonical bundled skills dir
+  independent of detection source.
+- [ ] **P3 — Preflight cost estimate is blind to ablation opts.** `preflight.ts:estimateCost`
+  doesn't know `optimizerMode`/`disableValidationGate`/`reflectMode`, so `--dry-run`
+  over-counts for `one-shot-rewrite` / `failure-only`. Low impact (eval-internal knobs;
+  runtime BudgetTracker enforcement is correct, no overspend) — just a lying preview.
+- [ ] **P3 — `maxRuntimeMin` is enforced only between optimization steps.** The baseline
+  eval, per-step held-out gate, one-shot rewrite, and final-test `scoreSkillOnTasks` calls
+  run unbounded LLM rollouts with no deadline check. BudgetTracker still caps spend; the
+  runtime guarantee is best-effort. Thread the deadline + abortSignal into those phases, or
+  document runtime as best-effort.
 ## v0.42.7.0 extract-in-default-loop follow-ups (v0.42+)
 
 Filed from the v0.42.2.0 wave (#1696 link/timeline extraction freshness

VERSION

@@ -1 +1 @@
-0.42.10.0
+0.42.11.0

docs/RELEASING.md

@@ -0,0 +1,433 @@
+# Releasing & contributing (gbrain)
+
+The full release + contributor process. CLAUDE.md keeps the ship-critical IRON RULES
+inline (the Version-locations table, branch=workspace, post-ship `/document-release`,
+the Privacy + Responsible-disclosure rules, PR-title-version-first, never-hand-roll-ship)
+and points here for everything else. **Before any ship, read this in full. Use `/ship` —
+never hand-roll a release.**
+
+## Pre-ship requirements
+
+Before shipping (/ship) or reviewing (/review), always run the full test suite.
+Two equivalent paths:
+
+**Path A — local CI gate (recommended, v0.23.1+):**
+- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
+  tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
+  fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
+  what nightly Tier 1 catches. Spins up + tears down postgres automatically via
+  `docker-compose.ci.yml`. Override the host port with
+  `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
+- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
+  (`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
+  schema/skills/package.json changes. Fast iteration during a focused branch.
+
+**Path B — manual lifecycle (still supported):**
+- `bun test` — unit tests (no database required)
+- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
+  run `bun run test:e2e`, then tear it down.
+
+Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
+
+**Always run typecheck before pushing.** `bun test` (the bun runner)
+skips TypeScript type checking — it only enforces runtime behavior.
+Three ways to actually gate on types:
+
+1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
+   plus the four shell pre-checks (`check-jsonb-pattern.sh`,
+   `check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
+   `check-wasm-embedded.sh`) before the runner. Use this mid-branch.
+2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
+3. `bun run ci:local` — the full local CI gate from Path A.
+
+The trap is: writing a new test, running `bun test test/foo.test.ts`,
+seeing it pass, pushing — and CI's separate typecheck stage rejects an
+invalid type literal that the runner accepted. Caught one of these
+shipping the v0.23.2 round-trip E2E (`type: 'reflection'` is not a
+member of `PageType`). Run `bun run typecheck` once before push, even
+when only test files changed.
+
+
+## CHANGELOG + VERSION are branch-scoped
+
+**VERSION and CHANGELOG describe what THIS branch adds vs master, not how we got
+here.** Every feature branch that ships gets its own version bump and CHANGELOG
+entry. The entry is product release notes for users; it is not a log of internal
+decisions, review rounds, or codex findings.
+
+**Write the CHANGELOG entry at /ship time, not during development.** Mid-branch
+iterations, review rounds (CEO/Eng/Codex/DX), and implementation detours belong
+in the plan file at `~/.claude/plans/`, not in the CHANGELOG. One unified entry
+per branch, covering what the branch added vs the base branch.
+
+**Never edit a CHANGELOG entry that already landed on master.** If master has
+v0.18.2 and your branch adds features, bump to the next version (v0.19.0, not
+editing master's v0.18.2). When merging master into your branch, master may
+bring new CHANGELOG entries above yours — push your entry above master's
+latest and verify:
+
+- Does CHANGELOG have your branch's own entry separate from master's entries?
+- Is VERSION higher than master's VERSION?
+- Is your entry the topmost `## [X.Y.Z]` entry?
+- `grep "^## \[" CHANGELOG.md` shows a contiguous version sequence?
+
+If any answer is no, fix it before continuing.
+
+**CHANGELOG is for users, not contributors.** Write like product release notes:
+
+- Lead with what the user can now **do** that they couldn't before. Sell the capability.
+- Plain language, not implementation details. "You can now..." not "Refactored the..."
+- **Never mention internal artifacts**: plan file IDs, decision tags (D-CX-#, F-ENG-#),
+  review rounds, codex findings, subcontractor credits. These are invisible to users.
+- Put contributor-facing changes in a separate `### For contributors` section at the bottom.
+- Every entry should make someone think "oh nice, I want to try that."
+
+**What to omit:**
+- "Codex caught X that the CEO review missed" — private process detail.
+- "D-CX-3 split errors/warnings" — tag is meaningless to users; name the feature instead.
+- "Fix-wave PR #N supersedes #M" — supersede chains belong in PR bodies, not release notes.
+- "215 new cases, 3 decisions applied, 7 reviews cleared" — these are planning-mode metrics.
+
+**What to keep:**
+- The user-facing change: what commands exist now, what flag was added, what behavior fixed.
+- Numbers that mean something to the user: TTHW, commands that timed out before, detection counts.
+- Upgrade instructions: `gbrain upgrade` + any manual step if needed.
+- Credit to external contributors when a community PR was incorporated.
+
+## CHANGELOG voice + release-summary format
+
+**IRON RULE: the CHANGELOG describes what the user gets, not how the work
+happened.** Nobody reading release notes cares that codex caught a bug, that
+the plan went through CEO + eng review, that the migration was originally
+numbered v68 and renumbered to v79 during master merge, or that two
+review rounds caught architectural mistakes. The reader cares what
+`gbrain brainstorm` does and how to use it. If a fact only exists because
+of the development process, it does NOT belong in the CHANGELOG.
+
+**Specifically forbidden in CHANGELOG entries:**
+
+- Any mention of review processes (CEO review, eng review, codex review,
+  plan-eng-review, outside voice, adversarial review, autoplan, /review).
+- "What we caught and fixed before merging" sections. Bugs found pre-merge
+  are not changes — they're things that didn't ship.
+- Plan file references, plan IDs, plan decision tags (D1, D14, D-CDX-3).
+- Migration version drama ("originally v68", "renumbered to v77", "claimed
+  by parallel waves") — just say "Migration v79 adds X." If the user
+  cares about migration ordering, they read the diff.
+- Round counts, finding counts, decision counts ("25 findings across 2
+  rounds", "8 architectural decisions", "5/6 expansions accepted").
+- Names of internal collaborators ("codex caught", "the reviewer flagged",
+  "Claude noticed").
+- "Plan + reviews" summary bullets. The plan lives in `~/.claude/plans/`;
+  if a future reader wants the backstory they can grep there.
+- Any wording that frames a shipped feature as a *recovery* from a planning
+  mistake ("the first plan was wrong", "we corrected the approach", "the
+  shipped version supersedes the original design").
+
+**Smell test:** read the entry as a stranger who has never touched gbrain.
+If any sentence makes them think "why are you telling me this?", cut it.
+Every sentence in the release-summary AND in the itemized changes must
+answer one of three questions: *What can I now do? How do I use it? What
+should I watch for after I upgrade?*
+
+Every version entry in `CHANGELOG.md` MUST start with a release-summary section in
+the GStack/Garry voice — one viewport's worth of prose + tables that lands like a
+verdict, not marketing. The itemized changelog (subsections, bullets, files) goes
+BELOW that summary, separated by a `### Itemized changes` header.
+
+The release-summary section gets read by humans, by the auto-update agent, and by
+anyone deciding whether to upgrade. The itemized list is for agents that need to
+know exactly what changed.
+
+### Release-summary template
+
+**Iron rule: lead ELI10, get precise after.** The first ~150 words of every entry
+must be readable by someone who does NOT know gbrain's internals. No file paths,
+no function names, no internal constants, no acronyms (no "RRF", no "knobsHash",
+no "MODE_BUNDLES", no "CDX-4"), no jargon that requires reading the codebase to
+parse. Lead with the user-visible behavior change, in everyday English, like
+you're explaining it to a smart engineer who has never opened the repo.
+
+THEN, once the reader knows what shipped and why they'd care, drill into the
+precise details: real file paths, real function names, real config keys, real
+numbers. The precision part is required (the entry is also the technical record
+of what changed), but it lives AFTER the plain-English lead, never before it.
+
+The shape:
+
+1. **One-line bold headline.** What changed for the user, in human English. No
+   jargon. No internal terms. Example good: "Your search stops boosting weak
+   pages just because they have a lot of links pointing at them." Example bad:
+   "PostFusionOpts gains floorRatio; KNOBS_HASH_VERSION bumped 2→3."
+2. **Plain-English opener** (~3-5 sentences). Describe the problem this fixes in
+   everyday terms. Pretend the reader has a brain full of meeting notes and
+   people pages and wants to know if this release helps them. Concrete example
+   beats abstract description.
+3. **A "How to turn it on" or "How to use it" section** with paste-ready
+   commands. Real flags, real config keys. This is where precision starts.
+4. **A "What you'd see in a concrete example" or "The X numbers that matter"
+   section** with a table. Use everyday-language column headers ("Page",
+   "Match quality", "Has many backlinks?") even when the underlying mechanism
+   is technical. The table teaches what the feature does without requiring the
+   reader to understand how.
+5. **A "What's safe to know about" or "Things to watch" section** for caveats,
+   side effects, cache invalidation, mid-deploy notes. Still in plain language.
+6. **A "What we caught and fixed before merging" section** if the work went
+   through review (CEO/eng/codex/outside-voice). Translate review findings into
+   plain English. "We caught a stale-cache bug" beats "knobsHash() did not
+   include floorRatio in the v=2 hash input."
+7. **`### Itemized changes`** (precision lives here). File paths, function
+   names, types, constants, line numbers. This section is for engineers who
+   need to know exactly what moved.
+
+Voice rules (apply throughout):
+- No em dashes (use commas, periods, "...").
+- No AI vocabulary (delve, robust, comprehensive, nuanced, fundamental, etc.) or
+  banned phrases ("here's the kicker", "the bottom line", etc.).
+- Real numbers, real file names, real commands AFTER the ELI10 lead. Not "fast"
+  but "~30s on 30K pages." In the ELI10 lead, "fast enough that you won't
+  notice" or "~30 seconds even on a big brain."
+- Short paragraphs, mix one-sentence punches with 2-3 sentence runs.
+- Connect to user outcomes: "the agent does ~3x less reading" beats "improved
+  precision."
+- Be direct about quality. "Well-designed" or "this is a mess." No dancing.
+
+**The smell test:** if someone who has never opened gbrain reads the first 150
+words and walks away knowing what shipped and whether they care, the entry
+passes. If they need to grep the codebase to follow along, rewrite the lead.
+
+**Canonical examples in this CHANGELOG:** v0.35.6.0 (floor-ratio gate, written
+ELI10-lead-first), v0.34.4.0 (embed stale fix wave). Use those shapes when in
+doubt. Avoid the shape of entries that lead with internal constants or release
+mechanics; those exist in older history but should not be the model for new
+work.
+
+Source material to pull from:
+- CHANGELOG.md previous entry for prior context
+- Latest `gbrain-evals/docs/benchmarks/[latest].md` for headline numbers (sibling repo)
+- Recent commits (`git log <prev-version>..HEAD --oneline`) for what shipped
+- Don't make up numbers. If a metric isn't in a benchmark or production data, don't
+  include it. Say "no measurement yet" if asked.
+
+Target length: ~250-350 words for the summary. Should render as one viewport.
+
+### "To take advantage of v[version]" block (required, v0.13+)
+
+After the release-summary and BEFORE `### Itemized changes`, every `## [X.Y.Z]`
+entry MUST include a human-readable self-repair block under the heading
+`## To take advantage of v[version]`.
+
+Why: `gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`.
+This chain has a known weak link — `upgrade.ts` catches post-upgrade failures as
+best-effort (so the binary still works). When that chain silently fails, users end
+up with half-upgraded brains. The self-repair block gives them a paste-ready
+recovery path; the v0.13+ `~/.gbrain/upgrade-errors.jsonl` trail + `gbrain doctor`
+integration close the loop.
+
+Template (adapt the verify commands per release):
+
+```markdown
+## To take advantage of v[version]
+
+`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
+warns about a partial migration:
+
+1. **Run the orchestrator manually:**
+   ```bash
+   gbrain apply-migrations --yes
+   ```
+2. **Your agent reads `skills/migrations/v[version].md` the next time you interact with it.**
+   [One sentence on whether headless agents need manual action, or whether the
+   orchestrator already handled the mechanical side.]
+3. **Verify the outcome:**
+   ```bash
+   [release-specific verify commands, e.g. `gbrain graph ... --depth 2`]
+   gbrain stats
+   ```
+4. **If any step fails or the numbers look wrong,** please file an issue:
+   https://github.com/garrytan/gbrain/issues with:
+   - output of `gbrain doctor`
+   - contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
+   - which step broke
+
+   This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
+```
+
+**Skip this block** for patches that are pure bug fixes with zero user-facing action
+(rare). If the release has a schema migration, data backfill, or new feature the
+user needs to verify, the block is required.
+
+The v0.13.0 entry in CHANGELOG.md is the canonical example.
+
+### Itemized changes (the existing rules)
+
+Below the release summary, write `### Itemized changes` and continue with the
+detailed subsections (Knowledge Graph Layer, Schema migrations, Security hardening,
+Tests, etc.). Same rules as before:
+
+- Lead with what the user can now DO that they couldn't before
+- Frame as benefits and capabilities, not files changed or code written
+- Make the user think "hell yeah, I want that"
+- Bad: "Added GBRAIN_VERIFY.md installation verification runbook"
+- Good: "Your agent now verifies the entire GBrain installation end-to-end, catching
+  silent sync failures and stale embeddings before they bite you"
+- Bad: "Setup skill Phase H and Phase I added"
+- Good: "New installs automatically set up live sync so your brain never falls behind"
+- **Always credit community contributions.** When a CHANGELOG entry includes work from
+  a community PR, name the contributor with `Contributed by @username`. Contributors
+  did real work. Thank them publicly every time, no exceptions.
+
+### Reference: v0.12.0 entry as canonical example
+
+The v0.12.0 entry in CHANGELOG.md is the canonical example of the format. Match its
+structure for every future version: bold headline, lead paragraph, "numbers that
+matter" with BrainBench-style before/after table, "what this means" closer, then
+`### Itemized changes` with the detailed sections below.
+
+## Version migrations
+
+Create a migration file at `skills/migrations/v[version].md` when a release
+includes changes that existing users need to act on. The auto-update agent
+reads these files post-upgrade (Section 17, Step 4) and executes them.
+
+**You need a migration file when:**
+- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
+  existing users need to set it up, not just new installs)
+- New SKILLPACK section with a MUST ADD setup requirement
+- Schema changes that require `gbrain init` or manual SQL
+- Changed defaults that affect existing behavior
+- Deprecated commands or flags that need replacement
+- New verification steps that should run on existing installs
+- New cron jobs or background processes that should be registered
+
+**You do NOT need a migration file when:**
+- Bug fixes with no behavior changes
+- Documentation-only improvements (the agent re-reads docs automatically)
+- New optional features that don't affect existing setups
+- Performance improvements that are transparent
+
+**The key test:** if an existing user upgrades and does nothing else, will their
+brain work worse than before? If yes, migration file. If no, skip it.
+
+Write migration files as agent instructions, not technical notes. Tell the agent
+what to do, step by step, with exact commands. See `skills/migrations/v0.5.0.md`
+for the pattern.
+
+## Migration is canonical, not advisory
+
+GBrain's job is to deliver a canonical, working setup to every user on upgrade.
+Anything that looks like a "host-repo change" — AGENTS.md, cron manifests,
+launchctl units, config files outside `~/.gbrain/` — is a GBrain migration
+step, not a nudge we leave for the host-repo maintainer. Migrations edit host
+files (with backups) to make the canonical setup real. Exceptions: changes
+that require human judgment (content edits, renames that break semantics,
+host-specific handler registration where shell-exec would be an RCE surface).
+Everything mechanical ships in the migration.
+
+**Test:** if shipping a feature requires a sentence that starts with "in
+your AGENTS.md, add…" or "in your cron/jobs.json, rewrite…", the migration
+orchestrator should be doing that edit, not the user.
+
+**The exception is host-specific code.** For custom Minion handlers
+(host-specific integrations like inbox sweeps or third-party API scanners), shipping them as a
+data file the worker would exec is an RCE surface. Those get registered in
+the host's own repo via the plugin contract (`docs/guides/plugin-handlers.md`);
+the migration orchestrator emits a structured TODO to
+`~/.gbrain/migrations/pending-host-work.jsonl` + the host agent walks the
+TODOs using `skills/migrations/v0.11.0.md` — stays host-agnostic, still
+canonical.
+
+
+## Schema state tracking
+
+`~/.gbrain/update-state.json` tracks which recommended schema directories the user
+adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
+reads this during upgrades to suggest new schema additions without re-suggesting
+things the user already declined. The setup skill writes the initial state during
+Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
+
+## GitHub Actions SHA maintenance
+
+All GitHub Actions in `.github/workflows/` are pinned to commit SHAs. Before shipping
+(`/ship`) or reviewing (`/review`), check for stale pins and update them:
+
+```bash
+for action in actions/checkout oven-sh/setup-bun actions/upload-artifact actions/download-artifact softprops/action-gh-release gitleaks/gitleaks-action; do
+  tag=$(grep -r "$action@" .github/workflows/ | head -1 | grep -o '#.*' | tr -d '# ')
+  [ -n "$tag" ] && echo "$action@$tag: $(gh api repos/$action/git/ref/tags/$tag --jq .object.sha 2>/dev/null)"
+done
+```
+
+If any SHA differs from what's in the workflow files, update the pin and version comment.
+
+
+## PR descriptions cover the whole branch
+
+Pull request titles and bodies must describe **everything in the PR diff against the
+base branch**, not just the most recent commit you made. When you open or update a
+PR, walk the full commit range with `git log --oneline <base>..<head>` and write the
+body to cover all of it. Group by feature area (schema, code, tests, docs) — not
+chronologically by commit.
+
+This matters because reviewers read the PR body to understand what's shipping. If
+the body only covers your last commit, they miss everything else and can't review
+properly. A 7-commit PR with a body that describes commit 7 is worse than no body
+at all — it actively misleads.
+
+When in doubt, run `gh pr view <N> --json commits --jq '[.commits[].messageHeadline]'`
+to see what's actually in the PR before writing the body.
+
+## Community PR wave process
+
+Never merge external PRs directly into master. Instead, use the "fix wave" workflow:
+
+1. **Categorize** — group PRs by theme (bug fixes, features, infra, docs)
+2. **Deduplicate** — if two PRs fix the same thing, pick the one that changes fewer
+   lines. Close the other with a note pointing to the winner.
+3. **Collector branch** — create a feature branch (e.g. `garrytan/fix-wave-N`), cherry-pick
+   or manually re-implement the best fixes from each PR. Do NOT merge PR branches directly —
+   read the diff, understand the fix, and write it yourself if needed.
+4. **Test the wave** — verify with `bun test && bun run test:e2e` (full E2E lifecycle).
+   Every fix in the wave must have test coverage.
+5. **Close with context** — every closed PR gets a comment explaining why and what (if
+   anything) supersedes it. Contributors did real work; respect that with clear communication
+   and thank them.
+6. **Ship as one PR** — single PR to master with all attributions preserved via
+   `Co-Authored-By:` trailers. Include a summary of what merged and what closed.
+
+**Community PR guardrails:**
+- Always AskUserQuestion before accepting commits that touch voice, tone, or
+  promotional material (README intro, CHANGELOG voice, skill templates).
+- Never auto-merge PRs that remove YC references or "neutralize" the founder perspective.
+- Preserve contributor attribution in commit messages.
+
+## Checking out PRs from garrytan-agents
+
+`garrytan-agents` is the AI-authored PR account and is NOT a collaborator on
+this repo. Its PRs live in a fork, so GitHub Actions triggered by
+`pull_request` events on those PRs do not receive base-repo secrets. Any CI
+job that needs `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or similar will fail
+with empty-env auth errors, regardless of what's set on the base repo. This
+is a GitHub security default, not a config bug.
+
+When the user says "check out <PR link>" and the PR is from `garrytan-agents`
+(or any other non-collaborator fork), move the branch into the base repo
+before running CI:
+
+1. `gh pr checkout <N>` — pull down the fork's branch. Note the PR number and
+   head branch name (`gh pr view <N> --json headRefName --jq .headRefName`).
+2. `git push origin HEAD:<branch-name>` — push the same branch to the base
+   repo (origin points at `garrytan/gbrain`, not the fork). This is the move
+   that gives CI access to secrets.
+3. `gh pr close <N> --comment "moving to base-repo branch for secret access"`
+   — close the fork PR so the queue stays clean.
+4. `gh pr create --base master --head <branch-name>` — open the replacement
+   PR from the base-repo branch. **Preserve the original PR's title and body
+   verbatim** (`gh pr view <N> --json title,body`); contributor attribution
+   moves to a `Co-Authored-By:` trailer if needed.
+
+Why this over alternatives: adding `garrytan-agents` as a collaborator, or
+flipping the repo-wide "send secrets to fork PRs" toggle, both broaden
+secret distribution to every fork PR from that account or any fork. Moving
+the branch keeps secret scope tight to just the one PR being shipped.
+

docs/TESTING.md

@@ -0,0 +1,289 @@
+# Testing (gbrain repo)
+
+On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
+only.
+
+### Test command tiers
+
+Seven test command tiers, each with a clear scope:
+
+| Command | What it runs | Wallclock | When to use |
+|---|---|---|---|
+| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
+| `bun run verify` | CI's authoritative pre-test gate set: `check:privacy && check:jsonb && check:progress && check:wasm && bun run typecheck`. The 4 checks `.github/workflows/test.yml` runs on shard 1 + typecheck. Single source of truth — CI literally calls `bun run verify`. | ~12s (wasm-compile dominates) | Before pushing; before `/ship`. |
+| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
+| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
+| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; runs at `--max-concurrency=1`). | ~1s per quarantined file | Debugging a specific quarantined file. |
+| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
+| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
+
+### CI vs local: intentionally divergent file sets
+
+- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` 4-way, which uses FNV-1a hash bucketing and INCLUDES `*.slow.test.ts`. CI EXCLUDES `*.serial.test.ts` from the hash buckets and runs them on shard 1 via `bun run test:serial` at `--max-concurrency=1` — keeping serial files out of the hash buckets is what preserves the `mock.module` quarantine (top-level mocks in serial files would otherwise leak into the parallel files they share a shard process with). CI is the ground truth for "did everything pass."
+- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
+
+This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
+
+### Failure-first logging
+
+When `bun run test` finds any failure, the wrapper:
+
+1. Writes failure blocks (each prefixed with `--- shard N: <test name> ---`) to `.context/test-failures.log` (workspace-local, gitignored). On systems without a writable `.context/`, falls back to `/tmp/gbrain-test-failures.log`.
+2. Prints a loud stderr banner with the absolute log path, plus the last 30 lines of the failure log inlined. Banner survives `| head` / `| tail` / agent-side log truncation.
+3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
+4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
+
+If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
+
+### File taxonomy
+
+- `*.test.ts` → fast loop (parallel 8-shard fan-out).
+- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
+- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
+- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
+- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
+- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
+
+### Test-isolation lint and helpers
+
+The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
+
+| Rule | What it bans | Fix |
+|---|---|---|
+| **R1** | `process.env.X = ...`, bracket assignment, `delete process.env.X`, `Object.assign(process.env, ...)`, `Reflect.set(process.env, ...)` | Use `withEnv()` from `test/helpers/with-env.ts`, OR rename file to `*.serial.test.ts` |
+| **R2** | `mock.module(...)` anywhere in the file | Rename file to `*.serial.test.ts` (no DI on production code for testability) |
+| **R3** | `new PGLiteEngine(` outside ~50 lines after a `beforeAll(` line | Use the canonical block (below) inside `beforeAll(` |
+| **R4** | Files creating `new PGLiteEngine(` without `engine.disconnect(` inside an `afterAll(` block | Add `afterAll(() => engine.disconnect())` |
+
+Files that violated these rules at the isolation-lint baseline are listed in `scripts/check-test-isolation.allowlist`. **The allow-list MUST shrink over time** — never add new entries.
+
+#### Canonical PGLite block (R3 + R4 compliant)
+
+Every test file that needs a PGLite engine should use this exact pattern:
+
+```ts
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+  engine = new PGLiteEngine();
+  await engine.connect({});
+  await engine.initSchema();
+});
+
+afterAll(async () => {
+  await engine.disconnect();
+});
+
+beforeEach(async () => {
+  await resetPgliteState(engine);
+});
+```
+
+Why this exact shape: `beforeAll` creates a single engine per file (PGLite WASM cold-start + initSchema is ~20s); `beforeEach` truncates user data via `resetPgliteState` ("two orders of magnitude faster" than fresh-engine-per-test); `afterAll` disconnects so the engine doesn't leak across file boundaries within a shard process.
+
+#### `withEnv` pattern (R1 fix)
+
+```ts
+import { withEnv } from './helpers/with-env.ts';
+
+test('reads OPENAI_API_KEY', async () => {
+  await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
+    expect(loadConfig().openai_key).toBe('sk-test');
+  });
+});
+
+// Delete a var (override is undefined):
+await withEnv({ GBRAIN_HOME: undefined }, fn);
+
+// Multiple keys:
+await withEnv({ A: '1', B: '2', C: undefined }, fn);
+```
+
+`withEnv` saves the prior value of every key it touches and restores via try/finally — including when the callback throws. **It is cross-test safe but NOT intra-file concurrent-safe.** `process.env` is process-global; two `test.concurrent()` calls in the same file both touching the same key will race. Files using `withEnv` stay outside the `test.concurrent()` codemod's eligibility filter.
+
+#### When to quarantine instead of fix
+
+Rename to `*.serial.test.ts` when:
+- The file uses `mock.module(...)` (R2 — there's no clean fix without changing production code).
+- The file is genuinely env-coupled (e.g. `gbrain-home-isolation.test.ts`, `claw-test-cli.test.ts`) — module-load env readers + ESM caching defeat dynamic-import-after-env tricks.
+- The file's tests intentionally share state across `it()` boundaries.
+
+Quarantine count cap: 10 (informational). Beyond that, push back on the design.
+
+### Unit test inventory
+
+`bun test` runs all tests without a database. E2E tests skip gracefully when `DATABASE_URL` is not set.
+
+Unit tests and what they cover:
+
+- `test/markdown.test.ts` — frontmatter parsing; `splitBody` sentinel precedence, horizontal-rule preservation, `inferType` wiki subtypes.
+- `test/chunkers/recursive.test.ts` — chunking.
+- `test/parity.test.ts` — operations contract parity.
+- `test/cli.test.ts` — CLI structure.
+- `test/config.test.ts` — config redaction.
+- `test/files.test.ts` — MIME/hash.
+- `test/import-file.test.ts` — import pipeline.
+- `test/upgrade.test.ts` — schema migrations.
+- `test/file-migration.test.ts` — file migration.
+- `test/file-resolver.test.ts` — file resolution.
+- `test/import-resume.test.ts` — import checkpoints.
+- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion.
+- `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage.
+- `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns.
+- `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.
+- `test/setup-branching.test.ts` — setup flow.
+- `test/slug-validation.test.ts` — slug validation.
+- `test/storage.test.ts` — storage backends.
+- `test/supabase-admin.test.ts` — Supabase admin.
+- `test/yaml-lite.test.ts` — YAML parsing.
+- `test/check-update.test.ts` — version check + update CLI.
+- `test/pglite-engine.test.ts` — PGLite engine, all BrainEngine methods including `addLinksBatch` / `addTimelineEntriesBatch` (empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100) plus `connect()` error-wrap assertion (original error nested, #223 link in message, lock released).
+- `test/engine-factory.test.ts` — engine factory + dynamic imports.
+- `test/integrations.test.ts` — recipe parsing, CLI routing, recipe validation.
+- `test/publish.test.ts` — content stripping, encryption, password generation, HTML output.
+- `test/backlinks.test.ts` — entity extraction, back-link detection, timeline entry generation.
+- `test/lint.test.ts` — LLM artifact detection, code fence stripping, frontmatter validation.
+- `test/report.test.ts` — report format, directory structure.
+- `test/skills-conformance.test.ts` — skill frontmatter + required sections validation.
+- `test/resolver.test.ts` — RESOLVER.md coverage, routing validation; round-trip that every quoted RESOLVER.md trigger matches a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md resolves to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`.
+- `test/search.test.ts` — RRF normalization, compiled truth boost, cosine similarity, dedup key.
+- `test/sql-ranking.test.ts` — source-boost helpers: longest-prefix-match in SQL CASE, `detail=high` temporal-bypass, three-meta-char LIKE escape (`%`, `_`, `\`), single-quote SQL-literal doubling, env override parsing for `GBRAIN_SOURCE_BOOST` + `GBRAIN_SEARCH_EXCLUDE`, `resolveBoostMap` / `resolveHardExcludes` merge semantics.
+- `test/dedup.test.ts` — source-aware dedup, compiled truth guarantee, layer interactions.
+- `test/intent.test.ts` — query intent classification: entity/temporal/event/general.
+- `test/eval.test.ts` — retrieval metrics: `precisionAtK`, `recallAtK`, `mrr`, `ndcgAtK`, `parseQrels`.
+- `test/check-resolvable.test.ts` — resolver reachability, MECE overlap, gap detection, proximity-based DRY detection, `extractDelegationTargets` coverage.
+- `test/dry-fix.test.ts` — auto-fix: three shape-aware expander pure-function tests; five guards (working-tree-dirty, no-git-backup, inside-code-fence, already-delegated within 40 lines, ambiguous-multi-match, block-is-callout).
+- `test/doctor-fix.test.ts` — `gbrain doctor --fix` CLI integration: dry-run preview, apply path, JSON output shape.
+- `test/backoff.test.ts` — load-aware throttling, concurrency limits, active hours.
+- `test/fail-improve.test.ts` — deterministic/LLM cascade, JSONL logging, test generation, rotation.
+- `test/transcription.test.ts` — provider detection, format validation, API key errors.
+- `test/enrichment-service.test.ts` — entity slugification, extraction, tier escalation.
+- `test/data-research.test.ts` — recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping.
+- `test/minions.test.ts` — Minions job queue: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, `child_done` inbox, attachments, removeOnComplete/Fail, `max_stalled` clamp/default/plumbing coverage.
+- `test/extract.test.ts` — link extraction, timeline extraction, frontmatter parsing, directory type inference.
+- `test/extract-db.test.ts` — `gbrain extract --source db`: typed link inference, idempotency, `--type` filter, `--dry-run` JSON output.
+- `test/extract-fs.test.ts` — `gbrain extract --source fs`: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard for the N+1 dedup bug.
+- `test/link-extraction.test.ts` — canonical `extractEntityRefs` both formats, `extractPageLinks` dedup, `inferLinkType` heuristics, `parseTimelineEntries` date variants, `isAutoLinkEnabled` config.
+- `test/graph-query.test.ts` — direction in/out/both, type filter, indented tree output.
+- `test/features.test.ts` — feature scanning, brain_score calculation, CLI routing, persistence.
+- `test/file-upload-security.test.ts` — symlink traversal, cwd confinement, slug + filename allowlists, remote vs local trust.
+- `test/query-sanitization.test.ts` — prompt-injection stripping, output sanitization, structural boundary.
+- `test/search-limit.test.ts` — `clampSearchLimit` default/cap behavior across `list_pages` and `get_ingest_log`.
+- `test/repair-jsonb.test.ts` — JSONB repair: TARGETS list, idempotency, engine-awareness.
+- `test/migrations-v0_12_2.test.ts` — JSONB-repair orchestrator phases: schema → repair → verify → record.
+- `test/orphans.test.ts` — orphans command: detection, pseudo filtering, text/json/count outputs, MCP op.
+- `test/postgres-engine.test.ts` — `statement_timeout` scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against a reintroduced bare `SET statement_timeout`.
+- `test/sync.test.ts` — sync logic + regression guard asserting top-level `engine.transaction` is not called.
+- `test/sync-concurrency.test.ts` — `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping; `shouldRunParallel()` explicit-bypasses-floor contract; `parseWorkers()` validation rejecting `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars.
+- `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract.
+- `test/sync-failures.test.ts` — `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` `AcknowledgeResult` shape + backfill on legacy entries.
+- `test/doctor.test.ts` — doctor command; assertions that `jsonb_integrity` scans the four JSONB write sites and `markdown_body_completeness` is present.
+- `test/utils.test.ts` — shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics.
+- `test/build-llms.test.ts` — `llms.txt`/`llms-full.txt` generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement.
+- `test/oauth.test.ts` — OAuth 2.1 provider: register, getClient, `client_credentials` grant exchange, `authorization_code` flow with PKCE challenge/verifier, refresh token rotation, `verifyAccessToken` with both OAuth + legacy `access_tokens` fallback, `revokeToken`, `sweepExpiredTokens`; contract test asserting `scope` + `localOnly` annotations on all operations; `coerceTimestamp` unit cases (null/undefined/string/number/throw-on-NaN); NULL-`expires_at`-as-expired contract for both refresh + access token paths; cascade-delete contract asserting `revoke-client` purges `oauth_tokens` + `oauth_codes` via FK CASCADE; cross-client isolation (wrong-client attempt MUST reject AND rightful owner MUST still succeed atomically afterward); empty-string `redirect_uri` bypass guard; PKCE DCR public-client gate (`token_endpoint_auth_method: "none"` returns no `client_secret`, default `client_secret_post` clients get the one-time-reveal secret, `getClient` NULL→undefined normalization, full PKCE `/authorize` → `/token` round-trip against a public client).
+- `test/mcp-dispatch-summarize.test.ts` — `summarizeMcpParams` invariants: declared-keys allow-list intersection, attacker-key-name leak guard (unknown keys counted not named), 1KB byte bucketing for size-probe defense, missing op falls through to fully-redacted shape, declared-keys sorted for deterministic output.
+- `test/trust-boundary-contract.test.ts` — fail-closed trust semantics under cast bypass: `ctx.remote === undefined` treated as remote/untrusted at every flipped call site; `as any` and `Partial<>` spreads can't downgrade trust by accident.
+- `test/check-resolvable-cli.test.ts` — CLI wrapper: exit codes, JSON envelope shape, AGENTS.md fallback chain.
+- `test/regression-v0_16_4.test.ts` — `findRepoRoot` regression guard, hermetic startDir parameterization.
+- `test/repo-root.test.ts` — `findRepoRoot` walk semantics + default-arg parity; the 4-tier `autoDetectSkillsDir` fallback chain (`$OPENCLAW_WORKSPACE` → `~/.openclaw/workspace` → repo-root → `./skills`); RESOLVER.md/AGENTS.md filename precedence; explicit-env-wins-over-repo-root; tier-0 `$GBRAIN_SKILLS_DIR` valid/invalid/precedence-over-`OPENCLAW_WORKSPACE`; the install-path walk in `autoDetectSkillsDirReadOnly`; no-drift on primary success; `AUTO_DETECT_HINT` + `AUTO_DETECT_HINT_READ_ONLY` content; regression guard asserting the shared `autoDetectSkillsDir` MUST NEVER return `'install_path'` source (how the read-path/write-path split stays safe).
+- `test/resolver-merge.test.ts` — multi-file resolver merge: `findAllResolverFiles` empty / RESOLVER.md-only / AGENTS.md-only / both-present (RESOLVER.md first); `checkResolvable` merge semantics across `skills/RESOLVER.md` + `../AGENTS.md` for the OpenClaw layout where the skillpack ships a thin RESOLVER.md and the real dispatcher lives at the workspace root; dedup by `skillPath` (first occurrence wins); AGENTS.md-at-workspace-root works alone.
+- `test/filing-audit.test.ts` — filing audit: `writes_pages` / `writes_to` frontmatter, filing-rules JSON validation.
+- `test/skill-brain-first.test.ts` — shared frontmatter parser; `analyzeSkillBrainFirst` compliance ladder across 9 fixtures under `test/fixtures/brain-first-skills/` (compliant-callout, compliant-phase, compliant-position, exempt-frontmatter, missing-brain-first, multi-pattern, negation-prose, no-external, typo-frontmatter); offset helpers; external-lookup regex shape; audit snapshot+diff transition logic; `FORMERLY_HARDCODED_EXEMPT` regression absorption.
+- `test/routing-eval.test.ts` — fixture parsing, structural routing, `ambiguous_with`, Haiku tie-break layer.
+- `test/skill-manifest.test.ts` — skill manifest parser: drift detection, managed-block markers.
+- `test/skillify-scaffold.test.ts` — `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures.
+- `test/skillpack-install.test.ts` — `gbrain skillpack install` managed-block install / update / no-clobber semantics.
+- `test/skillpack-sync-guard.test.ts` — sync-guard: bundled skills stay byte-identical to `skills/` source.
+- `test/http-transport.test.ts` — HTTP transport: bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass; dispatch.ts round-trip; invalid_params; application/json response shape (not SSE); CORS default-deny + allowlist; body cap on Content-Length AND chunked; two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB); `mcp_request_log` audit on success + auth_failed.
+- `test/restart-sweep.test.ts` — `recipes/restart-sweep.md` inlined script: sentinel-anchored fenced-block extraction with salted tmp filenames to bypass ESM cache; constructor-time env reads (proves no module-load snapshot); idempotency layer load/save/atomic-tmp-rename/corrupt-JSON-recovery/30-day-prune; `(sessionKey, lastAlertedAt)` cooldown gate with 6h threshold; AGGRESSIVE-gate two-state tests; execFile argv shape proving shell metachars in `OPENCLAW_TELEGRAM_GROUP` cannot reach `/bin/sh`; real-`\n`-not-literal alert formatting; `GBRAIN_HOME` state path override.
+- `test/eval-longmemeval.test.ts` — LongMemEval harness, hermetic with no `DATABASE_URL` and no API keys: PGLite create + reset over runtime-enumerated `pg_tables`, infrastructure-table preservation across resets, JSONL question parsing, retrieval-only and answer-gen modes via stubbed `ThinkLLMClient`, `--limit` cutoff, `--keyword-only` vs hybrid, default `--expansion=off` behavior, perf gate (p50 < 30ms / p99 < 50ms warm reset+import+search on Apple Silicon), `--help` works without a configured brain, fixture round-trip via `test/fixtures/longmemeval-mini.jsonl`.
+- `test/longmemeval-sanitize.test.ts` — sanitization parity pinning that `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` is the single source of truth (adding a pattern there must cover both `<take>` framing and `<chat_session>` framing, no per-surface regex drift).
+- `test/openai-compat-multimodal.test.ts` — gateway's openai-compatible multimodal path: happy-path single + multi-input embedding, unauthenticated proxy mode, dimension-mismatch guard (throws `AIConfigError` with model id + observed + expected pre-storage), default-dim fallback when recipe declares `default_dims`, HTTP 401 / 400 / malformed-JSON / non-array error paths, regression that the existing Voyage `/multimodalembeddings` recipe still routes through its dedicated path. Hermetic via the `__setEmbedTransportForTests` seam.
+- `test/serve-stdio-lifecycle.test.ts` — `MCP_STDIO=1` env guard: stdin EOF does NOT trigger shutdown when the env is set, SIGTERM still does (guard scope is correct), unset env preserves the CLI lifecycle. Exercises the `ServeOptions.mcpStdio?: boolean` test seam directly so tests don't mutate `process.env`.
+
+### E2E test inventory
+
+E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `DATABASE_URL`), except where noted as PGLite in-memory (no `DATABASE_URL` needed).
+
+- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
+- `test/e2e/search-quality.test.ts` — search quality against PGLite (no API keys, in-memory).
+- `test/e2e/graph-quality.test.ts` — knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory.
+- `test/e2e/postgres-jsonb.test.ts` — round-trips all 5 JSONB write sites (`pages.frontmatter`, `raw_data.data`, `ingest_log.pages_updated`, `files.metadata`, `page_versions.frontmatter`) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. Guards against the double-encode bug.
+- `test/e2e/integrity-batch.test.ts` — parity for `scanIntegrity`'s batch-load fast path vs sequential. Cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins multi-source overcounting; the "multi-source duplicate slugs scan once" case expects both batch + sequential paths to report 2.
+- `test/e2e/jsonb-roundtrip.test.ts` — companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface drifts from the actual write surface, one of these tests catches it.
+- `test/e2e/sync.test.ts` — `--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
+- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
+- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
+- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
+- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
+- `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
+- `test/e2e/engine-parity.test.ts` — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector` (Postgres ranks pages then picks best chunk while PGLite returns chunks directly, so the source-boost behavior needs parity coverage). Skips without `DATABASE_URL`.
+- `test/e2e/postgres-bootstrap.test.ts` — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`).
+- `test/e2e/http-transport.test.ts` — `gbrain serve --http` end-to-end against real Postgres: bearer auth round-trip, `last_used_at` SQL-level debounce, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the dispatch round-trip with a real operation. Skips without `DATABASE_URL`.
+- `test/e2e/serve-http-oauth.test.ts` — real-Postgres E2E against `gbrain serve --http` with full OAuth 2.1. Spawns a subprocess server, registers a client via the CLI, mints `client_credentials` tokens, exercises the `/mcp` JSON-RPC pipeline. Real DCR `/register` HTTP-level response-shape test (asserts `typeof body.client_id_issued_at === 'number'` over the wire, RFC 7591 §3.2.1); real CLI subprocess test for `revoke-client` (registers → mints token → revokes via `execSync` → asserts token rejected at `/mcp` → asserts re-run exits 1); server fixture flips on `--enable-dcr` so `/register` is reachable. **bun execSync env-inheritance contract:** bun's `execSync` does NOT inherit env mutations done via `process.env.X = ...`, only OS-level env from before bun started. helpers.ts loads `.env.testing` and sets `DATABASE_URL` via `process.env` mutation, which is invisible to subprocesses unless `env: { ...process.env }` is passed explicitly — every subprocess call in this file passes `env: { ...process.env }`. Reference fix for the same failure mode in sibling sync/cycle/dream/claw-test E2Es. `afterAll` cleanup is guarded on `clientId` (won't throw if `beforeAll` failed before registration); cleanup errors surface to stderr without throwing so real test failures aren't masked. Also covers the trust-boundary fix: an HTTP MCP `submit_job` for `name: "shell"` MUST reject with a permission error (request handler sets `remote: true` and `submit_job`'s protected-name guard fires), and the same guard rejects subagent submission. Skips without `DATABASE_URL`.
+- `test/e2e/sync-parallel.test.ts` — `DATABASE_URL`-gated. 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx`. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
+- `test/e2e/multi-source-bug-class.test.ts` — PGLite in-memory regression suite pinning every multi-source bug site: `listAllPageRefs` ordering by `(source_id, slug)`, `getPage` with sourceId picks the right `(source, slug)` row, `extract-takes` processes both overlapping `people/alice` rows independently, `listPages` filters correctly with `PageFilters.sourceId`, `addLinksBatch` with `from/to_source_id` targets the right rows, `validateSourceId` rejects path traversal, reverse-write disk layout uses `brainDir/.sources/<id>/<slug>.md` for non-default sources. No `DATABASE_URL` needed. Wired into `scripts/e2e-test-map.ts` so changes to extract-takes / patterns / synthesize / embed / extract / migrate-engine auto-trigger it.
+- `test/e2e/source-isolation-pglite.test.ts` — PGLite in-memory regression suite pinning the source-isolation seal at two layers. Engine layer: `searchKeyword` / `searchVector` / `searchKeywordChunks` / `listPages` / `getPage` / `traverseGraph` / `traversePaths` apply `sourceId` (scalar fast path) and `sourceIds` (array path) correctly across both engines. Op-handler layer: routes through `sourceScopeOpts(ctx)` so a `read+write`-scoped OAuth client bound to `--source dept-x` cannot see rows from neighboring sources via `search`, `query`, `list_pages`, `get_page`, or `find_experts`. Covers both `ctx.sourceId` (single-source clients) and `ctx.auth.allowedSources` (federated_read clients) precedence; federated array wins over scalar wins over nothing. No `DATABASE_URL` needed.
+- `test/e2e/skill-brain-first.test.ts` — doctor reports `skill_brain_first` check with structured issues; `--fix --dry-run` previews insertion without writing; `--fix` applies the canonical Convention callout idempotently; `brain_first: exempt` frontmatter resolves the warn; `brain_first_typo` surfaces a paste-ready hint; audit JSONL records `detected` / `resolved` / `fixed` transitions; stable brain emits 0 audit lines/run.
+- Tier 2 (`test/e2e/skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI.
+- If `.env.testing` doesn't exist in this directory, check sibling worktrees: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
+- **Run E2E tests without asking permission.** When you want to verify behavior, there's a relevant E2E test, or you're shipping anything covered by an E2E suite — spin up the test DB, run the tests, tear down. Don't ask, don't propose it, don't defer. The lifecycle is short (~2-30s startup, sub-minute tests, instant teardown) and the gate value is high. Skipping with "DATABASE_URL unset" is silent regression, not caution.
+
+### API keys and running ALL tests
+
+ALWAYS source the user's shell profile before running tests:
+
+```bash
+source ~/.zshrc 2>/dev/null || true
+```
+
+This loads `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`. Without these, Tier 2 tests
+skip silently. Do NOT skip Tier 2 tests just because they require API keys — load
+the keys and run them.
+
+When asked to "run all E2E tests" or "run tests", that means ALL tiers:
+- Tier 1: `bun run test:e2e` (mechanical, sync, upgrade — no API keys needed)
+- Tier 2: `test/e2e/skills.test.ts` (requires OpenAI + Anthropic + openclaw CLI)
+- Always spin up the test DB, source zshrc, run everything, tear down.
+
+### E2E test DB lifecycle (ALWAYS follow this)
+
+You are responsible for spinning up and tearing down the test Postgres container.
+Do not leave containers running after tests. Do not skip E2E tests, do not ask
+permission to run them — see the "run without asking" rule above.
+
+1. **Check for `.env.testing`** — if missing, copy from sibling worktree.
+   Read it to get the DATABASE_URL (it has the port number).
+2. **Check if the port is free:**
+   `docker ps --filter "publish=PORT"` — if another container is on that port,
+   pick a different port (try 5435, 5436, 5437) and start on that one instead.
+3. **Start the test DB:**
+   ```bash
+   docker run -d --name gbrain-test-pg \
+     -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres \
+     -e POSTGRES_DB=gbrain_test \
+     -p PORT:5432 pgvector/pgvector:pg16
+   ```
+   Wait for ready: `docker exec gbrain-test-pg pg_isready -U postgres`
+4. **Bootstrap the schema** (required — fresh containers have no `oauth_clients`,
+   `mcp_request_log`, `pages` etc.; tests like `serve-http-oauth.test.ts` will fail
+   with `relation "oauth_clients" does not exist` if you skip this):
+   ```bash
+   DATABASE_URL=postgresql://postgres:postgres@localhost:PORT/gbrain_test \
+     bun run src/cli.ts doctor --json > /dev/null 2>&1
+   ```
+   `gbrain doctor` triggers `initSchema()` on first connect, which is the canonical
+   way to bring a fresh DB to head. `apply-migrations --yes` alone does NOT seed
+   the base schema — it runs ALTER-style migrations on top of `initSchema`. Tests
+   that bypass the engine (raw `execSync`-spawned `auth register-client`) hit the
+   schema directly and need this step to have run first.
+5. **Run E2E tests:**
+   `DATABASE_URL=postgresql://postgres:postgres@localhost:PORT/gbrain_test bun run test:e2e`
+6. **Tear down immediately after tests finish (pass or fail):**
+   `docker stop gbrain-test-pg && docker rm gbrain-test-pg`
+
+Never leave `gbrain-test-pg` running. If you find a stale one from a previous run,
+stop and remove it before starting a new one.

docs/architecture/thin-client.md

@@ -0,0 +1,70 @@
+# Thin-client routing (remote MCP)
+
+On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
+only; release history lives in `CHANGELOG.md` + git.
+
+`gbrain init --mcp-only` (v0.29.2) sets up a thin-client install: no local
+brain content, just an OAuth client pointing at a remote `gbrain serve --http`.
+v0.29.2/v0.30.0 only refused 9 obvious local-only commands; the other ~25
+silently fell through to `connectEngine()` and opened the empty local PGLite,
+returning "No results." against a populated remote brain. v0.31.1 fixes the
+silent-empty-results bug class for every operation surface.
+
+Key files:
+
+- `src/cli.ts` — Routing seam INSIDE the existing op-dispatch path (CDX-1: no
+  parallel `src/core/thin-client/` module; routing is a ~80-line conditional
+  in `runThinClientRouted`). Detects `isThinClient(cfg)` BEFORE `connectEngine`
+  so thin-client installs never open the empty PGLite. localOnly ops on
+  thin-client refuse via `refuseThinClient` (with pinpoint hint table
+  `THIN_CLIENT_REFUSE_HINTS`). Banner via `printIdentityBannerBestEffort`
+  before each routed call (suppressed by `--quiet`, `GBRAIN_NO_BANNER=1`,
+  non-TTY default). Exhaustive TS `never` switch on `RemoteMcpError.reason`
+  for canned, actionable error messages. ENG-2 renderer parity: local-engine
+  path runs `JSON.parse(JSON.stringify(result))` so renderers see the same
+  shape on both paths (kills Date/bigint/Buffer drift class).
+- `src/core/mcp-client.ts` — `callRemoteTool(config, toolName, args, opts)`.
+  Hardened in v0.31.1 (CDX-4): all transport errors normalized to
+  `RemoteMcpError` via the `toRemoteMcpError` funnel. New `CallRemoteToolOptions
+  {timeoutMs, signal}`; `buildAbortController` composes external signal with
+  timeout. New `RemoteMcpErrorReason` stable union, `RemoteMcpErrorDetail.kind`
+  ('timeout' | 'aborted' | 'unreachable') sub-tag, `RemoteMcpErrorDetail.code`
+  field carrying server-supplied error codes (e.g. `missing_scope`).
+  `extractToolErrorCode` parses JSON envelopes first, falls back to substring
+  detection for legacy server messages. `unpackToolResult<T>(res)` unchanged
+  (parses tool-call JSON content). `_clearMcpClientTokenCache()` test escape.
+- `src/core/cli-options.ts` — `parseGlobalFlags` adds `--timeout=Ns` (accepts
+  `30s`, `2m`, `500ms`, plain ms). Default `null` = per-command default (30s
+  for most ops, 180s for `think`). `parseTimeout(s)` exported helper.
+- `src/core/doctor-remote.ts` — `gbrain remote doctor` adds the
+  `oauth_client_scopes_probe` check (CDX-5). Probes the read tier via
+  `get_brain_identity` and admin tier via `get_health`; reports per-tier
+  status with pinpoint remediation when admin is missing. `buildScopeCheck`
+  + `ScopeProbeResult` exported for test access. Skippable via
+  `GBRAIN_DOCTOR_SKIP_SCOPE_PROBE=1` for fixtures that mock /mcp at JSON-RPC
+  initialize level only (MCP SDK Client hangs on shape mismatch).
+- `src/core/ssrf-validate.ts` (v0.36 Commit 0) — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A AND AAAA record against the internal-IP deny list, returns the resolved IP so callers fetch by IP (defeats DNS rebinding: validation IP === fetch IP). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation, max 3 hops by default. Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
+- `src/core/search/query-intent.ts` extension (v0.36 cross-modal wave) — new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope `CROSS_MODAL_PATTERNS` regex array (compiles once at module load). `isAmbiguousModalityQuery(query)` heuristic gate fires when a visual noun + reference marker combination indicates genuinely ambiguous routing — used by the Commit 4 LLM tie-break to bound LLM calls to <1% of queries.
+- `src/core/search/mode.ts` extension (v0.36 cross-modal wave) — `ModeBundle` extended with 7 cross-modal knobs: `cross_modal_both_text_weight` / `cross_modal_both_image_weight` (D6 weighted RRF for `'both'` mode, defaults 0.6/0.4), `image_query_text_refinement_weight` / `image_query_image_refinement_weight` (D13 hybrid intersect for `searchByImage` query refinement, defaults 0.4/0.6), `unified_multimodal` + `unified_multimodal_only` (Phase 3 unified column routing flags), `cross_modal_llm_intent` (Commit 4 opt-in escalation). `SEARCH_MODE_CONFIG_KEYS` extended with 7 corresponding config keys. `KNOBS_HASH_VERSION` bumped 2→3 (D2 — closes the silent cache-hit class where a cached text-mode result could leak to an image-mode caller).
+- `src/core/search/hybrid.ts` extension (v0.36 cross-modal wave) — cross-modal routing branch at the embed step. Resolves `effectiveModality` from per-call `opts.crossModal` (normalized: literal `'auto'` → undefined per D22-1) → `suggestions.suggestedModality` → `'text'` default. Image route: `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_image'})`, skip expansion + keyword (D9 mode-bundle override). 'both' route: parallel text + image vector searches merged via `rrfFusionWeighted` with `effectiveRrfK(baseRrfK, weight)` from the configured cross-modal weights. Phase 3 unified routing fires when `cfg.search.unified_multimodal === true` — bypasses dual-column branching, runs `embedQueryMultimodal` + `searchVector({embeddingColumn: 'embedding_multimodal'})`, D8 fail-open on zero rows + not strict-mode falls through to dual-column. Commit 4 LLM escalation fires only when (no explicit per-call opt) AND (regex returned 'text') AND (`cfg.search.cross_modal.llm_intent` is true) AND (`isAmbiguousModalityQuery` returns true). Fail-open on every error.
+- `src/core/search/image-loader.ts` (v0.36 Phase 2) — `loadImageInput(input, opts)` accepts local path, `data:` URI, or `http(s)://` URL. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). For URLs: routes through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated. Pre-flight Content-Length check + post-fetch size guard for lying servers. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).
+- `src/core/search/by-image.ts` (v0.36 Phase 2) — `searchByImage(engine, input, opts)`. Always runs image branch (`embedQueryMultimodalImage` + `searchVector(embedding_image)`). D13 hybrid intersect: when caller provides optional `query`, runs parallel text branch via `embedQueryMultimodal(query)` and merges via `rrfFusionWeighted` with weights from resolved mode. Phase 3 widens to unified column once `search.unified_multimodal=true` (transparently upgrades the retrieval quality post-reindex).
+- `src/core/spend-log.ts` (v0.36 Phase 2 D23-#6) — per-OAuth-client paid-API spend tracking against the `mcp_spend_log` table (migration v74). `checkBudget(engine, clientId, capCents)` is the pre-flight gate; throws `BudgetExceededError` when today's spend has hit the cap. `recordSpend(engine, entry)` is best-effort post-call. UTC day-aligned aggregation so caps roll over deterministically regardless of server timezone. Local CLI callers (no clientId) bypass the gate. Pre-v0.36 brains without the table fail open to spend=0. `VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS` = 0.12 cents per image embed.
+- `src/core/search/llm-intent.ts` (v0.36 Commit 4) — opt-in LLM tie-break. `classifyModalityWithLLM(query, fallback)` routes through `gateway.chat()` with a fixed single-word-output system prompt. 1s timeout via AbortController. `parseModality(raw, fallback)` is the pure parser — tolerates trailing punctuation + casing. Fail-open on every error (gateway unavailable, timeout, parse failure, unrecognized output) — returns fallback so a misbehaving LLM can never break search. Cost-bounded by the ambiguity heuristic in `query-intent.ts` (fires <1% of queries when on).
+- `src/commands/reindex-multimodal.ts` (v0.36 Phase 3) — `gbrain reindex --multimodal [--limit N] [--dry-run] [--cost-estimate] [--no-embed] [--yes] [--json]`. Walks `content_chunks WHERE embedding_multimodal IS NULL`, batches via `embedMultimodalSafe` (Commit 0 partial-failure-aware), persists. D7 lock acquisition via `tryAcquireDbLock('gbrain-reindex-multimodal', 360min)`. Cost prompt + 10s Ctrl-C grace window in TTY. `GBRAIN_NO_REEMBED=1` bypass. Checkpoint at `~/.gbrain/reindex-multimodal-checkpoint.json` for resume. D23-#2 auto-flip prompt at coverage=100% completion (TTY: interactive; non-TTY: stderr hint with paste-ready command).
+- `src/core/backfill-registry.ts` extension (v0.36) — new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` AND `embedding_image IS NOT NULL` AND `(modality IS NULL OR modality != 'image')`. D22-7 defensive guard: never flag a non-image chunk that happens to have `embedding_image` populated. Idempotent — second run finds zero rows.
+- `src/core/migrate.ts` v74 (`mcp_spend_log`) + v75 (`embedding_multimodal_column`) — Phase 2 spend-log table + Phase 3 unified column ALTER. v75 is column-only (no HNSW index — deferred to post-reindex per pgvector best practice). v74 uses BTREE on `(client_id, created_at)` + `(token_name, created_at)` — `date_trunc('day', TIMESTAMPTZ)` is NOT IMMUTABLE so can't appear in index expressions; range scan on created_at covers the per-day rollup query.
+- `src/core/operations.ts` — `get_brain_identity` op (read scope, no params,
+  banner-only): cheap counter packet `{version, engine, page_count,
+  chunk_count, last_sync_iso}` for the thin-client identity banner. Reuses
+  `engine.getStats()`; banner's 60s client-side TTL bounds frequency to
+  ≤1/60s per CLI process (well below the Fly.io health-check cadence that
+  motivated the original `getStats` cost warning).
+- `src/commands/{salience,anomalies,graph-query,think}.ts` — Per-command
+  thin-client routing branches. These commands bypass the operation-layer
+  dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
+  `if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
+  to op params. `think` is a special case: the server's `think` op
+  intentionally disables `--save`/`--take` for remote callers
+  (operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
+  loudly when those flags are set.

docs/guides/skillopt.md

@@ -87,8 +87,9 @@ proposal (this lives in v0.42 follow-up; v1 emits the audit event).
 | `--judge-model MODEL` | tier.reasoning | Scores rollouts |
 | `--patch \| --rewrite` | patch | Edit ops only vs. full rewrites |
 | `--dry-run` | off | Cost preview, no LLM calls |
-| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md |
-| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills |
+| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md (no held-out needed) |
+| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills in place — ALSO requires `--held-out` (>=5 rows) or the run hard-refuses |
+| `--held-out <path>` | — | Independent test set (same JSONL shape as the benchmark, task IDs disjoint from it). A candidate that beats the benchmark but regresses on the held-out set is refused. Required for in-place bundled mutation. |
 | `--max-cost-usd N` | 5.00 | Hard cap; preflight refuses if exceeded |
 | `--max-runtime-min N` | 30 | Wall-clock cap |
 | `--force` | off | Bypass dirty-working-tree refusal |
@@ -123,7 +124,8 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
 | Validation gate is mandatory | D12 (paper) | Accepting LLM judge noise as improvement |
 | Frontmatter mutation forbidden | D5 | Routing surface drift (`check-resolvable` regression) |
 | Per-skill DB lock | D14 | Two concurrent runs corrupting history/versions |
-| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain |
+| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain (in-place mutation requires `--allow-mutate-bundled` + a `--held-out` set of >=5 benchmark-disjoint tasks; else hard-refuse + proposed.md) |
+| Held-out gate | F11 | Accepting a candidate that overfits its own benchmark — `--held-out` refuses a candidate whose held-out score regresses below baseline |
 | Bootstrap review sentinel | D15 | Self-referential benchmark gaming |
 | Read-only tool sandbox in rollouts | D13 | Optimization runs writing junk pages to your brain |
 | History-intent-first atomic commit | D8 | Half-written SKILL.md on crash |

docs/tutorials/improving-skills-with-skillopt.md

@@ -239,14 +239,23 @@ silently mutate a skill other people depend on. Two ways to handle that:
 ```bash
 # See the proposed improvement without touching SKILL.md (works for ANY skill):
 gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
-# → writes skills/meeting-prep/skillopt/best.md, prints its path. Copy what you want.
+# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
 
-# Actually rewrite a bundled skill (explicit opt-in):
-gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled
+# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
+gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+  --held-out skills/brain-ops/held-out.jsonl
 ```
 
-Rule of thumb: `--no-mutate` when you want to read the diff before trusting it;
-`--allow-mutate-bundled` only when you intend to commit a change to a shared skill.
+Rewriting a bundled skill in place now requires BOTH `--allow-mutate-bundled` AND
+`--held-out <path>` (a JSONL with the same shape as your benchmark, but at least 5
+tasks whose IDs don't appear in the benchmark). The held-out set is how the run
+proves the edit didn't just learn the benchmark: a candidate that climbs the
+benchmark but slips on the held-out tasks is refused. Drop `--held-out` and the
+run hard-refuses and points you at `proposed.md` instead.
+
+Rule of thumb: `--no-mutate` when you want to read the diff before trusting it
+(no held-out needed); `--allow-mutate-bundled --held-out` only when you intend to
+commit a proven change to a shared skill.
 
 ## Step 6: Iterate
 

llms.txt

@@ -7,7 +7,9 @@ Repo: https://github.com/garrytan/gbrain
 ## Core entry points
 
 - [AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md): Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.
-- [CLAUDE.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CLAUDE.md): Architecture reference. Key files, trust boundaries, engine factory, test layout.
+- [CLAUDE.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CLAUDE.md): Orientation + resolver. North Star, two axes, architecture + cross-cutting invariants, the reference map pointing at on-demand docs, and the inline ship IRON RULES.
+- [docs/architecture/KEY_FILES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/architecture/KEY_FILES.md): Per-file index for the gbrain repo: what each src/ file does + its load-bearing invariants. The on-demand detail CLAUDE.md's reference map routes to.
+- [docs/architecture/thin-client.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/architecture/thin-client.md): The thin-client / remote-MCP / cross-modal routing seam: isThinClient detection, callRemoteTool, SSRF-hardened URL validation, per-command routing.
 - [INSTALL_FOR_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md): 9-step agent installation.
 - [skills/RESOLVER.md](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/RESOLVER.md): Skill dispatcher. Read first for any task.
 - [README.md](https://raw.githubusercontent.com/garrytan/gbrain/master/README.md): Project overview, benchmarks, 30-minute setup.
@@ -42,6 +44,11 @@ Repo: https://github.com/garrytan/gbrain
 - [skills/migrations/](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/migrations/): Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.
 - [CHANGELOG.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CHANGELOG.md): Release-summary voice + itemized changes + self-repair block per version.
 
+## Contributing
+
+- [docs/TESTING.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/TESTING.md): Test command tiers, the test-isolation lint (R1-R4), the canonical PGLite block, withEnv, the E2E DB lifecycle, and the file taxonomy. Maintainer-facing.
+- [docs/RELEASING.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/RELEASING.md): Full release + contributor process: pre-ship test requirements, the CHANGELOG voice + release-summary template, the 'To take advantage of vX' block, version migrations, GitHub Actions SHA refresh, PR conventions, community-PR-wave. (Ship IRON RULES stay inline in CLAUDE.md.)
+
 ## Philosophy
 
 - [docs/ethos/THIN_HARNESS_FAT_SKILLS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/THIN_HARNESS_FAT_SKILLS.md): Why skills live in markdown.

package.json

@@ -47,9 +47,10 @@
     "check:system-of-record": "scripts/check-system-of-record.sh",
     "check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
     "check:cli-exec": "scripts/check-cli-executable.sh",
-    "check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
+    "check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
     "check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
     "check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
+    "check:doc-history": "scripts/check-key-files-current-state.sh",
     "check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
     "check:skill-brain-first": "scripts/check-skill-brain-first.sh",
     "check:wasm": "scripts/check-wasm-embedded.sh",
@@ -142,5 +143,5 @@
     "bun": ">=1.3.10"
   },
   "license": "MIT",
-  "version": "0.42.10.0"
+  "version": "0.42.11.0"
 }

scripts/check-key-files-current-state.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# scripts/check-key-files-current-state.sh — the anti-disease guard.
+#
+# CLAUDE.md grew to ~592KB / ~147k tokens (auto-loaded every session) once its
+# per-file index became append-only: one `**vX.Y.Z (#NNN):**` clause per release
+# per file. This guard makes that recurrence structurally impossible. A written
+# rule caused the disease; a CI guard cures it.
+#
+# TWO HARD GATES (fail the build):
+#   1. Bolded-release-clause ban — the reference docs (docs/architecture/KEY_FILES.md,
+#      docs/architecture/thin-client.md, docs/TESTING.md) describe CURRENT behavior
+#      only. Release history lives in CHANGELOG.md + git. The bolded `**v0.<digit>`
+#      marker is the disease signature; it must not appear in those docs. Plain prose
+#      ("as of pgvector 0.7", "Postgres 11+") is fine — only the bolded release
+#      marker is banned, so this never false-fires on legitimate version mentions.
+#   2. CLAUDE.md size cap — the structural backstop. Even if someone ignores the
+#      prose rule and pads CLAUDE.md, the size gate catches it.
+#
+# SOFT WARNS (stderr, non-fatal): prose history markers that suggest narration
+# creeping back ("pre-fix", ", then v0.", "superseded by") in the reference docs.
+#
+# Usage:
+#   bash scripts/check-key-files-current-state.sh
+#
+# Env overrides (for the guard's own test):
+#   GBRAIN_DOC_GUARD_ROOT        repo root to scan (default: script's ../)
+#   GBRAIN_CLAUDE_MD_MAX_BYTES   CLAUDE.md hard cap (default: 60000; post-restructure
+#                                CLAUDE.md is ~39KB, so this leaves headroom while
+#                                staying far below the ~592KB disease state)
+#
+# Exit codes:
+#   0   clean
+#   1   a hard gate failed
+
+set -uo pipefail
+
+ROOT="${GBRAIN_DOC_GUARD_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
+MAX_BYTES="${GBRAIN_CLAUDE_MD_MAX_BYTES:-60000}"
+
+# Reference docs that MUST stay current-state (history-free).
+REFERENCE_DOCS=(
+  "docs/architecture/KEY_FILES.md"
+  "docs/architecture/thin-client.md"
+  "docs/TESTING.md"
+)
+
+fail=0
+
+# ── Gate 1: bolded release-clause ban ──────────────────────────────────────
+for rel in "${REFERENCE_DOCS[@]}"; do
+  doc="$ROOT/$rel"
+  [ -f "$doc" ] || continue
+  hits=$(grep -nE '\*\*v0\.[0-9]' "$doc" || true)
+  if [ -n "$hits" ]; then
+    fail=1
+    echo "FAIL: $rel contains bolded release-clause markers (append-only history is the disease this guard prevents)." >&2
+    echo "      Reference docs describe CURRENT behavior only; release history goes in CHANGELOG.md + git." >&2
+    echo "      Collapse each version-clause chain into the single current truth. Offending lines:" >&2
+    printf '%s\n' "$hits" | sed 's/^/        /' | cut -c1-140 >&2
+  fi
+done
+
+# ── Gate 2: CLAUDE.md size cap ─────────────────────────────────────────────
+claude="$ROOT/CLAUDE.md"
+if [ -f "$claude" ]; then
+  bytes=$(wc -c < "$claude" | tr -d ' ')
+  if [ "$bytes" -gt "$MAX_BYTES" ]; then
+    fail=1
+    echo "FAIL: CLAUDE.md is $bytes bytes, over the $MAX_BYTES cap." >&2
+    echo "      CLAUDE.md is orientation + resolver, not the implementation spec. Per-file/" >&2
+    echo "      per-command/per-test detail belongs in the on-demand reference docs" >&2
+    echo "      (docs/architecture/KEY_FILES.md, docs/TESTING.md, docs/RELEASING.md), not here." >&2
+  fi
+fi
+
+# ── Soft warns: prose history markers creeping into reference docs ──────────
+for rel in "${REFERENCE_DOCS[@]}"; do
+  doc="$ROOT/$rel"
+  [ -f "$doc" ] || continue
+  warns=$(grep -cnE ', then v0\.|superseded by|pre-fix|post-fix' "$doc" || true)
+  if [ "${warns:-0}" -gt 0 ]; then
+    echo "WARN: $rel has $warns prose history marker(s) ('pre-fix' / ', then v0.' / 'superseded by'). Prefer current-state phrasing." >&2
+  fi
+done
+
+if [ "$fail" -ne 0 ]; then
+  exit 1
+fi
+echo "check-key-files-current-state: ok (reference docs history-free; CLAUDE.md within cap)"

scripts/ci-cache-hash.sh

@@ -30,6 +30,14 @@
 #   - everything else under src/, test/, scripts/, .github/, package.json,
 #     bun.lock, tsconfig*.json, the schema files — obviously test-affecting
 #
+# POLICY-DOC RE-ADMIT (the docs/ exception): some docs/*.md files carry
+# CI / release / test CONTRACTS that the test suite reads (e.g. the
+# build-llms content-contract test, the doc-history guard). The broad
+# `^docs/.*\.md$` deny above would let a policy edit to those skip CI — a
+# false-pass. The ALLOW_PATTERNS list below re-admits them into the hash
+# AFTER the deny. ADD a path there whenever you move a policy/contract doc
+# under docs/ (current entries: docs/TESTING.md, docs/RELEASING.md).
+#
 # Locale-stable: LC_ALL=C on the sort step so byte-order is identical
 # across runners (different default locales would re-order the line list
 # and change the final hash).
@@ -113,6 +121,33 @@ DENY_RE=$(printf '\t(%s)' "$DENY_ALT")
 # TODOS\.md$|docs/.*\.md$|...)`. Each alternative anchors its own end.
 INCLUDED=$(printf '%s\n' "$LS_FILES" | grep -vE "$DENY_RE" || true)
 
+# Re-admit test-affecting policy docs that live under docs/ but carry CI /
+# release / test contracts. The broad `^docs/.*\.md$` deny above removed
+# them; without this re-admit a policy edit to docs/TESTING.md or
+# docs/RELEASING.md would produce the SAME hash and skip the test shard
+# that runs the build-llms + doc-history guards — a false-pass. Patterns
+# anchor on the `\t<path>` boundary in `git ls-files -s` output, matching
+# the deny-list convention above. Re-admitted lines that don't exist yet
+# (pre-relocation) simply match nothing.
+# Path predicates only (no leading tab here) — the `\t` boundary is added
+# via printf below so it is a REAL tab byte, not the two-char string `\t`.
+# GNU grep (CI/Ubuntu) does not interpret `\t` in an ERE as a tab the way
+# BSD grep (macOS) does, so an inline `\t` matches nothing on CI and the
+# re-admit silently no-ops. Mirror the DENY_RE construction exactly.
+ALLOW_PATTERNS=(
+  'docs/TESTING\.md$'
+  'docs/RELEASING\.md$'
+)
+ALLOW_ALT=""
+for p in "${ALLOW_PATTERNS[@]}"; do
+  if [ -z "$ALLOW_ALT" ]; then ALLOW_ALT="$p"; else ALLOW_ALT="$ALLOW_ALT|$p"; fi
+done
+ALLOW_RE=$(printf '\t(%s)' "$ALLOW_ALT")
+READMIT=$(printf '%s\n' "$LS_FILES" | grep -E "$ALLOW_RE" || true)
+if [ -n "$READMIT" ]; then
+  INCLUDED=$(printf '%s\n%s\n' "$INCLUDED" "$READMIT" | grep -v '^$' | LC_ALL=C sort -u)
+fi
+
 if [ -z "$INCLUDED" ]; then
   echo "error: every tracked file is deny-listed — refusing to hash empty set" >&2
   exit 1

scripts/llms-config.ts

@@ -48,9 +48,26 @@ export const SECTIONS: DocSection[] = [
       {
         title: "CLAUDE.md",
         description:
-          "Architecture reference. Key files, trust boundaries, engine factory, test layout.",
+          "Orientation + resolver. North Star, two axes, architecture + cross-cutting invariants, the reference map pointing at on-demand docs, and the inline ship IRON RULES.",
         path: "CLAUDE.md",
       },
+      {
+        title: "docs/architecture/KEY_FILES.md",
+        description:
+          "Per-file index for the gbrain repo: what each src/ file does + its load-bearing invariants. The on-demand detail CLAUDE.md's reference map routes to.",
+        path: "docs/architecture/KEY_FILES.md",
+        // Link-only until compressed to current-state (still large pre-compression).
+        // Flip to inlined once the doc-history compression lands and the bundle
+        // budget is re-measured.
+        includeInFull: false,
+      },
+      {
+        title: "docs/architecture/thin-client.md",
+        description:
+          "The thin-client / remote-MCP / cross-modal routing seam: isThinClient detection, callRemoteTool, SSRF-hardened URL validation, per-command routing.",
+        path: "docs/architecture/thin-client.md",
+        includeInFull: false,
+      },
       {
         title: "INSTALL_FOR_AGENTS.md",
         description: "9-step agent installation.",
@@ -87,6 +104,9 @@ export const SECTIONS: DocSection[] = [
         includeInFull: false,
       },
       {
+        // Re-inlined: the CLAUDE.md resolver restructure (per-file index moved to
+        // docs/architecture/KEY_FILES.md, link-only) freed ~530KB of bundle
+        // headroom, so this value-explainer rides the single-fetch bundle again.
         title: "docs/what-schemas-unlock.md",
         description:
           "Why schemas matter: 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator) + the structural argument for typed page kinds. Read this before pitching schema authoring (v0.40.7.0).",
@@ -210,6 +230,26 @@ export const SECTIONS: DocSection[] = [
       },
     ],
   },
+  {
+    heading: "Contributing",
+    optional: true,
+    entries: [
+      {
+        title: "docs/TESTING.md",
+        description:
+          "Test command tiers, the test-isolation lint (R1-R4), the canonical PGLite block, withEnv, the E2E DB lifecycle, and the file taxonomy. Maintainer-facing.",
+        path: "docs/TESTING.md",
+        includeInFull: false,
+      },
+      {
+        title: "docs/RELEASING.md",
+        description:
+          "Full release + contributor process: pre-ship test requirements, the CHANGELOG voice + release-summary template, the 'To take advantage of vX' block, version migrations, GitHub Actions SHA refresh, PR conventions, community-PR-wave. (Ship IRON RULES stay inline in CLAUDE.md.)",
+        path: "docs/RELEASING.md",
+        includeInFull: false,
+      },
+    ],
+  },
   {
     heading: "Philosophy",
     optional: true,

scripts/run-verify-parallel.sh

@@ -55,6 +55,7 @@ CHECKS=(
   "check:operations-filter-bypass"
   "check:gateway-routed"
   "check:worker-pool-atomicity"
+  "check:doc-history"
   "check:fixture-privacy"
   "check:conversation-parser"
   "check:resolver"

skills/skill-optimizer/SKILL.md

@@ -32,10 +32,13 @@ The user wants to:
   + epsilon=0.05 margin against the sel-set before SKILL.md gets rewritten.
 - **Frontmatter mutation is FORBIDDEN.** The optimizer only edits the body.
   Routing surface (`triggers:`, `brain_first:`) stays invariant.
-- **Bundled skills require explicit opt-in.** Skills shipping with gbrain
-  cannot be auto-mutated; user passes `--allow-mutate-bundled` or
-  `--no-mutate` (default for the dream-cycle phase) writes proposed.md
-  for review.
+- **Bundled skills require explicit opt-in AND an independent held-out set.**
+  Skills shipping with gbrain cannot be auto-mutated. To rewrite one in place
+  the user passes BOTH `--allow-mutate-bundled` AND `--held-out <path>` with
+  at least 5 benchmark-disjoint tasks; without the held-out set the run
+  hard-refuses (exit 2). Drop `--allow-mutate-bundled` (or pass `--no-mutate`,
+  the default for the dream-cycle phase) to write proposed.md for review
+  instead — no held-out needed for review-only output.
 - **Bootstrap output requires human review.** Both `--bootstrap-from-skill`
   and `--bootstrap-from-routing` write a sentinel; you must review + STRENGTHEN
   the generated judges, delete the sentinel, and re-run with
@@ -127,8 +130,9 @@ attach >=2 rule checks each, save to `skills/X/skillopt-benchmark.jsonl`, run wi
 | Skill has a `routing-eval.jsonl` and you want a head start | `gbrain skillopt foo --bootstrap-from-routing` → review the generated tasks → `--bootstrap-reviewed` (routing tasks test dispatch; tighten them into quality tasks before trusting) |
 | Iterating on an existing skill | `gbrain skillopt foo --benchmark skills/foo/skillopt-benchmark.jsonl` |
 | Costly run, want preview | Add `--dry-run` |
-| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; add `--allow-mutate-bundled` to commit |
-| Want to review changes before applying | Add `--no-mutate` |
+| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; to commit in place add `--allow-mutate-bundled` AND `--held-out <path>` (>=5 benchmark-disjoint tasks) — else it hard-refuses |
+| Want to review changes before applying | Add `--no-mutate` (writes proposed.md, no held-out needed) |
+| Guard against benchmark overfitting | Add `--held-out <path>` — a candidate that beats the benchmark but regresses on the held-out set is refused |
 | Mid-run crash | `gbrain skillopt foo --resume <run-id>` |
 
 ## Output Format
@@ -146,8 +150,11 @@ When invoked, this skill produces:
 
 - **Don't bypass the validation gate.** The median-of-3 + epsilon=0.05 is
   load-bearing; without it, the optimizer accepts noise as improvement.
-- **Don't optimize bundled skills without `--allow-mutate-bundled`.** They
-  ship with gbrain and are load-bearing for downstream agents.
+- **Don't optimize bundled skills without `--allow-mutate-bundled` AND
+  `--held-out`.** They ship with gbrain and are load-bearing for downstream
+  agents. In-place mutation requires both flags (held-out >=5 benchmark-disjoint
+  tasks); without the held-out set the run hard-refuses and points you at
+  proposed.md.
 - **Don't use bootstrap output without strengthening it.** Both
   `--bootstrap-from-skill` and `--bootstrap-from-routing` have the optimizer
   model invent success criteria — generic and weak by default. Review and
@@ -163,7 +170,11 @@ When invoked, this skill produces:
 ```
 {
   outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored',
-  receipt: { run_id, skill_sha8, benchmark_sha8, models, scores, cost },
+  receipt: {
+    run_id, skill_sha8, benchmark_sha8, models, cost,
+    baseline_sel_score, best_sel_score,   // real measured baseline (no longer hardcoded 0)
+    baseline_test_score, test_score,      // final held-out test-split eval
+  },
   finalText: string,
   mutatedSkillFile: boolean,
   proposedPath?: string

src/commands/jobs.ts

@@ -1750,6 +1750,7 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
       noMutate: Boolean(data.no_mutate),
       allowMutateBundled: Boolean(data.allow_mutate_bundled),
       bootstrapReviewed: Boolean(data.bootstrap_reviewed),
+      ...(data.held_out_path ? { heldOutPath: String(data.held_out_path) } : {}),
       json: true,
       maxCostUsd: Number(data.max_cost_usd ?? 5.0),
       maxRuntimeMin: Number(data.max_runtime_min ?? 30),

src/commands/skillopt.ts

@@ -35,6 +35,8 @@ interface ParsedFlags {
   dryRun: boolean;
   noMutate: boolean;
   allowMutateBundled: boolean;
+  /** F11: optional held-out test set path. REQUIRED (non-empty) to mutate a bundled skill. */
+  heldOutPath?: string;
   json: boolean;
   maxCostUsd: number;
   maxRuntimeMin: number;
@@ -193,6 +195,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strin
         noMutate: parsed.noMutate,
         allowMutateBundled: parsed.allowMutateBundled,
         bootstrapReviewed: parsed.bootstrapReviewed,
+        ...(parsed.heldOutPath ? { heldOutPath: parsed.heldOutPath } : {}),
         maxCostUsd: parsed.maxCostUsd,
         maxRuntimeMin: parsed.maxRuntimeMin,
         force: parsed.force,
@@ -246,6 +249,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strin
           dry_run: parsed.dryRun,
           no_mutate: parsed.noMutate,
           allow_mutate_bundled: parsed.allowMutateBundled,
+          ...(parsed.heldOutPath ? { held_out_path: parsed.heldOutPath } : {}),
           bootstrap_reviewed: parsed.bootstrapReviewed,
           max_cost_usd: parsed.maxCostUsd,
           max_runtime_min: parsed.maxRuntimeMin,
@@ -289,6 +293,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strin
     dryRun: parsed.dryRun,
     noMutate: parsed.noMutate,
     allowMutateBundled: parsed.allowMutateBundled,
+    ...(parsed.heldOutPath ? { heldOutPath: parsed.heldOutPath } : {}),
     bootstrapReviewed: parsed.bootstrapReviewed,
     json: parsed.json,
     maxCostUsd: parsed.maxCostUsd,
@@ -345,6 +350,7 @@ export function parseFlags(args: string[]): ParsedFlags {
   let dryRun = false;
   let noMutate = false;
   let allowMutateBundled = false;
+  let heldOutPath: string | undefined;
   let json = false;
   let maxCostUsd = 5.0;
   let maxRuntimeMin = 30;
@@ -390,6 +396,7 @@ export function parseFlags(args: string[]): ParsedFlags {
     if (a === '--dry-run') { dryRun = true; i += 1; continue; }
     if (a === '--no-mutate') { noMutate = true; i += 1; continue; }
     if (a === '--allow-mutate-bundled') { allowMutateBundled = true; i += 1; continue; }
+    if (a === '--held-out') { heldOutPath = args[++i]; i += 1; continue; }
     if (a === '--json') { json = true; i += 1; continue; }
     if (a === '--max-cost-usd') { maxCostUsd = mustFloat(args[++i], '--max-cost-usd'); i += 1; continue; }
     if (a === '--max-runtime-min') { maxRuntimeMin = mustInt(args[++i], '--max-runtime-min'); i += 1; continue; }
@@ -466,6 +473,7 @@ export function parseFlags(args: string[]): ParsedFlags {
     dryRun,
     noMutate,
     allowMutateBundled,
+    ...(heldOutPath !== undefined ? { heldOutPath } : {}),
     json,
     maxCostUsd,
     maxRuntimeMin,

src/core/ai/gateway.ts

@@ -21,7 +21,7 @@
  *     rotation (via configureGateway()) invalidates stale entries.
  */
 
-import { embed as aiEmbed, embedMany, generateObject, generateText } from 'ai';
+import { embed as aiEmbed, embedMany, generateObject, generateText, jsonSchema } from 'ai';
 import { AsyncLocalStorage } from 'node:async_hooks';
 import { listRecipes } from './recipes/index.ts';
 import { createOpenAI } from '@ai-sdk/openai';
@@ -2177,6 +2177,52 @@ export interface ChatToolDef {
   inputSchema: Record<string, unknown>;
 }
 
+/**
+ * Convert gbrain's provider-neutral ChatMessage[] into AI SDK v6 ModelMessage[].
+ *
+ * The original code passed `opts.messages as any` straight to generateText,
+ * which worked on AI SDK v4/v5 but v6 tightened ModelMessage validation:
+ *   - tool results must be a `role: 'tool'` message (gbrain pushes them as
+ *     `role: 'user'` with tool-result blocks), and
+ *   - each tool-result `output` must be a structured `{ type, value }` part,
+ *     not a bare value.
+ * Without this conversion every multi-turn tool loop (skillopt rollouts AND
+ * production subagent jobs) throws "messages do not match the ModelMessage[]
+ * schema" the moment the model calls a tool. Surfaced by the SkillOpt eval.
+ */
+export function toModelMessages(messages: ChatMessage[]): unknown[] {
+  return messages.map((m) => {
+    if (typeof m.content === 'string') return { role: m.role, content: m.content };
+    const blocks = m.content;
+    if (blocks.some((b) => b.type === 'tool-result')) {
+      // v6: tool results ride on a dedicated `tool` role with structured output.
+      return {
+        role: 'tool' as const,
+        content: blocks
+          .filter((b): b is Extract<ChatBlock, { type: 'tool-result' }> => b.type === 'tool-result')
+          .map((b) => ({
+            type: 'tool-result' as const,
+            toolCallId: b.toolCallId,
+            toolName: b.toolName,
+            output: b.isError
+              ? { type: 'error-text' as const, value: typeof b.output === 'string' ? b.output : JSON.stringify(b.output) }
+              : (typeof b.output === 'string'
+                ? { type: 'text' as const, value: b.output }
+                : { type: 'json' as const, value: (b.output ?? null) as never }),
+          })),
+      };
+    }
+    return {
+      role: m.role,
+      content: blocks.map((b) => {
+        if (b.type === 'text') return { type: 'text' as const, text: b.text };
+        if (b.type === 'tool-call') return { type: 'tool-call' as const, toolCallId: b.toolCallId, toolName: b.toolName, input: b.input };
+        return b;
+      }),
+    };
+  });
+}
+
 export interface ChatResult {
   /** Final text content concatenated from text blocks. */
   text: string;
@@ -2528,7 +2574,12 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
   const tools = (opts.tools ?? []).reduce((acc, t) => {
     acc[t.name] = {
       description: t.description,
-      inputSchema: { jsonSchema: t.inputSchema } as any,
+      // AI SDK v6 requires a Schema (carrying the schema symbol), not a plain
+      // `{jsonSchema}` object — the bare object makes asSchema() treat it as a
+      // thunk and call schema(), throwing "schema is not a function". Wrap the
+      // raw JSON Schema with the SDK's jsonSchema() helper so tool calls work
+      // through the real toolLoop (skillopt rollouts + subagent jobs).
+      inputSchema: jsonSchema(t.inputSchema as any),
     };
     return acc;
   }, {} as Record<string, any>);
@@ -2558,7 +2609,7 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
     const result = await generateText({
       model,
       system: opts.system,
-      messages: opts.messages as any,
+      messages: toModelMessages(opts.messages) as any,
       tools: opts.tools && opts.tools.length > 0 ? tools : undefined,
       maxOutputTokens: opts.maxTokens ?? 4096,
       abortSignal: opts.abortSignal,

src/core/anthropic-pricing.ts

@@ -26,6 +26,11 @@ export const ANTHROPIC_PRICING: Record<string, ModelPricing> = {
   // https://platform.claude.com/docs/en/about-claude/models/overview (verified 2026-05-10).
   'claude-opus-4-7':            { input:  5.00, output: 25.00 },
   'claude-sonnet-4-6':          { input:  3.00, output: 15.00 },
+  // Both the dateless canonical id (TIER_DEFAULTS / aliases / every caller uses
+  // this) AND the dated snapshot. The dateless entry was missing pre-v0.42.9.0,
+  // so a budget-capped Haiku run (skillopt with --max-cost, eval harnesses) hit
+  // BudgetTracker no_pricing and every rollout silently scored 0.
+  'claude-haiku-4-5':           { input:  1.00, output:  5.00 },
   'claude-haiku-4-5-20251001':  { input:  1.00, output:  5.00 },
   // Older but still frequently aliased
   'claude-opus-4-6':            { input:  5.00, output: 25.00 },

src/core/operations.ts

@@ -4525,12 +4525,22 @@ const run_skillopt: Operation = {
     max_cost_usd: { type: 'number', description: 'Default 5.00' },
     no_mutate: { type: 'boolean', description: 'Write proposed.md without replacing SKILL.md' },
     allow_mutate_bundled: { type: 'boolean', description: 'Required to mutate bundled skills' },
+    held_out_path: { type: 'string', description: 'Path to a held-out test set (JSONL). REQUIRED (>=5 rows) to mutate a bundled skill in place — otherwise the run hard-refuses. Remote callers: must resolve within the skills directory.' },
     dry_run: { type: 'boolean', description: 'Cost preview, no LLM calls' },
   },
   mutating: true,
   scope: 'admin',
   localOnly: false,
   handler: async (ctx, p) => {
+    // SECURITY: skill_name is joined into filesystem paths (SKILL.md, default
+    // benchmark, checkpoint, history, best.md, proposed.md). A traversal-shaped
+    // name (`../`, absolute) would escape the skills dir even WITH the
+    // caller-supplied-path confinement below. Validate kebab-only up front so
+    // every derived path is contained by construction. Applies to all callers.
+    const skillNameRaw = (p.skill_name as string) ?? '';
+    if (!/^[a-z0-9][a-z0-9-]*$/.test(skillNameRaw)) {
+      throw new OperationError(`run_skillopt: skill_name must be kebab-case (matching ^[a-z0-9][a-z0-9-]*$); got '${skillNameRaw}'`, 'invalid_params');
+    }
     if (ctx.remote !== false) {
       // Remote: enforce per-skill allowlist read from config.
       // `skillopt.allowed_skills` is a JSON-array config of skill names
@@ -4560,6 +4570,37 @@ const run_skillopt: Operation = {
     const skillName = p.skill_name as string;
     const benchmarkPath = (p.benchmark_path as string) ??
       `${skillsDir}/${skillName}/skillopt-benchmark.jsonl`;
+    const heldOutPath = p.held_out_path as string | undefined;
+    // SECURITY: remote callers must NOT be able to point benchmark/held-out at
+    // arbitrary host files (loadBenchmark → fs.readFileSync would otherwise be an
+    // arbitrary-read + existence oracle). Confine any caller-supplied path to the
+    // skills directory. Local CLI callers (ctx.remote === false) are unconfined.
+    if (ctx.remote !== false) {
+      const nodePath = await import('node:path');
+      const nodeFs = await import('node:fs');
+      const rootReal = (() => {
+        try { return nodeFs.realpathSync(skillsDir); } catch { return nodePath.resolve(skillsDir); }
+      })();
+      const confine = (label: string, candidate: string | undefined): void => {
+        if (!candidate) return;
+        const resolved = nodePath.resolve(candidate);
+        let real = resolved;
+        try {
+          real = nodeFs.realpathSync(resolved);
+        } catch {
+          // Not yet present: canonicalize the nearest existing ancestor so a
+          // legit in-dir path under a symlinked skillsDir (e.g. macOS /tmp ->
+          // /private/tmp, Conductor worktrees) isn't wrongly rejected.
+          try { real = nodePath.join(nodeFs.realpathSync(nodePath.dirname(resolved)), nodePath.basename(resolved)); }
+          catch { /* parent also missing; fall back to resolved form */ }
+        }
+        if (real !== rootReal && !real.startsWith(rootReal + nodePath.sep)) {
+          throw new OperationError(`run_skillopt: ${label} must resolve within the skills directory for remote callers`, 'permission_denied');
+        }
+      };
+      confine('benchmark_path', p.benchmark_path as string | undefined);
+      confine('held_out_path', heldOutPath);
+    }
     const result = await runSkillOpt({
       engine: ctx.engine,
       skillName,
@@ -4578,6 +4619,7 @@ const run_skillopt: Operation = {
       noMutate: (p.no_mutate as boolean) === true,
       allowMutateBundled: (p.allow_mutate_bundled as boolean) === true,
       bootstrapReviewed: false,
+      ...(heldOutPath ? { heldOutPath } : {}),
       json: true,
       maxCostUsd: (p.max_cost_usd as number) ?? 5.0,
       maxRuntimeMin: 30,

src/core/skillopt/batch.ts

@@ -151,6 +151,8 @@ export interface FleetOpts {
   noMutate: boolean;
   allowMutateBundled: boolean;
   bootstrapReviewed: boolean;
+  /** F11: optional held-out test set (one skill, so a single path is valid here). */
+  heldOutPath?: string;
   maxCostUsd: number;
   maxRuntimeMin: number;
   force: boolean;
@@ -225,6 +227,7 @@ export async function runFleet(opts: FleetOpts): Promise<FleetResult> {
       noMutate,
       allowMutateBundled: opts.allowMutateBundled,
       bootstrapReviewed: opts.bootstrapReviewed,
+      ...(opts.heldOutPath ? { heldOutPath: opts.heldOutPath } : {}),
       json: true,
       maxCostUsd: opts.maxCostUsd,
       maxRuntimeMin: opts.maxRuntimeMin,

src/core/skillopt/bundled-skill-gate.ts

@@ -15,6 +15,8 @@
 import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { autoDetectSkillsDirReadOnly } from '../repo-root.ts';
+import { errorFor } from '../errors.ts';
+import { MIN_HELD_OUT_SIZE } from './held-out.ts';
 import type { BundledSkillContext } from './types.ts';
 
 /**
@@ -69,3 +71,30 @@ export function shouldMutateSkillFile(
   }
   return { mutate: true };
 }
+
+/**
+ * D16 ENFORCE (core mutation policy). Mutating a BUNDLED skill in place
+ * requires a NON-EMPTY held-out set (>= MIN_HELD_OUT_SIZE) — an empty/missing
+ * one passes the held-out gate vacuously and re-opens the benchmark-gaming
+ * hole. Throws (hard-refuse) when the requirement is unmet; no-op otherwise.
+ *
+ * Pure + side-effect-free (no fs, no engine) so every entry point that funnels
+ * through runSkillOpt — CLI, batch, fleet, cycle, background job, run_skillopt
+ * MCP op — inherits the check, and it's unit-testable without install-path
+ * detection (which is always false in a tempdir).
+ */
+export function assertBundledMutationHeldOut(input: {
+  isBundled: boolean;
+  willMutate: boolean;
+  heldOutCount: number;
+  skillName: string;
+}): void {
+  if (input.willMutate && input.isBundled && input.heldOutCount < MIN_HELD_OUT_SIZE) {
+    throw errorFor({
+      class: 'HeldOutRequired',
+      code: 'held_out_required_for_bundled',
+      message: `Mutating bundled skill '${input.skillName}' in place requires a non-empty held-out set (>= ${MIN_HELD_OUT_SIZE} rows); got ${input.heldOutCount}.`,
+      hint: `add --held-out <path> (>= ${MIN_HELD_OUT_SIZE} rows), or drop --allow-mutate-bundled to get proposed.md for review.`,
+    });
+  }
+}

src/core/skillopt/held-out.ts

@@ -29,11 +29,21 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import type { BrainEngine } from '../engine.ts';
 import { loadBenchmark } from './benchmark.ts';
+import { D_SEL_MIN_SIZE } from './types.ts';
 import type { BenchmarkTask } from './types.ts';
-import { runValidationGate } from './validate-gate.ts';
+import { scoreSkillOnTasks } from './validate-gate.ts';
 
 const CAPTURE_CONFIG_KEY = 'skillopt.capture_enabled';
 
+/**
+ * Minimum held-out task count required to gate a BUNDLED-skill mutation. A
+ * too-small (or empty) held-out set passes the gate vacuously, which would
+ * re-open the benchmark-gaming hole the gate exists to close. Derived from the
+ * D_sel floor so the two can't silently desync (the comment used to claim a
+ * mirror that the code didn't enforce).
+ */
+export const MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE;
+
 export function capturesDir(): string {
   const home = process.env.GBRAIN_HOME ?? process.env.HOME ?? '';
   return path.join(home, '.gbrain', 'skillopt-captures');
@@ -123,28 +133,19 @@ export async function runHeldOutGate(opts: HeldOutGateOpts): Promise<HeldOutGate
     return { baselineScore: 0, candidateScore: 0, passed: true };
   }
 
-  const baseline = await runValidationGate({
+  const scoreOpts = {
     engine: opts.engine,
-    candidateSkillText: opts.baselineSkillText,
-    selSet: opts.heldOutTasks,
-    bestScore: -1, // any score accepts; we want the score itself
+    tasks: opts.heldOutTasks,
     targetModel: opts.targetModel,
     judgeModel: opts.judgeModel,
     ...(opts.abortSignal ? { abortSignal: opts.abortSignal } : {}),
-  });
-  const candidate = await runValidationGate({
-    engine: opts.engine,
-    candidateSkillText: opts.candidateSkillText,
-    selSet: opts.heldOutTasks,
-    bestScore: -1,
-    targetModel: opts.targetModel,
-    judgeModel: opts.judgeModel,
-    ...(opts.abortSignal ? { abortSignal: opts.abortSignal } : {}),
-  });
+  };
+  const baselineScore = await scoreSkillOnTasks({ ...scoreOpts, skillText: opts.baselineSkillText });
+  const candidateScore = await scoreSkillOnTasks({ ...scoreOpts, skillText: opts.candidateSkillText });
 
   return {
-    baselineScore: baseline.selScore,
-    candidateScore: candidate.selScore,
-    passed: candidate.selScore >= baseline.selScore,
+    baselineScore,
+    candidateScore,
+    passed: candidateScore >= baselineScore,
   };
 }

src/core/skillopt/help.ts

@@ -40,7 +40,10 @@ Modes:
   --rewrite                     Allow full rewrites of sections
   --dry-run                     Plan + cost estimate, no LLM calls
   --no-mutate                   Write proposed.md without replacing SKILL.md
-  --allow-mutate-bundled        Required when target skill is bundled
+  --allow-mutate-bundled        Required to mutate a bundled skill in place.
+                                ALSO requires --held-out (>=5 rows); without it
+                                the run hard-refuses (exit 2). Drop this flag to
+                                get proposed.md for review instead.
   --json                        Machine-readable stdout
 
 Safety:
@@ -60,12 +63,11 @@ Batch + fleet + background:
                                 skills/<name>/skillopt/fleet/<slug>/
   --background                  Submit as a Minion job + print job_id; exits.
                                 Combine with --follow to attach.
-  --write-capture               Enable virtual put_page / submit_job /
-                                file_upload for write-flavored skills (no
-                                real writes — captured for judge inspection)
-  --held-out <path>             Independent held-out test set; gate refuses
-                                mutation if candidate's held-out score is
-                                below baseline.
+  --held-out <path>             Independent held-out test set (JSONL, same shape
+                                as the benchmark). The held-out gate refuses to
+                                promote a candidate whose held-out score is below
+                                baseline (benchmark-gaming defense). REQUIRED
+                                (>=5 rows) to mutate a bundled skill in place.
 
 Exit codes:
   0 = improved + accepted (or --no-mutate proposed.md written)
@@ -87,8 +89,9 @@ Examples:
   # Dry-run cost preview:
   gbrain skillopt meeting-prep --dry-run
 
-  # Optimize a bundled skill with explicit opt-in:
-  gbrain skillopt brain-ops --allow-mutate-bundled
+  # Optimize a bundled skill in place (requires an independent held-out set):
+  gbrain skillopt brain-ops --allow-mutate-bundled --held-out skills/brain-ops/held-out.jsonl
+  # ...or omit --allow-mutate-bundled to get proposed.md for manual review (no held-out needed).
 
   # Resume after interruption:
   gbrain skillopt meeting-prep --resume <run-id>

src/core/skillopt/orchestrator.ts

@@ -82,19 +82,21 @@ import * as fs from 'node:fs';
 import { BudgetTracker } from '../budget/budget-tracker.ts';
 import { withBudgetTracker } from '../ai/gateway.ts';
 import { errorFor } from '../errors.ts';
-import { applyEditBatch, getWorkingTreeStatusForFile } from './apply-edits.ts';
+import { applyEditBatch, getWorkingTreeStatusForFile, splitFrontmatter } from './apply-edits.ts';
 import { logEvent, sha8 } from './audit.ts';
 import { loadBenchmark, splitBench, parseSplit } from './benchmark.ts';
-import { getBundledSkillContext, shouldMutateSkillFile } from './bundled-skill-gate.ts';
+import { getBundledSkillContext, shouldMutateSkillFile, assertBundledMutationHeldOut } from './bundled-skill-gate.ts';
 import { loadCheckpoint, saveCheckpoint, deleteCheckpoint, type RunCheckpoint } from './checkpoint.ts';
+import { loadHeldOut, runHeldOutGate } from './held-out.ts';
 import { withSkilloptLock } from './lock.ts';
 import { resolveLrSchedule } from './lr-schedule.ts';
 import { preflight, formatPreflightReport } from './preflight.ts';
 import { isRejected, loadRejectedBuffer, makeRejectedEntry, saveRejectedBuffer } from './rejected-buffer.ts';
-import { runReflect } from './reflect.ts';
-import { acceptCandidate, bestPath, revertAllPending, skillPath } from './version-store.ts';
-import { runValidationGate } from './validate-gate.ts';
-import type { SkillOptOpts, EditOp, RunReceipt } from './types.ts';
+import { runReflect, runOneShotRewrite, describeJudges } from './reflect.ts';
+import { acceptCandidate, bestPath, revertAllPending, skillPath, writeProposed } from './version-store.ts';
+import { runValidationGate, scoreSkillOnTasks } from './validate-gate.ts';
+import { ROLLOUT_SUCCESS_THRESHOLD } from './types.ts';
+import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask } from './types.ts';
 
 export interface RunSkillOptResult {
   outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored';
@@ -141,10 +143,42 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResult
     allowMutateBundled: opts.allowMutateBundled,
   });
 
+  // Held-out set (F11). Loaded ONCE here so the bundled-mutation ENFORCE gate
+  // and the per-acceptance held-out gate share the same parsed tasks. Empty
+  // when --held-out is not passed.
+  const heldOutTasks: BenchmarkTask[] = opts.heldOutPath ? loadHeldOut(opts.heldOutPath) : [];
+
+  // D16 ENFORCE (core mutation policy — fires for EVERY caller: CLI, batch,
+  // fleet, cycle, background job, and the run_skillopt MCP op, since they all
+  // funnel through runSkillOpt). Fail-loud BEFORE the lock + any LLM spend.
+  assertBundledMutationHeldOut({
+    isBundled: bundledCtx.isBundled,
+    willMutate: mutateDecision.mutate,
+    heldOutCount: heldOutTasks.length,
+    skillName,
+  });
+
   // Load + validate benchmark (D17 floor enforcement, D15 sentinel check).
   const bench = loadBenchmark(opts.benchmarkPath, { bootstrapReviewed: opts.bootstrapReviewed });
   const split = splitBench(bench, opts.split);
 
+  // Held-out must be INDEPENDENT of the benchmark. If a held-out file shares
+  // task_ids with the benchmark, an overfit candidate passes the held-out gate
+  // vacuously — silently voiding the gate's entire benchmark-gaming defense
+  // (the load-bearing safety claim for bundled mutation). Reject on overlap.
+  if (heldOutTasks.length > 0) {
+    const benchIds = new Set(bench.tasks.map((t) => t.task_id));
+    const overlap = heldOutTasks.filter((t) => benchIds.has(t.task_id)).map((t) => t.task_id);
+    if (overlap.length > 0) {
+      throw errorFor({
+        class: 'HeldOutOverlap',
+        code: 'held_out_overlaps_benchmark',
+        message: `Held-out set shares ${overlap.length} task_id(s) with the benchmark (e.g. ${overlap.slice(0, 3).join(', ')}). The held-out set must be independent.`,
+        hint: `Use disjoint task_ids in the held-out file — an overlapping held-out can't catch benchmark overfitting.`,
+      });
+    }
+  }
+
   // ── Cost preflight (D3) ─────────────────────────────────────────────────
   const preflightResult = preflight({
     epochs: opts.epochs,
@@ -156,6 +190,7 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResult
     targetModel: opts.targetModel,
     judgeModel: opts.judgeModel,
     maxCostUsd: opts.maxCostUsd,
+    heldOutSize: heldOutTasks.length,
     interactive: process.stderr.isTTY === true,
   });
   if (opts.json !== true) {
@@ -198,7 +233,7 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResult
 
   // ── Acquire per-skill lock (D14) ────────────────────────────────────────
   return await withSkilloptLock(engine, skillName, async () => {
-    return runOptimizationLoop(opts, bench, split, bundledCtx, mutateDecision);
+    return runOptimizationLoop(opts, bench, split, bundledCtx, mutateDecision, heldOutTasks);
   });
 }
 
@@ -208,10 +243,20 @@ async function runOptimizationLoop(
   split: ReturnType<typeof splitBench>,
   bundledCtx: ReturnType<typeof getBundledSkillContext>,
   mutateDecision: ReturnType<typeof shouldMutateSkillFile>,
+  heldOutTasks: BenchmarkTask[],
 ): Promise<RunSkillOptResult> {
   const { skillName, skillsDir } = opts;
   const skillFile = skillPath(skillsDir, skillName);
 
+  // Plain-English success criteria from the benchmark judges, fed to the
+  // optimizer's reflect step so it targets WHAT the scorer rewards instead of
+  // guessing from a bare pass/fail score. Without this, rule-judged benchmarks
+  // (e.g. "must include a Confidence: line") are near-unoptimizable: the
+  // optimizer proposes plausible-but-off edits, the candidate scores 0, the gate
+  // rejects it, and the skill never changes. The held-out gate defends against
+  // the optimizer gaming these criteria at the expense of real quality.
+  const benchmarkCriteria = describeJudges([...split.train, ...split.sel, ...split.test]);
+
   // Crash-recovery sweep (D8): revert any pending rows from a prior crashed run.
   revertAllPending(skillsDir, skillName);
 
@@ -262,6 +307,10 @@ async function runOptimizationLoop(
     lr: opts.lr,
     lr_schedule: opts.lrSchedule,
     max_cost_usd: opts.maxCostUsd,
+    reflect_mode: opts.reflectMode ?? 'both',
+    validation_gate_disabled: opts.disableValidationGate === true,
+    optimizer_mode: opts.optimizerMode ?? 'reflect',
+    held_out_size: heldOutTasks.length,
   } as never);
 
   // Budget tracker for the whole run. BudgetExhausted propagates as
@@ -278,6 +327,14 @@ async function runOptimizationLoop(
   let outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored' = 'no_improvement';
   let finalText = checkpoint.best_skill_text;
   let totalStepsRun = 0;
+  // Hoisted for the receipt (computed inside the budget-tracker closure).
+  let baselineSelScore = 0;
+  let testScore: number | undefined;
+  let baselineTestScore: number | undefined;
+  // maxRuntimeMin enforcement: wall-clock deadline checked between steps. A
+  // breach throws `skillopt_runtime_exceeded`, caught below → outcome 'aborted'
+  // (no partial commit beyond whatever already passed every gate).
+  const deadline = Date.now() + opts.maxRuntimeMin * 60_000;
 
   try {
     await withBudgetTracker(tracker, async () => {
@@ -297,7 +354,75 @@ async function runOptimizationLoop(
         checkpoint!.best_skill_text = baselineText;
         saveCheckpoint(skillsDir, skillName, checkpoint!);
       }
-      const baselineSelScore = baselineGate.selScore;
+      baselineSelScore = baselineGate.selScore;
+
+      // ── Ablation (cat31 config C): one-shot rewrite ─────────────────────
+      // ONE rewrite call, no optimization loop, no D12 gate. Forward-pass on
+      // D_sel for signal, rewrite the body once, score the result, promote it.
+      // Sets last_completed_epoch = epochs so the epoch loop below no-ops, then
+      // falls through to the shared final-test + receipt path.
+      if (opts.optimizerMode === 'one-shot-rewrite') {
+        const { body: baselineBody, bodyStart } = splitFrontmatter(baselineText);
+        const fmBlock = baselineText.slice(0, bodyStart);
+        const fwd = await runValidationGate({
+          engine: opts.engine,
+          candidateSkillText: baselineText,
+          selSet: split.sel,
+          bestScore: -1,
+          targetModel: opts.targetModel,
+          judgeModel: opts.judgeModel,
+          runsPerTask: 1,
+        });
+        const rewrite = await runOneShotRewrite({
+          skillBodyText: baselineBody,
+          successes: fwd.scoredRollouts.filter((r) => r.score >= ROLLOUT_SUCCESS_THRESHOLD),
+          failures: fwd.scoredRollouts.filter((r) => r.score < ROLLOUT_SUCCESS_THRESHOLD),
+          rejected: [],
+          criteria: benchmarkCriteria,
+          optimizerModel: opts.optimizerModel,
+        });
+        if (rewrite.newBody) {
+          const candidate = fmBlock + rewrite.newBody;
+          // Held-out gate still applies (independent signal); skipped when unset.
+          let promote = true;
+          if (heldOutTasks.length > 0) {
+            const ho = await runHeldOutGate({
+              engine: opts.engine,
+              candidateSkillText: candidate,
+              baselineSkillText: baselineText,
+              heldOutTasks,
+              targetModel: opts.targetModel,
+              judgeModel: opts.judgeModel,
+            });
+            promote = ho.passed;
+          }
+          if (promote) {
+            const score = await scoreSkillOnTasks({
+              engine: opts.engine,
+              skillText: candidate,
+              tasks: split.sel,
+              targetModel: opts.targetModel,
+              judgeModel: opts.judgeModel,
+            });
+            checkpoint!.best_sel_score = score;
+            checkpoint!.best_skill_text = candidate;
+            totalStepsRun = 1;
+            if (mutateDecision.mutate) {
+              acceptCandidate({
+                skillsDir, skillName, runId, epoch: 1, step: 1,
+                edits: [], // whole-body rewrite — no granular edits to record
+                candidateText: candidate, selScore: score, delta: score - baselineSelScore,
+              });
+            } else {
+              writeProposed(skillsDir, skillName, candidate);
+            }
+            outcome = 'accepted';
+            finalText = candidate;
+          }
+        }
+        checkpoint!.last_completed_epoch = opts.epochs; // no-op the epoch loop below
+        saveCheckpoint(skillsDir, skillName, checkpoint!);
+      }
 
       // Epoch loop.
       for (let epoch = checkpoint!.last_completed_epoch + 1; epoch <= opts.epochs; epoch++) {
@@ -306,6 +431,7 @@ async function runOptimizationLoop(
         const epochStartBest = checkpoint!.best_sel_score;
 
         for (let step = startStep; step <= stepsPerEpoch; step++) {
+          if (Date.now() > deadline) throw new Error('skillopt_runtime_exceeded');
           totalStepsRun += 1;
           const globalStep = (epoch - 1) * stepsPerEpoch + step;
           const lrBudget = scheduleFn(opts.lr, globalStep, totalSteps);
@@ -330,8 +456,8 @@ async function runOptimizationLoop(
           // Partition into successes vs failures (>= 0.5 threshold). Reflect
           // gets the actual scored trajectories so failure-mode + success-mode
           // analysis can ground in real agent behavior (D7).
-          const successes = forwardGate.scoredRollouts.filter((r) => r.score >= 0.5);
-          const failures = forwardGate.scoredRollouts.filter((r) => r.score < 0.5);
+          const successes = forwardGate.scoredRollouts.filter((r) => r.score >= ROLLOUT_SUCCESS_THRESHOLD);
+          const failures = forwardGate.scoredRollouts.filter((r) => r.score < ROLLOUT_SUCCESS_THRESHOLD);
 
           // BACKWARD PASS: D7 two reflect calls (failures + successes).
           const rejected = loadRejectedBuffer(skillsDir, skillName);
@@ -340,7 +466,9 @@ async function runOptimizationLoop(
             successes,
             failures,
             rejected,
+            criteria: benchmarkCriteria,
             optimizerModel: opts.optimizerModel,
+            ...(opts.reflectMode ? { reflectMode: opts.reflectMode } : {}),
             abortSignal: undefined,
           });
 
@@ -385,9 +513,56 @@ async function runOptimizationLoop(
             judgeModel: opts.judgeModel,
           });
 
-          if (gate.accepted) {
+          // Ablation (cat31 config D): disableValidationGate greedy-accepts any
+          // applied edit, bypassing the D12 median+epsilon check. We still ran
+          // the gate above to GET the score (so sel_score tracking is honest),
+          // but ignore its accept verdict. We only reach here when at least one
+          // edit applied (the all-rejected case `continue`d earlier).
+          const stepAccepted = opts.disableValidationGate === true ? true : gate.accepted;
+          if (stepAccepted) {
+            // F11 HELD-OUT GATE — guards CHECKPOINT ACCEPTANCE (not just file
+            // mutation), so the no-mutate / proposed.md paths can't promote a
+            // held-out-failing candidate either. Independent signal: a candidate
+            // that improved D_sel but regresses on the held-out set is refused
+            // (benchmark-gaming defense). Skipped when no held-out is configured.
+            if (heldOutTasks.length > 0) {
+              const ho = await runHeldOutGate({
+                engine: opts.engine,
+                candidateSkillText: applied.newText,
+                baselineSkillText: baselineText,
+                heldOutTasks,
+                targetModel: opts.targetModel,
+                judgeModel: opts.judgeModel,
+              });
+              if (!ho.passed) {
+                const newRejections = fresh.map((e) =>
+                  makeRejectedEntry(checkpoint!.best_skill_text, [e], 'held_out_regression'),
+                );
+                saveRejectedBuffer(skillsDir, skillName, newRejections);
+                logEvent({
+                  kind: 'step',
+                  run_id: runId,
+                  skill: skillName,
+                  epoch,
+                  step,
+                  sel_score_median: gate.selScore,
+                  sel_score_runs: gate.perTaskMedians.map((t) => t.median),
+                  accepted: false,
+                  edits_attempted: fresh.length,
+                  edits_applied: applied.results.filter((r) => r.outcome === 'applied').length,
+                  delta: gate.selScore - checkpoint!.best_sel_score,
+                  reason: 'held_out_regression',
+                  held_out_baseline: ho.baselineScore,
+                  held_out_candidate: ho.candidateScore,
+                  cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,
+                } as never);
+                continue; // do NOT promote
+              }
+            }
+
             const delta = gate.selScore - checkpoint!.best_sel_score;
-            // ACCEPT: D8 commit via version-store.
+            // ACCEPT: D8 commit via version-store (mutate) OR write proposed.md
+            // for the --no-mutate / bundled-without-allow paths.
             if (mutateDecision.mutate) {
               acceptCandidate({
                 skillsDir,
@@ -400,6 +575,8 @@ async function runOptimizationLoop(
                 selScore: gate.selScore,
                 delta,
               });
+            } else {
+              writeProposed(skillsDir, skillName, applied.newText);
             }
             checkpoint!.best_sel_score = gate.selScore;
             checkpoint!.best_skill_text = applied.newText;
@@ -465,10 +642,26 @@ async function runOptimizationLoop(
         saveCheckpoint(skillsDir, skillName, checkpoint!);
       }
 
-      // FINAL TEST: score the best skill on D_test.
-      // For v1, we don't fire the test eval — that's a follow-up.
-      // The final receipt records the baseline + best sel scores.
-      void baselineSelScore;
+      // FINAL TEST: score the best skill AND the baseline on D_test so the
+      // receipt carries an honest held-out generalization signal (the eval
+      // harnesses also compute this themselves, but a truthful receipt removes
+      // the prior hardcoded-0 lie). Skipped when D_test is empty.
+      if (split.test.length > 0) {
+        testScore = await scoreSkillOnTasks({
+          engine: opts.engine,
+          skillText: checkpoint!.best_skill_text,
+          tasks: split.test,
+          targetModel: opts.targetModel,
+          judgeModel: opts.judgeModel,
+        });
+        baselineTestScore = await scoreSkillOnTasks({
+          engine: opts.engine,
+          skillText: baselineText,
+          tasks: split.test,
+          targetModel: opts.targetModel,
+          judgeModel: opts.judgeModel,
+        });
+      }
     });
   } catch (err) {
     const msg = err instanceof Error ? err.message : String(err);
@@ -481,6 +674,15 @@ async function runOptimizationLoop(
         reason: 'budget_exhausted',
         detail: msg,
       } as never);
+    } else if (msg.includes('skillopt_runtime_exceeded')) {
+      outcome = 'aborted';
+      logEvent({
+        kind: 'abort',
+        run_id: runId,
+        skill: skillName,
+        reason: 'runtime_exceeded',
+        detail: `exceeded --max-runtime-min ${opts.maxRuntimeMin}`,
+      } as never);
     } else {
       outcome = 'errored';
       logEvent({
@@ -500,10 +702,9 @@ async function runOptimizationLoop(
   // to the catch's assignment values only (it can't prove the async callback ran).
   const finalOutcome = outcome as 'accepted' | 'no_improvement' | 'aborted' | 'errored';
   if (!mutateDecision.mutate && finalOutcome === 'accepted') {
-    proposedPath = bestPath(skillsDir, skillName); // best.md doubles as proposed.md
-    // Note: acceptCandidate is gated by mutateDecision.mutate above, so best.md
-    // isn't written in --no-mutate mode. Write it explicitly here.
-    // (Simplified for v1 — a follow-up routes the proposed-only path cleanly.)
+    // best.md was written by writeProposed() in the accept branch (no-mutate
+    // path); it doubles as proposed.md for human review. SKILL.md untouched.
+    proposedPath = bestPath(skillsDir, skillName);
   } else if (mutateDecision.mutate) {
     mutatedSkillFile = finalOutcome === 'accepted';
   }
@@ -525,11 +726,17 @@ async function runOptimizationLoop(
     started_at: checkpoint.started_at,
     ended_at: new Date().toISOString(),
     outcome,
-    baseline_sel_score: 0,
+    baseline_sel_score: baselineSelScore,
     best_sel_score: checkpoint.best_sel_score,
+    ...(testScore !== undefined ? { test_score: testScore } : {}),
+    ...(baselineTestScore !== undefined ? { baseline_test_score: baselineTestScore } : {}),
     final_cost_usd: tracker.snapshot().cumulativeCostUsd,
     total_steps: totalStepsRun,
     epochs_completed: checkpoint.last_completed_epoch,
+    // Ablation provenance (cat31 replayability) — only when a non-default knob set.
+    ...(opts.reflectMode ? { reflect_mode: opts.reflectMode } : {}),
+    ...(opts.disableValidationGate ? { validation_gate_disabled: true } : {}),
+    ...(opts.optimizerMode ? { optimizer_mode: opts.optimizerMode } : {}),
   };
 
   logEvent({

src/core/skillopt/preflight.ts

@@ -46,6 +46,13 @@ export interface PreflightOpts {
   targetModel: string;
   judgeModel: string;
   maxCostUsd: number;
+  /**
+   * Held-out task count (F11). When > 0, the held-out gate scores baseline +
+   * candidate on the held-out set at every accepted step. Conservatively
+   * priced as if EVERY step accepts (upper bound) so the cap is honest.
+   * 0 / omitted = no held-out gate.
+   */
+  heldOutSize?: number;
   /** When true, print the prompt to stderr + use Ctrl-C grace. Default false (non-TTY). */
   interactive?: boolean;
 }
@@ -82,16 +89,25 @@ export function estimateCost(opts: PreflightOpts): PreflightEstimate {
   const reflectsPerStep = 2; // D7: two reflect calls
   const sel_runs_per_step = opts.selSize * VALIDATION_RUNS_PER_TASK;
 
+  // F11 held-out: baseline + candidate scored on the held-out set at every
+  // accepted step. Upper-bound: assume every step accepts (2 = baseline+candidate).
+  const heldOutSize = opts.heldOutSize ?? 0;
+  const heldOutRollouts = heldOutSize > 0
+    ? totalSteps * heldOutSize * VALIDATION_RUNS_PER_TASK * 2
+    : 0;
+
   // Cumulative counts across the whole run.
   const rollout_calls = totalSteps * rolloutsPerStep
     + opts.selSize * VALIDATION_RUNS_PER_TASK // baseline sel eval
     + opts.selSize * VALIDATION_RUNS_PER_TASK * totalSteps // per-step sel validation
-    + opts.testSize; // final test eval
+    + opts.testSize * 2 // final test eval (best + baseline)
+    + heldOutRollouts; // F11 held-out gate (baseline+candidate per accepted step)
   const reflect_calls = totalSteps * reflectsPerStep
     + opts.epochs; // slow-update meta calls
   const judge_calls = opts.selSize // baseline (1 per task; median-of-3 is in the rollout count already? — no, judge runs per rollout)
     + opts.selSize * VALIDATION_RUNS_PER_TASK * totalSteps // per-step validation
-    + opts.testSize; // final test judges
+    + opts.testSize * 2 // final test judges (best + baseline)
+    + heldOutRollouts; // F11 held-out judges (1 per held-out rollout)
 
   // Cost per call type.
   const targetPrice = lookupPrice(opts.targetModel);

src/core/skillopt/reflect.ts

@@ -14,9 +14,56 @@
  */
 
 import { chat as gatewayChat } from '../ai/gateway.ts';
-import type { EditOp, ScoredRollout } from './types.ts';
+import type { EditOp, ScoredRollout, Judge, RuleCheck } from './types.ts';
 import type { RejectedEntry } from './rejected-buffer.ts';
 
+/**
+ * Render ONE rule check as a plain-English requirement the optimizer can target.
+ */
+function describeCheck(c: RuleCheck): string {
+  switch (c.op) {
+    case 'contains': return `the output must contain the exact text \`${c.arg}\``;
+    case 'regex': return `the output must match the regular expression \`/${c.arg}/\``;
+    case 'section_present': return `the output must include a markdown heading titled "${c.arg}" (any heading level, case-insensitive)`;
+    case 'max_chars': return `the output must be at most ${c.arg} characters long`;
+    case 'min_citations': return `the output must include at least ${c.arg} citation(s)`;
+    case 'tool_called': return `the agent must call the \`${c.arg}\` tool at least once`;
+    case 'tool_not_called': return `the agent must NOT call the \`${c.arg}\` tool`;
+  }
+}
+
+/**
+ * Render a Judge into the plain-English criteria the scorer rewards, so the
+ * optimizer knows WHAT it is optimizing toward. Without this the optimizer only
+ * sees a pass/fail score and has to reverse-engineer the target from behavior
+ * alone — which fails for rule judges that require a specific structure (e.g. a
+ * literal "Confidence:" line): it proposes plausible-but-off edits that never
+ * satisfy the rule, the candidate scores 0, the gate rejects it, and the skill
+ * never changes. Reward-hacking is defended separately by the held-out gate.
+ */
+export function describeJudge(judge: Judge): string {
+  switch (judge.kind) {
+    case 'rule': return judge.checks.map((c) => `- ${describeCheck(c)}`).join('\n');
+    case 'llm': return `- the output is graded 0..1 by an LLM judge against this rubric:\n  "${judge.rubric}"`;
+    case 'qrels': return `- the agent must retrieve the expected pages (scored recall@${judge.k})`;
+  }
+}
+
+/**
+ * Describe the DISTINCT success criteria across a set of benchmark tasks. Most
+ * benchmarks use one judge shape for every task, so this collapses to a single
+ * block; heterogeneous benchmarks list each distinct shape once.
+ */
+export function describeJudges(tasks: ReadonlyArray<{ judge: Judge }>): string {
+  const seen = new Set<string>();
+  const blocks: string[] = [];
+  for (const t of tasks) {
+    const desc = describeJudge(t.judge);
+    if (!seen.has(desc)) { seen.add(desc); blocks.push(desc); }
+  }
+  return blocks.join('\n');
+}
+
 const FAILURE_REFLECT_SYSTEM = `You are SkillOpt's optimizer. You analyze AGENT FAILURE TRAJECTORIES and propose specific edits to a SKILL document so the agent does better next time.
 
 Output ONLY a single JSON object on one or more lines:
@@ -33,7 +80,8 @@ Rules:
 - Do NOT propose edits already in the rejected-edit history — those were tried and didn't help.
 - Be SURGICAL. Small targeted edits outperform large rewrites.
 - Do NOT modify the YAML frontmatter (triggers, brain_first, etc.) — that's out of scope.
-- Output at MOST 8 edits. The orchestrator's LR budget will rank-and-clip further.`;
+- Output at MOST 8 edits. The orchestrator's LR budget will rank-and-clip further.
+- You may be given SUCCESS CRITERIA describing exactly how the agent's output is scored. Make your edits cause the agent to SATISFY those criteria, through genuine, high-quality content (a real section with real substance, a justified confidence level) — never by inserting empty keywords. An independent held-out check rejects edits that game the score while hurting real quality.`;
 
 const SUCCESS_REFLECT_SYSTEM = `You are SkillOpt's optimizer. You analyze AGENT SUCCESS TRAJECTORIES and propose specific edits to a SKILL document so the agent CONSISTENTLY does what worked here.
 
@@ -51,7 +99,18 @@ export interface ReflectOpts {
   failures: ScoredRollout[];
   /** Rejected-edit buffer for anti-bias context. */
   rejected: readonly RejectedEntry[];
+  /**
+   * Plain-English description of how the agent's output is scored (from
+   * `describeJudges(benchmarkTasks)`). Threaded into the reflect prompt so the
+   * optimizer targets the actual criteria instead of guessing from score alone.
+   */
+  criteria?: string;
   optimizerModel: string;
+  /**
+   * Ablation (cat31 config B): 'failure-only' skips the D7 success-reflect call
+   * entirely (even when successes are present). Default 'both' (paper-faithful).
+   */
+  reflectMode?: 'both' | 'failure-only';
   /** Test seam — substitute for gateway.chat. */
   chatFn?: typeof gatewayChat;
   abortSignal?: AbortSignal;
@@ -84,13 +143,63 @@ export async function runReflect(opts: ReflectOpts): Promise<ReflectResult> {
   const failureEdits = opts.failures.length > 0
     ? await callReflect('failure', opts, FAILURE_REFLECT_SYSTEM, opts.failures, usage, errors)
     : [];
-  const successEdits = opts.successes.length > 0
+  // Ablation: 'failure-only' skips the success-reflect call regardless of data.
+  const successEdits = opts.reflectMode !== 'failure-only' && opts.successes.length > 0
     ? await callReflect('success', opts, SUCCESS_REFLECT_SYSTEM, opts.successes, usage, errors)
     : [];
 
   return { failureEdits, successEdits, usage, errors };
 }
 
+const ONE_SHOT_REWRITE_SYSTEM = `You are SkillOpt's optimizer in ONE-SHOT REWRITE mode. Given a SKILL document body and a batch of agent rollouts (some failing, some succeeding), rewrite the ENTIRE body ONCE to make the agent succeed more often.
+
+Output ONLY the rewritten skill body as markdown — no JSON, no code fence, no preamble, no commentary. Do NOT include or modify the YAML frontmatter (it is not shown to you and is out of scope). Keep the same general structure and headings unless a change clearly helps; be surgical, not verbose.`;
+
+export interface OneShotRewriteResult {
+  /** The rewritten skill body (frontmatter NOT included — caller re-attaches). */
+  newBody: string;
+  usage: ReflectResult['usage'];
+  /** Set when the rewrite call errored (caller treats as "no change"). */
+  error?: string;
+}
+
+/**
+ * Ablation baseline (cat31 config C): a single LLM rewrite of the whole skill
+ * body, no optimization loop and no validation gate. A real method (one-shot
+ * prompt rewrite) — the honest "do you even need the loop?" comparison. Runs
+ * through the SAME apply/score path as the loop (the orchestrator feeds the
+ * returned body to the gate), so the comparison is apples-to-apples.
+ */
+export async function runOneShotRewrite(opts: ReflectOpts): Promise<OneShotRewriteResult> {
+  const usage = { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 };
+  const chat = opts.chatFn ?? gatewayChat;
+  const userMsg = buildReflectUserMessage(opts.skillBodyText, [...opts.failures, ...opts.successes], opts.rejected, opts.criteria);
+  try {
+    const result = await chat({
+      model: opts.optimizerModel,
+      system: ONE_SHOT_REWRITE_SYSTEM,
+      messages: [{ role: 'user', content: userMsg }],
+      maxTokens: 4096,
+      cacheSystem: true,
+      abortSignal: opts.abortSignal,
+    });
+    usage.input_tokens += result.usage.input_tokens;
+    usage.output_tokens += result.usage.output_tokens;
+    usage.cache_read_tokens += result.usage.cache_read_tokens;
+    usage.cache_creation_tokens += result.usage.cache_creation_tokens;
+    // Unwrap a fence ONLY when the model wrapped the ENTIRE response in one
+    // (anchored ^```...```$). A non-anchored match would truncate a legitimate
+    // body that contains a code sample down to just that first fenced block.
+    const trimmed = result.text.trim();
+    const wholeFence = trimmed.match(/^```(?:markdown)?\s*\n([\s\S]*)\n```$/i);
+    const newBody = (wholeFence ? wholeFence[1]! : trimmed).trim();
+    return { newBody, usage };
+  } catch (err) {
+    const msg = err instanceof Error ? err.message : String(err);
+    return { newBody: '', usage, error: `one_shot_rewrite_failed: ${msg}` };
+  }
+}
+
 async function callReflect(
   mode: 'failure' | 'success',
   opts: ReflectOpts,
@@ -100,7 +209,7 @@ async function callReflect(
   errors: string[],
 ): Promise<EditOp[]> {
   const chat = opts.chatFn ?? gatewayChat;
-  const userMsg = buildReflectUserMessage(opts.skillBodyText, scoredRollouts, opts.rejected);
+  const userMsg = buildReflectUserMessage(opts.skillBodyText, scoredRollouts, opts.rejected, opts.criteria);
   try {
     const result = await chat({
       model: opts.optimizerModel,
@@ -126,6 +235,7 @@ function buildReflectUserMessage(
   skillBody: string,
   rollouts: ScoredRollout[],
   rejected: readonly RejectedEntry[],
+  criteria?: string,
 ): string {
   const trajectoryBlocks = rollouts.map((r, i) => {
     const tcSummary = r.trajectory.tool_calls
@@ -144,8 +254,12 @@ ${r.rationale ? `JUDGE RATIONALE: ${r.rationale}` : ''}`;
     ? `\n\n--- PREVIOUSLY REJECTED EDITS (do not re-propose) ---\n${rejected.slice(0, 20).map((r) => `- ${r.reason}: ${JSON.stringify(r.edits)}`).join('\n')}`
     : '';
 
+  const criteriaBlock = criteria
+    ? `\n\nSUCCESS CRITERIA (exactly how the agent's output is scored — make the agent satisfy these through genuine, high-quality content, never empty keywords):\n${criteria}`
+    : '';
+
   return `CURRENT SKILL BODY:
-${truncate(skillBody, 5000)}
+${truncate(skillBody, 5000)}${criteriaBlock}
 
 OBSERVED ROLLOUTS:
 ${trajectoryBlocks}${rejectedSummary}

src/core/skillopt/rollout.ts

@@ -18,6 +18,7 @@
 
 import { chat as gatewayChat, toolLoop, type ChatMessage, type ChatToolDef, type ToolHandler } from '../ai/gateway.ts';
 import { BRAIN_TOOL_ALLOWLIST } from '../minions/tools/brain-allowlist.ts';
+import { paramDefToSchema } from '../../mcp/tool-defs.ts';
 import { operations, type OperationContext } from '../operations.ts';
 import { loadConfig } from '../config.ts';
 import type { BrainEngine } from '../engine.ts';
@@ -207,12 +208,21 @@ function stripBrainPrefix(toolName: string): string {
   return toolName.startsWith('brain_') ? toolName.slice('brain_'.length) : toolName;
 }
 
-function paramsToSchema(params: Record<string, { type: string; description?: string; required?: boolean }>): Record<string, unknown> {
+/**
+ * Build a valid JSON Schema for a tool's params via the shared `paramDefToSchema`
+ * (the single source of truth, also used by the stdio MCP + subagent registries).
+ * The prior inline mapper dropped `items` on array params, producing an invalid
+ * `{type:'array'}` that AI SDK v6's tool-schema validation rejects — every real
+ * rollout crashed before this. Recursive on items/enum/default per param.
+ */
+function paramsToSchema(params: Record<string, unknown>): Record<string, unknown> {
   return {
     type: 'object' as const,
     properties: Object.fromEntries(
-      Object.entries(params).map(([k, v]) => [k, { type: v.type, description: v.description }]),
+      Object.entries(params).map(([k, v]) => [k, paramDefToSchema(v as never)]),
     ),
-    required: Object.entries(params).filter(([, v]) => v.required).map(([k]) => k),
+    required: Object.entries(params)
+      .filter(([, v]) => (v as { required?: boolean }).required === true)
+      .map(([k]) => k),
   };
 }

src/core/skillopt/types.ts

@@ -157,6 +157,18 @@ export interface SkillOptOpts {
   heldOutPath?: string;
   json: boolean;
 
+  // ─── Eval-internal ablation knobs (NOT exposed on the CLI) ───────────────
+  // These exist so the gbrain-evals ablation (cat31) can run the orchestrator
+  // in degraded modes for an apples-to-apples comparison. Defaults preserve
+  // the full production pipeline. `disableValidationGate` MUST NOT be wired to
+  // any user-facing flag — it disables the core safety gate.
+  /** Ablation: 'failure-only' skips the D7 success reflect call. Default 'both'. */
+  reflectMode?: 'both' | 'failure-only';
+  /** Ablation: greedy-accept every applied edit, skip the D12 median+epsilon gate. Default false. */
+  disableValidationGate?: boolean;
+  /** Ablation: 'one-shot-rewrite' does ONE rewrite call, no loop, no gate. Default 'reflect'. */
+  optimizerMode?: 'reflect' | 'one-shot-rewrite';
+
   // Safety.
   maxCostUsd: number;
   maxRuntimeMin: number;
@@ -201,6 +213,11 @@ export interface RunReceipt {
   final_cost_usd?: number;
   total_steps?: number;
   epochs_completed?: number;
+  // Ablation provenance (cat31 replayability) — present when a non-default
+  // ablation knob was set.
+  reflect_mode?: 'both' | 'failure-only';
+  validation_gate_disabled?: boolean;
+  optimizer_mode?: 'reflect' | 'one-shot-rewrite';
 }
 
 export interface HistoryRow {
@@ -220,6 +237,12 @@ export interface HistoryRow {
 export const VALIDATION_EPSILON = 0.05;
 /** D12: number of judge runs per sel-task for noise rejection. */
 export const VALIDATION_RUNS_PER_TASK = 3;
+/**
+ * Score at/above which a forward-pass rollout counts as a "success" (fed to the
+ * success-reflect call) vs a "failure" (fed to the failure-reflect call). Single
+ * source of truth for the D7 partition — used by every forward-pass site.
+ */
+export const ROLLOUT_SUCCESS_THRESHOLD = 0.5;
 
 export interface GateInput {
   candidateSkillText: string;

src/core/skillopt/validate-gate.ts

@@ -20,7 +20,7 @@
  * which is cached (D11), so the effective cost ~1.3x not 3x.
  */
 
-import { runWithLimit } from '../worker-pool.ts';
+import { runWithLimit, isMustAbortError } from '../worker-pool.ts';
 import { runRollout, type RolloutOpts } from './rollout.ts';
 import { scoreTrajectory } from './score.ts';
 import type { BenchmarkTask, GateInput, GateResult, ScoredRollout } from './types.ts';
@@ -94,8 +94,16 @@ export async function runValidationGate(opts: ValidateGateOpts): Promise<GateRes
     signal: opts.abortSignal,
   });
 
-  // SettledItem<TOut>[] — extract successful results; treat errors as score=0
-  // (pessimistic fallback consistent with the judge fail-open posture).
+  // MUST-ABORT errors (budget exhaustion / no-pricing) are NOT scoring noise —
+  // swallowing them as score=0 turns a pricing/cap crash into a fake "0/N" run
+  // (the bug the SkillOpt eval surfaced: a Haiku run with --max-cost hit
+  // no_pricing on every rollout and the whole gate reported a vacuous 0). Surface
+  // them loudly so the caller aborts instead of recording a hollow measurement.
+  const aborter = settled.find((s) => s && !s.ok && isMustAbortError(s.error));
+  if (aborter && !aborter.ok) throw aborter.error;
+
+  // SettledItem<TOut>[] — extract successful results; treat (non-abort) errors as
+  // score=0 (pessimistic fallback consistent with the judge fail-open posture).
   // Errored tasks contribute no scoredRollouts (caller's reflect sees fewer
   // trajectories rather than fabricated zero-score entries).
   const perTaskMedians = settled.map((s, idx) => {
@@ -120,6 +128,43 @@ export async function runValidationGate(opts: ValidateGateOpts): Promise<GateRes
   return { accepted, perTaskMedians, selScore, scoredRollouts, ...(reason ? { reason } : {}) };
 }
 
+export interface ScoreOnTasksOpts {
+  engine: BrainEngine;
+  skillText: string;
+  tasks: BenchmarkTask[];
+  targetModel: string;
+  judgeModel?: string;
+  /** Median-of-N runs per task. Defaults to `VALIDATION_RUNS_PER_TASK` (3). */
+  runsPerTask?: number;
+  abortSignal?: AbortSignal;
+  rolloutFn?: typeof runRollout;
+  scoreFn?: typeof scoreTrajectory;
+}
+
+/**
+ * Score a skill on an arbitrary task set and return the mean-of-per-task-medians
+ * (`selScore`). The single primitive for "how good is this skill on these tasks?"
+ * — used by the orchestrator's baseline + final-test eval, the held-out gate, and
+ * the Track B eval harnesses, so they can't drift on scoring semantics. Thin
+ * wrapper over `runValidationGate({ bestScore: -1 })` (any score "accepts"; we
+ * only read `.selScore`).
+ */
+export async function scoreSkillOnTasks(opts: ScoreOnTasksOpts): Promise<number> {
+  const gate = await runValidationGate({
+    engine: opts.engine,
+    candidateSkillText: opts.skillText,
+    selSet: opts.tasks,
+    bestScore: -1,
+    targetModel: opts.targetModel,
+    ...(opts.judgeModel !== undefined ? { judgeModel: opts.judgeModel } : {}),
+    ...(opts.runsPerTask !== undefined ? { runsPerTask: opts.runsPerTask } : {}),
+    ...(opts.abortSignal ? { abortSignal: opts.abortSignal } : {}),
+    ...(opts.rolloutFn ? { rolloutFn: opts.rolloutFn } : {}),
+    ...(opts.scoreFn ? { scoreFn: opts.scoreFn } : {}),
+  });
+  return gate.selScore;
+}
+
 /** Pure median for an array of numbers. Returns 0 for empty array. */
 export function median(values: readonly number[]): number {
   if (values.length === 0) return 0;

src/core/skillopt/version-store.ts

@@ -170,6 +170,20 @@ export function acceptCandidate(input: AcceptInput): AcceptResult {
   return { versionN, versionFilePath: verPath };
 }
 
+/**
+ * Write the candidate to `best.md` (which doubles as `proposed.md`) WITHOUT
+ * touching SKILL.md or the history ledger. Used by the `--no-mutate` /
+ * bundled-without-allow paths: the optimizer found a better candidate but the
+ * caller opted out of in-place mutation, so we surface it for human review.
+ * Returns the path written. Atomic (.tmp + rename).
+ */
+export function writeProposed(skillsDir: string, skillName: string, candidateText: string): string {
+  const p = bestPath(skillsDir, skillName);
+  fs.mkdirSync(path.dirname(p), { recursive: true });
+  atomicWrite(p, candidateText);
+  return p;
+}
+
 /**
  * Crash-recovery: walk history.json, find any row with `status: 'pending'`
  * whose committed counterpart doesn't exist, and revert:

test/build-llms.test.ts

@@ -96,3 +96,52 @@ describe("build-llms generator", () => {
     ).toBeLessThan(FULL_SIZE_BUDGET);
   });
 });
+
+// Content contracts for the CLAUDE.md resolver restructure. The restructure moved
+// the per-file index + testing discipline into on-demand docs and (per codex
+// outside-voice) keeps the ship-critical IRON RULES inline. These pin that the
+// safety-relevant content did NOT silently move out of CLAUDE.md, and that the
+// new docs are wired into the bundle the way intended (KEY_FILES link-only,
+// not inlined).
+describe("CLAUDE.md restructure content contracts", () => {
+  const claude = () => readFileSync(join(repoRoot, "CLAUDE.md"), "utf8");
+
+  test("CLAUDE.md keeps the inline ship IRON RULES (must NOT move to a doc)", () => {
+    const c = claude();
+    // Version format — the table stays inline (CI version-gate depends on it).
+    expect(c).toContain("MAJOR.MINOR.PATCH.MICRO");
+    // Post-ship discipline — /document-release stays referenced inline.
+    expect(c.toLowerCase()).toContain("document-release");
+    // Never hand-roll ship.
+    expect(c.toLowerCase()).toMatch(/hand-roll ship/);
+  });
+
+  test("CLAUDE.md carries the resolver + cross-cutting invariants (orientation survived)", () => {
+    const c = claude();
+    expect(c).toContain("## Reference map");
+    expect(c).toContain("docs/architecture/KEY_FILES.md");
+    // Invariants that used to live in the per-file index now load here.
+    expect(c).toContain("sourceScopeOpts");
+    expect(c).toContain("JSON.stringify"); // the JSONB trap rule
+  });
+
+  test("AGENTS.md keeps its boot order + points at the new docs (not reduced to a pointer)", () => {
+    const agents = readFileSync(join(repoRoot, "AGENTS.md"), "utf8");
+    expect(agents).toContain("Read this order");
+    expect(agents).toContain("docs/architecture/KEY_FILES.md");
+    expect(agents).toContain("docs/TESTING.md");
+  });
+
+  test("llms.txt indexes the relocated docs", () => {
+    const { llmsTxt } = buildLlmsFiles();
+    expect(llmsTxt).toContain("docs/architecture/KEY_FILES.md");
+    expect(llmsTxt).toContain("docs/TESTING.md");
+    expect(llmsTxt).toContain("docs/architecture/thin-client.md");
+  });
+
+  test("llms-full.txt does NOT inline KEY_FILES.md (link-only; keeps the bundle lean)", () => {
+    const { llmsFullTxt } = buildLlmsFiles();
+    // This H1 appears only in KEY_FILES.md; if it were inlined the bundle would carry it.
+    expect(llmsFullTxt).not.toContain("# Key files — per-file index (gbrain repo)");
+  });
+});

test/e2e/skillopt-loop.serial.test.ts

@@ -134,7 +134,30 @@ interface Fixture {
   cleanup: () => void;
 }
 
-function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
+/** 50 tasks all checking `contains: Citations` — baseline People-only fails them all. */
+const CITATIONS_BENCHMARK = Array.from({ length: 50 }, (_, i) => {
+  const n = String(i + 1).padStart(3, '0');
+  return {
+    task_id: `cit-${n}`,
+    task: `Process task ${i + 1}`,
+    judge: { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'Citations' }] },
+  };
+});
+
+/** Held-out set checking `contains: People` — baseline passes, a People-dropping candidate fails. */
+const PEOPLE_HELDOUT = Array.from({ length: 6 }, (_, i) => {
+  const n = String(i + 1).padStart(3, '0');
+  return {
+    task_id: `ho-${n}`,
+    task: `Held-out task ${i + 1}`,
+    judge: { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'People' }] },
+  };
+});
+
+function setupFixture(
+  skillBody: string = SKILL_PEOPLE_ONLY,
+  benchmark: ReadonlyArray<{ task_id: string; task: string; judge: unknown }> = SAMPLE_BENCHMARK,
+): Fixture {
   const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-loop-e2e-'));
   const skillDir = path.join(tmp, SKILL);
   fs.mkdirSync(skillDir, { recursive: true });
@@ -142,7 +165,7 @@ function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
   const benchmarkPath = path.join(skillDir, 'skillopt-benchmark.jsonl');
   fs.writeFileSync(
     benchmarkPath,
-    SAMPLE_BENCHMARK.map((t) => JSON.stringify(t)).join('\n') + '\n',
+    benchmark.map((t) => JSON.stringify(t)).join('\n') + '\n',
   );
   return {
     skillsDir: tmp,
@@ -153,6 +176,13 @@ function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
   };
 }
 
+/** Write a held-out JSONL into the fixture's skill dir; return its path. */
+function writeHeldOut(fixture: Fixture, tasks: ReadonlyArray<{ task_id: string; task: string; judge: unknown }>): string {
+  const p = path.join(fixture.skillsDir, SKILL, 'held-out.jsonl');
+  fs.writeFileSync(p, tasks.map((t) => JSON.stringify(t)).join('\n') + '\n');
+  return p;
+}
+
 // ─── Stub builder ───────────────────────────────────────────────────────────
 
 interface StubOpts {
@@ -176,10 +206,17 @@ interface StubOpts {
    * fast against a tight cap).
    */
   perCallUsage?: { input: number; output: number };
+  /** Raw body returned for ONE-SHOT REWRITE optimizer calls (optimizerMode test). */
+  oneShotBody?: string;
+  /** Counters incremented as the stub observes each call kind (ablation tests). */
+  stats?: { successReflectCalls: number; oneShotCalls: number };
 }
 
-const REFLECT_OPTIMIZER_PREFIX = "You are SkillOpt's optimizer.";
+// No trailing period: matches the FAILURE/SUCCESS reflect systems ("...optimizer.")
+// AND the one-shot system ("...optimizer in ONE-SHOT REWRITE mode.").
+const REFLECT_OPTIMIZER_PREFIX = "You are SkillOpt's optimizer";
 const FAILURE_REFLECT_MARKER = 'FAILURE TRAJECTORIES';
+const ONE_SHOT_MARKER = 'ONE-SHOT REWRITE';
 
 function defaultTargetText(skillText: string): string {
   // Faithful agent: read the skill's body, emit sections that exist there.
@@ -219,10 +256,16 @@ function installStub(opts: StubOpts): void {
 
     if (isOptimizerCall) {
       const model = chatOpts.model ?? 'anthropic:claude-opus-4-7';
+      // ONE-SHOT REWRITE mode returns a raw body, not edits JSON.
+      if (sys.includes(ONE_SHOT_MARKER)) {
+        if (opts.stats) opts.stats.oneShotCalls += 1;
+        return makeChatResult(opts.oneShotBody ?? '', model, usage);
+      }
       if (opts.optimizerRaw !== undefined) {
         return makeChatResult(opts.optimizerRaw, model, usage);
       }
       const isFailureMode = sys.includes(FAILURE_REFLECT_MARKER);
+      if (!isFailureMode && opts.stats) opts.stats.successReflectCalls += 1;
       const edit = isFailureMode ? opts.failureEdit : opts.successEdit;
       const text = JSON.stringify({ edits: edit ? [edit] : [] });
       return makeChatResult(text, model, usage);
@@ -245,6 +288,12 @@ interface RunOptsOverride {
   maxCostUsd?: number;
   epochs?: number;
   batchSize?: number;
+  noMutate?: boolean;
+  heldOutPath?: string;
+  optimizerMode?: 'reflect' | 'one-shot-rewrite';
+  reflectMode?: 'both' | 'failure-only';
+  disableValidationGate?: boolean;
+  maxRuntimeMin?: number;
 }
 
 async function runOnce(fixture: Fixture, over: RunOptsOverride = {}) {
@@ -263,13 +312,17 @@ async function runOnce(fixture: Fixture, over: RunOptsOverride = {}) {
     judgeModel: 'anthropic:claude-sonnet-4-6',
     mode: 'patch',
     dryRun: false,
-    noMutate: false,
+    noMutate: over.noMutate ?? false,
     allowMutateBundled: true,
     bootstrapReviewed: false,
     json: true,
     maxCostUsd: over.maxCostUsd ?? 100,
-    maxRuntimeMin: 1,
+    maxRuntimeMin: over.maxRuntimeMin ?? 1,
     force: true, // bypass dirty-tree (tempdir isn't a git repo)
+    ...(over.heldOutPath ? { heldOutPath: over.heldOutPath } : {}),
+    ...(over.optimizerMode ? { optimizerMode: over.optimizerMode } : {}),
+    ...(over.reflectMode ? { reflectMode: over.reflectMode } : {}),
+    ...(over.disableValidationGate ? { disableValidationGate: over.disableValidationGate } : {}),
   });
 }
 
@@ -629,3 +682,267 @@ describe('skillopt full-loop E2E (happy path + broken cases)', () => {
     }
   });
 });
+
+// ─── T3: held-out gate + ablation opts + no-DB-pollution ─────────────────────
+
+describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', () => {
+  test('F11 held-out BLOCKS: candidate passes D_sel but regresses held-out → no commit', async () => {
+    // baseline People-only fails the Citations benchmark; the failure edit
+    // REPLACES People with Citations → candidate passes D_sel (Citations) but
+    // tanks the held-out (People) → held-out gate refuses the commit.
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const heldOutPath = writeHeldOut(fixture, PEOPLE_HELDOUT);
+    try {
+      installStub({
+        failureEdit: {
+          op: 'replace',
+          target: '## People\nList people mentioned.',
+          replacement: '## Citations\nCite the source.',
+          reason: 'swap People for Citations to pass the benchmark',
+        },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { heldOutPath });
+          // Held-out gate blocked the promotion.
+          expect(result.outcome).toBe('no_improvement');
+          // SKILL.md unchanged: still People-only, never swapped to Citations.
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## People');
+          expect(skill).not.toContain('## Citations');
+          // No committed history row.
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('F11 held-out ALLOWS: candidate improves D_sel AND holds held-out → commit', async () => {
+    // ADD Citations (keep People): passes D_sel (Citations) and keeps held-out
+    // (People) at baseline → held-out gate allows → commit.
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const heldOutPath = writeHeldOut(fixture, PEOPLE_HELDOUT);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add Citations' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { heldOutPath });
+          expect(result.outcome).toBe('accepted');
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## People');
+          expect(skill).toContain('## Citations');
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(1);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('--no-mutate writes proposed.md (best.md), leaves SKILL.md untouched', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add Citations' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { noMutate: true });
+          expect(result.outcome).toBe('accepted');
+          expect(result.mutatedSkillFile).toBe(false);
+          expect(result.proposedPath).toBeDefined();
+          // proposed.md (best.md) exists and carries the improvement.
+          expect(fs.existsSync(result.proposedPath!)).toBe(true);
+          expect(fs.readFileSync(result.proposedPath!, 'utf8')).toContain('## Citations');
+          // SKILL.md on disk is UNCHANGED (still People-only).
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).not.toContain('## Citations');
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('optimizerMode one-shot-rewrite: single rewrite, no epoch loop, receipt records mode', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const stats = { successReflectCalls: 0, oneShotCalls: 0 };
+    try {
+      installStub({
+        stats,
+        // Body-only rewrite (frontmatter re-attached by the orchestrator).
+        oneShotBody: '# E2E Loop Test Skill\n\nProduce a structured output.\n\n## People\nList people.\n\n## Citations\nCite the source.\n',
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { optimizerMode: 'one-shot-rewrite' });
+          expect(result.outcome).toBe('accepted');
+          expect(result.receipt.optimizer_mode).toBe('one-shot-rewrite');
+          // Exactly ONE optimizer rewrite call — no epoch loop.
+          expect(stats.oneShotCalls).toBe(1);
+          expect(result.receipt.total_steps).toBe(1);
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## Citations');
+          // Frontmatter preserved (D5).
+          expect(skill).toContain('name: e2e-loop-skill');
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('reflectMode failure-only SKIPS the success reflect call; default fires it', async () => {
+    // Mixed benchmark → baseline has both successes (People tasks) and failures
+    // (Citations tasks), so default mode WOULD fire the success reflect.
+    const failOnly = { successReflectCalls: 0, oneShotCalls: 0 };
+    const fixtureA = setupFixture(SKILL_PEOPLE_ONLY, SAMPLE_BENCHMARK);
+    try {
+      installStub({
+        stats: failOnly,
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: { op: 'add', anchor: 'People', content: '<!-- success note -->', reason: 'note' },
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureA.skillsDir }, async () => {
+          await runOnce(fixtureA, { reflectMode: 'failure-only' });
+          expect(failOnly.successReflectCalls).toBe(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureA.cleanup(); }
+
+    const both = { successReflectCalls: 0, oneShotCalls: 0 };
+    const fixtureB = setupFixture(SKILL_PEOPLE_ONLY, SAMPLE_BENCHMARK);
+    try {
+      installStub({
+        stats: both,
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: { op: 'add', anchor: 'People', content: '<!-- success note -->', reason: 'note' },
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureB.skillsDir }, async () => {
+          await runOnce(fixtureB);
+          expect(both.successReflectCalls).toBeGreaterThan(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureB.cleanup(); }
+  });
+
+  test('disableValidationGate greedy-accepts a no-improvement edit the gate would reject', async () => {
+    // BOTH_SECTIONS already scores 1.0; a benign success edit yields delta 0,
+    // which the D12 gate rejects — unless disableValidationGate greedy-accepts.
+    const benignEdit = { op: 'add' as const, anchor: 'Citations', content: 'Extra note.', reason: 'benign' };
+
+    const fixtureGated = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: benignEdit, failureEdit: null });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureGated.skillsDir }, async () => {
+          const result = await runOnce(fixtureGated);
+          expect(result.outcome).toBe('no_improvement'); // gate rejected
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureGated.cleanup(); }
+
+    const fixtureGreedy = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: benignEdit, failureEdit: null });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureGreedy.skillsDir }, async () => {
+          const result = await runOnce(fixtureGreedy, { disableValidationGate: true });
+          expect(result.outcome).toBe('accepted'); // greedy
+          expect(result.receipt.validation_gate_disabled).toBe(true);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureGreedy.cleanup(); }
+  });
+
+  test('D2 no-DB-pollution: subagent_messages count unchanged across a full run', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const countMessages = async (): Promise<number> => {
+      const r = await engine.executeRaw('SELECT COUNT(*) AS c FROM subagent_messages', []);
+      const rows = Array.isArray(r) ? r : ((r as { rows?: unknown[] }).rows ?? []);
+      return Number((rows[0] as { c?: number | string })?.c ?? 0);
+    };
+    try {
+      const before = await countMessages();
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          await runOnce(fixture);
+        });
+      } finally { uninstallStub(); }
+      const after = await countMessages();
+      // Rollouts use gateway.toolLoop with no-op persistence (D2) → zero rows written.
+      expect(after).toBe(before);
+    } finally { fixture.cleanup(); }
+  });
+});
+
+// ─── T3 (review follow-ups): maxRuntimeMin abort + receipt honesty ───────────
+
+describe('skillopt T3 — runtime deadline + receipt score honesty', () => {
+  test('maxRuntimeMin deadline aborts cleanly with no commit', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          // maxRuntimeMin:0 → deadline == run start; the first step's deadline
+          // check fires after the baseline eval has already elapsed → abort.
+          const result = await runOnce(fixture, { maxRuntimeMin: 0 });
+          expect(result.outcome).toBe('aborted');
+          // No commit: SKILL.md unchanged, no committed history row.
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).not.toContain('## Citations');
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('receipt records the REAL baseline_sel_score + final-test scores (regression: was hardcoded 0)', async () => {
+    // BOTH_SECTIONS already scores ~1.0 on the alternating benchmark, so a real
+    // baseline read is ~1.0 — a hardcoded-0 receipt would fail this immediately.
+    const fixture = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: null, failureEdit: null }); // baseline already perfect; no edits
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const r = (await runOnce(fixture)).receipt;
+          // Real baseline, not the old hardcoded 0.
+          expect(r.baseline_sel_score).toBeGreaterThan(0.9);
+          // Final-test eval populated both test scores (D_test non-empty under 4:1:5).
+          expect(typeof r.test_score).toBe('number');
+          expect(typeof r.baseline_test_score).toBe('number');
+          expect(r.baseline_test_score!).toBeGreaterThan(0.9);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+});
+
+describe('skillopt T3 — held-out independence guard', () => {
+  test('held-out sharing task_ids with the benchmark is rejected (gaming defense)', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    // Held-out file reuses benchmark task_ids (cit-001..006) → must be rejected
+    // before any optimization, since an overlapping held-out can't catch overfit.
+    const heldOutPath = writeHeldOut(fixture, CITATIONS_BENCHMARK.slice(0, 6));
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          await expect(runOnce(fixture, { heldOutPath })).rejects.toThrow(/independent|shares .* task_id/i);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+});

test/gateway-model-messages.test.ts

@@ -0,0 +1,123 @@
+/**
+ * Pins `toModelMessages` — the gbrain ChatMessage[] → AI SDK v6 ModelMessage[]
+ * converter. v6 tightened ModelMessage validation: tool results must ride on a
+ * dedicated `role:'tool'` message with a structured `{type,value}` output part,
+ * not a `role:'user'` message with a bare-value tool-result block (which is how
+ * gbrain's toolLoop pushes them). Without this conversion every multi-turn tool
+ * loop — skillopt rollouts AND production subagent jobs — throws "messages do
+ * not match the ModelMessage[] schema" the moment the model calls a tool.
+ *
+ * Surfaced by the SkillOpt real-LLM eval (Track B). These cases pin the exact
+ * v6 shapes that `generateText` accepts (verified against AI SDK 6.0.174).
+ */
+import { describe, test, expect } from 'bun:test';
+import { toModelMessages, type ChatMessage } from '../src/core/ai/gateway.ts';
+
+describe('toModelMessages — v6 ModelMessage shape', () => {
+  test('string content passes through unchanged', () => {
+    const msgs: ChatMessage[] = [{ role: 'user', content: 'hello' }];
+    expect(toModelMessages(msgs)).toEqual([{ role: 'user', content: 'hello' }]);
+  });
+
+  test('assistant text block maps to {type:text,text}', () => {
+    const msgs: ChatMessage[] = [
+      { role: 'assistant', content: [{ type: 'text', text: 'hi' }] },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      { role: 'assistant', content: [{ type: 'text', text: 'hi' }] },
+    ]);
+  });
+
+  test('assistant tool-call block keeps {toolCallId,toolName,input}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'assistant',
+        content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'x' } }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'assistant',
+        content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'x' } }],
+      },
+    ]);
+  });
+
+  test('tool-result on a user-role message becomes role:tool with json output', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { hits: 0 } }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [
+          { type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { type: 'json', value: { hits: 0 } } },
+        ],
+      },
+    ]);
+  });
+
+  test('string tool-result output becomes {type:text,value}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'echo', output: 'done' }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'echo', output: { type: 'text', value: 'done' } }],
+      },
+    ]);
+  });
+
+  test('errored tool-result becomes {type:error-text,value}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { msg: 'boom' }, isError: true }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [
+          { type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { type: 'error-text', value: '{"msg":"boom"}' } },
+        ],
+      },
+    ]);
+  });
+
+  test('null tool-result output is preserved as json null (not dropped)', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'noop', output: null }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'noop', output: { type: 'json', value: null } }],
+      },
+    ]);
+  });
+
+  test('full multi-turn conversation: user → assistant(tool-call) → tool(result)', () => {
+    const msgs: ChatMessage[] = [
+      { role: 'user', content: 'find widget' },
+      { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'widget' } }] },
+      { role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { hits: 0 } }] },
+    ];
+    const out = toModelMessages(msgs);
+    expect(out).toHaveLength(3);
+    expect((out[0] as any).role).toBe('user');
+    expect((out[1] as any).role).toBe('assistant');
+    expect((out[2] as any).role).toBe('tool');
+    expect((out[2] as any).content[0].output).toEqual({ type: 'json', value: { hits: 0 } });
+  });
+});

test/scripts/check-key-files-current-state.test.ts

@@ -0,0 +1,113 @@
+/**
+ * check-key-files-current-state.test.ts — coverage of the anti-disease guard.
+ *
+ * The guard is the structural backstop that keeps CLAUDE.md from re-bloating:
+ * it bans bolded `**v0.<digit>` release markers in the reference docs and caps
+ * CLAUDE.md size. If the guard is broken, the append-only-history disease can
+ * silently return. This suite pins its contract against fixtures.
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "bun:test";
+import { spawnSync } from "node:child_process";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+const SCRIPT = resolve(import.meta.dir, "..", "..", "scripts/check-key-files-current-state.sh");
+
+let root: string;
+
+beforeEach(() => {
+  root = mkdtempSync(join(tmpdir(), "doc-guard-"));
+  mkdirSync(join(root, "docs/architecture"), { recursive: true });
+});
+afterEach(() => {
+  rmSync(root, { recursive: true, force: true });
+});
+
+function writeDoc(rel: string, content: string) {
+  const abs = join(root, rel);
+  mkdirSync(resolve(abs, ".."), { recursive: true });
+  writeFileSync(abs, content);
+}
+
+function run(extraEnv: Record<string, string> = {}) {
+  return spawnSync("bash", [SCRIPT], {
+    encoding: "utf8",
+    env: { ...process.env, GBRAIN_DOC_GUARD_ROOT: root, ...extraEnv },
+  });
+}
+
+// A minimal clean repo shape the guard is happy with.
+function seedClean() {
+  writeDoc("CLAUDE.md", "# CLAUDE.md\n\norientation only\n");
+  writeDoc(
+    "docs/architecture/KEY_FILES.md",
+    "# Key files\n\n- `src/core/db.ts` — connection management. Pinned by `test/db.test.ts`.\n",
+  );
+  writeDoc("docs/architecture/thin-client.md", "# Thin-client\n\nrouting seam\n");
+  writeDoc("docs/TESTING.md", "# Testing\n\ntiers\n");
+}
+
+describe("check-key-files-current-state.sh", () => {
+  it("passes on a clean current-state repo", () => {
+    seedClean();
+    const r = run();
+    expect(r.status).toBe(0);
+    expect(r.stdout).toContain("ok");
+  });
+
+  it("FAILS when a reference doc carries a bolded release-clause marker", () => {
+    seedClean();
+    writeDoc(
+      "docs/architecture/KEY_FILES.md",
+      "# Key files\n\n- `src/core/db.ts` — connection mgmt. **v0.41.2 (#9):** added pool reconnect.\n",
+    );
+    const r = run();
+    expect(r.status).toBe(1);
+    expect(r.stderr).toContain("bolded release-clause");
+    expect(r.stderr).toContain("KEY_FILES.md");
+  });
+
+  it("PASSES on a legitimate non-bolded version mention (no false positive)", () => {
+    seedClean();
+    writeDoc(
+      "docs/architecture/KEY_FILES.md",
+      "# Key files\n\n- `src/core/db.ts` — requires pgvector 0.7; on Postgres 11+ the ADD COLUMN is metadata-only.\n",
+    );
+    const r = run();
+    expect(r.status).toBe(0);
+  });
+
+  it("FAILS when CLAUDE.md exceeds the size cap", () => {
+    seedClean();
+    writeDoc("CLAUDE.md", "x".repeat(200_000));
+    const r = run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "90000" });
+    expect(r.status).toBe(1);
+    expect(r.stderr).toContain("over the");
+  });
+
+  it("size cap is configurable via env", () => {
+    seedClean();
+    writeDoc("CLAUDE.md", "x".repeat(5_000));
+    expect(run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "1000" }).status).toBe(1);
+    expect(run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "10000" }).status).toBe(0);
+  });
+
+  it("soft-warns (non-fatal) on prose history markers", () => {
+    seedClean();
+    writeDoc(
+      "docs/TESTING.md",
+      "# Testing\n\nThe tier set, then v0.26.7 added the parallel loop (pre-fix it was serial).\n",
+    );
+    const r = run();
+    expect(r.status).toBe(0); // warn, not fail
+    expect(r.stderr).toContain("WARN");
+  });
+
+  it("catches the marker in any of the three reference docs (thin-client)", () => {
+    seedClean();
+    writeDoc("docs/architecture/thin-client.md", "# Thin-client\n\n**v0.36.3:** added cross-modal.\n");
+    expect(run().status).toBe(1);
+  });
+});

test/scripts/ci-cache-hash.test.ts

@@ -400,6 +400,50 @@ describe("ci-cache-hash.sh — edge cases", () => {
   });
 });
 
+describe("ci-cache-hash.sh — policy-doc re-admit (test-affecting docs under docs/)", () => {
+  // These docs live under docs/ (normally deny-listed) but carry CI/release/
+  // test contracts the suite reads. The re-admit must make edits to them
+  // invalidate the hash, WITHOUT un-denying ordinary docs.
+  const POLICY_FILES: Record<string, string> = {
+    ...BASELINE_FILES,
+    "docs/TESTING.md": "# Testing\n\ntest tiers + isolation lint\n",
+    "docs/RELEASING.md": "# Releasing\n\nversion locations + ship process\n",
+  };
+
+  function withSandbox(test: (sb: Sandbox) => void) {
+    const sb = makeSandbox(POLICY_FILES);
+    try {
+      test(sb);
+    } finally {
+      rmSync(sb.dir, { recursive: true, force: true });
+    }
+  }
+
+  it("docs/TESTING.md edit MUST change hash (re-admitted policy doc)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/TESTING.md", "# Testing\n\nNEW POLICY\n");
+      expect(hash(sb)).not.toBe(before);
+    });
+  });
+
+  it("docs/RELEASING.md edit MUST change hash (re-admitted policy doc)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/RELEASING.md", "# Releasing\n\nNEW SHIP RULE\n");
+      expect(hash(sb)).not.toBe(before);
+    });
+  });
+
+  it("docs/guide.md edit STILL produces same hash (re-admit is scoped, not a blanket docs un-deny)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/guide.md", "# Guide v3\n");
+      expect(hash(sb)).toBe(before);
+    });
+  });
+});
+
 describe("ci-cache-hash.sh — usage errors", () => {
   it("--bogus arg exits 2", () => {
     const r = spawnSync("bash", [SCRIPT_SRC, "--bogus"], {

test/skillopt/held-out.test.ts

@@ -15,7 +15,9 @@ import {
   capturesDir,
   loadHeldOut,
   runHeldOutGate,
+  MIN_HELD_OUT_SIZE,
 } from '../../src/core/skillopt/held-out.ts';
+import { assertBundledMutationHeldOut } from '../../src/core/skillopt/bundled-skill-gate.ts';
 
 let tmp: string;
 
@@ -109,3 +111,35 @@ describe('F11 runHeldOutGate vacuous case', () => {
     expect(result.candidateScore).toBe(0);
   });
 });
+
+describe('D16 assertBundledMutationHeldOut (core ENFORCE)', () => {
+  test('bundled + mutate + too-small held-out → throws (hard refuse)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: MIN_HELD_OUT_SIZE - 1, skillName: 'brain-ops',
+    })).toThrow(/held-out/i);
+  });
+
+  test('bundled + mutate + empty held-out → throws (vacuous-pass hole closed)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: 0, skillName: 'brain-ops',
+    })).toThrow(/non-empty held-out/i);
+  });
+
+  test('bundled + mutate + sufficient held-out → no throw', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: MIN_HELD_OUT_SIZE, skillName: 'brain-ops',
+    })).not.toThrow();
+  });
+
+  test('not bundled → no throw even with zero held-out (user skills are free)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: false, willMutate: true, heldOutCount: 0, skillName: 'my-skill',
+    })).not.toThrow();
+  });
+
+  test('bundled but NOT mutating (no-mutate / proposed.md path) → no throw', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: false, heldOutCount: 0, skillName: 'brain-ops',
+    })).not.toThrow();
+  });
+});

test/skillopt/reflect.test.ts

@@ -18,7 +18,7 @@
  */
 
 import { describe, expect, test } from 'bun:test';
-import { parseEditsResponse, runReflect } from '../../src/core/skillopt/reflect.ts';
+import { parseEditsResponse, runReflect, runOneShotRewrite, describeJudge, describeJudges } from '../../src/core/skillopt/reflect.ts';
 import type { ChatOpts, ChatResult } from '../../src/core/ai/gateway.ts';
 import type { ScoredRollout, Trajectory } from '../../src/core/skillopt/types.ts';
 
@@ -287,3 +287,113 @@ describe('runReflect (D7 two-call contract)', () => {
     expect(observedUserMsg).toContain('validation_gate_below_baseline');
   });
 });
+
+// ─── runOneShotRewrite (cat31 config C baseline) ────────────────────────────
+
+function oneShotChatStub(text: string): NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']> {
+  return (async (_opts: ChatOpts): Promise<ChatResult> => ({
+    text,
+    blocks: [{ type: 'text', text }],
+    stopReason: 'end',
+    usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
+    model: 'm',
+    providerId: 'anthropic',
+  })) as NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']>;
+}
+
+const ONE_SHOT_BASE = { skillBodyText: '# Skill', successes: [] as ScoredRollout[], failures: [] as ScoredRollout[], rejected: [], optimizerModel: 'm' };
+
+describe('runOneShotRewrite', () => {
+  test('unwraps a whole-response code fence', async () => {
+    const body = '# Skill\n\n## People\nList people.';
+    const r = await runOneShotRewrite({ ...ONE_SHOT_BASE, chatFn: oneShotChatStub('```markdown\n' + body + '\n```') });
+    expect(r.newBody).toBe(body);
+    expect(r.error).toBeUndefined();
+  });
+
+  test('preserves an EMBEDDED code fence — does not truncate to the first block', async () => {
+    const body = '# Skill\n\nRun this:\n```bash\nls -la\n```\n\n## People\nList people.';
+    const r = await runOneShotRewrite({ ...ONE_SHOT_BASE, chatFn: oneShotChatStub(body) });
+    expect(r.newBody).toBe(body);
+    expect(r.newBody).toContain('## People'); // regression: old non-anchored regex truncated to the bash block
+  });
+
+  test('chat error returns empty newBody + error (caller treats as no-change)', async () => {
+    const r = await runOneShotRewrite({
+      ...ONE_SHOT_BASE,
+      chatFn: (async () => { throw new Error('boom'); }) as NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']>,
+    });
+    expect(r.newBody).toBe('');
+    expect(r.error).toContain('one_shot_rewrite_failed');
+  });
+});
+
+// ─── Success criteria threading (v0.42.9.0) ──────────────────────────────────
+// The optimizer must be TOLD how its output is scored, or it optimizes blind:
+// on a rule-judged benchmark it proposes plausible-but-off edits, every
+// candidate scores 0, the gate rejects them, and the skill never changes. These
+// pin that the judge criteria render to plain English AND reach the reflect prompt.
+describe('describeJudge / criteria threading', () => {
+  test('describeJudge renders each rule check as a plain requirement', () => {
+    const d = describeJudge({ kind: 'rule', checks: [
+      { op: 'section_present', arg: 'Key Risks' },
+      { op: 'regex', arg: '[Cc]onfidence\\s*[:=]' },
+      { op: 'max_chars', arg: 1200 },
+      { op: 'tool_called', arg: 'search' },
+    ] });
+    expect(d).toContain('Key Risks');
+    expect(d).toContain('[Cc]onfidence');
+    expect(d).toContain('1200');
+    expect(d).toContain('search');
+  });
+
+  test('describeJudge renders an llm rubric', () => {
+    expect(describeJudge({ kind: 'llm', rubric: 'reward genuine substance' }))
+      .toContain('reward genuine substance');
+  });
+
+  test('describeJudges dedupes identical judge shapes across tasks', () => {
+    const j = { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'X' }] };
+    const out = describeJudges([{ judge: j }, { judge: j }, { judge: j }]);
+    // One distinct shape → one block, not three.
+    expect(out.match(/contain the exact text/g)).toHaveLength(1);
+  });
+
+  test('runReflect injects the criteria block into the optimizer prompt', async () => {
+    let seenUser = '';
+    const chatFn = async (opts: ChatOpts): Promise<ChatResult> => {
+      const u = opts.messages[0]?.content;
+      seenUser = typeof u === 'string' ? u : '';
+      return makeChatResult(JSON.stringify({ edits: [] }));
+    };
+    await runReflect({
+      skillBodyText: '# Test',
+      successes: [],
+      failures: [makeScored('f-1', 0.0)],
+      rejected: [],
+      criteria: 'CRITERIA: must include a Confidence: line',
+      optimizerModel: 'anthropic:claude-opus-4-7',
+      chatFn,
+    });
+    expect(seenUser).toContain('SUCCESS CRITERIA');
+    expect(seenUser).toContain('must include a Confidence: line');
+  });
+
+  test('runReflect omits the criteria block when none is given', async () => {
+    let seenUser = '';
+    const chatFn = async (opts: ChatOpts): Promise<ChatResult> => {
+      const u = opts.messages[0]?.content;
+      seenUser = typeof u === 'string' ? u : '';
+      return makeChatResult(JSON.stringify({ edits: [] }));
+    };
+    await runReflect({
+      skillBodyText: '# Test',
+      successes: [],
+      failures: [makeScored('f-1', 0.0)],
+      rejected: [],
+      optimizerModel: 'anthropic:claude-opus-4-7',
+      chatFn,
+    });
+    expect(seenUser).not.toContain('SUCCESS CRITERIA');
+  });
+});

test/skillopt/rollout.test.ts

@@ -0,0 +1,123 @@
+/**
+ * SkillOpt rollout tests (the `toolLoopFn` DI seam — previously zero coverage).
+ *
+ * Covers: trajectory shape capture, READ_ONLY_BRAIN_TOOLS zero-write invariant,
+ * default read-only tool registry (no put_page/submit_job), --write-capture
+ * routing to the virtual write registry, and tool-call ordering via the
+ * onToolCallStart callback. Hermetic: the toolLoop transport is stubbed, so no
+ * LLM calls, no DB, no API keys. Handlers are never executed (the stub returns
+ * messages directly), so a `{}`-shaped engine is sufficient.
+ */
+
+import { describe, expect, test } from 'bun:test';
+import { runRollout, READ_ONLY_BRAIN_TOOLS } from '../../src/core/skillopt/rollout.ts';
+import type { BenchmarkTask } from '../../src/core/skillopt/types.ts';
+
+const TASK: BenchmarkTask = {
+  task_id: 't1',
+  task: 'do the thing',
+  judge: { kind: 'rule', checks: [] },
+};
+
+/** Build a stubbed toolLoop that records the tool defs it was handed. */
+function makeStubLoop(opts: { capturedTools?: Array<{ name: string }>; finalText?: string }) {
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  return async (loopOpts: any) => {
+    if (opts.capturedTools) opts.capturedTools.push(...loopOpts.tools);
+    return {
+      messages: [
+        { role: 'user', content: 'do the thing' },
+        { role: 'assistant', content: opts.finalText ?? 'done' },
+      ],
+      totalUsage: { input_tokens: 10, output_tokens: 5, cache_read_tokens: 0, cache_creation_tokens: 0 },
+      totalTurns: 2,
+      stopReason: 'end',
+    };
+  };
+}
+
+describe('rollout — READ_ONLY_BRAIN_TOOLS zero-write invariant', () => {
+  test('excludes put_page / submit_job / file_upload, includes a read op', () => {
+    expect(READ_ONLY_BRAIN_TOOLS.has('put_page')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('submit_job')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('file_upload')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('search')).toBe(true);
+  });
+});
+
+describe('rollout — trajectory capture', () => {
+  test('returns a Trajectory with the expected shape + threaded usage/stop_reason', async () => {
+    const traj = await runRollout({
+      engine: {} as never,
+      skillText: 'skill body',
+      task: TASK,
+      targetModel: 'anthropic:claude-sonnet-4-6',
+      toolLoopFn: makeStubLoop({ finalText: 'the answer' }) as never,
+    });
+    expect(traj.task_id).toBe('t1');
+    expect(traj.task).toBe('do the thing');
+    expect(traj.final_text).toBe('the answer');
+    expect(traj.stop_reason).toBe('end');
+    expect(traj.turns).toBe(2);
+    expect(traj.usage.input_tokens).toBe(10);
+    expect(traj.usage.output_tokens).toBe(5);
+    expect(typeof traj.duration_ms).toBe('number');
+    expect(Array.isArray(traj.tool_calls)).toBe(true);
+  });
+});
+
+describe('rollout — tool registry routing', () => {
+  test('default rollout passes ONLY read-only tools (no write defs)', async () => {
+    const capturedTools: Array<{ name: string }> = [];
+    await runRollout({
+      engine: {} as never,
+      skillText: 's',
+      task: TASK,
+      targetModel: 'm',
+      toolLoopFn: makeStubLoop({ capturedTools }) as never,
+    });
+    const names = capturedTools.map((d) => d.name);
+    expect(names).toContain('brain_search');
+    expect(names).not.toContain('brain_put_page');
+    expect(names).not.toContain('brain_submit_job');
+    expect(names).not.toContain('brain_file_upload');
+  });
+
+  test('--write-capture routes to the virtual write registry (put_page present, captured not real)', async () => {
+    const capturedTools: Array<{ name: string }> = [];
+    await runRollout({
+      engine: {} as never,
+      skillText: 's',
+      task: TASK,
+      targetModel: 'm',
+      writeCapture: true,
+      toolLoopFn: makeStubLoop({ capturedTools }) as never,
+    });
+    const names = capturedTools.map((d) => d.name);
+    expect(names).toContain('brain_put_page');
+    expect(names).toContain('brain_submit_job');
+    expect(names).toContain('brain_file_upload');
+  });
+});
+
+describe('rollout — tool-call ordering', () => {
+  test('onToolCallStart records calls in order with the brain_ prefix stripped', async () => {
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const loop = async (loopOpts: any) => {
+      await loopOpts.onToolCallStart?.(0, 0, 0, 'brain_search', { q: 'x' }, 'pc1');
+      await loopOpts.onToolCallStart?.(0, 1, 1, 'brain_get_page', { slug: 'y' }, 'pc2');
+      return {
+        messages: [{ role: 'assistant', content: 'ok' }],
+        totalUsage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
+        totalTurns: 1,
+        stopReason: 'end',
+      };
+    };
+    const traj = await runRollout({
+      engine: {} as never, skillText: 's', task: TASK, targetModel: 'm', toolLoopFn: loop as never,
+    });
+    expect(traj.tool_calls).toHaveLength(2);
+    expect(traj.tool_calls[0]!.name).toBe('search');
+    expect(traj.tool_calls[1]!.name).toBe('get_page');
+  });
+});

test/skillopt/validate-gate-abort.test.ts

@@ -0,0 +1,78 @@
+/**
+ * Regression: `runValidationGate` must SURFACE must-abort errors (budget
+ * exhaustion / no-pricing), not swallow them as score=0.
+ *
+ * The bug the SkillOpt real-LLM eval surfaced: a Haiku run with `--max-cost`
+ * hit `BudgetTracker` no_pricing on the FIRST chat() of every rollout, the
+ * error threw before any network call, and `runWithLimit` caught it as a
+ * `{ok:false}` settled item which the gate turned into `median:0`. Result: the
+ * whole gate reported a vacuous `selScore:0` in milliseconds with zero LLM
+ * calls — a pricing crash masquerading as a real "0/N" measurement. The fix
+ * re-throws any MUST_ABORT-class error so the caller aborts loudly.
+ */
+import { describe, test, expect } from 'bun:test';
+import { runValidationGate, scoreSkillOnTasks } from '../../src/core/skillopt/validate-gate.ts';
+import type { BenchmarkTask } from '../../src/core/skillopt/types.ts';
+
+const TASKS: BenchmarkTask[] = [
+  { task_id: 't1', task: 'do a thing', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'x' }] } } as never,
+  { task_id: 't2', task: 'do another', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'y' }] } } as never,
+];
+
+function budgetExhausted(): Error {
+  const e = new Error('no pricing entry for model "anthropic:claude-haiku-4-5" (kind=chat)');
+  (e as { tag?: string }).tag = 'BUDGET_EXHAUSTED';
+  return e;
+}
+
+describe('runValidationGate — must-abort errors surface', () => {
+  test('a BUDGET_EXHAUSTED rollout error is re-thrown, not scored 0', async () => {
+    const throwingRollout = (async () => {
+      throw budgetExhausted();
+    }) as never;
+    await expect(
+      runValidationGate({
+        engine: {} as never,
+        candidateSkillText: 'skill',
+        selSet: TASKS,
+        bestScore: -1,
+        targetModel: 'anthropic:claude-haiku-4-5',
+        runsPerTask: 1,
+        rolloutFn: throwingRollout,
+      }),
+    ).rejects.toThrow(/no pricing entry/);
+  });
+
+  test('scoreSkillOnTasks propagates the abort too (does not return a vacuous 0)', async () => {
+    const throwingRollout = (async () => {
+      throw budgetExhausted();
+    }) as never;
+    await expect(
+      scoreSkillOnTasks({
+        engine: {} as never,
+        skillText: 'skill',
+        tasks: TASKS,
+        targetModel: 'anthropic:claude-haiku-4-5',
+        runsPerTask: 1,
+        rolloutFn: throwingRollout,
+      }),
+    ).rejects.toThrow(/no pricing entry/);
+  });
+
+  test('an ordinary (non-abort) rollout error still scores 0 — fail-open preserved', async () => {
+    const flakyRollout = (async () => {
+      throw new Error('transient judge hiccup'); // no .tag → not must-abort
+    }) as never;
+    const gate = await runValidationGate({
+      engine: {} as never,
+      candidateSkillText: 'skill',
+      selSet: TASKS,
+      bestScore: -1,
+      targetModel: 'anthropic:claude-haiku-4-5',
+      runsPerTask: 1,
+      rolloutFn: flakyRollout,
+    });
+    expect(gate.selScore).toBe(0);
+    expect(gate.scoredRollouts).toHaveLength(0);
+  });
+});