From 127842e9ef8875c2c20cd0792be2f9622131948f Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 27 May 2026 07:16:30 -0700 Subject: [PATCH] v0.41.22.1 feat: brainstorm/lsd judge fixes (closes #1540 end-to-end) (#1562) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(core): add splitProviderModelId centralizer for pricing-side parsing New pure helper in src/core/model-id.ts that splits provider:model, provider/model, and bare model strings into a {provider, model} pair. Defensive contract: null/undefined/empty/whitespace returns {provider: null, model: ''}. Will be wired into the 5 pricing/budget sites in the next commit. Named splitProviderModelId (not parseModelId) to avoid the in-project collision with the gateway-side src/core/ai/model-resolver.ts:parseModelId which has a different bare-name contract. Pinned by 16 cases in test/model-id.test.ts covering all separator forms plus defensive + edge inputs. Co-Authored-By: Claude Opus 4.7 * feat(gateway): accept slash-form provider id in model-resolver src/core/ai/model-resolver.ts:parseModelId now accepts both provider:model (colon) and provider/model (slash) forms. Colon wins when both separators present so OpenRouter nested ids like openrouter:anthropic/claude-sonnet-4.6 route as {providerId: 'openrouter', modelId: 'anthropic/claude-sonnet-4.6'}. Pre-fix: every gateway entry point (chat / embed / rerank) threw AIConfigError 'missing a provider prefix' on slash form ids. That meant CLI users running gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6 would still fail mid-judge with AIConfigError even after pricing was relaxed to accept slash form. Closes the end-to-end bug class. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Existing tests pinning that throw (test/ai/capabilities.test.ts:43) stay green. Pinned by 10 cases in test/ai/model-resolver-slash.test.ts including a resolveRecipe round-trip that slash and colon forms land on the same recipe. Co-Authored-By: Claude Opus 4.7 * refactor: route 5 pricing/config sites through splitProviderModelId Five sites had inline ':'-only provider-prefix splits that silently missed slash-form ids. Centralizing through splitProviderModelId closes the bug class: - src/core/anthropic-pricing.ts:estimateMaxCostUsd - src/core/budget/budget-tracker.ts:lookupPricing (closes the headline BudgetExhausted no_pricing failure on --max-cost + slash-form --judge-model) - src/core/eval-contradictions/cost-tracker.ts:pricingFor (legacy silent-Haiku fallback preserved per plan D9) - src/core/minions/batch-projection.ts (deleted bareModel inline helper; inlined splitProviderModelId at 2 call sites) - src/core/model-config.ts:isAnthropicProvider (silently fixed v0.31.12 subagent-guard bypass for slash-form Anthropic ids) Test gates land together so any bisect step is green: - NEW test/anthropic-pricing.test.ts (7 cases including structural regression guard: every ANTHROPIC_PRICING key reachable via all three forms) - NEW test/eval-contradictions/cost-tracker-slash.test.ts (6 cases including legacy-Haiku-fallback pin) - EXTENDED test/batch-projection.test.ts (slash + double-separator cases) - EXTENDED test/model-config.serial.test.ts (2 slash-form isAnthropicProvider cases) - EXTENDED test/core/budget/budget-tracker.test.ts (2 slash + colon reserve() cases) Behavior changes for slash-prefix ids only; bare and colon ids unchanged. Co-Authored-By: Claude Opus 4.7 * feat(brainstorm): scale judge maxTokens with per-model output cap Replace the hard-coded maxTokens: 4000 with computeJudgeMaxTokens that scales with idea count and respects each model's actual output cap. Pre-fix: any judge call with 36+ ideas produced ~100 tokens/idea of JSON that got truncated mid-output. parseJudgeJSON threw, orchestrator surfaced judge_failed: true, all ideas saved unscored. Verified failure mode on 72-idea fixture: 0/72 passing before, 39/72 after. Formula: min(modelCap, max(LEGACY_MIN_MAX_TOKENS, ideaCount*150+500)) Named constants extracted at top of judges.ts: - TOKEN_BUDGET_PER_IDEA = 150 (1.5x headroom over observed ~100/idea) - TOKEN_BUDGET_ENVELOPE = 500 (JSON wrapper) - LEGACY_MIN_MAX_TOKENS = 4000 (pre-fix floor preserved for 1-idea) - MAX_OUTPUT_TOKENS_CEIL = 32_000 (fallback when model unknown) - ANTHROPIC_OUTPUT_CAPS (per-model: Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy 3.5 = 8K) When the caller passes no modelOverride, the cap routes through the gateway's actual configured chat model via getChatModel() so the formula matches what chat() will use, not whatever the override hints at. Pre-fix the undefined-override case fell back to 32K even if the configured default was a legacy 8K model. Pinned by 16 cases in test/brainstorm/judges-maxtokens.test.ts: formula at 1/10/36/96/200/300 ideas, per-model cap binding (Haiku 3.5 8K, Opus 4.7 32K, Sonnet 4.6 64K), and integration via runJudge with a stubbed chatFn that captures ChatOpts.maxTokens. Co-Authored-By: Claude Opus 4.7 * chore: bump version and changelog (v0.41.21.0) Brainstorm judge fix-wave: closes #1540 end-to-end. parseModelId centralizer + gateway resolver slash-form acceptance + per-model maxTokens cap. Co-Authored-By: Claude Opus 4.7 * docs: update project documentation for v0.41.21.0 CLAUDE.md: add v0.41.21.0 annotations to brainstorm/judges + model-config entries; add new key-files entry for src/core/model-id.ts (the shared splitProviderModelId centralizer) and src/core/ai/model-resolver.ts slash-form extension. README.md: add user-facing callout for the brainstorm judge_failed + slash-form pricing fix, mirroring the v0.41.19.0 callout shape. llms-full.txt: regenerated to absorb the CLAUDE.md + README changes (passes test/build-llms.test.ts drift guard). Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 --- CHANGELOG.md | 131 ++++++++++++ CLAUDE.md | 6 +- README.md | 10 + TODOS.md | 16 +- VERSION | 2 +- llms-full.txt | 16 +- package.json | 2 +- src/core/ai/model-resolver.ts | 39 +++- src/core/anthropic-pricing.ts | 20 +- src/core/brainstorm/judges.ts | 89 +++++++- src/core/budget/budget-tracker.ts | 9 +- src/core/eval-contradictions/cost-tracker.ts | 23 +- src/core/minions/batch-projection.ts | 15 +- src/core/model-config.ts | 21 +- src/core/model-id.ts | 64 ++++++ test/ai/model-resolver-slash.test.ts | 96 +++++++++ test/anthropic-pricing.test.ts | 67 ++++++ test/batch-projection.test.ts | 32 +++ test/brainstorm/judges-maxtokens.test.ts | 196 ++++++++++++++++++ test/core/budget/budget-tracker.test.ts | 32 +++ .../cost-tracker-slash.test.ts | 61 ++++++ test/model-config.serial.test.ts | 13 ++ test/model-id.test.ts | 140 +++++++++++++ 23 files changed, 1055 insertions(+), 45 deletions(-) create mode 100644 src/core/model-id.ts create mode 100644 test/ai/model-resolver-slash.test.ts create mode 100644 test/anthropic-pricing.test.ts create mode 100644 test/brainstorm/judges-maxtokens.test.ts create mode 100644 test/eval-contradictions/cost-tracker-slash.test.ts create mode 100644 test/model-id.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a82361101..56a062cc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,135 @@ All notable changes to GBrain will be documented in this file. +## [0.41.22.1] - 2026-05-27 + +**Your `gbrain brainstorm` and `gbrain lsd` calls now actually score the ideas they generate.** + +Since the calibration cold-start landed, every brainstorm or LSD run was +quietly returning `judge_failed: true` and saving the ideas with no +scores. You'd ask for 72 ideas, get 72 unscored entries, and have no +way to tell which ones the judge would have called good. v0.41.22.1 +closes the two bugs that caused it. On the same 72-idea fixture that +scored 0/72 before, you'll now see ~39/72 passing — real judgment, not +silence. + +The same patch closes a second silent bug: pricing lookup missed +slash-form model ids. If you ran `gbrain brainstorm --judge-model +anthropic/claude-sonnet-4-6 --max-cost 5` (slash, the form CLI flags +accept and OpenRouter recipes emit), the BudgetTracker refused to +start because the pricing table only matched the colon-prefix form +`anthropic:claude-...`. Now both forms work the same. + +**How to turn it on:** Nothing to do. `gbrain upgrade` is all you need. +No schema migration, no config change. + +**What you'd see in a concrete example:** + +| Command | Pre-fix | Post-fix | +|---|---|---| +| `gbrain brainstorm "topic" --max-cost 5` | judge_failed, 0/72 ideas scored | summary table with ~39/72 passing | +| `gbrain brainstorm ... --judge-model anthropic/claude-sonnet-4-6 --max-cost 5` | `BudgetExhausted reason=no_pricing` | runs to completion, cost tracked | +| `gbrain lsd ... --judge-model anthropic:claude-sonnet-4-6 --max-cost 5` | works both pre + post (colon form) | works both pre + post | + +**The fix in one paragraph.** Two bugs lived in the same release. Bug +1: the judge hard-coded `maxTokens: 4000` while emitting ~100 tokens +per idea, so any chunk past ~40 ideas got truncated mid-JSON and the +parser threw. Bug 2: every pricing lookup site (5 of them across the +codebase) re-implemented an inline `provider:model` split, and none of +them handled `provider/model` (slash). Three reviews refound the slash +bug in three separate places. The fix: one shared `parseModelId` +helper that 5 sites now route through, plus a maxTokens formula that +scales with idea count and respects each model's actual output cap. + +**A subagent-guard bug fixed in the same wave.** The v0.31.12 subagent +runtime guard (`isAnthropicProvider`) only handled the colon form too. +If anyone had configured their subagent tier as +`anthropic/claude-sonnet-4-6` (slash), the guard would have silently +returned false, and the subagent loop would have fallen back to +TIER_DEFAULTS instead of honoring the explicit config. The same +centralizer closes this bypass. + +**What's safe to know about.** This is pure-function refactoring + a +new constant. No schema change, no DB plane impact, no behavioral +change for existing colon-form or bare model ids. Brainstorm runs at +larger judge maxTokens budgets will see somewhat longer Anthropic API +latency (the call now actually completes instead of truncating). The +trade is slower-but-correct vs faster-but-broken. + +**What we caught and fixed before merging.** Adversarial review caught +four real issues that landed in the shipped version: + +1. The original "32K maxTokens cap, applied uniformly" was unsafe for + legacy Claude 3.5 models whose output cap is 8,192. The shipped + version uses a per-model cap map (`ANTHROPIC_OUTPUT_CAPS`) so legacy + models bind at 8K and modern 4-series at 32K or 64K. +2. The `splitProviderModelId` defensive contract should be in the type + signature, not just in tests — the shipped signature is + `splitProviderModelId(input: string | null | undefined)`. +3. The pricing-side fix would let BudgetTracker pass for slash-form ids, + but `gateway.chat()` would then throw via the OLDER gateway-side + `parseModelId` (in `src/core/ai/model-resolver.ts`). The shipped + version also relaxes the gateway resolver to accept slash form, + closing the bug class end-to-end. Bare names without ANY separator + still throw — gateway routing always needs an explicit provider. +4. The maxTokens cap was looking at `modelOverride` (caller-passed) but + ignoring the gateway's actual configured chat model — so an + `undefined` override fell back to 32K even if the configured default + was a legacy 8K model. The shipped version routes through the + gateway's `getChatModel()` so the cap matches what `chat()` will + actually use. + +The in-project name collision between my new +`src/core/model-id.ts:parseModelId` and the existing gateway-side +`src/core/ai/model-resolver.ts:parseModelId` was killed by renaming +the new helper to `splitProviderModelId`. Both functions now accept +the same input shapes; they differ only in how they handle bare names +(`splitProviderModelId` returns `{provider: null, model: 'bare'}`; +the gateway one throws because routing needs an explicit provider). + +Thanks to `@garrytan-agents` whose original bug report (PR #1540, +since closed as superseded by this wave) drove the whole investigation +and provided the first-pass diff for the two most visible sites. + +### Itemized changes + +- **`src/core/model-id.ts` (NEW)** — `splitProviderModelId(input): {provider, model}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Pinned by 16 cases in `test/model-id.test.ts`. +- **`src/core/ai/model-resolver.ts`** — gateway-side `parseModelId` extended to also accept slash form (`anthropic/claude-sonnet-4-6`). Pre-fix the colon-only check threw at every gateway entry point (chat / embed / rerank) so even with the pricing fix, slash-form judge models would still fail mid-judge. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. New test file `test/ai/model-resolver-slash.test.ts` (10 cases including a resolveRecipe round-trip pinning slash form resolves to the same recipe as colon form). +- **`src/core/anthropic-pricing.ts`** — `estimateMaxCostUsd` routes through `splitProviderModelId`. Now handles slash-form ids that previously returned null. New test file `test/anthropic-pricing.test.ts` (7 cases including a structural regression guard that every key in `ANTHROPIC_PRICING` is reachable via bare + colon + slash). +- **`src/core/budget/budget-tracker.ts`** — `lookupPricing` routes through `splitProviderModelId`. Closes the `BudgetExhausted reason=no_pricing` hard-fail on `--max-cost N` + `--judge-model anthropic/claude-...` (the headline brainstorm bug). 2 new cases in the existing budget-tracker test. +- **`src/core/eval-contradictions/cost-tracker.ts`** — `pricingFor` routes through `splitProviderModelId`. The duplicate ANTHROPIC_PRICING table (consolidation deferred to follow-up TODO) now correctly bills colon and slash forms of Sonnet/Opus instead of silently falling back to Haiku pricing. New test file `test/eval-contradictions/cost-tracker-slash.test.ts` (6 cases including a legacy-behavior pin for the unknown-model silent-Haiku fallback). +- **`src/core/minions/batch-projection.ts`** — deleted the 3-line inline `bareModel` helper; inlined `splitProviderModelId(model).model` at both call sites. Existing `test/batch-projection.test.ts` extended with slash-form + double-separator cases. +- **`src/core/model-config.ts:isAnthropicProvider`** — routes through `splitProviderModelId`. **Silently fixed a v0.31.12 subagent-guard bypass:** slash-form Anthropic ids (`anthropic/claude-sonnet-4-6`) now correctly classify as Anthropic, so the subagent loop honors them instead of falling back to TIER_DEFAULTS. 2 new cases in `test/model-config.serial.test.ts`. +- **`src/core/brainstorm/judges.ts`** — `maxTokens: 4000` replaced with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL` extracted at top of file with per-constant comment. New `ANTHROPIC_OUTPUT_CAPS` map per-model output ceilings (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy 3.5 = 8K). When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use, not whatever the override hints at. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. + +### For contributors + +Three follow-up TODOs filed in `TODOS.md` from the v0.41.22.1 plan review: + +- Config-write normalization (canonicalize provider IDs to `:` form on config write) +- Non-Anthropic pricing tables (OpenAI / Gemini / OpenRouter) +- Eval-contradictions duplicate ANTHROPIC_PRICING table consolidation + +The first two are v0.42+ scope. The third is deferred from this wave per the explicit Step 0 scope decision (cleanup-the-pricing-system would double the blast radius of a brainstorm fix). + +## To take advantage of v0.41.22.1 + +`gbrain upgrade` is all you need. No schema migration, no config change. + +**Verify the fix worked:** + +```bash +# Pre-fix this would silently exit with judge_failed in the report: +gbrain brainstorm "what should I work on next" --max-cost 1 +# Look for: "passing N/M ideas" in the summary — should be > 0 + +# Pre-fix this would refuse to start with BudgetExhausted no_pricing: +gbrain brainstorm "topic" --judge-model anthropic/claude-sonnet-4-6 --max-cost 1 +# Should run to completion and print a scored idea list +``` + +If `gbrain brainstorm` still hits `judge_failed` after upgrading, file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain --version` and the brainstorm command you ran. The fix is structural; failure post-upgrade indicates the fix didn't land properly. + ## [0.41.22.0] - 2026-05-27 **Your brain runs on a real taxonomy now. Not 94 types of cruft. Fifteen @@ -497,6 +626,8 @@ it exists. `{"schema_version"` envelope prefix instead of walking back from `"checks"` (which broke once `category_scores` introduced a nested object between). + + ## [0.41.19.0] - 2026-05-26 **Your dream cycle stops silently losing wiki links.** diff --git a/CLAUDE.md b/CLAUDE.md index 8ec2a3b03..ccc4c4f18 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -129,7 +129,7 @@ strict behavior when unset. - `src/core/diarize/payload-fitter.ts` (v0.37.x, P6 / Q3) — generic fit-arbitrarily-large-items-into-per-call-token-budget utility. `'batch'` strategy is deterministic token-budgeted chunking with no LLM calls. `'summarize'` strategy embed-clusters into ceil(items/4) groups via cheap deterministic nearest-neighbor on cosine, Haiku-summarizes each cluster via `Promise.allSettled` at parallelism=4 (Perf1). Each Haiku call composes the active BudgetTracker via T3's AsyncLocalStorage. The quality gate (codex outside-voice finding #4): when `success_ratio < min_success_ratio` (default 0.75), result is flagged `degraded: true` — the fitter preserves the successful subset; the caller decides whether to surface a partial result or abort. - `src/core/brainstorm/checkpoint.ts` (v0.37.x, P7 / TX3+TX4+A5 amended) — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB per run) so resume can MERGE the pre-crash ideas with the post-resume ideas before the judge runs (codex's load-bearing finding — a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume ` — the proposed `--retry-failed` was dropped per TX4: failed AND never-attempted crosses both go through `--resume`). `--list-runs` prints saved run_ids mtime-newest-first. `--force-resume` bypasses the 7-day staleness gate. The cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by 20 unit cases + 3 E2E cases in `test/e2e/brainstorm-resume.test.ts` including the load-bearing merge contract. - `src/core/remediation-checkpoint.ts` (v0.37.x, T7 / A4 amended) — `doctor --remediate` checkpoint at `~/.gbrain/remediation/.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned. Atomic write via `.tmp + rename`. `gbrain doctor --remediate --resume ` (or with no arg — picks the newest matching checkpoint) loads it and skips already-completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases. -- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. +- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. **v0.41.21.0:** `isAnthropicProvider` routes through the new `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly. Pre-fix the colon-only check silently returned false on slash form, so a user who set `models.tier.subagent` to the slash form had `enforceSubagentAnthropic` fall through to `TIER_DEFAULTS.subagent` AND skipped the warn — the explicit config was honored as if it had never been set. Now both shapes classify the same. Pinned by 2 new cases in `test/model-config.serial.test.ts`. - `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. **v0.31.12:** `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` gains an optional 4th `extendedModels: ReadonlySet` argument. When the modelId is in that set, the native-recipe allowlist throw is bypassed — the user explicitly opted into this model via config so we let provider rejection surface as `model_not_found` at HTTP call time (and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — typos in source code still fail fast. Replaces the earlier plan to soften the validator wholesale (Codex F4/F5 in plan review flagged that as too broad — it would have removed the fail-fast contract for chat + expand + embed all three). - `src/core/ai/gateway.ts` extension (v0.31.12) — new module-scoped `_extendedModels: Map>` registry feeds `assertTouchpoint`'s 4th-arg path. New `reconfigureGatewayWithEngine(engine)` async function is called from `cli.ts` after `engine.connect()` (and before every command except `CLI_ONLY` no-DB commands) — re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to expansion + chat both. `DEFAULT_CHAT_MODEL` corrected to `anthropic:claude-sonnet-4-6` (was the v0.31.6 phantom `-20250929`). New `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. - `src/core/minions/queue.ts` extension (v0.31.12) — `MinionQueue.add()` now rejects `subagent` jobs whose `data.model` resolves through `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (Codex F1+F2 in plan review). Layers 2 + 3 live in `src/core/model-config.ts` (`enforceSubagentAnthropic` runtime fallback) and `src/commands/doctor.ts` (`subagent_provider` check). Pinned by 3 cases in `test/agent-cli.test.ts`. @@ -247,7 +247,9 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. +- `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. +- `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. diff --git a/README.md b/README.md index 9db2f1a3f..dce17a1cf 100644 --- a/README.md +++ b/README.md @@ -314,6 +314,16 @@ Bad values surface at `gbrain doctor` startup with a paste-ready fix retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice. +**`gbrain brainstorm` returning `judge_failed: true` with 0 scored +ideas?** v0.41.21.0 closes the two bugs that caused it. The judge +hard-coded a 4K-token output cap; for any run past ~40 ideas the call +truncated mid-JSON and the parser threw. Same release closes a slash- +form pricing miss: `gbrain brainstorm --judge-model +anthropic/claude-sonnet-4-6 --max-cost 5` failed with +`BudgetExhausted reason=no_pricing` because every pricing site only +matched the colon form. Both shapes work now. No config change, no +schema migration — `gbrain upgrade` is the whole fix. + ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end diff --git a/TODOS.md b/TODOS.md index 00342d0a8..554b20e13 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,19 @@ # TODOS +## v0.41.22.1 brainstorm judge fix-wave follow-ups (v0.42+) + +Filed from the v0.41.22.1 plan-eng-review per cross-model-tension D13c. +Step 0 of that plan explicitly deferred a "full pricing-system DRY" +cleanup (Option C) to keep the brainstorm fix blast radius small. +These three items are what was deferred. None are user-reported bugs; +all are latent-debt cleanup. + +- [ ] **Config-write normalization.** Whenever a user writes `gbrain config set models.tier.deep anthropic/claude-opus-4-7` we silently store the slash form. v0.41.22.1 centralized the read-side via `splitProviderModelId`, but config writes still preserve whatever shape the user typed. Canonical form should be colon (`anthropic:claude-opus-4-7`). Fix: rewrite at config-write time in `src/core/config.ts`. Breaks existing config files that explicitly hold the slash form — defer to a v0.42+ config-migration wave that also handles the rewrite + once-per-process deprecation warn. Files: `src/core/config.ts`, `src/core/model-config.ts:saveConfig` path. Priority: P3 (latent, not user-visible). + +- [ ] **Non-Anthropic pricing tables.** `src/core/anthropic-pricing.ts` is the only pricing surface gbrain ships. Brainstorm + LSD users routing through OpenAI / Gemini / OpenRouter get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). The right shape: rename to `provider-pricing.ts`, add OpenAI / Gemini / OpenRouter tables, route `lookupPricing` through provider-routed table selection. OpenRouter is a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6` either way). Files: `src/core/anthropic-pricing.ts` (rename + extend), `src/core/budget/budget-tracker.ts`, `src/core/eval-contradictions/cost-tracker.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic). + +- [ ] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** `src/core/eval-contradictions/cost-tracker.ts:28-38` ships its OWN copy of the Anthropic pricing table with different keys (both bare and `anthropic:`-prefixed forms) and a silent-Haiku fallback on unknown. v0.41.22.1 routed both tables' lookups through `splitProviderModelId` but left the duplication. Right fix: delete the local table, import from `src/core/anthropic-pricing.ts`. Either (a) preserve the silent-Haiku-fallback semantic with an explicit `?? canonicalPricing['claude-haiku-4-5']` at the call site, or (b) tighten to warn-once on unknown (which changes the eval-contradictions soft-ceiling `--budget-usd` contract — coordinate with that subsystem). Files: `src/core/eval-contradictions/cost-tracker.ts`, `src/core/anthropic-pricing.ts`, `test/eval-contradictions/cost-tracker-slash.test.ts` (the legacy-Haiku-fallback pin would need updating). Priority: P3 (DRY cleanup, no user-visible impact). + ## v0.41.21.0 ops-fix-wave follow-ups (v0.41.22+) - **TODO-OPS-1 (P2)**: `gbrain sync print-cron` subcommand. Print the canonical @@ -112,7 +126,7 @@ takes_count). Today the MCP run_onboard op runs these server-side via runAllOnboardChecks; doctor-remote.ts would surface them on the thin-client dashboard for operators who only hit the brain via MCP. -======= + ## v0.41.17.0 `--workers N` cathedral follow-ups (v0.41.18+) These were filed during the ship of `garrytan/dar-es-salaam-v1` diff --git a/VERSION b/VERSION index 136fdf2e3..9ac51760f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.22.0 \ No newline at end of file +0.41.22.1 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 5bf205481..0c4946451 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -271,7 +271,7 @@ strict behavior when unset. - `src/core/diarize/payload-fitter.ts` (v0.37.x, P6 / Q3) — generic fit-arbitrarily-large-items-into-per-call-token-budget utility. `'batch'` strategy is deterministic token-budgeted chunking with no LLM calls. `'summarize'` strategy embed-clusters into ceil(items/4) groups via cheap deterministic nearest-neighbor on cosine, Haiku-summarizes each cluster via `Promise.allSettled` at parallelism=4 (Perf1). Each Haiku call composes the active BudgetTracker via T3's AsyncLocalStorage. The quality gate (codex outside-voice finding #4): when `success_ratio < min_success_ratio` (default 0.75), result is flagged `degraded: true` — the fitter preserves the successful subset; the caller decides whether to surface a partial result or abort. - `src/core/brainstorm/checkpoint.ts` (v0.37.x, P7 / TX3+TX4+A5 amended) — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB per run) so resume can MERGE the pre-crash ideas with the post-resume ideas before the judge runs (codex's load-bearing finding — a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume ` — the proposed `--retry-failed` was dropped per TX4: failed AND never-attempted crosses both go through `--resume`). `--list-runs` prints saved run_ids mtime-newest-first. `--force-resume` bypasses the 7-day staleness gate. The cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by 20 unit cases + 3 E2E cases in `test/e2e/brainstorm-resume.test.ts` including the load-bearing merge contract. - `src/core/remediation-checkpoint.ts` (v0.37.x, T7 / A4 amended) — `doctor --remediate` checkpoint at `~/.gbrain/remediation/.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned. Atomic write via `.tmp + rename`. `gbrain doctor --remediate --resume ` (or with no arg — picks the newest matching checkpoint) loads it and skips already-completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases. -- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. +- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. **v0.41.21.0:** `isAnthropicProvider` routes through the new `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly. Pre-fix the colon-only check silently returned false on slash form, so a user who set `models.tier.subagent` to the slash form had `enforceSubagentAnthropic` fall through to `TIER_DEFAULTS.subagent` AND skipped the warn — the explicit config was honored as if it had never been set. Now both shapes classify the same. Pinned by 2 new cases in `test/model-config.serial.test.ts`. - `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. **v0.31.12:** `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` gains an optional 4th `extendedModels: ReadonlySet` argument. When the modelId is in that set, the native-recipe allowlist throw is bypassed — the user explicitly opted into this model via config so we let provider rejection surface as `model_not_found` at HTTP call time (and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — typos in source code still fail fast. Replaces the earlier plan to soften the validator wholesale (Codex F4/F5 in plan review flagged that as too broad — it would have removed the fail-fast contract for chat + expand + embed all three). - `src/core/ai/gateway.ts` extension (v0.31.12) — new module-scoped `_extendedModels: Map>` registry feeds `assertTouchpoint`'s 4th-arg path. New `reconfigureGatewayWithEngine(engine)` async function is called from `cli.ts` after `engine.connect()` (and before every command except `CLI_ONLY` no-DB commands) — re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to expansion + chat both. `DEFAULT_CHAT_MODEL` corrected to `anthropic:claude-sonnet-4-6` (was the v0.31.6 phantom `-20250929`). New `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. - `src/core/minions/queue.ts` extension (v0.31.12) — `MinionQueue.add()` now rejects `subagent` jobs whose `data.model` resolves through `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (Codex F1+F2 in plan review). Layers 2 + 3 live in `src/core/model-config.ts` (`enforceSubagentAnthropic` runtime fallback) and `src/commands/doctor.ts` (`subagent_provider` check). Pinned by 3 cases in `test/agent-cli.test.ts`. @@ -389,7 +389,9 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. +- `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. +- `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. @@ -2953,6 +2955,16 @@ Bad values surface at `gbrain doctor` startup with a paste-ready fix retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice. +**`gbrain brainstorm` returning `judge_failed: true` with 0 scored +ideas?** v0.41.21.0 closes the two bugs that caused it. The judge +hard-coded a 4K-token output cap; for any run past ~40 ideas the call +truncated mid-JSON and the parser threw. Same release closes a slash- +form pricing miss: `gbrain brainstorm --judge-model +anthropic/claude-sonnet-4-6 --max-cost 5` failed with +`BudgetExhausted reason=no_pricing` because every pricing site only +matched the colon form. Both shapes work now. No config change, no +schema migration — `gbrain upgrade` is the whole fix. + ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end diff --git a/package.json b/package.json index 3b4eeb900..f6c0f5adb 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.22.0" + "version": "0.41.22.1" } diff --git a/src/core/ai/model-resolver.ts b/src/core/ai/model-resolver.ts index cc668c67a..ba03e3383 100644 --- a/src/core/ai/model-resolver.ts +++ b/src/core/ai/model-resolver.ts @@ -6,7 +6,22 @@ import type { ParsedModelId, Recipe, TouchpointKind, ChatTouchpoint, EmbeddingTo import { getRecipe, RECIPES } from './recipes/index.ts'; import { AIConfigError } from './errors.ts'; -/** Split "openai:text-embedding-3-large" into { providerId, modelId }. */ +/** + * Split "openai:text-embedding-3-large" or "openai/text-embedding-3-large" + * into { providerId, modelId }. Colon takes precedence so OpenRouter nested + * ids like "openrouter:anthropic/claude-sonnet-4-6" route as + * { providerId: 'openrouter', modelId: 'anthropic/claude-sonnet-4-6' }. + * + * v0.41.21.0: slash form added so users typing `anthropic/claude-sonnet-4-6` + * (the form OpenRouter recipes emit and CLI `--judge-model` accepts) reach + * the gateway successfully. Pre-fix the colon-only check threw at every + * gateway entry point (chat / embed / rerank), so a slash-form id passed + * pricing checks via splitProviderModelId in `src/core/model-id.ts` and + * then died here at the gateway resolver. Closes the end-to-end bug class. + * + * Bare names without ANY separator still throw — `claude-sonnet-4-6` alone + * doesn't tell us which provider to route through. + */ export function parseModelId(id: string): ParsedModelId { if (!id || typeof id !== 'string') { throw new AIConfigError( @@ -14,15 +29,23 @@ export function parseModelId(id: string): ParsedModelId { 'Expected format: provider:model (e.g. openai:text-embedding-3-large)', ); } + // Colon wins over slash (OpenRouter nested-id semantic). const colon = id.indexOf(':'); - if (colon === -1) { - throw new AIConfigError( - `Model id "${id}" is missing a provider prefix.`, - 'Use format provider:model, e.g. openai:text-embedding-3-large', - ); + let sepIdx: number; + if (colon !== -1) { + sepIdx = colon; + } else { + const slash = id.indexOf('/'); + if (slash === -1) { + throw new AIConfigError( + `Model id "${id}" is missing a provider prefix.`, + 'Use format provider:model (preferred) or provider/model, e.g. openai:text-embedding-3-large', + ); + } + sepIdx = slash; } - const providerId = id.slice(0, colon).trim().toLowerCase(); - const modelId = id.slice(colon + 1).trim(); + const providerId = id.slice(0, sepIdx).trim().toLowerCase(); + const modelId = id.slice(sepIdx + 1).trim(); if (!providerId || !modelId) { throw new AIConfigError( `Model id "${id}" has empty provider or model.`, diff --git a/src/core/anthropic-pricing.ts b/src/core/anthropic-pricing.ts index 22c771a21..b886ddc62 100644 --- a/src/core/anthropic-pricing.ts +++ b/src/core/anthropic-pricing.ts @@ -33,6 +33,8 @@ export const ANTHROPIC_PRICING: Record = { 'claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 }, }; +import { splitProviderModelId } from './model-id.ts'; + /** * Estimate the upper-bound USD cost of a single submit. * Uses (estimatedInputTokens × inputRate) + (maxOutputTokens × outputRate). @@ -41,20 +43,22 @@ export const ANTHROPIC_PRICING: Record = { * * Returns null when the model isn't in the pricing map. Callers warn-once * and treat as zero-cost (the cycle runs unbounded for that submit). + * + * Accepts bare (`claude-opus-4-7`), colon-prefixed (`anthropic:claude-opus-4-7`), + * and slash-prefixed (`anthropic/claude-opus-4-7`) ids. Routes through + * `splitProviderModelId` so the slash-form (which arrives via CLI `--judge-model` + * and OpenRouter recipe lists) hits the pricing table. Pre-v0.41.21.0 the inline + * `:`-only split missed slash form → BudgetTracker no_pricing hard-fail with + * `--max-cost N` (closes #1540). */ export function estimateMaxCostUsd( modelId: string, estimatedInputTokens: number, maxOutputTokens: number, ): number | null { - // Accept both bare (`claude-opus-4-7`) and provider-prefixed - // (`anthropic:claude-opus-4-7`) ids. Required since cebu-v4's - // model-config rewrite (commit c4f03a9d) prefixes every default — without - // tail fallback, every internal call would hit BUDGET_METER_NO_PRICING and - // silently disable the budget gate. - let p = ANTHROPIC_PRICING[modelId]; - if (!p && modelId.includes(':')) { - const tail = modelId.split(':', 2)[1]; + let p: ModelPricing | undefined = ANTHROPIC_PRICING[modelId]; + if (!p) { + const { model: tail } = splitProviderModelId(modelId); if (tail) p = ANTHROPIC_PRICING[tail]; } if (!p) return null; diff --git a/src/core/brainstorm/judges.ts b/src/core/brainstorm/judges.ts index ca7ef20b5..00cdad468 100644 --- a/src/core/brainstorm/judges.ts +++ b/src/core/brainstorm/judges.ts @@ -22,10 +22,91 @@ * `chatFn` injection point. */ -import { chat as defaultChat, type ChatResult, type ChatOpts } from '../ai/gateway.ts'; +import { chat as defaultChat, getChatModel, type ChatResult, type ChatOpts } from '../ai/gateway.ts'; +import { splitProviderModelId } from '../model-id.ts'; export const PROMPT_VERSION = 'brainstorm-judge-v1'; +// --------------------------------------------------------------------------- +// v0.41.20.0 — maxTokens scaling constants +// +// The judge emits ~100 tokens of JSON per idea (id + 5 axis scores + +// one-sentence note). With 36-96 ideas the response was consistently +// truncated mid-JSON when maxTokens was hard-coded at 4000 (closes #1540). +// The formula scales output budget with idea count while respecting the +// resolved model's actual output cap. +// +// Realistic chunks under default `maxIdeasPerCall=100` produce ≤15,500 +// tokens — comfortably under every supported modern Anthropic model. The +// per-model cap binds before any opaque provider HTTP 400 fires. +// --------------------------------------------------------------------------- + +/** Observed ~100 tok/idea; 1.5× headroom keeps malformed-row retries cheap. */ +export const TOKEN_BUDGET_PER_IDEA = 150; +/** JSON outer wrapper + leading/trailing markers + per-call overhead. */ +export const TOKEN_BUDGET_ENVELOPE = 500; +/** Pre-v0.41.20.0 hard-coded floor; preserved so 1-idea batches still get headroom. */ +export const LEGACY_MIN_MAX_TOKENS = 4000; +/** Fallback cap when the resolved model isn't in ANTHROPIC_OUTPUT_CAPS. Matches Opus 4.7's cap. */ +export const MAX_OUTPUT_TOKENS_CEIL = 32_000; + +/** + * Per-model max output tokens. Anthropic's published caps as of 2026-05. + * Lookup keyed on the bare model name (after parseModelId strip), so both + * `claude-sonnet-4-6` and `anthropic:claude-sonnet-4-6` and + * `anthropic/claude-sonnet-4-6` resolve to the same entry. + * + * Unknown models fall back to MAX_OUTPUT_TOKENS_CEIL — safe for every + * current Anthropic model but tight enough that misconfig hits OUR bound + * (with a readable error) instead of the provider's opaque HTTP 400. + */ +export const ANTHROPIC_OUTPUT_CAPS: Record = { + 'claude-opus-4-7': 32_000, + 'claude-sonnet-4-6': 64_000, + 'claude-haiku-4-5': 64_000, + 'claude-haiku-4-5-20251001': 64_000, + // Legacy 3.5 generation caps at 8,192 — much smaller. Without these + // entries, a `--judge-model anthropic:claude-3-5-haiku-20241022` with + // 96 ideas would request 14,900 tokens > 8K cap → HTTP 400. + 'claude-3-5-sonnet-20241022': 8_192, + 'claude-3-5-haiku-20241022': 8_192, +}; + +/** + * Resolve the per-model output cap for a (possibly provider-prefixed) model id. + * + * v0.41.21.0: when no explicit `modelId` is passed, resolve the actual default + * chat model via the gateway so the cap matches what `chat()` will use, not + * whatever the override hints at. Pre-fix the undefined-override case fell + * back to MAX_OUTPUT_TOKENS_CEIL=32K, which would request 14_900 tokens for + * a 96-idea batch even if the configured default was a legacy 8K model → + * provider HTTP 400. + */ +function resolveOutputCap(modelId: string | undefined): number { + let resolved = modelId; + if (!resolved) { + // Try the gateway's configured chat model. Wrap in try/catch because + // judges.ts is sometimes called in test contexts where the gateway + // isn't configured yet (`configureGateway` not called); fall through + // to the safe ceiling. + try { + resolved = getChatModel(); + } catch { + return MAX_OUTPUT_TOKENS_CEIL; + } + } + if (!resolved) return MAX_OUTPUT_TOKENS_CEIL; + const bare = splitProviderModelId(resolved).model; + return ANTHROPIC_OUTPUT_CAPS[bare] ?? MAX_OUTPUT_TOKENS_CEIL; +} + +/** Compute the maxTokens budget for a judge call given idea count + resolved model id. */ +export function computeJudgeMaxTokens(ideaCount: number, modelId: string | undefined): number { + const cap = resolveOutputCap(modelId); + const scaled = ideaCount * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE; + return Math.min(cap, Math.max(LEGACY_MIN_MAX_TOKENS, scaled)); +} + /** One idea handed to the judge. The orchestrator builds these from the cross output. */ export interface JudgeIdea { /** Stable id within this run (e.g. "01", "02"). */ @@ -448,7 +529,11 @@ async function runJudgeChunk( // knob isn't on ChatOpts (it's set per-provider in instantiateChat), // so we rely on the default. If we ever need temperature control here // we'd extend ChatOpts. - maxTokens: 4000, + // + // v0.41.20.0: maxTokens scales with idea count + per-model cap. + // See computeJudgeMaxTokens / ANTHROPIC_OUTPUT_CAPS above. Closes #1540 + // (judge truncation at default 36-96 idea batches). + maxTokens: computeJudgeMaxTokens(ideas.length, options.modelOverride), abortSignal: options.abortSignal, }); diff --git a/src/core/budget/budget-tracker.ts b/src/core/budget/budget-tracker.ts index d6b5e7961..a85fbc148 100644 --- a/src/core/budget/budget-tracker.ts +++ b/src/core/budget/budget-tracker.ts @@ -33,6 +33,7 @@ import { dirname } from 'node:path'; import { gbrainPath } from '../config.ts'; import { ANTHROPIC_PRICING, type ModelPricing } from '../anthropic-pricing.ts'; import { EMBEDDING_PRICING, lookupEmbeddingPrice } from '../embedding-pricing.ts'; +import { splitProviderModelId } from '../model-id.ts'; import { isoWeekFilename, resolveAuditDir } from '../audit-week-file.ts'; export type BudgetKind = 'chat' | 'embed' | 'rerank'; @@ -181,10 +182,14 @@ function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null { } return null; } - // chat or rerank: try bare key first, then provider:model + // chat or rerank: try bare key first, then provider:model or provider/model. + // v0.41.21.0: route through splitProviderModelId so slash-prefixed ids + // (the form `--judge-model` and OpenRouter recipes emit) hit the pricing + // table. Pre-fix, slash-form silently no_pricing-failed `--max-cost` on + // brainstorm/lsd. const bare = ANTHROPIC_PRICING[modelId]; if (bare) return bare; - const [providerId, modelTail] = modelId.includes(':') ? modelId.split(':', 2) : [null, modelId]; + const { provider: providerId, model: modelTail } = splitProviderModelId(modelId); if (modelTail) { const tailHit = ANTHROPIC_PRICING[modelTail]; if (tailHit) return tailHit; diff --git a/src/core/eval-contradictions/cost-tracker.ts b/src/core/eval-contradictions/cost-tracker.ts index 952096e1d..fb8c64dfd 100644 --- a/src/core/eval-contradictions/cost-tracker.ts +++ b/src/core/eval-contradictions/cost-tracker.ts @@ -20,10 +20,17 @@ */ import type { CostBreakdown } from './types.ts'; +import { splitProviderModelId } from '../model-id.ts'; /** * Per-million-token prices (USD). Update when models bump. These are * approximate — provider accounting after the call is authoritative. + * + * NOTE: duplicate of the canonical `src/core/anthropic-pricing.ts` table. + * Slated for consolidation (TODOS.md #3 from v0.41.20.0 plan); keys differ + * (this table uses both bare and `anthropic:`-prefixed forms; canonical + * is bare-only). For now we route lookup through `parseModelId` so the + * slash-prefix bug class is closed at this site too. */ const ANTHROPIC_PRICING: Record = { // Haiku 4.5: ~$1/Mtok in, $5/Mtok out (current as of 2026-05). @@ -48,7 +55,21 @@ const ESTIMATE_NOTE = 'approximate; provider accounting is post-call. --budget-usd is a soft ceiling — mid-run stop on cumulative > cap.'; function pricingFor(modelId: string): { input: number; output: number } { - return ANTHROPIC_PRICING[modelId] ?? ANTHROPIC_PRICING['claude-haiku-4-5']; + // v0.41.21.0: route through splitProviderModelId so slash-prefixed ids + // (`anthropic/claude-sonnet-4-6`) hit the pricing table. Pre-fix the + // exact-key match silently fell back to Haiku on every non-bare lookup + // (including colon-form Sonnet/Opus that the table DOES carry — caller + // bug class). Legacy silent-Haiku fallback for genuinely-unknown models + // is preserved by design — see TODOS.md #3 for the pricing-system + // consolidation that would tighten this to warn-once. + const direct = ANTHROPIC_PRICING[modelId]; + if (direct) return direct; + const { model: tail } = splitProviderModelId(modelId); + if (tail) { + const tailHit = ANTHROPIC_PRICING[tail]; + if (tailHit) return tailHit; + } + return ANTHROPIC_PRICING['claude-haiku-4-5']; } /** diff --git a/src/core/minions/batch-projection.ts b/src/core/minions/batch-projection.ts index 970488595..45e16c375 100644 --- a/src/core/minions/batch-projection.ts +++ b/src/core/minions/batch-projection.ts @@ -21,6 +21,7 @@ */ import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts'; +import { splitProviderModelId } from '../model-id.ts'; export interface RecentJobStats { /** How many jobs informed this window. 0 → cold start. */ @@ -56,20 +57,18 @@ export interface BatchProjection { raise_cap_hint?: string; } -/** Strip `provider:` prefix the same way the SDK call site does. */ -function bareModel(model: string): string { - const idx = model.indexOf(':'); - return idx > 0 ? model.slice(idx + 1) : model; -} - /** * Resolve per-token cost for a model. Returns null for unknown models. * Conservative: uses output-side pricing as a tight upper bound when * we don't have per-call usage stats yet. + * + * v0.41.21.0: routes through splitProviderModelId so slash-prefixed ids + * (`anthropic/claude-sonnet-4-6`) strip to the bare model name. Pre-fix + * the inline `bareModel(model)` helper only handled `:`-form. */ function modelDefaultMeanCostUsd(model: string): number | null { // Match the alias map's behavior loosely: bare names + the few we know. - const bare = bareModel(model); + const bare = splitProviderModelId(model).model; const p = ANTHROPIC_PRICING[bare]; if (!p) return null; // Assume a typical subagent turn: ~2k input + ~1k output tokens. @@ -94,7 +93,7 @@ export interface ProjectBatchInput { export function projectBatch(input: ProjectBatchInput): BatchProjection { const { job_count, model, stats, current_lease_cap } = input; - const bare = bareModel(model); + const bare = splitProviderModelId(model).model; const cold = stats.sample_size === 0; // Mean latency: historical → use it. Cold → 5s guess. diff --git a/src/core/model-config.ts b/src/core/model-config.ts index ba6ce9db6..9b993c8d0 100644 --- a/src/core/model-config.ts +++ b/src/core/model-config.ts @@ -21,6 +21,7 @@ */ import type { BrainEngine } from './engine.ts'; +import { splitProviderModelId } from './model-id.ts'; export type ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'; @@ -89,16 +90,18 @@ export const TIER_DEFAULTS: Record = { */ export function isAnthropicProvider(modelString: string): boolean { if (!modelString) return false; - const trimmed = modelString.trim(); - // `provider:model` form: check provider prefix. - const colon = trimmed.indexOf(':'); - if (colon !== -1) { - return trimmed.slice(0, colon).trim().toLowerCase() === 'anthropic'; + // v0.41.21.0: route through splitProviderModelId so slash form + // (`anthropic/claude-sonnet-4-6`) also classifies as Anthropic. + // Pre-fix the inline `:`-only split silently returned false for slash + // form → subagent guard bypass → silent fallback to TIER_DEFAULTS. + const { provider, model } = splitProviderModelId(modelString); + if (provider !== null) { + return provider.trim().toLowerCase() === 'anthropic'; } - // Bare model id: known Anthropic models start with `claude-`. Conservative: - // we'd rather warn-on-Anthropic-typo than silently route gpt-5 to the - // subagent loop. - return trimmed.toLowerCase().startsWith('claude-'); + // Bare model id (no separator): known Anthropic models start with `claude-`. + // Conservative: we'd rather warn-on-Anthropic-typo than silently route + // gpt-5 to the subagent loop. + return model.toLowerCase().startsWith('claude-'); } const _subagentTierWarningsEmitted = new Set(); diff --git a/src/core/model-id.ts b/src/core/model-id.ts new file mode 100644 index 000000000..512e296dc --- /dev/null +++ b/src/core/model-id.ts @@ -0,0 +1,64 @@ +/** + * v0.41.21.0 — single source of truth for model-id parsing (PRICING side). + * + * Splits `provider:model`, `provider/model`, and bare `model` strings into + * a `{provider, model}` pair. Five pricing/budget sites across the codebase + * used to inline their own ad-hoc split (colon-only); the slash-form miss + * kept refiring as a bug class (#1540 most recently). One helper kills it. + * + * **Name disambiguation:** the gateway-side resolver `src/core/ai/model-resolver.ts` + * has its own `parseModelId` that throws on bare names (a gateway routing + * decision needs an explicit provider; pricing can fall through to bare-key + * pricing-table lookup). To avoid in-project name collision, this helper is + * named `splitProviderModelId`. Both functions accept the same input shapes + * after v0.41.21.0; they differ in how they handle bare names (this returns + * `{provider: null, model: 'bare'}`; the gateway one throws). + * + * Separator precedence: `:` wins over `/`. The motivating case is + * OpenRouter's nested form `openrouter:anthropic/claude-sonnet-4.6` — + * the canonical transport-vs-vendor split is on the leading colon, so the + * helper returns `{provider: 'openrouter', model: 'anthropic/claude-sonnet-4.6'}`. + * Downstream pricing lookups that miss on the slash-bearing tail land in + * the caller's existing "unknown model" path (warn-once or no_pricing, + * depending on the caller). We do NOT recursively peel inner provider + * prefixes — that would conflate transport identity with billing identity + * (OpenRouter markup ≠ native Anthropic pricing). + * + * Defensive contract: null / undefined / empty / whitespace-only input + * returns `{provider: null, model: ''}` rather than throwing. The TypeScript + * signature reflects this so callers can pass uncertain input without + * `as any` casts (env-var-unset paths, optional config fields). + */ + +export interface SplitProviderModelId { + /** Provider prefix when separator present; null for bare or empty input. */ + provider: string | null; + /** Model tail after the separator; '' for empty input. */ + model: string; +} + +const EMPTY: SplitProviderModelId = { provider: null, model: '' }; + +export function splitProviderModelId(input: string | null | undefined): SplitProviderModelId { + if (input === null || input === undefined) return EMPTY; + const trimmed = input.trim(); + if (trimmed.length === 0) return EMPTY; + + const colon = trimmed.indexOf(':'); + if (colon !== -1) { + return { + provider: trimmed.slice(0, colon), + model: trimmed.slice(colon + 1), + }; + } + + const slash = trimmed.indexOf('/'); + if (slash !== -1) { + return { + provider: trimmed.slice(0, slash), + model: trimmed.slice(slash + 1), + }; + } + + return { provider: null, model: trimmed }; +} diff --git a/test/ai/model-resolver-slash.test.ts b/test/ai/model-resolver-slash.test.ts new file mode 100644 index 000000000..70cbbb633 --- /dev/null +++ b/test/ai/model-resolver-slash.test.ts @@ -0,0 +1,96 @@ +/** + * v0.41.21.0 — gateway-resolver parseModelId accepts slash form. + * + * Codex adversarial review caught a load-bearing gap: the v0.41.21.0 + * pricing-side fix (`src/core/model-id.ts:splitProviderModelId`) let + * BudgetTracker pass for slash-form ids, but `gateway.chat()` then routed + * through `src/core/ai/model-resolver.ts:parseModelId` which still hard- + * rejected no-colon ids → AIConfigError mid-judge → judge_failed for the + * end-to-end user. The fix here extends model-resolver.ts:parseModelId to + * also accept slash form, completing the end-to-end bug class closure. + * + * Bare names without ANY separator STILL throw — gateway routing always + * needs an explicit provider. This file pins both: + * - slash form parses successfully + * - bare names still throw (back-compat) + */ + +import { describe, test, expect } from 'bun:test'; +import { parseModelId, resolveRecipe } from '../../src/core/ai/model-resolver.ts'; +import { AIConfigError } from '../../src/core/ai/errors.ts'; + +describe('model-resolver parseModelId (gateway-side)', () => { + describe('happy paths', () => { + test('colon form parses (back-compat)', () => { + expect(parseModelId('anthropic:claude-sonnet-4-6')).toEqual({ + providerId: 'anthropic', + modelId: 'claude-sonnet-4-6', + }); + }); + + test('slash form parses (THE END-TO-END FIX)', () => { + // Pre-v0.41.21.0: threw AIConfigError "missing a provider prefix" + // → brainstorm/lsd judge_failed despite pricing fix. + expect(parseModelId('anthropic/claude-sonnet-4-6')).toEqual({ + providerId: 'anthropic', + modelId: 'claude-sonnet-4-6', + }); + }); + + test('colon wins over slash (OpenRouter nested-id semantic)', () => { + // openrouter:anthropic/claude-... → transport=openrouter, model + // includes the nested anthropic/ prefix verbatim. + expect(parseModelId('openrouter:anthropic/claude-sonnet-4.6')).toEqual({ + providerId: 'openrouter', + modelId: 'anthropic/claude-sonnet-4.6', + }); + }); + + test('provider name is lowercased', () => { + expect(parseModelId('Anthropic/claude-sonnet-4-6').providerId).toBe('anthropic'); + expect(parseModelId('OPENAI:gpt-5').providerId).toBe('openai'); + }); + }); + + describe('reject paths preserved', () => { + test('bare name with NO separator throws', () => { + expect(() => parseModelId('claude-sonnet-4-6')).toThrow(AIConfigError); + expect(() => parseModelId('claude-sonnet-4-6')).toThrow(/missing a provider prefix/); + }); + + test('empty string throws', () => { + expect(() => parseModelId('')).toThrow(AIConfigError); + }); + + test('null/undefined throws', () => { + expect(() => parseModelId(null as unknown as string)).toThrow(AIConfigError); + expect(() => parseModelId(undefined as unknown as string)).toThrow(AIConfigError); + }); + + test('trailing-only separator throws (empty model)', () => { + expect(() => parseModelId('anthropic:')).toThrow(AIConfigError); + expect(() => parseModelId('anthropic/')).toThrow(AIConfigError); + }); + + test('leading-only separator throws (empty provider)', () => { + expect(() => parseModelId(':claude-sonnet-4-6')).toThrow(AIConfigError); + expect(() => parseModelId('/claude-sonnet-4-6')).toThrow(AIConfigError); + }); + }); + + describe('resolveRecipe end-to-end with slash form', () => { + test('slash form resolves to the same recipe as colon form', () => { + const colonResult = resolveRecipe('anthropic:claude-sonnet-4-6'); + const slashResult = resolveRecipe('anthropic/claude-sonnet-4-6'); + expect(slashResult.recipe.id).toBe(colonResult.recipe.id); + expect(slashResult.parsed.providerId).toBe(colonResult.parsed.providerId); + expect(slashResult.parsed.modelId).toBe(colonResult.parsed.modelId); + }); + + test('slash form gives the expected recipe for opus', () => { + const result = resolveRecipe('anthropic/claude-opus-4-7'); + expect(result.parsed.providerId).toBe('anthropic'); + expect(result.parsed.modelId).toBe('claude-opus-4-7'); + }); + }); +}); diff --git a/test/anthropic-pricing.test.ts b/test/anthropic-pricing.test.ts new file mode 100644 index 000000000..c56af7918 --- /dev/null +++ b/test/anthropic-pricing.test.ts @@ -0,0 +1,67 @@ +/** + * v0.41.20.0 — pin estimateMaxCostUsd across bare/colon/slash/unknown ids. + * + * No prior coverage existed for this helper. The slash-form bug class + * (#1540) refired here for OpenRouter and CLI `--judge-model` users + * before this fix; this file pins the centralized parse path so any + * future refactor of parseModelId or estimateMaxCostUsd can't silently + * drop slash-form support. + */ + +import { describe, test, expect } from 'bun:test'; +import { ANTHROPIC_PRICING, estimateMaxCostUsd } from '../src/core/anthropic-pricing.ts'; + +describe('estimateMaxCostUsd', () => { + // Sonnet 4.6 = $3 input / $15 output per MTok. + // 1M input + 0 output → $3.00 + // 0 input + 1M output → $15.00 + + test('bare key claude-sonnet-4-6 → hits pricing', () => { + const cost = estimateMaxCostUsd('claude-sonnet-4-6', 1_000_000, 0); + expect(cost).toBeCloseTo(3.0, 5); + }); + + test('colon-prefixed anthropic:claude-sonnet-4-6 → hits pricing via tail', () => { + const cost = estimateMaxCostUsd('anthropic:claude-sonnet-4-6', 1_000_000, 0); + expect(cost).toBeCloseTo(3.0, 5); + }); + + test('slash-prefixed anthropic/claude-sonnet-4-6 → hits pricing via tail (THE FIX)', () => { + // Pre-v0.41.20.0: this returned null because the inline split only + // handled `:`. CLI `--judge-model anthropic/...` + `--max-cost N` then + // hit BudgetTracker no_pricing fail-closed. + const cost = estimateMaxCostUsd('anthropic/claude-sonnet-4-6', 1_000_000, 0); + expect(cost).toBeCloseTo(3.0, 5); + }); + + test('mixed input + output cost math', () => { + // 100K input + 50K output for opus 4.7 ($5/$25) + // = 0.1 * 5 + 0.05 * 25 = 0.5 + 1.25 = 1.75 + const cost = estimateMaxCostUsd('anthropic/claude-opus-4-7', 100_000, 50_000); + expect(cost).toBeCloseTo(1.75, 5); + }); + + test('unknown model → returns null (caller warn-once + bypass)', () => { + expect(estimateMaxCostUsd('mistral:medium', 1_000, 1_000)).toBeNull(); + expect(estimateMaxCostUsd('gpt-5', 1_000, 1_000)).toBeNull(); + }); + + test('OpenRouter nested form returns null — tail is `anthropic/claude-...` which is not a pricing key', () => { + // Per D2 architecture: parseModelId returns {provider:'openrouter', + // model:'anthropic/claude-sonnet-4-6'}; lookup on the tail + // 'anthropic/claude-sonnet-4-6' misses (table has bare 'claude-sonnet-4-6'). + // OpenRouter pricing is intentionally out of scope (TODO #2). + expect(estimateMaxCostUsd('openrouter:anthropic/claude-sonnet-4-6', 1_000, 1_000)).toBeNull(); + }); + + test('every key in ANTHROPIC_PRICING is reachable via bare/colon/slash form', () => { + // Regression guard: if someone adds a new entry to ANTHROPIC_PRICING, + // it should be reachable via all three forms automatically (the route + // is structural, not per-key). + for (const key of Object.keys(ANTHROPIC_PRICING)) { + expect(estimateMaxCostUsd(key, 1_000_000, 0)).not.toBeNull(); + expect(estimateMaxCostUsd(`anthropic:${key}`, 1_000_000, 0)).not.toBeNull(); + expect(estimateMaxCostUsd(`anthropic/${key}`, 1_000_000, 0)).not.toBeNull(); + } + }); +}); diff --git a/test/batch-projection.test.ts b/test/batch-projection.test.ts index 61d7b42fa..f2a969b23 100644 --- a/test/batch-projection.test.ts +++ b/test/batch-projection.test.ts @@ -132,6 +132,38 @@ describe('projectBatch', () => { }); expect(p.raise_cap_hint).toBeUndefined(); }); + + describe('v0.41.20.0 — slash-prefix model id routing (THE FIX)', () => { + test('slash-form anthropic/claude-sonnet-4-6 strips to bare name + pricing hits', () => { + // Pre-fix: inline bareModel(model) only handled `:`; slash-form fell + // through to the unknown_model branch silently. Post-fix: parseModelId + // handles both forms; pricing lookup succeeds; cold-start path produces + // a non-null cost estimate. + const p = projectBatch({ + job_count: 100, + model: 'anthropic/claude-sonnet-4-6', + stats: { sample_size: 0, effective_concurrency: 4 }, + current_lease_cap: 32, + }); + expect(p.unknown_model).toBeUndefined(); + expect(p.total_cost_usd).not.toBeNull(); + expect(p.total_cost_usd).toBeGreaterThan(0); + }); + + test('double-separator openrouter:anthropic/X → unknown_model branch fires', () => { + // Per D2: colon wins; tail is `anthropic/claude-sonnet-4-6` which + // doesn't match ANTHROPIC_PRICING keys. Confirms the deliberate + // OpenRouter-out-of-scope posture is observable downstream. + const p = projectBatch({ + job_count: 100, + model: 'openrouter:anthropic/claude-sonnet-4-6', + stats: { sample_size: 0, effective_concurrency: 4 }, + current_lease_cap: 32, + }); + expect(p.unknown_model).toBe('anthropic/claude-sonnet-4-6'); + expect(p.total_cost_usd).toBeNull(); + }); + }); }); describe('formatProjection', () => { diff --git a/test/brainstorm/judges-maxtokens.test.ts b/test/brainstorm/judges-maxtokens.test.ts new file mode 100644 index 000000000..a372af091 --- /dev/null +++ b/test/brainstorm/judges-maxtokens.test.ts @@ -0,0 +1,196 @@ +/** + * v0.41.20.0 — pin the judge maxTokens scaling formula + per-model cap. + * + * Bug 1 from #1540: `maxTokens: 4000` hard-coded in judges.ts → response + * truncated mid-JSON at any chunk ≥ ~40 ideas → parseJudgeJSON throw → + * judge_failed: true → ideas saved unscored. Verified failure mode: + * 0/72 ideas passing before fix; 39/72 after. + * + * The fix is in two pieces and this file pins both: + * + * 1. `computeJudgeMaxTokens` (pure formula) — scales with idea count, + * respects per-model output cap, floors at 4000 for tiny batches. + * 2. `runJudgeChunk` wires the formula into the `chat({maxTokens})` call + * — integration test via the existing `chatFn` DI seam captures the + * ChatOpts and asserts maxTokens matches the formula. + */ + +import { describe, test, expect } from 'bun:test'; +import { + computeJudgeMaxTokens, + TOKEN_BUDGET_PER_IDEA, + TOKEN_BUDGET_ENVELOPE, + LEGACY_MIN_MAX_TOKENS, + MAX_OUTPUT_TOKENS_CEIL, + ANTHROPIC_OUTPUT_CAPS, + runJudge, + BRAINSTORM_JUDGE_CONFIG, + type ChatFn, +} from '../../src/core/brainstorm/judges.ts'; +import type { ChatOpts, ChatResult } from '../../src/core/ai/gateway.ts'; + +describe('computeJudgeMaxTokens (pure formula)', () => { + test('1 idea: formula yields 650 → floor binds at LEGACY_MIN_MAX_TOKENS (4000)', () => { + // 1 * 150 + 500 = 650; max(4000, 650) = 4000. + expect(computeJudgeMaxTokens(1, 'claude-sonnet-4-6')).toBe(LEGACY_MIN_MAX_TOKENS); + }); + + test('10 ideas: formula yields 2000 → floor binds', () => { + // 10 * 150 + 500 = 2000; max(4000, 2000) = 4000. + expect(computeJudgeMaxTokens(10, 'claude-sonnet-4-6')).toBe(LEGACY_MIN_MAX_TOKENS); + }); + + test('36 ideas: formula yields 5900 → above floor', () => { + // 36 * 150 + 500 = 5900; max(4000, 5900) = 5900; below 64K Sonnet cap. + expect(computeJudgeMaxTokens(36, 'claude-sonnet-4-6')).toBe(36 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE); + }); + + test('96 ideas: formula yields 14_900 → above floor, under modern cap', () => { + // 96 * 150 + 500 = 14_900; below 32K Opus cap. + expect(computeJudgeMaxTokens(96, 'claude-opus-4-7')).toBe(96 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE); + }); + + test('300 ideas on Opus 4.7: formula yields 45_500 → CAP binds at 32K', () => { + // 300 * 150 + 500 = 45_500; min(32_000, 45_500) = 32_000. + expect(computeJudgeMaxTokens(300, 'claude-opus-4-7')).toBe(32_000); + }); + + test('300 ideas on Sonnet 4.6: formula yields 45_500 → fits under 64K Sonnet cap', () => { + expect(computeJudgeMaxTokens(300, 'claude-sonnet-4-6')).toBe(300 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE); + }); + + test('96 ideas on legacy Haiku 3.5 (8K cap): CAP binds at 8192 (D11 codex fix)', () => { + // 96 * 150 + 500 = 14_900 > 8192; legacy 3.5 caps at 8K — without + // ANTHROPIC_OUTPUT_CAPS this would have been the next opaque HTTP 400. + expect(computeJudgeMaxTokens(96, 'claude-3-5-haiku-20241022')).toBe(8_192); + }); + + test('unknown model: falls back to MAX_OUTPUT_TOKENS_CEIL', () => { + expect(computeJudgeMaxTokens(300, 'mistral:medium')).toBe(MAX_OUTPUT_TOKENS_CEIL); + expect(computeJudgeMaxTokens(300, 'gpt-5')).toBe(MAX_OUTPUT_TOKENS_CEIL); + }); + + // v0.41.21.0: when modelId is undefined the cap routes through the gateway's + // configured chat model via getChatModel(). The actual returned cap therefore + // depends on whether the gateway has been initialized in this test process + // (cross-test side effect of any earlier import that called configureGateway). + // We test the explicit-modelId path comprehensively above; the undefined path + // is exercised end-to-end below via runJudge() without modelOverride. + + test('colon-prefixed id resolves through splitProviderModelId', () => { + expect(computeJudgeMaxTokens(96, 'anthropic:claude-opus-4-7')).toBe(96 * 150 + 500); + }); + + test('slash-prefixed id resolves through splitProviderModelId (THE FIX combined with site routing)', () => { + expect(computeJudgeMaxTokens(96, 'anthropic/claude-opus-4-7')).toBe(96 * 150 + 500); + // Legacy 3.5 via slash form still hits the 8K cap. + expect(computeJudgeMaxTokens(96, 'anthropic/claude-3-5-haiku-20241022')).toBe(8_192); + }); + + test('every entry in ANTHROPIC_OUTPUT_CAPS is reachable by lookup', () => { + for (const [key, cap] of Object.entries(ANTHROPIC_OUTPUT_CAPS)) { + // Pick an idea count high enough that the cap binds. + const huge = Math.ceil(cap / TOKEN_BUDGET_PER_IDEA) + 10; + expect(computeJudgeMaxTokens(huge, key)).toBe(cap); + } + }); +}); + +describe('runJudge wires computeJudgeMaxTokens into chat({maxTokens})', () => { + function makeCapturingChatFn(captured: ChatOpts[]): ChatFn { + return async (opts: ChatOpts): Promise => { + captured.push(opts); + // Return a valid-shape judge response so parseJudgeJSON succeeds and + // we can pin maxTokens without dealing with parse failures. + const ideasJson = (opts.messages[0].content as string) + .match(/^- id=(\S+)/gm) + ?.map((line) => line.replace(/^- id=/, '')) ?? []; + const ideas = ideasJson.map((id) => ({ + id, + scores: { + originality: 3, + resistance: 3, + thesis_density: 3, + concrete_grounding: 3, + cognitive_load: 3, + }, + note: 'stub', + })); + const text = JSON.stringify({ ideas }); + return { + text, + blocks: [{ type: 'text', text }], + stopReason: 'end', + model: opts.model ?? 'noop', + providerId: 'anthropic', + usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, + }; + }; + } + + function makeIdeas(n: number) { + return Array.from({ length: n }, (_, i) => ({ + id: String(i + 1).padStart(2, '0'), + text: 'stub idea text', + close_slug: 'concepts/foo', + far_slug: 'wiki/bar', + })); + } + + test('1 idea → maxTokens = LEGACY_MIN_MAX_TOKENS (4000)', async () => { + const captured: ChatOpts[] = []; + await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(1), { + chatFn: makeCapturingChatFn(captured), + modelOverride: 'claude-sonnet-4-6', + }); + expect(captured.length).toBe(1); + expect(captured[0].maxTokens).toBe(LEGACY_MIN_MAX_TOKENS); + }); + + test('96 ideas on Opus 4.7 → maxTokens = formula (14_900)', async () => { + const captured: ChatOpts[] = []; + await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(96), { + chatFn: makeCapturingChatFn(captured), + modelOverride: 'anthropic:claude-opus-4-7', + }); + expect(captured.length).toBe(1); + expect(captured[0].maxTokens).toBe(96 * 150 + 500); + }); + + test('slash-form modelOverride routes through parseModelId for the cap lookup', async () => { + // Pre-v0.41.20.0 the inline maxTokens was a constant; this test would + // not have caught anything. Post-fix: slash-form is honored for the + // per-model cap because computeJudgeMaxTokens routes through parseModelId. + const captured: ChatOpts[] = []; + await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(96), { + chatFn: makeCapturingChatFn(captured), + modelOverride: 'anthropic/claude-3-5-haiku-20241022', + }); + // 14_900 formula > 8K legacy cap → cap binds. + expect(captured[0].maxTokens).toBe(8_192); + }); + + test('200-idea chunk size set via maxIdeasPerCall → maxTokens scales (single chunk)', async () => { + const captured: ChatOpts[] = []; + await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(200), { + chatFn: makeCapturingChatFn(captured), + modelOverride: 'claude-sonnet-4-6', + maxIdeasPerCall: 200, + }); + expect(captured.length).toBe(1); + expect(captured[0].maxTokens).toBe(200 * 150 + 500); // 30_500, under 64K Sonnet cap + }); + + test('chunking is independent of cap — multi-chunk each gets its own scaled budget', async () => { + const captured: ChatOpts[] = []; + // 250 ideas at default chunk 100 → 3 chunks of [100, 100, 50]. + await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(250), { + chatFn: makeCapturingChatFn(captured), + modelOverride: 'claude-sonnet-4-6', + }); + expect(captured.length).toBe(3); + expect(captured[0].maxTokens).toBe(100 * 150 + 500); // 15_500 + expect(captured[1].maxTokens).toBe(100 * 150 + 500); + expect(captured[2].maxTokens).toBe(Math.max(LEGACY_MIN_MAX_TOKENS, 50 * 150 + 500)); // 8000 + }); +}); diff --git a/test/core/budget/budget-tracker.test.ts b/test/core/budget/budget-tracker.test.ts index f30834e26..9dd236bb4 100644 --- a/test/core/budget/budget-tracker.test.ts +++ b/test/core/budget/budget-tracker.test.ts @@ -138,6 +138,38 @@ describe('BudgetTracker.reserve', () => { expect((caught as Error).message).toMatch(/anthropic-pricing\.ts/); }); + test('v0.41.20.0: slash-prefix anthropic/claude-* under --max-cost does NOT no_pricing throw (THE FIX)', () => { + // Pre-v0.41.20.0: lookupPricing only split modelId on ':'. CLI users + // running `gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6 + // --max-cost 5` hit TX2 no_pricing because the slash-form id silently + // missed ANTHROPIC_PRICING (closes #1540). + const t = new BudgetTracker({ maxCostUsd: 10.0, label: 'test', auditPath }); + expect(() => + t.reserve({ + modelId: 'anthropic/claude-sonnet-4-6', + estimatedInputTokens: 100, + maxOutputTokens: 100, + kind: 'chat', + }), + ).not.toThrow(); + const audit = readAudit(); + expect(audit[0].event).toBe('reserve'); + }); + + test('v0.41.20.0: colon-prefix anthropic:claude-* under --max-cost still works (regression guard)', () => { + // Same path as slash, exercised separately so a future refactor that + // accidentally drops colon support fires this test loudly. + const t = new BudgetTracker({ maxCostUsd: 10.0, label: 'test', auditPath }); + expect(() => + t.reserve({ + modelId: 'anthropic:claude-sonnet-4-6', + estimatedInputTokens: 100, + maxOutputTokens: 100, + kind: 'chat', + }), + ).not.toThrow(); + }); + test('no cap + unknown pricing: warns once per process, no throw', () => { const t = new BudgetTracker({ label: 'test', auditPath }); expect(() => diff --git a/test/eval-contradictions/cost-tracker-slash.test.ts b/test/eval-contradictions/cost-tracker-slash.test.ts new file mode 100644 index 000000000..e631ba062 --- /dev/null +++ b/test/eval-contradictions/cost-tracker-slash.test.ts @@ -0,0 +1,61 @@ +/** + * v0.41.20.0 — pin slash-prefix model id routing through + * `eval-contradictions/cost-tracker.ts:pricingFor`. + * + * The cost-tracker carries its own duplicate ANTHROPIC_PRICING table + * (consolidation deferred to TODOS.md #3 from the v0.41.20.0 plan). + * Pre-fix, `pricingFor` did exact-key match only, so every non-bare + * lookup silently fell back to Haiku — including colon-form Sonnet/Opus + * that the table DOES carry as explicit `anthropic:claude-*` keys. After + * fix, parseModelId handles bare/colon/slash uniformly; the silent-Haiku + * fallback is preserved only for genuinely-unknown models (legacy behavior + * pinned here per D9). + */ + +import { describe, test, expect } from 'bun:test'; +import { CostTracker } from '../../src/core/eval-contradictions/cost-tracker.ts'; + +describe('eval-contradictions/cost-tracker pricingFor (via recordJudgeCall)', () => { + function spendCheck(modelId: string, inputTokens: number, outputTokens: number): number { + const t = new CostTracker({ capUsd: 999 }); + t.recordJudgeCall(modelId, { inputTokens, outputTokens }); + return t.judge(); + } + + test('bare claude-sonnet-4-6 → Sonnet pricing (1M in = $3)', () => { + expect(spendCheck('claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5); + }); + + test('colon-form anthropic:claude-sonnet-4-6 → Sonnet pricing', () => { + // Table has this key explicitly; pre-fix path also worked. + expect(spendCheck('anthropic:claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5); + }); + + test('slash-form anthropic/claude-sonnet-4-6 → Sonnet pricing (THE FIX)', () => { + // Pre-v0.41.20.0: silently billed as Haiku ($1/MTok instead of $3/MTok). + expect(spendCheck('anthropic/claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5); + }); + + test('slash-form anthropic/claude-opus-4-7 → Opus pricing ($5/MTok in)', () => { + expect(spendCheck('anthropic/claude-opus-4-7', 1_000_000, 0)).toBeCloseTo(5.0, 5); + }); + + test('LEGACY BEHAVIOR PIN: unknown model silently falls back to Haiku pricing', () => { + // D9 from the v0.41.20.0 plan: we deliberately preserve the silent-Haiku + // fallback in this duplicate pricing table. The right fix is unifying + // the two pricing systems (TODOS.md #3) — tightening cost-tracker in + // isolation would surprise existing eval-contradictions callers who + // depend on the soft-ceiling --budget-usd contract. + expect(spendCheck('mistral/medium', 1_000_000, 0)).toBeCloseTo(1.0, 5); + expect(spendCheck('gpt-5', 1_000_000, 0)).toBeCloseTo(1.0, 5); + }); + + test('OpenRouter nested form falls back to Haiku (legacy behavior preserved)', () => { + // Per D2: parseModelId returns {provider:'openrouter', model:'anthropic/...'}; + // the tail 'anthropic/claude-sonnet-4-6' is not a pricing key in this + // duplicate table (which doesn't carry slash-form keys), so the silent + // Haiku fallback fires. Matches the deliberate OpenRouter-pricing-deferred + // posture from TODOS.md #2. + expect(spendCheck('openrouter:anthropic/claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(1.0, 5); + }); +}); diff --git a/test/model-config.serial.test.ts b/test/model-config.serial.test.ts index 5b462c5c3..94f28bc19 100644 --- a/test/model-config.serial.test.ts +++ b/test/model-config.serial.test.ts @@ -220,6 +220,19 @@ describe('resolveModel — v0.31.12 tier system', () => { expect(isAnthropicProvider('')).toBe(false); }); + test('v0.41.20.0: isAnthropicProvider classifies slash-form (subagent-guard fix)', () => { + // Pre-fix: 'anthropic/claude-sonnet-4-6' had no colon and didn't start + // with 'claude-' (started with 'anthropic') → returned false → silent + // subagent-guard bypass → fall back to TIER_DEFAULTS.subagent without + // honoring the user's explicit slash-form config. + expect(isAnthropicProvider('anthropic/claude-sonnet-4-6')).toBe(true); + expect(isAnthropicProvider('anthropic/claude-opus-4-7')).toBe(true); + // Non-Anthropic slash forms STILL return false (don't accidentally + // widen the guard). + expect(isAnthropicProvider('openai/gpt-5')).toBe(false); + expect(isAnthropicProvider('google/gemini-3-pro')).toBe(false); + }); + test('alias-chain conflict: forward + reverse for same id (Codex F6)', async () => { // Codex F6: if both forward and reverse aliases exist, depth cap (2) // prevents infinite loop. Canonicalization is deterministic — terminates diff --git a/test/model-id.test.ts b/test/model-id.test.ts new file mode 100644 index 000000000..37d9dd8f5 --- /dev/null +++ b/test/model-id.test.ts @@ -0,0 +1,140 @@ +/** + * v0.41.21.0 — splitProviderModelId centralizer contract. + * + * Pins every shape the helper must handle so future refactors of the 5 + * downstream consumers (anthropic-pricing, budget-tracker, cost-tracker, + * batch-projection, model-config) can't silently regress on the slash-prefix + * bug class. + * + * Sibling: `src/core/ai/model-resolver.ts:parseModelId` — the gateway-side + * resolver. Both accept the same input shapes post-v0.41.21.0; this helper + * is defensive (returns `{provider: null, model: 'bare'}` for bare names) + * and the gateway one throws (routing needs an explicit provider). + */ + +import { describe, test, expect } from 'bun:test'; +import { splitProviderModelId } from '../src/core/model-id.ts'; + +describe('splitProviderModelId', () => { + describe('happy paths', () => { + test('bare model id → no provider', () => { + expect(splitProviderModelId('claude-sonnet-4-6')).toEqual({ + provider: null, + model: 'claude-sonnet-4-6', + }); + }); + + test('colon-separated provider:model', () => { + expect(splitProviderModelId('anthropic:claude-sonnet-4-6')).toEqual({ + provider: 'anthropic', + model: 'claude-sonnet-4-6', + }); + }); + + test('slash-separated provider/model — THE BUG CLASS FIX', () => { + // Pre-fix: every site's inline split missed this shape, silently + // returning the whole string as the "model" and failing pricing lookups. + expect(splitProviderModelId('anthropic/claude-sonnet-4-6')).toEqual({ + provider: 'anthropic', + model: 'claude-sonnet-4-6', + }); + }); + + test('double-separator openrouter:anthropic/X — colon wins, tail as-is', () => { + // Per D2 architecture: do NOT recursively peel. Transport=openrouter; + // pricing-vendor-identity is intentionally deferred to TODO #2 (non- + // Anthropic pricing). Pricing lookups will miss on the slash-bearing + // tail and land in the caller's existing unknown-model path. + expect(splitProviderModelId('openrouter:anthropic/claude-sonnet-4.6')).toEqual({ + provider: 'openrouter', + model: 'anthropic/claude-sonnet-4.6', + }); + }); + + test('slash-separated openrouter form openai/gpt-5', () => { + expect(splitProviderModelId('openai/gpt-5')).toEqual({ + provider: 'openai', + model: 'gpt-5', + }); + }); + }); + + describe('defensive contract', () => { + test('null → {provider: null, model: ""}', () => { + expect(splitProviderModelId(null)).toEqual({ provider: null, model: '' }); + }); + + test('undefined → {provider: null, model: ""}', () => { + expect(splitProviderModelId(undefined)).toEqual({ provider: null, model: '' }); + }); + + test('empty string → {provider: null, model: ""}', () => { + expect(splitProviderModelId('')).toEqual({ provider: null, model: '' }); + }); + + test('whitespace-only → {provider: null, model: ""}', () => { + expect(splitProviderModelId(' ')).toEqual({ provider: null, model: '' }); + expect(splitProviderModelId('\t\n ')).toEqual({ provider: null, model: '' }); + }); + + test('leading/trailing whitespace is trimmed before split', () => { + expect(splitProviderModelId(' anthropic:claude-sonnet-4-6 ')).toEqual({ + provider: 'anthropic', + model: 'claude-sonnet-4-6', + }); + expect(splitProviderModelId(' anthropic/claude-sonnet-4-6 ')).toEqual({ + provider: 'anthropic', + model: 'claude-sonnet-4-6', + }); + }); + }); + + describe('edge inputs', () => { + test('leading separator ":foo" → provider is empty string, not null', () => { + // Distinguish "no separator present" (null provider) from "separator + // with empty left side" (empty-string provider). Empty-string provider + // is a malformed input but we preserve the distinction so downstream + // callers can detect it without re-parsing. + expect(splitProviderModelId(':claude-foo')).toEqual({ + provider: '', + model: 'claude-foo', + }); + }); + + test('leading slash "/foo" → provider is empty string', () => { + expect(splitProviderModelId('/claude-foo')).toEqual({ + provider: '', + model: 'claude-foo', + }); + }); + + test('trailing separator "anthropic:" → model is empty string', () => { + expect(splitProviderModelId('anthropic:')).toEqual({ + provider: 'anthropic', + model: '', + }); + }); + + test('only ":" → empty provider AND empty model', () => { + expect(splitProviderModelId(':')).toEqual({ + provider: '', + model: '', + }); + }); + + test('only "/" → empty provider AND empty model', () => { + expect(splitProviderModelId('/')).toEqual({ + provider: '', + model: '', + }); + }); + + test('mixed-case provider is preserved (no normalization)', () => { + // Callers that care (e.g. isAnthropicProvider) lowercase themselves. + expect(splitProviderModelId('Anthropic:claude-foo')).toEqual({ + provider: 'Anthropic', + model: 'claude-foo', + }); + }); + }); +});