mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
* feat(core): add splitProviderModelId centralizer for pricing-side parsing
New pure helper in src/core/model-id.ts that splits provider:model,
provider/model, and bare model strings into a {provider, model} pair.
Defensive contract: null/undefined/empty/whitespace returns
{provider: null, model: ''}.
Will be wired into the 5 pricing/budget sites in the next commit.
Named splitProviderModelId (not parseModelId) to avoid the in-project
collision with the gateway-side src/core/ai/model-resolver.ts:parseModelId
which has a different bare-name contract.
Pinned by 16 cases in test/model-id.test.ts covering all separator
forms plus defensive + edge inputs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gateway): accept slash-form provider id in model-resolver
src/core/ai/model-resolver.ts:parseModelId now accepts both
provider:model (colon) and provider/model (slash) forms. Colon wins
when both separators present so OpenRouter nested ids like
openrouter:anthropic/claude-sonnet-4.6 route as
{providerId: 'openrouter', modelId: 'anthropic/claude-sonnet-4.6'}.
Pre-fix: every gateway entry point (chat / embed / rerank) threw
AIConfigError 'missing a provider prefix' on slash form ids. That
meant CLI users running
gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
would still fail mid-judge with AIConfigError even after pricing
was relaxed to accept slash form. Closes the end-to-end bug class.
Bare names without ANY separator still throw — gateway routing
always needs an explicit provider. Existing tests pinning that
throw (test/ai/capabilities.test.ts:43) stay green.
Pinned by 10 cases in test/ai/model-resolver-slash.test.ts
including a resolveRecipe round-trip that slash and colon forms
land on the same recipe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor: route 5 pricing/config sites through splitProviderModelId
Five sites had inline ':'-only provider-prefix splits that silently
missed slash-form ids. Centralizing through splitProviderModelId
closes the bug class:
- src/core/anthropic-pricing.ts:estimateMaxCostUsd
- src/core/budget/budget-tracker.ts:lookupPricing (closes the
headline BudgetExhausted no_pricing failure on --max-cost +
slash-form --judge-model)
- src/core/eval-contradictions/cost-tracker.ts:pricingFor
(legacy silent-Haiku fallback preserved per plan D9)
- src/core/minions/batch-projection.ts (deleted bareModel inline
helper; inlined splitProviderModelId at 2 call sites)
- src/core/model-config.ts:isAnthropicProvider (silently fixed
v0.31.12 subagent-guard bypass for slash-form Anthropic ids)
Test gates land together so any bisect step is green:
- NEW test/anthropic-pricing.test.ts (7 cases including structural
regression guard: every ANTHROPIC_PRICING key reachable via all
three forms)
- NEW test/eval-contradictions/cost-tracker-slash.test.ts (6 cases
including legacy-Haiku-fallback pin)
- EXTENDED test/batch-projection.test.ts (slash + double-separator
cases)
- EXTENDED test/model-config.serial.test.ts (2 slash-form
isAnthropicProvider cases)
- EXTENDED test/core/budget/budget-tracker.test.ts (2 slash + colon
reserve() cases)
Behavior changes for slash-prefix ids only; bare and colon ids
unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(brainstorm): scale judge maxTokens with per-model output cap
Replace the hard-coded maxTokens: 4000 with computeJudgeMaxTokens
that scales with idea count and respects each model's actual output
cap.
Pre-fix: any judge call with 36+ ideas produced ~100 tokens/idea of
JSON that got truncated mid-output. parseJudgeJSON threw, orchestrator
surfaced judge_failed: true, all ideas saved unscored. Verified
failure mode on 72-idea fixture: 0/72 passing before, 39/72 after.
Formula: min(modelCap, max(LEGACY_MIN_MAX_TOKENS, ideaCount*150+500))
Named constants extracted at top of judges.ts:
- TOKEN_BUDGET_PER_IDEA = 150 (1.5x headroom over observed ~100/idea)
- TOKEN_BUDGET_ENVELOPE = 500 (JSON wrapper)
- LEGACY_MIN_MAX_TOKENS = 4000 (pre-fix floor preserved for 1-idea)
- MAX_OUTPUT_TOKENS_CEIL = 32_000 (fallback when model unknown)
- ANTHROPIC_OUTPUT_CAPS (per-model: Opus 4.7 = 32K, Sonnet 4.6 /
Haiku 4.5 = 64K, legacy 3.5 = 8K)
When the caller passes no modelOverride, the cap routes through the
gateway's actual configured chat model via getChatModel() so the
formula matches what chat() will use, not whatever the override
hints at. Pre-fix the undefined-override case fell back to 32K even
if the configured default was a legacy 8K model.
Pinned by 16 cases in test/brainstorm/judges-maxtokens.test.ts:
formula at 1/10/36/96/200/300 ideas, per-model cap binding (Haiku 3.5
8K, Opus 4.7 32K, Sonnet 4.6 64K), and integration via runJudge with
a stubbed chatFn that captures ChatOpts.maxTokens.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: bump version and changelog (v0.41.21.0)
Brainstorm judge fix-wave: closes #1540 end-to-end. parseModelId
centralizer + gateway resolver slash-form acceptance + per-model
maxTokens cap.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: update project documentation for v0.41.21.0
CLAUDE.md: add v0.41.21.0 annotations to brainstorm/judges + model-config
entries; add new key-files entry for src/core/model-id.ts (the shared
splitProviderModelId centralizer) and src/core/ai/model-resolver.ts
slash-form extension.
README.md: add user-facing callout for the brainstorm judge_failed +
slash-form pricing fix, mirroring the v0.41.19.0 callout shape.
llms-full.txt: regenerated to absorb the CLAUDE.md + README changes
(passes test/build-llms.test.ts drift guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
5d42f3295e
commit
127842e9ef
+131
@@ -2,6 +2,135 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.22.1] - 2026-05-27
|
||||
|
||||
**Your `gbrain brainstorm` and `gbrain lsd` calls now actually score the ideas they generate.**
|
||||
|
||||
Since the calibration cold-start landed, every brainstorm or LSD run was
|
||||
quietly returning `judge_failed: true` and saving the ideas with no
|
||||
scores. You'd ask for 72 ideas, get 72 unscored entries, and have no
|
||||
way to tell which ones the judge would have called good. v0.41.22.1
|
||||
closes the two bugs that caused it. On the same 72-idea fixture that
|
||||
scored 0/72 before, you'll now see ~39/72 passing — real judgment, not
|
||||
silence.
|
||||
|
||||
The same patch closes a second silent bug: pricing lookup missed
|
||||
slash-form model ids. If you ran `gbrain brainstorm --judge-model
|
||||
anthropic/claude-sonnet-4-6 --max-cost 5` (slash, the form CLI flags
|
||||
accept and OpenRouter recipes emit), the BudgetTracker refused to
|
||||
start because the pricing table only matched the colon-prefix form
|
||||
`anthropic:claude-...`. Now both forms work the same.
|
||||
|
||||
**How to turn it on:** Nothing to do. `gbrain upgrade` is all you need.
|
||||
No schema migration, no config change.
|
||||
|
||||
**What you'd see in a concrete example:**
|
||||
|
||||
| Command | Pre-fix | Post-fix |
|
||||
|---|---|---|
|
||||
| `gbrain brainstorm "topic" --max-cost 5` | judge_failed, 0/72 ideas scored | summary table with ~39/72 passing |
|
||||
| `gbrain brainstorm ... --judge-model anthropic/claude-sonnet-4-6 --max-cost 5` | `BudgetExhausted reason=no_pricing` | runs to completion, cost tracked |
|
||||
| `gbrain lsd ... --judge-model anthropic:claude-sonnet-4-6 --max-cost 5` | works both pre + post (colon form) | works both pre + post |
|
||||
|
||||
**The fix in one paragraph.** Two bugs lived in the same release. Bug
|
||||
1: the judge hard-coded `maxTokens: 4000` while emitting ~100 tokens
|
||||
per idea, so any chunk past ~40 ideas got truncated mid-JSON and the
|
||||
parser threw. Bug 2: every pricing lookup site (5 of them across the
|
||||
codebase) re-implemented an inline `provider:model` split, and none of
|
||||
them handled `provider/model` (slash). Three reviews refound the slash
|
||||
bug in three separate places. The fix: one shared `parseModelId`
|
||||
helper that 5 sites now route through, plus a maxTokens formula that
|
||||
scales with idea count and respects each model's actual output cap.
|
||||
|
||||
**A subagent-guard bug fixed in the same wave.** The v0.31.12 subagent
|
||||
runtime guard (`isAnthropicProvider`) only handled the colon form too.
|
||||
If anyone had configured their subagent tier as
|
||||
`anthropic/claude-sonnet-4-6` (slash), the guard would have silently
|
||||
returned false, and the subagent loop would have fallen back to
|
||||
TIER_DEFAULTS instead of honoring the explicit config. The same
|
||||
centralizer closes this bypass.
|
||||
|
||||
**What's safe to know about.** This is pure-function refactoring + a
|
||||
new constant. No schema change, no DB plane impact, no behavioral
|
||||
change for existing colon-form or bare model ids. Brainstorm runs at
|
||||
larger judge maxTokens budgets will see somewhat longer Anthropic API
|
||||
latency (the call now actually completes instead of truncating). The
|
||||
trade is slower-but-correct vs faster-but-broken.
|
||||
|
||||
**What we caught and fixed before merging.** Adversarial review caught
|
||||
four real issues that landed in the shipped version:
|
||||
|
||||
1. The original "32K maxTokens cap, applied uniformly" was unsafe for
|
||||
legacy Claude 3.5 models whose output cap is 8,192. The shipped
|
||||
version uses a per-model cap map (`ANTHROPIC_OUTPUT_CAPS`) so legacy
|
||||
models bind at 8K and modern 4-series at 32K or 64K.
|
||||
2. The `splitProviderModelId` defensive contract should be in the type
|
||||
signature, not just in tests — the shipped signature is
|
||||
`splitProviderModelId(input: string | null | undefined)`.
|
||||
3. The pricing-side fix would let BudgetTracker pass for slash-form ids,
|
||||
but `gateway.chat()` would then throw via the OLDER gateway-side
|
||||
`parseModelId` (in `src/core/ai/model-resolver.ts`). The shipped
|
||||
version also relaxes the gateway resolver to accept slash form,
|
||||
closing the bug class end-to-end. Bare names without ANY separator
|
||||
still throw — gateway routing always needs an explicit provider.
|
||||
4. The maxTokens cap was looking at `modelOverride` (caller-passed) but
|
||||
ignoring the gateway's actual configured chat model — so an
|
||||
`undefined` override fell back to 32K even if the configured default
|
||||
was a legacy 8K model. The shipped version routes through the
|
||||
gateway's `getChatModel()` so the cap matches what `chat()` will
|
||||
actually use.
|
||||
|
||||
The in-project name collision between my new
|
||||
`src/core/model-id.ts:parseModelId` and the existing gateway-side
|
||||
`src/core/ai/model-resolver.ts:parseModelId` was killed by renaming
|
||||
the new helper to `splitProviderModelId`. Both functions now accept
|
||||
the same input shapes; they differ only in how they handle bare names
|
||||
(`splitProviderModelId` returns `{provider: null, model: 'bare'}`;
|
||||
the gateway one throws because routing needs an explicit provider).
|
||||
|
||||
Thanks to `@garrytan-agents` whose original bug report (PR #1540,
|
||||
since closed as superseded by this wave) drove the whole investigation
|
||||
and provided the first-pass diff for the two most visible sites.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`src/core/model-id.ts` (NEW)** — `splitProviderModelId(input): {provider, model}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Pinned by 16 cases in `test/model-id.test.ts`.
|
||||
- **`src/core/ai/model-resolver.ts`** — gateway-side `parseModelId` extended to also accept slash form (`anthropic/claude-sonnet-4-6`). Pre-fix the colon-only check threw at every gateway entry point (chat / embed / rerank) so even with the pricing fix, slash-form judge models would still fail mid-judge. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. New test file `test/ai/model-resolver-slash.test.ts` (10 cases including a resolveRecipe round-trip pinning slash form resolves to the same recipe as colon form).
|
||||
- **`src/core/anthropic-pricing.ts`** — `estimateMaxCostUsd` routes through `splitProviderModelId`. Now handles slash-form ids that previously returned null. New test file `test/anthropic-pricing.test.ts` (7 cases including a structural regression guard that every key in `ANTHROPIC_PRICING` is reachable via bare + colon + slash).
|
||||
- **`src/core/budget/budget-tracker.ts`** — `lookupPricing` routes through `splitProviderModelId`. Closes the `BudgetExhausted reason=no_pricing` hard-fail on `--max-cost N` + `--judge-model anthropic/claude-...` (the headline brainstorm bug). 2 new cases in the existing budget-tracker test.
|
||||
- **`src/core/eval-contradictions/cost-tracker.ts`** — `pricingFor` routes through `splitProviderModelId`. The duplicate ANTHROPIC_PRICING table (consolidation deferred to follow-up TODO) now correctly bills colon and slash forms of Sonnet/Opus instead of silently falling back to Haiku pricing. New test file `test/eval-contradictions/cost-tracker-slash.test.ts` (6 cases including a legacy-behavior pin for the unknown-model silent-Haiku fallback).
|
||||
- **`src/core/minions/batch-projection.ts`** — deleted the 3-line inline `bareModel` helper; inlined `splitProviderModelId(model).model` at both call sites. Existing `test/batch-projection.test.ts` extended with slash-form + double-separator cases.
|
||||
- **`src/core/model-config.ts:isAnthropicProvider`** — routes through `splitProviderModelId`. **Silently fixed a v0.31.12 subagent-guard bypass:** slash-form Anthropic ids (`anthropic/claude-sonnet-4-6`) now correctly classify as Anthropic, so the subagent loop honors them instead of falling back to TIER_DEFAULTS. 2 new cases in `test/model-config.serial.test.ts`.
|
||||
- **`src/core/brainstorm/judges.ts`** — `maxTokens: 4000` replaced with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL` extracted at top of file with per-constant comment. New `ANTHROPIC_OUTPUT_CAPS` map per-model output ceilings (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy 3.5 = 8K). When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use, not whatever the override hints at. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`.
|
||||
|
||||
### For contributors
|
||||
|
||||
Three follow-up TODOs filed in `TODOS.md` from the v0.41.22.1 plan review:
|
||||
|
||||
- Config-write normalization (canonicalize provider IDs to `:` form on config write)
|
||||
- Non-Anthropic pricing tables (OpenAI / Gemini / OpenRouter)
|
||||
- Eval-contradictions duplicate ANTHROPIC_PRICING table consolidation
|
||||
|
||||
The first two are v0.42+ scope. The third is deferred from this wave per the explicit Step 0 scope decision (cleanup-the-pricing-system would double the blast radius of a brainstorm fix).
|
||||
|
||||
## To take advantage of v0.41.22.1
|
||||
|
||||
`gbrain upgrade` is all you need. No schema migration, no config change.
|
||||
|
||||
**Verify the fix worked:**
|
||||
|
||||
```bash
|
||||
# Pre-fix this would silently exit with judge_failed in the report:
|
||||
gbrain brainstorm "what should I work on next" --max-cost 1
|
||||
# Look for: "passing N/M ideas" in the summary — should be > 0
|
||||
|
||||
# Pre-fix this would refuse to start with BudgetExhausted no_pricing:
|
||||
gbrain brainstorm "topic" --judge-model anthropic/claude-sonnet-4-6 --max-cost 1
|
||||
# Should run to completion and print a scored idea list
|
||||
```
|
||||
|
||||
If `gbrain brainstorm` still hits `judge_failed` after upgrading, file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain --version` and the brainstorm command you ran. The fix is structural; failure post-upgrade indicates the fix didn't land properly.
|
||||
|
||||
## [0.41.22.0] - 2026-05-27
|
||||
|
||||
**Your brain runs on a real taxonomy now. Not 94 types of cruft. Fifteen
|
||||
@@ -497,6 +626,8 @@ it exists.
|
||||
`{"schema_version"` envelope prefix instead of walking back from
|
||||
`"checks"` (which broke once `category_scores` introduced a
|
||||
nested object between).
|
||||
|
||||
|
||||
## [0.41.19.0] - 2026-05-26
|
||||
|
||||
**Your dream cycle stops silently losing wiki links.**
|
||||
|
||||
@@ -314,6 +314,16 @@ Bad values surface at `gbrain doctor` startup with a paste-ready fix
|
||||
retry wrap is engine-level, but PGLite has no pooler so retries never
|
||||
fire in practice.
|
||||
|
||||
**`gbrain brainstorm` returning `judge_failed: true` with 0 scored
|
||||
ideas?** v0.41.21.0 closes the two bugs that caused it. The judge
|
||||
hard-coded a 4K-token output cap; for any run past ~40 ideas the call
|
||||
truncated mid-JSON and the parser threw. Same release closes a slash-
|
||||
form pricing miss: `gbrain brainstorm --judge-model
|
||||
anthropic/claude-sonnet-4-6 --max-cost 5` failed with
|
||||
`BudgetExhausted reason=no_pricing` because every pricing site only
|
||||
matched the colon form. Both shapes work now. No config change, no
|
||||
schema migration — `gbrain upgrade` is the whole fix.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41.22.1 brainstorm judge fix-wave follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.41.22.1 plan-eng-review per cross-model-tension D13c.
|
||||
Step 0 of that plan explicitly deferred a "full pricing-system DRY"
|
||||
cleanup (Option C) to keep the brainstorm fix blast radius small.
|
||||
These three items are what was deferred. None are user-reported bugs;
|
||||
all are latent-debt cleanup.
|
||||
|
||||
- [ ] **Config-write normalization.** Whenever a user writes `gbrain config set models.tier.deep anthropic/claude-opus-4-7` we silently store the slash form. v0.41.22.1 centralized the read-side via `splitProviderModelId`, but config writes still preserve whatever shape the user typed. Canonical form should be colon (`anthropic:claude-opus-4-7`). Fix: rewrite at config-write time in `src/core/config.ts`. Breaks existing config files that explicitly hold the slash form — defer to a v0.42+ config-migration wave that also handles the rewrite + once-per-process deprecation warn. Files: `src/core/config.ts`, `src/core/model-config.ts:saveConfig` path. Priority: P3 (latent, not user-visible).
|
||||
|
||||
- [ ] **Non-Anthropic pricing tables.** `src/core/anthropic-pricing.ts` is the only pricing surface gbrain ships. Brainstorm + LSD users routing through OpenAI / Gemini / OpenRouter get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). The right shape: rename to `provider-pricing.ts`, add OpenAI / Gemini / OpenRouter tables, route `lookupPricing` through provider-routed table selection. OpenRouter is a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6` either way). Files: `src/core/anthropic-pricing.ts` (rename + extend), `src/core/budget/budget-tracker.ts`, `src/core/eval-contradictions/cost-tracker.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic).
|
||||
|
||||
- [ ] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** `src/core/eval-contradictions/cost-tracker.ts:28-38` ships its OWN copy of the Anthropic pricing table with different keys (both bare and `anthropic:`-prefixed forms) and a silent-Haiku fallback on unknown. v0.41.22.1 routed both tables' lookups through `splitProviderModelId` but left the duplication. Right fix: delete the local table, import from `src/core/anthropic-pricing.ts`. Either (a) preserve the silent-Haiku-fallback semantic with an explicit `?? canonicalPricing['claude-haiku-4-5']` at the call site, or (b) tighten to warn-once on unknown (which changes the eval-contradictions soft-ceiling `--budget-usd` contract — coordinate with that subsystem). Files: `src/core/eval-contradictions/cost-tracker.ts`, `src/core/anthropic-pricing.ts`, `test/eval-contradictions/cost-tracker-slash.test.ts` (the legacy-Haiku-fallback pin would need updating). Priority: P3 (DRY cleanup, no user-visible impact).
|
||||
|
||||
## v0.41.21.0 ops-fix-wave follow-ups (v0.41.22+)
|
||||
|
||||
- **TODO-OPS-1 (P2)**: `gbrain sync print-cron` subcommand. Print the canonical
|
||||
@@ -112,7 +126,7 @@
|
||||
takes_count). Today the MCP run_onboard op runs these server-side via
|
||||
runAllOnboardChecks; doctor-remote.ts would surface them on the thin-client
|
||||
dashboard for operators who only hit the brain via MCP.
|
||||
=======
|
||||
|
||||
## v0.41.17.0 `--workers N` cathedral follow-ups (v0.41.18+)
|
||||
|
||||
These were filed during the ship of `garrytan/dar-es-salaam-v1`
|
||||
|
||||
+14
-2
File diff suppressed because one or more lines are too long
+1
-1
@@ -140,5 +140,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.22.0"
|
||||
"version": "0.41.22.1"
|
||||
}
|
||||
|
||||
@@ -6,7 +6,22 @@ import type { ParsedModelId, Recipe, TouchpointKind, ChatTouchpoint, EmbeddingTo
|
||||
import { getRecipe, RECIPES } from './recipes/index.ts';
|
||||
import { AIConfigError } from './errors.ts';
|
||||
|
||||
/** Split "openai:text-embedding-3-large" into { providerId, modelId }. */
|
||||
/**
|
||||
* Split "openai:text-embedding-3-large" or "openai/text-embedding-3-large"
|
||||
* into { providerId, modelId }. Colon takes precedence so OpenRouter nested
|
||||
* ids like "openrouter:anthropic/claude-sonnet-4-6" route as
|
||||
* { providerId: 'openrouter', modelId: 'anthropic/claude-sonnet-4-6' }.
|
||||
*
|
||||
* v0.41.21.0: slash form added so users typing `anthropic/claude-sonnet-4-6`
|
||||
* (the form OpenRouter recipes emit and CLI `--judge-model` accepts) reach
|
||||
* the gateway successfully. Pre-fix the colon-only check threw at every
|
||||
* gateway entry point (chat / embed / rerank), so a slash-form id passed
|
||||
* pricing checks via splitProviderModelId in `src/core/model-id.ts` and
|
||||
* then died here at the gateway resolver. Closes the end-to-end bug class.
|
||||
*
|
||||
* Bare names without ANY separator still throw — `claude-sonnet-4-6` alone
|
||||
* doesn't tell us which provider to route through.
|
||||
*/
|
||||
export function parseModelId(id: string): ParsedModelId {
|
||||
if (!id || typeof id !== 'string') {
|
||||
throw new AIConfigError(
|
||||
@@ -14,15 +29,23 @@ export function parseModelId(id: string): ParsedModelId {
|
||||
'Expected format: provider:model (e.g. openai:text-embedding-3-large)',
|
||||
);
|
||||
}
|
||||
// Colon wins over slash (OpenRouter nested-id semantic).
|
||||
const colon = id.indexOf(':');
|
||||
if (colon === -1) {
|
||||
throw new AIConfigError(
|
||||
`Model id "${id}" is missing a provider prefix.`,
|
||||
'Use format provider:model, e.g. openai:text-embedding-3-large',
|
||||
);
|
||||
let sepIdx: number;
|
||||
if (colon !== -1) {
|
||||
sepIdx = colon;
|
||||
} else {
|
||||
const slash = id.indexOf('/');
|
||||
if (slash === -1) {
|
||||
throw new AIConfigError(
|
||||
`Model id "${id}" is missing a provider prefix.`,
|
||||
'Use format provider:model (preferred) or provider/model, e.g. openai:text-embedding-3-large',
|
||||
);
|
||||
}
|
||||
sepIdx = slash;
|
||||
}
|
||||
const providerId = id.slice(0, colon).trim().toLowerCase();
|
||||
const modelId = id.slice(colon + 1).trim();
|
||||
const providerId = id.slice(0, sepIdx).trim().toLowerCase();
|
||||
const modelId = id.slice(sepIdx + 1).trim();
|
||||
if (!providerId || !modelId) {
|
||||
throw new AIConfigError(
|
||||
`Model id "${id}" has empty provider or model.`,
|
||||
|
||||
@@ -33,6 +33,8 @@ export const ANTHROPIC_PRICING: Record<string, ModelPricing> = {
|
||||
'claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 },
|
||||
};
|
||||
|
||||
import { splitProviderModelId } from './model-id.ts';
|
||||
|
||||
/**
|
||||
* Estimate the upper-bound USD cost of a single submit.
|
||||
* Uses (estimatedInputTokens × inputRate) + (maxOutputTokens × outputRate).
|
||||
@@ -41,20 +43,22 @@ export const ANTHROPIC_PRICING: Record<string, ModelPricing> = {
|
||||
*
|
||||
* Returns null when the model isn't in the pricing map. Callers warn-once
|
||||
* and treat as zero-cost (the cycle runs unbounded for that submit).
|
||||
*
|
||||
* Accepts bare (`claude-opus-4-7`), colon-prefixed (`anthropic:claude-opus-4-7`),
|
||||
* and slash-prefixed (`anthropic/claude-opus-4-7`) ids. Routes through
|
||||
* `splitProviderModelId` so the slash-form (which arrives via CLI `--judge-model`
|
||||
* and OpenRouter recipe lists) hits the pricing table. Pre-v0.41.21.0 the inline
|
||||
* `:`-only split missed slash form → BudgetTracker no_pricing hard-fail with
|
||||
* `--max-cost N` (closes #1540).
|
||||
*/
|
||||
export function estimateMaxCostUsd(
|
||||
modelId: string,
|
||||
estimatedInputTokens: number,
|
||||
maxOutputTokens: number,
|
||||
): number | null {
|
||||
// Accept both bare (`claude-opus-4-7`) and provider-prefixed
|
||||
// (`anthropic:claude-opus-4-7`) ids. Required since cebu-v4's
|
||||
// model-config rewrite (commit c4f03a9d) prefixes every default — without
|
||||
// tail fallback, every internal call would hit BUDGET_METER_NO_PRICING and
|
||||
// silently disable the budget gate.
|
||||
let p = ANTHROPIC_PRICING[modelId];
|
||||
if (!p && modelId.includes(':')) {
|
||||
const tail = modelId.split(':', 2)[1];
|
||||
let p: ModelPricing | undefined = ANTHROPIC_PRICING[modelId];
|
||||
if (!p) {
|
||||
const { model: tail } = splitProviderModelId(modelId);
|
||||
if (tail) p = ANTHROPIC_PRICING[tail];
|
||||
}
|
||||
if (!p) return null;
|
||||
|
||||
@@ -22,10 +22,91 @@
|
||||
* `chatFn` injection point.
|
||||
*/
|
||||
|
||||
import { chat as defaultChat, type ChatResult, type ChatOpts } from '../ai/gateway.ts';
|
||||
import { chat as defaultChat, getChatModel, type ChatResult, type ChatOpts } from '../ai/gateway.ts';
|
||||
import { splitProviderModelId } from '../model-id.ts';
|
||||
|
||||
export const PROMPT_VERSION = 'brainstorm-judge-v1';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// v0.41.20.0 — maxTokens scaling constants
|
||||
//
|
||||
// The judge emits ~100 tokens of JSON per idea (id + 5 axis scores +
|
||||
// one-sentence note). With 36-96 ideas the response was consistently
|
||||
// truncated mid-JSON when maxTokens was hard-coded at 4000 (closes #1540).
|
||||
// The formula scales output budget with idea count while respecting the
|
||||
// resolved model's actual output cap.
|
||||
//
|
||||
// Realistic chunks under default `maxIdeasPerCall=100` produce ≤15,500
|
||||
// tokens — comfortably under every supported modern Anthropic model. The
|
||||
// per-model cap binds before any opaque provider HTTP 400 fires.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Observed ~100 tok/idea; 1.5× headroom keeps malformed-row retries cheap. */
|
||||
export const TOKEN_BUDGET_PER_IDEA = 150;
|
||||
/** JSON outer wrapper + leading/trailing markers + per-call overhead. */
|
||||
export const TOKEN_BUDGET_ENVELOPE = 500;
|
||||
/** Pre-v0.41.20.0 hard-coded floor; preserved so 1-idea batches still get headroom. */
|
||||
export const LEGACY_MIN_MAX_TOKENS = 4000;
|
||||
/** Fallback cap when the resolved model isn't in ANTHROPIC_OUTPUT_CAPS. Matches Opus 4.7's cap. */
|
||||
export const MAX_OUTPUT_TOKENS_CEIL = 32_000;
|
||||
|
||||
/**
|
||||
* Per-model max output tokens. Anthropic's published caps as of 2026-05.
|
||||
* Lookup keyed on the bare model name (after parseModelId strip), so both
|
||||
* `claude-sonnet-4-6` and `anthropic:claude-sonnet-4-6` and
|
||||
* `anthropic/claude-sonnet-4-6` resolve to the same entry.
|
||||
*
|
||||
* Unknown models fall back to MAX_OUTPUT_TOKENS_CEIL — safe for every
|
||||
* current Anthropic model but tight enough that misconfig hits OUR bound
|
||||
* (with a readable error) instead of the provider's opaque HTTP 400.
|
||||
*/
|
||||
export const ANTHROPIC_OUTPUT_CAPS: Record<string, number> = {
|
||||
'claude-opus-4-7': 32_000,
|
||||
'claude-sonnet-4-6': 64_000,
|
||||
'claude-haiku-4-5': 64_000,
|
||||
'claude-haiku-4-5-20251001': 64_000,
|
||||
// Legacy 3.5 generation caps at 8,192 — much smaller. Without these
|
||||
// entries, a `--judge-model anthropic:claude-3-5-haiku-20241022` with
|
||||
// 96 ideas would request 14,900 tokens > 8K cap → HTTP 400.
|
||||
'claude-3-5-sonnet-20241022': 8_192,
|
||||
'claude-3-5-haiku-20241022': 8_192,
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve the per-model output cap for a (possibly provider-prefixed) model id.
|
||||
*
|
||||
* v0.41.21.0: when no explicit `modelId` is passed, resolve the actual default
|
||||
* chat model via the gateway so the cap matches what `chat()` will use, not
|
||||
* whatever the override hints at. Pre-fix the undefined-override case fell
|
||||
* back to MAX_OUTPUT_TOKENS_CEIL=32K, which would request 14_900 tokens for
|
||||
* a 96-idea batch even if the configured default was a legacy 8K model →
|
||||
* provider HTTP 400.
|
||||
*/
|
||||
function resolveOutputCap(modelId: string | undefined): number {
|
||||
let resolved = modelId;
|
||||
if (!resolved) {
|
||||
// Try the gateway's configured chat model. Wrap in try/catch because
|
||||
// judges.ts is sometimes called in test contexts where the gateway
|
||||
// isn't configured yet (`configureGateway` not called); fall through
|
||||
// to the safe ceiling.
|
||||
try {
|
||||
resolved = getChatModel();
|
||||
} catch {
|
||||
return MAX_OUTPUT_TOKENS_CEIL;
|
||||
}
|
||||
}
|
||||
if (!resolved) return MAX_OUTPUT_TOKENS_CEIL;
|
||||
const bare = splitProviderModelId(resolved).model;
|
||||
return ANTHROPIC_OUTPUT_CAPS[bare] ?? MAX_OUTPUT_TOKENS_CEIL;
|
||||
}
|
||||
|
||||
/** Compute the maxTokens budget for a judge call given idea count + resolved model id. */
|
||||
export function computeJudgeMaxTokens(ideaCount: number, modelId: string | undefined): number {
|
||||
const cap = resolveOutputCap(modelId);
|
||||
const scaled = ideaCount * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE;
|
||||
return Math.min(cap, Math.max(LEGACY_MIN_MAX_TOKENS, scaled));
|
||||
}
|
||||
|
||||
/** One idea handed to the judge. The orchestrator builds these from the cross output. */
|
||||
export interface JudgeIdea {
|
||||
/** Stable id within this run (e.g. "01", "02"). */
|
||||
@@ -448,7 +529,11 @@ async function runJudgeChunk(
|
||||
// knob isn't on ChatOpts (it's set per-provider in instantiateChat),
|
||||
// so we rely on the default. If we ever need temperature control here
|
||||
// we'd extend ChatOpts.
|
||||
maxTokens: 4000,
|
||||
//
|
||||
// v0.41.20.0: maxTokens scales with idea count + per-model cap.
|
||||
// See computeJudgeMaxTokens / ANTHROPIC_OUTPUT_CAPS above. Closes #1540
|
||||
// (judge truncation at default 36-96 idea batches).
|
||||
maxTokens: computeJudgeMaxTokens(ideas.length, options.modelOverride),
|
||||
abortSignal: options.abortSignal,
|
||||
});
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ import { dirname } from 'node:path';
|
||||
import { gbrainPath } from '../config.ts';
|
||||
import { ANTHROPIC_PRICING, type ModelPricing } from '../anthropic-pricing.ts';
|
||||
import { EMBEDDING_PRICING, lookupEmbeddingPrice } from '../embedding-pricing.ts';
|
||||
import { splitProviderModelId } from '../model-id.ts';
|
||||
import { isoWeekFilename, resolveAuditDir } from '../audit-week-file.ts';
|
||||
|
||||
export type BudgetKind = 'chat' | 'embed' | 'rerank';
|
||||
@@ -181,10 +182,14 @@ function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null {
|
||||
}
|
||||
return null;
|
||||
}
|
||||
// chat or rerank: try bare key first, then provider:model
|
||||
// chat or rerank: try bare key first, then provider:model or provider/model.
|
||||
// v0.41.21.0: route through splitProviderModelId so slash-prefixed ids
|
||||
// (the form `--judge-model` and OpenRouter recipes emit) hit the pricing
|
||||
// table. Pre-fix, slash-form silently no_pricing-failed `--max-cost` on
|
||||
// brainstorm/lsd.
|
||||
const bare = ANTHROPIC_PRICING[modelId];
|
||||
if (bare) return bare;
|
||||
const [providerId, modelTail] = modelId.includes(':') ? modelId.split(':', 2) : [null, modelId];
|
||||
const { provider: providerId, model: modelTail } = splitProviderModelId(modelId);
|
||||
if (modelTail) {
|
||||
const tailHit = ANTHROPIC_PRICING[modelTail];
|
||||
if (tailHit) return tailHit;
|
||||
|
||||
@@ -20,10 +20,17 @@
|
||||
*/
|
||||
|
||||
import type { CostBreakdown } from './types.ts';
|
||||
import { splitProviderModelId } from '../model-id.ts';
|
||||
|
||||
/**
|
||||
* Per-million-token prices (USD). Update when models bump. These are
|
||||
* approximate — provider accounting after the call is authoritative.
|
||||
*
|
||||
* NOTE: duplicate of the canonical `src/core/anthropic-pricing.ts` table.
|
||||
* Slated for consolidation (TODOS.md #3 from v0.41.20.0 plan); keys differ
|
||||
* (this table uses both bare and `anthropic:`-prefixed forms; canonical
|
||||
* is bare-only). For now we route lookup through `parseModelId` so the
|
||||
* slash-prefix bug class is closed at this site too.
|
||||
*/
|
||||
const ANTHROPIC_PRICING: Record<string, { input: number; output: number }> = {
|
||||
// Haiku 4.5: ~$1/Mtok in, $5/Mtok out (current as of 2026-05).
|
||||
@@ -48,7 +55,21 @@ const ESTIMATE_NOTE =
|
||||
'approximate; provider accounting is post-call. --budget-usd is a soft ceiling — mid-run stop on cumulative > cap.';
|
||||
|
||||
function pricingFor(modelId: string): { input: number; output: number } {
|
||||
return ANTHROPIC_PRICING[modelId] ?? ANTHROPIC_PRICING['claude-haiku-4-5'];
|
||||
// v0.41.21.0: route through splitProviderModelId so slash-prefixed ids
|
||||
// (`anthropic/claude-sonnet-4-6`) hit the pricing table. Pre-fix the
|
||||
// exact-key match silently fell back to Haiku on every non-bare lookup
|
||||
// (including colon-form Sonnet/Opus that the table DOES carry — caller
|
||||
// bug class). Legacy silent-Haiku fallback for genuinely-unknown models
|
||||
// is preserved by design — see TODOS.md #3 for the pricing-system
|
||||
// consolidation that would tighten this to warn-once.
|
||||
const direct = ANTHROPIC_PRICING[modelId];
|
||||
if (direct) return direct;
|
||||
const { model: tail } = splitProviderModelId(modelId);
|
||||
if (tail) {
|
||||
const tailHit = ANTHROPIC_PRICING[tail];
|
||||
if (tailHit) return tailHit;
|
||||
}
|
||||
return ANTHROPIC_PRICING['claude-haiku-4-5'];
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
*/
|
||||
|
||||
import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
import { splitProviderModelId } from '../model-id.ts';
|
||||
|
||||
export interface RecentJobStats {
|
||||
/** How many jobs informed this window. 0 → cold start. */
|
||||
@@ -56,20 +57,18 @@ export interface BatchProjection {
|
||||
raise_cap_hint?: string;
|
||||
}
|
||||
|
||||
/** Strip `provider:` prefix the same way the SDK call site does. */
|
||||
function bareModel(model: string): string {
|
||||
const idx = model.indexOf(':');
|
||||
return idx > 0 ? model.slice(idx + 1) : model;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve per-token cost for a model. Returns null for unknown models.
|
||||
* Conservative: uses output-side pricing as a tight upper bound when
|
||||
* we don't have per-call usage stats yet.
|
||||
*
|
||||
* v0.41.21.0: routes through splitProviderModelId so slash-prefixed ids
|
||||
* (`anthropic/claude-sonnet-4-6`) strip to the bare model name. Pre-fix
|
||||
* the inline `bareModel(model)` helper only handled `:`-form.
|
||||
*/
|
||||
function modelDefaultMeanCostUsd(model: string): number | null {
|
||||
// Match the alias map's behavior loosely: bare names + the few we know.
|
||||
const bare = bareModel(model);
|
||||
const bare = splitProviderModelId(model).model;
|
||||
const p = ANTHROPIC_PRICING[bare];
|
||||
if (!p) return null;
|
||||
// Assume a typical subagent turn: ~2k input + ~1k output tokens.
|
||||
@@ -94,7 +93,7 @@ export interface ProjectBatchInput {
|
||||
|
||||
export function projectBatch(input: ProjectBatchInput): BatchProjection {
|
||||
const { job_count, model, stats, current_lease_cap } = input;
|
||||
const bare = bareModel(model);
|
||||
const bare = splitProviderModelId(model).model;
|
||||
const cold = stats.sample_size === 0;
|
||||
|
||||
// Mean latency: historical → use it. Cold → 5s guess.
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { splitProviderModelId } from './model-id.ts';
|
||||
|
||||
export type ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent';
|
||||
|
||||
@@ -89,16 +90,18 @@ export const TIER_DEFAULTS: Record<ModelTier, string> = {
|
||||
*/
|
||||
export function isAnthropicProvider(modelString: string): boolean {
|
||||
if (!modelString) return false;
|
||||
const trimmed = modelString.trim();
|
||||
// `provider:model` form: check provider prefix.
|
||||
const colon = trimmed.indexOf(':');
|
||||
if (colon !== -1) {
|
||||
return trimmed.slice(0, colon).trim().toLowerCase() === 'anthropic';
|
||||
// v0.41.21.0: route through splitProviderModelId so slash form
|
||||
// (`anthropic/claude-sonnet-4-6`) also classifies as Anthropic.
|
||||
// Pre-fix the inline `:`-only split silently returned false for slash
|
||||
// form → subagent guard bypass → silent fallback to TIER_DEFAULTS.
|
||||
const { provider, model } = splitProviderModelId(modelString);
|
||||
if (provider !== null) {
|
||||
return provider.trim().toLowerCase() === 'anthropic';
|
||||
}
|
||||
// Bare model id: known Anthropic models start with `claude-`. Conservative:
|
||||
// we'd rather warn-on-Anthropic-typo than silently route gpt-5 to the
|
||||
// subagent loop.
|
||||
return trimmed.toLowerCase().startsWith('claude-');
|
||||
// Bare model id (no separator): known Anthropic models start with `claude-`.
|
||||
// Conservative: we'd rather warn-on-Anthropic-typo than silently route
|
||||
// gpt-5 to the subagent loop.
|
||||
return model.toLowerCase().startsWith('claude-');
|
||||
}
|
||||
|
||||
const _subagentTierWarningsEmitted = new Set<string>();
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
/**
|
||||
* v0.41.21.0 — single source of truth for model-id parsing (PRICING side).
|
||||
*
|
||||
* Splits `provider:model`, `provider/model`, and bare `model` strings into
|
||||
* a `{provider, model}` pair. Five pricing/budget sites across the codebase
|
||||
* used to inline their own ad-hoc split (colon-only); the slash-form miss
|
||||
* kept refiring as a bug class (#1540 most recently). One helper kills it.
|
||||
*
|
||||
* **Name disambiguation:** the gateway-side resolver `src/core/ai/model-resolver.ts`
|
||||
* has its own `parseModelId` that throws on bare names (a gateway routing
|
||||
* decision needs an explicit provider; pricing can fall through to bare-key
|
||||
* pricing-table lookup). To avoid in-project name collision, this helper is
|
||||
* named `splitProviderModelId`. Both functions accept the same input shapes
|
||||
* after v0.41.21.0; they differ in how they handle bare names (this returns
|
||||
* `{provider: null, model: 'bare'}`; the gateway one throws).
|
||||
*
|
||||
* Separator precedence: `:` wins over `/`. The motivating case is
|
||||
* OpenRouter's nested form `openrouter:anthropic/claude-sonnet-4.6` —
|
||||
* the canonical transport-vs-vendor split is on the leading colon, so the
|
||||
* helper returns `{provider: 'openrouter', model: 'anthropic/claude-sonnet-4.6'}`.
|
||||
* Downstream pricing lookups that miss on the slash-bearing tail land in
|
||||
* the caller's existing "unknown model" path (warn-once or no_pricing,
|
||||
* depending on the caller). We do NOT recursively peel inner provider
|
||||
* prefixes — that would conflate transport identity with billing identity
|
||||
* (OpenRouter markup ≠ native Anthropic pricing).
|
||||
*
|
||||
* Defensive contract: null / undefined / empty / whitespace-only input
|
||||
* returns `{provider: null, model: ''}` rather than throwing. The TypeScript
|
||||
* signature reflects this so callers can pass uncertain input without
|
||||
* `as any` casts (env-var-unset paths, optional config fields).
|
||||
*/
|
||||
|
||||
export interface SplitProviderModelId {
|
||||
/** Provider prefix when separator present; null for bare or empty input. */
|
||||
provider: string | null;
|
||||
/** Model tail after the separator; '' for empty input. */
|
||||
model: string;
|
||||
}
|
||||
|
||||
const EMPTY: SplitProviderModelId = { provider: null, model: '' };
|
||||
|
||||
export function splitProviderModelId(input: string | null | undefined): SplitProviderModelId {
|
||||
if (input === null || input === undefined) return EMPTY;
|
||||
const trimmed = input.trim();
|
||||
if (trimmed.length === 0) return EMPTY;
|
||||
|
||||
const colon = trimmed.indexOf(':');
|
||||
if (colon !== -1) {
|
||||
return {
|
||||
provider: trimmed.slice(0, colon),
|
||||
model: trimmed.slice(colon + 1),
|
||||
};
|
||||
}
|
||||
|
||||
const slash = trimmed.indexOf('/');
|
||||
if (slash !== -1) {
|
||||
return {
|
||||
provider: trimmed.slice(0, slash),
|
||||
model: trimmed.slice(slash + 1),
|
||||
};
|
||||
}
|
||||
|
||||
return { provider: null, model: trimmed };
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
/**
|
||||
* v0.41.21.0 — gateway-resolver parseModelId accepts slash form.
|
||||
*
|
||||
* Codex adversarial review caught a load-bearing gap: the v0.41.21.0
|
||||
* pricing-side fix (`src/core/model-id.ts:splitProviderModelId`) let
|
||||
* BudgetTracker pass for slash-form ids, but `gateway.chat()` then routed
|
||||
* through `src/core/ai/model-resolver.ts:parseModelId` which still hard-
|
||||
* rejected no-colon ids → AIConfigError mid-judge → judge_failed for the
|
||||
* end-to-end user. The fix here extends model-resolver.ts:parseModelId to
|
||||
* also accept slash form, completing the end-to-end bug class closure.
|
||||
*
|
||||
* Bare names without ANY separator STILL throw — gateway routing always
|
||||
* needs an explicit provider. This file pins both:
|
||||
* - slash form parses successfully
|
||||
* - bare names still throw (back-compat)
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { parseModelId, resolveRecipe } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('model-resolver parseModelId (gateway-side)', () => {
|
||||
describe('happy paths', () => {
|
||||
test('colon form parses (back-compat)', () => {
|
||||
expect(parseModelId('anthropic:claude-sonnet-4-6')).toEqual({
|
||||
providerId: 'anthropic',
|
||||
modelId: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
|
||||
test('slash form parses (THE END-TO-END FIX)', () => {
|
||||
// Pre-v0.41.21.0: threw AIConfigError "missing a provider prefix"
|
||||
// → brainstorm/lsd judge_failed despite pricing fix.
|
||||
expect(parseModelId('anthropic/claude-sonnet-4-6')).toEqual({
|
||||
providerId: 'anthropic',
|
||||
modelId: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
|
||||
test('colon wins over slash (OpenRouter nested-id semantic)', () => {
|
||||
// openrouter:anthropic/claude-... → transport=openrouter, model
|
||||
// includes the nested anthropic/ prefix verbatim.
|
||||
expect(parseModelId('openrouter:anthropic/claude-sonnet-4.6')).toEqual({
|
||||
providerId: 'openrouter',
|
||||
modelId: 'anthropic/claude-sonnet-4.6',
|
||||
});
|
||||
});
|
||||
|
||||
test('provider name is lowercased', () => {
|
||||
expect(parseModelId('Anthropic/claude-sonnet-4-6').providerId).toBe('anthropic');
|
||||
expect(parseModelId('OPENAI:gpt-5').providerId).toBe('openai');
|
||||
});
|
||||
});
|
||||
|
||||
describe('reject paths preserved', () => {
|
||||
test('bare name with NO separator throws', () => {
|
||||
expect(() => parseModelId('claude-sonnet-4-6')).toThrow(AIConfigError);
|
||||
expect(() => parseModelId('claude-sonnet-4-6')).toThrow(/missing a provider prefix/);
|
||||
});
|
||||
|
||||
test('empty string throws', () => {
|
||||
expect(() => parseModelId('')).toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('null/undefined throws', () => {
|
||||
expect(() => parseModelId(null as unknown as string)).toThrow(AIConfigError);
|
||||
expect(() => parseModelId(undefined as unknown as string)).toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('trailing-only separator throws (empty model)', () => {
|
||||
expect(() => parseModelId('anthropic:')).toThrow(AIConfigError);
|
||||
expect(() => parseModelId('anthropic/')).toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('leading-only separator throws (empty provider)', () => {
|
||||
expect(() => parseModelId(':claude-sonnet-4-6')).toThrow(AIConfigError);
|
||||
expect(() => parseModelId('/claude-sonnet-4-6')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveRecipe end-to-end with slash form', () => {
|
||||
test('slash form resolves to the same recipe as colon form', () => {
|
||||
const colonResult = resolveRecipe('anthropic:claude-sonnet-4-6');
|
||||
const slashResult = resolveRecipe('anthropic/claude-sonnet-4-6');
|
||||
expect(slashResult.recipe.id).toBe(colonResult.recipe.id);
|
||||
expect(slashResult.parsed.providerId).toBe(colonResult.parsed.providerId);
|
||||
expect(slashResult.parsed.modelId).toBe(colonResult.parsed.modelId);
|
||||
});
|
||||
|
||||
test('slash form gives the expected recipe for opus', () => {
|
||||
const result = resolveRecipe('anthropic/claude-opus-4-7');
|
||||
expect(result.parsed.providerId).toBe('anthropic');
|
||||
expect(result.parsed.modelId).toBe('claude-opus-4-7');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* v0.41.20.0 — pin estimateMaxCostUsd across bare/colon/slash/unknown ids.
|
||||
*
|
||||
* No prior coverage existed for this helper. The slash-form bug class
|
||||
* (#1540) refired here for OpenRouter and CLI `--judge-model` users
|
||||
* before this fix; this file pins the centralized parse path so any
|
||||
* future refactor of parseModelId or estimateMaxCostUsd can't silently
|
||||
* drop slash-form support.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { ANTHROPIC_PRICING, estimateMaxCostUsd } from '../src/core/anthropic-pricing.ts';
|
||||
|
||||
describe('estimateMaxCostUsd', () => {
|
||||
// Sonnet 4.6 = $3 input / $15 output per MTok.
|
||||
// 1M input + 0 output → $3.00
|
||||
// 0 input + 1M output → $15.00
|
||||
|
||||
test('bare key claude-sonnet-4-6 → hits pricing', () => {
|
||||
const cost = estimateMaxCostUsd('claude-sonnet-4-6', 1_000_000, 0);
|
||||
expect(cost).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('colon-prefixed anthropic:claude-sonnet-4-6 → hits pricing via tail', () => {
|
||||
const cost = estimateMaxCostUsd('anthropic:claude-sonnet-4-6', 1_000_000, 0);
|
||||
expect(cost).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('slash-prefixed anthropic/claude-sonnet-4-6 → hits pricing via tail (THE FIX)', () => {
|
||||
// Pre-v0.41.20.0: this returned null because the inline split only
|
||||
// handled `:`. CLI `--judge-model anthropic/...` + `--max-cost N` then
|
||||
// hit BudgetTracker no_pricing fail-closed.
|
||||
const cost = estimateMaxCostUsd('anthropic/claude-sonnet-4-6', 1_000_000, 0);
|
||||
expect(cost).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('mixed input + output cost math', () => {
|
||||
// 100K input + 50K output for opus 4.7 ($5/$25)
|
||||
// = 0.1 * 5 + 0.05 * 25 = 0.5 + 1.25 = 1.75
|
||||
const cost = estimateMaxCostUsd('anthropic/claude-opus-4-7', 100_000, 50_000);
|
||||
expect(cost).toBeCloseTo(1.75, 5);
|
||||
});
|
||||
|
||||
test('unknown model → returns null (caller warn-once + bypass)', () => {
|
||||
expect(estimateMaxCostUsd('mistral:medium', 1_000, 1_000)).toBeNull();
|
||||
expect(estimateMaxCostUsd('gpt-5', 1_000, 1_000)).toBeNull();
|
||||
});
|
||||
|
||||
test('OpenRouter nested form returns null — tail is `anthropic/claude-...` which is not a pricing key', () => {
|
||||
// Per D2 architecture: parseModelId returns {provider:'openrouter',
|
||||
// model:'anthropic/claude-sonnet-4-6'}; lookup on the tail
|
||||
// 'anthropic/claude-sonnet-4-6' misses (table has bare 'claude-sonnet-4-6').
|
||||
// OpenRouter pricing is intentionally out of scope (TODO #2).
|
||||
expect(estimateMaxCostUsd('openrouter:anthropic/claude-sonnet-4-6', 1_000, 1_000)).toBeNull();
|
||||
});
|
||||
|
||||
test('every key in ANTHROPIC_PRICING is reachable via bare/colon/slash form', () => {
|
||||
// Regression guard: if someone adds a new entry to ANTHROPIC_PRICING,
|
||||
// it should be reachable via all three forms automatically (the route
|
||||
// is structural, not per-key).
|
||||
for (const key of Object.keys(ANTHROPIC_PRICING)) {
|
||||
expect(estimateMaxCostUsd(key, 1_000_000, 0)).not.toBeNull();
|
||||
expect(estimateMaxCostUsd(`anthropic:${key}`, 1_000_000, 0)).not.toBeNull();
|
||||
expect(estimateMaxCostUsd(`anthropic/${key}`, 1_000_000, 0)).not.toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -132,6 +132,38 @@ describe('projectBatch', () => {
|
||||
});
|
||||
expect(p.raise_cap_hint).toBeUndefined();
|
||||
});
|
||||
|
||||
describe('v0.41.20.0 — slash-prefix model id routing (THE FIX)', () => {
|
||||
test('slash-form anthropic/claude-sonnet-4-6 strips to bare name + pricing hits', () => {
|
||||
// Pre-fix: inline bareModel(model) only handled `:`; slash-form fell
|
||||
// through to the unknown_model branch silently. Post-fix: parseModelId
|
||||
// handles both forms; pricing lookup succeeds; cold-start path produces
|
||||
// a non-null cost estimate.
|
||||
const p = projectBatch({
|
||||
job_count: 100,
|
||||
model: 'anthropic/claude-sonnet-4-6',
|
||||
stats: { sample_size: 0, effective_concurrency: 4 },
|
||||
current_lease_cap: 32,
|
||||
});
|
||||
expect(p.unknown_model).toBeUndefined();
|
||||
expect(p.total_cost_usd).not.toBeNull();
|
||||
expect(p.total_cost_usd).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('double-separator openrouter:anthropic/X → unknown_model branch fires', () => {
|
||||
// Per D2: colon wins; tail is `anthropic/claude-sonnet-4-6` which
|
||||
// doesn't match ANTHROPIC_PRICING keys. Confirms the deliberate
|
||||
// OpenRouter-out-of-scope posture is observable downstream.
|
||||
const p = projectBatch({
|
||||
job_count: 100,
|
||||
model: 'openrouter:anthropic/claude-sonnet-4-6',
|
||||
stats: { sample_size: 0, effective_concurrency: 4 },
|
||||
current_lease_cap: 32,
|
||||
});
|
||||
expect(p.unknown_model).toBe('anthropic/claude-sonnet-4-6');
|
||||
expect(p.total_cost_usd).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('formatProjection', () => {
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* v0.41.20.0 — pin the judge maxTokens scaling formula + per-model cap.
|
||||
*
|
||||
* Bug 1 from #1540: `maxTokens: 4000` hard-coded in judges.ts → response
|
||||
* truncated mid-JSON at any chunk ≥ ~40 ideas → parseJudgeJSON throw →
|
||||
* judge_failed: true → ideas saved unscored. Verified failure mode:
|
||||
* 0/72 ideas passing before fix; 39/72 after.
|
||||
*
|
||||
* The fix is in two pieces and this file pins both:
|
||||
*
|
||||
* 1. `computeJudgeMaxTokens` (pure formula) — scales with idea count,
|
||||
* respects per-model output cap, floors at 4000 for tiny batches.
|
||||
* 2. `runJudgeChunk` wires the formula into the `chat({maxTokens})` call
|
||||
* — integration test via the existing `chatFn` DI seam captures the
|
||||
* ChatOpts and asserts maxTokens matches the formula.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
computeJudgeMaxTokens,
|
||||
TOKEN_BUDGET_PER_IDEA,
|
||||
TOKEN_BUDGET_ENVELOPE,
|
||||
LEGACY_MIN_MAX_TOKENS,
|
||||
MAX_OUTPUT_TOKENS_CEIL,
|
||||
ANTHROPIC_OUTPUT_CAPS,
|
||||
runJudge,
|
||||
BRAINSTORM_JUDGE_CONFIG,
|
||||
type ChatFn,
|
||||
} from '../../src/core/brainstorm/judges.ts';
|
||||
import type { ChatOpts, ChatResult } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
describe('computeJudgeMaxTokens (pure formula)', () => {
|
||||
test('1 idea: formula yields 650 → floor binds at LEGACY_MIN_MAX_TOKENS (4000)', () => {
|
||||
// 1 * 150 + 500 = 650; max(4000, 650) = 4000.
|
||||
expect(computeJudgeMaxTokens(1, 'claude-sonnet-4-6')).toBe(LEGACY_MIN_MAX_TOKENS);
|
||||
});
|
||||
|
||||
test('10 ideas: formula yields 2000 → floor binds', () => {
|
||||
// 10 * 150 + 500 = 2000; max(4000, 2000) = 4000.
|
||||
expect(computeJudgeMaxTokens(10, 'claude-sonnet-4-6')).toBe(LEGACY_MIN_MAX_TOKENS);
|
||||
});
|
||||
|
||||
test('36 ideas: formula yields 5900 → above floor', () => {
|
||||
// 36 * 150 + 500 = 5900; max(4000, 5900) = 5900; below 64K Sonnet cap.
|
||||
expect(computeJudgeMaxTokens(36, 'claude-sonnet-4-6')).toBe(36 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE);
|
||||
});
|
||||
|
||||
test('96 ideas: formula yields 14_900 → above floor, under modern cap', () => {
|
||||
// 96 * 150 + 500 = 14_900; below 32K Opus cap.
|
||||
expect(computeJudgeMaxTokens(96, 'claude-opus-4-7')).toBe(96 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE);
|
||||
});
|
||||
|
||||
test('300 ideas on Opus 4.7: formula yields 45_500 → CAP binds at 32K', () => {
|
||||
// 300 * 150 + 500 = 45_500; min(32_000, 45_500) = 32_000.
|
||||
expect(computeJudgeMaxTokens(300, 'claude-opus-4-7')).toBe(32_000);
|
||||
});
|
||||
|
||||
test('300 ideas on Sonnet 4.6: formula yields 45_500 → fits under 64K Sonnet cap', () => {
|
||||
expect(computeJudgeMaxTokens(300, 'claude-sonnet-4-6')).toBe(300 * TOKEN_BUDGET_PER_IDEA + TOKEN_BUDGET_ENVELOPE);
|
||||
});
|
||||
|
||||
test('96 ideas on legacy Haiku 3.5 (8K cap): CAP binds at 8192 (D11 codex fix)', () => {
|
||||
// 96 * 150 + 500 = 14_900 > 8192; legacy 3.5 caps at 8K — without
|
||||
// ANTHROPIC_OUTPUT_CAPS this would have been the next opaque HTTP 400.
|
||||
expect(computeJudgeMaxTokens(96, 'claude-3-5-haiku-20241022')).toBe(8_192);
|
||||
});
|
||||
|
||||
test('unknown model: falls back to MAX_OUTPUT_TOKENS_CEIL', () => {
|
||||
expect(computeJudgeMaxTokens(300, 'mistral:medium')).toBe(MAX_OUTPUT_TOKENS_CEIL);
|
||||
expect(computeJudgeMaxTokens(300, 'gpt-5')).toBe(MAX_OUTPUT_TOKENS_CEIL);
|
||||
});
|
||||
|
||||
// v0.41.21.0: when modelId is undefined the cap routes through the gateway's
|
||||
// configured chat model via getChatModel(). The actual returned cap therefore
|
||||
// depends on whether the gateway has been initialized in this test process
|
||||
// (cross-test side effect of any earlier import that called configureGateway).
|
||||
// We test the explicit-modelId path comprehensively above; the undefined path
|
||||
// is exercised end-to-end below via runJudge() without modelOverride.
|
||||
|
||||
test('colon-prefixed id resolves through splitProviderModelId', () => {
|
||||
expect(computeJudgeMaxTokens(96, 'anthropic:claude-opus-4-7')).toBe(96 * 150 + 500);
|
||||
});
|
||||
|
||||
test('slash-prefixed id resolves through splitProviderModelId (THE FIX combined with site routing)', () => {
|
||||
expect(computeJudgeMaxTokens(96, 'anthropic/claude-opus-4-7')).toBe(96 * 150 + 500);
|
||||
// Legacy 3.5 via slash form still hits the 8K cap.
|
||||
expect(computeJudgeMaxTokens(96, 'anthropic/claude-3-5-haiku-20241022')).toBe(8_192);
|
||||
});
|
||||
|
||||
test('every entry in ANTHROPIC_OUTPUT_CAPS is reachable by lookup', () => {
|
||||
for (const [key, cap] of Object.entries(ANTHROPIC_OUTPUT_CAPS)) {
|
||||
// Pick an idea count high enough that the cap binds.
|
||||
const huge = Math.ceil(cap / TOKEN_BUDGET_PER_IDEA) + 10;
|
||||
expect(computeJudgeMaxTokens(huge, key)).toBe(cap);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('runJudge wires computeJudgeMaxTokens into chat({maxTokens})', () => {
|
||||
function makeCapturingChatFn(captured: ChatOpts[]): ChatFn {
|
||||
return async (opts: ChatOpts): Promise<ChatResult> => {
|
||||
captured.push(opts);
|
||||
// Return a valid-shape judge response so parseJudgeJSON succeeds and
|
||||
// we can pin maxTokens without dealing with parse failures.
|
||||
const ideasJson = (opts.messages[0].content as string)
|
||||
.match(/^- id=(\S+)/gm)
|
||||
?.map((line) => line.replace(/^- id=/, '')) ?? [];
|
||||
const ideas = ideasJson.map((id) => ({
|
||||
id,
|
||||
scores: {
|
||||
originality: 3,
|
||||
resistance: 3,
|
||||
thesis_density: 3,
|
||||
concrete_grounding: 3,
|
||||
cognitive_load: 3,
|
||||
},
|
||||
note: 'stub',
|
||||
}));
|
||||
const text = JSON.stringify({ ideas });
|
||||
return {
|
||||
text,
|
||||
blocks: [{ type: 'text', text }],
|
||||
stopReason: 'end',
|
||||
model: opts.model ?? 'noop',
|
||||
providerId: 'anthropic',
|
||||
usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
function makeIdeas(n: number) {
|
||||
return Array.from({ length: n }, (_, i) => ({
|
||||
id: String(i + 1).padStart(2, '0'),
|
||||
text: 'stub idea text',
|
||||
close_slug: 'concepts/foo',
|
||||
far_slug: 'wiki/bar',
|
||||
}));
|
||||
}
|
||||
|
||||
test('1 idea → maxTokens = LEGACY_MIN_MAX_TOKENS (4000)', async () => {
|
||||
const captured: ChatOpts[] = [];
|
||||
await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(1), {
|
||||
chatFn: makeCapturingChatFn(captured),
|
||||
modelOverride: 'claude-sonnet-4-6',
|
||||
});
|
||||
expect(captured.length).toBe(1);
|
||||
expect(captured[0].maxTokens).toBe(LEGACY_MIN_MAX_TOKENS);
|
||||
});
|
||||
|
||||
test('96 ideas on Opus 4.7 → maxTokens = formula (14_900)', async () => {
|
||||
const captured: ChatOpts[] = [];
|
||||
await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(96), {
|
||||
chatFn: makeCapturingChatFn(captured),
|
||||
modelOverride: 'anthropic:claude-opus-4-7',
|
||||
});
|
||||
expect(captured.length).toBe(1);
|
||||
expect(captured[0].maxTokens).toBe(96 * 150 + 500);
|
||||
});
|
||||
|
||||
test('slash-form modelOverride routes through parseModelId for the cap lookup', async () => {
|
||||
// Pre-v0.41.20.0 the inline maxTokens was a constant; this test would
|
||||
// not have caught anything. Post-fix: slash-form is honored for the
|
||||
// per-model cap because computeJudgeMaxTokens routes through parseModelId.
|
||||
const captured: ChatOpts[] = [];
|
||||
await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(96), {
|
||||
chatFn: makeCapturingChatFn(captured),
|
||||
modelOverride: 'anthropic/claude-3-5-haiku-20241022',
|
||||
});
|
||||
// 14_900 formula > 8K legacy cap → cap binds.
|
||||
expect(captured[0].maxTokens).toBe(8_192);
|
||||
});
|
||||
|
||||
test('200-idea chunk size set via maxIdeasPerCall → maxTokens scales (single chunk)', async () => {
|
||||
const captured: ChatOpts[] = [];
|
||||
await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(200), {
|
||||
chatFn: makeCapturingChatFn(captured),
|
||||
modelOverride: 'claude-sonnet-4-6',
|
||||
maxIdeasPerCall: 200,
|
||||
});
|
||||
expect(captured.length).toBe(1);
|
||||
expect(captured[0].maxTokens).toBe(200 * 150 + 500); // 30_500, under 64K Sonnet cap
|
||||
});
|
||||
|
||||
test('chunking is independent of cap — multi-chunk each gets its own scaled budget', async () => {
|
||||
const captured: ChatOpts[] = [];
|
||||
// 250 ideas at default chunk 100 → 3 chunks of [100, 100, 50].
|
||||
await runJudge(BRAINSTORM_JUDGE_CONFIG, makeIdeas(250), {
|
||||
chatFn: makeCapturingChatFn(captured),
|
||||
modelOverride: 'claude-sonnet-4-6',
|
||||
});
|
||||
expect(captured.length).toBe(3);
|
||||
expect(captured[0].maxTokens).toBe(100 * 150 + 500); // 15_500
|
||||
expect(captured[1].maxTokens).toBe(100 * 150 + 500);
|
||||
expect(captured[2].maxTokens).toBe(Math.max(LEGACY_MIN_MAX_TOKENS, 50 * 150 + 500)); // 8000
|
||||
});
|
||||
});
|
||||
@@ -138,6 +138,38 @@ describe('BudgetTracker.reserve', () => {
|
||||
expect((caught as Error).message).toMatch(/anthropic-pricing\.ts/);
|
||||
});
|
||||
|
||||
test('v0.41.20.0: slash-prefix anthropic/claude-* under --max-cost does NOT no_pricing throw (THE FIX)', () => {
|
||||
// Pre-v0.41.20.0: lookupPricing only split modelId on ':'. CLI users
|
||||
// running `gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6
|
||||
// --max-cost 5` hit TX2 no_pricing because the slash-form id silently
|
||||
// missed ANTHROPIC_PRICING (closes #1540).
|
||||
const t = new BudgetTracker({ maxCostUsd: 10.0, label: 'test', auditPath });
|
||||
expect(() =>
|
||||
t.reserve({
|
||||
modelId: 'anthropic/claude-sonnet-4-6',
|
||||
estimatedInputTokens: 100,
|
||||
maxOutputTokens: 100,
|
||||
kind: 'chat',
|
||||
}),
|
||||
).not.toThrow();
|
||||
const audit = readAudit();
|
||||
expect(audit[0].event).toBe('reserve');
|
||||
});
|
||||
|
||||
test('v0.41.20.0: colon-prefix anthropic:claude-* under --max-cost still works (regression guard)', () => {
|
||||
// Same path as slash, exercised separately so a future refactor that
|
||||
// accidentally drops colon support fires this test loudly.
|
||||
const t = new BudgetTracker({ maxCostUsd: 10.0, label: 'test', auditPath });
|
||||
expect(() =>
|
||||
t.reserve({
|
||||
modelId: 'anthropic:claude-sonnet-4-6',
|
||||
estimatedInputTokens: 100,
|
||||
maxOutputTokens: 100,
|
||||
kind: 'chat',
|
||||
}),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('no cap + unknown pricing: warns once per process, no throw', () => {
|
||||
const t = new BudgetTracker({ label: 'test', auditPath });
|
||||
expect(() =>
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* v0.41.20.0 — pin slash-prefix model id routing through
|
||||
* `eval-contradictions/cost-tracker.ts:pricingFor`.
|
||||
*
|
||||
* The cost-tracker carries its own duplicate ANTHROPIC_PRICING table
|
||||
* (consolidation deferred to TODOS.md #3 from the v0.41.20.0 plan).
|
||||
* Pre-fix, `pricingFor` did exact-key match only, so every non-bare
|
||||
* lookup silently fell back to Haiku — including colon-form Sonnet/Opus
|
||||
* that the table DOES carry as explicit `anthropic:claude-*` keys. After
|
||||
* fix, parseModelId handles bare/colon/slash uniformly; the silent-Haiku
|
||||
* fallback is preserved only for genuinely-unknown models (legacy behavior
|
||||
* pinned here per D9).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { CostTracker } from '../../src/core/eval-contradictions/cost-tracker.ts';
|
||||
|
||||
describe('eval-contradictions/cost-tracker pricingFor (via recordJudgeCall)', () => {
|
||||
function spendCheck(modelId: string, inputTokens: number, outputTokens: number): number {
|
||||
const t = new CostTracker({ capUsd: 999 });
|
||||
t.recordJudgeCall(modelId, { inputTokens, outputTokens });
|
||||
return t.judge();
|
||||
}
|
||||
|
||||
test('bare claude-sonnet-4-6 → Sonnet pricing (1M in = $3)', () => {
|
||||
expect(spendCheck('claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('colon-form anthropic:claude-sonnet-4-6 → Sonnet pricing', () => {
|
||||
// Table has this key explicitly; pre-fix path also worked.
|
||||
expect(spendCheck('anthropic:claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('slash-form anthropic/claude-sonnet-4-6 → Sonnet pricing (THE FIX)', () => {
|
||||
// Pre-v0.41.20.0: silently billed as Haiku ($1/MTok instead of $3/MTok).
|
||||
expect(spendCheck('anthropic/claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(3.0, 5);
|
||||
});
|
||||
|
||||
test('slash-form anthropic/claude-opus-4-7 → Opus pricing ($5/MTok in)', () => {
|
||||
expect(spendCheck('anthropic/claude-opus-4-7', 1_000_000, 0)).toBeCloseTo(5.0, 5);
|
||||
});
|
||||
|
||||
test('LEGACY BEHAVIOR PIN: unknown model silently falls back to Haiku pricing', () => {
|
||||
// D9 from the v0.41.20.0 plan: we deliberately preserve the silent-Haiku
|
||||
// fallback in this duplicate pricing table. The right fix is unifying
|
||||
// the two pricing systems (TODOS.md #3) — tightening cost-tracker in
|
||||
// isolation would surprise existing eval-contradictions callers who
|
||||
// depend on the soft-ceiling --budget-usd contract.
|
||||
expect(spendCheck('mistral/medium', 1_000_000, 0)).toBeCloseTo(1.0, 5);
|
||||
expect(spendCheck('gpt-5', 1_000_000, 0)).toBeCloseTo(1.0, 5);
|
||||
});
|
||||
|
||||
test('OpenRouter nested form falls back to Haiku (legacy behavior preserved)', () => {
|
||||
// Per D2: parseModelId returns {provider:'openrouter', model:'anthropic/...'};
|
||||
// the tail 'anthropic/claude-sonnet-4-6' is not a pricing key in this
|
||||
// duplicate table (which doesn't carry slash-form keys), so the silent
|
||||
// Haiku fallback fires. Matches the deliberate OpenRouter-pricing-deferred
|
||||
// posture from TODOS.md #2.
|
||||
expect(spendCheck('openrouter:anthropic/claude-sonnet-4-6', 1_000_000, 0)).toBeCloseTo(1.0, 5);
|
||||
});
|
||||
});
|
||||
@@ -220,6 +220,19 @@ describe('resolveModel — v0.31.12 tier system', () => {
|
||||
expect(isAnthropicProvider('')).toBe(false);
|
||||
});
|
||||
|
||||
test('v0.41.20.0: isAnthropicProvider classifies slash-form (subagent-guard fix)', () => {
|
||||
// Pre-fix: 'anthropic/claude-sonnet-4-6' had no colon and didn't start
|
||||
// with 'claude-' (started with 'anthropic') → returned false → silent
|
||||
// subagent-guard bypass → fall back to TIER_DEFAULTS.subagent without
|
||||
// honoring the user's explicit slash-form config.
|
||||
expect(isAnthropicProvider('anthropic/claude-sonnet-4-6')).toBe(true);
|
||||
expect(isAnthropicProvider('anthropic/claude-opus-4-7')).toBe(true);
|
||||
// Non-Anthropic slash forms STILL return false (don't accidentally
|
||||
// widen the guard).
|
||||
expect(isAnthropicProvider('openai/gpt-5')).toBe(false);
|
||||
expect(isAnthropicProvider('google/gemini-3-pro')).toBe(false);
|
||||
});
|
||||
|
||||
test('alias-chain conflict: forward + reverse for same id (Codex F6)', async () => {
|
||||
// Codex F6: if both forward and reverse aliases exist, depth cap (2)
|
||||
// prevents infinite loop. Canonicalization is deterministic — terminates
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* v0.41.21.0 — splitProviderModelId centralizer contract.
|
||||
*
|
||||
* Pins every shape the helper must handle so future refactors of the 5
|
||||
* downstream consumers (anthropic-pricing, budget-tracker, cost-tracker,
|
||||
* batch-projection, model-config) can't silently regress on the slash-prefix
|
||||
* bug class.
|
||||
*
|
||||
* Sibling: `src/core/ai/model-resolver.ts:parseModelId` — the gateway-side
|
||||
* resolver. Both accept the same input shapes post-v0.41.21.0; this helper
|
||||
* is defensive (returns `{provider: null, model: 'bare'}` for bare names)
|
||||
* and the gateway one throws (routing needs an explicit provider).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { splitProviderModelId } from '../src/core/model-id.ts';
|
||||
|
||||
describe('splitProviderModelId', () => {
|
||||
describe('happy paths', () => {
|
||||
test('bare model id → no provider', () => {
|
||||
expect(splitProviderModelId('claude-sonnet-4-6')).toEqual({
|
||||
provider: null,
|
||||
model: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
|
||||
test('colon-separated provider:model', () => {
|
||||
expect(splitProviderModelId('anthropic:claude-sonnet-4-6')).toEqual({
|
||||
provider: 'anthropic',
|
||||
model: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
|
||||
test('slash-separated provider/model — THE BUG CLASS FIX', () => {
|
||||
// Pre-fix: every site's inline split missed this shape, silently
|
||||
// returning the whole string as the "model" and failing pricing lookups.
|
||||
expect(splitProviderModelId('anthropic/claude-sonnet-4-6')).toEqual({
|
||||
provider: 'anthropic',
|
||||
model: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
|
||||
test('double-separator openrouter:anthropic/X — colon wins, tail as-is', () => {
|
||||
// Per D2 architecture: do NOT recursively peel. Transport=openrouter;
|
||||
// pricing-vendor-identity is intentionally deferred to TODO #2 (non-
|
||||
// Anthropic pricing). Pricing lookups will miss on the slash-bearing
|
||||
// tail and land in the caller's existing unknown-model path.
|
||||
expect(splitProviderModelId('openrouter:anthropic/claude-sonnet-4.6')).toEqual({
|
||||
provider: 'openrouter',
|
||||
model: 'anthropic/claude-sonnet-4.6',
|
||||
});
|
||||
});
|
||||
|
||||
test('slash-separated openrouter form openai/gpt-5', () => {
|
||||
expect(splitProviderModelId('openai/gpt-5')).toEqual({
|
||||
provider: 'openai',
|
||||
model: 'gpt-5',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('defensive contract', () => {
|
||||
test('null → {provider: null, model: ""}', () => {
|
||||
expect(splitProviderModelId(null)).toEqual({ provider: null, model: '' });
|
||||
});
|
||||
|
||||
test('undefined → {provider: null, model: ""}', () => {
|
||||
expect(splitProviderModelId(undefined)).toEqual({ provider: null, model: '' });
|
||||
});
|
||||
|
||||
test('empty string → {provider: null, model: ""}', () => {
|
||||
expect(splitProviderModelId('')).toEqual({ provider: null, model: '' });
|
||||
});
|
||||
|
||||
test('whitespace-only → {provider: null, model: ""}', () => {
|
||||
expect(splitProviderModelId(' ')).toEqual({ provider: null, model: '' });
|
||||
expect(splitProviderModelId('\t\n ')).toEqual({ provider: null, model: '' });
|
||||
});
|
||||
|
||||
test('leading/trailing whitespace is trimmed before split', () => {
|
||||
expect(splitProviderModelId(' anthropic:claude-sonnet-4-6 ')).toEqual({
|
||||
provider: 'anthropic',
|
||||
model: 'claude-sonnet-4-6',
|
||||
});
|
||||
expect(splitProviderModelId(' anthropic/claude-sonnet-4-6 ')).toEqual({
|
||||
provider: 'anthropic',
|
||||
model: 'claude-sonnet-4-6',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge inputs', () => {
|
||||
test('leading separator ":foo" → provider is empty string, not null', () => {
|
||||
// Distinguish "no separator present" (null provider) from "separator
|
||||
// with empty left side" (empty-string provider). Empty-string provider
|
||||
// is a malformed input but we preserve the distinction so downstream
|
||||
// callers can detect it without re-parsing.
|
||||
expect(splitProviderModelId(':claude-foo')).toEqual({
|
||||
provider: '',
|
||||
model: 'claude-foo',
|
||||
});
|
||||
});
|
||||
|
||||
test('leading slash "/foo" → provider is empty string', () => {
|
||||
expect(splitProviderModelId('/claude-foo')).toEqual({
|
||||
provider: '',
|
||||
model: 'claude-foo',
|
||||
});
|
||||
});
|
||||
|
||||
test('trailing separator "anthropic:" → model is empty string', () => {
|
||||
expect(splitProviderModelId('anthropic:')).toEqual({
|
||||
provider: 'anthropic',
|
||||
model: '',
|
||||
});
|
||||
});
|
||||
|
||||
test('only ":" → empty provider AND empty model', () => {
|
||||
expect(splitProviderModelId(':')).toEqual({
|
||||
provider: '',
|
||||
model: '',
|
||||
});
|
||||
});
|
||||
|
||||
test('only "/" → empty provider AND empty model', () => {
|
||||
expect(splitProviderModelId('/')).toEqual({
|
||||
provider: '',
|
||||
model: '',
|
||||
});
|
||||
});
|
||||
|
||||
test('mixed-case provider is preserved (no normalization)', () => {
|
||||
// Callers that care (e.g. isAnthropicProvider) lowercase themselves.
|
||||
expect(splitProviderModelId('Anthropic:claude-foo')).toEqual({
|
||||
provider: 'Anthropic',
|
||||
model: 'claude-foo',
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user