mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
fix(cycle): BudgetMeter prices from the derived Anthropic view, so the cap is off for canonically-priced models (#3951)
* fix(cycle): price BudgetMeter through the canonical table, not the derived Anthropic view BudgetMeter estimated submit cost with `estimateMaxCostUsd`, which reads ANTHROPIC_PRICING. CLAUDE.md defines that table as a DERIVED view of the one canonical chat-pricing table, so any non-Anthropic model was unpriceable here even when CANONICAL_PRICING carries its rates — and an unpriceable model does not merely lose accuracy, it takes the `cost === null` branch and returns `allowed: true` with cost 0. The cap is off for that model. Measured on130d321d: openai:gpt-5.2 and deepseek:deepseek-chat are both in CANONICAL_PRICING and both absent from the derived view, so the gate was disabled for them. claude-opus-4-7 resolves through either. The estimate now goes through canonicalLookup, falling back to the existing call for anything canonical does not carry. Anthropic numbers are unchanged by construction — the derived view is generated from canonical — and a test pins that parity. Deliberately NOT changed: a model absent from canonical too keeps the documented warn-and-allow bypass, with the same BUDGET_METER_NO_PRICING warning, the same submit_unpriced ledger event, and the same `unpricedSubmits` counter. Whether those should be priced at a conservative fallback rate instead — as synthesize-concepts (#3915) and skillopt/preflight already do — is a policy call, not this fix. The ledger schema is untouched; `test/fixtures/dream-budget-schema-v1.jsonl` still describes it. Related to #2149 (self-closed by its reporter without a fix; the pricing half of what it described is still live). Distinct from #2504, which is the opposite failure mode in BudgetTracker: hard-throw on a missing entry rather than silent bypass. Verification on130d321d(v0.42.76.0): - bun test budget-meter + auto-think-phase + propose-takes + model-pricing -> 93 pass / 0 fail - red check: with upstream/master's budget-meter.ts -> 10 pass / 1 fail on the new gating test - bun run typecheck -> clean - bun run verify -> 34/34 green Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(cycle): guard the non-finite estimate, and correct the docs the change makes stale Codex review, both verified before applying: - Both pricing tables are object literals, so a model id colliding with an inherited key ('constructor', 'toString', '__proto__') resolves to a truthy Object.prototype value whose .input/.output are undefined and the rate arithmetic yields NaN. Measured: canonicalLookup('constructor') returns a function, and estimateMaxCostUsd('constructor', …) already returns NaN on master, so the shape predates this change — but it sits on the path this patch touches. `cumulative + NaN` is NaN and `NaN > budget` is false, so one such submit would disable the gate for the rest of the cycle. The estimate helper now treats a non-finite result as unpriceable, which routes that one submit into the documented warn-and-allow branch and leaves the running total finite. Regression test submits the poison id, then a submit that must be denied. The same shape on the estimateMaxCostUsd path is left alone. - Two docs asserted the old behaviour and are now current-state: the anthropic-pricing.ts header ("non-Anthropic models … bypass … runs unbounded") and the KEY_FILES.md entry for the same file. Both now say the bypass applies only when canonical has no rates either. `bun run build:llms` regenerated; the bundles are unchanged because KEY_FILES.md is linked rather than inlined, and test/build-llms.test.ts passes. Verification on130d321d(v0.42.76.0): - bun test budget-meter + auto-think-phase + propose-takes + model-pricing + build-llms -> 106 pass / 0 fail - bun run typecheck -> clean - bun run verify -> 34/34 green Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Masa
@@ -132,7 +132,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. Declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio), avoiding the backfill loop where tiktoken-grounded budgeting undercounted Voyage's actual token usage. Declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. Recipe docstring at `:7-16` names the seven hosted flexible-dim models that accept `output_dimension` (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and notes `voyage-4-nano` is the open-weight variant fixed at 1024-dim that does NOT accept the parameter (negative regression assertion in `test/ai/gateway.test.ts`: `dimsProviderOptions` returns `undefined` for `voyage-4-nano`). `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`); discoverability surfaces: decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`.
|
||||
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). Canonical id is `claude-sonnet-4-6` (no date suffix); a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize`). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts`.
|
||||
- `src/core/model-pricing.ts` — single source of truth for paid-cloud CHAT/completion model pricing (USD per 1M tokens, input | output). `CANONICAL_PRICING` is a `provider:model`-keyed table (Anthropic Opus 5/4.8/4.7/4.6 `$5/$25`, Sonnet 4.6 `$3/$15`, Haiku 4.5 `$1/$5` both dateless + dated, plus OpenAI / Google / Together / DeepSeek panel models). `canonicalLookup(modelId)` resolves bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and slash (`anthropic/...`) forms — bare ids default to the `anthropic:` provider; nested OpenRouter ids (`openrouter:anthropic/...`) intentionally MISS so OpenRouter markup isn't repriced as the inner vendor. Every other chat-pricing table is a DERIVED view of this one (NOT a hand-copied duplicate), so cross-table price drift is structurally impossible. Embeddings live separately in `embedding-pricing.ts` (different unit). Pinned by `test/model-pricing.test.ts` whose drift guard asserts each derived view equals canonical and that the cross-modal panel models are all present.
|
||||
- `src/core/anthropic-pricing.ts` — bare-keyed Anthropic VIEW of `model-pricing.ts` (the `anthropic:` canonical entries with the prefix stripped). Kept distinct because many callers look up by bare Claude id and because `estimateMaxCostUsd(modelId, inTokens, maxOutTokens)` carries the null-on-miss contract the dream-cycle budget gate depends on (non-Anthropic ids return null → caller warns `BUDGET_METER_NO_PRICING` once and runs unbounded). `estimateMaxCostUsd` routes bare/colon/slash ids through `splitProviderModelId`. Do NOT hand-edit prices here — the map is derived from canonical, so it cannot drift. `ANTHROPIC_PRICING` is consumed by `budget/budget-tracker.ts`, `minions/batch-projection.ts`, and `cycle/budget-meter.ts`.
|
||||
- `src/core/anthropic-pricing.ts` — bare-keyed Anthropic VIEW of `model-pricing.ts` (the `anthropic:` canonical entries with the prefix stripped). Kept distinct because many callers look up by bare Claude id and because `estimateMaxCostUsd(modelId, inTokens, maxOutTokens)` carries the null-on-miss contract the dream-cycle budget gate depends on (non-Anthropic ids return null; `BudgetMeter` tries `canonicalLookup` first and only falls back here, so it warns `BUDGET_METER_NO_PRICING` and runs unbounded only when canonical has no rates either). `estimateMaxCostUsd` routes bare/colon/slash ids through `splitProviderModelId`. Do NOT hand-edit prices here — the map is derived from canonical, so it cannot drift. `ANTHROPIC_PRICING` is consumed by `budget/budget-tracker.ts`, `minions/batch-projection.ts`, and `cycle/budget-meter.ts`.
|
||||
- `src/core/takes-quality-eval/pricing.ts` — fail-closed budget pricing for `eval takes-quality run --budget-usd N`. `MODEL_PRICING` is a curated `provider:model` allowlist (default panel + likely overrides) whose VALUES are derived from `model-pricing.ts` via `canonicalLookup`; an allowlisted id missing from canonical throws at module load. Schema is `{input_per_1m, output_per_1m}`. A model NOT on the allowlist aborts the run with an actionable error rather than guessing (distinct from `cross-modal-eval/runner.ts`, which silently estimates zero on unknown models — both now source numbers from canonical).
|
||||
- `src/core/budget/budget-tracker.ts` — keystone primitive for the brainstorm cost-cathedral wave. One typed error (`BudgetExhausted` with `reason: 'cost' | 'runtime' | 'no_pricing'`), one schema-stable audit JSONL at `~/.gbrain/audit/budget-YYYY-Www.jsonl`. Contracts: `record()` throws when cumulative spend exceeds cap (the cap is a real ceiling, not a suggestion); `reserve()` hard-fails with `reason: 'no_pricing'` when `maxCostUsd` is set AND the model is missing from pricing maps (warn-once preserved when cap is unset); `extractUsageFromError(err, fallback)` returns `err.usage` when the SDK provides it, else the pessimistic fallback (caller passes `maxOutputTokens`, not the optimistic pre-call estimate). `onExhausted(cb)` fires once synchronously BEFORE the throw propagates so callers can persist checkpoints. Replaces three parallel copies (inline brainstorm class, cycle/budget-meter, eval-contradictions). Adapts the old `BudgetMeter` (public shape preserved + `schema_version: 1` stamped on every dream-budget audit line). Pinned by 18 unit cases.
|
||||
- `src/core/audit-week-file.ts` — single source of truth for ISO-week audit JSONL filename math. Exports `isoWeek(d)`, `isoWeekFilename(prefix, now?)`, `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). Year-boundary correctness pinned by tests at 2020-W53 (the 53-week year), 2025-W01 rolling in from 2024-12-30 (Monday), 2026-W01. Four call sites migrated: `src/core/minions/handlers/shell-audit.ts`, `src/core/facts/phantom-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/cycle/budget-meter.ts`. Each keeps its `compute<X>AuditFilename` thin wrapper for back-compat with existing tests.
|
||||
|
||||
@@ -11,9 +11,12 @@
|
||||
* takes-quality-eval/pricing.ts duplicated the numbers and drifted: Opus 4.7
|
||||
* read $15/$75 in one and $5/$25 in the other.)
|
||||
*
|
||||
* Codex P1 #10 fold: non-Anthropic models (gemini, gpt, anything not in this
|
||||
* map) bypass the budget gate with a `BUDGET_METER_NO_PRICING` warn once per
|
||||
* process. The cycle still runs unbounded for those models.
|
||||
* Codex P1 #10 fold: `estimateMaxCostUsd` keeps its null-on-miss contract, and
|
||||
* a null estimate makes the dream-cycle budget gate wave the submit through.
|
||||
* BudgetMeter therefore prices through `canonicalLookup` first and only falls
|
||||
* back here, so a model is unbounded ONLY when the canonical table has no
|
||||
* rates for it either — that case still warns `BUDGET_METER_NO_PRICING` once
|
||||
* per process.
|
||||
*/
|
||||
|
||||
import { CANONICAL_PRICING, type ModelPricing } from './model-pricing.ts';
|
||||
|
||||
@@ -11,7 +11,9 @@
|
||||
*
|
||||
* Per Codex P1 #10: each subagent submit estimates max-cost from
|
||||
* `model + max_output_tokens`, accumulates per-cycle, refuses next submit
|
||||
* if cumulative > budget. Non-Anthropic models bypass the gate with a
|
||||
* if cumulative > budget. Pricing resolves through the canonical chat table
|
||||
* (`canonicalLookup`), so any provider carried there is gated. Only a model
|
||||
* absent from canonical too bypasses the gate, with a
|
||||
* `BUDGET_METER_NO_PRICING` warn (once per process).
|
||||
*
|
||||
* Ledger lives at `~/.gbrain/audit/dream-budget-YYYY-Www.jsonl` (ISO-week
|
||||
@@ -24,6 +26,7 @@ import { mkdirSync, appendFileSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { isoWeekFilename, resolveAuditDir } from '../audit-week-file.ts';
|
||||
import { estimateMaxCostUsd, ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
import { canonicalLookup } from '../model-pricing.ts';
|
||||
|
||||
export interface BudgetMeterOpts {
|
||||
/** USD cap for the whole cycle. 0 or negative disables the gate. */
|
||||
@@ -81,21 +84,58 @@ export class BudgetMeter {
|
||||
this.auditPath = auditFilePath(opts.auditPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Max-cost estimate for a planned submit.
|
||||
*
|
||||
* Prices through `canonicalLookup` first. `estimateMaxCostUsd` reads
|
||||
* ANTHROPIC_PRICING, which CLAUDE.md defines as a DERIVED view of the one
|
||||
* canonical chat-pricing table — so reaching for it directly made every
|
||||
* non-Anthropic model unpriceable here even when the canonical table has
|
||||
* its rates, and an unpriceable model disables the gate entirely (see
|
||||
* `check`). Anthropic ids resolve identically either way, since the derived
|
||||
* view is generated from canonical.
|
||||
*
|
||||
* Returns null only for models absent from the canonical table too; the
|
||||
* caller keeps the existing warn-and-allow behaviour for those.
|
||||
*/
|
||||
private estimateCost(estimate: SubmitEstimate): number | null {
|
||||
const p = canonicalLookup(estimate.modelId);
|
||||
const raw = p
|
||||
? (estimate.estimatedInputTokens / 1_000_000) * p.input +
|
||||
(estimate.maxOutputTokens / 1_000_000) * p.output
|
||||
: estimateMaxCostUsd(
|
||||
estimate.modelId,
|
||||
estimate.estimatedInputTokens,
|
||||
estimate.maxOutputTokens,
|
||||
);
|
||||
// A non-finite estimate must not reach the accumulator. Both tables are
|
||||
// plain object literals, so a model id colliding with an inherited key
|
||||
// ('constructor', 'toString', '__proto__') resolves to a truthy
|
||||
// Object.prototype value whose .input/.output are undefined, and the
|
||||
// arithmetic yields NaN. `cumulative + NaN` is NaN, `NaN > budget` is
|
||||
// false, so a single such submit would silently disable the gate for the
|
||||
// rest of the cycle. Treated as unpriceable instead, which routes into
|
||||
// the documented warn-and-allow branch for that one submit and leaves
|
||||
// the running total intact. (The same shape exists on the
|
||||
// estimateMaxCostUsd path today; not changed here.)
|
||||
return raw !== null && Number.isFinite(raw) ? raw : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check whether a planned submit fits within the remaining budget.
|
||||
* Records the attempt to the ledger regardless of allow/deny.
|
||||
* Caller is responsible for skipping the actual LLM call when allowed=false.
|
||||
*/
|
||||
check(estimate: SubmitEstimate): BudgetCheckResult {
|
||||
const cost = estimateMaxCostUsd(estimate.modelId, estimate.estimatedInputTokens, estimate.maxOutputTokens);
|
||||
const cost = this.estimateCost(estimate);
|
||||
|
||||
// Codex P1 #10: non-Anthropic / unpriced models bypass the gate.
|
||||
// Codex P1 #10: models absent from the canonical table bypass the gate.
|
||||
if (cost === null) {
|
||||
this.unpricedSubmitsThisCycle++;
|
||||
if (!_unpricedWarnings.has(estimate.modelId)) {
|
||||
_unpricedWarnings.add(estimate.modelId);
|
||||
process.stderr.write(
|
||||
`[budget] BUDGET_METER_NO_PRICING: model "${estimate.modelId}" not in ANTHROPIC_PRICING. ` +
|
||||
`[budget] BUDGET_METER_NO_PRICING: model "${estimate.modelId}" has no canonical pricing. ` +
|
||||
`Budget gate disabled for this submit. (Per-provider pricing modules: TODO v0.29.)\n`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ import { mkdtempSync, rmSync, readFileSync, existsSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { BudgetMeter, _resetBudgetMeterWarningsForTest, ANTHROPIC_PRICING } from '../src/core/cycle/budget-meter.ts';
|
||||
import { estimateMaxCostUsd } from '../src/core/anthropic-pricing.ts';
|
||||
|
||||
let tmpDir: string;
|
||||
let auditPath: string;
|
||||
@@ -60,6 +61,65 @@ describe('BudgetMeter', () => {
|
||||
expect(meter.unpricedSubmits).toBe(2);
|
||||
});
|
||||
|
||||
test('a canonical-priced non-Anthropic model is gated, not waved through', () => {
|
||||
// openai:gpt-5.2 is in CANONICAL_PRICING but not in the derived
|
||||
// ANTHROPIC_PRICING view, so the pre-canonical lookup returned null and
|
||||
// check() took the unpriced bypass: allowed, cost 0, gate disabled.
|
||||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||||
const r = meter.check({
|
||||
modelId: 'openai:gpt-5.2',
|
||||
estimatedInputTokens: 1_000_000,
|
||||
maxOutputTokens: 1_000_000,
|
||||
label: 'canonical-priced',
|
||||
});
|
||||
expect(r.unpriced).toBeFalsy();
|
||||
expect(r.estimatedCostUsd).toBeGreaterThan(0);
|
||||
expect(r.allowed).toBe(false); // 1M+1M tokens cannot fit $0.001
|
||||
expect(meter.unpricedSubmits).toBe(0);
|
||||
expect(readLedger().at(-1)!.event).toBe('submit_denied');
|
||||
});
|
||||
|
||||
test('a model absent from the canonical table keeps the documented bypass', () => {
|
||||
// Unchanged behaviour: gemini-3-pro is in neither table. Whether these
|
||||
// should also be gated (via a conservative fallback rate, as
|
||||
// synthesize-concepts and skillopt/preflight do) is a policy call and is
|
||||
// deliberately not decided here.
|
||||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||||
const r = meter.check({ modelId: 'gemini-3-pro', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'absent' });
|
||||
expect(r.unpriced).toBe(true);
|
||||
expect(r.allowed).toBe(true);
|
||||
expect(readLedger().at(-1)!.event).toBe('submit_unpriced');
|
||||
});
|
||||
|
||||
test('Anthropic ids price identically through canonical and the derived view', () => {
|
||||
// The derived view is generated from canonical, so routing through
|
||||
// canonicalLookup must not move any Anthropic number.
|
||||
const meter = new BudgetMeter({ budgetUsd: 1000, phase: 'auto_think', auditPath });
|
||||
const r = meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'parity' });
|
||||
const viaView = estimateMaxCostUsd('claude-opus-4-7', 5000, 4000);
|
||||
expect(viaView).not.toBeNull();
|
||||
expect(r.estimatedCostUsd).toBeCloseTo(viaView!, 10);
|
||||
});
|
||||
|
||||
test('an inherited-key model id cannot poison the running total', () => {
|
||||
// Both pricing tables are object literals, so 'constructor' resolves to a
|
||||
// truthy Object.prototype value and the rate arithmetic yields NaN. If
|
||||
// that reached cumulativeUsd, every later submit would pass the gate.
|
||||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||||
const poison = meter.check({ modelId: 'constructor', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'poison' });
|
||||
expect(poison.unpriced).toBe(true); // treated as unpriceable
|
||||
expect(Number.isFinite(poison.cumulativeCostUsd)).toBe(true);
|
||||
|
||||
const after = meter.check({
|
||||
modelId: 'claude-opus-4-7',
|
||||
estimatedInputTokens: 1_000_000,
|
||||
maxOutputTokens: 1_000_000,
|
||||
label: 'after-poison',
|
||||
});
|
||||
expect(after.allowed).toBe(false); // gate still enforcing
|
||||
expect(Number.isFinite(after.cumulativeCostUsd)).toBe(true);
|
||||
});
|
||||
|
||||
test('ledger captures every submit (allowed + denied + unpriced)', () => {
|
||||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||||
meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'a' });
|
||||
|
||||
Reference in New Issue
Block a user