mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
v0.44.1.0 feat(ai): recipes resolve any model — remove the native-recipe runtime allowlist (#4014)
* feat(ai): recipes resolve any model — remove the native-recipe runtime allowlist Frontier models ship weekly; a curated models: array can never stay current. assertTouchpoint now checks only the provider's touchpoint capability (anthropic has no embeddings, voyage has no chat) and never gates on the model id. Any id the user names goes to the provider, which is the real authority on what exists — a nonexistent model surfaces as the provider's own model_not_found at call time, and gbrain models doctor live-probes the configured models for a pre-flight check. With the gate gone, the entire extendedModels bypass machinery is dead and deleted: the _extendedModels registry, registerExtendedModel, registerConfigSelectedChatModel (+ its one caller in the contextual-reindex handler), both registration loops, and the tier-resolution loop that existed only to feed them. This also structurally closes the per-task-key gap where models.think / models.dream.* / facts.extraction_model selections were rejected while identical models.default selections worked. Recipe models: arrays remain informational — models[0] default selection for --model <provider> shorthand, guard-test fixtures pinning the repo's own hardcoded defaults, and gbrain providers list display. gateway.rerank() keeps its own model-list check deliberately: each listed reranker id maps to a known request/response wire shape. unknown_model still fires for providers lacking the touchpoint, so every probe reason stays reachable; tests that pinned the allowlist rejection now pin the pass-through contract (or the missing-touchpoint trigger). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v0.44.1.0 feat(ai): recipes resolve any model — remove the native-recipe runtime allowlist Frontier models ship weekly; a curated models: array can never stay current. assertTouchpoint now checks only the provider's touchpoint capability and never gates on the model id — any id the user names goes to the provider, and a nonexistent one surfaces as the provider's own model_not_found at call time (gbrain models doctor stays the token-free pre-flight). The extendedModels bypass machinery is deleted end to end, which also closes the per-task-key gap: models.think / models.dream.* / facts.extraction_model selections now behave exactly like models.default. think's graceful sentinel surfaces the thrown AIConfigError's own message + fix instead of generic key advice, so a provider 4xx is never key-blamed. gateway.rerank() keeps its own model-list check (each listed id maps to a known wire shape). Ship chores: VERSION/package.json → 0.44.1.0, CHANGELOG entry, gitleaks action pin refreshed to current v2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sina Matian
co-authored by
Claude Fable 5
Garry Tan
Claude Fable 5
Garry Tan
parent
52306ed438
commit
fc310db3ea
@@ -87,7 +87,7 @@ jobs:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2
|
||||
- uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
|
||||
@@ -2,6 +2,56 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.44.1.0] - 2026-08-11
|
||||
|
||||
**Any current model works now. gbrain stops rejecting model ids it hasn't heard of.**
|
||||
|
||||
New models ship every week. Until now, gbrain kept a built-in list of "known" models for each major provider (Anthropic, OpenAI, Google), and if you configured a model that list hadn't learned yet, gbrain refused to run it, even when the provider was already serving that model to everyone else. In practice that meant a brand-new model like `openai:gpt-5.6-sol` read as "not available" on an install whose binary shipped three weeks earlier, and the error message steered people toward older models instead. Worse, the workaround was inconsistent: the same model id worked when set as `models.default` but was rejected when set as `models.think`, for no reason a user could see.
|
||||
|
||||
That whole class of failure is gone. gbrain now checks only what a provider can do (Anthropic has no embedding models, Voyage has no chat models). Which model id you use is your call. If you name one that does not exist, the provider says so at call time, in its own words, and gbrain shows you that message instead of guessing.
|
||||
|
||||
How to use it:
|
||||
|
||||
```bash
|
||||
gbrain config set models.default openai:gpt-5.6-sol # any current model id
|
||||
gbrain config set models.think google:gemini-3.6-flash # per-task keys work identically now
|
||||
gbrain models doctor # pre-flight: probes your configured models live
|
||||
```
|
||||
|
||||
What changed in each case:
|
||||
|
||||
| You do | Before | Now |
|
||||
|---|---|---|
|
||||
| Configure a model newer than your gbrain binary | Rejected: "not listed... Known models: ..." | Runs |
|
||||
| Set that model via `models.think` / `models.dream.*` | Rejected even when `models.default` worked | Identical behavior on every key |
|
||||
| Typo a model id | Caught instantly, locally | Fails at the provider with the provider's own message; `gbrain models doctor` still catches it pre-flight without spending tokens |
|
||||
| Use a chat model from an embeddings-only provider | Rejected | Still rejected (that check is about the provider, not the model) |
|
||||
|
||||
Things to watch: a typo'd model id now costs one failed provider call instead of failing free and instantly. Run `gbrain models doctor` after changing model config if you want the old fail-fast feel. `gbrain think`'s fallback answer now carries the provider's actual error text, so a bad model id no longer reads as an API-key problem.
|
||||
|
||||
## To take advantage of v0.44.1.0
|
||||
|
||||
No migration, no schema change. `gbrain upgrade` is enough.
|
||||
|
||||
1. **Set any current model:**
|
||||
```bash
|
||||
gbrain config set models.default <provider>:<model>
|
||||
```
|
||||
2. **Verify:**
|
||||
```bash
|
||||
gbrain models doctor
|
||||
```
|
||||
3. **If a model you know is real still fails,** the error now comes from the provider; check the id spelling and your key, then file an issue at https://github.com/garrytan/gbrain/issues with the `gbrain models doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/ai/model-resolver.ts` — `assertTouchpoint(recipe, touchpoint, modelId)` checks provider touchpoint capability only; the native-recipe model allowlist throw is removed. Recipe `models:` arrays remain as data: `models[0]` default selection for `--model <provider>` shorthand and env-ready pickers, guard-test fixtures for the repo's own hardcoded defaults, and `gbrain providers list` display.
|
||||
- `src/core/ai/gateway.ts` — the extended-models registry (`_extendedModels`, `registerExtendedModel`, `registerConfigSelectedChatModel`, both registration loops, and the tier-resolution loop that fed them) is deleted. Every per-task model key (`models.think`, `models.dream.*`, `facts.extraction_model`, ...) now behaves exactly like `models.default`. `gateway.rerank()` keeps its own model-list check: each listed reranker id maps to a known request/response wire shape.
|
||||
- `src/core/think/index.ts` — the graceful "no LLM available" sentinel surfaces the thrown `AIConfigError`'s own message and fix (which key is missing, or what the provider rejected) instead of generic key advice.
|
||||
- `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — drops the now-dead chat-model registration call.
|
||||
- Tests — `test/gateway-tier-extended-models.test.ts` deleted (pinned the removed machinery); rejection tests across `test/ai/`, `test/think-*`, and `test/cycle/` now pin the pass-through contract; `unknown_model` still fires for providers lacking the touchpoint, so every probe reason stays reachable; new test pins the sentinel carrying the provider's error text.
|
||||
- Docs — `docs/architecture/KEY_FILES.md` entries for the resolver, gateway, and reindex handler updated to the new contract.
|
||||
|
||||
## [0.44.0.0] - 2026-06-12
|
||||
|
||||
**BrainBench: agent memory now has a scorecard.** `gbrain eval brainbench` is a public, reproducible, cross-harness conformance suite for the four ways agent memory fails — and from this release forward, every memory PR must hold or move its numbers against a committed baseline that CI compares against master's own copy.
|
||||
|
||||
@@ -2589,8 +2589,9 @@ contributor traps.
|
||||
- [ ] **P2: Document `FREE_LOCAL_RERANK_PROVIDERS` invariant.** `src/core/budget/budget-tracker.ts:lookupPricing`
|
||||
returns `{input:0, output:0}` for any model id under the `llama-server-reranker:`
|
||||
provider on the rerank kind. The contract relies on all callers going through
|
||||
`gateway.rerank()`'s `assertTouchpoint`-with-extended-models check (which validates
|
||||
the model exists before pricing fires). Theoretical bypass: a future caller that
|
||||
`gateway.rerank()`'s own model-list check (rerank-specific; it validates the
|
||||
model exists before pricing fires — note this was never `assertTouchpoint`,
|
||||
which checks provider touchpoints only). Theoretical bypass: a future caller that
|
||||
reserves directly against BudgetTracker with `kind: 'rerank'` and an arbitrary
|
||||
`llama-server-reranker:<anything>` model id gets free pricing. Fix: code comment
|
||||
documenting the invariant, OR move the freeness check to gateway.rerank() where
|
||||
|
||||
@@ -143,8 +143,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/brainstorm/checkpoint.ts` — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB/run) so resume MERGES pre-crash ideas with post-resume ideas before the judge runs (a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume <run_id>` covers both failed AND never-attempted crosses); `--list-runs` prints run_ids mtime-newest-first; `--force-resume` bypasses the 7-day staleness gate. Cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by `test/e2e/brainstorm-resume.test.ts` (20 unit + 3 E2E cases incl. the merge contract).
|
||||
- `src/core/remediation-checkpoint.ts` — `doctor --remediate` checkpoint at `~/.gbrain/remediation/<plan_hash>.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned, atomic `.tmp + rename`. `gbrain doctor --remediate --resume <plan_hash>` (no arg picks newest matching) loads it and skips completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases.
|
||||
- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). Four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. 8-step resolution chain: cliFlag → deprecated key → config key → `models.default` → `models.tier.<tier>` → env var → `TIER_DEFAULTS[tier]` → caller fallback. `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern (routes through `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids like `anthropic/claude-sonnet-4-6` classify correctly). `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves non-Anthropic, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` (the Anthropic Messages API tool-loop can't run on OpenAI/Gemini). `_resetDeprecationWarningsForTest()` also clears `_subagentTierWarningsEmitted`. Pinned by `test/model-config.serial.test.ts`.
|
||||
- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` takes an optional 4th `extendedModels: ReadonlySet<string>`: when the modelId is in that set the native-recipe allowlist throw is bypassed (user explicitly opted in via config, so provider rejection surfaces as `model_not_found` at HTTP call time and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — source typos still fail fast (the fail-fast contract for chat + expand + embed stays intact).
|
||||
- `src/core/ai/gateway.ts` extension — module-scoped `_extendedModels: Map<providerId, Map<touchpoint, Set<modelId>>>` registry feeds `assertTouchpoint`'s extended-model path without broadening unrelated surfaces. `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `registerConfigSelectedChatModel(model)` is the narrow runtime seam for a dedicated contextual-synopsis model: the ID joins the chat allowlist but remains rejected for embedding, expansion, and reranking. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport.
|
||||
- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. `assertTouchpoint(recipe, touchpoint, modelId)` checks the PROVIDER's capability only (anthropic has no embedding touchpoint; voyage/ollama have no chat) and never gates on the model id. Recipe `models:` arrays are informational — default-model selection (`models[0]` for `--model <provider>` shorthand and env-ready pickers), guard-test fixtures for the repo's own hardcoded defaults, and `gbrain providers list` display — NOT a runtime allowlist, so frontier models newer than a recipe work without a recipe PR. A nonexistent id surfaces as the provider's own `model_not_found` at call time; `gbrain models doctor` live-probes the configured models for a pre-flight check. Exception: `gateway.rerank()` keeps its own model-list check because each listed reranker id maps to a known request/response wire shape.
|
||||
- `src/core/ai/gateway.ts` extension — `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport.
|
||||
- `src/core/minions/queue.ts` extension — `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`.
|
||||
- `src/commands/models.ts` — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain), every per-task override (13 `PER_TASK_KEYS`, now including provider-neutral `models.contextual_synopsis` with legacy-key/env attribution), the alias map, and a source-of-truth column (`default` / `config: <key>` / `env: <VAR>`). `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. Wired into `cli.ts` dispatch + `CLI_ONLY` set. A zero-token `embedding_config` probe runs FIRST, before any chat/expansion probes spend money: `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. `ProbeStatus` variant `'config'` + optional `fix?: string` on `ProbeResult` surface a paste-ready `gbrain config set ...` line in human + JSON output; touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`.
|
||||
- `src/core/init-embed-check.ts` — embedding-key validation at `gbrain init`. `runInitEmbedCheck(opts)` runs a config-only `diagnoseEmbedding` (catches a missing key for ANY provider) plus a best-effort `liveTestEmbed` (1-token `gateway.embed(['probe'], {inputType:'query', abortSignal})`, 5s `AbortController` timeout, never throws — catches an invalid/expired key). Loud warning to stderr; init still exits 0 (`--no-embedding` is the deferred-setup escape; `--skip-embed-check` / `GBRAIN_INIT_SKIP_EMBED_CHECK=1` skip the check). Builds the effective env (`process.env` + file-plane `openai/anthropic/zeroentropy_api_key` from `loadConfigFileOnly()` + `opts.apiKey`) and configures the gateway via `buildGatewayConfig` before diagnose/probe, so the check sees the same keys AND provider base URLs runtime will (no false "missing key" for config.json-keyed users; the probe hits the right endpoint). Init-specific warning text names `--no-embedding` / `--skip-embed-check`, not the sync-flavored `--no-embed`. Wired into `initPGLite` + `initPostgres` in `src/commands/init.ts`, with the result added to the `--json` envelope as `embedding_check {ok, reason?, live_ok?}`. Pinned by `test/init-embed-check.test.ts` (hermetic via the gateway embed-transport seam + `withEnv`).
|
||||
@@ -272,7 +272,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` — `subagent_aggregator` handler. Claims AFTER all children resolve (queue guarantees every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds a deterministic mixed-outcome markdown summary. No LLM call.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback for `gbrain agent logs`.
|
||||
- `src/core/minions/rate-leases.ts` — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
|
||||
- `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — per-page contextual re-embed handler. Resolves `models.contextual_synopsis` once, registers native chat models at the chat touchpoint, and isolates cross-worker leases by the full resolved model id. `GBRAIN_CONTEXTUAL_SYNOPSIS_RPM` controls the cap; `GBRAIN_CONTEXTUAL_HAIKU_RPM` is the compatibility alias.
|
||||
- `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — per-page contextual re-embed handler. Resolves `models.contextual_synopsis` once and isolates cross-worker leases by the full resolved model id. `GBRAIN_CONTEXTUAL_SYNOPSIS_RPM` controls the cap; `GBRAIN_CONTEXTUAL_HAIKU_RPM` is the compatibility alias.
|
||||
- `src/core/minions/wait-for-completion.ts` — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/transcript.ts` — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
|
||||
- `src/core/minions/plugin-loader.ts` — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
|
||||
|
||||
+1
-1
@@ -149,7 +149,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.44.0.0",
|
||||
"version": "0.44.1.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.5",
|
||||
|
||||
@@ -85,12 +85,10 @@ export function getProviderCapabilities(modelString: string): ProviderCapabiliti
|
||||
);
|
||||
}
|
||||
|
||||
// For native providers, the model must be in the recipe's allow-list. For
|
||||
// openai-compatible recipes (litellm, ollama, llama-server), arbitrary model
|
||||
// ids are accepted because the gateway behind the proxy decides what's real.
|
||||
// We don't error here — `assertTouchpoint` already enforces this at gateway
|
||||
// boundary; this function returns capabilities for whatever the user asked
|
||||
// for, on the assumption it'll be validated elsewhere.
|
||||
// Model ids are never validated against recipe model lists (any id goes to
|
||||
// the provider, which is the real authority on what exists). This function
|
||||
// returns capabilities for whatever the user asked for; a nonexistent model
|
||||
// surfaces as the provider's own model_not_found at call time.
|
||||
|
||||
const promptCache = chat.supports_prompt_cache;
|
||||
|
||||
|
||||
+26
-124
@@ -51,7 +51,7 @@ import {
|
||||
OPENROUTER_CACHE_HEADER,
|
||||
openrouterRequiresExplicitPromptCache,
|
||||
} from './recipes/openrouter.ts';
|
||||
import { resolveModel, TIER_DEFAULTS } from '../model-config.ts';
|
||||
import { resolveModel } from '../model-config.ts';
|
||||
import { parseLlmJson } from '../llm-json.ts';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { dimsProviderOptions } from './dims.ts';
|
||||
@@ -136,71 +136,6 @@ export function configureGatewayIfUninitialized(): void {
|
||||
if (config) configureGateway(buildGatewayConfig(config));
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.12 recipe-models merge: per-gateway-instance set of model ids the
|
||||
* user opted into via config. Keyed by provider id (`anthropic`, `openai`,
|
||||
* etc.) AND touchpoint so a chat-only selection cannot silently authorize
|
||||
* expansion/embedding/reranker. Passed into `assertTouchpoint` so native-recipe
|
||||
* allowlist checks skip these models — provider 404s surface at HTTP call time
|
||||
* instead of config-build time.
|
||||
*
|
||||
* Replaces the earlier plan to soften `assertTouchpoint` from throw to
|
||||
* warn (Codex F4/F5 — too broad, removed fail-fast for chat/expand/embed
|
||||
* across all callers). This narrower approach preserves fail-fast for
|
||||
* source-code typos while allowing config-time model selection of any id.
|
||||
*/
|
||||
const _extendedModels: Map<string, Map<TouchpointKind, Set<string>>> = new Map();
|
||||
|
||||
/**
|
||||
* v0.31.12 — register a model id under its provider+touchpoint so
|
||||
* `assertTouchpoint` (called via the gateway's chat/embed/expand entry points)
|
||||
* permits it there even when it isn't in the recipe's declared `models:` array.
|
||||
*
|
||||
* Idempotent + safe to call before/after configureGateway. Exported only
|
||||
* for the `gbrain models doctor` probe path (where the operator may want
|
||||
* to probe any user-supplied id without re-running configure).
|
||||
*/
|
||||
function registerExtendedModel(touchpoint: TouchpointKind, modelStr: string): void {
|
||||
if (!modelStr) return;
|
||||
try {
|
||||
const { providerId, modelId } = parseModelId(modelStr);
|
||||
let byTouchpoint = _extendedModels.get(providerId);
|
||||
if (!byTouchpoint) {
|
||||
byTouchpoint = new Map();
|
||||
_extendedModels.set(providerId, byTouchpoint);
|
||||
}
|
||||
let set = byTouchpoint.get(touchpoint);
|
||||
if (!set) {
|
||||
set = new Set();
|
||||
byTouchpoint.set(touchpoint, set);
|
||||
}
|
||||
set.add(modelId);
|
||||
} catch {
|
||||
// Malformed model strings will fail at parseModelId — ignore here;
|
||||
// the actual chat/embed call will surface the error.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a model that was selected through a DB/env config resolver for a
|
||||
* chat-backed call site outside the gateway's built-in chat/expansion defaults.
|
||||
*
|
||||
* This preserves `assertTouchpoint`'s native-provider fail-fast behavior for
|
||||
* hardcoded source models while allowing an explicit operator-selected chat
|
||||
* model (for example `models.contextual_synopsis`) to reach the provider even
|
||||
* when the recipe's curated model list has not yet learned the new id.
|
||||
*/
|
||||
export function registerConfigSelectedChatModel(modelStr: string): void {
|
||||
registerExtendedModel('chat', modelStr);
|
||||
}
|
||||
|
||||
function getExtendedModelsForProvider(
|
||||
providerId: string,
|
||||
touchpoint: TouchpointKind,
|
||||
): ReadonlySet<string> | undefined {
|
||||
return _extendedModels.get(providerId)?.get(touchpoint);
|
||||
}
|
||||
|
||||
/**
|
||||
* The function the gateway calls to actually run a batch through the AI SDK.
|
||||
* Defaults to the imported `embedMany`. Tests inject a stub via
|
||||
@@ -516,17 +451,6 @@ export function configureGateway(config: AIGatewayConfig): void {
|
||||
};
|
||||
_modelCache.clear();
|
||||
_shrinkState.clear();
|
||||
_extendedModels.clear();
|
||||
// Register configured models so assertTouchpoint allows them even when
|
||||
// they aren't in the recipe's declared models: array (v0.31.12).
|
||||
if (_config.embedding_model) registerExtendedModel('embedding', _config.embedding_model);
|
||||
if (_config.embedding_multimodal_model) registerExtendedModel('embedding', _config.embedding_multimodal_model);
|
||||
if (_config.expansion_model) registerExtendedModel('expansion', _config.expansion_model);
|
||||
if (_config.chat_model) registerExtendedModel('chat', _config.chat_model);
|
||||
if (_config.reranker_model) registerExtendedModel('reranker', _config.reranker_model);
|
||||
for (const m of _config.chat_fallback_chain ?? []) {
|
||||
if (m) registerExtendedModel('chat', m);
|
||||
}
|
||||
warnRecipesMissingBatchTokens();
|
||||
}
|
||||
|
||||
@@ -572,35 +496,9 @@ export async function reconfigureGatewayWithEngine(engine: BrainEngine): Promise
|
||||
const expansionFull = newExpansion.includes(':') ? newExpansion : prefixWithProviderFrom(cfg.expansion_model ?? DEFAULT_EXPANSION_MODEL, newExpansion);
|
||||
const chatFull = newChat.includes(':') ? newChat : prefixWithProviderFrom(cfg.chat_model ?? DEFAULT_CHAT_MODEL, newChat);
|
||||
|
||||
// ALSO resolve the four tier models and register them as extended models.
|
||||
// assertTouchpoint's contract (model-resolver.ts) says config-chosen models —
|
||||
// `models.default` and `models.tier.*` included — bypass the native recipe
|
||||
// allowlist, but pre-fix only chat/expansion/embedding/reranker were
|
||||
// registered. A model reachable ONLY through a tier (e.g. `models.tier.deep`
|
||||
// set to an Opus newer than the recipe list) failed `probeChatModel` at call
|
||||
// time and silently degraded think/auto_think to the gather-only stub.
|
||||
// Resolving per-tier also honors `models.default` (it sits above tiers in
|
||||
// the resolveModel chain).
|
||||
const tierModels: string[] = [];
|
||||
for (const tier of ['utility', 'reasoning', 'deep', 'subagent'] as const) {
|
||||
tierModels.push(await resolveModel(engine, { tier, fallback: TIER_DEFAULTS[tier] }));
|
||||
}
|
||||
|
||||
_config = { ...cfg, expansion_model: expansionFull, chat_model: chatFull };
|
||||
_modelCache.clear();
|
||||
_shrinkState.clear();
|
||||
_extendedModels.clear();
|
||||
if (_config.embedding_model) registerExtendedModel('embedding', _config.embedding_model);
|
||||
if (_config.embedding_multimodal_model) registerExtendedModel('embedding', _config.embedding_multimodal_model);
|
||||
if (_config.expansion_model) registerExtendedModel('expansion', _config.expansion_model);
|
||||
if (_config.chat_model) registerExtendedModel('chat', _config.chat_model);
|
||||
if (_config.reranker_model) registerExtendedModel('reranker', _config.reranker_model);
|
||||
for (const m of _config.chat_fallback_chain ?? []) {
|
||||
if (m) registerExtendedModel('chat', m);
|
||||
}
|
||||
for (const m of tierModels) {
|
||||
if (m) registerExtendedModel('chat', m);
|
||||
}
|
||||
return _config;
|
||||
}
|
||||
|
||||
@@ -706,20 +604,19 @@ function clearGatewayState(): void {
|
||||
_embedTransportInstalled = false;
|
||||
_chatTransport = null;
|
||||
_warnedRecipes.clear();
|
||||
_extendedModels.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset (for tests). Clears all module state (config, model cache, shrink
|
||||
* state, transports, warned recipes, extended models), then — if a test
|
||||
* baseline is registered — re-applies it so the gateway returns to the
|
||||
* process-wide test default instead of an unconfigured limbo (#3554).
|
||||
* state, transports, warned recipes), then — if a test baseline is
|
||||
* registered — re-applies it so the gateway returns to the process-wide
|
||||
* test default instead of an unconfigured limbo (#3554).
|
||||
*/
|
||||
export function resetGateway(): void {
|
||||
clearGatewayState();
|
||||
// configureGateway re-clears _modelCache/_shrinkState/_extendedModels and
|
||||
// registers the baseline's models; transports are NOT touched by it, so a
|
||||
// stale test transport can never leak back in through this path.
|
||||
// configureGateway re-clears _modelCache/_shrinkState; transports are NOT
|
||||
// touched by it, so a stale test transport can never leak back in through
|
||||
// this path.
|
||||
if (_resetBaseline) configureGateway(_resetBaseline());
|
||||
}
|
||||
|
||||
@@ -1505,7 +1402,7 @@ export const perplexityCompatFetch = (async (input: RequestInfo | URL, init?: Re
|
||||
|
||||
async function resolveEmbeddingProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
|
||||
const { parsed, recipe } = resolveRecipe(modelStr);
|
||||
assertTouchpoint(recipe, 'embedding', parsed.modelId, getExtendedModelsForProvider(parsed.providerId, 'embedding'));
|
||||
assertTouchpoint(recipe, 'embedding', parsed.modelId);
|
||||
const cfg = requireConfig();
|
||||
|
||||
const cacheKey = `emb:${recipe.id}:${parsed.modelId}:${cfg.base_urls?.[recipe.id] ?? ''}`;
|
||||
@@ -2464,7 +2361,7 @@ export async function embedMultimodalSafe(
|
||||
|
||||
async function resolveExpansionProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
|
||||
const { parsed, recipe } = resolveRecipe(modelStr);
|
||||
assertTouchpoint(recipe, 'expansion', parsed.modelId, getExtendedModelsForProvider(parsed.providerId, 'expansion'));
|
||||
assertTouchpoint(recipe, 'expansion', parsed.modelId);
|
||||
const cfg = requireConfig();
|
||||
|
||||
const cacheKey = `exp:${recipe.id}:${parsed.modelId}:${cfg.base_urls?.[recipe.id] ?? ''}`;
|
||||
@@ -2634,7 +2531,7 @@ export async function expand(query: string): Promise<string[]> {
|
||||
|
||||
/**
|
||||
* Cherry-1: opt-in OCR pass for ingested images. Uses the configured
|
||||
* expansion model (default: openai:gpt-4o-mini) with a prompt explicitly
|
||||
* expansion model (default: DEFAULT_EXPANSION_MODEL) with a prompt explicitly
|
||||
* instructing the model to NOT interpret instructions embedded in the
|
||||
* image (mitigation for OCR-as-prompt-injection).
|
||||
*
|
||||
@@ -2949,14 +2846,18 @@ export interface ChatOpts {
|
||||
* (via `probeChatModel`) AND `makeJudgeClient` in `cycle/synthesize.ts`.
|
||||
*
|
||||
* Validates that a `provider:model` string resolves to a real recipe AND that the
|
||||
* recipe supports the chat touchpoint (catches typo'd native models like
|
||||
* `anthropic:claude-bogus-9`). Both checks read the recipe REGISTRY, not gateway
|
||||
* `_config`, so this works before `configureGateway()` has run — which is why
|
||||
* `makeJudgeClient` reuses this layer instead of the full `probeChatModel` (whose
|
||||
* `isAvailable` layer would reject non-Anthropic-no-key + unconfigured-gateway).
|
||||
* recipe supports the chat touchpoint (catches chat-less providers like
|
||||
* `voyage:*` / `ollama:*` embeddings-only recipes — NOT model-id typos: there
|
||||
* is no runtime model allowlist, so an unlisted id passes here and a
|
||||
* nonexistent one surfaces as the provider's own model_not_found at call
|
||||
* time). Both checks read the recipe REGISTRY, not gateway `_config`, so this
|
||||
* works before `configureGateway()` has run — which is why `makeJudgeClient`
|
||||
* reuses this layer instead of the full `probeChatModel` (whose `isAvailable`
|
||||
* layer would reject non-Anthropic-no-key + unconfigured-gateway).
|
||||
*
|
||||
* Order matters: `resolveRecipe` first (unknown_provider), then `assertTouchpoint`
|
||||
* (unknown_model). `isAvailable` alone collapses both into a bare `false`.
|
||||
* (unknown_model = provider lacks the touchpoint). `isAvailable` alone collapses
|
||||
* both into a bare `false`.
|
||||
*/
|
||||
export type ModelIdValidity =
|
||||
| { ok: true; parsed: ParsedModelId; recipe: Recipe }
|
||||
@@ -2975,7 +2876,7 @@ export function validateModelId(
|
||||
throw e;
|
||||
}
|
||||
try {
|
||||
assertTouchpoint(recipe, touchpoint, parsed.modelId, getExtendedModelsForProvider(parsed.providerId, touchpoint));
|
||||
assertTouchpoint(recipe, touchpoint, parsed.modelId);
|
||||
} catch (e) {
|
||||
if (e instanceof AIConfigError) return { ok: false, reason: 'unknown_model', detail: e.message, fix: e.fix };
|
||||
throw e;
|
||||
@@ -3029,7 +2930,7 @@ function chatSupportsPromptCache(recipe: Recipe, modelId: string): boolean {
|
||||
|
||||
async function resolveChatProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
|
||||
const { parsed, recipe } = resolveRecipe(modelStr);
|
||||
assertTouchpoint(recipe, 'chat', parsed.modelId, getExtendedModelsForProvider(parsed.providerId, 'chat'));
|
||||
assertTouchpoint(recipe, 'chat', parsed.modelId);
|
||||
const cfg = requireConfig();
|
||||
|
||||
const cacheKey = `chat:${recipe.id}:${parsed.modelId}:${cfg.base_urls?.[recipe.id] ?? ''}`;
|
||||
@@ -3942,9 +3843,10 @@ const DEFAULT_RERANK_TIMEOUT_MS = 5000;
|
||||
* the fail-open path so search never throws.
|
||||
*
|
||||
* Errors classified into RerankError.reason for the caller's fail-open
|
||||
* decision table. The model allowlist check is done HERE (not via
|
||||
* assertTouchpoint), because assertTouchpoint doesn't enforce allowlists for
|
||||
* openai-compatible recipes — CDX2-F11 in the plan.
|
||||
* decision table. The model list check below is rerank-specific and
|
||||
* deliberate (assertTouchpoint never checks model ids): each listed reranker
|
||||
* model maps to a known request/response wire shape, so an unknown id could
|
||||
* mis-parse a response rather than fail cleanly.
|
||||
*/
|
||||
export async function rerank(input: RerankInput): Promise<RerankResult[]> {
|
||||
if (!input.query) {
|
||||
|
||||
@@ -89,27 +89,19 @@ function getTouchpoint(recipe: Recipe, touchpoint: TouchpointKind): EmbeddingTou
|
||||
/**
|
||||
* Assert the resolved recipe actually offers the requested touchpoint.
|
||||
*
|
||||
* @param extendedModels Per-gateway-instance Set of additional models the
|
||||
* user opted into via `cfg.chat_model` / `cfg.embedding_model` /
|
||||
* `cfg.expansion_model` / `models.default` / `models.tier.*`. When the
|
||||
* modelId is in this set, the native-recipe allowlist check is skipped
|
||||
* (the user explicitly chose this model via config — provider rejection
|
||||
* surfaces at HTTP call time, with a clear `model_not_found` from the
|
||||
* provider).
|
||||
*
|
||||
* Default code paths (hardcoded model strings in source code) MUST NOT
|
||||
* pass this argument — typos in code still fail fast. Only config-derived
|
||||
* model selection extends the allowlist.
|
||||
*
|
||||
* v0.31.12 — replaces the earlier plan to soften the validator from throw
|
||||
* to warn (which would have removed the fail-fast contract for chat/expand/
|
||||
* embed all three; per Codex F4/F5 in plan review).
|
||||
* This checks the PROVIDER's capability (anthropic has no embeddings; voyage
|
||||
* has no chat), never the model id. Recipe `models:` arrays are informational
|
||||
* — defaults for `--model <provider>` shorthand, guard-test fixtures for the
|
||||
* repo's own hardcoded defaults, display in `gbrain providers list` — not a
|
||||
* runtime allowlist. Frontier models ship weekly; any id the user names goes
|
||||
* to the provider, and a nonexistent one surfaces as the provider's own
|
||||
* `model_not_found` at call time (`gbrain models doctor` probes the configured
|
||||
* models live for a pre-flight check).
|
||||
*/
|
||||
export function assertTouchpoint(
|
||||
recipe: Recipe,
|
||||
touchpoint: TouchpointKind,
|
||||
modelId: string,
|
||||
extendedModels?: ReadonlySet<string>,
|
||||
): void {
|
||||
const tp = getTouchpoint(recipe, touchpoint);
|
||||
if (!tp) {
|
||||
@@ -122,23 +114,6 @@ export function assertTouchpoint(
|
||||
: undefined,
|
||||
);
|
||||
}
|
||||
const supportedModels = tp.models ?? [];
|
||||
if (supportedModels.length > 0 && !supportedModels.includes(modelId)) {
|
||||
// Non-fatal: providers like ollama/litellm accept arbitrary model ids. We only warn for native providers.
|
||||
if (recipe.tier === 'native') {
|
||||
// v0.31.12 recipe-models merge: if the user opted into this model via
|
||||
// config (cfg.chat_model, models.default, models.tier.*), skip the
|
||||
// throw. The model goes to the provider; provider 404s surface as
|
||||
// `model_not_found` via `gbrain models doctor`.
|
||||
if (extendedModels && extendedModels.has(modelId)) {
|
||||
return;
|
||||
}
|
||||
throw new AIConfigError(
|
||||
`Model "${modelId}" is not listed for ${recipe.name} ${touchpoint}.`,
|
||||
`Known models: ${supportedModels.join(', ')}. Use one of these or add it to the recipe (or add an alias).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function knownProviderIds(): string[] {
|
||||
|
||||
@@ -974,7 +974,8 @@ export function makeJudgeClient(verdictModel: string): JudgeClient | null {
|
||||
const modelStr = normalizeModelId(verdictModel);
|
||||
|
||||
// #1698 (C1): id-validity via the shared `validateModelId` core (resolveRecipe +
|
||||
// assertTouchpoint) — catches unknown provider AND typo'd native model. We do NOT
|
||||
// assertTouchpoint) — catches unknown provider AND chat-less provider (model-id
|
||||
// typos pass locally and fail at the provider; no runtime allowlist). We do NOT
|
||||
// use the full `probeChatModel` here: its `isAvailable` layer would reject
|
||||
// non-Anthropic-no-key providers and an unconfigured gateway, breaking the
|
||||
// deliberate per-transcript-degrade contract (and test A9). validateModelId reads
|
||||
|
||||
@@ -50,7 +50,6 @@ import {
|
||||
import { resolveSearchMode, loadSearchModeConfig } from '../../search/mode.ts';
|
||||
import { resolveModel } from '../../model-config.ts';
|
||||
import { DEFAULT_SYNOPSIS_MODEL } from '../../page-summary.ts';
|
||||
import { registerConfigSelectedChatModel } from '../../ai/gateway.ts';
|
||||
|
||||
/**
|
||||
* Default global concurrency cap for contextual synopsis calls. The public
|
||||
@@ -168,7 +167,6 @@ export function makeContextualReindexHandler(opts: MakeContextualReindexHandlerO
|
||||
const chunkConcurrency = resolveContextualChunkConcurrency();
|
||||
const synopsisModel = await resolveContextualSynopsisModel(engine);
|
||||
const leaseSettings = resolveContextualSynopsisLeaseSettings(synopsisModel);
|
||||
registerConfigSelectedChatModel(synopsisModel);
|
||||
|
||||
const result: ReembedPageResult = await reembedPage({
|
||||
engine,
|
||||
|
||||
+22
-8
@@ -741,7 +741,9 @@ async function tryBuildGatewayClient(
|
||||
const modelStr = normalizeModelId(modelUsed);
|
||||
|
||||
// #1698: ONE shared probe (resolveRecipe + assertTouchpoint + isAvailable).
|
||||
// assertTouchpoint catches typo'd native models; isAvailable catches missing keys.
|
||||
// assertTouchpoint catches chat-less providers (voyage/ollama); isAvailable
|
||||
// catches missing keys. Model-id typos are NOT caught locally (no runtime
|
||||
// allowlist) — a nonexistent id fails at the provider with model_not_found.
|
||||
// For an EXPLICIT model the user typed, an unusable model is a HARD ERROR (throw)
|
||||
// — never silently degrade to the no-LLM stub. For the default/configured-model
|
||||
// path, return null so the caller falls through to the graceful "no LLM" stub
|
||||
@@ -787,7 +789,7 @@ async function tryBuildGatewayClient(
|
||||
// existing JSON-parse path produces the graceful degradation answer.
|
||||
if (e instanceof AIConfigError) {
|
||||
if (opts.explicitModel) throw e;
|
||||
return buildGracefulMessage(modelStr) as unknown as Anthropic.Message;
|
||||
return buildGracefulMessage(modelStr, e) as unknown as Anthropic.Message;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
@@ -836,12 +838,19 @@ function mapStopReason(s: ChatResult['stopReason']): 'end_turn' | 'max_tokens' |
|
||||
}
|
||||
|
||||
/**
|
||||
* Sentinel Message returned when gateway.chat throws AIConfigError (typically
|
||||
* missing API key for the resolved provider). The caller's JSON parser will
|
||||
* fail on this text, fall through to `LLM_OUTPUT_NOT_JSON`, and surface the
|
||||
* sentinel as the answer — matches the legacy graceful-degradation shape.
|
||||
* Sentinel Message returned when gateway.chat throws AIConfigError (missing
|
||||
* API key, or the provider rejecting the model/config with a 4xx — with no
|
||||
* runtime model allowlist, a nonexistent model id surfaces here as the
|
||||
* provider's model_not_found). The caller's JSON parser will fail on this
|
||||
* text, fall through to `LLM_OUTPUT_NOT_JSON`, and surface the sentinel as
|
||||
* the answer — matches the legacy graceful-degradation shape.
|
||||
*
|
||||
* When the thrown error is in hand, its own message + fix are surfaced (they
|
||||
* name the actual cause: which key is missing, or what the provider rejected)
|
||||
* instead of the generic key advice — the generic text key-blamed provider
|
||||
* 4xxs like model_not_found.
|
||||
*/
|
||||
function buildGracefulMessage(modelStr: string): {
|
||||
function buildGracefulMessage(modelStr: string, err?: AIConfigError): {
|
||||
id: string;
|
||||
type: 'message';
|
||||
role: 'assistant';
|
||||
@@ -855,7 +864,12 @@ function buildGracefulMessage(modelStr: string): {
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: modelStr,
|
||||
content: [{ type: 'text', text: '(no LLM available — set anthropic_api_key via gbrain config or ANTHROPIC_API_KEY env)' }],
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: err
|
||||
? `(no LLM available — ${err.message}${err.fix ? ` Fix: ${err.fix}` : ''})`
|
||||
: '(no LLM available — set anthropic_api_key via gbrain config or ANTHROPIC_API_KEY env)',
|
||||
}],
|
||||
usage: { input_tokens: 0, output_tokens: 0 },
|
||||
stop_reason: 'end_turn',
|
||||
};
|
||||
|
||||
@@ -176,14 +176,15 @@ describe('chat touchpoint — model resolver + aliases (Codex F-OV-5)', () => {
|
||||
.toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('assertTouchpoint rejects unknown native model with the model list in the fix hint', () => {
|
||||
try {
|
||||
assertTouchpoint(getRecipe('anthropic')!, 'chat', 'claude-opus-9-99');
|
||||
throw new Error('should have thrown');
|
||||
} catch (e) {
|
||||
expect(e).toBeInstanceOf(AIConfigError);
|
||||
expect((e as AIConfigError).message).toContain('claude-opus-9-99');
|
||||
}
|
||||
test('assertTouchpoint accepts unlisted models on native recipes (no runtime allowlist)', () => {
|
||||
// Frontier models ship weekly; recipe models: arrays are informational
|
||||
// (defaults, guard-test fixtures, display), not a gate. A nonexistent id
|
||||
// surfaces as the provider's own model_not_found at call time.
|
||||
expect(() => assertTouchpoint(getRecipe('anthropic')!, 'chat', 'claude-opus-9-99')).not.toThrow();
|
||||
expect(() => assertTouchpoint(getRecipe('openai')!, 'chat', 'gpt-5.6-sol')).not.toThrow();
|
||||
expect(() => assertTouchpoint(getRecipe('google')!, 'chat', 'gemini-9-flash')).not.toThrow();
|
||||
expect(() => assertTouchpoint(getRecipe('openai')!, 'expansion', 'gpt-5.6-luna')).not.toThrow();
|
||||
expect(() => assertTouchpoint(getRecipe('openai')!, 'embedding', 'text-embedding-9-huge')).not.toThrow();
|
||||
});
|
||||
|
||||
test('assertTouchpoint accepts arbitrary model on openai-compat tier', () => {
|
||||
|
||||
@@ -60,8 +60,13 @@ describe('validateModelId (#1698 C1 core)', () => {
|
||||
if (!v.ok) expect(v.reason).toBe('unknown_provider');
|
||||
});
|
||||
|
||||
test('unknown_model for a typo native model', () => {
|
||||
const v = validateModelId('anthropic:claude-bogus-9');
|
||||
test('ok for an unlisted native model (no runtime allowlist — provider decides)', () => {
|
||||
expect(validateModelId('anthropic:claude-bogus-9').ok).toBe(true);
|
||||
expect(validateModelId('openai:gpt-5.6-sol').ok).toBe(true);
|
||||
});
|
||||
|
||||
test('unknown_model when the provider lacks the touchpoint entirely', () => {
|
||||
const v = validateModelId('voyage:voyage-3', 'chat');
|
||||
expect(v.ok).toBe(false);
|
||||
if (!v.ok) expect(v.reason).toBe('unknown_model');
|
||||
});
|
||||
@@ -85,7 +90,10 @@ describe('probeChatModel (#1698 = validity + key, config-independent)', () => {
|
||||
test('unknown_provider / unknown_model classify regardless of key (validity runs first)', async () => {
|
||||
await withEnv(withKeyEnv(), async () => {
|
||||
expect(probeChatModel('bogusprovider:x')).toMatchObject({ ok: false, reason: 'unknown_provider' });
|
||||
expect(probeChatModel('anthropic:claude-bogus-9')).toMatchObject({ ok: false, reason: 'unknown_model' });
|
||||
// unknown_model now fires only for a missing touchpoint (voyage has no
|
||||
// chat); unlisted ids on chat-capable providers pass local validation.
|
||||
expect(probeChatModel('voyage:voyage-3')).toMatchObject({ ok: false, reason: 'unknown_model' });
|
||||
expect(probeChatModel('anthropic:claude-bogus-9').ok).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -317,7 +317,6 @@ async function expectHandlerRegistersNativeSynopsisRoute(opts: {
|
||||
|
||||
expect(generatedModels).toEqual([expectedModel]);
|
||||
expect(validateModelId(expectedModel, 'chat').ok).toBe(true);
|
||||
expect(validateModelId(expectedModel, 'expansion').ok).toBe(false);
|
||||
}
|
||||
|
||||
async function captureModelsReport(engine: StubConfigEngine): Promise<{
|
||||
|
||||
@@ -87,12 +87,12 @@ describe('makeJudgeClient — construction-time provider probe', () => {
|
||||
});
|
||||
});
|
||||
|
||||
test('A8b (#1698): typo native model → null at construction (validateModelId unknown_model)', async () => {
|
||||
// Pre-#1698 makeJudgeClient only checked the provider (resolveRecipe) and would
|
||||
// have returned a client that failed at call time. Now the shared validateModelId
|
||||
// core runs assertTouchpoint, so a typo'd native model is rejected up front.
|
||||
test('A8b (#1698): chat-less-provider model → null at construction (validateModelId unknown_model)', async () => {
|
||||
// validateModelId rejects a provider with no chat touchpoint (voyage).
|
||||
// Unlisted ids on chat-capable providers pass local validation now (no
|
||||
// runtime allowlist) and fail at the provider instead.
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test-A8b' }, async () => {
|
||||
const judge = makeJudgeClient('anthropic:claude-bogus-9');
|
||||
const judge = makeJudgeClient('voyage:voyage-3');
|
||||
expect(judge).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
/**
|
||||
* reconfigureGatewayWithEngine — tier-resolved models join the extended set.
|
||||
*
|
||||
* assertTouchpoint's extended-models contract (model-resolver.ts) says models
|
||||
* the user opted into via config — `models.default` and `models.tier.*`
|
||||
* included — bypass the native recipe allowlist. Pre-fix, only chat/expansion/
|
||||
* embedding/reranker were registered, so a model reachable ONLY through a tier
|
||||
* (e.g. `models.tier.deep` set to an Opus newer than the recipe list) failed
|
||||
* `probeChatModel` and silently degraded think/auto_think to the gather-only
|
||||
* stub — mislabeled NO_ANTHROPIC_API_KEY.
|
||||
*
|
||||
* Uses a deliberately fictional model id so the test stays valid no matter how
|
||||
* current the recipe list is.
|
||||
*/
|
||||
import { describe, test, expect, afterEach } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
reconfigureGatewayWithEngine,
|
||||
resetGateway,
|
||||
validateModelId,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
function stubEngine(config: Record<string, string>): BrainEngine {
|
||||
return { getConfig: async (k: string) => config[k] ?? null } as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('reconfigureGatewayWithEngine — tier models extend the allowlist', () => {
|
||||
test('a models.tier.deep model unknown to the recipe validates after reconfigure', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { ANTHROPIC_API_KEY: 'sk-fake', OPENAI_API_KEY: 'sk-fake' },
|
||||
});
|
||||
// Pre-reconfigure: an id absent from the recipe allowlist is rejected.
|
||||
expect(validateModelId('anthropic:claude-hypothetical-9').ok).toBe(false);
|
||||
|
||||
await reconfigureGatewayWithEngine(
|
||||
stubEngine({ 'models.tier.deep': 'anthropic:claude-hypothetical-9' }),
|
||||
);
|
||||
|
||||
// Post-reconfigure: the tier-configured model is in the extended set.
|
||||
expect(validateModelId('anthropic:claude-hypothetical-9').ok).toBe(true);
|
||||
// An id configured NOWHERE stays rejected — the allowlist still bites.
|
||||
expect(validateModelId('anthropic:claude-never-configured-1').ok).toBe(false);
|
||||
});
|
||||
|
||||
test('models.default reaches the extended set through tier resolution', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { ANTHROPIC_API_KEY: 'sk-fake', OPENAI_API_KEY: 'sk-fake' },
|
||||
});
|
||||
await reconfigureGatewayWithEngine(
|
||||
stubEngine({ 'models.default': 'anthropic:claude-hypothetical-10' }),
|
||||
);
|
||||
expect(validateModelId('anthropic:claude-hypothetical-10').ok).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -18,6 +18,7 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { __thinkAdapter } from '../src/core/think/index.ts';
|
||||
import { resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { AIConfigError } from '../src/core/ai/errors.ts';
|
||||
import { withEnv, emptyHome } from './helpers/with-env.ts';
|
||||
|
||||
describe('think gateway adapter — response shape conversion', () => {
|
||||
@@ -108,9 +109,11 @@ describe('think gateway adapter — #1698 slash form + explicit-model fork', ()
|
||||
).rejects.toThrow(/not usable.*unknown_provider/);
|
||||
});
|
||||
|
||||
test('explicit typo native model THROWS (unknown_model)', async () => {
|
||||
test('explicit chat-less-provider model THROWS (unknown_model)', async () => {
|
||||
// voyage has no chat touchpoint. Unlisted ids on chat-capable providers
|
||||
// pass local validation (no runtime allowlist) — the provider decides.
|
||||
await expect(
|
||||
__thinkAdapter.tryBuildGatewayClient('anthropic:claude-bogus-9', { explicitModel: true }),
|
||||
__thinkAdapter.tryBuildGatewayClient('voyage:voyage-3', { explicitModel: true }),
|
||||
).rejects.toThrow(/not usable.*unknown_model/);
|
||||
});
|
||||
|
||||
@@ -215,4 +218,18 @@ describe('think gateway adapter — graceful fallback shape', () => {
|
||||
expect(m.usage.output_tokens).toBe(0);
|
||||
expect(m.stop_reason).toBe('end_turn');
|
||||
});
|
||||
|
||||
test('the sentinel surfaces the AIConfigError cause, not generic key advice', () => {
|
||||
// With no runtime model allowlist, a nonexistent model id reaches the
|
||||
// provider and comes back as a 4xx wrapped in AIConfigError. The sentinel
|
||||
// must name that cause — the generic text key-blamed model problems.
|
||||
const err = new AIConfigError(
|
||||
'[chat(openai:gpt-bogus)] The model `gpt-bogus` does not exist',
|
||||
'Check your model id + provider options match the provider API.',
|
||||
);
|
||||
const m = __thinkAdapter.buildGracefulMessage('openai:gpt-bogus', err);
|
||||
expect(m.content[0].text).toContain('does not exist');
|
||||
expect(m.content[0].text).toContain('Check your model id');
|
||||
expect(m.content[0].text).not.toContain('ANTHROPIC_API_KEY');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -286,7 +286,9 @@ describe('runThink (with stub client)', () => {
|
||||
// used to be stamped NO_ANTHROPIC_API_KEY, sending operators to debug
|
||||
// env/keychain when the fix was the model id. Model validity beats the key
|
||||
// check in probeChatModel, so the honest label holds even keyless.
|
||||
await engine.setConfig('models.think', 'anthropic:claude-bogus-9');
|
||||
// voyage has no chat touchpoint — the surviving unknown_model trigger now
|
||||
// that unlisted ids on chat-capable providers pass through to the provider.
|
||||
await engine.setConfig('models.think', 'voyage:voyage-3');
|
||||
try {
|
||||
const result = await withoutAnthropicKey(() => runThink(engine, { question: 'bad model test' }));
|
||||
expect(result.warnings).toContain('MODEL_NOT_USABLE:unknown_model');
|
||||
@@ -358,9 +360,9 @@ describe('runThink — #1698 explicit-model hard error', () => {
|
||||
).rejects.toThrow(/not usable.*unknown_provider/);
|
||||
});
|
||||
|
||||
test('explicit typo native --model THROWS (unknown_model)', async () => {
|
||||
test('explicit --model on a chat-less provider THROWS (unknown_model)', async () => {
|
||||
await expect(
|
||||
runThink(engine, { question: 'x', model: 'anthropic:claude-bogus-9', modelExplicit: true }),
|
||||
runThink(engine, { question: 'x', model: 'voyage:voyage-3', modelExplicit: true }),
|
||||
).rejects.toThrow(/not usable.*unknown_model/);
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user