diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a8784a43..1c7419b33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,53 @@ All notable changes to GBrain will be documented in this file. +## [0.42.68.0] - 2026-07-28 + +**The default embedding model is now `openai:text-embedding-3-small` at 1280 dimensions. Existing brains keep their column and index — only the vectors are rebuilt.** + +ZeroEntropy's hosted API shuts down 2026-09-04. It has been GBrain's default embedder since v0.36.2.0, which means every brain that never picked a model explicitly was going to lose semantic retrieval on that date — not just for new writes, but for existing vectors, because query embedding runs through the same endpoint. + +The new default is OpenAI `text-embedding-3-small` at **1280** dimensions. 1280, not 1536, is the whole point: OpenAI's `text-embedding-3-*` family is Matryoshka and accepts any output width up to the model's native size, so a brain created under the previous 1280-wide default keeps its existing `vector(1280)` column *and* its HNSW index. Migrating rebuilds vectors only — no dimension transition, no `ALTER`, no index rebuild. + +This release also adds a **default-provider policy** to `CLAUDE.md`: a GBrain default embedding or reranking model must be either open-weight, or from the vendor with the longest proven model-lifetime record. Novel providers can ship as opt-in recipes, never as the default. The v0.36 default stranded every default-config brain on about six weeks' notice; the policy exists so that cannot repeat. + +## To take advantage of v0.42.68.0 + +Fresh installs get the new default with no action. **Existing brains on ZeroEntropy must migrate before 2026-09-04** — `gbrain upgrade` prints a one-time banner with the exact command for your brain's width. + +1. **Upgrade and read the banner:** + ```bash + gbrain upgrade + ``` +2. **Migrate off ZeroEntropy** (resumable; preview the cost first). Pass `--dim` at your brain's current width so the existing column and index are reused: + ```bash + gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --dry-run + gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 + ``` + Check your current width with `gbrain doctor` if you are unsure. A killed run resumes from where it stopped — re-run the same command. +3. **If you use the ZeroEntropy reranker**, it sunsets on the same date. Either point the `llama-server-reranker` recipe at the Apache-2.0 `zerank` weights you self-host, pick another reranker, or turn it off: + ```bash + gbrain config set search.reranker.enabled false + ``` +4. **Prefer to stay on zembed-1?** The weights are Apache-2.0. Self-host via llama-server or Ollama and point `embedding_model` at the local endpoint; your existing vectors stay valid and no migration is needed. +5. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`. + +### Itemized changes + +#### Changed + +- `DEFAULT_EMBEDDING_MODEL` is `openai:text-embedding-3-small`; `DEFAULT_EMBEDDING_DIMENSIONS` stays `1280` (`src/core/ai/defaults.ts`). +- The `openai` recipe lists `text-embedding-3-small` first, so a fresh install with only `OPENAI_API_KEY` set resolves the declared default instead of the recipe's largest tier. +- `1280` joins the `openai` recipe's `dims_options`. That list is Tier 1 in dimension validation and wins over the Matryoshka range check, so without it `gbrain init` rejected its own shipped default. +- The ZeroEntropy sunset banner in `gbrain upgrade` now names the concrete migration target and passes `--dim` at the brain's current width, and reads the database config plane as well as the file plane so brains that never wrote `embedding_model` to `~/.gbrain/config.json` are still detected. +- `gbrain advisor`'s missing-embedding-key smell checks for an OpenAI key rather than a ZeroEntropy one. +- `gbrain init`'s no-provider hint leads with the current default. +- Default-provider policy added to `CLAUDE.md`; README, `INSTALL_FOR_AGENTS.md`, the provider matrix, both tutorials, and the migration guides updated to current truth. + +#### Fixed + +- `test/e2e/fresh-install-pglite.test.ts` clears every non-OpenAI embedding provider key from the recipe registry instead of a hardcoded pair, so the file no longer passes or fails based on which provider keys happen to be set on the developer's machine. + ## [0.42.66.1] - 2026-07-27 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index c23e645bc..4c4eb061f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -239,6 +239,23 @@ The install picker fires inside `gbrain init` AFTER `engine.initSchema()` (non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade` in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`. +## Default-provider policy + +**Default-provider policy.** A gbrain DEFAULT embedding or reranking model must +be either open-weight, or from the vendor with the longest proven model-lifetime +record. Novel/startup providers may ship as opt-in recipes, never as the +default. Rationale: the v0.36 zembed-1 default stranded every default-config +brain when ZeroEntropy was acquired and gave ~6 weeks notice. + +Current defaults live in `src/core/ai/defaults.ts`: +`DEFAULT_EMBEDDING_MODEL = 'openai:text-embedding-3-small'`, +`DEFAULT_EMBEDDING_DIMENSIONS = 1280` (v0.42.68.0, #3390). 1280 — not 1536 — so +brains created under the previous ZeroEntropy default keep their existing +`vector(1280)` column AND its HNSW index: OpenAI text-embedding-3-* is +Matryoshka and `isValidOpenAITextEmbedding3Dim` accepts any width ≤ the model's +native size, so `gbrain migrate embeddings --to openai:text-embedding-3-small +--dim 1280` rebuilds vectors only, with no dimension transition. + ## Eval discipline (v0.32.3) Every metric printed by any `gbrain eval *` or `gbrain search stats` command diff --git a/INSTALL_FOR_AGENTS.md b/INSTALL_FOR_AGENTS.md index 421d2ff09..fb504ef79 100644 --- a/INSTALL_FOR_AGENTS.md +++ b/INSTALL_FOR_AGENTS.md @@ -40,16 +40,19 @@ restart the shell or add the PATH export to the shell profile. ## Step 2: API Keys -Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack -(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config -set embedding_model `. +Ask the user for these. gbrain's default embedder is `openai:text-embedding-3-small` +at 1280 dimensions (as of v0.42.68.0); Voyage/Google/local providers are supported via +`gbrain config set embedding_model `. ```bash -export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+) -export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models +export OPENAI_API_KEY=sk-... # default embedding (text-embedding-3-small, 1280d); also chat models export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion ``` +> ZeroEntropy was the default embedder + reranker from v0.36.2.0 through v0.42.67. +> Its hosted API shuts down 2026-09-04. Brains still on it should run +> `gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280`. + Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider, keyword search still works. Without Anthropic, search works but skips query expansion. diff --git a/README.md b/README.md index 2cd26578d..58aa06fe6 100644 --- a/README.md +++ b/README.md @@ -290,8 +290,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h - **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md). - **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md). -- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). -- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). +- **Embedding providers**: 16 recipes covering OpenAI (`text-embedding-3-small` at 1280d is the default since v0.42.68.0), OpenRouter, Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). ZeroEntropy was the default from v0.36.2.0 to v0.42.67; its hosted API sunsets 2026-09-04 — existing brains move forward with `gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280` (same column width, no schema change). Default-provider policy lives in [`CLAUDE.md`](CLAUDE.md). +- **Rerankers**: ZeroEntropy `zerank-2` hosted (still the `tokenmax`-mode default; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the switch instructions, and the `zerank` weights are Apache-2.0 so the local recipe below keeps working) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). - **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md). - **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup. @@ -459,4 +459,4 @@ MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production Origin story: [`docs/ethos/ORIGIN.md`](docs/ethos/ORIGIN.md). -Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage. +Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that shipped as the default from v0.36.2.0 to v0.42.67. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage. diff --git a/VERSION b/VERSION index bdb592ae4..eb4542bbf 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.66.1 +0.42.68.0 \ No newline at end of file diff --git a/docs/ai-providers/zeroentropy.md b/docs/ai-providers/zeroentropy.md index b465c0e3f..3ba8ad31a 100644 --- a/docs/ai-providers/zeroentropy.md +++ b/docs/ai-providers/zeroentropy.md @@ -1,5 +1,22 @@ # ZeroEntropy — zembed-1 + zerank-2 +> **Deprecated as a hosted provider. The ZeroEntropy hosted API shuts down +> 2026-09-04.** `zembed-1` was GBrain's default embedder from v0.36.2.0 +> through v0.42.67; the default is now `openai:text-embedding-3-small` at +> 1280 dimensions (see the Default-provider policy in `CLAUDE.md`). If your +> brain still embeds through ZeroEntropy, migrate before that date: +> +> ```bash +> gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --dry-run +> gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 +> ``` +> +> `--dim` at your brain's current width reuses the existing `vector(N)` +> column and its HNSW index — only the vectors are rebuilt. The `zembed-1` +> and `zerank` weights are Apache-2.0, so self-hosting via llama-server or +> Ollama is the other forward path and preserves your existing vectors +> outright. See [`../guides/embedding-migration.md`](../guides/embedding-migration.md). + [ZeroEntropy](https://zeroentropy.dev) ships two specialized small models for retrieval pipelines: diff --git a/docs/embedding-migrations.md b/docs/embedding-migrations.md index 49e08765b..542fc501a 100644 --- a/docs/embedding-migrations.md +++ b/docs/embedding-migrations.md @@ -2,9 +2,10 @@ GBrain stores embeddings in a fixed-dimension `vector(N)` column on `content_chunks`. If you switch to a model with a different dimension -(e.g. `openai:text-embedding-3-large` 1536 → `zeroentropyai:zembed-1` -1280, or `voyage:voyage-4-large` 2048), the on-disk column type doesn't -change automatically. +(e.g. `openai:text-embedding-3-large` 1536 → `voyage:voyage-4-large` +2048), the on-disk column type doesn't change automatically. Staying at +the SAME width — the v0.42.68.0 ZeroEntropy→OpenAI default swap keeps +1280 precisely so it can — needs no column change at all. `gbrain init`, `gbrain doctor`, and `gbrain embed --stale` all detect this mismatch and refuse to silently proceed. This doc is the recipe @@ -63,7 +64,7 @@ single-command wrapper: ```bash gbrain reinit-pglite \ - --embedding-model zeroentropyai:zembed-1 \ + --embedding-model openai:text-embedding-3-small \ --embedding-dimensions 1280 ``` @@ -84,7 +85,7 @@ mv ~/.gbrain/brain.pglite ~/.gbrain/brain.pglite.bak # every other field in ~/.gbrain/config.json (chat model, # expansion model, API keys). gbrain init --pglite \ - --embedding-model zeroentropyai:zembed-1 \ + --embedding-model openai:text-embedding-3-small \ --embedding-dimensions 1280 # 3. Re-import your brain repo. `gbrain sync` reads the brain repo diff --git a/docs/guides/embedding-migration.md b/docs/guides/embedding-migration.md index ac748609e..710eccb65 100644 --- a/docs/guides/embedding-migration.md +++ b/docs/guides/embedding-migration.md @@ -3,9 +3,22 @@ `gbrain migrate embeddings` re-embeds an entire brain onto a different embedding provider/model, safely and resumably. It is the forward path off a sunsetting provider (for example ZeroEntropy's hosted API, which shuts down -2026-09-04 and is the shipped default for brains that never picked a model) — -but it is provider-agnostic: any configured `provider:model` works as a -target. +2026-09-04 and was the shipped default from v0.36.2.0 through v0.42.67) — but +it is provider-agnostic: any configured `provider:model` works as a target. + +**Coming off the ZeroEntropy default?** Run: + +```bash +gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --dry-run +gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 +``` + +`openai:text-embedding-3-small` @ 1280 is the v0.42.68.0 default. `--dim 1280` +matters: OpenAI text-embedding-3-* is Matryoshka and accepts any width up to +its native size, so migrating at the width you already have reuses the existing +`vector(1280)` column and its HNSW index — only the vectors are rebuilt. Omit +`--dim` and the target resolves to the recipe's 1536, forcing a needless +dimension transition (schema change + index rebuild). Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the README reference). diff --git a/docs/integrations/embedding-providers.md b/docs/integrations/embedding-providers.md index ffd8114bf..e70ec7d56 100644 --- a/docs/integrations/embedding-providers.md +++ b/docs/integrations/embedding-providers.md @@ -15,7 +15,7 @@ gbrain init --pglite --model voyage # use a non-default provider ## Init resolves your provider from env keys -As of v0.37, `gbrain init --pglite` auto-detects which provider to use from your env vars. With `OPENAI_API_KEY` set, you get OpenAI. With `ZEROENTROPY_API_KEY` set, you get ZeroEntropy. If multiple provider keys are set, init fires an interactive picker. If no provider keys are set in a non-TTY context (CI, Docker build), init exits 1 with a paste-ready setup hint. Explicit flags (`--embedding-model`, `--no-embedding`) always win over env detection. +As of v0.37, `gbrain init --pglite` auto-detects which provider to use from your env vars. With `OPENAI_API_KEY` set, you get the default `openai:text-embedding-3-small` at 1280 dimensions. With `ZEROENTROPY_API_KEY` set, you get ZeroEntropy (deprecated — hosted API sunsets 2026-09-04). If multiple provider keys are set, init fires an interactive picker. If no provider keys are set in a non-TTY context (CI, Docker build), init exits 1 with a paste-ready setup hint. Explicit flags (`--embedding-model`, `--no-embedding`) always win over env detection. The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atomically, so subsequent runs are deterministic across releases. @@ -23,8 +23,8 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom | Provider | env vars | default dims | cost ($/1M tokens) | local? | multimodal? | |---|---|---|---|---|---| +| `openai` (**default**) | `OPENAI_API_KEY` | `text-embedding-3-small` @ 1280 (Matryoshka, any width ≤1536); `text-embedding-3-large` @ 1536 | 0.02 (-small) / 0.13 (-large) | no | no | | `zeroentropyai` | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no | -| `openai` | `OPENAI_API_KEY` | 1536 | 0.13 | no | no | | `openrouter` | `OPENROUTER_API_KEY` | 1536 | 0.02 | no | model-dependent | | `voyage` | `VOYAGE_API_KEY` | 1024 | 0.18 | no | yes (`voyage-multimodal-3`) | | `google` | `GOOGLE_GENERATIVE_AI_API_KEY` | 768 | 0.025 | no | no | @@ -75,7 +75,9 @@ The doctor distinguishes two repair paths: ### OpenAI -Default. Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 default), `text-embedding-3-small` (1536). Matryoshka via the `dimensions` field — gbrain pins it from `embedding_dimensions` config so existing 1536-dim brains stay aligned across SDK upgrades. +**Default since v0.42.68.0: `openai:text-embedding-3-small` at 1280 dimensions.** Set `OPENAI_API_KEY`. Models: `text-embedding-3-small` (1536 native, $0.02/Mtok), `text-embedding-3-large` (3072 native, 1536 recipe default, $0.13/Mtok). Both are Matryoshka: `isValidOpenAITextEmbedding3Dim` accepts **any integer width up to the model's native size**, so 1280 is a first-class width — that is exactly why the v0.42.68.0 default swap needs no schema change on brains created under the previous 1280-wide ZeroEntropy default. gbrain pins `dimensions` from `embedding_dimensions` config so existing brains stay aligned across SDK upgrades. + +Note the split: `DEFAULT_EMBEDDING_DIMENSIONS` (1280) is the zero-config brain width; the openai recipe's `default_dims` (1536) is what `gbrain migrate embeddings --to openai:text-embedding-3-*` resolves when you pass no `--dim`. Pass `--dim 1280` when you want to keep an existing 1280-wide column and its HNSW index in place. Optional `OPENAI_BASE_URL` — point the native OpenAI provider at an OpenAI-compatible gateway. A bare host is normalized to carry the `/v1` suffix automatically (so `https://gw.example.com` and `https://gw.example.com/v1` both work); when unset, the SDK's default endpoint is untouched. `ANTHROPIC_BASE_URL` gets the same normalization for Anthropic chat/expansion calls. diff --git a/docs/tutorials/company-brain.md b/docs/tutorials/company-brain.md index 6ebe643e1..fd6f826c5 100644 --- a/docs/tutorials/company-brain.md +++ b/docs/tutorials/company-brain.md @@ -492,9 +492,12 @@ OAuth source scoping only guards the HTTP MCP path. If the brain's Postgres and ## Part 13: Cost and speed expectations -Real numbers from the published benchmark, running the default stack (GBrain with ZeroEntropy for embedding + reranker): +Real numbers from the published benchmark. The benchmark run used ZeroEntropy for +embedding + reranker, which was the default through v0.42.67; the default embedder is +now `openai:text-embedding-3-small` at 1280 dimensions (see the Default-provider policy +in `CLAUDE.md`). -- **Embedding cost:** $0.05 per million tokens. For comparison, GBrain configured with OpenAI is $0.13 (2.6× more expensive), Voyage is $0.18 (3.6× more). +- **Embedding cost:** $0.02 per million tokens on the current default (`text-embedding-3-small`). For comparison, `text-embedding-3-large` is $0.13 (6.5× more), Voyage is $0.18 (9× more). - **Ingest speed:** about 22 seconds for a small test corpus of 164 pages on the host machine. For a 10K-page corpus, expect about 20 minutes the first time, then most syncs are incremental and finish in seconds. - **Query latency:** about 122 ms median for a `gbrain search`. For comparison, the same query through GBrain with OpenAI takes about 282 ms. - **Synthesized-answer latency:** a few seconds, dominated by the Anthropic API. @@ -502,7 +505,7 @@ Real numbers from the published benchmark, running the default stack (GBrain wit Full methodology and per-run receipt JSONs live in [the gbrain-evals repo](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-05-23-v0.40.6.0-snapshot.md). -For a 25-person company at sustained use, expect about $35 a month in embeddings (ZeroEntropy at $0.05/million tokens), $50 a month in Anthropic calls for the synthesized-answer queries, plus your hosting bill. Under $100 a month for the AI side at most companies your size. +For a 25-person company at sustained use, expect about $15 a month in embeddings (the default `text-embedding-3-small` at $0.02/million tokens), $50 a month in Anthropic calls for the synthesized-answer queries, plus your hosting bill. Under $100 a month for the AI side at most companies your size. --- @@ -514,7 +517,7 @@ Check `gbrain auth list` on the host and confirm their client has `--source` set ### "Sync is slow and feels stuck" -The first sync embeds every page, which takes time. Check `gbrain sources status` for the live page count. If it's climbing you're not stuck, you're just embedding. If you've got a 10K-page corpus and ZeroEntropy is being throttled, the per-source parallel sync looks like progress on three sources at once rather than one source moving fast. +The first sync embeds every page, which takes time. Check `gbrain sources status` for the live page count. If it's climbing you're not stuck, you're just embedding. If you've got a 10K-page corpus and your embedding provider is being throttled, the per-source parallel sync looks like progress on three sources at once rather than one source moving fast. ### "I see a page I shouldn't see" diff --git a/docs/tutorials/personal-brain.md b/docs/tutorials/personal-brain.md index 8aaaca957..de6fe2ed9 100644 --- a/docs/tutorials/personal-brain.md +++ b/docs/tutorials/personal-brain.md @@ -103,11 +103,10 @@ Render will build a Docker container with the harness. First deploy takes about In the AlphaClaw UI (Providers tab): -- **OpenAI API Key.** Required for embeddings if you use the OpenAI provider. +- **OpenAI API Key.** Required. GBrain's default embedder is `openai:text-embedding-3-small` at 1280 dimensions ($0.02 per million tokens). - **Anthropic API Key.** Required for Claude (the main model the agent talks through). - **Perplexity API Key.** Optional, for web search. - **Voyage API Key.** Optional, alternative to OpenAI for embeddings. -- **ZeroEntropy API Key.** Recommended. GBrain ships with ZeroEntropy as the default embedder + reranker because it's about 2× faster than OpenAI and about 2.6× cheaper. You can use the same keys across multiple agents. @@ -236,7 +235,7 @@ Brains share through git. My main agent can populate another agent's brain by pu |-----------|-------------| | Render Pro (minimum viable) | about $85 | | Supabase (small) | free to $25 | -| OpenAI API (embeddings) | $5 to $20 (much less if you use ZeroEntropy as the default) | +| OpenAI API (embeddings) | $5 to $20 (the default `text-embedding-3-small` is the cheap end of that range) | | Anthropic API (Claude) | $50 to $500 (usage dependent) | | **Total minimum** | **about $100 to $150 a month** | diff --git a/llms-full.txt b/llms-full.txt index 0184bd797..a68497fcc 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -388,6 +388,23 @@ The install picker fires inside `gbrain init` AFTER `engine.initSchema()` (non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade` in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`. +## Default-provider policy + +**Default-provider policy.** A gbrain DEFAULT embedding or reranking model must +be either open-weight, or from the vendor with the longest proven model-lifetime +record. Novel/startup providers may ship as opt-in recipes, never as the +default. Rationale: the v0.36 zembed-1 default stranded every default-config +brain when ZeroEntropy was acquired and gave ~6 weeks notice. + +Current defaults live in `src/core/ai/defaults.ts`: +`DEFAULT_EMBEDDING_MODEL = 'openai:text-embedding-3-small'`, +`DEFAULT_EMBEDDING_DIMENSIONS = 1280` (v0.42.68.0, #3390). 1280 — not 1536 — so +brains created under the previous ZeroEntropy default keep their existing +`vector(1280)` column AND its HNSW index: OpenAI text-embedding-3-* is +Matryoshka and `isValidOpenAITextEmbedding3Dim` accepts any width ≤ the model's +native size, so `gbrain migrate embeddings --to openai:text-embedding-3-small +--dim 1280` rebuilds vectors only, with no dimension transition. + ## Eval discipline (v0.32.3) Every metric printed by any `gbrain eval *` or `gbrain search stats` command @@ -1030,16 +1047,19 @@ restart the shell or add the PATH export to the shell profile. ## Step 2: API Keys -Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack -(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config -set embedding_model `. +Ask the user for these. gbrain's default embedder is `openai:text-embedding-3-small` +at 1280 dimensions (as of v0.42.68.0); Voyage/Google/local providers are supported via +`gbrain config set embedding_model `. ```bash -export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+) -export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models +export OPENAI_API_KEY=sk-... # default embedding (text-embedding-3-small, 1280d); also chat models export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion ``` +> ZeroEntropy was the default embedder + reranker from v0.36.2.0 through v0.42.67. +> Its hosted API shuts down 2026-09-04. Brains still on it should run +> `gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280`. + Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider, keyword search still works. Without Anthropic, search works but skips query expansion. @@ -1784,8 +1804,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h - **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md). - **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md). -- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). -- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). +- **Embedding providers**: 16 recipes covering OpenAI (`text-embedding-3-small` at 1280d is the default since v0.42.68.0), OpenRouter, Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). ZeroEntropy was the default from v0.36.2.0 to v0.42.67; its hosted API sunsets 2026-09-04 — existing brains move forward with `gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280` (same column width, no schema change). Default-provider policy lives in [`CLAUDE.md`](CLAUDE.md). +- **Rerankers**: ZeroEntropy `zerank-2` hosted (still the `tokenmax`-mode default; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the switch instructions, and the `zerank` weights are Apache-2.0 so the local recipe below keeps working) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md). - **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md). - **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup. @@ -1953,7 +1973,7 @@ MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production Origin story: [`docs/ethos/ORIGIN.md`](docs/ethos/ORIGIN.md). -Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage. +Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that shipped as the default from v0.36.2.0 to v0.42.67. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage. --- diff --git a/package.json b/package.json index 13979ea4f..6bc05d71f 100644 --- a/package.json +++ b/package.json @@ -146,7 +146,7 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.66.1", + "version": "0.42.68.0", "overrides": { "@hono/node-server": "^2.0.5", "fast-uri": "^3.1.4", diff --git a/src/commands/init.ts b/src/commands/init.ts index ca01c611c..33143ad9b 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -494,9 +494,9 @@ export async function findEnvKeyTypos( /** Emit the fail-loud "no embedding provider" message + paste-ready setup. */ function printNoEmbeddingProviderHint(typos: Array<{ userSet: string; suggested: string }>): void { console.error('\nNo embedding provider configured. Set one of:'); - console.error(' export OPENAI_API_KEY=sk-… # openai:text-embedding-3-large (1536d)'); - console.error(' export ZEROENTROPY_API_KEY=ze-… # zeroentropyai:zembed-1 (2560d, Matryoshka)'); + console.error(' export OPENAI_API_KEY=sk-… # default: openai:text-embedding-3-small (1280d)'); console.error(' export VOYAGE_API_KEY=pa-… # voyage:voyage-3-large (1024d)'); + console.error(' export GOOGLE_GENERATIVE_AI_API_KEY=… # google:gemini-embedding-001 (768d)'); console.error('Then re-run: gbrain init --pglite'); console.error(''); console.error('Or pick explicitly:'); @@ -527,9 +527,9 @@ async function resolveEmbeddingByEnv(out: ResolvedAIOptions, nonInteractive: boo // (DEFAULT_EMBEDDING_MODEL), use the gateway's // DEFAULT_EMBEDDING_DIMENSIONS instead of the recipe's `default_dims` // (which is the recipe's "largest sensible" tier). This keeps - // fresh-install schema width aligned with the v0.37.11.0 system - // default — for ZE that means 1280 (the Matryoshka step closest to - // legacy OpenAI 1536), not the recipe's 2560. + // fresh-install schema width aligned with the system default — for + // openai:text-embedding-3-small that means 1280 (v0.42.68.0), not + // the recipe's 1536. const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = await import('../core/ai/defaults.ts'); const { embeddingDimsForModel } = await import('../core/ai/model-resolver.ts'); diff --git a/src/commands/upgrade.ts b/src/commands/upgrade.ts index 313ab4120..5619cf8b9 100644 --- a/src/commands/upgrade.ts +++ b/src/commands/upgrade.ts @@ -470,10 +470,30 @@ export async function runPostUpgrade(args: string[] = []): Promise { // embedding uses the same endpoint, so existing vectors become // unqueryable. One-shot per install, gated by // `ze_sunset_notice_shown` (same pattern as the search-mode banner). + // + // v0.42.68.0: DEFAULT_EMBEDDING_MODEL is no longer a ZE model, so the + // file plane alone would stop detecting brains that were created under + // the v0.36–v0.42.67 ZE default and never wrote `embedding_model` to + // ~/.gbrain/config.json. Those brains DO carry the DB-plane row seeded + // by initSchema (pglite-schema.ts `('embedding_model', …)`), so read + // the DB plane as the second source before falling back to the default. try { const shown = await engine.getConfig('ze_sunset_notice_shown'); - const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts'); - const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL; + const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = + await import('../core/ai/defaults.ts'); + const dbModel = await engine.getConfig('embedding_model'); + const effectiveModel = cfgSchema.embedding_model ?? dbModel ?? DEFAULT_EMBEDDING_MODEL; + // Migrating AT THE CURRENT WIDTH is what keeps the existing + // vector(N) column + HNSW index in place (applyEmbeddingMigration + // only runs runSchemaTransition when col.dims !== plan.to_dims). + // Bare `--to openai:text-embedding-3-small` would resolve to the + // recipe's 1536 and force a needless dimension transition, so the + // hint carries `--dim` explicitly. + const dbDims = await engine.getConfig('embedding_dimensions'); + const parsedDbDims = dbDims ? parseInt(dbDims, 10) : NaN; + const currentDims = cfgSchema.embedding_dimensions + ?? (Number.isFinite(parsedDbDims) && parsedDbDims > 0 ? parsedDbDims : undefined) + ?? DEFAULT_EMBEDDING_DIMENSIONS; const rerankerModel = await engine.getConfig('search.reranker.model'); const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:'); const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:'); @@ -492,9 +512,13 @@ export async function runPostUpgrade(args: string[] = []): Promise { } console.log('═══════════════════════════════════════════════════════════════'); console.log(''); - console.log('Migrate before the sunset (resumable; preview cost first):'); - console.log(' gbrain migrate embeddings --to --dry-run'); - console.log(' gbrain migrate embeddings --to '); + console.log('Migrate before 2026-09-04 (resumable; preview cost first):'); + console.log(` gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${currentDims} --dry-run`); + console.log(` gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${currentDims}`); + console.log(''); + console.log(`${DEFAULT_EMBEDDING_MODEL} is the v0.42.68.0 default. At --dim`); + console.log(`${currentDims} it reuses your existing vector(${currentDims}) column and HNSW`); + console.log('index — only the vectors are rebuilt, no schema transition.'); console.log(''); console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /'); console.log('ollama also works and preserves your existing vectors — point'); diff --git a/src/core/advisor/collect-setup-smells.ts b/src/core/advisor/collect-setup-smells.ts index de7884e24..0793008b7 100644 --- a/src/core/advisor/collect-setup-smells.ts +++ b/src/core/advisor/collect-setup-smells.ts @@ -36,14 +36,16 @@ export const collectSetupSmells: AdvisorCollector = { collector: 'setup-smells', ask_user: true, }); - } else if (!cfg.embedding_model && !cfg.zeroentropy_api_key && !process.env.ZEROENTROPY_API_KEY) { + } else if (!cfg.embedding_model && !cfg.openai_api_key && !process.env.OPENAI_API_KEY) { // Default provider needs a key; none present anywhere → embeds will fail. + // v0.42.68.0 (#3390): the default is openai:text-embedding-3-small, so + // the key this checks for is OpenAI's, not ZeroEntropy's. findings.push({ id: 'embedding_key_missing', severity: 'warn', title: 'No embedding provider key is set — embedding will fail at write time.', - detail: 'Set zeroentropy_api_key (or choose another provider via embedding_model).', - fix: { command_argv: ['gbrain', 'config', 'set', 'zeroentropy_api_key', ''] }, + detail: 'Set openai_api_key (or choose another provider via embedding_model).', + fix: { command_argv: ['gbrain', 'config', 'set', 'openai_api_key', ''] }, collector: 'setup-smells', ask_user: true, }); diff --git a/src/core/ai/defaults.ts b/src/core/ai/defaults.ts index b00131a38..11305044c 100644 --- a/src/core/ai/defaults.ts +++ b/src/core/ai/defaults.ts @@ -12,10 +12,20 @@ * install AND every doctor consistency check. */ -// v0.36.0 chose ZeroEntropy as the system default after evals showed -// 11/20 wins vs OpenAI (6) and Voyage (4) on real-corpus benchmarks. -// 1280 is the closest analog to legacy OpenAI 1536d while staying on -// the high-recall section of ZE's Matryoshka curve. Valid ZE Matryoshka -// steps: {2560, 1280, 640, 320, 160, 80, 40} — see ai/dims.ts. -export const DEFAULT_EMBEDDING_MODEL = 'zeroentropyai:zembed-1'; +// v0.42.68.0 (#3390): the default moved OFF ZeroEntropy. ZE's hosted API +// (including /models/embed) shuts down 2026-09-04, which would have taken +// semantic retrieval with it on every default-config brain. See the +// Default-provider policy in CLAUDE.md: a gbrain DEFAULT must be +// open-weight or from the vendor with the longest proven model-lifetime +// record. OpenAI's text-embedding-3-* has been stable since 2024-01. +// +// Why 1280 and NOT 1536 (load-bearing — do not "round up"): +// OpenAI text-embedding-3-* is Matryoshka, and +// `isValidOpenAITextEmbedding3Dim` accepts ANY integer 1..1536 for +// text-embedding-3-small (ai/dims.ts). Keeping 1280 means every brain +// created under the v0.36–v0.42.67 ZE default keeps its existing +// `vector(1280)` column AND its HNSW index — `gbrain migrate embeddings +// --to openai:text-embedding-3-small` rebuilds the VECTORS only, with no +// dimension transition, no ALTER, no index rebuild. +export const DEFAULT_EMBEDDING_MODEL = 'openai:text-embedding-3-small'; export const DEFAULT_EMBEDDING_DIMENSIONS = 1280; diff --git a/src/core/ai/recipes/openai.ts b/src/core/ai/recipes/openai.ts index d453a4db0..dcaf93d0f 100644 --- a/src/core/ai/recipes/openai.ts +++ b/src/core/ai/recipes/openai.ts @@ -12,11 +12,30 @@ export const openai: Recipe = { }, touchpoints: { embedding: { - models: ['text-embedding-3-large', 'text-embedding-3-small'], + // v0.42.68.0 (#3390): -small leads because it IS + // DEFAULT_EMBEDDING_MODEL (ai/defaults.ts). `init`'s env detection + // picks models[0]; when that equals the canonical default it also + // adopts DEFAULT_EMBEDDING_DIMENSIONS (1280) instead of default_dims. + // Reordering keeps "the declared default" and "what a fresh + // OPENAI_API_KEY-only install actually gets" the same thing. + models: ['text-embedding-3-small', 'text-embedding-3-large'], default_dims: 1536, - dims_options: [256, 512, 768, 1024, 1536, 3072], - cost_per_1m_tokens_usd: 0.13, - price_last_verified: '2026-04-20', + // 1280 is here because it IS DEFAULT_EMBEDDING_DIMENSIONS (v0.42.68.0). + // `dims_options` is Tier 1 in isCustomDimValidForProvider — it wins over + // the Tier-2 `isValidOpenAITextEmbedding3Dim` range check, so a width + // missing from this list is rejected before the real Matryoshka rule is + // ever consulted. Without 1280 the shipped default config fails + // `resolveSchemaEmbeddingDim` and `gbrain init` refuses its own default. + // ponytail: curated list, not the true rule (OpenAI accepts ANY integer + // ≤ the model's native size). Delete `dims_options` here and let Tier 2 + // govern if arbitrary widths ever need to work. + dims_options: [256, 512, 768, 1024, 1280, 1536, 3072], + // Tracks models[0] (`text-embedding-3-small`), same convention as the + // openrouter recipe. Display-only, for `gbrain providers list/explain`; + // all actual cost math routes through the per-model table in + // src/core/embedding-pricing.ts (-small $0.02 / -large $0.13). + cost_per_1m_tokens_usd: 0.02, + price_last_verified: '2026-07-28', // OpenAI per-request hard cap is 300K tokens. Free/Tier-1 TPM is 1M. // Cap batches conservatively at 100K to handle token-dense content // (Discord/Slack markdown+JSON tokenizes at ~chars/2.7, not the chars/4 diff --git a/test/ai/dims-openai.test.ts b/test/ai/dims-openai.test.ts index 2d05a4dcb..4eaa5eccf 100644 --- a/test/ai/dims-openai.test.ts +++ b/test/ai/dims-openai.test.ts @@ -161,3 +161,65 @@ describe('dimsProviderOptions — prefixed model IDs (OpenRouter / proxy provide } }); }); + +// v0.42.68.0 (#3390) — the default embedding model moved off ZeroEntropy +// (hosted API sunsets 2026-09-04) to openai:text-embedding-3-small, and +// KEPT 1280 dimensions on purpose. These tests pin the two properties the +// swap rests on, so a future "tidy up to 1536" can't land silently. +describe('default embedding config (v0.42.68.0 #3390)', () => { + test('DEFAULT_EMBEDDING_MODEL / DIMENSIONS are openai:text-embedding-3-small @ 1280', async () => { + const defaults = await import('../../src/core/ai/defaults.ts'); + expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('openai:text-embedding-3-small'); + expect(defaults.DEFAULT_EMBEDDING_DIMENSIONS).toBe(1280); + }); + + test('the default width is a valid Matryoshka width for the default model', async () => { + // The no-schema-change property: brains created under the previous + // 1280-wide ZeroEntropy default keep their vector(1280) column and its + // HNSW index because OpenAI text-embedding-3-* accepts any width up to + // its native size. Derived from the constants, never hardcoded. + const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = + await import('../../src/core/ai/defaults.ts'); + const bareModel = DEFAULT_EMBEDDING_MODEL.split(':')[1]; + expect(isOpenAITextEmbedding3Model(bareModel)).toBe(true); + expect(isValidOpenAITextEmbedding3Dim(bareModel, DEFAULT_EMBEDDING_DIMENSIONS)).toBe(true); + expect(DEFAULT_EMBEDDING_DIMENSIONS) + .toBeLessThanOrEqual(maxOpenAITextEmbedding3Dim(bareModel)!); + }); + + test('dimsProviderOptions passes the default width through to the wire', async () => { + const { DEFAULT_EMBEDDING_DIMENSIONS } = await import('../../src/core/ai/defaults.ts'); + expect(dimsProviderOptions('native-openai', 'text-embedding-3-small', DEFAULT_EMBEDDING_DIMENSIONS)) + .toEqual({ openai: { dimensions: 1280 } }); + }); + + test('resolveSchemaEmbeddingDim ACCEPTS the shipped default config', async () => { + // Regression guard: `dims_options` on the openai recipe is Tier 1 in + // isCustomDimValidForProvider and wins over the Matryoshka range check. + // It omitted 1280 until #3390, which made `gbrain init` reject its own + // default. Drop 1280 from the recipe and this test fails. + const { resolveSchemaEmbeddingDim } = await import('../../src/core/embedding-dim-check.ts'); + const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = + await import('../../src/core/ai/defaults.ts'); + const got = resolveSchemaEmbeddingDim({ + embedding_model: DEFAULT_EMBEDDING_MODEL, + embedding_dimensions: DEFAULT_EMBEDDING_DIMENSIONS, + }); + expect(got.ok).toBe(true); + if (got.ok) { + expect(got.dim).toBe(1280); + expect(got.model).toBe(DEFAULT_EMBEDDING_MODEL); + expect(got.provider).toBe('openai'); + } + }); + + test('init env-detection lands on the default model when OPENAI_API_KEY is the only key', async () => { + // `resolveEmbeddingByEnv` picks touchpoints.embedding.models[0] and only + // adopts DEFAULT_EMBEDDING_DIMENSIONS when that equals the canonical + // default. If the recipe's model order regresses, a fresh install silently + // gets text-embedding-3-large @ 1536 instead of the declared default. + const { openai } = await import('../../src/core/ai/recipes/openai.ts'); + const { DEFAULT_EMBEDDING_MODEL } = await import('../../src/core/ai/defaults.ts'); + expect(`openai:${openai.touchpoints.embedding!.models![0]}`).toBe(DEFAULT_EMBEDDING_MODEL); + }); +}); diff --git a/test/ai/gateway.test.ts b/test/ai/gateway.test.ts index 10aaceeea..faaac6056 100644 --- a/test/ai/gateway.test.ts +++ b/test/ai/gateway.test.ts @@ -40,12 +40,14 @@ describe('gateway configuration', () => { expect(getExpansionModel()).toBe('anthropic:claude-haiku-4-5-20251001'); }); - test('defaults are ZE 1280d as of v0.36.0.0 (D3)', () => { - // The default flipped from openai:text-embedding-3-large 1536d to - // zeroentropyai:zembed-1 1280d in v0.36.0.0. The cost story is in - // CHANGELOG.md; the rationale lives in src/core/ai/gateway.ts:45-54. + test('defaults are openai:text-embedding-3-small 1280d as of v0.42.68.0 (#3390)', () => { + // v0.36.0.0 flipped the default to zeroentropyai:zembed-1 @ 1280d. + // v0.42.68.0 flipped the MODEL off ZeroEntropy (hosted API sunsets + // 2026-09-04) but deliberately KEPT 1280 so existing ZE-default brains + // reuse their vector(1280) column + HNSW index. Rationale + policy in + // src/core/ai/defaults.ts and CLAUDE.md's Default-provider policy. configureGateway({ env: {} }); - expect(getEmbeddingModel()).toBe('zeroentropyai:zembed-1'); + expect(getEmbeddingModel()).toBe('openai:text-embedding-3-small'); expect(getEmbeddingDimensions()).toBe(1280); expect(getExpansionModel()).toBe('anthropic:claude-haiku-4-5-20251001'); }); diff --git a/test/ai/schema-templating.test.ts b/test/ai/schema-templating.test.ts index 5ab5e2d47..6dbf59d4b 100644 --- a/test/ai/schema-templating.test.ts +++ b/test/ai/schema-templating.test.ts @@ -3,15 +3,15 @@ import { getPGLiteSchema, PGLITE_SCHEMA_SQL } from '../../src/core/pglite-schema import { getPostgresSchema } from '../../src/core/postgres-engine.ts'; describe('getPGLiteSchema', () => { - test('default produces gateway-default schema (v0.37+: 1280d + zeroentropyai:zembed-1)', () => { + test('default produces gateway-default schema (v0.42.68.0+: 1280d + openai:text-embedding-3-small)', () => { // v0.37 fix wave Lane A.1 + CDX2-1: defaults now track the canonical // gateway constants in `ai/defaults.ts` instead of the stale v0.13 // OpenAI literals (1536 / text-embedding-3-large). Fixes the // headline bug where bare `gbrain init --pglite` produced a 1536 - // schema while the ZE default model emitted 1280-dim vectors. + // schema while the default model emitted 1280-dim vectors. const sql = getPGLiteSchema(); expect(sql).toMatch(/vector\(1280\)/); - expect(sql).toMatch(/'zeroentropyai:zembed-1'/); + expect(sql).toMatch(/'openai:text-embedding-3-small'/); expect(sql).not.toMatch(/__EMBEDDING_DIMS__/); expect(sql).not.toMatch(/__EMBEDDING_MODEL__/); }); diff --git a/test/e2e/embedding-column-pglite.test.ts b/test/e2e/embedding-column-pglite.test.ts index 7254806cf..a4ab9609b 100644 --- a/test/e2e/embedding-column-pglite.test.ts +++ b/test/e2e/embedding-column-pglite.test.ts @@ -219,12 +219,13 @@ describe('hybridSearch + resolver — unknown column at entry (D11)', () => { describe('upsertChunks — model provenance uses gateway-resolved model, not compiled default', () => { // Regression (zbrain-rfi): when a caller builds ChunkInputs without an // explicit `model` (as src/commands/embed.ts does), the engine used to - // stamp the compile-time DEFAULT_EMBEDDING_MODEL ('zeroentropyai:zembed-1') - // onto content_chunks.model — even though the vector was produced by the + // stamp the compile-time DEFAULT_EMBEDDING_MODEL onto + // content_chunks.model — even though the vector was produced by the // config-resolved model. That corrupted provenance the signature-drift + // dim-migration logic trusts. The engine must fall back to the model the - // gateway ACTUALLY resolves at write time. - test('unspecified chunk.model records the resolved model, not zeroentropyai:zembed-1', async () => { + // gateway ACTUALLY resolves at write time. Asserts against the LIVE + // constant so the guard survives a default swap (v0.42.68.0 #3390). + test('unspecified chunk.model records the resolved model, not the compiled default', async () => { configureGateway({ embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: 1536, @@ -248,7 +249,8 @@ describe('upsertChunks — model provenance uses gateway-resolved model, not com ); expect(rows.length).toBe(1); expect(rows[0].model).toBe('openai:text-embedding-3-large'); - expect(rows[0].model).not.toBe('zeroentropyai:zembed-1'); + const { DEFAULT_EMBEDDING_MODEL } = await import('../../src/core/ai/defaults.ts'); + expect(rows[0].model).not.toBe(DEFAULT_EMBEDDING_MODEL); resetGateway(); }); diff --git a/test/e2e/fresh-install-pglite.test.ts b/test/e2e/fresh-install-pglite.test.ts index 1b3f3f3bc..3ffdfbfe3 100644 --- a/test/e2e/fresh-install-pglite.test.ts +++ b/test/e2e/fresh-install-pglite.test.ts @@ -3,7 +3,8 @@ * * The headline behavior the v0.37 fix wave exists to fix. Pre-fix, this * exact path broke: schema sized to 1536 (stale default), embed pipeline - * used ZE/1280, first chunk insert failed with vector dim mismatch. + * used the 1280-wide default, first chunk insert failed with vector dim + * mismatch. * * Hermetic: in-process (NOT a CLI subprocess), GBRAIN_HOME pinned to a * tmpdir, embed transport stubbed via `__setEmbedTransportForTests` so we @@ -26,36 +27,49 @@ import { describe('E2E: fresh gbrain init --pglite → import → embed works end-to-end', () => { let tmpHome: string; let origHome: string | undefined; - let origZeKey: string | undefined; - let origOpenaiKey: string | undefined; - let origVoyageKey: string | undefined; + // Every embedding-provider auth env var we cleared, so afterEach restores + // exactly what was there. Enumerated from the recipe registry rather than + // hardcoded: a dev machine with MINIMAX_API_KEY (or any of the other 14 + // providers) set used to trip init's disambiguation gate + // ("Multiple embedding providers env-ready: openai, minimax") before the + // test body ran, which made this file pass or fail on ambient environment. + let clearedProviderKeys: Record = {}; - beforeEach(() => { + async function clearNonOpenAIEmbeddingKeys(): Promise { + const { RECIPES } = await import('../../src/core/ai/recipes/index.ts'); + clearedProviderKeys = {}; + for (const recipe of RECIPES.values()) { + if (recipe.id === 'openai') continue; + if (!recipe.touchpoints.embedding) continue; + for (const key of recipe.auth_env?.required ?? []) { + clearedProviderKeys[key] = process.env[key]; + delete process.env[key]; + } + } + } + + beforeEach(async () => { tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-e2e-fresh-')); origHome = process.env.GBRAIN_HOME; - origZeKey = process.env.ZEROENTROPY_API_KEY; - // Save + clear OPENAI_API_KEY + VOYAGE_API_KEY so init only sees - // one provider as env-ready (ZE). Without this, dev machines with - // multi-provider env (Garry's setup) fail init's disambiguation gate - // ("Multiple embedding providers env-ready: openai, voyage, - // zeroentropyai") before the test body runs. - origOpenaiKey = process.env.OPENAI_API_KEY; - origVoyageKey = process.env.VOYAGE_API_KEY; - delete process.env.OPENAI_API_KEY; - delete process.env.VOYAGE_API_KEY; + // Leave openai as the ONLY env-ready embedding provider, so bare + // `init --pglite` resolves the v0.42.68.0 default + // (openai:text-embedding-3-small @ DEFAULT_EMBEDDING_DIMENSIONS). + await clearNonOpenAIEmbeddingKeys(); + clearedProviderKeys.OPENAI_API_KEY = process.env.OPENAI_API_KEY; process.env.GBRAIN_HOME = tmpHome; // Stub key so init's setup-hint check passes. - process.env.ZEROENTROPY_API_KEY = 'sk-test-ze'; + process.env.OPENAI_API_KEY = 'sk-test-openai'; }); afterEach(() => { rmSync(tmpHome, { recursive: true, force: true }); if (origHome === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = origHome; - if (origZeKey === undefined) delete process.env.ZEROENTROPY_API_KEY; - else process.env.ZEROENTROPY_API_KEY = origZeKey; - if (origOpenaiKey !== undefined) process.env.OPENAI_API_KEY = origOpenaiKey; - if (origVoyageKey !== undefined) process.env.VOYAGE_API_KEY = origVoyageKey; + for (const [key, value] of Object.entries(clearedProviderKeys)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + clearedProviderKeys = {}; __setEmbedTransportForTests(null); // Restore legacy-preload gateway state. configureGateway({ @@ -65,7 +79,7 @@ describe('E2E: fresh gbrain init --pglite → import → embed works end-to-end' }); }); - test('bare `init --pglite`: schema sized to gateway defaults (ZE/1280)', async () => { + test('bare `init --pglite`: schema sized to gateway defaults (openai text-embedding-3-small/1280)', async () => { // Reset gateway so init.ts has to resolve defaults from // ai/defaults.ts. This is the actual production code path for a // fresh install: bare `gbrain init --pglite` with no env or file diff --git a/test/e2e/init-fresh-pglite.test.ts b/test/e2e/init-fresh-pglite.test.ts index 3cc1b74f2..556e4d65f 100644 --- a/test/e2e/init-fresh-pglite.test.ts +++ b/test/e2e/init-fresh-pglite.test.ts @@ -74,15 +74,21 @@ describe('v0.37 T12 — fresh init env-detection (D1, D2, D3) + persistence (D5) // Init may or may not succeed (depends on whether OpenAI key is real for // any side effect — but init.ts has no live embed call, just config // writes + schema). Assert the auto-pick stderr notice fired. - expect(r.stderr).toMatch(/Detected OPENAI_API_KEY|Using openai:text-embedding-3-large/); + expect(r.stderr).toMatch(/Detected OPENAI_API_KEY|Using openai:text-embedding-3-small/); expect(r.exitCode).toBe(0); - // Config persisted with the right embedding fields. + // Config persisted with the right embedding fields. v0.42.68.0 (#3390): + // env-detected OpenAI now lands on the canonical default + // (openai:text-embedding-3-small @ DEFAULT_EMBEDDING_DIMENSIONS = 1280), + // not the recipe's largest tier. Asserted against the live constants so + // this test tracks a future default swap instead of pinning stale literals. + const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = + await import('../../src/core/ai/defaults.ts'); const cfgPath = join(tmpHome, '.gbrain', 'config.json'); expect(existsSync(cfgPath)).toBe(true); const cfg = JSON.parse(readFileSync(cfgPath, 'utf-8')); - expect(cfg.embedding_model).toBe('openai:text-embedding-3-large'); - expect(cfg.embedding_dimensions).toBe(1536); + expect(cfg.embedding_model).toBe(DEFAULT_EMBEDDING_MODEL); + expect(cfg.embedding_dimensions).toBe(DEFAULT_EMBEDDING_DIMENSIONS); expect(cfg.engine).toBe('pglite'); }, 240000); }); @@ -104,8 +110,8 @@ describe('v0.37 T12 — D3 non-TTY no-key fail-loud', () => { // Fail-loud message includes the canonical env var list. expect(r.stderr).toContain('No embedding provider configured'); expect(r.stderr).toContain('OPENAI_API_KEY'); - expect(r.stderr).toContain('ZEROENTROPY_API_KEY'); expect(r.stderr).toContain('VOYAGE_API_KEY'); + expect(r.stderr).toContain('GOOGLE_GENERATIVE_AI_API_KEY'); // Suggests --no-embedding alternative. expect(r.stderr).toContain('--no-embedding'); }, 60000); diff --git a/test/helpers/legacy-embedding-preload.ts b/test/helpers/legacy-embedding-preload.ts index 257c21d35..bcb2973de 100644 --- a/test/helpers/legacy-embedding-preload.ts +++ b/test/helpers/legacy-embedding-preload.ts @@ -3,9 +3,10 @@ * so tests written before v0.37 (with hardcoded `new Float32Array(1536)` * fixtures) keep working without per-file edits. * - * v0.37 fix wave changed the canonical gateway defaults to - * `zeroentropyai:zembed-1` / 1280-d (matching the system default chosen - * in v0.36.0). Tests that don't explicitly configure the gateway + * v0.37 fix wave changed the canonical gateway defaults to a 1280-d model + * (zeroentropyai:zembed-1 then; openai:text-embedding-3-small as of + * v0.42.68.0 — the width is what matters here). Tests that don't + * explicitly configure the gateway * previously got 1536-d schemas via the stale `getPGLiteSchema()` * default; v0.37 fixed that so the schema tracks the gateway default * (1280 out of the box). Tests with 1536-d fixtures need the schema to @@ -14,7 +15,7 @@ * Imported by `bunfig.toml` via `preload = ["./test/helpers/legacy-embedding-preload.ts"]`. * * Tests that need a different embedding shape (the new v0.37 tests, - * future ZE-1280 tests, or specific-provider tests) should call + * future 1280-d tests, or specific-provider tests) should call * `configureGateway()` explicitly in their own beforeAll, which * overwrites this preload. */ diff --git a/test/v0_37_fix_wave.serial.test.ts b/test/v0_37_fix_wave.serial.test.ts index 500309570..eda25de4a 100644 --- a/test/v0_37_fix_wave.serial.test.ts +++ b/test/v0_37_fix_wave.serial.test.ts @@ -19,13 +19,13 @@ describe('v0.37 Lane A — defaults sweep', () => { // CDX2-1: these were file-private const; Lane A consumers (schema // helpers, registry) need them exported. Importing here is the test. const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = await import('../src/core/ai/gateway.ts'); - expect(DEFAULT_EMBEDDING_MODEL).toBe('zeroentropyai:zembed-1'); + expect(DEFAULT_EMBEDDING_MODEL).toBe('openai:text-embedding-3-small'); expect(DEFAULT_EMBEDDING_DIMENSIONS).toBe(1280); }); test('A.0: ai/defaults.ts is the canonical source (leaf module, no SDK pulls)', async () => { const defaults = await import('../src/core/ai/defaults.ts'); - expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('zeroentropyai:zembed-1'); + expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('openai:text-embedding-3-small'); expect(defaults.DEFAULT_EMBEDDING_DIMENSIONS).toBe(1280); }); @@ -52,17 +52,18 @@ describe('v0.37 Lane A — defaults sweep', () => { expect(sql).toContain('voyage:voyage-4-large'); }); - test('A.5: embedding-column registry builtin defaults to ZE/1280 on empty config + gateway', async () => { + test('A.5: embedding-column registry builtin defaults to openai/1280 on empty config + gateway', async () => { // The registry's resolution chain is cfg > gateway > DEFAULT. With // no cfg AND no gateway, it should fall through to the canonical - // default (ZE/1280). Reset gateway first to exercise that path. + // default (openai:text-embedding-3-small/1280 as of v0.42.68.0). + // Reset gateway first to exercise that path. const { resetGateway } = await import('../src/core/ai/gateway.ts'); const { getEmbeddingColumnRegistry } = await import('../src/core/search/embedding-column.ts'); resetGateway(); try { const reg = getEmbeddingColumnRegistry({ engine: 'pglite' } as any); expect(reg['embedding']).toBeDefined(); - expect(reg['embedding'].provider).toBe('zeroentropyai:zembed-1'); + expect(reg['embedding'].provider).toBe('openai:text-embedding-3-small'); expect(reg['embedding'].dimensions).toBe(1280); } finally { // Re-apply legacy preload defaults so the rest of the file's tests