Compare commits

..
Author SHA1 Message Date
Time Attakc 7bee2e48f3 Merge branch 'master' into fix/docs-nonexistent-install-command-3502 2026-07-28 17:09:00 -07:00
Garry Tan 53bb974eaf merge origin/master: union CLI_ONLY additions (pages, bench + backfill from #3529) 2026-07-28 15:57:54 -07:00
Garry Tan bdd23cdede fix(build): restore the executable bit on src/cli.ts
check:cli-exec requires mode 100755; the edit in this branch landed it as
100644, failing `bun run verify` (1/32) on an otherwise-green PR. Mode only,
no content change.
2026-07-28 13:47:11 -07:00
Garry TanandClaude Opus 5 45689dd1bd fix(docs): remove references to the nonexistent gbrain install command (#3502)
`docs/tutorials/personal-brain.md` Step 6 instructed `gbrain install`,
which fails with "Unknown command: install" — the managed-install model
was retired in v0.36.0.0. Step 6 now documents the current flow
(`gbrain init --supabase` in the brain repo, `gbrain skillpack scaffold
--all` in the agent workspace), states which repo each command runs in,
and how it feeds the Step 7 Supabase setup.

Full sweep of README/docs/skills for `gbrain <verb>` invocations against
the live CLI surface (CLI_ONLY + op cliHints names + aliases) found and
fixed every other dead reference: the second `gbrain install` site in
the ethos doc, `put-page`/`put_page`/`get_page` → `put`/`get`,
`add_link` → `link`, `add_timeline_entry --entry` → `timeline-add`
positional form, `get_links`/`put_raw_data`/`get_raw_data` (MCP-only
ops) → `gbrain call <op> '<json>'`, `find_trajectory` →
`find-trajectory`, `gbrain file upload` → `gbrain files upload`,
`gbrain pages restore` → `gbrain restore`, the never-shipped
`gbrain rebuild` → the real recovery sequence, and stale forward-notes
(`gbrain cron`, `gbrain transcription`, `gbrain research init`,
`gbrain plugin list`).

Two documented surfaces turned out to be unwired CODE, not wrong docs,
so they're wired instead of rewritten: `pages` had a live handleCliOnly
case but was missing from CLI_ONLY (the #2035 calibration bug class),
and `bench publish` (bench-publish.ts, referenced by eval-gate's own
--help) was never dispatched — same promised-but-unwired class as
retrieval-upgrade (#3390).

New CI guard: test/docs-cli-commands.test.ts scans README/docs/skills
code blocks + inline spans for `gbrain <verb>` and fails on any verb not
in the live command set (historical docs excluded by design; prose kept
out via comment/diagram/command-position heuristics).

llms bundle regenerated (`bun run build:llms`) for the inlined docs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 13:32:09 -07:00
76 changed files with 428 additions and 2016 deletions
+3 -6
View File
@@ -61,10 +61,7 @@ jobs:
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
# --timeout also raises bun's 5s default hook budget (beforeAll/afterAll
# do NOT inherit a test's third-arg timeout; verified on bun 1.3.x).
# Every runner script in scripts/ passes it; bare invocations must too.
run: bun test --timeout=60000 test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
@@ -91,7 +88,7 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -158,7 +155,7 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test --timeout=60000 test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+1 -3
View File
@@ -29,9 +29,7 @@ jobs:
with:
bun-version: 1.3.13
- run: bun install
# --timeout matches every scripts/ runner and covers hook budgets too
# (bunfig.toml's timeout key is ignored by bun; hooks default to 5s).
- run: bun test --timeout=60000
- run: bun test
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Attest build provenance
-5
View File
@@ -113,11 +113,6 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
# default regardless of per-test third-arg timeouts. Runs directly
# (not via verify's CHECKS array) to avoid a package.json edit.
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
+1 -21
View File
@@ -239,27 +239,7 @@ The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
## Default-provider policy
**A gbrain DEFAULT embedding or reranking model must be either open-weight, or
from the vendor with the longest proven model-lifetime record. Novel/startup
providers may ship as opt-in recipes, never as the default.** Rationale: the
v0.36 zembed-1 default stranded every default-config brain when ZeroEntropy
was acquired and gave ~6 weeks notice (hosted API sunsets 2026-09-04).
Current defaults live in `src/core/ai/defaults.ts`:
`DEFAULT_EMBEDDING_MODEL = 'ollama:bge-m3'`,
`DEFAULT_EMBEDDING_DIMENSIONS = 1024` (bge-m3's native width — open-weight,
local, cannot be sunset by anyone). Because the default has no hosted vendor,
it also has a **declared hosted fallback**: `FALLBACK_EMBEDDING_MODEL =
'openai:text-embedding-3-small'` at 1024 (Matryoshka width pinned to bge-m3's
so a later fallback→default migration rebuilds vectors only — no column ALTER,
no HNSW rebuild). `gbrain init` probes Ollama ONCE (`src/core/ai/ollama-detect.ts`,
≤1.5s, fail-open), persists the resolved choice to config.json, and falls back
LOUDLY — the notice names the multilingual quality cost and the paste-ready way
back, and the `embedding_default_fallback` config marker makes `gbrain doctor`
re-probe on every run so "installed Ollama later" surfaces the switch. Embed
calls never probe. A silent downgrade onto the fallback is a bug.
## Eval discipline (v0.32.3)
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
resolves through `src/core/eval/metric-glossary.ts` so industry terms
+5 -12
View File
@@ -40,23 +40,16 @@ restart the shell or add the PATH export to the shell profile.
## Step 2: API Keys
gbrain's default embedder is `ollama:bge-m3` at 1024 dimensions — local,
open-weight, no API key. If Ollama is installed with the model pulled
(`ollama pull bge-m3`), `gbrain init` detects it automatically. Otherwise
init falls back (loudly) to the hosted `openai:text-embedding-3-small` when
`OPENAI_API_KEY` is set; other providers via `gbrain config set
embedding_model <provider:model>`.
Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack
(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config
set embedding_model <provider:model>`.
```bash
ollama pull bge-m3 # default embedding (local; install Ollama from https://ollama.ai)
export OPENAI_API_KEY=sk-... # hosted fallback embedding; also used for chat models
export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+)
export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion
```
> ZeroEntropy was the default embedder + reranker from v0.36.2.0 through
> v0.42.x. Its hosted API shuts down 2026-09-04. Brains still on it get a
> one-time `gbrain upgrade` banner with the exact migration commands.
Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically
or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider,
keyword search still works. Without Anthropic, search works but skips query expansion.
+3 -3
View File
@@ -290,8 +290,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
- **Embedding providers**: 16 recipes covering Ollama (local — `bge-m3` at 1024d is the default: open-weight, multilingual, cannot be sunset), OpenAI (`text-embedding-3-small` is the hosted fallback when Ollama isn't available), OpenRouter, Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). ZeroEntropy was the default from v0.36.2.0 to v0.42.x; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the migration commands. Default-provider policy lives in [`CLAUDE.md`](CLAUDE.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (still the `tokenmax`-mode default; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the switch instructions, and the `zerank` weights are Apache-2.0 so the local recipe below keeps working) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
@@ -459,4 +459,4 @@ MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production
Origin story: [`docs/ethos/ORIGIN.md`](docs/ethos/ORIGIN.md).
Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that shipped as the default from v0.36.2.0 to v0.42.x. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.
Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.
-12
View File
@@ -1,17 +1,5 @@
# ZeroEntropy — zembed-1 + zerank-2
> **Deprecated as a hosted provider. The ZeroEntropy hosted API shuts down
> 2026-09-04.** `zembed-1` was GBrain's default embedder from v0.36.2.0
> through v0.42.x; the default is now `ollama:bge-m3` at 1024 dimensions
> (open-weight, local — see the Default-provider policy in `CLAUDE.md`),
> with `openai:text-embedding-3-small` as the hosted fallback. If your
> brain still embeds through ZeroEntropy, migrate before that date —
> `gbrain upgrade` prints the exact commands for your brain, and
> [`../guides/embedding-migration.md`](../guides/embedding-migration.md)
> walks both targets. The `zembed-1` and `zerank` weights are Apache-2.0,
> so self-hosting via llama-server or Ollama is the other forward path and
> preserves your existing vectors outright.
[ZeroEntropy](https://zeroentropy.dev) ships two specialized small models
for retrieval pipelines:
+1 -1
View File
@@ -396,7 +396,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
- `src/commands/pages.ts``gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/commands/pages.ts``gbrain purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `recordCompleted` binds its array through `$3::text::jsonb` (NOT a bare `$3::jsonb`) so postgres.js `.unsafe()` doesn't double-encode `JSON.stringify(sorted)` into the scalar string that CHECK rejects — the #2339 bug that aborted every multi-source sync at the first pin write (PGLite parsed it silently, so it shipped). A DATABASE_URL-gated `test/e2e/op-checkpoint-jsonb-parity.test.ts` (its own CI job) asserts the array shape on real Postgres. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
+1 -1
View File
@@ -229,7 +229,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
+1 -1
View File
@@ -214,7 +214,7 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
v0.39-typed pages that no longer have a matching type in the active
pack.
+10 -10
View File
@@ -19,11 +19,13 @@ entire DB from scratch.
This means:
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Disaster recovery is a short, boring sequence.** If your DB volume
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
don't need a backup. You wipe the derived tables (on PGLite,
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
your brain repo with `gbrain sync`, and `gbrain extract all`
regenerates the derived state. See "Disaster recovery" below for the
exact commands.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
@@ -146,11 +148,9 @@ The promise the rule makes:
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
# Wipe and rebuild — delete the derived tables (pages + content_chunks
# survive the CASCADE-safe design), then re-derive from the repo.
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
+2 -2
View File
@@ -108,8 +108,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
+1 -1
View File
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain install voice-agent`
`gbrain skillpack scaffold voice-agent`
That's it.
+1 -1
View File
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain add_timeline_entry {slug} {
gbrain timeline-add {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
+9 -9
View File
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
+3 -24
View File
@@ -3,30 +3,9 @@
`gbrain migrate embeddings` re-embeds an entire brain onto a different
embedding provider/model, safely and resumably. It is the forward path off a
sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
2026-09-04 and was the shipped default from v0.36.2.0 through v0.42.x) — but
it is provider-agnostic: any configured `provider:model` works as a target.
**Coming off the ZeroEntropy default?** Two targets, pick one:
```bash
# A) The current default — open-weight, local, free. Requires Ollama with
# `ollama pull bge-m3`. Changes column width (e.g. 1280 → 1024), so this
# includes a dimension transition + index rebuild:
gbrain migrate embeddings --to ollama:bge-m3 --dim 1024 --dry-run
gbrain migrate embeddings --to ollama:bge-m3 --dim 1024
# B) Hosted, no schema change — pass --dim at your brain's CURRENT width
# (check `gbrain doctor`). OpenAI text-embedding-3-* is Matryoshka, so
# migrating at the width you already have reuses the existing vector(N)
# column and its HNSW index; only the vectors are rebuilt. Note: weaker
# on non-English content than bge-m3.
gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280 --dry-run
gbrain migrate embeddings --to openai:text-embedding-3-small --dim 1280
```
Omitting `--dim` resolves the target recipe's default width (1536 for the
OpenAI recipe), which forces a needless dimension transition — always pass it
explicitly.
2026-09-04 and is the shipped default for brains that never picked a model) —
but it is provider-agnostic: any configured `provider:model` works as a
target.
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
README reference).
+9 -9
View File
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
gbrain call put_raw_data \
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
+5 -5
View File
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain get_links <attendee_slug>,
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
}
else:
briefing[attendee] = "No brain page -- consider enriching"
@@ -67,14 +67,14 @@ on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
gbrain timeline-add <sender_slug> {date} \
"Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
+7 -7
View File
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
gbrain timeline-add <person_slug> {date} \
"Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
gbrain timeline-add <company_slug> {date} \
"Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
---
+3 -3
View File
@@ -91,7 +91,7 @@ first):
6. The seeded `default` source.
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
the `gstack` source. Outside any registered directory with no env/dotfile
set, it writes to the default.
@@ -188,10 +188,10 @@ citations keep working.
```bash
# Pass --source explicitly
gbrain put-page topics/ai ... --source wiki
gbrain put topics/ai ... --source wiki
# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put-page plans/multi-repo ...
cd ~/.gstack && gbrain put plans/multi-repo ...
# → source auto-resolves to gstack
```
+8 -8
View File
@@ -20,8 +20,8 @@ on every_inbound_message(message):
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
# The brain COMPOUNDS overnight.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list_pages
pages = gbrain list
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain get_links <page.slug>
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
for mention in mentions:
if mention not in existing_links:
gbrain add_link <page.slug> <mention_slug> # fix broken graph
gbrain link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain get_timeline <page.slug>
timeline = gbrain timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+3 -3
View File
@@ -47,8 +47,8 @@ on user_message(message):
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
+1 -1
View File
@@ -87,7 +87,7 @@ expect it.
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Shown in future `gbrain plugin list`. |
| `description` | string | no | Shown in a future plugin-listing command. |
## Subagent definition files
+10 -22
View File
@@ -15,13 +15,7 @@ gbrain init --pglite --model voyage # use a non-default provider
## Init resolves your provider from env keys
As of v0.37, `gbrain init --pglite` auto-detects which provider to use. Detection order:
1. **The default: `ollama:bge-m3` (1024d, local, open-weight).** Init probes the local Ollama daemon once (≤1.5s, `OLLAMA_BASE_URL` honored). If it's running with `bge-m3` pulled, the default wins — over every env key, since it needs no key and costs nothing.
2. **The hosted fallback: `openai:text-embedding-3-small` (1024d).** If Ollama is unreachable (or the model isn't pulled) and `OPENAI_API_KEY` is set, init falls back — loudly. The notice names the trade-off (text-embedding-3-small is the weakest multilingual performer among the candidates we evaluated) and the paste-ready way back; a config marker makes `gbrain doctor` re-probe for Ollama on every run, so installing Ollama later surfaces the switch automatically.
3. **Other single env keys** auto-pick that provider; multiple keys fire an interactive picker. If nothing resolves in a non-TTY context (CI, Docker build), init exits 1 with a paste-ready setup hint. Explicit flags (`--embedding-model`, `--no-embedding`) always win over detection.
The probe runs ONCE at init — never on embed calls.
As of v0.37, `gbrain init --pglite` auto-detects which provider to use from your env vars. With `OPENAI_API_KEY` set, you get OpenAI. With `ZEROENTROPY_API_KEY` set, you get ZeroEntropy. If multiple provider keys are set, init fires an interactive picker. If no provider keys are set in a non-TTY context (CI, Docker build), init exits 1 with a paste-ready setup hint. Explicit flags (`--embedding-model`, `--no-embedding`) always win over env detection.
The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atomically, so subsequent runs are deterministic across releases.
@@ -29,9 +23,8 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
| Provider | env vars | default dims | cost ($/1M tokens) | local? | multimodal? |
|---|---|---|---|---|---|
| `ollama` (**default**: `bge-m3` @ 1024) | (none — runs locally; `OLLAMA_BASE_URL` optional) | per-model (bge-m3 1024, nomic 768, ...) | 0 | yes | no |
| `openai` (**hosted fallback**: `text-embedding-3-small` @ 1024) | `OPENAI_API_KEY` | recipe 1536; fallback pins 1024 | 0.02 (-small) / 0.13 (-large) | no | no |
| `zeroentropyai` (sunsets 2026-09-04) | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no |
| `zeroentropyai` | `ZEROENTROPY_API_KEY` | 2560 (Matryoshka to 1280/640/320/...) | 0.05 | no | no |
| `openai` | `OPENAI_API_KEY` | 1536 | 0.13 | no | no |
| `openrouter` | `OPENROUTER_API_KEY` | 1536 | 0.02 | no | model-dependent |
| `voyage` | `VOYAGE_API_KEY` | 1024 | 0.18 | no | yes (`voyage-multimodal-3`) |
| `google` | `GOOGLE_GENERATIVE_AI_API_KEY` | 768 | 0.025 | no | no |
@@ -39,6 +32,7 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
| `minimax` | `MINIMAX_API_KEY` | 1536 | 0.07 | no | no |
| `dashscope` | `DASHSCOPE_API_KEY` | 1024 | varies | no | no |
| `zhipu` | `ZHIPUAI_API_KEY` | 1024 | varies | no | no |
| `ollama` | (none — runs locally) | 768 | 0 | yes | no |
| `llama-server` | (none — runs locally) | user-set | 0 | yes | no |
| `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | yes (backend permitting) |
| `together` | `TOGETHER_API_KEY` | 768 | varies | no | no |
@@ -46,7 +40,7 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
| `deepseek` | (no embedding model — chat only) | — | — | — | — |
| `groq` | (no embedding model — chat only) | — | — | — | — |
**Note on local providers.** llama-server has no required API key, so it never shows up in env-detection auto-pick — pick it explicitly with `--embedding-model llama-server:<model>`. Ollama is special-cased as the system default: init verifies the daemon is actually reachable AND `bge-m3` is pulled before selecting it, so it can never silently route to a daemon that isn't running. Other Ollama models remain explicit-only (`--embedding-model ollama:<model>`).
**Note on local providers.** Ollama and llama-server have no required API key, so they don't show up in env-detection auto-pick. Pick them explicitly with `--embedding-model ollama:<model>` to avoid silently routing to a daemon that may not be running.
## If first import fails
@@ -81,9 +75,7 @@ The doctor distinguishes two repair paths:
### OpenAI
**The hosted fallback.** When Ollama isn't available at init and `OPENAI_API_KEY` is set, init lands on `openai:text-embedding-3-small` at **1024** dimensions — pinned to bge-m3's width so a later switch to the default rebuilds vectors only (no column change, no HNSW rebuild). Be aware of the trade-off: `text-embedding-3-small` was the weakest multilingual performer among the candidates we evaluated; if your brain carries substantial non-English content, prefer the bge-m3 default.
Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 recipe default), `text-embedding-3-small` (1536 native). Both are Matryoshka via the `dimensions` field (any integer width ≤ native) — gbrain pins it from `embedding_dimensions` config so existing brains stay aligned across SDK upgrades.
Default. Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 default), `text-embedding-3-small` (1536). Matryoshka via the `dimensions` field — gbrain pins it from `embedding_dimensions` config so existing 1536-dim brains stay aligned across SDK upgrades.
Optional `OPENAI_BASE_URL` — point the native OpenAI provider at an OpenAI-compatible gateway. A bare host is normalized to carry the `/v1` suffix automatically (so `https://gw.example.com` and `https://gw.example.com/v1` both work); when unset, the SDK's default endpoint is untouched. `ANTHROPIC_BASE_URL` gets the same normalization for Anthropic chat/expansion calls.
@@ -147,17 +139,13 @@ CJK-dominant content tokenizes denser than OpenAI tiktoken; gbrain declares `cha
Set `ZHIPUAI_API_KEY`. Models: `embedding-3` (current; Matryoshka 256-2048 dims), `embedding-2`. v0.32 default is 1024 (HNSW-compatible). The 2048-dim option works but falls into the exact-scan branch (see Voyage 4 Large note above).
### Ollama (local) — the default
### Ollama (local)
**`ollama:bge-m3` at its native 1024 dimensions is gbrain's default embedder.** Open-weight (nobody can sunset it), free, local, and the strongest open-weight multilingual retriever among the candidates we evaluated — it holds retrieval quality on non-Latin-script content where small hosted models degrade sharply. Setup: install Ollama from https://ollama.ai, then `ollama pull bge-m3`; `gbrain init` detects it automatically.
No env required — Ollama runs unauthenticated locally. Optional `OLLAMA_BASE_URL` (default `http://localhost:11434/v1`) and `OLLAMA_API_KEY` (for auth-enabled deployments).
**Throughput expectation (one-time):** local bge-m3 embeds roughly 8× slower than hosted APIs (~7 docs/s vs ~60 docs/s on the eval hardware). For a 10K-page brain, that's the first full embed taking ~23 minutes instead of ~3. After the initial sync, embeds are incremental and the difference is unnoticeable — and queries are unaffected.
Recipe ships with `nomic-embed-text` (768d, recommended), `mxbai-embed-large` (1024d), `all-minilm` (384d), plus the larger modern embedders `qwen3-embed-8b` (4096d) and `snowflake-arctic-embed-l-v2` (1024d). `gbrain providers test --model ollama:nomic-embed-text` smoke-tests the local install.
No env required — Ollama runs unauthenticated locally. Optional `OLLAMA_BASE_URL` (default `http://localhost:11434/v1`; also honored by init's availability probe) and `OLLAMA_API_KEY` (for auth-enabled deployments).
Recipe also ships `nomic-embed-text` (768d), `mxbai-embed-large` (1024d), `all-minilm` (384d), plus the larger modern embedders `qwen3-embed-8b` (4096d) and `snowflake-arctic-embed-l-v2` (1024d). `gbrain providers test --model ollama:bge-m3` smoke-tests the local install.
Widths resolve per model (`model_dims` in the recipe); for models not named there, declare the native dimension with `--embedding-dimensions <N>` at init — gbrain trusts the value you declare for local recipes.
The recipe default is `nomic-embed-text`'s 768 dims. If you run one of the larger models, declare its native dimension with `--embedding-dimensions <N>` at init — gbrain trusts the value you declare for local recipes instead of rejecting a non-768 width.
### llama-server (local, llama.cpp)
+1 -1
View File
@@ -250,7 +250,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+1 -1
View File
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect``suggest``review-candidates` so the brain learns your shape instead of forcing you to learn its.
+4 -8
View File
@@ -492,11 +492,7 @@ OAuth source scoping only guards the HTTP MCP path. If the brain's Postgres and
## Part 13: Cost and speed expectations
Real numbers from the published benchmark. The benchmark run used ZeroEntropy
for embedding + reranker, which was the default through v0.42.x; the default
embedder is now the local `ollama:bge-m3` at 1024 dimensions ($0 per token,
see the Default-provider policy in `CLAUDE.md`), with
`openai:text-embedding-3-small` ($0.02/M tokens) as the hosted fallback:
Real numbers from the published benchmark, running the default stack (GBrain with ZeroEntropy for embedding + reranker):
- **Embedding cost:** $0.05 per million tokens. For comparison, GBrain configured with OpenAI is $0.13 (2.6× more expensive), Voyage is $0.18 (3.6× more).
- **Ingest speed:** about 22 seconds for a small test corpus of 164 pages on the host machine. For a 10K-page corpus, expect about 20 minutes the first time, then most syncs are incremental and finish in seconds.
@@ -506,7 +502,7 @@ see the Default-provider policy in `CLAUDE.md`), with
Full methodology and per-run receipt JSONs live in [the gbrain-evals repo](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-05-23-v0.40.6.0-snapshot.md).
For a 25-person company at sustained use, expect $0 a month in embeddings on the default (local `ollama:bge-m3`) or about $15 a month on the hosted fallback (`text-embedding-3-small` at $0.02/million tokens), $50 a month in Anthropic calls for the synthesized-answer queries, plus your hosting bill. Under $100 a month for the AI side at most companies your size.
For a 25-person company at sustained use, expect about $35 a month in embeddings (ZeroEntropy at $0.05/million tokens), $50 a month in Anthropic calls for the synthesized-answer queries, plus your hosting bill. Under $100 a month for the AI side at most companies your size.
---
@@ -518,7 +514,7 @@ Check `gbrain auth list` on the host and confirm their client has `--source` set
### "Sync is slow and feels stuck"
The first sync embeds every page, which takes time. Check `gbrain sources status` for the live page count. If it's climbing you're not stuck, you're just embedding. If you've got a 10K-page corpus and your embedding provider is slow or throttled (the local bge-m3 default embeds at roughly an eighth of hosted-API speed — about 23 minutes for a 10K-page first sync instead of ~3, one time), the per-source parallel sync looks like progress on three sources at once rather than one source moving fast.
The first sync embeds every page, which takes time. Check `gbrain sources status` for the live page count. If it's climbing you're not stuck, you're just embedding. If you've got a 10K-page corpus and ZeroEntropy is being throttled, the per-source parallel sync looks like progress on three sources at once rather than one source moving fast.
### "I see a page I shouldn't see"
@@ -558,7 +554,7 @@ What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
+11 -10
View File
@@ -103,10 +103,11 @@ Render will build a Docker container with the harness. First deploy takes about
In the AlphaClaw UI (Providers tab):
- **OpenAI API Key.** Recommended. GBrain's default embedder is the local `ollama:bge-m3`; on a hosted deployment without Ollama, init falls back to `openai:text-embedding-3-small` via this key.
- **OpenAI API Key.** Required for embeddings if you use the OpenAI provider.
- **Anthropic API Key.** Required for Claude (the main model the agent talks through).
- **Perplexity API Key.** Optional, for web search.
- **Voyage API Key.** Optional, alternative to OpenAI for embeddings.
- **ZeroEntropy API Key.** Recommended. GBrain ships with ZeroEntropy as the default embedder + reranker because it's about 2× faster than OpenAI and about 2.6× cheaper.
You can use the same keys across multiple agents.
@@ -114,21 +115,21 @@ You can use the same keys across multiple agents.
## Step 6: Install GBrain
Once OpenClaw is running:
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
```bash
gbrain install
# In the BRAIN repo (the git repo that holds your markdown pages):
gbrain init --supabase
# In the AGENT WORKSPACE repo (where OpenClaw runs):
gbrain skillpack scaffold --all
```
This installs:
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
From this point, the agent has working memory and access to every skill.
---
+3 -3
View File
@@ -32,7 +32,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
+13 -40
View File
@@ -388,27 +388,7 @@ The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
## Default-provider policy
**A gbrain DEFAULT embedding or reranking model must be either open-weight, or
from the vendor with the longest proven model-lifetime record. Novel/startup
providers may ship as opt-in recipes, never as the default.** Rationale: the
v0.36 zembed-1 default stranded every default-config brain when ZeroEntropy
was acquired and gave ~6 weeks notice (hosted API sunsets 2026-09-04).
Current defaults live in `src/core/ai/defaults.ts`:
`DEFAULT_EMBEDDING_MODEL = 'ollama:bge-m3'`,
`DEFAULT_EMBEDDING_DIMENSIONS = 1024` (bge-m3's native width — open-weight,
local, cannot be sunset by anyone). Because the default has no hosted vendor,
it also has a **declared hosted fallback**: `FALLBACK_EMBEDDING_MODEL =
'openai:text-embedding-3-small'` at 1024 (Matryoshka width pinned to bge-m3's
so a later fallback→default migration rebuilds vectors only — no column ALTER,
no HNSW rebuild). `gbrain init` probes Ollama ONCE (`src/core/ai/ollama-detect.ts`,
≤1.5s, fail-open), persists the resolved choice to config.json, and falls back
LOUDLY — the notice names the multilingual quality cost and the paste-ready way
back, and the `embedding_default_fallback` config marker makes `gbrain doctor`
re-probe on every run so "installed Ollama later" surfaces the switch. Embed
calls never probe. A silent downgrade onto the fallback is a bug.
## Eval discipline (v0.32.3)
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
resolves through `src/core/eval/metric-glossary.ts` so industry terms
@@ -1050,23 +1030,16 @@ restart the shell or add the PATH export to the shell profile.
## Step 2: API Keys
gbrain's default embedder is `ollama:bge-m3` at 1024 dimensions — local,
open-weight, no API key. If Ollama is installed with the model pulled
(`ollama pull bge-m3`), `gbrain init` detects it automatically. Otherwise
init falls back (loudly) to the hosted `openai:text-embedding-3-small` when
`OPENAI_API_KEY` is set; other providers via `gbrain config set
embedding_model <provider:model>`.
Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack
(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config
set embedding_model <provider:model>`.
```bash
ollama pull bge-m3 # default embedding (local; install Ollama from https://ollama.ai)
export OPENAI_API_KEY=sk-... # hosted fallback embedding; also used for chat models
export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+)
export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion
```
> ZeroEntropy was the default embedder + reranker from v0.36.2.0 through
> v0.42.x. Its hosted API shuts down 2026-09-04. Brains still on it get a
> one-time `gbrain upgrade` banner with the exact migration commands.
Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically
or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider,
keyword search still works. Without Anthropic, search works but skips query expansion.
@@ -1811,8 +1784,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
- **Embedding providers**: 16 recipes covering Ollama (local — `bge-m3` at 1024d is the default: open-weight, multilingual, cannot be sunset), OpenAI (`text-embedding-3-small` is the hosted fallback when Ollama isn't available), OpenRouter, Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md). ZeroEntropy was the default from v0.36.2.0 to v0.42.x; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the migration commands. Default-provider policy lives in [`CLAUDE.md`](CLAUDE.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (still the `tokenmax`-mode default; its hosted API sunsets 2026-09-04 — `gbrain upgrade` prints the switch instructions, and the `zerank` weights are Apache-2.0 so the local recipe below keeps working) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Embedding providers**: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
- **Rerankers**: ZeroEntropy `zerank-2` hosted (default in `tokenmax` mode) plus the v0.40.6.1 `llama-server-reranker` recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same `gateway.rerank()` seam. Setup walkthrough in [`docs/ai-providers/llama-server-reranker.md`](docs/ai-providers/llama-server-reranker.md).
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
@@ -1980,7 +1953,7 @@ MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production
Origin story: [`docs/ethos/ORIGIN.md`](docs/ethos/ORIGIN.md).
Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that shipped as the default from v0.36.2.0 to v0.42.x. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.
Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.
---
@@ -2343,7 +2316,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -2373,7 +2346,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -2454,7 +2427,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
@@ -3924,7 +3897,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
-41
View File
@@ -1,41 +0,0 @@
#!/usr/bin/env bash
# CI guard: every `bun test` invocation in workflows and runner scripts must
# pass an explicit --timeout.
#
# Why: bun ignores bunfig.toml's `timeout` key (verified on 1.3.14), so a bare
# `bun test` gets the 5000ms default for BOTH tests and beforeAll/beforeEach/
# afterAll/afterEach hooks. Hooks do NOT inherit a test's third-arg timeout —
# a file whose tests all declare `}, 30_000)` still has a 5s hook budget, and
# slow setup (Postgres connect + migrations, PGLite cold start) flakes on
# loaded CI runners with the signature `(unnamed) [5001ms] ... hook timed out`
# (the #3545 jsonb-parity failure). The CLI --timeout flag is the one measured
# mechanism that raises the hook budget uniformly; per-hook second-arg
# timeouts work too but don't scale to ~400 slow hooks.
#
# Usage: scripts/check-bun-test-timeout.sh
# Exit: 0 when clean, 1 when a bare `bun test` invocation is found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Match executable `bun test` invocations. Exclude comment lines (#, //, *)
# and lines that already carry --timeout anywhere.
# Scope: workflows + runner scripts (the surfaces CI executes). package.json
# script bodies route through scripts/ already; editing it is out of scope here.
violations="$(grep -rnE '\bbun test\b' .github/workflows scripts 2>/dev/null \
| grep -v -- '--timeout' \
| grep -vE ':[[:space:]]*(#|//|\*)' \
| grep -v 'check-bun-test-timeout' \
|| true)"
if [ -n "$violations" ]; then
echo "FAIL: bare 'bun test' without --timeout (5s default kills slow setup hooks):" >&2
echo "$violations" >&2
echo "" >&2
echo "Add --timeout=60000 (see scripts/run-unit-shard.sh for the convention)." >&2
exit 1
fi
echo "OK: every bun test invocation passes an explicit --timeout."
-103
View File
@@ -1,103 +0,0 @@
#!/usr/bin/env node
/**
* Import an envelope-v0 file (a JSON serialization of AI chat history; format
* spec: github.com/memvelope/memvelope) into a brain repo as one Markdown page
* per conversation, which `gbrain sync` ingests.
*
* Usage:
* node scripts/envelope-to-gbrain.mjs <envelope.mve.json> [outDir]
*
* Zero dependencies. Deterministic. No network. It does NOT call gbrain it
* only writes Markdown files.
*
* Output layout:
* - One page per conversation, filename = date + conversation id (shared
* titles cannot collide; the id is the natural key). A duplicate id
* overwrites its own filename and warns on stderr; stdout reports DISTINCT
* files written, not write calls.
* - Frontmatter: `type: conversation` (keeps pages eligible for
* conversation-facts extraction and chronicle behavior after sync), the
* source provider, the conversation id, and `origin: memvelope/envelope-v0`.
* - Page `date` is the first 10 chars of the conversation's ISO-8601
* `created_at`. Body keeps message-id citations beside each speaker turn.
*
* Memory: the whole envelope is held in memory (no streaming); envelopes are
* far smaller than the vendor exports they serialize.
*
* Verify:
* node scripts/envelope-to-gbrain.mjs test/fixtures/memvelope/sample.mve.json /tmp/out
* -> expect "wrote 1 markdown page(s)"
* bun test test/envelope-to-gbrain.test.ts
*
* STATUS: live-verified against gbrain v0.42.56.0 on 2026-07-03: the sample
* fixture -> 1 page; a real 662MB Claude export -> 353 conversations = 353
* distinct pages (no collisions), searchable after sync with provenance and
* message-id citations intact.
*/
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
const [, , envelopePath, outDir = './brain/conversations'] = process.argv;
if (!envelopePath) {
console.error('usage: node envelope-to-gbrain.mjs <envelope.mve.json> [outDir]');
process.exit(1);
}
const env = JSON.parse(readFileSync(envelopePath, 'utf8'));
if (env.memvelope !== 'envelope-v0') {
console.error(`not an envelope-v0 file (memvelope field = ${JSON.stringify(env.memvelope)})`);
process.exit(1);
}
const slug = (s, fallback) =>
(String(s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '') || fallback).slice(0, 60);
mkdirSync(outDir, { recursive: true });
const filesWritten = new Set();
let collisions = 0;
const conversations = env.conversations || [];
for (const [i, c] of conversations.entries()) {
const date = (c.created_at || '').slice(0, 10);
// Name the file by the conversation's own id — the natural unique key — so two
// conversations that share a date and title can never silently overwrite each
// other. The date only leads as a human/chronological sort prefix; the id
// carries uniqueness. Positional fallback keeps names unique and deterministic
// when an envelope omits an id.
const convId = (typeof c.id === 'string' && c.id.trim()) ? c.id.trim() : `conv-${i + 1}`;
const name = `${date || '0000-00-00'}-${slug(convId, `conv-${i + 1}`)}.md`;
// gbrain reads YAML frontmatter + markdown body; keep provenance in frontmatter.
// Emit `type: conversation` so gbrain stores these as conversation pages rather
// than defaulting to the generic `concept`. gbrain is open-typed — it takes an
// explicit frontmatter `type` verbatim — and its conversation-aware features
// (conversation-facts extraction, the conversation_format_coverage check,
// chronicle eligibility) key off `type == 'conversation'`.
const front = [
'---',
'type: conversation',
`title: ${JSON.stringify(c.title || 'Untitled conversation')}`,
`date: ${date || 'null'}`,
`source: ${env.meta?.source_provider || 'unknown'}`,
`memvelope_conversation_id: ${JSON.stringify(c.id)}`,
'origin: memvelope/envelope-v0',
'---',
'',
].join('\n');
const body = (c.messages || [])
.map((m) => `**${m.role === 'user' ? 'Me' : 'Assistant'}** (${m.ts || 'no timestamp'} · ${m.id}):\n\n${m.text}`)
.join('\n\n---\n\n');
// Never lose a page silently: if two conversations still map to the same
// filename (e.g. an envelope carrying duplicate ids), warn loudly instead of
// overwriting in silence, and report the count of DISTINCT files written — not
// the number of write calls, which is what hid the old title-collision bug.
if (filesWritten.has(name)) {
collisions += 1;
console.warn(`warning: filename collision on "${name}" — conversation id ${JSON.stringify(c.id)} is not unique; overwriting the earlier page.`);
}
writeFileSync(join(outDir, name), front + `# ${c.title || 'Conversation'}\n\n` + body + '\n');
filesWritten.add(name);
}
console.log(`wrote ${filesWritten.size} markdown page(s) to ${outDir} — point gbrain's sync at this directory.`);
if (collisions) {
console.warn(`warning: ${collisions} filename collision(s) — ${collisions} page(s) overwritten. Deduplicate conversation ids in the envelope to avoid data loss.`);
}
+2 -3
View File
@@ -162,9 +162,8 @@ for f in "${files[@]}"; do
if [ -n "${DATABASE_URL:-}" ]; then
psql "$DATABASE_URL" -At -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND datname = current_database()" >/dev/null 2>&1 || true
fi
# Hard outer timeout (180s per file). bun's --timeout covers tests AND
# hooks (measured on 1.3.14), but it's timer-based: a PGLite WASM call
# that blocks the event loop synchronously never lets the timer fire and
# Hard outer timeout (180s per file). bun's --timeout is per-test; if a
# PGLite WASM call hangs in beforeAll/afterAll, --timeout never fires and
# the file wedges indefinitely. gtimeout/timeout SIGKILLs the file so the
# suite advances. gtimeout (macOS via coreutils) preferred; timeout (Linux)
# fallback; bare bun (no outer cap) if neither is installed.
+1 -1
View File
@@ -248,7 +248,7 @@ before submission.
After the brain page is written, render to PDF using `skills/brain-pdf`:
```bash
gbrain put_page # already done by the CLI; nothing to add here
gbrain put # already done by the CLI; nothing to add here
# Then invoke brain-pdf:
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
```
+6 -6
View File
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
using `agentTurn`. Respect that. No auto-rewrite.
## Forward note (v0.12.0)
## Forward note
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
`gbrain jobs work` that owns cron expressions natively — no more
handing off to host schedulers. Until v0.12.0 lands, the host
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
layer (what the cron trigger *does*), not the scheduling layer.
A native scheduler loop inside `gbrain jobs work` (owning cron
expressions directly, with no host-scheduler hand-off) has been on the
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
firing on schedule; this convention only replaces the execution layer
(what the cron trigger *does*), not the scheduling layer.
## Related
+2 -2
View File
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
- Define a custom recipe with: source queries, classification rules, extraction schema,
tracker page path, tracker format
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
to scaffold a new one.
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
copying a built-in recipe file and editing its fields.
### Phase 2: Search Sources
+1 -1
View File
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
### 4b. Entity pages (people, companies)
For each entity mentioned:
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
- If exists: update State, append Timeline entry citing this research.
- If not: create with enrichment.
+1 -1
View File
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
# 4. Write the structured research page via put_page:
gbrain put_page research/<slug> # via the put_page operation
gbrain put research/<slug> # via the put_page operation
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
```
+4 -4
View File
@@ -11,7 +11,7 @@ tools:
- gbrain schema active
- gbrain schema use
- gbrain schema stats
- gbrain pages restore
- gbrain restore
- mcp:run_onboard
triggers:
- "unify my types"
@@ -143,7 +143,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
```bash
gbrain pages restore <slug>
gbrain restore <slug>
```
Revert the active pack flip:
@@ -197,7 +197,7 @@ Outputs:
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
Side effects:
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
@@ -212,7 +212,7 @@ DON'T:
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
## Output Format
+6 -4
View File
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
The user sends an audio or voice message via any channel (Telegram, voice
memo upload, openclaw audio attachment). The host agent typically provides
the transcript text. If not, transcribe via `gbrain transcription` (Groq
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
the transcript text. If not, transcribe it with your host's transcription
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
audio > 25MB via ffmpeg first).
## The pipeline
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
1. STORE → Upload original audio to gbrain storage backend
(S3 / Supabase Storage / local — pluggable per
src/core/storage.ts).
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
gbrain transcription if no transcript was supplied.
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
transcribe the audio yourself (see "When to invoke")
if no transcript was supplied.
3. ROUTE → Apply the decision tree (below) to find the right
destination directory.
4. WRITE → Create / update the destination brain page; preserve the
+20 -1
View File
@@ -55,12 +55,17 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'backfill']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
const CLI_ONLY_SELF_HELP = new Set([
'upgrade', 'post-upgrade', 'check-update',
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
// bench-publish.ts printHelp). Both were documented but undispatchable —
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
// (the #2035 calibration bug class); `bench` was never wired at all.
'pages', 'bench',
'embed', 'config',
'skillpack', 'skillpack-check',
'integrations', 'friction',
@@ -1265,6 +1270,20 @@ async function handleCliOnly(command: string, args: string[]) {
await runInit(args);
return;
}
if (command === 'bench') {
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
if (args[0] === 'publish') {
const { runBenchPublish } = await import('./commands/bench-publish.ts');
await runBenchPublish(args.slice(1));
return;
}
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
console.error('Run `gbrain bench publish --help` for the full flag list.');
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
}
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
// Spawns its own engine internally so no pre-bound engine needed.
if (command === 'reinit-pglite') {
+2 -75
View File
@@ -2337,65 +2337,6 @@ export async function checkZeEmbeddingHealth(engine: BrainEngine): Promise<Check
}
}
/**
* embedding_default_fallback doctor check.
*
* When `gbrain init` couldn't reach Ollama (or bge-m3 wasn't pulled) it
* lands on the hosted fallback and writes the `embedding_default_fallback`
* marker to config.json. This check is how "install Ollama later" becomes
* visible: while the marker is set AND the brain is still on the fallback
* model, re-probe Ollama (bounded 1.5s, fail-open) and print the paste-ready
* migrate command back to the default. Exported for test/doctor tests.
*/
export async function checkEmbeddingDefaultFallback(_engine: BrainEngine): Promise<Check> {
const name = 'embedding_default_fallback';
try {
const { loadConfigFileOnly } = await import('../core/config.ts');
const cfg = loadConfigFileOnly();
if (!cfg?.embedding_default_fallback) {
return { name, status: 'ok', message: 'Not on the hosted embedding fallback — skip.' };
}
const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS, FALLBACK_EMBEDDING_MODEL } =
await import('../core/ai/defaults.ts');
if (cfg.embedding_model !== FALLBACK_EMBEDDING_MODEL) {
// Stale marker (user moved to another model since); ignore.
return {
name,
status: 'ok',
message: `Fallback marker present but embedding_model="${cfg.embedding_model}" is no longer the fallback — stale, ignoring.`,
};
}
const bareModel = DEFAULT_EMBEDDING_MODEL.split(':')[1];
const { probeOllamaModel } = await import('../core/ai/ollama-detect.ts');
const probe = await probeOllamaModel(bareModel);
if (probe.ok) {
return {
name,
status: 'warn',
message:
`This brain is on the hosted embedding fallback (${FALLBACK_EMBEDDING_MODEL}), but Ollama ` +
`with ${bareModel} is now available. The default (${DEFAULT_EMBEDDING_MODEL}) is local, free, ` +
`open-weight, and stronger on non-English content. Switch (same column width, vectors ` +
`rebuilt, no schema change): gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${DEFAULT_EMBEDDING_DIMENSIONS}`,
};
}
const why = probe.serverUp
? `Ollama is running but ${bareModel} is not pulled (fix: ollama pull ${bareModel})`
: 'Ollama is not reachable';
return {
name,
status: 'ok',
message:
`On the hosted embedding fallback (${FALLBACK_EMBEDDING_MODEL}); ${why}. ` +
`To move to the default: install Ollama, \`ollama pull ${bareModel}\`, then ` +
`\`gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${DEFAULT_EMBEDDING_DIMENSIONS}\`.`,
};
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
return { name, status: 'warn', message: `Could not check embedding fallback state: ${msg}` };
}
}
/**
* v0.36.0.0 (A5): embedding_width_consistency doctor check.
*
@@ -4408,18 +4349,8 @@ export async function checkCycleFreshness(
: `'${source.id}'`;
const raw = source.config?.last_full_cycle_at;
if (typeof raw !== 'string') {
// #2540: WARN, not FAIL. This check iterates EVERY local_path source,
// so on a multi-source install where only some vaults are cycled
// (e.g. one nightly `gbrain dream --dir <vault>`), a never-cycled
// sibling source turned doctor permanently red — which erodes the
// check's signal until real staleness hides inside the noise (the
// reporter's install masked genuinely stale sources for weeks this
// way). "Never cycled" also fires on a source added minutes ago.
// A source that HAS cycled and then went stale still escalates
// through the warn/fail age thresholds below — that is the
// regression signal this check exists for.
issues.push(`Source ${display} has never completed a full cycle`);
hasWarnings = true;
hasFailures = true;
continue;
}
const last = new Date(raw).getTime();
@@ -4455,7 +4386,7 @@ export async function checkCycleFreshness(
return {
name: 'cycle_freshness',
status: 'warn',
message: `${issues.join('; ')}. Run \`gbrain dream --source <id>\` to cycle a source, or start \`gbrain autopilot\`.`,
message: `${issues.join('; ')}.`,
};
}
return {
@@ -7737,10 +7668,6 @@ export async function buildChecks(
checks.push(await checkZeEmbeddingHealth(engine));
progress.heartbeat('embedding_width_consistency');
checks.push(await checkEmbeddingWidthConsistency(engine));
// Hosted-fallback re-check: nags (warn) only when Ollama+bge-m3 became
// available after an install that fell back to the hosted embedder.
progress.heartbeat('embedding_default_fallback');
checks.push(await checkEmbeddingDefaultFallback(engine));
// v0.41.15.0 (T6, codex #19/#20) — facts.embedding column drift
// parity check. Same drift class as content_chunks, separate column.
progress.heartbeat('facts_embedding_width_consistency');
+15 -104
View File
@@ -229,13 +229,6 @@ export interface ResolvedAIOptions {
chat_model?: string;
/** v0.37 (D9): user opted into deferred embedding setup. */
noEmbedding?: boolean;
/**
* Set when init landed on the hosted FALLBACK_EMBEDDING_MODEL because the
* declared default (ollama:bge-m3) was unavailable. Persisted to
* config.json as `embedding_default_fallback` so `gbrain doctor` can
* re-check for Ollama later and offer the way back to the default.
*/
embeddingFallback?: boolean;
}
/**
@@ -500,15 +493,14 @@ export async function findEnvKeyTypos(
/** Emit the fail-loud "no embedding provider" message + paste-ready setup. */
function printNoEmbeddingProviderHint(typos: Array<{ userSet: string; suggested: string }>): void {
console.error('\nNo embedding provider configured. The default is local + open-weight:');
console.error(' ollama pull bge-m3 # default: ollama:bge-m3 (1024d) — install Ollama from https://ollama.ai');
console.error('Or set a hosted provider key:');
console.error(' export OPENAI_API_KEY=sk-… # fallback: openai:text-embedding-3-small (1024d)');
console.error('\nNo embedding provider configured. Set one of:');
console.error(' export OPENAI_API_KEY=sk-… # openai:text-embedding-3-large (1536d)');
console.error(' export ZEROENTROPY_API_KEY=ze-… # zeroentropyai:zembed-1 (2560d, Matryoshka)');
console.error(' export VOYAGE_API_KEY=pa-… # voyage:voyage-3-large (1024d)');
console.error('Then re-run: gbrain init --pglite');
console.error('');
console.error('Or pick explicitly:');
console.error(' gbrain init --pglite --embedding-model openai:text-embedding-3-small');
console.error(' gbrain init --pglite --embedding-model openai:text-embedding-3-large');
console.error('');
console.error('Or defer setup: gbrain init --pglite --no-embedding');
console.error(' (you can configure later with `gbrain config set embedding_model <id>`)');
@@ -521,83 +513,30 @@ function printNoEmbeddingProviderHint(typos: Array<{ userSet: string; suggested:
}
}
/** Exported for unit tests (probe stubbed via __setOllamaProbeForTests). */
export async function resolveEmbeddingByEnv(out: ResolvedAIOptions, nonInteractive: boolean): Promise<void> {
// --- Tier 3a: the declared default (ollama:bge-m3, open-weight, local). ---
// One cheap probe (≤1.5s, instant ECONNREFUSED when no daemon) per init;
// the resolved choice persists into config.json so no other code path
// ever probes. When Ollama is up with bge-m3 pulled, the default wins
// over every env key — it needs no key, costs nothing, and cannot be
// sunset. See the Default-provider policy in CLAUDE.md.
const {
DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS,
FALLBACK_EMBEDDING_MODEL, FALLBACK_EMBEDDING_DIMENSIONS,
} = await import('../core/ai/defaults.ts');
const { probeOllamaModel } = await import('../core/ai/ollama-detect.ts');
const defaultBareModel = DEFAULT_EMBEDDING_MODEL.split(':')[1];
const probe = await probeOllamaModel(defaultBareModel);
if (probe.ok) {
out.embedding_model = DEFAULT_EMBEDDING_MODEL;
out.embedding_dimensions = DEFAULT_EMBEDDING_DIMENSIONS;
console.error(
`Detected Ollama with ${defaultBareModel}. ` +
`Using ${DEFAULT_EMBEDDING_MODEL} (${DEFAULT_EMBEDDING_DIMENSIONS}d, local, open-weight — the default). ` +
`Override with --embedding-model.`,
);
return;
}
async function resolveEmbeddingByEnv(out: ResolvedAIOptions, nonInteractive: boolean): Promise<void> {
const ready = await groupReadyByProvider('embedding');
const isTTY = !nonInteractive && !!process.stdin.isTTY;
// --- Tier 3b: hosted fallback (loud, never silent). --------------------
// Ollama is unreachable or bge-m3 isn't pulled. Rather than failing the
// install, land on the designated hosted fallback when its key is
// present — and say exactly what that costs and how to get back to the
// default. The `embedding_default_fallback` marker persists to config so
// `gbrain doctor` re-checks for Ollama on every run.
const fallbackProvider = FALLBACK_EMBEDDING_MODEL.split(':')[0];
if (ready.some(p => p.recipeId === fallbackProvider)) {
out.embedding_model = FALLBACK_EMBEDDING_MODEL;
out.embedding_dimensions = FALLBACK_EMBEDDING_DIMENSIONS;
out.embeddingFallback = true;
const why = probe.serverUp
? `Ollama is running but ${defaultBareModel} is not pulled`
: 'Ollama is not reachable';
console.error('');
console.error(`NOTE: gbrain's default embedder is ${DEFAULT_EMBEDDING_MODEL} (local, open-weight), but ${why}.`);
console.error(`Falling back to the hosted ${FALLBACK_EMBEDDING_MODEL} (${FALLBACK_EMBEDDING_DIMENSIONS}d) via OPENAI_API_KEY.`);
console.error('Trade-off: the fallback is noticeably weaker on non-English content (worst multilingual');
console.error('performer among evaluated candidates), and embedding stops working if the key is removed.');
console.error('To move to the default later:');
console.error(` ollama pull ${defaultBareModel} # after installing Ollama from https://ollama.ai`);
console.error(` gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${DEFAULT_EMBEDDING_DIMENSIONS}`);
console.error(`(same ${FALLBACK_EMBEDDING_DIMENSIONS}d column width — vectors are rebuilt, no schema change.`);
console.error(' `gbrain doctor` will remind you when Ollama becomes available.)');
console.error('');
return;
}
if (ready.length === 1) {
const r = ready[0].recipe;
const tp = r.touchpoints.embedding!;
if (Array.isArray(tp.models) && tp.models.length > 0) {
const model = tp.models[0];
const fullModel = `${r.id}:${model}`;
// When the resolved provider matches the canonical default model
// (DEFAULT_EMBEDDING_MODEL), use the gateway's
// DEFAULT_EMBEDDING_DIMENSIONS instead of the recipe's `default_dims`
// (which is the recipe's "largest sensible" tier). This keeps
// fresh-install schema width aligned with the v0.37.11.0 system
// default — for ZE that means 1280 (the Matryoshka step closest to
// legacy OpenAI 1536), not the recipe's 2560.
const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } =
await import('../core/ai/defaults.ts');
const { embeddingDimsForModel } = await import('../core/ai/model-resolver.ts');
// #2051: non-canonical models resolve per-model, not recipe-wide.
// The DEFAULT_EMBEDDING_MODEL check is kept for the day a keyed
// provider becomes the default again; today's default (ollama) never
// appears in `ready` (local-only providers are excluded above) and
// resolves in Tier 3a. The zembed-1 pin preserves the width the
// v0.36v0.42 canonical default gave ZE-key installs (1280, not the
// recipe's 2560) so new ZE brains stay consistent with the migration
// docs until the 2026-09-04 sunset.
const dims = fullModel === DEFAULT_EMBEDDING_MODEL
? DEFAULT_EMBEDDING_DIMENSIONS
: fullModel === 'zeroentropyai:zembed-1'
? 1280
: embeddingDimsForModel(r, model);
: embeddingDimsForModel(r, model);
out.embedding_model = fullModel;
out.embedding_dimensions = dims;
console.error(
@@ -1096,13 +1035,6 @@ async function initPGLite(opts: {
: (resolvedModel && resolvedDim)
? { embedding_model: resolvedModel, embedding_dimensions: resolvedDim }
: {}),
// Fallback marker: records that this install WANTED the default
// (ollama:bge-m3) but landed on the hosted fallback because Ollama
// was unavailable at init. `gbrain doctor` probes Ollama while this
// is set and prints the way back to the default.
...(opts.aiOpts?.embeddingFallback
? { embedding_default_fallback: (await import('../core/ai/defaults.ts')).DEFAULT_EMBEDDING_MODEL }
: {}),
...(opts.aiOpts?.expansion_model ? { expansion_model: opts.aiOpts.expansion_model } : {}),
...(opts.aiOpts?.chat_model ? { chat_model: opts.aiOpts.chat_model } : {}),
// v0.42 (T17): default new brains to the schema_pack selected at init
@@ -1117,13 +1049,6 @@ async function initPGLite(opts: {
// gbrain invocation). mode_prompted=true so the upgrade-time banner doesn't
// also fire on a fresh install. Hands-off: gbrain config set self_upgrade.mode auto
config.self_upgrade = { mode: 'notify', mode_prompted: true, ...(config.self_upgrade ?? {}) };
// Stale-marker hygiene: a re-init that resolves anything other than the
// hosted fallback (e.g. --embedding-model ollama:bge-m3 once Ollama is
// installed) clears the fallback marker so doctor stops re-probing.
if (!opts.aiOpts?.embeddingFallback && config.embedding_default_fallback) {
const { FALLBACK_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
if (config.embedding_model !== FALLBACK_EMBEDDING_MODEL) delete config.embedding_default_fallback;
}
saveConfig(config);
if (opts.schemaPack) {
process.stderr.write(
@@ -1360,13 +1285,6 @@ async function initPostgres(opts: {
: (resolvedModel && resolvedDim)
? { embedding_model: resolvedModel, embedding_dimensions: resolvedDim }
: {}),
// Fallback marker: records that this install WANTED the default
// (ollama:bge-m3) but landed on the hosted fallback because Ollama
// was unavailable at init. `gbrain doctor` probes Ollama while this
// is set and prints the way back to the default.
...(opts.aiOpts?.embeddingFallback
? { embedding_default_fallback: (await import('../core/ai/defaults.ts')).DEFAULT_EMBEDDING_MODEL }
: {}),
...(opts.aiOpts?.expansion_model ? { expansion_model: opts.aiOpts.expansion_model } : {}),
...(opts.aiOpts?.chat_model ? { chat_model: opts.aiOpts.chat_model } : {}),
// v0.42 (T17): same schema_pack default as PGLite path.
@@ -1379,13 +1297,6 @@ async function initPostgres(opts: {
// gbrain invocation). mode_prompted=true so the upgrade-time banner doesn't
// also fire on a fresh install. Hands-off: gbrain config set self_upgrade.mode auto
config.self_upgrade = { mode: 'notify', mode_prompted: true, ...(config.self_upgrade ?? {}) };
// Stale-marker hygiene: a re-init that resolves anything other than the
// hosted fallback (e.g. --embedding-model ollama:bge-m3 once Ollama is
// installed) clears the fallback marker so doctor stops re-probing.
if (!opts.aiOpts?.embeddingFallback && config.embedding_default_fallback) {
const { FALLBACK_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
if (config.embedding_model !== FALLBACK_EMBEDDING_MODEL) delete config.embedding_default_fallback;
}
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
if (opts.schemaPack) {
+4 -81
View File
@@ -2874,17 +2874,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
: await resolveSlugByPathOrSourcePath(engine, from, undefined);
// The new path doesn't yet have a row, so resolve from path only.
const newSlug = resolveSlugForPath(to);
// #3056: the cheap rename is OBSERVED, not assumed. A zero-row UPDATE
// doesn't throw, and a thrown collision used to be swallowed by an
// empty catch — both fell through to importFile, which created/updated
// the row at the new path while the old row stayed behind live. Both
// shapes now fall through to the reconcile below.
let renameApplied = false;
try {
renameApplied = (await engine.updateSlug(oldSlug, newSlug, renameOpts)) > 0;
await engine.updateSlug(oldSlug, newSlug, renameOpts);
} catch {
// Destination slug occupied or invalid — treat as add; the reconcile
// below removes the stale old row once the destination materialized.
// Slug doesn't exist or collision, treat as add
}
// Reimport at new path (picks up content changes). Wrapped to match the
// deletes/adds loops: a malformed renamed file is recorded to failedFiles
@@ -2897,11 +2890,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
// repo (committed symlink pointing out).
const filePath = join(gitContextRoot, to);
let importResult: Awaited<ReturnType<typeof importFile>> | undefined;
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
try {
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
importResult = result;
if (result.status === 'imported') chunksCreated += result.chunks;
else if (result.status === 'skipped' && (result as { error?: string }).error) {
failedFiles.push({ path: to, error: String((result as { error?: string }).error) });
@@ -2910,68 +2901,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
failedFiles.push({ path: to, error: e instanceof Error ? e.message : String(e) });
}
}
// #3056 reconcile: the rename fell back to add semantics, so the row
// that still represents the OLD path is the stale half of the rename
// (git reported the old path gone; a plain delete of that path would
// remove this row). Two safety rails, both from the #3252 review:
//
// 1. Delete only after the destination demonstrably materialized —
// `imported`, or an errorless `skipped` AT the new slug. Identity
// dedup can skip against the OLD row (result.slug === oldSlug),
// in which case nothing landed at newSlug and deleting the old
// row would destroy the only copy.
// 2. Locate the stale row POSITIVELY by `source_path = from`, never
// by the oldSlug guess — after a collision, a path-derived
// fallback slug could name an unrelated (e.g. manually curated)
// row. No source_path match → nothing is deleted (this also means
// code-strategy imports, which don't populate source_path, fall
// back safely to leaving the old row rather than guessing).
//
// A failed delete records a `<rename:…>` SENTINEL (not an ordinary
// path failure): the gate hard-blocks the bookmark, and — unlike a
// plain path row — the auto-skip valve can never chronic-skip it after
// N attempts, which would advance the bookmark and make a transient
// delete outage a permanent duplicate. The sentinel clears through the
// ordinary success path once the rename converges on a later run.
let reconcileFailed = false;
if (!renameApplied && importResult !== undefined) {
const destMaterialized = importResult.status === 'imported' ||
(importResult.status === 'skipped' && !importResult.error && importResult.slug === newSlug);
if (destMaterialized) {
try {
const staleMap = await engine.resolveSlugsByPaths([from], { sourceId: opts.sourceId ?? DEFAULT_SOURCE_ID });
const staleSlug = staleMap.get(from);
if (staleSlug !== undefined && staleSlug !== newSlug) {
await engine.deletePage(staleSlug, renameOpts);
deletedSlugs.add(staleSlug); // never hand a deleted slug to auto-embed
serr(` [sync] rename reconciled: removed stale row ${staleSlug} (${from} -> ${to} fell back to add).`);
} else if (staleSlug === undefined) {
serr(` [sync] rename fallback: no row has source_path ${from}; stale row (if any) left in place.`);
}
} catch (e: unknown) {
reconcileFailed = true;
failedFiles.push({
path: `<rename:${to}>`,
error: `rename reconcile failed (stale row for ${from} not removed): ` +
`${e instanceof Error ? e.message : String(e)}`,
});
}
} else {
serr(
` [sync] rename fallback: ${from} -> ${to} did not materialize at ${newSlug} ` +
`(import ${importResult.status}); old row left in place.`,
);
}
}
// Converged (cheap rename, clean reconcile, or nothing to reconcile):
// clear any `<rename:…>` sentinel a previous failing run recorded.
if (!reconcileFailed) succeededPaths.push(`<rename:${to}>`);
pagesAffected.push(newSlug);
deletedSlugs.delete(newSlug); // #1284: rename landed on a previously-deleted slug → embeddable again
// A failed reconcile must NOT checkpoint: banking `to` would make the
// resume filter skip this rename on the retry run, turning a transient
// delete failure into a permanent duplicate — the exact bug being fixed.
if (!reconcileFailed) await markCompleted(to);
await markCompleted(to);
progress.tick(1, newSlug);
}
progress.finish();
@@ -3430,10 +3362,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
if (!gate.advanced) {
const codeBreakdown = formatCodeBreakdown(failedFiles);
// Two sentinel classes block here: `<head>` (pin ancestry broken) and
// `<rename:…>` (#3056 — a rename-reconcile delete failed and advancing
// would permanently bank the duplicate). Pick the message by which fired.
if (gate.sentinelBlocked && failedFiles.some(f => f.path === '<head>')) {
if (gate.sentinelBlocked) {
serr(
`\nSync blocked: repository history changed during sync (force-push / reset).\n` +
`${codeBreakdown}\n\n` +
@@ -3441,12 +3370,6 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
`a commit that doesn't match the indexed tree. Re-run sync to re-pin against ` +
`current HEAD.`,
);
} else if (gate.sentinelBlocked) {
serr(
`\nSync blocked: a rename left a stale duplicate that could not be removed:\n` +
`${codeBreakdown}\n\n` +
`The next 'gbrain sync' retries the reconcile from the same diff.`,
);
} else {
const fileFailCount = failedFiles.filter(f => isSkippablePath(f.path)).length;
serr(
+5 -30
View File
@@ -472,27 +472,8 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
// `ze_sunset_notice_shown` (same pattern as the search-mode banner).
try {
const shown = await engine.getConfig('ze_sunset_notice_shown');
const {
DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS,
FALLBACK_EMBEDDING_MODEL,
} = await import('../core/ai/defaults.ts');
// DEFAULT_EMBEDDING_MODEL is no longer a ZE model, so the file
// plane alone would stop detecting brains created under the
// v0.36v0.42 ZE default that never wrote `embedding_model` to
// ~/.gbrain/config.json. Those brains DO carry the DB-plane row
// seeded by initSchema, so read the DB plane as the second source
// before falling back to the compiled default.
const dbModel = await engine.getConfig('embedding_model');
const effectiveModel = cfgSchema.embedding_model ?? dbModel ?? DEFAULT_EMBEDDING_MODEL;
// Current width, for the keep-your-column hosted option
// (applyEmbeddingMigration only runs a schema transition when
// col.dims !== plan.to_dims — migrating AT the current width
// rebuilds vectors only).
const dbDims = await engine.getConfig('embedding_dimensions');
const parsedDbDims = dbDims ? parseInt(dbDims, 10) : NaN;
const currentDims = cfgSchema.embedding_dimensions
?? (Number.isFinite(parsedDbDims) && parsedDbDims > 0 ? parsedDbDims : undefined)
?? 1280; // the v0.36v0.42 ZE default width
const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL;
const rerankerModel = await engine.getConfig('search.reranker.model');
const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:');
const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:');
@@ -511,15 +492,9 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
}
console.log('═══════════════════════════════════════════════════════════════');
console.log('');
console.log('Migrate before 2026-09-04 (resumable; preview cost first with --dry-run):');
console.log('');
console.log(`Option A — the default (open-weight, local, free; requires Ollama +`);
console.log(`\`ollama pull ${DEFAULT_EMBEDDING_MODEL.split(':')[1]}\`; changes column width ${currentDims}${DEFAULT_EMBEDDING_DIMENSIONS}):`);
console.log(` gbrain migrate embeddings --to ${DEFAULT_EMBEDDING_MODEL} --dim ${DEFAULT_EMBEDDING_DIMENSIONS}`);
console.log('');
console.log(`Option B — hosted (needs OPENAI_API_KEY; keeps your vector(${currentDims})`);
console.log('column and HNSW index — vectors rebuilt only; weaker on non-English content):');
console.log(` gbrain migrate embeddings --to ${FALLBACK_EMBEDDING_MODEL} --dim ${currentDims}`);
console.log('Migrate before the sunset (resumable; preview cost first):');
console.log(' gbrain migrate embeddings --to <provider:model> --dry-run');
console.log(' gbrain migrate embeddings --to <provider:model>');
console.log('');
console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /');
console.log('ollama also works and preserves your existing vectors — point');
+5 -13
View File
@@ -36,22 +36,14 @@ export const collectSetupSmells: AdvisorCollector = {
collector: 'setup-smells',
ask_user: true,
});
} else if (!cfg.embedding_model && !cfg.openai_api_key && !process.env.OPENAI_API_KEY) {
// No embedding_model configuredthe compiled default (ollama:bge-m3)
// applies at embed time, which needs a running Ollama daemon with the
// model pulled. Post-v0.37 installs always persist embedding_model at
// init, so landing here usually means setup never completed. No key
// for the hosted fallback either → flag it. (Deliberately no network
// probe here — advisor collectors stay cheap; `gbrain doctor` probes.)
} else if (!cfg.embedding_model && !cfg.zeroentropy_api_key && !process.env.ZEROENTROPY_API_KEY) {
// Default provider needs a key; none present anywhere → embeds will fail.
findings.push({
id: 'embedding_key_missing',
severity: 'warn',
title: 'No embedding provider configured — embedding may fail at write time.',
detail:
'The default (ollama:bge-m3) needs Ollama running with the model pulled ' +
'(`ollama pull bge-m3`). Alternatively set openai_api_key for the hosted ' +
'fallback, or pick a provider via embedding_model. Run `gbrain doctor` to verify.',
fix: { command_argv: ['gbrain', 'doctor'] },
title: 'No embedding provider key is set — embedding will fail at write time.',
detail: 'Set zeroentropy_api_key (or choose another provider via embedding_model).',
fix: { command_argv: ['gbrain', 'config', 'set', 'zeroentropy_api_key', '<key>'] },
collector: 'setup-smells',
ask_user: true,
});
+7 -34
View File
@@ -12,37 +12,10 @@
* install AND every doctor consistency check.
*/
// The default moved OFF ZeroEntropy (its hosted API — including
// /models/embed — shuts down 2026-09-04, which would have taken semantic
// retrieval with it on every default-config brain). See the
// Default-provider policy in CLAUDE.md: a gbrain DEFAULT must be
// open-weight or from the vendor with the longest proven model-lifetime
// record. bge-m3 is open-weight (MIT), served locally through Ollama —
// nobody can sunset it — and it is the strongest open-weight multilingual
// retriever in the 8-candidate eval that drove this choice (vector
// nDCG@10 ≥ 0.89 on every language slice tested, including the
// non-Latin-script slices where hosted small models collapse).
//
// 1024 is bge-m3's NATIVE width (see the ollama recipe's model_dims).
// Do not "round up": the Matryoshka free-truncation property measured
// for other families was NOT tested for bge-m3.
export const DEFAULT_EMBEDDING_MODEL = 'ollama:bge-m3';
export const DEFAULT_EMBEDDING_DIMENSIONS = 1024;
// Hosted fallback when Ollama is unreachable (or bge-m3 isn't pulled) at
// `gbrain init` time. text-embedding-3-small is the key most users
// already have (OPENAI_API_KEY), cheap ($0.02/MTok), and from the vendor
// with the longest hosted-embedding lifetime record — but it is the
// WEAKEST multilingual performer among the evaluated candidates
// (nDCG@10 0.645 on the Hebrew slice vs bge-m3's 0.901). That is why the
// fallback is loud, never silent: init prints the trade-off + the path
// back to the default, and `gbrain doctor` re-checks for Ollama.
//
// 1024 (not the model's native 1536, and not the legacy 1280): OpenAI
// text-embedding-3-* is Matryoshka (`isValidOpenAITextEmbedding3Dim`
// accepts any width ≤ native), so pinning the fallback at bge-m3's width
// means a later `gbrain migrate embeddings --to ollama:bge-m3 --dim 1024`
// rebuilds VECTORS only — the vector(1024) column and its HNSW index
// stay in place, no dimension transition.
export const FALLBACK_EMBEDDING_MODEL = 'openai:text-embedding-3-small';
export const FALLBACK_EMBEDDING_DIMENSIONS = 1024;
// v0.36.0 chose ZeroEntropy as the system default after evals showed
// 11/20 wins vs OpenAI (6) and Voyage (4) on real-corpus benchmarks.
// 1280 is the closest analog to legacy OpenAI 1536d while staying on
// the high-recall section of ZE's Matryoshka curve. Valid ZE Matryoshka
// steps: {2560, 1280, 640, 320, 160, 80, 40} — see ai/dims.ts.
export const DEFAULT_EMBEDDING_MODEL = 'zeroentropyai:zembed-1';
export const DEFAULT_EMBEDDING_DIMENSIONS = 1280;
-65
View File
@@ -1,65 +0,0 @@
/**
* Cheap, non-blocking Ollama availability probe for the default embedding
* model (`ollama:bge-m3`).
*
* Called exactly ONCE per `gbrain init` (the resolved choice persists into
* config.json, so embed calls never probe) and on demand by
* `gbrain doctor`'s fallback re-check. Bounded by a short timeout and
* fail-open: any error means "not available", never a thrown exception
* a probe bug must not break an install.
*
* Leaf module (no SDK imports) so init/doctor can load it without pulling
* the full gateway.
*/
export interface OllamaProbeResult {
/** Server reachable AND the model is pulled. */
ok: boolean;
/** Server responded to /api/tags at all. */
serverUp: boolean;
reason: 'ok' | 'model_missing' | 'unreachable';
}
/**
* Ollama's native API base (NOT the /v1 OpenAI-compat suffix the recipe's
* base_url_default carries). Honors OLLAMA_BASE_URL the same env var the
* gateway's openai-compat transport uses with any trailing `/v1` stripped
* so both spellings work.
*/
export function ollamaApiBase(env: NodeJS.ProcessEnv = process.env): string {
const raw = env.OLLAMA_BASE_URL?.trim() || 'http://localhost:11434';
return raw.replace(/\/+$/, '').replace(/\/v1$/, '');
}
/**
* Probe `{base}/api/tags` and check the given model is pulled. Matches
* bare names against Ollama's `name:tag` form (`bge-m3` matches
* `bge-m3:latest` and `bge-m3:567m`).
*/
/** Test seam (same pattern as gateway's __setEmbedTransportForTests). */
let probeOverride: ((model: string) => Promise<OllamaProbeResult>) | null = null;
export function __setOllamaProbeForTests(fn: typeof probeOverride): void {
probeOverride = fn;
}
export async function probeOllamaModel(
model: string,
opts: { env?: NodeJS.ProcessEnv; timeoutMs?: number } = {},
): Promise<OllamaProbeResult> {
if (probeOverride) return probeOverride(model);
const base = ollamaApiBase(opts.env ?? process.env);
try {
const res = await fetch(`${base}/api/tags`, {
signal: AbortSignal.timeout(opts.timeoutMs ?? 1500),
});
if (!res.ok) return { ok: false, serverUp: false, reason: 'unreachable' };
const body = (await res.json()) as { models?: Array<{ name?: string }> };
const names = (body.models ?? []).map(m => m.name ?? '');
const has = names.some(n => n === model || n.split(':')[0] === model);
return has
? { ok: true, serverUp: true, reason: 'ok' }
: { ok: false, serverUp: true, reason: 'model_missing' };
} catch {
return { ok: false, serverUp: false, reason: 'unreachable' };
}
}
+1 -10
View File
@@ -69,18 +69,9 @@ export interface GBrainConfig {
azure_openai_endpoint?: string;
azure_openai_deployment?: string;
azure_openai_use_entra?: string;
/** AI gateway config (v0.14+). Default: "ollama:bge-m3" / 1024 / "anthropic:claude-haiku-4-5-20251001" (see src/core/ai/defaults.ts). */
/** AI gateway config (v0.14+). v0.36+ default: "zeroentropyai:zembed-1" / 1280 / "anthropic:claude-haiku-4-5-20251001". */
embedding_model?: string;
embedding_dimensions?: number;
/**
* Set by `gbrain init` when the declared default embedder (ollama:bge-m3)
* was unavailable and init landed on the hosted fallback instead. Holds
* the default model that was skipped. While set AND embedding_model still
* equals the fallback, `gbrain doctor` probes Ollama and prints the
* migrate command back to the default. Cleared by a re-init that resolves
* anything other than the fallback.
*/
embedding_default_fallback?: string;
/**
* v0.37 (D9): user opted into deferred-setup mode at init time via
* `gbrain init --no-embedding`. When true, embed callsites and `gbrain
+1 -10
View File
@@ -895,17 +895,8 @@ export async function resolveSourceForDir(
// (the cycleSourceId precedence) or 'default'.
if (brainDir === null) return undefined;
try {
// #2540: exclude archived rows (dream's --source guard refuses to stamp
// them, so an archived alias winning here means the stamp silently never
// lands and doctor's cycle_freshness stays red on a healthy install) and
// order deterministically so a duplicate registration of the same path
// can't shadow the active source on whichever row the engine scans first.
// Ordering matches listAllSources/sources-ops for operator-output parity.
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources
WHERE local_path = $1 AND archived = false
ORDER BY (id = 'default') DESC, id
LIMIT 1`,
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
[brainDir],
);
if (rows[0]) return rows[0].id;
-4
View File
@@ -26,10 +26,6 @@ export interface EmbeddingPricing {
* gateway model strings (e.g. 'openai:text-embedding-3-large').
*/
export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
// The system default (src/core/ai/defaults.ts): local Ollama, no API cost.
// Listed so migrate/upgrade cost previews for the default show $0 rather
// than "estimate unavailable".
'ollama:bge-m3': { pricePerMTok: 0 },
// OpenAI (https://openai.com/api/pricing/, verified 2026-05-11)
'openai:text-embedding-3-large': { pricePerMTok: 0.13 },
'openai:text-embedding-3-small': { pricePerMTok: 0.02 },
+1 -6
View File
@@ -1951,13 +1951,8 @@ export interface BrainEngine {
* preserved via stable page_id). `opts.sourceId` scopes the UPDATE without
* it, the bare `WHERE slug = old` matches every row across every source and
* would either rename them all OR violate the (source_id, slug) UNIQUE.
*
* Returns the number of rows moved. 0 means the old slug had no row in the
* scoped source an UPDATE that matches nothing does NOT throw, so callers
* that need to know whether the rename actually happened (the sync rename
* path, #3056) must check the return value rather than rely on the catch.
*/
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number>;
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void>;
rewriteLinks(oldSlug: string, newSlug: string): Promise<void>;
/**
+5 -13
View File
@@ -5332,16 +5332,12 @@ export class PGLiteEngine implements BrainEngine {
// pages_with_timeline) and v0.10.3 graph layer (link_coverage, timeline_coverage,
// most_connected). Both coexist: master's brain_score is the composite
// dashboard, v0.10.3 metrics give entity-page-level granularity.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const { rows: [h] } = await this.db.query(`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
)
SELECT
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5366,7 +5362,7 @@ export class PGLiteEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
WHERE p.type IN ('entity', 'person', 'company')
ORDER BY link_count DESC
LIMIT 5
`);
@@ -5385,7 +5381,6 @@ export class PGLiteEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`);
const r = h as Record<string, unknown>;
@@ -5480,18 +5475,15 @@ export class PGLiteEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
newSlug = validateSlug(newSlug);
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (mirrors postgres-engine.ts).
const result = await this.db.query(
await this.db.query(
`UPDATE pages SET slug = $1, updated_at = now() WHERE slug = $2 AND source_id = $3`,
[newSlug, oldSlug, sourceId]
);
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.affectedRows ?? 0;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
+5 -13
View File
@@ -5432,16 +5432,12 @@ export class PostgresEngine implements BrainEngine {
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
)
SELECT
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5463,7 +5459,7 @@ export class PostgresEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
WHERE p.type IN ('entity', 'person', 'company')
ORDER BY link_count DESC
LIMIT 5
`;
@@ -5482,7 +5478,6 @@ export class PostgresEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`;
const pageCount = Number(h.page_count);
@@ -5574,17 +5569,14 @@ export class PostgresEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
newSlug = validateSlug(newSlug);
const sql = this.sql;
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (which would either rename them all OR fail the
// (source_id, slug) UNIQUE if the new slug already exists in another source).
const result = await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.count ?? 0;
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
+3 -29
View File
@@ -48,32 +48,6 @@ import {
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
/**
* Which detail levels get the compiled_truth boost (#3430).
*
* ONLY `low`. The documented contract (`src/core/operations.ts`) is
* "low (compiled truth only), medium (default, all with dedup), high (all
* chunks)" so `low` is the level that privileges compiled truth, and both
* `medium` and `high` are supposed to see everything on equal footing.
*
* This was previously spelled `detail !== 'high'`, i.e. written as though
* `high` were the special case. Because COMPILED_TRUTH_BOOST is applied AFTER
* RRF normalization, and RRF's whole range over a 100-deep pool is 1/60 1/160,
* a 2.0x multiplier is not a tilt break-even is `2/(60+r) >= 1/60`, so any
* boosted chunk inside the first 60 ranks outranks an unboosted rank-1 chunk.
* At the default detail that made search categorically compiled-truth-only:
* a page whose answer lived in a `fenced_code` chunk returned the prose chunk,
* and the code chunk fell out of the window entirely.
*
* Extracted as a named predicate rather than left inline at three call sites so
* the detailboost mapping is directly testable. An inline expression can only
* be covered through a full `hybridSearch` round trip, which is why the
* original inversion went unnoticed.
*/
export function shouldBoostCompiledTruth(detail: string | null | undefined): boolean {
return detail === 'low';
}
const pendingCacheWrites = new Set<Promise<unknown>>();
/**
@@ -1195,7 +1169,7 @@ export async function hybridSearch(
const noEmbedLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
noEmbedResults = rrfFusionWeighted(noEmbedLists, shouldBoostCompiledTruth(detailResolved));
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
}
if (noEmbedResults.length > 0) {
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
@@ -1439,7 +1413,7 @@ export async function hybridSearch(
const fallbackLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
fallbackResults = rrfFusionWeighted(fallbackLists, shouldBoostCompiledTruth(detail));
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
}
if (fallbackResults.length > 0) {
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
@@ -1526,7 +1500,7 @@ export async function hybridSearch(
// arms BEFORE fusion so the compiled-truth authority boost skips them.
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
let fused = rrfFusionWeighted(allLists, shouldBoostCompiledTruth(detail));
let fused = rrfFusionWeighted(allLists, detail !== 'high');
// Cosine re-scoring before dedup so semantically better chunks survive.
// v0.36 (D9): hydrate from the active embedding column so rescore happens
+1 -1
View File
@@ -766,7 +766,7 @@ export function attributeKnob<K extends keyof ModeBundle>(
// written between the #3391 stale-fix (which changes which chunks count as
// current) and the operator's migration run. Same one-time global cold-miss
// pattern as the bumps above.
export const KNOBS_HASH_VERSION = 14;
export const KNOBS_HASH_VERSION = 13;
/**
* v0.36 (D8 / CDX-2) second-arg context for the cache key. The
+6 -8
View File
@@ -41,15 +41,13 @@ describe('gateway configuration', () => {
expect(getExpansionModel()).toBe('anthropic:claude-haiku-4-5-20251001');
});
test('defaults are ollama:bge-m3 1024d (open-weight default, ZE sunset)', () => {
// v0.36.0.0 flipped the default to zeroentropyai:zembed-1 @ 1280d. With
// ZE's hosted API sunsetting 2026-09-04, the default moved to the
// open-weight ollama:bge-m3 at its native 1024d — a model nobody can
// sunset. Rationale + hosted-fallback policy in src/core/ai/defaults.ts
// and CLAUDE.md's Default-provider policy.
test('defaults are ZE 1280d as of v0.36.0.0 (D3)', () => {
// The default flipped from openai:text-embedding-3-large 1536d to
// zeroentropyai:zembed-1 1280d in v0.36.0.0. The cost story is in
// CHANGELOG.md; the rationale lives in src/core/ai/gateway.ts:45-54.
configureGateway({ env: {} });
expect(getEmbeddingModel()).toBe('ollama:bge-m3');
expect(getEmbeddingDimensions()).toBe(1024);
expect(getEmbeddingModel()).toBe('zeroentropyai:zembed-1');
expect(getEmbeddingDimensions()).toBe(1280);
expect(getExpansionModel()).toBe('anthropic:claude-haiku-4-5-20251001');
});
});
+4 -4
View File
@@ -3,15 +3,15 @@ import { getPGLiteSchema, PGLITE_SCHEMA_SQL } from '../../src/core/pglite-schema
import { getPostgresSchema } from '../../src/core/postgres-engine.ts';
describe('getPGLiteSchema', () => {
test('default produces gateway-default schema (1024d + ollama:bge-m3)', () => {
test('default produces gateway-default schema (v0.37+: 1280d + zeroentropyai:zembed-1)', () => {
// v0.37 fix wave Lane A.1 + CDX2-1: defaults now track the canonical
// gateway constants in `ai/defaults.ts` instead of the stale v0.13
// OpenAI literals (1536 / text-embedding-3-large). Fixes the
// headline bug where bare `gbrain init --pglite` produced a 1536
// schema while the default model emitted a different width.
// schema while the ZE default model emitted 1280-dim vectors.
const sql = getPGLiteSchema();
expect(sql).toMatch(/vector\(1024\)/);
expect(sql).toMatch(/'ollama:bge-m3'/);
expect(sql).toMatch(/vector\(1280\)/);
expect(sql).toMatch(/'zeroentropyai:zembed-1'/);
expect(sql).not.toMatch(/__EMBEDDING_DIMS__/);
expect(sql).not.toMatch(/__EMBEDDING_MODEL__/);
});
+2 -3
View File
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
return resolveSearchMode({ mode: 'balanced' });
}
test('KNOBS_HASH_VERSION is 14 (cross-modal still appended; 13→14 compiled_truth boost scope #3430)', () => {
test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => {
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
@@ -146,8 +146,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
// finally reaches asymmetric providers — pre-fix rows were keyed on
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
// #3430: 13→14 compiled_truth boost no longer applies at detail=medium.
expect(KNOBS_HASH_VERSION).toBe(14);
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('flipping unified_multimodal changes the hash', () => {
+8 -14
View File
@@ -38,7 +38,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv, emptyHome } from './helpers/with-env.ts';
import { runCycle, ALL_PHASES } from '../src/core/cycle.ts';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { mkdtempSync, writeFileSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
@@ -139,25 +139,19 @@ describe('#2540 (i) — pack omitting optional phases, all enabled phases comple
describe('#2540 (ii) — an enabled phase that never completes still prevents the stamp', () => {
test('every selected phase failing reports status=failed and does NOT stamp last_full_cycle_at', async () => {
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
await withEnv({ GBRAIN_HOME: gbrainHome, OPENAI_API_KEY: undefined, ANTHROPIC_API_KEY: undefined }, async () => {
await seedSource('always-fails');
expect(await readLastFullCycleAt('always-fails')).toBeNull();
// Deterministic, environment-independent failure: run the sync phase
// against a brain directory that no longer exists. The previous shape
// ('embed' with OPENAI_API_KEY/ANTHROPIC_API_KEY unset) was
// environment-sensitive — on a machine where any OTHER embedding
// provider resolves (Voyage, ZeroEntropy, a local endpoint, …), embed
// with zero stale chunks succeeds and the cycle reports 'clean',
// flipping this test's expectation. A vanished checkout fails the
// sync phase on every machine. This is NOT the fix under test; it's
// the pre-existing "an enabled phase genuinely never completes" case
// the issue says must keep failing doctor's check.
rmSync(brainDir, { recursive: true, force: true });
// embed is a real, always-enabled phase (no pack gate, no config
// .enabled toggle). With no embedding provider key configured it
// deterministically fails — this is NOT the fix under test, it's
// the pre-existing "an enabled phase genuinely never completes"
// case the issue says must keep failing doctor's check.
const report = await runCycle(engine, {
brainDir,
sourceId: 'always-fails',
phases: ['sync'],
phases: ['embed'],
});
expect(report.status).toBe('failed');
+138
View File
@@ -0,0 +1,138 @@
/**
* #3502: docs must not reference nonexistent gbrain commands.
*
* `docs/tutorials/personal-brain.md` shipped a `gbrain install` step for two
* months after the command it replaced was retired every reader hit
* "Unknown command: install". This guard scans README.md, docs/, and skills/
* for `gbrain <verb>` invocations in code (fenced blocks + inline code spans)
* and checks each verb against the live CLI surface: CLI_ONLY, operation
* cliHints names (non-hidden), and aliases.
*
* Deliberately excluded (historical or speculative by design, per CLAUDE.md's
* "historical docs are never rewritten" rule):
* - docs/GBRAIN_V0.md the v0 spec; documents v0's CLI
* - docs/designs/, docs/plans/ future/speculative design docs
* - docs/migrations/, skills/migrations/ per-release migration notes,
* written against that release's CLI
* - docs/UPGRADING_DOWNSTREAM_AGENTS.md per-release upgrade chronicle
*
* Heuristics keep prose out: only fenced code + inline spans are scanned,
* comment lines and diagram lines are skipped, and the verb must sit in
* command position (start of command text, or after a shell operator).
*/
import { describe, expect, test } from 'bun:test';
import { readdirSync, readFileSync, statSync } from 'fs';
import { dirname, join, relative } from 'path';
import { CLI_ONLY, cliAliases } from '../src/cli.ts';
import { operations } from '../src/core/operations.ts';
const ROOT = dirname(import.meta.dir);
const EXCLUDED = [
'docs/GBRAIN_V0.md',
'docs/UPGRADING_DOWNSTREAM_AGENTS.md',
'docs/designs/',
'docs/plans/',
'docs/migrations/',
'skills/migrations/',
];
/** Known-intentional references to commands that deliberately don't exist. */
const ALLOWLIST: Record<string, string[]> = {
// The doc explains that gbrain does NOT ship this command, on purpose.
'docs/guides/rls-and-you.md': ['rls-exempt'],
};
function validCommands(): Set<string> {
const valid = new Set<string>(CLI_ONLY);
for (const op of operations) {
const name = op.cliHints?.name;
if (name && !op.cliHints?.hidden) valid.add(name);
}
for (const alias of cliAliases.keys()) valid.add(alias);
return valid;
}
function* mdFiles(dir: string): Generator<string> {
for (const entry of readdirSync(dir)) {
const p = join(dir, entry);
if (statSync(p).isDirectory()) yield* mdFiles(p);
else if (p.endsWith('.md')) yield p;
}
}
interface CodeLine { code: string; line: number }
/** Fenced-block lines + inline code spans that START with `gbrain `. */
function codeRegions(text: string): CodeLine[] {
const out: CodeLine[] = [];
const lines = text.split('\n');
let inFence = false;
for (let i = 0; i < lines.length; i++) {
const l = lines[i];
if (/^\s*(```|~~~)/.test(l)) { inFence = !inFence; continue; }
if (inFence) {
const t = l.trim();
if (/^(#|\/\/|--|\*)/.test(t)) continue; // comment lines
if (/[│┌┐└┘├┤─═╔╗╚╝]/.test(l)) continue; // ASCII-art diagrams
out.push({ code: l, line: i + 1 });
continue;
}
for (const m of l.matchAll(/`(gbrain [^`]+)`/g)) out.push({ code: m[1], line: i + 1 });
}
return out;
}
/** True when `gbrain` sits at command position (not mid-prose). */
function commandPosition(prefix: string): boolean {
const p = prefix.trimEnd();
return p === '' || /[|;&`(={[]$/.test(p) || /\$$/.test(p);
}
function scan(): string[] {
const valid = validCommands();
const violations: string[] = [];
const files = [
join(ROOT, 'README.md'),
...mdFiles(join(ROOT, 'docs')),
...mdFiles(join(ROOT, 'skills')),
];
for (const file of files) {
const rel = relative(ROOT, file);
if (EXCLUDED.some((e) => rel === e || rel.startsWith(e))) continue;
const text = readFileSync(file, 'utf-8');
for (const { code, line } of codeRegions(text)) {
for (const m of code.matchAll(/\bgbrain\s+([A-Za-z][\w-]*)/g)) {
const verb = m[1];
if (!/^[a-z][a-z0-9_-]{2,}$/.test(verb)) continue; // flags, <slots>, v0.x
if (!commandPosition(code.slice(0, m.index))) continue;
if (valid.has(verb)) continue;
if (ALLOWLIST[rel]?.includes(verb)) continue;
violations.push(`${rel}:${line}: \`gbrain ${verb}\` is not a real command — ${code.trim().slice(0, 90)}`);
}
}
}
return violations;
}
describe('#3502 — docs reference only real gbrain commands', () => {
test('every `gbrain <verb>` in README/docs/skills resolves to a live command', () => {
const violations = scan();
expect(violations).toEqual([]);
});
test('the sanity anchors: install is dead, init/put/skillpack are live', () => {
const valid = validCommands();
expect(valid.has('install')).toBe(false); // retired v0.36.0.0 — the #3502 bug
expect(valid.has('init')).toBe(true);
expect(valid.has('put')).toBe(true);
expect(valid.has('skillpack')).toBe(true);
});
test('pages + bench are dispatchable (documented surfaces; #2035 bug class)', () => {
// `pages` had a live handleCliOnly case but was dropped from CLI_ONLY;
// `bench` (bench-publish.ts) was documented but never wired at all.
expect(CLI_ONLY.has('pages')).toBe(true);
expect(CLI_ONLY.has('bench')).toBe(true);
});
});
+2 -28
View File
@@ -79,38 +79,12 @@ describe('doctor checkCycleFreshness', () => {
expect(result.message).toMatch(/gbrain dream --source/);
});
test('source with NO last_full_cycle_at (never cycled) returns warn, not fail (#2540)', async () => {
// #2540: never-cycled used to FAIL, which turned doctor permanently red
// on any install that doesn't cycle every local_path source (e.g. one
// nightly `dream --dir <vault>` plus other federated sources) — and on
// any source added minutes ago. It surfaces as a warning; only a source
// that HAS cycled and then went stale escalates to fail.
test('source with NO last_full_cycle_at (never cycled) returns fail', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('virgin');
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('warn');
expect(result.message).toMatch(/never completed a full cycle/);
expect(result.message).toMatch(/gbrain dream --source/);
});
test('reporter case (#2540): one cycled vault + never-cycled siblings is warn, not permanent fail', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('nightly-vault', agoH(2)); // the one vault dreamt via --dir
await seed('federated-a'); // never cycled
await seed('federated-b'); // never cycled
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('warn');
expect(result.message).toMatch(/federated-a/);
expect(result.message).toMatch(/federated-b/);
expect(result.message).not.toMatch(/nightly-vault/);
});
test('a previously-cycled source gone stale still fails even next to never-cycled sources', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('stale', agoH(72)); // real regression signal
await seed('virgin'); // never cycled — warn-only
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('fail');
expect(result.message).toMatch(/never completed a full cycle/);
});
test('mixed sources: highest severity wins (fail > warn > ok)', async () => {
-22
View File
@@ -96,28 +96,6 @@ describe('gbrain dream --dir <path> freshness stamp (#1869)', () => {
expect(await readLastFullCycleAt('mothballed')).toBeNull();
});
}, 60_000);
test('an ARCHIVED alias of the same path does not shadow the active source (#2540)', async () => {
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
// Ordinary shape: a source was archived and re-added under a new id
// pointing at the same checkout. Seed the archived twin FIRST so a
// filterless `LIMIT 1` scan finds it first.
await seedSource('retired-twin', true);
await seedSource('active-twin', false);
const report = await runDream(engine, ['--dir', brainDir, '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
// Pre-fix, resolveSourceForDir's exact match had no `archived = false`
// filter and no ORDER BY, so the archived twin won the lookup; dream's
// archived guard then (correctly) refused to stamp it — and the ACTIVE
// source silently never got its stamp, leaving doctor's cycle_freshness
// permanently stale on a healthy install.
expect(await readLastFullCycleAt('active-twin')).not.toBeNull();
expect(await readLastFullCycleAt('retired-twin')).toBeNull();
});
}, 60_000);
});
/**
+16 -24
View File
@@ -26,44 +26,36 @@ import {
describe('E2E: fresh gbrain init --pglite → import → embed works end-to-end', () => {
let tmpHome: string;
let origHome: string | undefined;
let origZeKey: string | undefined;
let origOpenaiKey: string | undefined;
let origOllamaUrl: string | undefined;
let fakeOllama: ReturnType<typeof Bun.serve> | null = null;
let origVoyageKey: string | undefined;
beforeEach(() => {
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-e2e-fresh-'));
origHome = process.env.GBRAIN_HOME;
// The default embedder is ollama:bge-m3, resolved via a one-shot
// /api/tags probe at init. Serve a fake Ollama daemon so the bare-init
// happy path is hermetic and deterministic regardless of whether the
// dev machine runs a real Ollama (or has hosted provider keys set —
// the probe wins before env-key detection runs, so no key clearing
// is needed beyond OPENAI_API_KEY hygiene for the embed-check path).
fakeOllama = Bun.serve({
port: 0,
fetch(req) {
if (new URL(req.url).pathname === '/api/tags') {
return Response.json({ models: [{ name: 'bge-m3:latest' }] });
}
return new Response('not found', { status: 404 });
},
});
origOllamaUrl = process.env.OLLAMA_BASE_URL;
process.env.OLLAMA_BASE_URL = `http://127.0.0.1:${fakeOllama.port}`;
origZeKey = process.env.ZEROENTROPY_API_KEY;
// Save + clear OPENAI_API_KEY + VOYAGE_API_KEY so init only sees
// one provider as env-ready (ZE). Without this, dev machines with
// multi-provider env (Garry's setup) fail init's disambiguation gate
// ("Multiple embedding providers env-ready: openai, voyage,
// zeroentropyai") before the test body runs.
origOpenaiKey = process.env.OPENAI_API_KEY;
origVoyageKey = process.env.VOYAGE_API_KEY;
delete process.env.OPENAI_API_KEY;
delete process.env.VOYAGE_API_KEY;
process.env.GBRAIN_HOME = tmpHome;
// Stub key so init's setup-hint check passes.
process.env.ZEROENTROPY_API_KEY = 'sk-test-ze';
});
afterEach(() => {
fakeOllama?.stop(true);
fakeOllama = null;
rmSync(tmpHome, { recursive: true, force: true });
if (origHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = origHome;
if (origOllamaUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = origOllamaUrl;
if (origZeKey === undefined) delete process.env.ZEROENTROPY_API_KEY;
else process.env.ZEROENTROPY_API_KEY = origZeKey;
if (origOpenaiKey !== undefined) process.env.OPENAI_API_KEY = origOpenaiKey;
if (origVoyageKey !== undefined) process.env.VOYAGE_API_KEY = origVoyageKey;
__setEmbedTransportForTests(null);
// Restore legacy-preload gateway state.
configureGateway({
@@ -73,7 +65,7 @@ describe('E2E: fresh gbrain init --pglite → import → embed works end-to-end'
});
});
test('bare `init --pglite`: schema sized to gateway defaults (ollama:bge-m3/1024)', async () => {
test('bare `init --pglite`: schema sized to gateway defaults (ZE/1280)', async () => {
// Reset gateway so init.ts has to resolve defaults from
// ai/defaults.ts. This is the actual production code path for a
// fresh install: bare `gbrain init --pglite` with no env or file
+13 -20
View File
@@ -58,11 +58,6 @@ function makeTempHome(): string {
return mkdtempSync(join(tmpdir(), 'gbrain-e2e-init-'));
}
// Pin the Ollama probe at a dead port so env-detection tests are
// deterministic on machines that run a real Ollama daemon (the default
// embedder ollama:bge-m3 would otherwise win over every env key).
const DEAD_OLLAMA = { OLLAMA_BASE_URL: 'http://127.0.0.1:9' };
// ============================================================================
describe('v0.37 T12 — fresh init env-detection (D1, D2, D3) + persistence (D5)', () => {
@@ -71,25 +66,23 @@ describe('v0.37 T12 — fresh init env-detection (D1, D2, D3) + persistence (D5)
beforeAll(() => { tmpHome = makeTempHome(); });
afterAll(() => { rmSync(tmpHome, { recursive: true, force: true }); });
test('OPENAI_API_KEY with Ollama absent lands on the hosted fallback, loudly', async () => {
test('OPENAI_API_KEY auto-picks OpenAI, persists embedding_model + embedding_dimensions', async () => {
const r = await runCli(['init', '--pglite'], {
gbrainHome: tmpHome,
env: { OPENAI_API_KEY: 'sk-test-only-for-init-resolution-NOT-CALLED', GBRAIN_INIT_SKIP_EMBED_CHECK: '1', ...DEAD_OLLAMA },
env: { OPENAI_API_KEY: 'sk-test-only-for-init-resolution-NOT-CALLED' },
});
// The declared default is ollama:bge-m3; with Ollama unreachable and an
// OpenAI key present, init lands on the designated hosted fallback —
// and says so (never a silent downgrade).
expect(r.stderr).toMatch(/default embedder is ollama:bge-m3/);
expect(r.stderr).toMatch(/Falling back to the hosted openai:text-embedding-3-small/);
// Init may or may not succeed (depends on whether OpenAI key is real for
// any side effect — but init.ts has no live embed call, just config
// writes + schema). Assert the auto-pick stderr notice fired.
expect(r.stderr).toMatch(/Detected OPENAI_API_KEY|Using openai:text-embedding-3-large/);
expect(r.exitCode).toBe(0);
// Config persisted with the fallback + the doctor-recheck marker.
// Config persisted with the right embedding fields.
const cfgPath = join(tmpHome, '.gbrain', 'config.json');
expect(existsSync(cfgPath)).toBe(true);
const cfg = JSON.parse(readFileSync(cfgPath, 'utf-8'));
expect(cfg.embedding_model).toBe('openai:text-embedding-3-small');
expect(cfg.embedding_dimensions).toBe(1024);
expect(cfg.embedding_default_fallback).toBe('ollama:bge-m3');
expect(cfg.embedding_model).toBe('openai:text-embedding-3-large');
expect(cfg.embedding_dimensions).toBe(1536);
expect(cfg.engine).toBe('pglite');
}, 240000);
});
@@ -105,13 +98,13 @@ describe('v0.37 T12 — D3 non-TTY no-key fail-loud', () => {
test('--non-interactive with zero provider keys → exit 1 + paste-ready hint', async () => {
const r = await runCli(['init', '--pglite', '--non-interactive'], {
gbrainHome: tmpHome,
env: { ...DEAD_OLLAMA }, // no provider keys, no Ollama
env: {}, // no provider keys
});
expect(r.exitCode).toBe(1);
// Fail-loud message leads with the default and lists hosted keys.
// Fail-loud message includes the canonical env var list.
expect(r.stderr).toContain('No embedding provider configured');
expect(r.stderr).toContain('ollama pull bge-m3');
expect(r.stderr).toContain('OPENAI_API_KEY');
expect(r.stderr).toContain('ZEROENTROPY_API_KEY');
expect(r.stderr).toContain('VOYAGE_API_KEY');
// Suggests --no-embedding alternative.
expect(r.stderr).toContain('--no-embedding');
@@ -120,7 +113,7 @@ describe('v0.37 T12 — D3 non-TTY no-key fail-loud', () => {
test('--non-interactive with env-key typo surfaces Levenshtein hint', async () => {
const r = await runCli(['init', '--pglite', '--non-interactive'], {
gbrainHome: tmpHome,
env: { OPENAPI_API_KEY: 'sk-test-typo', ...DEAD_OLLAMA },
env: { OPENAPI_API_KEY: 'sk-test-typo' },
});
expect(r.exitCode).toBe(1);
// D13 typo detection: surfaces "did you mean OPENAI_API_KEY"
+2 -6
View File
@@ -26,12 +26,8 @@ if (skip) {
}
describeE2E('E2E: JSONB roundtrip — v0.12.1 reliability wave', () => {
// 60s hook budget: setupDB runs connect + the full migration chain, which
// exceeds bun's default 5s hook timeout on loaded CI runners. Hooks do NOT
// inherit a test's third-arg timeout (verified on bun 1.3.14) — they need
// their own second-arg budget. Same pattern as op-checkpoint-jsonb-parity.
beforeAll(async () => { await setupDB(); }, 60_000);
afterAll(async () => { await teardownDB(); }, 60_000);
beforeAll(async () => { await setupDB(); });
afterAll(async () => { await teardownDB(); });
test('putPage writes frontmatter as object, not double-encoded string', async () => {
const engine = getEngine();
-159
View File
@@ -1,159 +0,0 @@
/**
* Pins the Memvelope envelope importer contract: deterministic markdown output,
* provenance frontmatter, citation-bearing bodies, and loud collision handling.
*/
import { afterAll, describe, expect, test } from 'bun:test';
import { mkdtempSync, rmSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const SCRIPT_PATH = join(import.meta.dir, '..', 'scripts', 'envelope-to-gbrain.mjs');
const FIXTURE_PATH = join(import.meta.dir, 'fixtures', 'memvelope', 'sample.mve.json');
const TEMP_DIRS: string[] = [];
afterAll(() => {
for (const dir of TEMP_DIRS) {
rmSync(dir, { recursive: true, force: true });
}
});
function tempDir(): string {
const dir = mkdtempSync(join(tmpdir(), 'envelope-to-gbrain-'));
TEMP_DIRS.push(dir);
return dir;
}
async function runImporter(envelopePath: string, outDir = tempDir()) {
// The script is plain Node-compatible ESM; Bun can execute it directly in CI
// without requiring a separate node toolchain.
const proc = Bun.spawn([process.execPath, SCRIPT_PATH, envelopePath, outDir], {
stdout: 'pipe',
stderr: 'pipe',
});
await proc.exited;
const stdout = await new Response(proc.stdout).text();
const stderr = await new Response(proc.stderr).text();
return { exitCode: proc.exitCode, stdout, stderr, outDir };
}
function markdownFiles(dir: string): string[] {
return readdirSync(dir).filter((name) => name.endsWith('.md')).sort();
}
function readOnlyMarkdown(dir: string): string {
const files = markdownFiles(dir);
expect(files).toHaveLength(1);
return readFileSync(join(dir, files[0]), 'utf8');
}
describe('envelope-to-gbrain importer', () => {
test('sample envelope writes exactly one markdown page and reports count', async () => {
const result = await runImporter(FIXTURE_PATH);
expect(result.exitCode).toBe(0);
expect(markdownFiles(result.outDir)).toHaveLength(1);
expect(result.stdout).toContain('wrote 1 markdown page(s)');
});
test('filename is keyed by conversation id with date prefix', async () => {
const result = await runImporter(FIXTURE_PATH);
expect(result.exitCode).toBe(0);
expect(markdownFiles(result.outDir)).toEqual(['2025-11-02-c-3f9a2b.md']);
});
test('frontmatter carries conversation provenance fields', async () => {
const result = await runImporter(FIXTURE_PATH);
const page = readOnlyMarkdown(result.outDir);
expect(result.exitCode).toBe(0);
expect(page).toContain('type: conversation');
expect(page).toContain('title: "Onboarding Checklist Draft"');
expect(page).toContain('date: 2025-11-02');
expect(page).toContain('source: chatgpt');
expect(page).toContain('memvelope_conversation_id: "c-3f9a2b"');
expect(page).toContain('origin: memvelope/envelope-v0');
});
test('body carries role labels and message-id citations', async () => {
const result = await runImporter(FIXTURE_PATH);
const page = readOnlyMarkdown(result.outDir);
expect(result.exitCode).toBe(0);
expect(page).toContain('· m1');
expect(page).toContain('· m4');
expect(page).toContain('**Me**');
expect(page).toContain('**Assistant**');
});
test('output is deterministic across repeated runs', async () => {
const first = await runImporter(FIXTURE_PATH);
const second = await runImporter(FIXTURE_PATH);
expect(first.exitCode).toBe(0);
expect(second.exitCode).toBe(0);
expect(readOnlyMarkdown(first.outDir)).toBe(readOnlyMarkdown(second.outDir));
});
test('duplicate conversation ids warn and report distinct files written', async () => {
const inputDir = tempDir();
const envelopePath = join(inputDir, 'duplicate.mve.json');
writeFileSync(envelopePath, JSON.stringify({
memvelope: 'envelope-v0',
meta: { source_provider: 'chatgpt' },
conversations: [
{
id: 'c-repeat',
title: 'First repeated id',
created_at: '2025-11-02T14:22:51.000Z',
messages: [{ id: 'm1', role: 'user', ts: '2025-11-02T14:22:51.000Z', text: 'alice-example noted the first checklist draft.' }],
},
{
id: 'c-repeat',
title: 'Second repeated id',
created_at: '2025-11-02T15:22:51.000Z',
messages: [{ id: 'm2', role: 'assistant', ts: '2025-11-02T15:22:51.000Z', text: 'Assistant noted the repeated id collision.' }],
},
],
}));
const result = await runImporter(envelopePath);
expect(result.exitCode).toBe(0);
expect(result.stderr).toContain('warning: filename collision on "2025-11-02-c-repeat.md"');
expect(result.stdout).toContain('wrote 1 markdown page(s)');
expect(markdownFiles(result.outDir)).toHaveLength(1);
});
test('missing or foreign format is rejected', async () => {
const inputDir = tempDir();
const envelopePath = join(inputDir, 'not-envelope.json');
writeFileSync(envelopePath, JSON.stringify({ conversations: [] }));
const result = await runImporter(envelopePath);
expect(result.exitCode).toBe(1);
expect(result.stderr).toContain('envelope-v0');
});
test('missing conversation id uses positional fallback filename', async () => {
const inputDir = tempDir();
const envelopePath = join(inputDir, 'missing-id.mve.json');
writeFileSync(envelopePath, JSON.stringify({
memvelope: 'envelope-v0',
meta: { source_provider: 'chatgpt' },
conversations: [
{
title: 'Missing id example',
created_at: '2025-11-02T14:22:51.000Z',
messages: [{ id: 'm1', role: 'user', ts: '2025-11-02T14:22:51.000Z', text: 'alice-example asked for a fallback filename.' }],
},
],
}));
const result = await runImporter(envelopePath);
expect(result.exitCode).toBe(0);
expect(markdownFiles(result.outDir)).toEqual(['2025-11-02-conv-1.md']);
});
});
-42
View File
@@ -1,42 +0,0 @@
{
"memvelope": "envelope-v0",
"meta": {
"source_provider": "chatgpt",
"conversation_count": 1,
"message_count": 4
},
"conversations": [
{
"id": "c-3f9a2b",
"title": "Onboarding Checklist Draft",
"created_at": "2025-11-02T14:22:51.000Z",
"updated_at": "2025-11-02T14:31:12.000Z",
"messages": [
{
"id": "m1",
"role": "user",
"ts": "2025-11-02T14:22:51.000Z",
"text": "alice-example is drafting acme-example's widget-co onboarding checklist and wants a concise first pass."
},
{
"id": "m2",
"role": "assistant",
"ts": "2025-11-02T14:24:03.000Z",
"text": "Start with account setup, workspace access, sample widget review, and a first-week check-in with the acme-example owner."
},
{
"id": "m3",
"role": "user",
"ts": "2025-11-02T14:28:19.000Z",
"text": "Add a note that bob-example should compare fund-a and fund-b reporting needs before the kickoff."
},
{
"id": "m4",
"role": "assistant",
"ts": "2025-11-02T14:31:12.000Z",
"text": "Include a pre-kickoff step for bob-example to list fund-a and fund-b reporting questions, then confirm owners with charlie-example."
}
]
}
]
}
-123
View File
@@ -1,123 +0,0 @@
/**
* #1305 getHealth() must exclude soft-deleted pages from every
* page-scoped count, the same posture getStats() has had since v0.26.5.
*
* Pre-fix, getHealth counted raw `pages` rows: page_count and orphan_pages
* included soft-deleted pages, the entity_pages CTE kept deleted entities in
* the link/timeline coverage denominators and in most_connected, and
* brain_score therefore never moved when a user soft-deleted pages.
*
* Boundary (deliberate): chunk- and link-scoped counts (embed_coverage,
* missing_embeddings, link_count, dead_links) stay RAW they occupy storage
* until the autopilot purge phase, matching getStats. Destructive-removal
* counts (purge paths, #2235) also deliberately count all rows and are
* untouched here.
*
* Runs against PGLite the fixed SQL shapes are identical in both engines.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
for (const t of ['links', 'content_chunks', 'timeline_entries', 'tags', 'page_versions', 'pages']) {
await (engine as any).db.exec(`DELETE FROM ${t}`);
}
});
async function seedNote(slug: string): Promise<void> {
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `content of ${slug}`, frontmatter: {} });
}
async function pageId(slug: string): Promise<number> {
return (await (engine as any).db.query(`SELECT id FROM pages WHERE slug=$1`, [slug])).rows[0].id;
}
describe('#1305 — getHealth excludes soft-deleted pages', () => {
test('page_count and orphan_pages match getStats after soft-delete (the issue repro)', async () => {
for (let i = 0; i < 10; i++) await seedNote(`wiki/note-${i}`);
for (let i = 0; i < 6; i++) await engine.softDeletePage(`wiki/note-${i}`);
const stats = await engine.getStats();
const health = await engine.getHealth();
expect(stats.page_count).toBe(4);
// Pre-fix: 10 (raw rows). getHealth must agree with getStats.
expect(health.page_count).toBe(4);
// Pre-fix: 10 — deleted pages stayed in the islanded scan.
expect(health.orphan_pages).toBe(4);
});
test('brain_score moves when the user soft-deletes the islanded pages', async () => {
// 2 connected pages + 8 islanded ones.
await seedNote('wiki/hub');
await seedNote('wiki/leaf');
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
[await pageId('wiki/hub'), await pageId('wiki/leaf')],
);
for (let i = 0; i < 8; i++) await seedNote(`wiki/clutter-${i}`);
const before = await engine.getHealth();
for (let i = 0; i < 8; i++) await engine.softDeletePage(`wiki/clutter-${i}`);
const after = await engine.getHealth();
// Pre-fix both assertions fail: orphan_pages stayed 8 and brain_score
// was byte-identical before/after the delete.
expect(after.orphan_pages).toBe(0);
expect(after.brain_score).toBeGreaterThan(before.brain_score);
});
test('entity coverage denominators and most_connected exclude deleted entities', async () => {
// Live entity: inbound link + timeline entry → full coverage.
await engine.putPage('people/alice-example', { type: 'person', title: 'Alice', compiled_truth: 'a person', frontmatter: {} });
await engine.putPage('people/bob-example', { type: 'person', title: 'Bob', compiled_truth: 'another person', frontmatter: {} });
await seedNote('wiki/mentions-alice');
const aliceId = await pageId('people/alice-example');
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
[await pageId('wiki/mentions-alice'), aliceId],
);
await (engine as any).db.query(
`INSERT INTO timeline_entries (page_id, date, summary) VALUES ($1, '2026-01-01', 'met alice')`,
[aliceId],
);
await engine.softDeletePage('people/bob-example');
const h = await engine.getHealth();
// Pre-fix: bob stayed in the entity_pages CTE → coverage 0.5 each,
// and bob appeared in most_connected.
expect(h.link_coverage).toBe(1);
expect(h.timeline_coverage).toBe(1);
expect(h.most_connected.map((c) => c.slug)).not.toContain('people/bob-example');
});
test('chunk storage counts stay raw (the deliberate boundary)', async () => {
await seedNote('wiki/kept');
await seedNote('wiki/gone');
for (const slug of ['wiki/kept', 'wiki/gone']) {
await (engine as any).db.query(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text) VALUES ($1, 0, 'chunk')`,
[await pageId(slug)],
);
}
await engine.softDeletePage('wiki/gone');
const h = await engine.getHealth();
// Soft-deleted pages' chunks still occupy storage until purge; the
// missing_embeddings count keeps seeing them, same as getStats.
expect(h.missing_embeddings).toBe(2);
});
});
-254
View File
@@ -1,254 +0,0 @@
/**
* Default-embedder swap: ollama:bge-m3 @ 1024 with a loud hosted fallback.
*
* ZeroEntropy's hosted API (the previous default) sunsets 2026-09-04. The
* new default is open-weight + local (cannot be sunset); when Ollama is
* unreachable at `gbrain init`, init lands on the hosted fallback
* (openai:text-embedding-3-small @ 1024) loudly, with the way back and
* persists the `embedding_default_fallback` marker so `gbrain doctor`
* re-checks for Ollama on every run.
*
* Reachability is stubbed via `__setOllamaProbeForTests` (no fake daemon);
* the probe's own network behavior is covered only for the no-server case
* (dead port fail-open), which needs no listener.
*
* Master-discrimination: the "declared default" and "fallback resolution"
* tests fail BEHAVIORALLY on pre-swap code (wrong model/dims resolved, no
* marker, no notice) see also test/e2e/init-fresh-pglite.test.ts, whose
* updated subprocess tests prove the same through the real CLI.
*/
import { describe, test, expect, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { withEnv } from './helpers/with-env.ts';
import {
__setOllamaProbeForTests,
probeOllamaModel,
ollamaApiBase,
type OllamaProbeResult,
} from '../src/core/ai/ollama-detect.ts';
/**
* withEnv overrides that clear every embedding-provider auth key
* (enumerated from the recipe registry, not hardcoded) so resolution is
* deterministic on dev machines with ambient keys.
*/
async function embeddingKeyClears(): Promise<Record<string, undefined>> {
const { RECIPES } = await import('../src/core/ai/recipes/index.ts');
const overrides: Record<string, undefined> = {};
for (const recipe of RECIPES.values()) {
if (!recipe.touchpoints.embedding) continue;
for (const key of recipe.auth_env?.required ?? []) overrides[key] = undefined;
}
return overrides;
}
describe('declared default: ollama:bge-m3 @ 1024', () => {
test('DEFAULT_EMBEDDING_MODEL / DIMENSIONS are ollama:bge-m3 @ 1024', async () => {
const defaults = await import('../src/core/ai/defaults.ts');
expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('ollama:bge-m3');
// bge-m3's NATIVE width. Matryoshka free-truncation was measured for
// other families, not bge-m3 — do not "round" this in either direction.
expect(defaults.DEFAULT_EMBEDDING_DIMENSIONS).toBe(1024);
});
test('fallback is openai:text-embedding-3-small @ 1024 (same width as the default)', async () => {
const defaults = await import('../src/core/ai/defaults.ts');
expect(defaults.FALLBACK_EMBEDDING_MODEL).toBe('openai:text-embedding-3-small');
// 1024, not the model's native 1536: pinning the fallback at bge-m3's
// width makes the later fallback→default migration a vector-only
// rebuild (no column ALTER, no HNSW rebuild). Valid because OpenAI
// text-embedding-3-* accepts any Matryoshka width ≤ native.
expect(defaults.FALLBACK_EMBEDDING_DIMENSIONS).toBe(1024);
const { isValidOpenAITextEmbedding3Dim } = await import('../src/core/ai/dims.ts');
expect(isValidOpenAITextEmbedding3Dim('text-embedding-3-small', 1024)).toBe(true);
});
test('resolveSchemaEmbeddingDim ACCEPTS both the default and the fallback config', async () => {
const { resolveSchemaEmbeddingDim } = await import('../src/core/embedding-dim-check.ts');
const {
DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS,
FALLBACK_EMBEDDING_MODEL, FALLBACK_EMBEDDING_DIMENSIONS,
} = await import('../src/core/ai/defaults.ts');
for (const [model, dims] of [
[DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS],
[FALLBACK_EMBEDDING_MODEL, FALLBACK_EMBEDDING_DIMENSIONS],
] as const) {
const got = resolveSchemaEmbeddingDim({ embedding_model: model, embedding_dimensions: dims });
expect(got.ok).toBe(true);
if (got.ok) expect(got.dim).toBe(dims);
}
});
test('the default costs $0 in the embedding price table', async () => {
const { lookupEmbeddingPrice } = await import('../src/core/embedding-pricing.ts');
const { DEFAULT_EMBEDDING_MODEL } = await import('../src/core/ai/defaults.ts');
const price = lookupEmbeddingPrice(DEFAULT_EMBEDDING_MODEL);
expect(price.kind).toBe('known');
if (price.kind === 'known') expect(price.pricePerMTok).toBe(0);
});
});
describe('ollama probe (no daemon involved)', () => {
test('ollamaApiBase strips /v1 and trailing slashes from OLLAMA_BASE_URL', () => {
expect(ollamaApiBase({} as NodeJS.ProcessEnv)).toBe('http://localhost:11434');
expect(ollamaApiBase({ OLLAMA_BASE_URL: 'http://box:11434/v1' } as NodeJS.ProcessEnv)).toBe('http://box:11434');
expect(ollamaApiBase({ OLLAMA_BASE_URL: 'http://box:11434/' } as NodeJS.ProcessEnv)).toBe('http://box:11434');
});
test('unreachable server → fail-open {ok:false, serverUp:false}, never a throw', async () => {
const res = await probeOllamaModel('bge-m3', {
env: { OLLAMA_BASE_URL: 'http://127.0.0.1:9' } as NodeJS.ProcessEnv,
timeoutMs: 800,
});
expect(res.ok).toBe(false);
expect(res.serverUp).toBe(false);
expect(res.reason).toBe('unreachable');
});
});
describe('init embedding resolution (probe stubbed)', () => {
afterEach(() => {
__setOllamaProbeForTests(null);
});
/** Run resolveEmbeddingByEnv with stubbed probe + controlled env, capturing stderr. */
async function resolveWith(
probe: OllamaProbeResult,
envKeys: Record<string, string>,
): Promise<{ out: import('../src/commands/init.ts').ResolvedAIOptions; notice: string }> {
__setOllamaProbeForTests(async () => probe);
const clears = await embeddingKeyClears();
const errLines: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => { errLines.push(args.join(' ')); };
try {
return await withEnv({ ...clears, ...envKeys }, async () => {
const { resolveEmbeddingByEnv } = await import('../src/commands/init.ts');
const out: import('../src/commands/init.ts').ResolvedAIOptions = {};
await resolveEmbeddingByEnv(out, /* nonInteractive */ true);
return { out, notice: errLines.join('\n') };
});
} finally {
console.error = origError;
}
}
test('happy path: Ollama + bge-m3 available → the default wins, even over env keys', async () => {
// A hosted key present must NOT shadow the default.
const { out, notice } = await resolveWith(
{ ok: true, serverUp: true, reason: 'ok' },
{ OPENAI_API_KEY: 'sk-test' },
);
expect(out.embedding_model).toBe('ollama:bge-m3');
expect(out.embedding_dimensions).toBe(1024);
expect(out.embeddingFallback).toBeUndefined();
expect(notice).toContain('Detected Ollama with bge-m3');
});
test('fallback path: Ollama absent + OPENAI_API_KEY → hosted fallback, marker, LOUD notice', async () => {
const { out, notice } = await resolveWith(
{ ok: false, serverUp: false, reason: 'unreachable' },
{ OPENAI_API_KEY: 'sk-test' },
);
expect(out.embedding_model).toBe('openai:text-embedding-3-small');
expect(out.embedding_dimensions).toBe(1024);
expect(out.embeddingFallback).toBe(true);
// Visible, not silent: names the default, the reason, the trade-off,
// and the paste-ready way back.
expect(notice).toContain('default embedder is ollama:bge-m3');
expect(notice).toContain('Ollama is not reachable');
expect(notice).toContain('weaker on non-English content');
expect(notice).toContain('ollama pull bge-m3');
expect(notice).toContain('gbrain migrate embeddings --to ollama:bge-m3 --dim 1024');
});
test('fallback notice distinguishes "running but model not pulled"', async () => {
const { out, notice } = await resolveWith(
{ ok: false, serverUp: true, reason: 'model_missing' },
{ OPENAI_API_KEY: 'sk-test' },
);
expect(out.embeddingFallback).toBe(true);
expect(notice).toContain('running but bge-m3 is not pulled');
});
test('no Ollama, no OpenAI key, one other provider key → existing single-key auto-pick unchanged', async () => {
const { out } = await resolveWith(
{ ok: false, serverUp: false, reason: 'unreachable' },
{ VOYAGE_API_KEY: 'pa-test' },
);
expect(out.embedding_model?.startsWith('voyage:')).toBe(true);
expect(out.embeddingFallback).toBeUndefined();
});
});
describe('doctor re-check: embedding_default_fallback', () => {
afterEach(() => {
__setOllamaProbeForTests(null);
});
/** Write a config.json into a throw-away GBRAIN_HOME and run the check there. */
async function checkWith(
cfg: Record<string, unknown>,
probe: OllamaProbeResult | null,
): Promise<{ status: string; message: string }> {
if (probe) __setOllamaProbeForTests(async () => probe);
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-fallback-doctor-'));
mkdirSync(join(tmpHome, '.gbrain'), { recursive: true });
writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify(cfg));
try {
return await withEnv({ GBRAIN_HOME: tmpHome }, async () => {
const { checkEmbeddingDefaultFallback } = await import('../src/commands/doctor.ts');
return checkEmbeddingDefaultFallback({} as never);
});
} finally {
rmSync(tmpHome, { recursive: true, force: true });
}
}
test('warns with the migrate command once Ollama becomes available', async () => {
const check = await checkWith({
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-small',
embedding_dimensions: 1024,
embedding_default_fallback: 'ollama:bge-m3',
}, { ok: true, serverUp: true, reason: 'ok' });
expect(check.status).toBe('warn');
expect(check.message).toContain('gbrain migrate embeddings --to ollama:bge-m3 --dim 1024');
});
test('stays ok (informational) while Ollama is still unavailable', async () => {
const check = await checkWith({
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-small',
embedding_dimensions: 1024,
embedding_default_fallback: 'ollama:bge-m3',
}, { ok: false, serverUp: false, reason: 'unreachable' });
expect(check.status).toBe('ok');
expect(check.message).toContain('hosted embedding fallback');
});
test('stale marker (user moved off the fallback) is ignored', async () => {
// Probe stubbed "available" to prove the staleness guard short-circuits
// before the probe even matters.
const check = await checkWith({
engine: 'pglite',
embedding_model: 'voyage:voyage-3-large',
embedding_dimensions: 1024,
embedding_default_fallback: 'ollama:bge-m3',
}, { ok: true, serverUp: true, reason: 'ok' });
expect(check.status).toBe('ok');
expect(check.message).toContain('stale');
});
test('no marker → skip', async () => {
const check = await checkWith(
{ engine: 'pglite', embedding_model: 'openai:text-embedding-3-small' },
null,
);
expect(check.status).toBe('ok');
expect(check.message).toContain('skip');
});
});
+2 -2
View File
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
});
describe('KNOBS_HASH_VERSION', () => {
it('is 14 (13→14 compiled_truth boost no longer applies at detail=medium, so pre-fix rankings must be unreachable, #3430)', () => {
expect(KNOBS_HASH_VERSION).toBe(14);
it('is 13 (12→13 embedding-provider migration invalidates rows written against the prior embedding space, #3390)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
});
});
@@ -1,112 +0,0 @@
/**
* #3430: the compiled_truth boost must not apply at `detail=medium`.
*
* `COMPILED_TRUTH_BOOST = 2.0` is applied AFTER RRF score normalization. RRF's
* entire dynamic range over a 100-deep pool is 1/60 1/160 (a factor of 2.67),
* so a 2.0x multiplier consumes roughly three quarters of it. Break-even is
* `2/(60+r) >= 1/60`, i.e. r <= 60 so ANY boosted chunk in the first 60 ranks
* outranks an unboosted rank-1 chunk. That is a categorical filter, not a tilt:
* a page whose actual answer is in a `fenced_code` chunk returns the prose
* chunk instead, and the code chunk leaves the result window entirely.
*
* The gate was written as `detail !== 'high'` "high is special" but the
* documented contract in `src/core/operations.ts` is:
*
* low (compiled truth only), medium (default, all with dedup), high (all chunks)
*
* which makes LOW the special one. `low` already restricts to compiled_truth,
* so a boost there is a no-op among equals; `medium` and `high` are both
* supposed to see everything. Hence `detail === 'low'`.
*
* These tests pin the arithmetic, not the constant they would still fail if
* someone reintroduced a boost at medium with a different multiplier or behind
* a score floor, which is why they assert final RANK rather than score.
*/
import { describe, test, expect } from 'bun:test';
import { rrfFusion, RRF_K, shouldBoostCompiledTruth } from '../src/core/search/hybrid.ts';
import { KNOBS_HASH_VERSION } from '../src/core/search/mode.ts';
import type { SearchResult } from '../src/core/types.ts';
function chunk(slug: string, chunkSource: string): SearchResult {
return { slug, chunk_source: chunkSource, chunk_text: 'x', title: slug, score: 0 } as unknown as SearchResult;
}
/** One vector arm: the correct answer at rank 0, then `n` compiled_truth chunks. */
function poolWithAnswerFirst(n: number): SearchResult[] {
const list = [chunk('code/answer', 'fenced_code')];
for (let i = 0; i < n; i++) list.push(chunk(`prose/p${i}`, 'compiled_truth'));
return list;
}
function rankOfAnswer(results: SearchResult[]): number {
return results.findIndex((r) => r.slug === 'code/answer');
}
describe('#3430: the detail→boost mapping itself', () => {
// These are the assertions that actually FAIL on master. The rrfFusion tests
// below pin the arithmetic but pass either way, because they pass the boost
// flag explicitly — they cannot see how hybridSearch decides it. This is the
// wiring.
test('ONLY detail=low boosts compiled_truth', () => {
expect(shouldBoostCompiledTruth('low')).toBe(true);
expect(shouldBoostCompiledTruth('medium')).toBe(false);
expect(shouldBoostCompiledTruth('high')).toBe(false);
});
test('an absent detail does not boost — medium is the documented default', () => {
// Callers that omit detail get medium semantics, so the unset case must
// match medium, not low. A `!== 'high'` spelling gets this backwards.
expect(shouldBoostCompiledTruth(undefined)).toBe(false);
expect(shouldBoostCompiledTruth(null)).toBe(false);
});
test('an unrecognized detail value does not boost', () => {
// Fail-open toward showing everything rather than silently filtering.
expect(shouldBoostCompiledTruth('')).toBe(false);
expect(shouldBoostCompiledTruth('LOW')).toBe(false);
expect(shouldBoostCompiledTruth('detailed')).toBe(false);
});
test('the cache version was bumped so pre-fix rankings are unreachable', () => {
// Results are cached AFTER fusion, so rows written under the old boost
// semantics would otherwise be served under the new ones for the whole TTL.
// 13 was the pre-fix value.
expect(KNOBS_HASH_VERSION).toBeGreaterThanOrEqual(14);
});
});
describe('#3430: compiled_truth boost scope', () => {
test('boost OFF (detail=medium/high) keeps the vector-ranked answer at rank 0', () => {
// The regression this file exists for. Pre-fix, medium passed applyBoost=true
// and the answer landed at rank n — outside a 20-result window for n >= 20.
for (const n of [10, 20, 40, 80]) {
const fused = rrfFusion([poolWithAnswerFirst(n)], RRF_K, false);
expect(rankOfAnswer(fused), `n=${n}: answer must stay first without the boost`).toBe(0);
}
});
test('boost ON demonstrates the categorical displacement it causes', () => {
// Documents WHY the boost cannot be on at medium. Not an endorsement of
// these numbers — a characterization of the mechanism, so a future reader
// sees the cost rather than re-deriving it.
const observed = [10, 20, 40].map((n) => ({
n,
rank: rankOfAnswer(rrfFusion([poolWithAnswerFirst(n)], RRF_K, true)),
}));
// Displacement scales with pool composition: the answer is pushed back by
// roughly one position per boosted chunk ahead of the break-even rank.
for (const { n, rank } of observed) {
expect(rank, `n=${n}: boosted chunks should displace the answer`).toBeGreaterThan(0);
}
// And past ~20 compiled_truth chunks it leaves a default-size window.
expect(observed.find((o) => o.n === 20)!.rank).toBeGreaterThanOrEqual(20);
});
test('with the boost off, compiled_truth still wins when the vector arm ranks it first', () => {
// Guard against over-correcting: removing the boost must not penalize
// compiled_truth, only stop privileging it.
const list = [chunk('prose/answer', 'compiled_truth'), chunk('code/other', 'fenced_code')];
const fused = rrfFusion([list], RRF_K, false);
expect(fused[0].slug).toBe('prose/answer');
});
});
+3 -6
View File
@@ -413,10 +413,7 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
// #3390/#3391: bumped 12→13 for the embedding-provider migration wave —
// legacy callers hash prov=default before AND after a provider swap, so
// pre-migration cache rows must become unreachable on upgrade.
// v0.42.67.x bumped 13→14: the compiled_truth boost no longer applies at
// detail=medium (#3430). Cached rows were ranked under the old semantics,
// so they must become unreachable rather than be served under the new ones.
expect(KNOBS_HASH_VERSION).toBe(14);
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
@@ -581,8 +578,8 @@ describe('v0.40.4 — graph_signals knob', () => {
});
describe('v0.42.3.0 — autocut knobs', () => {
test('KNOBS_HASH_VERSION is 14 (13→14 compiled_truth boost scope fix, #3430)', () => {
expect(KNOBS_HASH_VERSION).toBe(14);
test('KNOBS_HASH_VERSION is 13 (12→13 embedding-migration wave, #3390/#3391)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
+1 -4
View File
@@ -64,10 +64,7 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
// pre-fix document-side query vectors must not be served.
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
// #3430: 13→14 — the compiled_truth boost no longer applies at
// detail=medium. Results are cached after fusion, so rows ranked under
// the old boost semantics must not be served under the new ones.
expect(KNOBS_HASH_VERSION).toBe(14);
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('hash is 16 hex chars regardless of reranker config', () => {
-280
View File
@@ -1,280 +0,0 @@
/**
* #3056 sync rename path: a failed `updateSlug` must not leave a live
* duplicate of the renamed page behind.
*
* Before the fix, the rename loop swallowed `updateSlug` failures with an
* empty catch ("treat as add") and could not see a zero-row UPDATE at all
* (updateSlug returned void). The run then fell through to importFile,
* which created/updated the row at the new path while the old row stayed
* behind, live, with its slug occupied. Nothing was logged, no counter
* moved, and the duplicate was permanent.
*
* The fix reconciles: when the cheap rename didn't move a row AND the
* destination demonstrably materialized, the stale old row is located
* positively by `source_path = from` and deleted. Two safety rails:
*
* - dedup-skip protection: identity dedup can skip the import against
* the OLD row, in which case nothing landed at the destination and
* deleting the old row would destroy the only copy no reconcile.
* - no slug-guess deletes: the stale row is found by source_path only;
* an unrelated row that happens to sit at the guessed slug survives.
*
* A failed reconcile delete lands in failedFiles so the existing failure
* gate blocks the bookmark and the next run retries the same rename diff.
*/
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { execSync } from 'node:child_process';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
let engine: PGLiteEngine;
const repos: string[] = [];
// Serial-file requirement: blocked runs write real rows to the sync-failure
// ledger under the gbrain home — isolate it per test so the operator's
// actual ledger is never touched (GBRAIN_HOME is the isolation lever;
// process.env.HOME does not redirect Bun's os.homedir()).
let tmpHome: string;
const originalGbrainHome = process.env.GBRAIN_HOME;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-3056-home-'));
process.env.GBRAIN_HOME = tmpHome;
await resetPgliteState(engine);
});
afterEach(() => {
if (originalGbrainHome !== undefined) process.env.GBRAIN_HOME = originalGbrainHome;
else delete process.env.GBRAIN_HOME;
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
while (repos.length) {
const d = repos.pop();
if (d) rmSync(d, { recursive: true, force: true });
}
});
function personMd(title: string, body: string): string {
return ['---', 'type: person', `title: ${title}`, '---', '', body].join('\n');
}
/** Create a temp git repo seeded with the given files + an initial commit. */
function mkRepo(files: Record<string, string>): string {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-3056-'));
repos.push(dir);
execSync('git init', { cwd: dir, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: dir, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: dir, stdio: 'pipe' });
for (const [rel, content] of Object.entries(files)) {
mkdirSync(join(dir, rel, '..'), { recursive: true });
writeFileSync(join(dir, rel), content);
}
execSync('git add -A && git commit -m "initial"', { cwd: dir, stdio: 'pipe' });
return dir;
}
const SYNC_OPTS = { noPull: true, noEmbed: true, noExtract: true, sourceId: 'default' } as const;
async function countPages(): Promise<number> {
const rows = await engine.executeRaw<{ n: number | string }>(
`SELECT count(*)::int AS n FROM pages WHERE source_id = 'default'`,
);
return Number(rows[0]?.n ?? 0);
}
describe('updateSlug engine contract (#3056)', () => {
test('returns 1 when the old slug row is moved', async () => {
await engine.putPage('people/old', {
type: 'person', title: 'Old', compiled_truth: 'body',
}, { sourceId: 'default' });
const moved = await engine.updateSlug('people/old', 'people/new', { sourceId: 'default' });
expect(moved).toBe(1);
expect(await engine.getPage('people/new')).not.toBeNull();
});
test('returns 0 when the old slug has no row (the silent no-op case)', async () => {
const moved = await engine.updateSlug('people/ghost', 'people/new', { sourceId: 'default' });
expect(moved).toBe(0);
});
});
describe('#3056: rename fallback reconciles the stale old row', () => {
test('collision: destination slug occupied → stale old row deleted after import lands', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(await engine.getPage('people/carol')).not.toBeNull();
// A pre-existing row already occupies the rename destination, so
// updateSlug throws (source_id, slug) UNIQUE and the loop falls back.
await engine.putPage('people/dana', {
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
}, { sourceId: 'default' });
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(result.status).toBe('synced');
// The destination carries the renamed file's content...
const dana = await engine.getPage('people/dana');
expect(dana).not.toBeNull();
expect(dana!.compiled_truth).toContain('Carol is a person.');
// ...and the stale old row is gone — no live duplicate.
expect(await engine.getPage('people/carol')).toBeNull();
expect(await countPages()).toBe(1);
});
test('dedup-skip against the old row must NOT reconcile: the only copy survives', async () => {
const { performSync } = await import('../src/commands/sync.ts');
// frontmatter.id gives identity dedup a handle: the import at the new
// path can skip as "identical to <old row>" — in which case NOTHING
// landed at the destination and deleting the old row would destroy the
// only copy of the content.
const md = ['---', 'type: person', 'title: Carol', 'id: ext-3056', '---', '', 'Carol is a person.'].join('\n');
const repo = mkRepo({ 'people/carol.md': md });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(await engine.getPage('people/carol')).not.toBeNull();
// Destination occupied → updateSlug throws → fallback path.
await engine.putPage('people/dana', {
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
}, { sourceId: 'default' });
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
// The import skipped against the OLD row (identity dedup), so the
// destination never materialized with the renamed content — the
// reconcile must not have deleted the old row, which still holds the
// only copy.
const carol = await engine.getPage('people/carol');
expect(carol).not.toBeNull();
expect(carol!.compiled_truth).toContain('Carol is a person.');
});
test('reconcile never deletes by slug guess: unrelated manual row survives', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
// The file's real row drifts to a divergent slug with no source_path
// (unlocatable), and an UNRELATED manually-curated page happens to sit
// at the path-derived slug a naive reconcile would guess.
await engine.executeRaw(
`UPDATE pages SET slug = 'people/carol-divergent', source_path = NULL
WHERE source_id = 'default' AND slug = 'people/carol'`,
);
await engine.putPage('people/carol', {
type: 'person', title: 'Manual Carol', compiled_truth: 'hand-authored, not from the file',
}, { sourceId: 'default' });
// Destination occupied → updateSlug throws UNIQUE → fallback path.
await engine.putPage('people/dana', {
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
}, { sourceId: 'default' });
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(result.status).toBe('synced');
// The destination materialized with the file's content...
const dana = await engine.getPage('people/dana');
expect(dana).not.toBeNull();
expect(dana!.compiled_truth).toContain('Carol is a person.');
// ...but no row had source_path = from, so the reconcile deleted
// NOTHING: the unrelated manual row at the guessed slug survives.
const manual = await engine.getPage('people/carol');
expect(manual).not.toBeNull();
expect(manual!.compiled_truth).toContain('hand-authored');
});
test('happy path: clean git mv rename keeps page_id and touches nothing else', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
const before = await engine.getPage('people/carol');
expect(before).not.toBeNull();
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(result.status).toBe('synced');
const after = await engine.getPage('people/dana');
expect(after).not.toBeNull();
expect(after!.id).toBe(before!.id); // cheap-path rename preserved the row
expect(await engine.getPage('people/carol')).toBeNull();
expect(await countPages()).toBe(1);
});
test('reconcile failure blocks the bookmark and the next run retries to convergence', async () => {
const { performSync } = await import('../src/commands/sync.ts');
const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') });
await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
await engine.putPage('people/dana', {
type: 'person', title: 'Dana (stale)', compiled_truth: 'occupies the destination slug',
}, { sourceId: 'default' });
execSync('git mv people/carol.md people/dana.md', { cwd: repo, stdio: 'pipe' });
execSync('git commit -m "rename carol to dana"', { cwd: repo, stdio: 'pipe' });
// Inject a transient failure into the reconcile delete.
const origDelete = engine.deletePage.bind(engine);
engine.deletePage = async () => { throw new Error('injected transient delete failure'); };
let blocked;
try {
blocked = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
} finally {
engine.deletePage = origDelete;
}
// The failed reconcile is not checkpointed past: the run blocks and the
// stale duplicate is still visible. The failure is recorded as a
// `<rename:…>` SENTINEL, which the auto-skip valve can never
// chronic-skip — an outage lasting longer than the threshold must not
// quietly bank the duplicate.
expect(blocked.status).toBe('blocked_by_failures');
expect(blocked.failedFiles).toBe(1);
expect(await engine.getPage('people/carol')).not.toBeNull();
const { loadSyncFailures } = await import('../src/core/sync-failure-ledger.ts');
const openSentinels = loadSyncFailures().filter(
f => f.path === '<rename:people/dana.md>' && f.state === 'open',
);
expect(openSentinels).toHaveLength(1);
// Next run (failure gone) retries the same rename diff and converges.
const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS });
expect(result.status).toBe('synced');
expect(await engine.getPage('people/carol')).toBeNull();
const dana = await engine.getPage('people/dana');
expect(dana).not.toBeNull();
expect(dana!.compiled_truth).toContain('Carol is a person.');
expect(await countPages()).toBe(1);
// The convergence also clears the sentinel row — doctor must not keep
// warning about a rename that has since reconciled.
const remaining = loadSyncFailures().filter(
f => f.path === '<rename:people/dana.md>' && f.state === 'open',
);
expect(remaining).toHaveLength(0);
});
});
+12 -17
View File
@@ -19,30 +19,28 @@ describe('v0.37 Lane A — defaults sweep', () => {
// CDX2-1: these were file-private const; Lane A consumers (schema
// helpers, registry) need them exported. Importing here is the test.
const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } = await import('../src/core/ai/gateway.ts');
expect(DEFAULT_EMBEDDING_MODEL).toBe('ollama:bge-m3');
expect(DEFAULT_EMBEDDING_DIMENSIONS).toBe(1024);
expect(DEFAULT_EMBEDDING_MODEL).toBe('zeroentropyai:zembed-1');
expect(DEFAULT_EMBEDDING_DIMENSIONS).toBe(1280);
});
test('A.0: ai/defaults.ts is the canonical source (leaf module, no SDK pulls)', async () => {
const defaults = await import('../src/core/ai/defaults.ts');
expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('ollama:bge-m3');
expect(defaults.DEFAULT_EMBEDDING_DIMENSIONS).toBe(1024);
expect(defaults.DEFAULT_EMBEDDING_MODEL).toBe('zeroentropyai:zembed-1');
expect(defaults.DEFAULT_EMBEDDING_DIMENSIONS).toBe(1280);
});
// T-11 / T-12: registry + schema defaults track gateway constants.
test('A.1: getPGLiteSchema() default-args produce a vector(1024) column', async () => {
test('A.1: getPGLiteSchema() default-args produce a vector(1280) column', async () => {
const { getPGLiteSchema } = await import('../src/core/pglite-schema.ts');
const sql = getPGLiteSchema(); // no args — uses defaults
expect(sql).toContain('vector(1024)');
expect(sql).toContain("'ollama:bge-m3'");
expect(sql).toContain('vector(1280)');
expect(sql).not.toContain('vector(1536)');
});
test('A.2: getPostgresSchema() default-args produce a vector(1024) column', async () => {
test('A.2: getPostgresSchema() default-args produce a vector(1280) column', async () => {
const { getPostgresSchema } = await import('../src/core/postgres-engine.ts');
const sql = getPostgresSchema();
expect(sql).toContain('vector(1024)');
expect(sql).toContain("'ollama:bge-m3'");
expect(sql).toContain('vector(1280)');
expect(sql).not.toContain('vector(1536)');
});
@@ -50,14 +48,11 @@ describe('v0.37 Lane A — defaults sweep', () => {
const { getPostgresSchema } = await import('../src/core/postgres-engine.ts');
const sql = getPostgresSchema(2048, 'voyage:voyage-4-large');
expect(sql).toContain('vector(2048)');
// The default model must not leak through when overridden. (Width 1024
// can't be asserted absent — the schema carries a fixed vector(1024)
// auxiliary embedding column unrelated to the default.)
expect(sql).not.toContain("'ollama:bge-m3'");
expect(sql).not.toContain('vector(1280)');
expect(sql).toContain('voyage:voyage-4-large');
});
test('A.5: embedding-column registry builtin defaults to ollama/1024 on empty config + gateway', async () => {
test('A.5: embedding-column registry builtin defaults to ZE/1280 on empty config + gateway', async () => {
// The registry's resolution chain is cfg > gateway > DEFAULT. With
// no cfg AND no gateway, it should fall through to the canonical
// default (ZE/1280). Hard-unconfigure first to exercise that path —
@@ -68,8 +63,8 @@ describe('v0.37 Lane A — defaults sweep', () => {
try {
const reg = getEmbeddingColumnRegistry({ engine: 'pglite' } as any);
expect(reg['embedding']).toBeDefined();
expect(reg['embedding'].provider).toBe('ollama:bge-m3');
expect(reg['embedding'].dimensions).toBe(1024);
expect(reg['embedding'].provider).toBe('zeroentropyai:zembed-1');
expect(reg['embedding'].dimensions).toBe(1280);
} finally {
// Restore the preload's legacy baseline so the rest of the file's
// tests (and subsequent files in this shard) see a configured gateway.