Compare commits

...
Author SHA1 Message Date
Garry Tan bb756d90c8 fix(test): recipe-ollama-dims pins a truncated width — native-width requests now omit the param (#3699)
Master's #1072 test asserted bare qwen3-embedding@1024 threads dimensions:1024;
wave PR #3699 suppresses the param when the request equals the native width
(1024 for the bare id), so the same input now correctly returns undefined —
pinned by dims-qwen3-native.test.ts. Moved this test to 512 to keep its actual
intent (bare id recognized as Matryoshka-capable) without contradicting the
suppression. Third composition defect of the wave; caught by CI shard 8.
2026-08-01 07:26:11 +08:00
Garry Tan d64c3f54ff v0.42.69.0 fix: community fix wave — 22 contributed fixes for silent-failure paths, local-model support, and multi-source routing 2026-08-01 07:12:23 +08:00
Garry Tan aa6f52c464 fix(skills,docs): unify-types playbooks must pass apply:true after the #3574 default flip
#3574 flipped the unify-types worker default to dry-run (jobs.ts:2221,
apply: data.apply ?? false) and updated the architecture docs, but three
agent-facing surfaces still presented the bare submit as the Apply step:
skills/schema-unify/SKILL.md 'Phase 3: Apply', skills/conventions/
schema-evolution.md, and README.md. Because #3545 also edited SKILL.md in
this wave, each PR looked self-consistent in isolation — only the composed
branch shipped a playbook whose apply step silently retypes nothing and
never flips the active pack. Skills distribute downstream via the skillpack,
so this would have propagated. Found by an independent cross-PR review pass.
2026-08-01 07:06:36 +08:00
Garry Tan a06dded2f6 fix(budget): reconcile #3691 and #3541 — free local providers are priceable at $0
#3691's regression test used llama-server as an 'unknown provider' example.
#3541 (same wave) prices ollama/llama-server at $0 via FREE_LOCAL_CHAT_PROVIDERS,
so that example is now priceable and the assertion inverted. Swapped in groq —
the paid-but-unpriced case #3691's own description cites — and added the
positive assertion that free local providers keep their cap enforced.
2026-08-01 07:06:36 +08:00
Garry TanandChristoph 682476e1cd feat(by-mention): Unicode-aware word tokenizer — Vietnamese/diacritic name extraction (#3563)
Co-Authored-By: Christoph <astaran@herr-der-ringe-film.de>
2026-08-01 05:40:40 +08:00
Garry Tanand老赵 193c8ca2bc test(cli): cover import side-effect guard (#3581)
Co-Authored-By: 老赵 <273731059@qq.com>
2026-08-01 05:40:37 +08:00
Garry TanandKyzcreig 577d4fddf0 fix(import): checkpoint on a time interval and before preserve, not only every 100 files (#3585)
Co-Authored-By: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
2026-08-01 05:40:34 +08:00
Garry TanandVyacheslav Zakharov 4e23c1564a fix(test): kill timeout-resistant unit shards (#3631)
Co-Authored-By: Vyacheslav Zakharov <vyacheslav.zakharov@avers.kz>
2026-08-01 05:40:31 +08:00
Garry TanandAlexey 659aa1027f fix: health metric that cannot converge, and facts lost to a row_num collision (#3634)
Co-Authored-By: Alexey (CTO) <cto@phrase.local>
2026-08-01 05:40:29 +08:00
Garry TanandBen Young 707127a7bf fix(budget): price free local chat providers at $0, matching embed and rerank (#3541)
Co-Authored-By: Ben Young <Grimnoth@users.noreply.github.com>
2026-08-01 05:40:25 +08:00
Garry TanandAustin Wilhite 239954cbd5 fix(cycle): stop extract_atoms silently skipping all work on unpriced models (#3691)
Co-Authored-By: Austin Wilhite <austinw80@gmail.com>
2026-08-01 05:40:22 +08:00
Garry Tanandzenspam d84f5a9182 fix(dims): skip dimensions param when it equals qwen3-embedding native width (#3699)
Co-Authored-By: zenspam <zenspam@gmail.com>
2026-08-01 05:40:19 +08:00
Garry TanandTime Attakc c79d238ff9 fix(docs): remove references to the nonexistent gbrain install command (#3502) (#3545)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:40:17 +08:00
Garry Tan dd4c10cab3 fix(embed): stop one bad chunk darkening a page, and exit non-zero on failures (#3037) (#3532)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 05:40:14 +08:00
Garry TanandRyan Ayers 2b36c40de4 fix(cycle): surface stamp-write failure on CycleReport and degrade status (#3504) (#3589)
Co-Authored-By: Ryan Ayers <rayers@dividia.net>
2026-08-01 05:40:11 +08:00
Garry Tan 0a188cbcfd fix(jobs): unify-types worker defaults to dry-run per its handler contract (#1575) (#3574)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 05:40:08 +08:00
Garry TanandTime Attakc c9dc923f16 fix(skillpack): resolve gbrain root from module path when cwd walk fails (#3144)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:40:06 +08:00
Garry TanandBrett f7a6516a60 fix(models): hint the /v1 base-URL suffix when a doctor chat probe 401s on an openai-compatible proxy (#3553)
Co-Authored-By: Brett <brettdavies@users.noreply.github.com>
2026-08-01 05:40:03 +08:00
Garry TanandTime Attakc 9cfb72e3b9 fix(doctor): bound embedding provider health probe (#3364)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:40:00 +08:00
Garry TanandTime Attakc 2ba9c795e9 fix(doctor): stop reporting Windows-drive image paths as missing under WSL (#1835) (#3523)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:57 +08:00
Garry TanandTime Attakc 9ac6c8685f fix(dream): honor a configured 0 in synthesize + auto_think config resolution (stop coercing to the default) (#3552)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:54 +08:00
Garry TanandTime Attakc 4e536ca0ec fix(minions): reconnect the stall timer, stop failJob masking job errors, unify retry matchers (#1720) (#3555)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:48 +08:00
Garry TanandTime Attakc 470e95fc57 fix(entities): resolveEntitySlug fallback keeps the path separator (#3447) (#3567)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:45 +08:00
Garry TanandTime Attakc 99ffe4cce2 fix(cycle): log swallowed lock.release() failures (#1470) (#3572)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:43 +08:00
Garry TanandTime Attakc 6d56967407 fix(operations): sync_brain threads ctx.sourceId to performSync (#2830) (#3568)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:40 +08:00
Garry TanandTime Attakc 23edfc71f7 fix(cli): route init --help to its own usage text (#3652)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:39:37 +08:00
88 changed files with 3407 additions and 318 deletions
+47
View File
@@ -2,6 +2,53 @@
All notable changes to GBrain will be documented in this file.
## [0.42.69.0] - 2026-08-01
**A community fix wave: 22 contributed fixes, most of them for work your brain was quietly not doing.**
The theme of this release is silent failure. A nightly cycle that reported `ok` while extracting nothing. An `embed` run that left a whole page unsearchable because one chunk in it failed, then exited 0. A health metric that recommended the same step forever because it counted one thing and the fix measured another. None of these looked broken from the outside, which is exactly why they lasted.
**If you run a local or non-Anthropic model, atom extraction was doing nothing.** With a cost cap set, any model absent from the pricing tables made the first work item hard-fail, which latched a budget flag and skipped every remaining item — while the phase still reported success. Local models (`ollama`, `llama-server`) now price at $0, because local inference costs electricity rather than tokens, so their caps stay enforceable. Genuinely unpriced paid providers still skip, but loudly now instead of silently.
**`gbrain embed` no longer lets one bad chunk darken an entire page,** and it exits non-zero when embeddings actually fail. If you have a cron wrapping `gbrain embed`, a brain holding permanently un-embeddable content will now turn that cron red. That is the intended change — it was previously green while silently incomplete.
**Non-Latin and diacritic names now survive mention extraction.** The by-mention tokenizer matched ASCII letters and digits only, so `Đà Nẵng` shredded into one- and two-character fragments and never matched anything. Names in Vietnamese, and any script outside ASCII, are now tokenized properly.
**Self-hosted embedding backends work.** Fixed-dimension OpenAI-compatible servers that reject an explicit `dimensions` parameter no longer get sent one when the requested width already matches the model's native width. A vector search on the embedded database also now asks the index for as many candidates as it was told to consider, instead of silently truncating the pool to the driver default.
**Multi-source brains route correctly in two more places.** A programmatic `sync_brain` call now syncs the source it was handed rather than the global default, and entity slug resolution keeps its path separators instead of flattening `people/alice-example` into an id no page can hold.
**Safer default on a destructive migration.** Submitting the type-unification job without an explicit `apply` now previews instead of applying. If you have that command in a runbook, add `"apply":true` — the playbooks and README were updated to show it.
Also: interrupted imports keep their tail instead of losing progress below the next 100-file boundary; `gbrain init --help` prints its own help instead of a stub; `doctor` stops reporting Windows drive paths as missing files under WSL and bounds its embedding health probe instead of retrying a permanent auth failure three times; cycle lock-release and stamp-write failures are visible instead of swallowed; and references to a `gbrain install` command that never existed are gone from the docs.
### To take advantage of v0.42.69.0
```bash
gbrain upgrade # or: bun install -g gbrain@0.42.69.0
gbrain doctor # confirms the health metric now converges
gbrain embed --stale # exits non-zero if anything is genuinely un-embeddable
```
If you use a local chat model for the nightly cycle, re-run it once and check that atoms actually land:
```bash
gbrain dream --json | jq '.phases[] | select(.name=="extract_atoms")'
```
If you have `gbrain jobs submit unify-types` in a runbook or script, add `"apply":true` to its `--params` or it will now preview only.
### For contributors
Two defects existed only in the *combination* of otherwise-sound fixes, and were caught by reviewing the composed branch rather than the individual changes:
- `isModelPriceable` was introduced with a test asserting `llama-server` is unpriced, while a second fix in the same wave priced `llama-server` at $0. Together the assertion inverted. Reconciled by using a genuinely unpriced provider in the test and pinning the positive case: free local providers are priceable at $0, so their caps stay enforced.
- The type-unification default flipped to dry-run, but three agent-facing playbooks still presented a bare submit as the apply step. Because a second fix in the same wave also edited one of those files, each change looked self-consistent alone. Skills ship downstream via the skillpack, so this would have propagated a playbook whose apply step silently did nothing.
One reviewed fix was deliberately held back: extending the inline subagent drain to Postgres composes badly with this wave's minion connection-recovery work, since the drain calls the same queue operations without the new recovery path and can strand a child job in a per-run queue no worker will claim.
Contributed by @alexey-metaengage (#3652), @time-attack (#3568, #3572, #3567, #3555, #3523, #3144, #3574, #3532, #3545), @brettdavies (#3552, #3553), @mattchronicle (#3364), @rayers (#3589), @zenspam (#3699), @awilhite (#3691), @Grimnoth (#3541), @georgell-ceo (#3634), @Vyacheslav-Zakharov (#3631), @Kyzcreig (#3585), @HammerTech-Z (#3581), @cfeddersen (#3563).
## [0.42.68.1] - 2026-07-30
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
+1 -1
View File
@@ -216,7 +216,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
+1 -1
View File
@@ -2580,7 +2580,7 @@ contributor traps.
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
- [ ] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
+1 -1
View File
@@ -1 +1 @@
0.42.68.1
0.42.69.0
+2 -2
View File
@@ -192,7 +192,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/extraction-review.ts` — Extraction quarantine lane markers (issue #160), sibling of `src/core/quarantine.ts` / `embed-skip.ts` (frontmatter-key pattern, no schema migration). Auto-extracted stubs from untrusted input carry the PAIR `provenance: 'auto-extracted'` + `status: 'unverified'` (both required — user pages with their own `status`/`provenance` never match). Exports `quarantineMarkers()`, `isUnverifiedExtraction()` (JS predicate) and `unverifiedExtractionFragment(alias)` — the single SQL source of truth consumed by `buildSourceFactorCase` (namespace source-boost guard), both engines' `getUnverifiedExtractionPageIds`, the `extraction_pending` op, and the `unverified_extractions` doctor check, so filter and marker keys can never drift. Consequences: unverified stubs are excluded from the compiled-truth fusion boost + the `people/`/`companies/` source-boost (rank as ordinary content), stamped `unverified: true` in search results (`stampUnverifiedExtractions`, hybrid.ts), listed by `extraction_pending`, promoted (status → `verified`, provenance kept for audit) or rejected (soft-delete) by the owner-only `extraction_review` op. Pinned by `test/extraction-review.test.ts` (PGLite) + `test/e2e/extraction-review-postgres.test.ts` (live Postgres parity).
- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts``gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich:<src>:<slug>')``getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1``runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `<context>…</context>` data-envelope delimiters (injection escape, mirrors the `</trajectory>` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity).
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping.
- `src/commands/embed.ts``gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`.
- `src/commands/embed.ts``gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp per page when every chunk embedded cleanly. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. Embed failures are never silent (#3037): all three page paths embed via `embedPageTexts`, which tries the page's chunks in one batch and, on a PERMANENT request-shaped failure (non-429, non-`AITransientError`, non-auth), retries once per chunk so one bad chunk costs one chunk instead of darkening the whole page (failed chunks stay `embedding IS NULL` for the next `--stale` pass; a partially-failed page is never signature-stamped). Rate-limit/outage/auth failures do NOT fan out (cost bounding — `embedBatchWithBackoff` already owns 429 backoff). Failed chunk counts land on `EmbedResult.failures` + capped `failure_samples`, and `src/cli.ts`'s embed case sets a non-zero exit verdict on `failures > 0` (mirror of the `import` errors>0 guard). Pinned by `test/embed-partial-failure-3037.serial.test.ts` + `test/embed-exit-code-3037.serial.test.ts` (real spawned CLI).
- `src/core/retrieval-upgrade-planner.ts``runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL THREE dim-pinned text-embedding-space columns at `targetDim``content_chunks.embedding`, `query_cache.embedding`, `facts.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache + facts are created at brain-birth width by `migrate.ts` and NO migration ever ALTERs them, so omitting either leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all three widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`.
- `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state``runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions``persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})``SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector).
- `src/commands/migrate-embeddings.ts``gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite).
@@ -396,7 +396,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
- `src/commands/pages.ts``gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/commands/pages.ts``gbrain purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `recordCompleted` binds its array through `$3::text::jsonb` (NOT a bare `$3::jsonb`) so postgres.js `.unsafe()` doesn't double-encode `JSON.stringify(sorted)` into the scalar string that CHECK rejects — the #2339 bug that aborted every multi-source sync at the first pin write (PGLite parsed it silently, so it shipped). A DATABASE_URL-gated `test/e2e/op-checkpoint-jsonb-parity.test.ts` (its own CI job) asserts the array shape on real Postgres. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
+3 -2
View File
@@ -56,7 +56,8 @@ that tuple lights up the `pack_upgrade_available` onboard check.
│ gbrain onboard --check --explain shows per-cluster narrative │
│ User reviews; if OK, runs: │
│ gbrain jobs submit unify-types --allow-protected \ │
│ --params '{"target_pack":"gbrain-base-v2"}'
│ --params '{"target_pack":"gbrain-base-v2","apply":true}'
│ (omit "apply":true for a dry-run; that is the default) │
│ (Autopilot never auto-fires this; manual_only) │
└──────────────────────────┬─────────────────────────────────────┘
@@ -229,7 +230,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
+1 -1
View File
@@ -214,7 +214,7 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
v0.39-typed pages that no longer have a matching type in the active
pack.
+10 -10
View File
@@ -19,11 +19,13 @@ entire DB from scratch.
This means:
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Disaster recovery is a short, boring sequence.** If your DB volume
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
don't need a backup. You wipe the derived tables (on PGLite,
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
your brain repo with `gbrain sync`, and `gbrain extract all`
regenerates the derived state. See "Disaster recovery" below for the
exact commands.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
@@ -146,11 +148,9 @@ The promise the rule makes:
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
# Wipe and rebuild — delete the derived tables (pages + content_chunks
# survive the CASCADE-safe design), then re-derive from the repo.
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
+6 -4
View File
@@ -76,7 +76,8 @@ gbrain onboard --check --explain # per-cluster narrative dry-run
gbrain jobs submit unify-types \ # PROTECTED + manual_only
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
# omit "apply":true → dry-run (default)
Handler runs 4 phases:
┌─────────────────────────────────────┐
@@ -108,8 +109,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
@@ -127,7 +128,8 @@ For brains with substantial custom types that deserve their own canonical
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
--params '{"target_pack":"my-pack"}'`
--params '{"target_pack":"my-pack","apply":true}'` (omit `"apply":true`
for a dry-run preview — that is the default)
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
version: "1.x"}` to register itself as a successor — future agents
+1 -1
View File
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain install voice-agent`
`gbrain skillpack scaffold voice-agent`
That's it.
+1 -1
View File
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain add_timeline_entry {slug} {
gbrain timeline-add {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
+9 -9
View File
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
+9 -9
View File
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
gbrain call put_raw_data \
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
+5 -5
View File
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain get_links <attendee_slug>,
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
}
else:
briefing[attendee] = "No brain page -- consider enriching"
@@ -67,14 +67,14 @@ on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
gbrain timeline-add <sender_slug> {date} \
"Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
+7 -7
View File
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
gbrain timeline-add <person_slug> {date} \
"Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
gbrain timeline-add <company_slug> {date} \
"Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
---
+3 -3
View File
@@ -91,7 +91,7 @@ first):
6. The seeded `default` source.
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
the `gstack` source. Outside any registered directory with no env/dotfile
set, it writes to the default.
@@ -188,10 +188,10 @@ citations keep working.
```bash
# Pass --source explicitly
gbrain put-page topics/ai ... --source wiki
gbrain put topics/ai ... --source wiki
# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put-page plans/multi-repo ...
cd ~/.gstack && gbrain put plans/multi-repo ...
# → source auto-resolves to gstack
```
+8 -8
View File
@@ -20,8 +20,8 @@ on every_inbound_message(message):
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
# The brain COMPOUNDS overnight.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list_pages
pages = gbrain list
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain get_links <page.slug>
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
for mention in mentions:
if mention not in existing_links:
gbrain add_link <page.slug> <mention_slug> # fix broken graph
gbrain link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain get_timeline <page.slug>
timeline = gbrain timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+3 -3
View File
@@ -47,8 +47,8 @@ on user_message(message):
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
+1 -1
View File
@@ -87,7 +87,7 @@ expect it.
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Shown in future `gbrain plugin list`. |
| `description` | string | no | Shown in a future plugin-listing command. |
## Subagent definition files
+1 -1
View File
@@ -250,7 +250,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+1 -1
View File
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect``suggest``review-candidates` so the brain learns your shape instead of forcing you to learn its.
+1 -1
View File
@@ -554,7 +554,7 @@ What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
+9 -9
View File
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
## Step 6: Install GBrain
Once OpenClaw is running:
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
```bash
gbrain install
# In the BRAIN repo (the git repo that holds your markdown pages):
gbrain init --supabase
# In the AGENT WORKSPACE repo (where OpenClaw runs):
gbrain skillpack scaffold --all
```
This installs:
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
From this point, the agent has working memory and access to every skill.
---
+3 -3
View File
@@ -32,7 +32,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
+5 -5
View File
@@ -1730,7 +1730,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -2346,7 +2346,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -2376,7 +2376,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -2457,7 +2457,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
@@ -3927,7 +3927,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+1 -1
View File
@@ -147,7 +147,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.68.1",
"version": "0.42.69.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
+10 -5
View File
@@ -13,7 +13,8 @@
#
# Env overrides:
# SHARDS=N same as --shards
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 600)
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 1500)
# GBRAIN_TEST_SHARD_KILL_AFTER grace after TERM before KILL (default 30)
# GBRAIN_TEST_MAX_CONCURRENCY passed through to bun test (default 4)
#
# Output files (workspace-local; falls back to /tmp if .context/ unwritable):
@@ -79,6 +80,10 @@ INTRA_CONC="${MAX_CONCURRENCY_OVERRIDE:-${GBRAIN_TEST_MAX_CONCURRENCY:-4}}"
# 4-shard wallclock; real hangs still hit it. Override via
# GBRAIN_TEST_SHARD_TIMEOUT=N.
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-1500}"
SHARD_KILL_AFTER="${GBRAIN_TEST_SHARD_KILL_AFTER:-30}"
if ! printf '%s' "$SHARD_KILL_AFTER" | grep -qE '^[0-9]+$' || [ "$SHARD_KILL_AFTER" -lt 1 ]; then
echo "ERROR: invalid shard kill-after: $SHARD_KILL_AFTER" >&2; exit 2
fi
# ──────────────────────────────────────────────────────────────────────────
# Output directories. Prefer workspace-local .context/, fall back to /tmp.
@@ -109,7 +114,7 @@ elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
fi
START_TS=$(date +%s)
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | logs=$LOG_DIR" >&2
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | kill-after=${SHARD_KILL_AFTER}s | logs=$LOG_DIR" >&2
if [ "$DRY_RUN" = "1" ]; then
echo "[unit-parallel] dry-run: would spawn $N shards with the above settings."
@@ -129,7 +134,7 @@ for i in $(seq 1 "$N"); do
(
SHARD_LOG="$LOG_DIR/shard-$i.log"
if [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" "${SHARD_TIMEOUT}s" \
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${SHARD_TIMEOUT}s" \
env SHARD="$i/$N" \
bash scripts/run-unit-shard.sh --max-concurrency="$INTRA_CONC" \
> "$SHARD_LOG" 2>&1
@@ -140,7 +145,7 @@ for i in $(seq 1 "$N"); do
> "$SHARD_LOG" 2>&1 &
pid=$!
( sleep "$SHARD_TIMEOUT" && kill -TERM "$pid" 2>/dev/null && \
sleep 5 && kill -KILL "$pid" 2>/dev/null ) &
sleep "$SHARD_KILL_AFTER" && kill -KILL "$pid" 2>/dev/null ) &
cap_pid=$!
wait "$pid" 2>/dev/null
# Capture the shard's exit code from ITS `wait`, before any watchdog
@@ -158,7 +163,7 @@ for i in $(seq 1 "$N"); do
wait "$cap_pid" 2>/dev/null
fi
echo "$rc" > "$LOG_DIR/shard-$i.exit"
[ "$rc" = "124" ] && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
{ [ "$rc" = "124" ] || [ "$rc" = "137" ]; } && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
) &
SHARD_PIDS+=($!)
done
+1 -1
View File
@@ -248,7 +248,7 @@ before submission.
After the brain page is written, render to PDF using `skills/brain-pdf`:
```bash
gbrain put_page # already done by the CLI; nothing to add here
gbrain put # already done by the CLI; nothing to add here
# Then invoke brain-pdf:
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
```
+6 -6
View File
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
using `agentTurn`. Respect that. No auto-rewrite.
## Forward note (v0.12.0)
## Forward note
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
`gbrain jobs work` that owns cron expressions natively — no more
handing off to host schedulers. Until v0.12.0 lands, the host
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
layer (what the cron trigger *does*), not the scheduling layer.
A native scheduler loop inside `gbrain jobs work` (owning cron
expressions directly, with no host-scheduler hand-off) has been on the
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
firing on schedule; this convention only replaces the execution layer
(what the cron trigger *does*), not the scheduling layer.
## Related
+2 -1
View File
@@ -125,7 +125,8 @@ v0.41.22 ships **gbrain-base-v2** as the declared successor to
gbrain-base@1.x — collapses 94 noisy types to 15 canonical via
declarative mapping_rules. Run via `gbrain onboard --check --explain`
(preview) → `gbrain jobs submit unify-types --allow-protected --params
'{"target_pack":"gbrain-base-v2"}'` (apply). See
'{"target_pack":"gbrain-base-v2","apply":true}'` (apply — `apply`
defaults to false, so a bare submit is a dry run). See
`skills/schema-unify/SKILL.md` for the full playbook.
Authoring a successor pack: declare
+2 -2
View File
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
- Define a custom recipe with: source queries, classification rules, extraction schema,
tracker page path, tracker format
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
to scaffold a new one.
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
copying a built-in recipe file and editing its fields.
### Phase 2: Search Sources
+1 -1
View File
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
### 4b. Entity pages (people, companies)
For each entity mentioned:
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
- If exists: update State, append Timeline entry citing this research.
- If not: create with enrichment.
+1 -1
View File
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
# 4. Write the structured research page via put_page:
gbrain put_page research/<slug> # via the put_page operation
gbrain put research/<slug> # via the put_page operation
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
```
+9 -5
View File
@@ -11,7 +11,7 @@ tools:
- gbrain schema active
- gbrain schema use
- gbrain schema stats
- gbrain pages restore
- gbrain restore
- mcp:run_onboard
triggers:
- "unify my types"
@@ -90,9 +90,13 @@ The handler is PROTECTED (manual_only per D17) — autopilot will never auto-fir
```bash
gbrain jobs submit unify-types \
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
```
`apply` defaults to **false** (dry-run) per the handler contract, so
`"apply":true` is required here or the job reports success having retyped
nothing and left the active pack unflipped. Omit it to preview.
Watch progress per phase:
```bash
@@ -143,7 +147,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
```bash
gbrain pages restore <slug>
gbrain restore <slug>
```
Revert the active pack flip:
@@ -197,7 +201,7 @@ Outputs:
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
Side effects:
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
@@ -212,7 +216,7 @@ DON'T:
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
## Output Format
+6 -4
View File
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
The user sends an audio or voice message via any channel (Telegram, voice
memo upload, openclaw audio attachment). The host agent typically provides
the transcript text. If not, transcribe via `gbrain transcription` (Groq
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
the transcript text. If not, transcribe it with your host's transcription
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
audio > 25MB via ffmpeg first).
## The pipeline
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
1. STORE → Upload original audio to gbrain storage backend
(S3 / Supabase Storage / local — pluggable per
src/core/storage.ts).
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
gbrain transcription if no transcript was supplied.
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
transcribe the audio yourself (see "When to invoke")
if no transcript was supplied.
3. ROUTE → Apply the decision tree (below) to find the right
destination directory.
4. WRITE → Create / update the destination brain page; preserve the
+33 -2
View File
@@ -55,12 +55,17 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'backfill']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
const CLI_ONLY_SELF_HELP = new Set([
'upgrade', 'post-upgrade', 'check-update',
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
// bench-publish.ts printHelp). Both were documented but undispatchable —
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
// (the #2035 calibration bug class); `bench` was never wired at all.
'pages', 'bench',
'embed', 'config',
'skillpack', 'skillpack-check',
'integrations', 'friction',
@@ -107,6 +112,11 @@ const CLI_ONLY_SELF_HELP = new Set([
// `gbrain connect --help` prints its own usage (flags + examples) from
// runConnect; route around the generic one-line short-circuit.
'connect',
// `gbrain init --help` prints its own usage from runInit; route around the
// generic one-line short-circuit (matches `connect`). Without this, `init`
// is in CLI_ONLY but not CLI_ONLY_SELF_HELP, so the dispatcher's generic
// short-circuit fires and the printInitHelp() guard in init.ts is dead code.
'init',
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
// (engine transfer) keeps its own dispatch too.
@@ -1265,6 +1275,20 @@ async function handleCliOnly(command: string, args: string[]) {
await runInit(args);
return;
}
if (command === 'bench') {
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
if (args[0] === 'publish') {
const { runBenchPublish } = await import('./commands/bench-publish.ts');
await runBenchPublish(args.slice(1));
return;
}
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
console.error('Run `gbrain bench publish --help` for the full flag list.');
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
}
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
// Spawns its own engine internally so no pre-bound engine needed.
if (command === 'reinit-pglite') {
@@ -1827,7 +1851,14 @@ async function handleCliOnly(command: string, args: string[]) {
}
case 'embed': {
const { runEmbed } = await import('./commands/embed.ts');
await runEmbed(engine, args);
// #3037: mirror the `import` case above — the CLI was discarding the
// result, so a run where every chunk failed to embed still exited 0
// and cron/CI/health gates read total silence as success. Surface
// non-zero on failures > 0. (undefined = backgrounded via --background.)
const embedResult = await runEmbed(engine, args);
if (embedResult && embedResult.failures > 0) {
setCliExitVerdict(1);
}
break;
}
case 'serve': {
+108
View File
@@ -0,0 +1,108 @@
/**
* #1835 storage_path resolution for the doctor `image_assets` check.
*
* `files.storage_path` rows written by a Windows gbrain install carry Windows
* drive paths (`D:/foo/img.jpg`, `D:\foo\img.jpg`). On POSIX,
* `path.isAbsolute()` is false for those, so the old code joined them onto the
* repo root and produced a path that can never exist a false-positive
* "missing from disk, restore from git" WARN under WSL and macOS.
*
* Policy:
* - win32: drive paths are absolute; stat them as-is.
* - WSL (linux + "microsoft" in /proc/version): translate `D:/x` to
* `<automount root>/d/x` (automount root read from /etc/wsl.conf
* `[automount] root`, default `/mnt`) and stat that.
* - any other POSIX host (macOS, plain Linux): the path is unresolvable on
* this platform report it as foreign so the caller SKIPS the stat
* instead of inventing a path that will never exist.
*
* Kept in its own module (not doctor.ts) so the pure tests don't pull the
* 7k-line doctor dep graph, and so open PRs rewriting the image_assets block
* (e.g. a `resolveImageAssetPath` helper) can adopt it with a one-line call.
*/
import { readFileSync } from 'node:fs';
import { join, posix, win32 } from 'node:path';
const WINDOWS_DRIVE_RE = /^([A-Za-z]):[\\/](.*)$/;
export interface AssetPathResolution {
/** Absolute path to stat, or null when the path is unresolvable here. */
abs: string | null;
/** True when storage_path is a Windows drive path this host cannot stat. */
foreign: boolean;
}
/**
* Resolve a files.storage_path to a stat-able absolute path.
* `opts.platform` / `opts.wslMountRoot` exist for tests; production callers
* pass neither (process.platform + detected WSL automount root).
* `wslMountRoot: null` means "not under WSL".
*/
export function resolveAssetPath(
storagePath: string,
repoRoot: string,
opts: { platform?: NodeJS.Platform; wslMountRoot?: string | null } = {},
): AssetPathResolution {
const platform = opts.platform ?? process.platform;
if (platform !== 'win32') {
const m = WINDOWS_DRIVE_RE.exec(storagePath);
if (m) {
const root = opts.wslMountRoot !== undefined ? opts.wslMountRoot : detectWslMountRoot();
if (root === null) return { abs: null, foreign: true };
const abs = `${root.replace(/\/+$/, '')}/${m[1].toLowerCase()}/${m[2].replace(/\\/g, '/')}`;
return { abs, foreign: false };
}
}
// Platform-appropriate absoluteness (not the host's) so injected-platform
// tests behave identically everywhere; in production platform === host.
const isAbs = platform === 'win32' ? win32.isAbsolute(storagePath) : posix.isAbsolute(storagePath);
return {
abs: isAbs ? storagePath : join(repoRoot, storagePath),
foreign: false,
};
}
/**
* Extract the `[automount] root` value from /etc/wsl.conf content.
* Defaults to `/mnt` (WSL's own default) when absent/unparseable.
*/
export function parseWslAutomountRoot(conf: string): string {
let inAutomount = false;
for (const raw of conf.split(/\r?\n/)) {
const line = raw.replace(/[#;].*$/, '').trim();
if (line.startsWith('[')) {
inAutomount = /^\[automount\]$/i.test(line);
continue;
}
if (!inAutomount) continue;
const m = /^root\s*=\s*"?([^"]+?)"?\s*$/.exec(line);
if (m) return m[1];
}
return '/mnt';
}
let cachedWslMountRoot: string | null | undefined;
/**
* Detect the WSL Windows-drive automount root. Returns null when not running
* under WSL (including macOS and plain Linux). Memoized per process.
*/
export function detectWslMountRoot(): string | null {
if (cachedWslMountRoot === undefined) cachedWslMountRoot = computeWslMountRoot();
return cachedWslMountRoot;
}
function computeWslMountRoot(): string | null {
if (process.platform !== 'linux') return null;
try {
// The standard WSL tell: kernel version string names Microsoft.
if (!/microsoft/i.test(readFileSync('/proc/version', 'utf8'))) return null;
} catch {
return null;
}
try {
return parseWslAutomountRoot(readFileSync('/etc/wsl.conf', 'utf8'));
} catch {
return '/mnt'; // WSL default when wsl.conf is absent.
}
}
+22 -8
View File
@@ -6102,7 +6102,10 @@ export async function buildChecks(
} else {
// Live embed test
const start = Date.now();
const vec = await embedOne('gbrain doctor embedding smoke test');
// Doctor is itself the provider-health circuit breaker. A permanent
// billing/auth failure must be sampled once, not multiplied by the AI
// SDK's default retries (which can add ~90s to every health check).
const vec = await embedOne('gbrain doctor embedding smoke test', { maxRetries: 0 });
const ms = Date.now() - start;
const actualDims = vec.length;
@@ -7582,33 +7585,44 @@ export async function buildChecks(
`SELECT storage_path FROM files WHERE mime_type LIKE 'image/%' LIMIT 1000`
);
let vanished = 0;
let foreign = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
const { resolveAssetPath } = await import('./doctor-asset-paths.ts');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
// #1835: Windows drive paths (D:/…) translate to the WSL automount
// (/mnt/d/…) under WSL, and are SKIPPED (not "missing") on hosts
// where they cannot exist (macOS / plain Linux) — never joined onto
// repoRoot, which produced a false "restore from git" WARN.
const resolved = resolveAssetPath(r.storage_path, repoRoot);
if (resolved.abs === null) {
foreign++;
continue;
}
try {
fs.statSync(abs);
fs.statSync(resolved.abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
}
}
const checked = rows.length - foreign;
const foreignNote = foreign > 0
? ` (${foreign} Windows-drive path(s) skipped — not resolvable on this platform)`
: '';
if (rows.length === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: 'No image assets indexed yet' });
} else if (vanished === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: `${rows.length} image(s) all present on disk` });
checks.push({ name: 'image_assets', status: 'ok', message: `${checked} image(s) all present on disk${foreignNote}` });
} else {
checks.push({
name: 'image_assets',
status: 'warn',
message: `${vanished} of ${rows.length} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')}). ` +
message: `${vanished} of ${checked} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')})${foreignNote}. ` +
`Fix: restore from git, or \`gbrain sync --skip-failed\` to acknowledge.`,
});
}
+217 -36
View File
@@ -19,10 +19,26 @@ import {
} from '../core/pace-mode.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
import { AITransientError } from '../core/ai/errors.ts';
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
import type { Page } from '../core/types.ts';
/** #3037: cap failure samples so a corpus-wide outage doesn't bloat --json. */
const FAILURE_SAMPLE_CAP = 10;
/**
* #3037: record embed failures on the run result. `chunkCount` is the number
* of chunks left un-embedded by this failure (1 for page-level errors where
* the chunk count isn't known at the catch site).
*/
function recordFailure(result: EmbedResult, chunkCount: number, slug: string, e: unknown): void {
result.failures += chunkCount;
if (result.failure_samples.length < FAILURE_SAMPLE_CAP) {
result.failure_samples.push(`${slug}: ${e instanceof Error ? e.message : String(e)}`);
}
}
/**
* #3507 after a plain re-embed fully re-embedded a `per_chunk_synopsis`
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
@@ -166,6 +182,20 @@ export interface EmbedResult {
total_chunks: number;
/** Number of pages processed (whether or not they had stale chunks). */
pages_processed: number;
/**
* #3037: chunks that FAILED to embed this run (batch failures + per-chunk
* isolation failures). Callers must not read total silence as success:
* `src/cli.ts` turns `failures > 0` into a non-zero exit verdict (mirrors
* the `import` errors>0 guard), and structured consumers (--json, minion
* handlers) can surface it. 0 on a clean run. Additive field.
*/
failures: number;
/**
* #3037: up to 10 `slug: error-message` samples of what failed, so the
* operator gets a diagnosis without scrolling stderr. Capped so a
* corpus-wide outage doesn't bloat structured output. Additive field.
*/
failure_samples: string[];
/** True if this run was a dry-run. */
dryRun: boolean;
/**
@@ -284,6 +314,8 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
would_embed: 0,
total_chunks: 0,
pages_processed: 0,
failures: 0,
failure_samples: [],
dryRun: !!opts.dryRun,
};
@@ -293,6 +325,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal, opts.quiet);
} catch (e: unknown) {
if (isAborted(opts.signal)) break; // shutdown, not a failure
// #3037: a page-level error (not found, DB write) must not exit 0.
// Chunk-level embed failures are counted inside embedPage; this
// counts the page itself (chunk count unknown at this site).
recordFailure(result, 1, s, e);
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
}
@@ -535,6 +572,12 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
try {
const result = await runEmbedCore(engine, opts);
if (progressStarted) progress.finish();
// #3037: loud end-of-run summary so failures are visible even when the
// per-page stderr lines scrolled away. cli.ts turns failures>0 into a
// non-zero exit verdict.
if (result.failures > 0) {
serr(`[embed] ${result.failures} chunk(s) failed to embed. First error: ${result.failure_samples[0] ?? 'unknown'}`);
}
return result;
} catch (e) {
if (progressStarted) progress.finish();
@@ -623,10 +666,32 @@ async function embedPage(
// contextual prefix when the page was embedded wrapped), not raw
// chunk_text — otherwise a re-embed silently strips the contextual
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
// #3037: per-chunk failure isolation — one bad chunk must not leave the
// page's sibling chunks NULL. The wrapped texts (computed once) feed the
// fan-out too, so an isolation retry never strips the prefixes. Total
// embed failure is recorded here (where the chunk count is known) and
// swallowed: the page stays NULL exactly as before, but the run now
// reports it (result.failures → non-zero exit) instead of pretending
// success. Abort (shutdown) still propagates.
let embeddings: (Float32Array | null)[];
let failed = 0;
let firstError: unknown;
try {
({ embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
));
} catch (e: unknown) {
if (isAborted(signal)) throw e;
recordFailure(result, toEmbed.length, slug, e);
result.pages_processed++;
serr(` Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
return;
}
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
@@ -643,16 +708,21 @@ async function embedPage(
// Guard: only stamp when EVERY chunk was (re)embedded this pass. If some
// chunks were preserved from a prior embed (unknown/old provenance), the
// page is mixed — don't claim it's current. `embed --all` fully re-embeds
// such a page and then stamps it.
if (toEmbed.length === chunks.length) {
// such a page and then stamps it. #3037: a partial failure leaves failed
// chunks NULL, so don't stamp then either.
if (failed === 0 && toEmbed.length === chunks.length) {
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest.
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
}
result.embedded += toEmbed.length;
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(` ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
result.pages_processed++;
if (!quiet) slog(`${slug}: embedded ${toEmbed.length} chunks`);
if (!quiet) slog(`${slug}: embedded ${toEmbed.length - failed} chunks`);
}
/**
@@ -791,11 +861,18 @@ async function embedAll(
try {
// #3507: reproduce the page's stored wrapping convention (see embedPage).
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
// #3037: per-chunk failure isolation — one bad chunk costs one chunk,
// not the whole page's siblings. The wrapped texts feed the fan-out
// too, so an isolation retry never strips the contextual prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
);
// Build a map of new embeddings by chunk_index
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
@@ -809,17 +886,30 @@ async function embedAll(
}));
await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts));
// v0.41.31: stamp embedding provenance so a later model swap is
// detectable as stale.
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest.
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
result.embedded += toEmbed.length;
// detectable as stale. #3037: not on partial failure — failed chunks
// stay NULL under unknown provenance.
if (failed === 0) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest. #3037: gated on
// failed === 0 — a partially-failed page was NOT fully re-embedded,
// so restamping would make contextual_retrieval_mode lie again
// (the exact #3461 bug).
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
}
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, page.slug, firstError);
serr(`\n ${page.slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
} catch (e: unknown) {
// #3037: count the darkened page so the run can't exit 0 (abort is a
// shutdown, not a failure).
if (!isAborted(signal)) recordFailure(result, toEmbed.length, page.slug, e);
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
}
@@ -1037,10 +1127,8 @@ async function embedAllStale(
let afterUpdatedAt: string | null = null;
let totalChunksLoaded = 0;
let budgetExitNotified = false;
// #1946 (OV2a): track chunks that errored out so a catch-up pass that finishes
// with stale chunks still remaining (un-embeddable for a non-transient reason)
// surfaces that loudly instead of looking like a clean run.
let embedFailures = 0;
// #1946 (OV2a) + #3037: embed failures are tracked on result.failures so
// the catch-up warning below AND the CLI exit verdict both see them.
// E-3 (paced-backfill): bounded end-of-run re-entry. A longer paced run gives
// a live writer (sync / put_page) more time to insert NEW stale rows BEHIND
@@ -1137,12 +1225,19 @@ async function embedAllStale(
// NORMAL post-model-migration path, so raw-text embedding here
// quietly converted whole corpora to the unwrapped convention.
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
// #3037: per-chunk failure isolation — one bad chunk costs one
// chunk, not the whole page's siblings. The wrapped texts feed the
// fan-out too, so an isolation retry never strips the prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(pageRow, stale),
{ abortSignal: effectiveSignal },
);
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
const staleIdxToEmbedding = new Map<number, Float32Array>();
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) staleIdxToEmbedding.set(stale[j].chunk_index, emb);
}
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
@@ -1160,7 +1255,8 @@ async function embedAllStale(
// A partially-stale page keeps preserved chunks of unknown/old
// provenance, so don't claim it's current. (After invalidate, a
// signature-drifted page IS fully stale → this stamps it.)
if (signature && stale.length === existing.length) {
// #3037: not on partial failure — failed chunks stay NULL.
if (signature && failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
@@ -1168,17 +1264,24 @@ async function embedAllStale(
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest. Partially-stale pages
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
if (stale.length === existing.length) {
// #3037: `failed === 0` is part of "fully re-embedded" — if the
// per-chunk isolation left some chunks NULL, restamping would make
// contextual_retrieval_mode lie again (the exact #3461 bug).
if (failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
);
}
result.embedded += stale.length;
result.embedded += stale.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(`\n ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${stale.length - failed}`);
}
} catch (e: unknown) {
// Budget/abort-fired cancellations are expected on the way out; don't
// spam per-page "Error embedding" lines when we're shutting down.
if (effectiveSignal.aborted) return;
embedFailures++;
recordFailure(result, stale.length, slug, e);
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
totalProcessedPages++;
@@ -1231,14 +1334,14 @@ async function embedAllStale(
// chunks unembedded means those chunks are stuck (a non-transient embed
// failure), not that we ran out of time. Surface it loudly so it doesn't read
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
if (staleOpts?.catchUp && !effectiveSignal.aborted && result.failures > 0) {
const remaining = await engine.countStaleChunks(
signature
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
: (sourceId ? { sourceId } : undefined),
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${result.failures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
}
}
}
@@ -1358,12 +1461,7 @@ export async function embedBatchWithBackoff(
// If the budget fired we may have been aborted mid-fetch; bubble out.
if (signal?.aborted) throw e;
const msg = e instanceof Error ? e.message : String(e);
// D4: structured detection first (handles gateway-wrapped errors via
// cause chain); message-match as fallback for providers whose wrappers
// strip `cause.status`.
const isRateLimit = detect429FromCause(e)
|| /rate.?limit|429/i.test(msg);
if (!isRateLimit || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
if (!isRateLimitError(e) || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
const delayMs = parseRetryDelayMs(msg);
serr(` [rate-limit] attempt ${attempt + 1}/${MAX_RATE_LIMIT_RETRIES}, waiting ${delayMs}ms...`);
@@ -1373,3 +1471,86 @@ export async function embedBatchWithBackoff(
// Unreachable, but TypeScript needs it.
return embedBatch(texts);
}
/**
* 429 judgment shared by embedBatchWithBackoff (retry decision) and
* embedPageTexts (fan-out decision). D4: structured detection first
* (gateway-wrapped errors via cause chain); message-match as fallback for
* providers whose wrappers strip `cause.status`.
*/
function isRateLimitError(e: unknown): boolean {
const msg = e instanceof Error ? e.message : String(e);
return detect429FromCause(e) || /rate.?limit|429/i.test(msg);
}
/** Walk the cause chain (like detect429FromCause) for the first HTTP status. */
function statusFromCause(e: unknown): number | undefined {
let cur: unknown = e;
for (let depth = 0; depth < 5 && cur !== undefined && cur !== null; depth++) {
const obj = cur as { status?: unknown; statusCode?: unknown; cause?: unknown };
if (typeof obj.status === 'number') return obj.status;
if (typeof obj.statusCode === 'number') return obj.statusCode;
cur = obj.cause;
}
return undefined;
}
/**
* #3037: embed one page's chunk texts with per-chunk failure isolation.
*
* All three embed paths used to send a page's chunks in ONE
* embedBatch call, so one bad chunk (e.g. an oversized chunk the provider
* 400s) left EVERY sibling chunk NULL an ~8.6x blast radius. This wrapper
* tries the batch first (the cheap, common path), and only on a
* PERMANENT-looking batch failure retries once per chunk so one bad chunk
* costs one chunk.
*
* Cost bounding when we do NOT fan out (rethrow instead):
* - 429 / rate limit: embedBatchWithBackoff already retried with backoff;
* fanning out N single-chunk calls would hammer the same limiter N-fold.
* - AITransientError (5xx / network / unknown, per normalizeAIError): the
* batch CONTENT isn't the problem, so isolation can't help during an
* outage it would just multiply failing calls per page.
* - 401/403 (auth): nothing chunk-specific; every call would fail.
* When we DO fan out (permanent request-shaped 4xx like 400/413/422), the
* per-chunk pass happens at most ONCE per page per run and re-spends roughly
* the same tokens the failed batch would have bounded, no recursion. A
* fresh 429 arising DURING the fan-out still gets the normal backoff (each
* single-chunk call goes through embedBatchWithBackoff).
*
* Throws when nothing could be embedded (total failure same contract as
* the pre-#3037 single batch call). Returns `null` at the index of each
* failed chunk otherwise.
*/
async function embedPageTexts(
texts: string[],
opts: EmbedBatchWithBackoffOpts = {},
): Promise<{ embeddings: (Float32Array | null)[]; failed: number; firstError?: unknown }> {
try {
return { embeddings: await embedBatchWithBackoff(texts, opts), failed: 0 };
} catch (e: unknown) {
if (opts.abortSignal?.aborted) throw e; // shutdown, not a chunk problem
if (texts.length <= 1) throw e; // nothing to isolate
if (isRateLimitError(e) || e instanceof AITransientError) throw e;
const status = statusFromCause(e);
if (status === 401 || status === 403) throw e;
const embeddings: (Float32Array | null)[] = [];
let failed = 0;
let firstError: unknown;
for (const t of texts) {
try {
const single = await embedBatchWithBackoff([t], opts);
embeddings.push(single[0] ?? null);
if (single[0] === undefined) { failed++; firstError ??= e; }
} catch (chunkErr: unknown) {
if (opts.abortSignal?.aborted) throw chunkErr;
embeddings.push(null);
failed++;
firstError ??= chunkErr;
}
}
if (failed === texts.length) throw firstError ?? e; // total failure: pre-#3037 contract
return { embeddings, failed, firstError };
}
}
+41 -1
View File
@@ -268,6 +268,12 @@ export async function runImport(
let skipped = 0;
let errors = 0;
let processed = 0;
// Time-based checkpoint floor (see the save site below). Chunking cost scales
// with paragraph count, not bytes, so a single reference-style file can take
// many minutes; a count-only trigger leaves that work undurable.
const CHECKPOINT_MAX_INTERVAL_MS = 120_000;
let lastCheckpointMs = Date.now();
let lastCheckpointSize = completed.size;
let chunksCreated = 0;
const importedSlugs: string[] = [];
const errorCounts: Record<string, number> = {};
@@ -343,7 +349,17 @@ export async function runImport(
// Save checkpoint every 100 SUCCESSFUL adds (not every 100 processed).
// Failed files never enter `completed`, so a flaky file can't push the
// checkpoint past it — the next run will retry it.
if (completed.size > 0 && completed.size % 100 === 0) {
// ...and ALSO save on a time interval. On a corpus with an expensive tail
// `completed` can advance ~1 file per several minutes, so the next
// 100-boundary may be hours away; any kill before it discards every file
// since the last boundary and the run can never converge.
const nowMs = Date.now();
const dueByCount = completed.size > 0 && completed.size % 100 === 0;
const dueByTime = completed.size > lastCheckpointSize
&& nowMs - lastCheckpointMs >= CHECKPOINT_MAX_INTERVAL_MS;
if (dueByCount || dueByTime) {
lastCheckpointMs = nowMs;
lastCheckpointSize = completed.size;
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
try { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
@@ -429,6 +445,30 @@ export async function runImport(
}
}
// Final checkpoint save BEFORE the clear/preserve decision below. The
// periodic triggers above are gated on a 100-file boundary or an interval,
// so a run that ends between them would otherwise leave its tail unsaved.
// This must run before clearCheckpoint() so a clean run still ends with no
// checkpoint file — it only makes the ERROR path's preserved checkpoint
// complete.
if (errors > 0 && completed.size > lastCheckpointSize) {
try {
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
const { mkdirSync } = await import('fs');
mkdirSync(cpDir, { recursive: true });
}
saveCheckpoint(checkpointPath, {
schema_version: 1,
owner: 'gbrain',
kind: 'import',
dir,
completedPaths: Array.from(completed),
timestamp: new Date().toISOString(),
});
} catch { /* non-fatal: the next run simply redoes the tail */ }
}
// Clear checkpoint on clean completion. On error, the path-based checkpoint
// preserves only the successfully-completed paths, so the next run retries
// failed files automatically (they never entered `completed`).
+8 -3
View File
@@ -2194,8 +2194,9 @@ export async function registerBuiltinHandlers(
// migration that retypes 25K+ pages, creates alias rows, converts edge-
// shaped pages to link rows, AND flips the active pack at end of run.
// manual_only via src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS.
// Operator path: `gbrain jobs submit unify-types --allow-protected --params
// '{"target_pack":"gbrain-base-v2"}'`.
// Dry-run preview: `gbrain jobs submit unify-types --allow-protected
// --params '{"target_pack":"gbrain-base-v2"}'`; apply with
// '{"target_pack":"gbrain-base-v2","apply":true}'.
worker.register('unify-types', async (job) => {
const { runUnifyTypes } = await import('../core/schema-pack/unify-types-handler.ts');
const data = (job.data ?? {}) as {
@@ -2213,7 +2214,11 @@ export async function registerBuiltinHandlers(
} as unknown as import('../core/operations.ts').OperationContext;
return await runUnifyTypes(ctx, {
target_pack: data.target_pack,
apply: data.apply ?? true,
// #1575: default matches the handler interface's "Default false
// (dry-run)" — a destructive one-shot migration must be opted into
// with apply:true (the onboard remediation + the printed migration
// command both carry it explicitly).
apply: data.apply ?? false,
sourceId: data.sourceId,
onProgress: (msg: string) => {
job.updateProgress({ phase: 'unify-types', message: msg }).catch(() => {});
+62 -1
View File
@@ -34,6 +34,7 @@ import {
resolveModel,
type ModelTier,
} from '../core/model-config.ts';
import { resolveRecipe } from '../core/ai/model-resolver.ts';
const TIERS: ModelTier[] = ['utility', 'reasoning', 'deep', 'subagent'];
@@ -186,6 +187,44 @@ function classifyError(err: unknown): { status: ProbeStatus; message: string } {
return { status: 'unknown', message: msg };
}
const OPENAI_COMPAT_V1_HINT =
'If the API key is correct, the base URL may be missing the /v1 suffix. ' +
'OpenAI-shaped proxies (codex-proxy, Azure-OpenAI mirrors, LiteLLM fronting an OpenAI route) ' +
'serve /v1/chat/completions and 401 on the bare path. ' +
'Confirm with: `curl <base>/models` returns 200 with the same bearer, then append /v1 to the base URL.';
/**
* Fix-hint for the openai-compatible-proxy `/v1`-suffix trap.
*
* An OpenAI-shaped proxy whose base URL omits `/v1` (codex-proxy, some
* Azure-OpenAI mirrors, a LiteLLM proxy fronting an OpenAI-route backend)
* serves `/v1/chat/completions` and returns 401 on the bare `/chat/completions`
* the AI SDK appends to the base. `classifyError` reads that 401 as `auth` and
* points the operator at the bearer token, when the real fix is the URL shape.
*
* Returns the corrective hint only when the model routes through an
* openai-compatible recipe (proxy tier, not native anthropic/openai/google),
* `baseURL` is set, and `baseURL` does not already end in `/v1` (optionally with
* a trailing slash). Pure: recipe resolution is synchronous and does no
* network/engine work; any resolution failure returns undefined.
*
* @internal exported for tests.
*/
export function openAiCompatV1Hint(
modelStr: string,
baseURL: string | undefined | null,
): string | undefined {
if (!baseURL || !baseURL.trim()) return undefined;
if (/\/v1\/?$/.test(baseURL.trim())) return undefined;
try {
const { recipe } = resolveRecipe(modelStr);
if (recipe.tier !== 'openai-compat') return undefined;
return OPENAI_COMPAT_V1_HINT;
} catch {
return undefined;
}
}
/**
* Validate the configured embedding model + dims combo without spending tokens.
* Catches the bug class where a brain configured for Voyage with a missing or
@@ -523,7 +562,29 @@ async function probeModel(modelStr: string, touchpoint: 'chat' | 'expansion'): P
}
} catch (err) {
const { status, message } = classifyError(err);
return { model: modelStr, touchpoint, status, message, elapsed_ms: Date.now() - start };
const result: ProbeResult = { model: modelStr, touchpoint, status, message, elapsed_ms: Date.now() - start };
// An openai-compatible proxy whose base URL omits `/v1` returns 401 (not
// 404) on the bare `/chat/completions` path, which classifyError reads as
// `auth`. Attach the URL-shape hint so the operator doesn't chase the
// bearer token. Fail open: any error resolving the base URL yields no hint
// and never breaks the probe.
if (status === 'auth') {
try {
const { loadConfig } = await import('../core/config.ts');
const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts');
const fileCfg = loadConfig();
if (fileCfg) {
const cfg = buildGatewayConfig(fileCfg);
const { recipe } = resolveRecipe(modelStr);
const baseURL = cfg.base_urls?.[recipe.id] ?? recipe.base_url_default;
const hint = openAiCompatV1Hint(modelStr, baseURL);
if (hint) result.fix = hint;
}
} catch {
// fail open — no hint
}
}
return result;
}
}
+1 -1
View File
@@ -251,7 +251,7 @@ async function renderPackUpgradeExplain(
` Page-to-link: ${result.per_phase.page_to_link.would_convert} edges across ${result.per_phase.page_to_link.rules} rules\n` +
` Page-to-alias: ${result.per_phase.page_to_alias.would_alias} aliases across ${result.per_phase.page_to_alias.rules} rules\n` +
`\nRun the migration with:\n` +
` gbrain jobs submit unify-types --allow-protected --params '${JSON.stringify({ target_pack: targetPack })}'\n`,
` gbrain jobs submit unify-types --allow-protected --params '${JSON.stringify({ target_pack: targetPack, apply: true })}'\n`,
);
if (result.warnings.length > 0) {
process.stdout.write(`\nWarnings:\n`);
+13
View File
@@ -312,6 +312,19 @@ export function dimsProviderOptions(
// widths hard-fail with a dim-mismatch error. Pattern match the bare
// model name + any `:tag` (e.g. `qwen3-embedding:4b`, `qwen3-embedding:0.6b`).
if (modelId === 'qwen3-embedding' || modelId.startsWith('qwen3-embedding:')) {
// Only send `dimensions` when it actually differs from the model's
// native width. Fixed-dim OpenAI-compatible backends serving this
// family (e.g. vLLM) reject the parameter outright with HTTP 400
// ("does not support matryoshka representation") even when the
// requested value equals the native size; omitting it in the equal
// case is semantically identical for Ollama and keeps vLLM working.
const QWEN3_EMBEDDING_NATIVE_DIMS: Record<string, number> = {
'qwen3-embedding': 1024,
'qwen3-embedding:0.6b': 1024,
'qwen3-embedding:4b': 2560,
'qwen3-embedding:8b': 4096,
};
if (QWEN3_EMBEDDING_NATIVE_DIMS[modelId] === dims) return undefined;
return { openaiCompatible: { dimensions: dims } };
}
// MiniMax embo-01 takes a `type: 'db' | 'query'` field for asymmetric
+2 -2
View File
@@ -1951,8 +1951,8 @@ async function embedSubBatch(
}
/** Embed one text (convenience wrapper). */
export async function embedOne(text: string): Promise<Float32Array> {
const [v] = await embed([text]);
export async function embedOne(text: string, opts?: EmbedOpts): Promise<Float32Array> {
const [v] = await embed([text], opts);
return v;
}
+38
View File
@@ -156,6 +156,27 @@ const FREE_LOCAL_EMBED_PROVIDERS: ReadonlySet<string> = new Set([
'llama-server',
]);
/**
* Chat sibling of FREE_LOCAL_EMBED_PROVIDERS / FREE_LOCAL_RERANK_PROVIDERS.
*
* Local inference costs electricity, not tokens, so these providers price at
* $0 rather than TX2 hard-failing. Without this a caller that sets ANY cost cap
* cannot use a local chat model at all: CANONICAL_PRICING has no `ollama:*`
* keys, so `reserve()` throws no_pricing before the first call and every work
* item is skipped with `budget_exhausted: true` at $0 spent.
*
* That is not theoretical `cycle.extract_atoms` always constructs its tracker
* with `maxCostUsd` (config only accepts `n > 0`, so the cap can't be unset),
* which made `models.dream.extract_atoms: ollama:*` silently extract nothing.
*
* `litellm` is excluded on purpose, matching the embed set: a LiteLLM proxy can
* front a paid provider, so pricing-unknown is the honest state there.
*/
const FREE_LOCAL_CHAT_PROVIDERS: ReadonlySet<string> = new Set([
'ollama',
'llama-server',
]);
/**
* Look up `modelId` in the chat or embedding pricing maps. Returns a
* per-1M-token price tuple, or null when unknown.
@@ -220,9 +241,26 @@ function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null {
// above is only the bare-keyed Claude view.
const canon = canonicalLookup(modelId);
if (canon) return canon;
// Local-inference chat providers cost electricity, not tokens. Checked AFTER
// the canonical table so an explicitly-priced local entry, should one ever be
// added, still wins over the blanket zero.
if (kind === 'chat' && providerId && FREE_LOCAL_CHAT_PROVIDERS.has(providerId)) {
return { input: 0, output: 0 };
}
return null;
}
/**
* True when the budget tracker can price this model, i.e. when setting a cost
* cap is meaningful. Callers that apply a *default* cap (rather than one the
* user asked for) should skip the cap when this returns false otherwise
* `reserve()` hard-fails with BudgetExhausted(reason:'no_pricing') and the
* caller silently does no work.
*/
export function isModelPriceable(modelId: string, kind: BudgetKind): boolean {
return lookupPricing(modelId, kind) !== null;
}
function costForUsage(modelId: string, inputTokens: number, outputTokens: number, kind: BudgetKind): number | null {
const p = lookupPricing(modelId, kind);
if (!p) return null;
+164 -64
View File
@@ -27,6 +27,7 @@
*/
import type { BrainEngine } from './engine.ts';
import { CJK_SLUG_CHARS } from './cjk.ts';
import { stripCodeBlocks } from './link-extraction.ts';
/** D2: hardcoded entity types for v1. Pack-aware extension is TODO-1. */
@@ -105,18 +106,101 @@ export interface FindMentionsOpts {
// ============================================================
/**
* Token-only tokenizer. Returns `[token, offset]` pairs.
* The CJK character set this module treats as char-level, declared ONCE.
*
* ASCII: each `[a-zA-Z0-9]+` run is a single token, lowercased.
* CJK: each CJK character (Chinese/Japanese/Korean) is an individual
* token, lowercased. This allows the normal maximal-munch scan path
* to reach CJK gazetteer entries without a separate substring pass.
* `CJK_SLUG_CHARS` (src/core/cjk.ts) is the repo-wide single source of truth
* Han U+4E009FFF, Hiragana, Katakana, Hangul syllables and this module
* now uses it verbatim.
*
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
* run) single-word "Acme" lookup succeeds at offset 0; the trailing 's'
* is harmless noise.
* Note the deliberate behaviour change: the walkers here used to carry their
* own copy of the ranges that also covered Han Extension A (U+34004DBF),
* which cjk.ts scopes out repo-wide (see its header). Aligning on the shared
* constant means Ext-A characters are no longer treated as CJK by
* by-mention: they tokenize as word runs and, being a single sub-4-character
* token, an Ext-A-only entity title now falls below MIN_NAME_LENGTH instead
* of qualifying under MIN_CJK_NAME_LENGTH. Search, chunking and slug grammar
* already ignore Ext-A, so this makes by-mention consistent with them rather
* than being the one subsystem that disagrees.
*
* Everything below TOKEN_RE, hasCJK(), cjkCharCount() and the two
* per-character walkers derives from this one import. There are no copies
* of the ranges in this file.
*/
const TOKEN_RE = /[a-zA-Z0-9]+/g;
const CJK_CHAR_RE = new RegExp(`^[${CJK_SLUG_CHARS}]$`, 'u');
/**
* Conservative code-point bounds for CJK_SLUG_CHARS, derived from the range
* string itself (strip the `-` separators and the remaining characters are
* exactly the range endpoints) so they can never drift from it. Used only
* as a cheap pre-filter Latin/Vietnamese text short-circuits before the
* regex in the per-character walkers, which run over every body byte.
*/
const CJK_BOUNDS = ((): { min: number; max: number } => {
let min = 0x10ffff;
let max = 0;
for (const ch of CJK_SLUG_CHARS.replace(/-/g, '')) {
const cp = ch.codePointAt(0)!;
if (cp < min) min = cp;
if (cp > max) max = cp;
}
return { min, max };
})();
function isCJKChar(ch: string): boolean {
const cp = ch.codePointAt(0) ?? 0;
if (cp < CJK_BOUNDS.min || cp > CJK_BOUNDS.max) return false;
return CJK_CHAR_RE.test(ch);
}
/**
* Word-run tokenizer: a letter or ASCII digit, followed by any run of
* letters, ASCII digits and combining marks CJK excluded throughout, so
* CJK keeps flowing through the per-character path in the walkers below.
*
* Latin scripts with diacritics tokenize as whole words instead of
* fragmenting on every accented character "Nguyễn" is one token, not
* ["nguy","n"], and "Đà Nẵng" is ["đà","nẵng"], not ["n","ng"].
*
* Four deliberate boundaries, each of which was a real regression:
*
* - The LEAD must be a letter or digit, so a token can never consist of
* combining marks alone. U+FE0F (VARIATION SELECTOR-16, category Mn)
* rides on most emoji, so a mark-only token would hijack the gazetteer
* key of every emoji-prefixed entity title ("❤️ Health Notes" keying on
* U+FE0F instead of "health") and collapse all of them into one shared,
* mutually-confusable bucket.
* - Combining marks ARE allowed after the lead. NFD Vietnamese is base
* letter + mark, so excluding \p{M} would re-fragment the exact names
* this tokenizer exists to keep whole.
* - Digits are ASCII-only, exactly as the previous /[a-zA-Z0-9]+/ was.
* \p{N} would additionally mint tokens for ¹ ½ (Nl/No/non-ASCII Nd),
* and findMentionedEntities requires gazetteer tokens to be STRICTLY
* ADJACENT in the body so a superscript between the words of
* "Acme Corp" would silently break a match that used to work.
* - Plain `u` flag, not `v`: the CJK exclusion is a negative lookahead
* over CJK_SLUG_CHARS, the same construction src/core/think/gather.ts
* already uses. No es2024 target requirement, no set-subtraction syntax.
*/
const TOKEN_RE = new RegExp(
`(?![${CJK_SLUG_CHARS}])[\\p{L}0-9]` +
`(?:(?![${CJK_SLUG_CHARS}])[\\p{L}\\p{M}0-9])*`,
'gu',
);
/**
* Canonical form for a single token. NFC only canonical composition, no
* compatibility folding so an NFD body and an NFC gazetteer title produce
* the same token, while diacritics stay significant ("Hồng" still must not
* match "Hong").
*
* Applied PER TOKEN, never to the whole text: `Mention.offset` is contracted
* to index into the ORIGINAL body (extract-ner.ts slices a context window
* from it to infer the link verb), and normalizing the text up front would
* silently shift every offset.
*/
function normalizeToken(s: string): string {
return s.normalize('NFC').toLowerCase();
}
interface ScannedToken {
text: string; // lowercase
@@ -124,48 +208,64 @@ interface ScannedToken {
length: number; // original length (for span tracking)
}
function tokenizeForScan(text: string): ScannedToken[] {
/**
* Body-text tokenizer. Returns `[token, offset]` pairs.
*
* Word runs: each TOKEN_RE match is one token, NFC-normalized and
* lowercased. Covers ASCII and diacritic Latin scripts like Vietnamese
* ("Nguyễn" one token, not ["nguy","n"]).
* CJK: each CJK character (Chinese/Japanese/Korean) is an individual
* token. This allows the normal maximal-munch scan path to reach CJK
* gazetteer entries without a separate substring pass.
*
* `offset` and `length` index into the ORIGINAL string callers slice
* context windows out of the untouched body with them.
*
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
* run) single-word "Acme" lookup succeeds at offset 0; the trailing 's'
* is harmless noise.
*
* Exported so tests can assert on TOKENIZATION rather than only on the
* resolved mention (see tokenizeTitle).
*/
export function tokenizeForScan(text: string): ScannedToken[] {
const out: ScannedToken[] = [];
TOKEN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
// Collect ASCII token spans first.
const asciiSpans: Array<{ start: number; end: number }> = [];
// Collect word-run token spans first.
const wordSpans: Array<{ start: number; end: number }> = [];
while ((m = TOKEN_RE.exec(text)) !== null) {
asciiSpans.push({ start: m.index, end: m.index + m[0].length });
wordSpans.push({ start: m.index, end: m.index + m[0].length });
}
// Walk character-by-character: emit ASCII tokens at their start positions,
// then emit individual CJK characters for non-ASCII positions that fall
// outside ASCII token spans.
let asciiIdx = 0;
// Walk character-by-character: emit word-run tokens at their start
// positions, then emit individual CJK characters for positions that fall
// outside every word-run span.
let spanIdx = 0;
for (let i = 0; i < text.length;) {
const cp = text.codePointAt(i) ?? 0;
const isCJK = (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af);
// Advance asciiIdx past any spans that end before or at i.
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) {
asciiIdx++;
// Advance spanIdx past any spans that end before or at i.
while (spanIdx < wordSpans.length && wordSpans[spanIdx]!.end <= i) {
spanIdx++;
}
// If position i is inside an ASCII token span, emit the full ASCII token
// and jump past it.
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
const span = asciiSpans[asciiIdx]!;
// If position i is inside a word-run span, emit the full token and jump
// past it.
if (spanIdx < wordSpans.length && i >= wordSpans[spanIdx]!.start && i < wordSpans[spanIdx]!.end) {
const span = wordSpans[spanIdx]!;
const token = text.slice(span.start, span.end);
out.push({ text: token.toLowerCase(), offset: span.start, length: token.length });
out.push({ text: normalizeToken(token), offset: span.start, length: token.length });
i = span.end;
asciiIdx++;
spanIdx++;
continue;
}
// CJK: emit as individual character token.
if (isCJK) {
const charLen = cp > 0xffff ? 2 : 1; // surrogate pair
const charStr = text.slice(i, i + charLen);
out.push({ text: charStr.toLowerCase(), offset: i, length: charLen });
const cp = text.codePointAt(i) ?? 0;
const charLen = cp > 0xffff ? 2 : 1; // surrogate pair
const charStr = text.slice(i, i + charLen);
if (isCJKChar(charStr)) {
out.push({ text: normalizeToken(charStr), offset: i, length: charLen });
i += charLen;
} else {
i++;
@@ -176,10 +276,7 @@ function tokenizeForScan(text: string): ScannedToken[] {
function hasCJK(s: string): boolean {
for (const ch of s) {
const cp = ch.codePointAt(0) ?? 0;
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af)) return true;
if (isCJKChar(ch)) return true;
}
return false;
}
@@ -187,10 +284,7 @@ function hasCJK(s: string): boolean {
function cjkCharCount(s: string): number {
let count = 0;
for (const ch of s) {
const cp = ch.codePointAt(0) ?? 0;
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af)) count++;
if (isCJKChar(ch)) count++;
}
return count;
}
@@ -198,40 +292,46 @@ function cjkCharCount(s: string): number {
/**
* Tokenize a page title for gazetteer insertion.
*
* ASCII titles: standard `[a-zA-Z0-9]+` tokenization, lowercased.
* CJK titles (no ASCII content): split into individual characters
* Word-run titles: TOKEN_RE tokenization, NFC-normalized and lowercased
* ASCII plus diacritic Latin scripts (Vietnamese, etc.).
* CJK titles (no word-run content): split into individual characters
* e.g. "纳瓦尔" ["纳","瓦","尔"]. This allows normal multi-token
* maximal-munch matching to work with character-level CJK tokens
* produced by `tokenizeForScan`.
* Mixed CJK+ASCII titles: ASCII parts tokenized normally, CJK parts
* Mixed CJK+word-run titles: word-run parts tokenized normally, CJK parts
* split into individual characters.
*
* Exported so tests can assert on TOKENIZATION rather than only on the
* resolved mention a mention-only assertion passes even with a tokenizer
* that fragments the title and the body symmetrically.
*/
function tokenizeTitle(title: string): string[] {
export function tokenizeTitle(title: string): string[] {
const tokens: string[] = [];
TOKEN_RE.lastIndex = 0;
const hasAscii = TOKEN_RE.test(title);
if (hasAscii) {
// Mixed ASCII+CJK or pure ASCII: tokenize ASCII normally, then
// append individual CJK characters in order.
const hasWordRun = TOKEN_RE.test(title);
if (hasWordRun) {
// Mixed word-run+CJK or pure word-run: tokenize word runs normally,
// then append individual CJK characters in order.
TOKEN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
const asciiSpans: Array<{ start: number; end: number; text: string }> = [];
const wordSpans: Array<{ start: number; end: number; text: string }> = [];
while ((m = TOKEN_RE.exec(title)) !== null) {
asciiSpans.push({ start: m.index, end: m.index + m[0].length, text: m[0].toLowerCase() });
wordSpans.push({ start: m.index, end: m.index + m[0].length, text: normalizeToken(m[0]) });
}
let asciiIdx = 0;
let spanIdx = 0;
for (let i = 0; i < title.length;) {
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) asciiIdx++;
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
tokens.push(asciiSpans[asciiIdx]!.text);
i = asciiSpans[asciiIdx]!.end;
asciiIdx++;
while (spanIdx < wordSpans.length && wordSpans[spanIdx]!.end <= i) spanIdx++;
if (spanIdx < wordSpans.length && i >= wordSpans[spanIdx]!.start && i < wordSpans[spanIdx]!.end) {
tokens.push(wordSpans[spanIdx]!.text);
i = wordSpans[spanIdx]!.end;
spanIdx++;
continue;
}
const cp = title.codePointAt(i) ?? 0;
if (hasCJK(title[i]!)) {
const charLen = cp > 0xffff ? 2 : 1;
tokens.push(title.slice(i, i + charLen).toLowerCase());
const charLen = cp > 0xffff ? 2 : 1;
const charStr = title.slice(i, i + charLen);
if (isCJKChar(charStr)) {
tokens.push(normalizeToken(charStr));
i += charLen;
} else {
i++;
@@ -239,12 +339,12 @@ function tokenizeTitle(title: string): string[] {
}
return tokens;
}
// Pure CJK (no ASCII content): split into individual characters.
// Pure CJK (no word-run content): split into individual characters.
if (hasCJK(title)) {
for (let i = 0; i < title.length;) {
const cp = title.codePointAt(i) ?? 0;
const charLen = cp > 0xffff ? 2 : 1;
tokens.push(title.slice(i, i + charLen).toLowerCase());
tokens.push(normalizeToken(title.slice(i, i + charLen)));
i += charLen;
}
return tokens;
+55 -10
View File
@@ -362,8 +362,22 @@ export interface CycleReport {
* - 'failed' : lock acquired but all attempted phases failed
*/
status: CycleStatus;
/** Present when status = 'skipped'. E.g., 'cycle_already_running' or 'no_database'. Also 'aborted' when the cycle was cancelled mid-flight (#1972). */
/** Present when status = 'skipped'. E.g., 'cycle_already_running' or 'no_database'. Also 'aborted' when the cycle was cancelled mid-flight (#1972), or 'stamp_write_failed' (#3504). */
reason?: string;
/**
* #3504: the cycle ran, but persisting `last_source_cycle_at` /
* `last_full_cycle_at` threw. Set ONLY on a real write error never for a
* pack that merely omits optional phases (those come back 'skipped' and
* `deriveStatus` correctly ignores them).
*
* When present, `status` is degraded away from success, because a cycle that
* cannot record that it finished is not a cycle that finished as far as every
* downstream freshness reader is concerned. Before this existed the failure
* was a `console.warn` only: `dream --json` reported `status: 'ok'`, doctor
* separately reported `cycle_freshness` stale, and nothing connected the two,
* so re-running (the advice doctor gives) could never fix it.
*/
stamp_write_failed?: { source_id: string; error: string };
/**
* #1972: dead-holder sync/cycle locks the cycle-start reaper cleared this
* run (count + lock ids). Omitted when nothing was reaped or no engine.
@@ -1702,7 +1716,14 @@ export async function runCycle(
await pgliteFileLock!.refresh();
},
release: async () => {
try { await dbLock!.release(); } catch { /* fall through to file release */ }
try {
await dbLock!.release();
} catch (e) {
// #1470: best-effort, but never silent — a swallowed release
// failure strands a row in gbrain_cycle_locks and the next
// cycle skips with a phantom `cycle_already_running`.
console.error(`[cycle] DB lock release failed: ${e instanceof Error ? e.message : String(e)} — a row may remain in gbrain_cycle_locks until TTL expiry`);
}
await pgliteFileLock!.release();
},
}
@@ -2491,7 +2512,14 @@ export async function runCycle(
}
} finally {
if (lock) {
try { await lock.release(); } catch { /* best-effort */ }
try {
await lock.release();
} catch (e) {
// #1470: best-effort, but never silent — a swallowed release failure
// strands a row in gbrain_cycle_locks and the next cycle within the
// TTL skips with a phantom `cycle_already_running`.
console.error(`[cycle] lock.release() failed: ${e instanceof Error ? e.message : String(e)} — a row may remain in gbrain_cycle_locks until TTL expiry`);
}
}
}
@@ -2535,9 +2563,14 @@ export async function runCycle(
// - status is 'failed' or 'skipped' (don't mark a non-run as fresh)
// - dryRun (writes are out of scope)
//
// Best-effort: a write failure does NOT change the CycleReport status.
// The cost of writing the wrong timestamp post-failure is higher than
// the cost of missing a successful write (next cycle will redo work).
// #3504: the write is still best-effort in the sense that it never throws out
// of runCycle and never aborts the run (the phases already did their work).
// But a failure is no longer invisible: it is recorded on the report and
// degrades `status` away from success, so a cycle that could not persist its
// "done" stamp stops claiming it finished. The cost of writing the wrong
// timestamp post-failure is still higher than missing a successful write, so
// the stamp itself is unchanged — only the reporting is.
let stampWriteFailed: { source_id: string; error: string } | undefined;
if (opts.sourceId && engine && !dryRun && !aborted && (status === 'ok' || status === 'clean' || status === 'partial')) {
try {
const nowIso = new Date().toISOString();
@@ -2553,17 +2586,29 @@ export async function runCycle(
last_full_cycle_at: nowIso,
});
} catch (e) {
// Best-effort; cycle already succeeded by the time we get here.
console.warn(`[cycle] failed to write last_source_cycle_at for source ${opts.sourceId}: ${e instanceof Error ? e.message : String(e)}`);
const message = e instanceof Error ? e.message : String(e);
// Record it so `--json` consumers and the autopilot runner can see it.
// stderr alone does not survive a cron run, which is how #2251 stayed
// invisible while every stamp write failed for weeks.
stampWriteFailed = { source_id: opts.sourceId, error: message };
console.warn(`[cycle] failed to write last_source_cycle_at for source ${opts.sourceId}: ${message}`);
}
}
// #3504: a stamp-write failure degrades a successful run to 'partial'. It
// cannot upgrade or downgrade anything else: 'partial' is already non-success,
// and 'failed'/'skipped' never reach the stamp block at all. `aborted` still
// wins the reason slot, since an aborted run is the more fundamental fact.
const degradedByStamp = stampWriteFailed !== undefined && (status === 'ok' || status === 'clean');
const effectiveStatus: CycleStatus = aborted ? 'partial' : degradedByStamp ? 'partial' : status;
return {
schema_version: '1',
timestamp,
duration_ms,
status: aborted ? 'partial' : status,
...(aborted ? { reason: 'aborted' } : {}),
status: effectiveStatus,
...(aborted ? { reason: 'aborted' } : stampWriteFailed ? { reason: 'stamp_write_failed' } : {}),
...(stampWriteFailed ? { stamp_write_failed: stampWriteFailed } : {}),
...(reapedLocks ? { reaped_dead_holder_locks: reapedLocks } : {}),
brain_dir: opts.brainDir,
phases: phaseResults,
+19 -4
View File
@@ -56,8 +56,6 @@ async function loadConfig(engine: BrainEngine): Promise<AutoThinkConfig> {
const enabledStr = await engine.getConfig('dream.auto_think.enabled');
const questionsStr = await engine.getConfig('dream.auto_think.questions');
const maxPerStr = await engine.getConfig('dream.auto_think.max_per_cycle');
const budgetStr = await engine.getConfig('dream.auto_think.budget');
const cooldownStr = await engine.getConfig('dream.auto_think.cooldown_days');
const autoCommitStr = await engine.getConfig('dream.auto_think.auto_commit');
let questions: string[] = [];
@@ -68,16 +66,30 @@ async function loadConfig(engine: BrainEngine): Promise<AutoThinkConfig> {
} catch { /* ignore */ }
}
// getNumberConfig (not `parse* || N`) so a configured 0 is honored — a bare
// `|| N` coerces an explicit 0 back to the default (budget 0 = "spend nothing",
// cooldown 0 = "no cooldown"). max_per_cycle stays inline: its Math.max(1, ...)
// floor already makes 0 invalid there, so no configured value is lost.
const budgetUsd = Math.max(0, await getNumberConfig(engine, 'dream.auto_think.budget', 2.0));
const cooldownDays = Math.max(0, await getNumberConfig(engine, 'dream.auto_think.cooldown_days', 30));
return {
enabled: enabledStr === 'true',
questions,
maxPerCycle: maxPerStr ? Math.max(1, parseInt(maxPerStr, 10) || 5) : 5,
budgetUsd: budgetStr ? Math.max(0, parseFloat(budgetStr) || 2.0) : 2.0,
cooldownDays: cooldownStr ? Math.max(0, parseInt(cooldownStr, 10) || 30) : 30,
budgetUsd,
cooldownDays,
autoCommit: autoCommitStr === 'true',
};
}
async function getNumberConfig(engine: BrainEngine, key: string, fallback: number): Promise<number> {
const raw = await engine.getConfig(key);
if (raw === undefined || raw === null) return fallback;
const value = Number(raw);
return Number.isNaN(value) ? fallback : value;
}
async function isCoolingDown(engine: BrainEngine, days: number): Promise<boolean> {
if (days <= 0) return false;
const last = await engine.getConfig('dream.auto_think.last_completion_ts');
@@ -201,3 +213,6 @@ export async function runPhaseAutoThink(
duration_ms: Date.now() - start,
};
}
// Test-only export: pin config-resolution behavior at function granularity.
export const __testing = { loadConfig };
+25 -3
View File
@@ -52,7 +52,7 @@ import type { PhaseResult } from '../cycle.ts';
import type { GBrainConfig } from '../config.ts';
import type { ProgressReporter } from '../progress.ts';
import { chat as gatewayChat, withBudgetTracker } from '../ai/gateway.ts';
import { BudgetExhausted, BudgetTracker } from '../budget/budget-tracker.ts';
import { BudgetExhausted, BudgetTracker, isModelPriceable } from '../budget/budget-tracker.ts';
import { writeReceipt } from '../extract/receipt-writer.ts';
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
import { createHash } from 'crypto';
@@ -577,8 +577,21 @@ export async function runPhaseExtractAtoms(
} catch {
// Keep safe defaults: Haiku + $0.30.
}
// A cost cap is only meaningful for a model the tracker can price.
// BudgetTracker.reserve() hard-fails with BudgetExhausted(reason:'no_pricing')
// when the model is absent from the pricing maps AND a cap is set; with no cap
// it warns once and proceeds. Because this phase always set a cap, every
// non-Anthropic model tripped that hard-fail on the first item, latched
// `budgetExhausted`, and skipped the entire workload while reporting ok.
const priceable = isModelPriceable(extractModel, 'chat');
if (!priceable) {
console.error(
`[extract_atoms] model "${extractModel}" is not in the pricing maps; ` +
`running without a cost gate (a cap cannot be enforced on an unpriced model).`,
);
}
const budgetTracker = new BudgetTracker({
maxCostUsd: budgetCap,
maxCostUsd: priceable ? budgetCap : undefined,
label: 'cycle.extract_atoms',
});
@@ -754,7 +767,16 @@ export async function runPhaseExtractAtoms(
return {
phase: 'extract_atoms',
status: failures.length > 0 ? 'warn' : 'ok',
// A phase that skipped every work item and produced nothing did not
// succeed, even though skips are not failures and leave failures[] empty.
// Reporting 'ok' there hides a total no-op behind a green status.
status:
failures.length > 0 ||
(work.length > 0 &&
totalAtomsExtracted === 0 &&
transcriptsSkipped + pagesSkipped === work.length)
? 'warn'
: 'ok',
duration_ms: 0,
summary:
`extract_atoms: ${totalAtomsExtracted} atoms from ` +
+7 -4
View File
@@ -804,7 +804,6 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
// Explicit enabled=false still wins for pausing synthesis without removing corpus config.
const enabled = enabledRaw === 'false' ? false : (enabledRaw === 'true' || !!corpusDir);
const meetingTranscriptsDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir');
const minCharsStr = await engine.getConfig('dream.synthesize.min_chars');
const excludeStr = await engine.getConfig('dream.synthesize.exclude_patterns');
// v0.28: resolveModel() unifies CLI flag > new key > deprecated key > models.default > env > fallback
const { resolveModel } = await import('../model-config.ts');
@@ -820,7 +819,10 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
tier: 'utility',
fallback: 'haiku',
});
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
// getNumberConfig (not `parseInt(str, 10) || N`) so a configured 0 is honored — a bare
// `|| N` coerces an explicit 0 back to the default (cooldown 0 = "no cooldown").
const cooldownHours = Math.max(0, await getNumberConfig(engine, 'dream.synthesize.cooldown_hours', 12));
const minChars = Math.max(0, await getNumberConfig(engine, 'dream.synthesize.min_chars', 2000));
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
const subagentTimeoutMs = await getNumberConfig(
@@ -863,11 +865,11 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
enabled,
corpusDir: corpusDir ?? null,
meetingTranscriptsDir: meetingTranscriptsDir ?? null,
minChars: minCharsStr ? Math.max(0, parseInt(minCharsStr, 10) || 2000) : 2000,
minChars,
excludePatterns,
model,
verdictModel,
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
cooldownHours,
maxPromptTokens,
maxChunksPerTranscript,
outputRoot: await loadOutputRoot(engine),
@@ -1599,4 +1601,5 @@ export const __testing = {
stampDreamProvenance,
reverseWriteRefs,
runPgliteSubagentsInline,
loadSynthConfig,
};
+9 -13
View File
@@ -3,6 +3,7 @@ import { GBrainError, type EngineConfig } from './types.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import type { BrainEngine } from './engine.ts';
import { verifySchema } from './schema-verify.ts';
import { isRetryableConnError } from './retry-matcher.ts';
let sql: ReturnType<typeof postgres> | null = null;
let connectedUrl: string | null = null;
@@ -322,19 +323,14 @@ export async function withTransaction<T>(fn: (tx: ReturnType<typeof postgres>) =
}) as Promise<T>;
}
const RETRYABLE_DB_CONNECT_PATTERNS = [
/password authentication failed/i,
/connection refused/i,
/the database system is starting up/i,
/Connection terminated unexpectedly/i,
/ECONNRESET/i,
];
export function isRetryableDbConnectError(err: unknown): boolean {
const msg = err instanceof Error ? err.message : String(err);
if (!msg) return false;
return RETRYABLE_DB_CONNECT_PATTERNS.some(p => p.test(msg));
}
// issue #1720 (proposal 4): the startup connect matcher and the runtime
// matcher drifted — this used to be a private 5-pattern list that predated
// /connection.*closed/i and the CONNECTION_ENDED/CONNECTION_CLOSED codes, so
// a pooler close hitting connectWithRetry was treated as permanent. One
// canonical source now: retry-matcher.ts's isRetryableConnError (a strict
// superset of the old list). Do NOT reintroduce a local pattern list here;
// the agreement guard in test/worker-conn-resilience-1720.test.ts pins it.
export const isRetryableDbConnectError = isRetryableConnError;
export interface ConnectWithRetryOpts {
attempts?: number;
+16 -1
View File
@@ -74,6 +74,21 @@ export async function resolveEntitySlug(
}
// 4. Fallback: deterministic slugify.
return fallbackSlugify(trimmed);
}
/**
* #3447 shared fallback for both resolvers. slugify()'s `[^a-z0-9]+ → '-'`
* rule rewrites the path separator, so running slug-shaped input through it
* corrupts a well-formed slug (`people/alice-example` `people-alice-example`)
* into one no page can ever have and the flattened slug never resolves, so
* every re-extraction re-mints it. Path-shaped input is slugified PER SEGMENT
* (identity for already-well-formed slugs); display names keep plain slugify.
*/
function fallbackSlugify(trimmed: string): string {
if (trimmed.includes('/')) {
return trimmed.split('/').map(slugify).filter(Boolean).join('/');
}
return slugify(trimmed);
}
@@ -141,7 +156,7 @@ export async function resolveEntitySlugWithSource(
if (fuzzy) return { slug: fuzzy, source: 'fuzzy_match' };
}
return { slug: slugify(trimmed), source: 'fallback_slugify' };
return { slug: fallbackSlugify(trimmed), source: 'fallback_slugify' };
}
/**
+41
View File
@@ -219,10 +219,51 @@ export async function writeFactsToFence(
// 2. Upsert each fact onto the fence in input order. row_num
// monotonically increases (max-existing + 1 per call, append-only).
//
// Seed the counter from the DB as well as the fence file. Uniqueness
// is enforced by idx_facts_fence_key on
// (source_id, source_markdown_slug, row_num) in Postgres, but
// upsertFactRow derives the next value from the fence in the markdown
// alone — and falls back to 1 when the file has no fence at all. Any
// write path that rewrites a page without preserving its facts fence
// (put_page write-through, sync, dream-cycle reverse-render) therefore
// resets the counter below what the DB already holds, and the next
// absorb re-issues a row_num that is already taken. That surfaces as
// "duplicate key value violates unique constraint idx_facts_fence_key"
// and the whole batch of facts is dropped.
//
// Symptom in the wild: a page whose fence had been rewritten away had
// 24 facts in the DB and none in the file, so every subsequent absorb
// on it failed permanently. Taking the max of both sources keeps the
// file as the readable mirror while the DB stays authoritative about
// which row_nums have been issued.
//
// Degrades to the previous file-only behaviour if the lookup fails
// (pre-v51 brain without the fence columns, or a transient DB error):
// a fence write must not become impossible just because the counter
// hint is unavailable.
let dbMaxRowNum = 0;
try {
const rows = await engine.executeRaw<{ max_row_num: number | null }>(
`SELECT MAX(row_num) AS max_row_num FROM facts
WHERE source_id = $1 AND source_markdown_slug = $2`,
[target.sourceId, target.slug],
);
dbMaxRowNum = Number(rows[0]?.max_row_num ?? 0);
} catch {
dbMaxRowNum = 0;
}
const { facts: existingFenceFacts } = parseFactsFence(body);
const fileMaxRowNum = existingFenceFacts.length > 0
? Math.max(...existingFenceFacts.map(f => f.rowNum))
: 0;
let nextRowNum = Math.max(fileMaxRowNum, dbMaxRowNum) + 1;
const assignedRowNums: number[] = [];
for (const f of facts) {
const validFromStr = (f.validFrom ?? new Date()).toISOString().slice(0, 10);
const { body: updated, rowNum } = upsertFactRow(body, {
rowNum: nextRowNum++,
claim: f.fact,
kind: (f.kind ?? 'fact') as 'fact' | 'event' | 'preference' | 'commitment' | 'belief',
confidence: f.confidence ?? 1.0,
+43 -1
View File
@@ -295,18 +295,32 @@ export class MinionWorker extends EventEmitter {
// so a stalled job (lock_until expired) gets requeued before handleTimeouts'
// `lock_until > now()` guard would skip it. Stall → retry, timeout → dead.
const stalledTimer = setInterval(async () => {
// issue #1720: a dead pool used to spray "Stall detection error: write
// CONNECTION_CLOSED ..." every tick forever — this interval was the only
// background loop without the #1491-style reconnect. Rebuild the
// worker-owned pool AT MOST ONCE per tick, shared across the three
// sweeps: a dead pool fails all three, and one rebuild is enough (three
// back-to-back connect attempts would just add pooler pressure).
let reconnectedThisTick = false;
const recoverConnection = async (site: string, e: unknown): Promise<void> => {
if (reconnectedThisTick || !isRetryableConnError(e)) return;
reconnectedThisTick = true;
await this.reconnectAfterConnectionError(site, e);
};
try {
const { requeued, dead } = await this.queue.handleStalled();
if (requeued.length > 0) console.log(`Stall detector: requeued ${requeued.length} jobs`);
if (dead.length > 0) console.log(`Stall detector: dead-lettered ${dead.length} jobs`);
} catch (e) {
console.error('Stall detection error:', e instanceof Error ? e.message : String(e));
await recoverConnection('handleStalled', e);
}
try {
const timedOut = await this.queue.handleTimeouts();
if (timedOut.length > 0) console.log(`Timeout detector: dead-lettered ${timedOut.length} jobs (timeout exceeded)`);
} catch (e) {
console.error('Timeout detection error:', e instanceof Error ? e.message : String(e));
await recoverConnection('handleTimeouts', e);
}
try {
const wallClockTimedOut = await this.queue.handleWallClockTimeouts(this.opts.lockDuration);
@@ -315,6 +329,7 @@ export class MinionWorker extends EventEmitter {
}
} catch (e) {
console.error('Wall-clock timeout detection error:', e instanceof Error ? e.message : String(e));
await recoverConnection('handleWallClockTimeouts', e);
}
}, this.opts.stalledInterval);
@@ -1125,7 +1140,34 @@ export class MinionWorker extends EventEmitter {
attempts_made: job.attempts_made + 1,
}) : 0;
const failed = await this.queue.failJob(job.id, lockToken, errorText, newStatus, backoffMs);
// issue #1720: failJob can itself throw during the same DB outage that
// failed the job. Pre-fix the rejection escaped to launchJob's .catch
// and the ORIGINAL job error was never logged anywhere — the recording
// error masked it. Log the original FIRST (it must survive no matter
// what), then reconnect + retry the recording once. If it still fails,
// leave the row to the stall detector: the lock has stopped renewing,
// so handleStalled requeues it cleanly on a live pool (the D8a path).
let failed: MinionJob | null;
try {
failed = await this.queue.failJob(job.id, lockToken, errorText, newStatus, backoffMs);
} catch (recordErr) {
const recordMsg = recordErr instanceof Error ? recordErr.message : String(recordErr);
console.error(
`Job ${job.id} (${job.name}) failed with: ${errorText} — and recording the failure threw: ${recordMsg}`,
);
if (!isRetryableConnError(recordErr)) throw recordErr;
await this.reconnectAfterConnectionError('failJob', recordErr);
try {
failed = await this.queue.failJob(job.id, lockToken, errorText, newStatus, backoffMs);
} catch (retryErr) {
console.error(
`Job ${job.id} (${job.name}) failure-recording retry also failed ` +
`(${retryErr instanceof Error ? retryErr.message : String(retryErr)}); ` +
`leaving the row for the stall detector to requeue after lock expiry`,
);
return;
}
}
if (!failed) {
console.warn(`Job ${job.id} failure dropped (lock token mismatch)`);
return;
+3 -1
View File
@@ -426,7 +426,9 @@ export async function checkPackUpgradeAvailable(
makeRemediationStep({
id: 'onboard.pack_upgrade_' + successor.manifest.name,
job: 'unify-types',
params: { target_pack: successor.manifest.name },
// #1575: the worker defaults `apply` to false (dry-run); a
// remediation step is a consented apply, so carry it explicitly.
params: { target_pack: successor.manifest.name, apply: true },
severity: 'medium',
est_seconds: 600, // ~10min on 186K-page brain (production proxy)
est_usd_cost: 0, // pure SQL; no LLM spend
+6
View File
@@ -2761,12 +2761,18 @@ const sync_brain: Operation = {
localOnly: true,
handler: async (ctx, p) => {
const { performSync } = await import('../commands/sync.ts');
// #2830: thread ctx.sourceId (D7 pattern, same as revert_version /
// put_page) so a no-`repo` call resolves the CALLER's sync anchor.
// Without it, performSync read the default source's repo_path/last_commit
// and silently synced against the wrong repo on multi-source brains.
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
return performSync(ctx.engine, {
repoPath: p.repo as string | undefined,
dryRun: ctx.dryRun || (p.dry_run as boolean) || false,
noEmbed: (p.no_embed as boolean) || false,
noPull: (p.no_pull as boolean) || false,
full: (p.full as boolean) || false,
...sourceOpts,
});
},
cliHints: { name: 'sync', hidden: true },
+11 -2
View File
@@ -2407,7 +2407,7 @@ export class PGLiteEngine implements BrainEngine {
}
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
// embedded_at must reset to NULL so 'embed --stale' correctly picks up the row for re-embedding.
// See postgres-engine.ts upsertChunks for the full rationale — pglite mirrors it for parity.
//
// v0.40.3.0 D24 NULL→non-NULL race fix mirrors postgres-engine.ts. Two writers
@@ -5392,7 +5392,16 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
-- Parity with postgres-engine.ts: same predicate as
-- buildStaleChunkWhere / countStaleChunks, i.e. what 'embed --stale'
-- actually processes. 'embedding IS NULL' (not embedded_at, which can
-- be non-NULL while embedding is NULL) and embed_skip excluded, so the
-- count can reach zero and the embed.stale remediation can converge.
(SELECT count(*) FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT jsonb_exists(COALESCE(p.frontmatter, '{}'::jsonb), 'embed_skip')
) as missing_embeddings,
(SELECT count(*) FROM links) as link_count,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
+22 -4
View File
@@ -2527,15 +2527,15 @@ export class PostgresEngine implements BrainEngine {
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL.
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
// embedded_at must reset to NULL so 'embed --stale' correctly picks up the row for re-embedding.
// Without this, embedded_at lies (says "embedded" while embedding=NULL), and any staleness
// predicate on embedded_at would silently skip the row. This is why the egress fix predicates
// on `embedding IS NULL` rather than `embedded_at IS NULL` — and it's why we now keep both
// on 'embedding IS NULL' rather than `embedded_at IS NULL` — and it's why we now keep both
// columns honest at write time.
//
// v0.40.3.0 D24 NULL→non-NULL race fix (TODOS.md v0.35.x item).
// Two writers racing on the same chunk (e.g., autopilot sync + manual
// `embed --stale` + contextual reindex) previously raced last-write-wins
// 'embed --stale' + contextual reindex) previously raced last-write-wins
// via `COALESCE(EXCLUDED.embedding, content_chunks.embedding)`. With
// per-chunk Haiku synopsis the cost of an overwrite jumped from
// ~$0.000001 to ~$0.0003. New rule for the text-unchanged branch:
@@ -5489,7 +5489,25 @@ export class PostgresEngine implements BrainEngine {
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
-- missing_embeddings uses the same predicate as the thing that
-- resolves it: buildStaleChunkWhere / countStaleChunks, i.e. what
-- 'embed --stale' actually processes. Two divergences existed:
-- 1. embedded_at vs embedding. upsertChunks resets BOTH to NULL
-- when chunk_text changes, but the stale-chunk predicate keys
-- on 'embedding IS NULL' deliberately (see the CONSISTENCY note
-- on that upsert) because embedded_at can be non-NULL while
-- embedding is NULL. Health should agree with the embedder.
-- 2. embed_skip pages were counted here but excluded there, so
-- chunks the author opted out of read as permanently "missing"
-- and the count could never reach zero.
-- Effect of the mismatch: computeRecommendations emits an embed.stale
-- step from a number that 'embed --stale' reports as 0, so the step
-- cannot move it and 'doctor --remediate' re-plans it every pass.
(SELECT count(*) FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT jsonb_exists(COALESCE(p.frontmatter, '{}'::jsonb), 'embed_skip')
) as missing_embeddings,
(SELECT count(*) FROM links) as link_count,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
+5
View File
@@ -112,6 +112,11 @@ export function isRetryableConnError(err: unknown): boolean {
// v0.42.5.0 (issue #1678): postgres.js's library-level connection-ended
// code. Not an 08xxx SQLSTATE, so the /^08/ test above misses it.
if (code === 'CONNECTION_ENDED') return true;
// issue #1720: postgres.js also throws code 'CONNECTION_CLOSED' when the
// pooler closes the socket mid-query ("write CONNECTION_CLOSED host:port").
// The message form is already caught by /connection.*closed/i below; match
// the code too for wrappers that rethrow with the code but a new message.
if (code === 'CONNECTION_CLOSED') return true;
// v0.42.x (#1794): SQLSTATE 53300 too_many_connections — pool/pooler
// exhaustion. Starts with 53 not 08, so the /^08/ test above misses it.
// Transient: the spike clears as in-flight queries release connections.
+38 -11
View File
@@ -9,6 +9,7 @@
import { existsSync, readFileSync, statSync, readdirSync } from 'fs';
import { join, dirname, isAbsolute, resolve } from 'path';
import { fileURLToPath } from 'url';
import { parseMarkdown } from '../markdown.ts';
@@ -38,19 +39,45 @@ export class BundleError extends Error {
/**
* Walk up from `start` (default cwd) looking for an `openclaw.plugin.json`
* sibling to `src/cli.ts`. That pair identifies a gbrain repo root.
*
* When no explicit `start` is given and the cwd walk fails (e.g. gbrain was
* installed globally via `bun install -g` and the user is in an unrelated
* directory, #1917), fall back to walking up from this module's own location
* and from the running entrypoint (`process.argv[1]`). Both resolve the
* bun-global layout (~/.bun/install/global/node_modules/gbrain/) and the
* in-repo compiled binary (bin/gbrain).
*/
export function findGbrainRoot(start: string = process.cwd()): string | null {
let dir = resolve(start);
for (let i = 0; i < 10; i++) {
if (
existsSync(join(dir, 'openclaw.plugin.json')) &&
existsSync(join(dir, 'src', 'cli.ts'))
) {
return dir;
export function findGbrainRoot(start?: string): string | null {
const walkUp = (from: string): string | null => {
let dir = resolve(from);
for (let i = 0; i < 10; i++) {
if (
existsSync(join(dir, 'openclaw.plugin.json')) &&
existsSync(join(dir, 'src', 'cli.ts'))
) {
return dir;
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
return null;
};
const found = walkUp(start ?? process.cwd());
if (found !== null || start !== undefined) return found;
const fallbacks: string[] = [];
try {
// Not a file:// URL inside a compiled binary; skip on error.
fallbacks.push(dirname(fileURLToPath(import.meta.url)));
} catch {
/* ignore */
}
if (process.argv[1]) fallbacks.push(dirname(resolve(process.argv[1])));
for (const candidate of fallbacks) {
const root = walkUp(candidate);
if (root !== null) return root;
}
return null;
}
+88
View File
@@ -271,3 +271,91 @@ describe('buildGatewayConfig env empty-string clobber guard (#1249)', () => {
);
});
});
/**
* Side-effect guard (v0.37.x): importing buildGatewayConfig from src/cli.ts
* must NOT trigger the CLI's top-level main() and dump help to stdout. The
* helper is exported specifically so test/agent/daemon consumers can call
* it as a library the import side effect was historically the loudest
* source of test-runner noise. Wrap is in src/cli.ts: `if (import.meta.main)`
* around the `main().catch(...)` invocation.
*/
import { spawnSync } from 'child_process';
describe('buildGatewayConfig import side effect guard', () => {
/**
* Spawn `bun run src/cli.ts --help` as a subprocess. Process-global stdout
* capture avoids any contamination from the test runner's own TTY hooks.
* The CLI dispatcher MUST print help when invoked as the entry point
* (no side-effect regression). When main() accidentally fires during an
* import, this help text is what ends up leaking into test output.
*/
function runCliHelp(): { stdout: string; stderr: string; status: number | null } {
const result = spawnSync(
'bun',
['run', 'src/cli.ts', '--help'],
{
cwd: import.meta.dir + '/../..',
encoding: 'utf8',
timeout: 30_000,
},
);
return {
stdout: result.stdout ?? '',
stderr: result.stderr ?? '',
status: result.status,
};
}
test('direct CLI entry --help still prints help to stdout (regression guard)', () => {
const { stdout, status } = runCliHelp();
expect(status).toBe(0);
expect(stdout).toContain('gbrain');
expect(stdout.toLowerCase()).toMatch(/usage|commands|search|init/);
});
test('importing buildGatewayConfig does NOT trigger main() (no help on stdout)', async () => {
// The helper import at the top of this file is the unit under test — it
// already ran by the time describe() executes. Re-importing here is
// belt-and-suspenders: any future test that splits the suite would still
// exercise the side-effect contract from a fresh module record.
const mod = await import('../../src/cli.ts');
expect(typeof mod.buildGatewayConfig).toBe('function');
// Build a synthetic config and call it. Pre-fix behavior: importing the
// module executed `main()` which read argv and called printHelp(). The
// printHelp output landed on stdout during test bootstrap — observable
// as a leading "gbrain" banner before the test runner's own output.
// Post-fix: the import is silent; only the call below produces output,
// and the helper itself writes nothing.
const cfg = mod.buildGatewayConfig({} as unknown as GBrainConfig);
expect(cfg).toBeDefined();
});
test('subprocess importing buildGatewayConfig sees no CLI help on stdout', () => {
// Independent subprocess so the test runner's own process state cannot
// mask a leak. The spawned bun evaluates the same import the test file
// does, then exits. Pre-fix: stdout includes the help banner (and the
// process would exit 0 because main() returns normally after printHelp()).
// Post-fix: stdout is empty; only the bun runtime header / warnings may
// appear on stderr.
const inline = `
import { buildGatewayConfig } from './src/cli.ts';
const cfg = buildGatewayConfig({});
// Touch the result so the engine does not dead-code-eliminate the call.
if (!cfg) process.exit(2);
`;
const result = spawnSync(
'bun',
['--eval', inline],
{
cwd: import.meta.dir + '/../..',
encoding: 'utf8',
timeout: 30_000,
},
);
expect(result.status).toBe(0);
expect(result.stdout).not.toMatch(/usage|commands available|gbrain v?\\d/);
});
});
+40
View File
@@ -0,0 +1,40 @@
/**
* Qwen3-Embedding native-width dims suppression tests.
*
* Pins:
* - dimsProviderOptions returns undefined when the configured dim equals
* the model's native width (1024/2560/4096 for 0.6B/4B/8B) fixed-dim
* OpenAI-compatible backends serving this family (e.g. vLLM) reject the
* `dimensions` parameter with HTTP 400 "does not support matryoshka
* representation" even when the value equals the native size, and
* omitting it in the equal case is a no-op for Ollama.
* - Matryoshka truncation is preserved: a dim that differs from the native
* width still emits { openaiCompatible: { dimensions } } for Ollama.
*/
import { describe, test, expect } from 'bun:test';
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
describe('qwen3-embedding native-width suppression', () => {
test('bare model at native 1024 emits no dimensions param', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 1024)).toBeUndefined();
});
test('tagged variants at their native width emit no dimensions param', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:0.6b', 1024)).toBeUndefined();
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:4b', 2560)).toBeUndefined();
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:8b', 4096)).toBeUndefined();
});
test('non-native dim still requests Matryoshka truncation', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 512))
.toEqual({ openaiCompatible: { dimensions: 512 } });
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:4b', 1024))
.toEqual({ openaiCompatible: { dimensions: 1024 } });
});
test('unknown tag falls through to sending the configured dim', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:32b', 1024))
.toEqual({ openaiCompatible: { dimensions: 1024 } });
});
});
+35 -1
View File
@@ -5,6 +5,8 @@ import {
__unconfigureGatewayForTests,
isAvailable,
embed,
embedOne,
__setEmbedTransportForTests,
getEmbeddingModel,
getEmbeddingDimensions,
getExpansionModel,
@@ -17,7 +19,10 @@ import {
// (capture / ingest-capture tests), where it produced "Incorrect API key
// provided: openai-fake" against the real OpenAI endpoint and wedged
// the shard. Reset once at file teardown so no caller sees the residue.
afterAll(() => resetGateway());
afterAll(() => {
resetGateway();
__setEmbedTransportForTests(null);
});
import { parseModelId, resolveRecipe } from '../../src/core/ai/model-resolver.ts';
import {
dimsProviderOptions,
@@ -52,6 +57,35 @@ describe('gateway configuration', () => {
});
});
describe('gateway.embedOne options', () => {
beforeEach(() => {
resetGateway();
__setEmbedTransportForTests(null);
});
test('passes maxRetries=0 to the provider transport for health probes', async () => {
let observedMaxRetries: number | undefined;
configureGateway({
embedding_model: 'google:gemini-embedding-001',
embedding_dimensions: 3,
env: { GOOGLE_GENERATIVE_AI_API_KEY: 'fake-google' },
});
__setEmbedTransportForTests(async (args: any) => {
observedMaxRetries = args.maxRetries;
return {
embeddings: [new Array(3).fill(0.1)],
usage: { tokens: 1 },
} as any;
});
const vector = await embedOne('health probe', { maxRetries: 0 });
expect(observedMaxRetries).toBe(0);
expect(vector.length).toBe(3);
__setEmbedTransportForTests(null);
});
});
describe('gateway.isAvailable (silent-drop regression surface)', () => {
beforeEach(() => resetGateway());
+6 -2
View File
@@ -28,8 +28,12 @@ describe('dims: ollama Matryoshka models', () => {
});
test('bare qwen3-embedding (no quant tag) also recognized', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 1024))
.toEqual({ openaiCompatible: { dimensions: 1024 } });
// 512 not 1024: bare qwen3-embedding's native width is 1024, and a
// request equal to the native width omits the param entirely (fixed-dim
// vLLM backends 400 on it) — pinned by dims-qwen3-native.test.ts. This
// test's job is only that the bare id is recognized as Matryoshka-capable.
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 512))
.toEqual({ openaiCompatible: { dimensions: 512 } });
});
test('unrelated openai-compat model returns undefined (regression guard)', () => {
@@ -0,0 +1,58 @@
import { describe, test, expect } from 'bun:test';
import { BudgetTracker, BudgetExhausted } from '../../src/core/budget/budget-tracker.ts';
/**
* Regression guard for a silent-zero-yield bug: `cycle.extract_atoms` always
* constructs its BudgetTracker with a cap (config only accepts `n > 0`, so it
* cannot be unset). Local chat models have no CANONICAL_PRICING entry, so TX2
* hard-failed `no_pricing` before the first call and every page was skipped
* with `budget_exhausted: true` at $0 spent extraction reported success and
* produced nothing.
*/
const est = (modelId: string, kind: 'chat' | 'embed' | 'rerank' = 'chat') => ({
modelId, kind, estimatedInputTokens: 12_000, maxOutputTokens: 4096,
});
describe('free local chat providers under a cost cap', () => {
test('ollama chat reserves at $0 instead of hard-failing', () => {
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
expect(() => t.reserve(est('ollama:gemma4:26b'))).not.toThrow();
expect(t.totalSpent).toBe(0);
});
test('llama-server chat also reserves at $0', () => {
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
expect(() => t.reserve(est('llama-server:qwen3-32b'))).not.toThrow();
});
test('a genuinely unpriced remote provider still hard-fails (TX2 intact)', () => {
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
let err: unknown;
try { t.reserve(est('some-unknown-vendor:mystery-model')); } catch (e) { err = e; }
expect(err).toBeInstanceOf(BudgetExhausted);
expect((err as BudgetExhausted).reason).toBe('no_pricing');
});
test('litellm is NOT free — a proxy can front a paid provider', () => {
// Mirrors the embed set's deliberate exclusion. Pricing-unknown is the
// honest state for a proxy, so the cap must still hard-fail.
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
expect(() => t.reserve(est('litellm:gpt-5.4'))).toThrow(BudgetExhausted);
});
test('priced models are unaffected — real cost still projected', () => {
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
expect(() => t.reserve(est('anthropic:claude-haiku-4-5'))).not.toThrow();
// And a cap smaller than the projected cost still throws on cost, not pricing.
const tight = new BudgetTracker({ maxCostUsd: 0.000001, label: 'test' });
let err: unknown;
try { tight.reserve(est('anthropic:claude-opus-4-7')); } catch (e) { err = e; }
expect(err).toBeInstanceOf(BudgetExhausted);
expect((err as BudgetExhausted).reason).toBe('cost');
});
test('embed and rerank paths are untouched by the chat addition', () => {
const t = new BudgetTracker({ maxCostUsd: 0.3, label: 'test' });
expect(() => t.reserve(est('ollama:nomic-embed-text', 'embed'))).not.toThrow();
});
});
+237 -20
View File
@@ -33,6 +33,8 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import {
buildGazetteer,
findMentionedEntities,
tokenizeForScan,
tokenizeTitle,
LINKABLE_ENTITY_TYPES,
type Gazetteer,
type GazetteerEntry,
@@ -56,29 +58,15 @@ beforeEach(async () => {
});
// Tiny gazetteer builder for pure-fn cases that don't need engine.
//
// Deliberately calls the PRODUCTION `tokenizeTitle` rather than re-declaring
// the tokenizer. A duplicated copy makes every test here non-discriminating:
// reverting the source tokenizer would leave the fixture on the new one, so
// title and body would keep agreeing and the tests would pass either way.
function gazetteerFromEntries(entries: Omit<GazetteerEntry, 'tokens'>[]): Gazetteer {
const TOKEN_RE = /[a-zA-Z0-9]+/g;
const isCJK = (s: string): boolean => {
const cp = s.codePointAt(0) ?? 0;
return (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af);
};
const hasCJKTitle = (s: string): boolean => [...s].some(isCJK);
const tokenize = (s: string): string[] => {
TOKEN_RE.lastIndex = 0;
if (!hasCJKTitle(s)) {
const out: string[] = [];
let m: RegExpExecArray | null;
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
return out;
}
// CJK: split into individual characters, lowercased.
return [...s].map(c => isCJK(c) ? c.toLowerCase() : '').filter(Boolean);
};
const g: Gazetteer = new Map();
for (const raw of entries) {
const tokens = tokenize(raw.title);
const tokens = tokenizeTitle(raw.title);
if (tokens.length === 0) continue;
const key = tokens[0]!;
const entry: GazetteerEntry = { ...raw, tokens };
@@ -392,6 +380,221 @@ describe('findMentionedEntities — CJK cases', () => {
});
});
// ============================================================
// Vietnamese (diacritic Latin) — entity extraction tests
// ============================================================
// Fictional Vietnamese names only (privacy rule: no real people in fixtures).
// "Đà Nẵng" is a public city, not a person, and is the canonical đ-diacritic case.
//
// Every case below asserts TOKENIZATION, not just the resolved mention. A
// mention-only assertion does not discriminate: the previous ASCII tokenizer
// fragmented the gazetteer title and the body symmetrically, so a 5-fragment
// entry still matched a 5-fragment body run, and `Mention.name` is copied from
// the untouched `title` column rather than derived from tokens.
describe('findMentionedEntities — Vietnamese cases', () => {
test('VN multi-syllable name matches as a WHOLE (regression: no diacritic fragmentation)', () => {
// Discriminating assertion: the ASCII tokenizer produced
// ['nguy','n','v','n','c'] for this title.
expect(tokenizeTitle('Nguyễn Văn Đức')).toEqual(['nguyễn', 'văn', 'đức']);
const body = 'Hôm nay mình học bài của thầy Nguyễn Văn Đức.';
expect(tokenizeForScan(body).map(t => t.text)).toContain('nguyễn');
const g = gazetteerFromEntries([
{ slug: 'people/nguyen-van-duc', source_id: 'default', title: 'Nguyễn Văn Đức' },
]);
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
expect(mentions[0]!.slug).toBe('people/nguyen-van-duc');
expect(mentions[0]!.name).toBe('Nguyễn Văn Đức');
});
test('VN place name with đ/diacritics — "Đà Nẵng" matched', () => {
// Discriminating assertion: the ASCII tokenizer produced ['n','ng'],
// which is what made this entity match 820 pages instead of 440.
expect(tokenizeTitle('Đà Nẵng')).toEqual(['đà', 'nẵng']);
const body = 'Gia đình mình chuyển tới Đà Nẵng năm ngoái.';
expect(tokenizeForScan(body).map(t => t.text)).toContain('nẵng');
const g = gazetteerFromEntries([
{ slug: 'places/da-nang', source_id: 'default', title: 'Đà Nẵng' },
]);
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
expect(mentions[0]!.slug).toBe('places/da-nang');
expect(mentions[0]!.name).toBe('Đà Nẵng');
});
test('VN NFD body matches an NFC gazetteer title (and the reverse)', () => {
const nfc = 'Nguyễn Văn';
const nfd = nfc.normalize('NFD');
expect(nfd).not.toBe(nfc); // fixture really is decomposed
expect(tokenizeTitle(nfd)).toEqual(tokenizeTitle(nfc));
expect(tokenizeTitle(nfd)).toEqual(['nguyễn', 'văn']);
const opts = { fromSlug: 'writing/post-1', fromSourceId: 'default' };
const gNfc = gazetteerFromEntries([{ slug: 'people/nvd', source_id: 'default', title: nfc }]);
expect(findMentionedEntities(`Thầy ${nfd} nói.`, gNfc, opts)).toHaveLength(1);
const gNfd = gazetteerFromEntries([{ slug: 'people/nvd', source_id: 'default', title: nfd }]);
expect(findMentionedEntities(`Thầy ${nfc} nói.`, gNfd, opts)).toHaveLength(1);
});
test('VN diacritics are significant — "Hồng" title does NOT match diacritic-free "Hong"', () => {
// The title must survive tokenization intact for this to mean anything:
// under the ASCII tokenizer it became ['l','th','h','ng'] and missed for
// the wrong reason.
expect(tokenizeTitle('Lê Thị Hồng')).toEqual(['lê', 'thị', 'hồng']);
const g = gazetteerFromEntries([
{ slug: 'people/le-thi-hong', source_id: 'default', title: 'Lê Thị Hồng' },
]);
// Body uses the ASCII-typed variant "Le Thi Hong" — tokens differ, no false match.
const mentions = findMentionedEntities('Gặp Le Thi Hong hôm qua.', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toEqual([]);
});
test('VN mixed with ASCII — Vietnamese name + ASCII company in one body', () => {
const body = 'Phạm Quốc Bảo hợp tác với Acme.';
// Discriminating: the ASCII tokenizer emitted ['ph','m','qu','c','b','o',
// 'h','p','t','c','v','i','acme'] here — only the ASCII control survived.
expect(tokenizeForScan(body).map(t => t.text))
.toEqual(['phạm', 'quốc', 'bảo', 'hợp', 'tác', 'với', 'acme']);
const g = gazetteerFromEntries([
{ slug: 'people/pham-quoc-bao', source_id: 'default', title: 'Phạm Quốc Bảo' },
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
]);
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(2);
const slugs = mentions.map(m => m.slug);
expect(slugs).toContain('people/pham-quoc-bao');
expect(slugs).toContain('companies/acme');
});
test('VN longest-match wins — "Nguyễn Văn Đức" beats a shorter "Nguyễn Văn" entry', () => {
// Both entries must share a real first token for maximal-munch to be
// exercised at all; under the ASCII tokenizer both keyed on 'nguy'.
expect(tokenizeTitle('Nguyễn Văn')).toEqual(['nguyễn', 'văn']);
expect(tokenizeTitle('Nguyễn Văn Đức')[0]).toBe('nguyễn');
const g = gazetteerFromEntries([
{ slug: 'people/nguyen-van-duc', source_id: 'default', title: 'Nguyễn Văn Đức' },
{ slug: 'people/nguyen-van', source_id: 'default', title: 'Nguyễn Văn' },
]);
const mentions = findMentionedEntities('Bài giảng của Nguyễn Văn Đức rất hay.', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
expect(mentions[0]!.slug).toBe('people/nguyen-van-duc');
});
test('VN first-mention-only cap — repeated name → single link', () => {
const g = gazetteerFromEntries([
{ slug: 'people/nguyen-van-duc', source_id: 'default', title: 'Nguyễn Văn Đức' },
]);
const body = 'Nguyễn Văn Đức nói. Sau đó Nguyễn Văn Đức nói tiếp.';
// The cap must be capping a WHOLE-name match, not a fragment run.
expect(tokenizeForScan(body).map(t => t.text).slice(0, 3)).toEqual(['nguyễn', 'văn', 'đức']);
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
});
test('VN determinism — identical output across 10 calls', () => {
const g = gazetteerFromEntries([
{ slug: 'people/nguyen-van-duc', source_id: 'default', title: 'Nguyễn Văn Đức' },
{ slug: 'places/da-nang', source_id: 'default', title: 'Đà Nẵng' },
]);
const body = 'Thầy Nguyễn Văn Đức ở Đà Nẵng. Nguyễn Văn Đức lần nữa.';
expect(tokenizeForScan(body).map(t => t.text))
.toEqual(['thầy', 'nguyễn', 'văn', 'đức', 'ở', 'đà', 'nẵng', 'nguyễn', 'văn', 'đức', 'lần', 'nữa']);
const refs = new Set<string>();
for (let i = 0; i < 10; i++) {
refs.add(JSON.stringify(findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
})));
}
expect(refs.size).toBe(1);
});
});
// ============================================================
// Tokenizer boundaries — non-word glyphs
// ============================================================
// Guards for the two ways a Unicode tokenizer regresses against the ASCII one
// it replaces. Both were found in review of the first version of this change,
// which used /[[\p{L}\p{M}\p{N}]--[CJK]]+/gv: \p{M} let a token consist of
// combining marks alone, and \p{N} minted tokens the ASCII regex never emitted.
describe('tokenizer boundaries — marks and non-ASCII numerics', () => {
const opts = { fromSlug: 'writing/post-1', fromSourceId: 'default' };
test('a token can never be combining marks alone (U+FE0F does not become a key)', () => {
// VARIATION SELECTOR-16 is \p{Mn} and rides on most emoji. Allowing a
// mark-only token made every emoji-prefixed entity title key on a bare
// U+FE0F, collapsing them into one mutually-confusable bucket.
expect(tokenizeTitle('❤️ Health Notes')).toEqual(['health', 'notes']);
expect(tokenizeTitle('⭐️ Budget Notes')).toEqual(['budget', 'notes']);
expect(tokenizeForScan('❤️').map(t => t.text)).toEqual([]);
const g = gazetteerFromEntries([
{ slug: 'companies/health-notes', source_id: 'default', title: '❤️ Health Notes' },
{ slug: 'companies/budget-notes', source_id: 'default', title: '⭐️ Budget Notes' },
]);
expect([...g.keys()].sort()).toEqual(['budget', 'health']);
// The plain-text link survives...
expect(findMentionedEntities('Plain health notes, no emoji.', g, opts).map(m => m.slug))
.toEqual(['companies/health-notes']);
// ...and an unrelated emoji in the body does not drag in the other entity.
expect(findMentionedEntities('Sprint ⚠️ health notes were fine.', g, opts).map(m => m.slug))
.toEqual(['companies/health-notes']);
});
test('non-ASCII numerics do not break strict token adjacency of an ASCII name', () => {
// findMentionedEntities requires an entry's tokens to be STRICTLY
// ADJACENT in the body, so any glyph that newly tokenizes between the
// words of "Acme Corp" silently kills a match that used to work.
const g = gazetteerFromEntries([
{ slug: 'companies/acme-corp', source_id: 'default', title: 'Acme Corp' },
]);
for (const body of [
'We met Acme Corp today.', // control
'We met Acme¹ Corp today.', // U+00B9 superscript one (No)
'Acme ½ Corp', // U+00BD vulgar fraction (No)
'Acme Corp', // U+FF11 fullwidth digit one (Nd)
'Acme ❤️ Corp', // emoji + VS16 (So + Mn)
]) {
expect(findMentionedEntities(body, g, opts)).toHaveLength(1);
}
// ASCII digits still tokenize exactly as /[a-zA-Z0-9]+/ did.
expect(tokenizeForScan('web3 and h2o').map(t => t.text)).toEqual(['web3', 'and', 'h2o']);
});
test('Han Extension A is no longer CJK here — aligned with cjk.ts scope', () => {
// Deliberate behaviour change. by-mention's walkers used to carry their
// own range copy covering Ext-A (U+34004DBF); cjk.ts scopes Ext-A out
// repo-wide, and this module now uses CJK_SLUG_CHARS verbatim. Ext-A
// therefore tokenizes as a word run instead of per character, and an
// Ext-A-only title is one sub-MIN_NAME_LENGTH token rather than N
// char-level ones. Search, chunking and slug grammar already ignore
// Ext-A, so this removes by-mention as the lone subsystem that disagreed.
expect(tokenizeForScan('㐀㐁').map(t => t.text)).toEqual(['㐀㐁']);
expect(tokenizeTitle('㐀㐁')).toEqual(['㐀㐁']);
// In-scope CJK is untouched: still char-level.
expect(tokenizeTitle('纳瓦尔')).toEqual(['纳', '瓦', '尔']);
expect(tokenizeForScan('纳瓦尔说').map(t => t.text)).toEqual(['纳', '瓦', '尔', '说']);
});
});
// ============================================================
// buildGazetteer — engine-backed tests
// ============================================================
@@ -521,4 +724,18 @@ describe('buildGazetteer — engine integration', () => {
const g = await buildGazetteer(engine);
expect(g.size).toBe(0);
});
test('VN person title enters gazetteer keyed on first diacritic-preserving token', async () => {
await engine.putPage('people/nguyen-van-duc', {
type: 'person', title: 'Nguyễn Văn Đức', compiled_truth: 'b', timeline: '', frontmatter: {},
});
const g = await buildGazetteer(engine);
// "Nguyễn Văn Đức" → ["nguyễn","văn","đức"], keyed on "nguyễn" (NOT fragmented to "nguy").
expect(g.has('nguyễn')).toBe(true);
const bucket = g.get('nguyễn')!;
expect(bucket[0]!.tokens).toEqual(['nguyễn', 'văn', 'đức']);
expect(bucket[0]!.slug).toBe('people/nguyen-van-duc');
// Regression guard: the old ASCII tokenizer would have keyed on "nguy".
expect(g.has('nguy')).toBe(false);
});
});
+6 -1
View File
@@ -195,7 +195,12 @@ describe('CLI dispatch integration', () => {
});
const stdout = await new Response(proc.stdout).text();
const exitCode = await proc.exited;
expect(stdout).toContain('Usage: gbrain init');
// init prints its OWN detailed help (printInitHelp), not the generic
// CLI-only one-line stub. Assert on markers unique to the real help...
expect(stdout).toContain('gbrain init [flags]');
expect(stdout).toContain('ENGINE SELECTION');
// ...and confirm the generic stub (printCliOnlyHelp) did NOT fire.
expect(stdout).not.toContain('run gbrain --help for the full command list');
expect(existsSync(join(home, '.gbrain', 'config.json'))).toBe(false);
expect(exitCode).toBe(0);
} finally {
+52
View File
@@ -0,0 +1,52 @@
import { describe, test, expect } from 'bun:test';
import { __testing } from '../src/core/cycle/auto-think.ts';
import type { BrainEngine } from '../src/core/engine.ts';
// A configured 0 must survive config resolution. The pre-fix
// `parse*(str) || <default>` coerced an explicit "0" back to the default
// (budget 0 = "spend nothing", cooldown 0 = "no cooldown"); loadConfig now
// routes budget + cooldown_days through getNumberConfig, which honors 0.
function stubEngine(config: Record<string, string>): BrainEngine {
return { getConfig: async (key: string) => config[key] ?? null } as unknown as BrainEngine;
}
describe('auto_think loadConfig honors a configured 0', () => {
test('budget = "0" resolves 0, not $2', async () => {
const cfg = await __testing.loadConfig(stubEngine({ 'dream.auto_think.budget': '0' }));
expect(cfg.budgetUsd).toBe(0);
});
test('cooldown_days = "0" resolves 0, not the 30d default', async () => {
const cfg = await __testing.loadConfig(stubEngine({ 'dream.auto_think.cooldown_days': '0' }));
expect(cfg.cooldownDays).toBe(0);
});
test('absent keys keep the defaults', async () => {
const cfg = await __testing.loadConfig(stubEngine({}));
expect(cfg.budgetUsd).toBe(2.0);
expect(cfg.cooldownDays).toBe(30);
});
test('unparseable values fall back to the defaults', async () => {
const cfg = await __testing.loadConfig(stubEngine({
'dream.auto_think.budget': 'abc',
'dream.auto_think.cooldown_days': 'xyz',
}));
expect(cfg.budgetUsd).toBe(2.0);
expect(cfg.cooldownDays).toBe(30);
});
test('positive values round-trip (budget accepts fractions)', async () => {
const cfg = await __testing.loadConfig(stubEngine({
'dream.auto_think.budget': '0.5',
'dream.auto_think.cooldown_days': '7',
}));
expect(cfg.budgetUsd).toBe(0.5);
expect(cfg.cooldownDays).toBe(7);
});
test('a negative value clamps to 0', async () => {
const cfg = await __testing.loadConfig(stubEngine({ 'dream.auto_think.budget': '-1' }));
expect(cfg.budgetUsd).toBe(0);
});
});
+3 -1
View File
@@ -9,7 +9,9 @@
* - status is 'ok' | 'clean' | 'partial' (failed/skipped don't mark fresh)
* - dryRun is false
*
* Best-effort: a write failure does NOT change the CycleReport status.
* Best-effort in that it never throws out of runCycle. As of #3504 a write
* failure IS surfaced: it sets `stamp_write_failed` on the report and degrades
* a successful status to 'partial'. See test/cycle-stamp-write-failure.test.ts.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -0,0 +1,80 @@
/**
* #1470 runCycle swallowed lock.release() errors with empty catches. When
* the release SQL throws (e.g. CONNECTION_ENDED after the pool was ended out
* from under the cycle), the row in gbrain_cycle_locks persists with zero
* operator-visible signal; the next `gbrain dream` within the TTL then skips
* with a phantom `cycle_already_running`.
*
* Behavioral pin: a cycle whose DB-lock release throws still completes
* (release stays best-effort) but emits a one-line stderr diagnostic naming
* the failure, instead of silence.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runCycle } from '../src/core/cycle.ts';
let engine: PGLiteEngine;
let tmpHome: string;
let prevHome: string | undefined;
beforeAll(async () => {
// Point the file-lock path (gbrainPath('cycle.lock')) at a throwaway dir so
// the test never touches the operator's real ~/.gbrain.
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-cycle-diag-'));
prevHome = process.env.GBRAIN_HOME;
process.env.GBRAIN_HOME = tmpHome;
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
if (engine) await engine.disconnect();
if (prevHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = prevHome;
if (tmpHome) rmSync(tmpHome, { recursive: true, force: true });
});
describe('runCycle lock-release failure diagnostic (#1470)', () => {
test('a throwing DB-lock release is logged, not silently swallowed', async () => {
// Simulate the pool being ended under the cycle: the release DELETE on
// gbrain_cycle_locks throws; everything else passes through untouched.
const db = (engine as unknown as { db: { query: (...a: unknown[]) => Promise<unknown> } }).db;
const realQuery = db.query.bind(db);
db.query = (...args: unknown[]) => {
if (typeof args[0] === 'string' && (args[0] as string).includes('DELETE FROM gbrain_cycle_locks')) {
return Promise.reject(new Error('CONNECTION_ENDED (simulated)'));
}
return realQuery(...args);
};
const errLines: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => { errLines.push(args.map(String).join(' ')); };
let report: Awaited<ReturnType<typeof runCycle>>;
try {
// 'lint' needs the cycle lock; brainDir null skips the phase body fast.
report = await runCycle(engine, { phases: ['lint'], brainDir: null });
} finally {
console.error = origError;
db.query = realQuery;
}
// Release stays best-effort — the cycle itself still completes...
expect(report.status).not.toBe('failed');
// ...but the swallowed release error now leaves a diagnostic naming the
// failure and the stranded-lock consequence.
const diagnostic = errLines.find(l => l.includes('release') && l.includes('CONNECTION_ENDED (simulated)'));
expect(diagnostic).toBeDefined();
// The stranded row really is there (release never ran) — the situation
// the diagnostic points the operator at.
const rows = await engine.executeRaw<{ id: string }>(`SELECT id FROM gbrain_cycle_locks`);
expect(rows.length).toBeGreaterThan(0);
}, 60_000);
});
+159
View File
@@ -0,0 +1,159 @@
/**
* #3504 a cycle that cannot persist its freshness stamp must stop reporting
* success.
*
* Before this, `updateSourceConfig` throwing was a `console.warn` and nothing
* else. `gbrain dream --json` reported `status: 'ok'`, doctor separately
* reported `cycle_freshness` stale, and no signal connected them so the fix
* doctor recommends (re-run the cycle) could never work, because the cycle was
* already succeeding. That is the loop #2251 sat in while every stamp write
* failed on a corrupted `sources.config`.
*
* Contract pinned here:
* - a stamp-write error sets `stamp_write_failed: {source_id, error}`
* - it degrades 'ok' / 'clean' to 'partial' and sets reason 'stamp_write_failed'
* - it NEVER throws out of runCycle (the phases already did their work)
* - a pack that merely omits optional phases is NOT affected: those phases come
* back 'skipped', `deriveStatus` ignores them by design, and the status stays
* a success status. This is the conflation the maintainer flagged on #3504.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv } from './helpers/with-env.ts';
import { runCycle } from '../src/core/cycle.ts';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
let engine: PGLiteEngine;
let brainDir: string;
// Per-test GBRAIN_HOME isolation: the PGLite cycle path takes a file lock at
// `~/.gbrain/cycle.lock`, unscoped by source. Without isolation, a sibling
// worktree running its own tests makes runCycle return 'skipped' and the stamp
// hook silently no-ops. Same rationale as cycle-last-full-cycle-at.test.ts.
let gbrainHome: string;
const SOURCE = 'stamp-fail-src';
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-stamp-brain-'));
gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-stamp-home-'));
await engine.executeRaw(
`INSERT INTO sources (id, name, config) VALUES ($1, $2, $3::text::jsonb)
ON CONFLICT (id) DO NOTHING`,
[SOURCE, 'Stamp Fail Source', '{}'],
);
});
/** Run a per-source cycle with updateSourceConfig forced to throw. */
async function runWithFailingStamp(message: string) {
const original = engine.updateSourceConfig.bind(engine);
let calls = 0;
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = async () => {
calls += 1;
throw new Error(message);
};
try {
const report = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
runCycle(engine, { brainDir, sourceId: SOURCE, phases: ['lint'] }),
);
return { report, calls };
} finally {
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = original;
}
}
describe('#3504 stamp-write failure is surfaced on the report', () => {
test('sets stamp_write_failed with the source id and the error message', async () => {
const { report, calls } = await runWithFailingStamp('jsonb_each on a non-object');
expect(calls).toBeGreaterThan(0);
expect(report.stamp_write_failed).toBeDefined();
expect(report.stamp_write_failed!.source_id).toBe(SOURCE);
expect(report.stamp_write_failed!.error).toContain('jsonb_each on a non-object');
});
test('degrades a successful status to partial with reason stamp_write_failed', async () => {
const { report } = await runWithFailingStamp('write blew up');
expect(report.status).toBe('partial');
expect(report.reason).toBe('stamp_write_failed');
});
test('does NOT throw out of runCycle — the phases already ran', async () => {
const { report } = await runWithFailingStamp('write blew up');
// The run still produced a report with its phase results intact.
expect(report.schema_version).toBe('1');
expect(report.phases.length).toBeGreaterThan(0);
});
});
describe('#3504 no false positives', () => {
test('a healthy per-source cycle has no stamp_write_failed and keeps a success status', async () => {
const report = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
runCycle(engine, { brainDir, sourceId: SOURCE, phases: ['lint'] }),
);
expect(report.stamp_write_failed).toBeUndefined();
expect(report.reason).toBeUndefined();
expect(['ok', 'clean']).toContain(report.status);
});
test('a pack that omits optional phases is not conflated with a stamp failure', async () => {
// The distinction the maintainer called out on #3504: `deriveStatus`
// deliberately ignores 'skipped' phases, so omitting optional phases is not
// a failure. Only a real write error may degrade the status.
const report = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
runCycle(engine, { brainDir, sourceId: SOURCE, phases: ['lint'] }),
);
const skipped = report.phases.filter((p) => p.status === 'skipped');
// Whether or not any phase skipped in this environment, the invariant holds:
// a success status must not carry a stamp-failure marker.
expect(report.stamp_write_failed).toBeUndefined();
if (skipped.length > 0) {
expect(['ok', 'clean']).toContain(report.status);
}
});
test('dryRun does not attempt the write and cannot report a stamp failure', async () => {
const original = engine.updateSourceConfig.bind(engine);
let called = false;
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = async () => {
called = true;
throw new Error('should never run under dryRun');
};
try {
const report = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
runCycle(engine, { brainDir, sourceId: SOURCE, phases: ['lint'], dryRun: true }),
);
expect(called).toBe(false);
expect(report.stamp_write_failed).toBeUndefined();
} finally {
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = original;
}
});
test('a legacy caller with no sourceId cannot report a stamp failure', async () => {
const original = engine.updateSourceConfig.bind(engine);
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = async () => {
throw new Error('should never run without sourceId');
};
try {
const report = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
runCycle(engine, { brainDir, phases: ['lint'] }),
);
expect(report.stamp_write_failed).toBeUndefined();
} finally {
(engine as unknown as { updateSourceConfig: unknown }).updateSourceConfig = original;
}
});
});
+53
View File
@@ -0,0 +1,53 @@
import { describe, test, expect } from 'bun:test';
import { __testing } from '../src/core/cycle/synthesize.ts';
import type { BrainEngine } from '../src/core/engine.ts';
// A configured 0 must survive config resolution. The pre-fix `parseInt(str, 10)
// || <default>` coerced an explicit "0" back to the default (cooldown 0 = "no
// cooldown"); loadSynthConfig now routes cooldown_hours + min_chars through
// getNumberConfig, which honors 0. Only engine.getConfig is exercised, so a
// stub engine is sufficient (no PGLite).
function stubEngine(config: Record<string, string>): BrainEngine {
return { getConfig: async (key: string) => config[key] ?? null } as unknown as BrainEngine;
}
describe('loadSynthConfig honors a configured 0', () => {
test('cooldown_hours = "0" resolves 0, not the 12h default', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({ 'dream.synthesize.cooldown_hours': '0' }));
expect(cfg.cooldownHours).toBe(0);
});
test('min_chars = "0" resolves 0, not 2000', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({ 'dream.synthesize.min_chars': '0' }));
expect(cfg.minChars).toBe(0);
});
test('absent keys keep the defaults', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({}));
expect(cfg.cooldownHours).toBe(12);
expect(cfg.minChars).toBe(2000);
});
test('unparseable values fall back to the defaults', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({
'dream.synthesize.cooldown_hours': 'abc',
'dream.synthesize.min_chars': 'xyz',
}));
expect(cfg.cooldownHours).toBe(12);
expect(cfg.minChars).toBe(2000);
});
test('positive values round-trip', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({
'dream.synthesize.cooldown_hours': '6',
'dream.synthesize.min_chars': '500',
}));
expect(cfg.cooldownHours).toBe(6);
expect(cfg.minChars).toBe(500);
});
test('a negative value clamps to 0', async () => {
const cfg = await __testing.loadSynthConfig(stubEngine({ 'dream.synthesize.cooldown_hours': '-5' }));
expect(cfg.cooldownHours).toBe(0);
});
});
+138
View File
@@ -0,0 +1,138 @@
/**
* #3502: docs must not reference nonexistent gbrain commands.
*
* `docs/tutorials/personal-brain.md` shipped a `gbrain install` step for two
* months after the command it replaced was retired every reader hit
* "Unknown command: install". This guard scans README.md, docs/, and skills/
* for `gbrain <verb>` invocations in code (fenced blocks + inline code spans)
* and checks each verb against the live CLI surface: CLI_ONLY, operation
* cliHints names (non-hidden), and aliases.
*
* Deliberately excluded (historical or speculative by design, per CLAUDE.md's
* "historical docs are never rewritten" rule):
* - docs/GBRAIN_V0.md the v0 spec; documents v0's CLI
* - docs/designs/, docs/plans/ future/speculative design docs
* - docs/migrations/, skills/migrations/ per-release migration notes,
* written against that release's CLI
* - docs/UPGRADING_DOWNSTREAM_AGENTS.md per-release upgrade chronicle
*
* Heuristics keep prose out: only fenced code + inline spans are scanned,
* comment lines and diagram lines are skipped, and the verb must sit in
* command position (start of command text, or after a shell operator).
*/
import { describe, expect, test } from 'bun:test';
import { readdirSync, readFileSync, statSync } from 'fs';
import { dirname, join, relative } from 'path';
import { CLI_ONLY, cliAliases } from '../src/cli.ts';
import { operations } from '../src/core/operations.ts';
const ROOT = dirname(import.meta.dir);
const EXCLUDED = [
'docs/GBRAIN_V0.md',
'docs/UPGRADING_DOWNSTREAM_AGENTS.md',
'docs/designs/',
'docs/plans/',
'docs/migrations/',
'skills/migrations/',
];
/** Known-intentional references to commands that deliberately don't exist. */
const ALLOWLIST: Record<string, string[]> = {
// The doc explains that gbrain does NOT ship this command, on purpose.
'docs/guides/rls-and-you.md': ['rls-exempt'],
};
function validCommands(): Set<string> {
const valid = new Set<string>(CLI_ONLY);
for (const op of operations) {
const name = op.cliHints?.name;
if (name && !op.cliHints?.hidden) valid.add(name);
}
for (const alias of cliAliases.keys()) valid.add(alias);
return valid;
}
function* mdFiles(dir: string): Generator<string> {
for (const entry of readdirSync(dir)) {
const p = join(dir, entry);
if (statSync(p).isDirectory()) yield* mdFiles(p);
else if (p.endsWith('.md')) yield p;
}
}
interface CodeLine { code: string; line: number }
/** Fenced-block lines + inline code spans that START with `gbrain `. */
function codeRegions(text: string): CodeLine[] {
const out: CodeLine[] = [];
const lines = text.split('\n');
let inFence = false;
for (let i = 0; i < lines.length; i++) {
const l = lines[i];
if (/^\s*(```|~~~)/.test(l)) { inFence = !inFence; continue; }
if (inFence) {
const t = l.trim();
if (/^(#|\/\/|--|\*)/.test(t)) continue; // comment lines
if (/[│┌┐└┘├┤─═╔╗╚╝]/.test(l)) continue; // ASCII-art diagrams
out.push({ code: l, line: i + 1 });
continue;
}
for (const m of l.matchAll(/`(gbrain [^`]+)`/g)) out.push({ code: m[1], line: i + 1 });
}
return out;
}
/** True when `gbrain` sits at command position (not mid-prose). */
function commandPosition(prefix: string): boolean {
const p = prefix.trimEnd();
return p === '' || /[|;&`(={[]$/.test(p) || /\$$/.test(p);
}
function scan(): string[] {
const valid = validCommands();
const violations: string[] = [];
const files = [
join(ROOT, 'README.md'),
...mdFiles(join(ROOT, 'docs')),
...mdFiles(join(ROOT, 'skills')),
];
for (const file of files) {
const rel = relative(ROOT, file);
if (EXCLUDED.some((e) => rel === e || rel.startsWith(e))) continue;
const text = readFileSync(file, 'utf-8');
for (const { code, line } of codeRegions(text)) {
for (const m of code.matchAll(/\bgbrain\s+([A-Za-z][\w-]*)/g)) {
const verb = m[1];
if (!/^[a-z][a-z0-9_-]{2,}$/.test(verb)) continue; // flags, <slots>, v0.x
if (!commandPosition(code.slice(0, m.index))) continue;
if (valid.has(verb)) continue;
if (ALLOWLIST[rel]?.includes(verb)) continue;
violations.push(`${rel}:${line}: \`gbrain ${verb}\` is not a real command — ${code.trim().slice(0, 90)}`);
}
}
}
return violations;
}
describe('#3502 — docs reference only real gbrain commands', () => {
test('every `gbrain <verb>` in README/docs/skills resolves to a live command', () => {
const violations = scan();
expect(violations).toEqual([]);
});
test('the sanity anchors: install is dead, init/put/skillpack are live', () => {
const valid = validCommands();
expect(valid.has('install')).toBe(false); // retired v0.36.0.0 — the #3502 bug
expect(valid.has('init')).toBe(true);
expect(valid.has('put')).toBe(true);
expect(valid.has('skillpack')).toBe(true);
});
test('pages + bench are dispatchable (documented surfaces; #2035 bug class)', () => {
// `pages` had a live handleCliOnly case but was dropped from CLI_ONLY;
// `bench` (bench-publish.ts) was documented but never wired at all.
expect(CLI_ONLY.has('pages')).toBe(true);
expect(CLI_ONLY.has('bench')).toBe(true);
});
});
+105
View File
@@ -0,0 +1,105 @@
/**
* #1835 pure unit coverage for src/commands/doctor-asset-paths.ts.
*
* Everything here is path/string-based with injected platform + WSL mount
* root, so it runs identically on macOS / Linux / CI. The WSL translation
* itself is UNVERIFIED-ON-PLATFORM (no real WSL host in this environment);
* these tests pin the intended mapping.
*/
import { describe, expect, test } from 'bun:test';
import { resolveAssetPath, parseWslAutomountRoot } from '../src/commands/doctor-asset-paths.ts';
const REPO = '/mnt/d/brain-repo';
describe('resolveAssetPath — Windows drive paths', () => {
test('WSL: D:/ forward-slash path maps to <root>/d/…', () => {
const r = resolveAssetPath('D:/cicada3301/lost9999/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: '/mnt/d/cicada3301/lost9999/img.jpg', foreign: false });
});
test('WSL: D:\\ backslash path maps with separators normalized', () => {
const r = resolveAssetPath('D:\\cicada3301\\lost9999\\img.jpg', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: '/mnt/d/cicada3301/lost9999/img.jpg', foreign: false });
});
test('WSL: drive letter is lowercased, custom automount root honored', () => {
const r = resolveAssetPath('C:/Users/a/img.png', REPO, {
platform: 'linux',
wslMountRoot: '/windir/',
});
expect(r.abs).toBe('/windir/c/Users/a/img.png');
});
test('macOS: drive path is foreign (skip, never joined onto repoRoot)', () => {
const r = resolveAssetPath('D:/cicada3301/img.jpg', REPO, {
platform: 'darwin',
wslMountRoot: null,
});
expect(r).toEqual({ abs: null, foreign: true });
});
test('plain Linux (non-WSL): drive path is foreign', () => {
const r = resolveAssetPath('D:/x/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: null,
});
expect(r).toEqual({ abs: null, foreign: true });
});
test('win32: drive path stats natively, untouched', () => {
const r = resolveAssetPath('D:/x/img.jpg', REPO, { platform: 'win32' });
expect(r).toEqual({ abs: 'D:/x/img.jpg', foreign: false });
});
});
describe('resolveAssetPath — non-drive paths keep pre-#1835 behavior', () => {
test('POSIX absolute path passes through', () => {
const r = resolveAssetPath('/var/data/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: null,
});
expect(r).toEqual({ abs: '/var/data/img.jpg', foreign: false });
});
test('relative path joins onto repoRoot', () => {
const r = resolveAssetPath('assets/img.jpg', REPO, {
platform: 'darwin',
wslMountRoot: null,
});
expect(r).toEqual({ abs: `${REPO}/assets/img.jpg`, foreign: false });
});
test('lookalike without separator after colon is NOT treated as a drive', () => {
const r = resolveAssetPath('notes:draft.md', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: `${REPO}/notes:draft.md`, foreign: false });
});
});
describe('parseWslAutomountRoot', () => {
test('defaults to /mnt on empty or unrelated config', () => {
expect(parseWslAutomountRoot('')).toBe('/mnt');
expect(parseWslAutomountRoot('[boot]\nsystemd=true\n')).toBe('/mnt');
});
test('reads [automount] root', () => {
expect(parseWslAutomountRoot('[automount]\nroot = /custom\n')).toBe('/custom');
});
test('ignores root under a different section', () => {
expect(parseWslAutomountRoot('[network]\nroot = /nope\n')).toBe('/mnt');
});
test('handles quotes, comments, and CRLF', () => {
const conf = '[automount]\r\nroot = "/win" # drives here\r\noptions = "metadata"\r\n';
expect(parseWslAutomountRoot(conf)).toBe('/win');
});
});
+104
View File
@@ -0,0 +1,104 @@
/**
* #1835 doctor `image_assets`: Windows drive paths (`D:/…`, `D:\`) written
* by a Windows gbrain install must not be reported as "missing from disk"
* on POSIX hosts that cannot resolve them.
*
* Behavioral test through the master-existing `buildChecks` seam it
* deliberately imports NOTHING introduced by this fix, so running this file
* against an unmodified master demonstrates the bug (image_assets WARNs
* "restore from git" for a drive path that was never lost).
*
* Pure translation-logic coverage (WSL /mnt mapping, wsl.conf parsing) lives
* in test/doctor-asset-paths.test.ts.
*/
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
import { readFileSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { buildChecks, type Check } from '../src/commands/doctor.ts';
let engine: PGLiteEngine;
let repoRoot: string;
// These assertions describe non-WSL POSIX hosts (macOS, plain Linux — every
// dev box + CI runner here). On real WSL the drive path is translated and
// statted instead; on win32 it stats natively. Skip there.
const onWsl = (() => {
try {
return process.platform === 'linux' && /microsoft/i.test(readFileSync('/proc/version', 'utf8'));
} catch {
return false;
}
})();
const skip = onWsl || process.platform === 'win32';
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
repoRoot = mkdtempSync(join(tmpdir(), 'gbrain-1835-'));
await engine.setConfig('sync.repo_path', repoRoot);
});
async function insertImage(storagePath: string, hash: string): Promise<void> {
await engine.executeRaw(
`INSERT INTO files (source_id, filename, storage_path, mime_type, content_hash)
VALUES ('default', 'img.jpg', $1, 'image/jpeg', $2)`,
[storagePath, hash],
);
}
async function imageAssetsCheck(): Promise<Check> {
const checks = await buildChecks(engine, []);
const check = checks.find((c) => c.name === 'image_assets');
expect(check).toBeDefined();
return check!;
}
describe('doctor image_assets — Windows drive paths on POSIX (#1835)', () => {
test.skipIf(skip)('D:/ path is skipped with a note, not reported missing', async () => {
await insertImage('D:/cicada3301/lost9999/img.jpg', 'h1');
const check = await imageAssetsCheck();
// Master joins the drive path onto repoRoot and WARNs "missing from
// disk … restore from git" — a false data-loss report.
expect(check.status).toBe('ok');
expect(check.message).toContain('Windows-drive path(s) skipped');
expect(check.message).not.toContain('restore from git');
});
test.skipIf(skip)('backslash D:\\ path is also skipped', async () => {
await insertImage('D:\\cicada3301\\lost9999\\img.jpg', 'h2');
const check = await imageAssetsCheck();
expect(check.status).toBe('ok');
expect(check.message).toContain('Windows-drive path(s) skipped');
});
test.skipIf(skip)('drive path skip does not mask a genuinely vanished asset', async () => {
await insertImage('D:/cicada3301/lost9999/img.jpg', 'h3');
await insertImage('assets/really-gone.png', 'h4');
const check = await imageAssetsCheck();
expect(check.status).toBe('warn');
expect(check.message).toContain('assets/really-gone.png');
// The unresolvable drive path is excluded from the checked denominator.
expect(check.message).toContain('1 of 1 image(s) missing');
expect(check.message).toContain('Windows-drive path(s) skipped');
});
test('present relative asset still resolves against repoRoot (regression guard)', async () => {
writeFileSync(join(repoRoot, 'here.png'), 'x');
await insertImage('here.png', 'h5');
const check = await imageAssetsCheck();
expect(check.status).toBe('ok');
expect(check.message).toContain('all present on disk');
});
});
+132
View File
@@ -0,0 +1,132 @@
/**
* #3037 `gbrain embed` must exit non-zero when chunks failed to embed.
*
* Pre-fix, src/cli.ts discarded runEmbed's result entirely, so a run where
* EVERY chunk failed to embed still exited 0 cron, CI and health gates read
* total failure as success. The fix mirrors the `import` case's
* `errors > 0 → setCliExitVerdict(1)` guard.
*
* Real spawned CLI against a tmpdir PGLite brain, with the embedding
* provider pointed at a local mock llama-server (OpenAI-compatible, no auth)
* that can be flipped between failing and healthy. Single test, single
* brain: every spawn pays a cold transpile cost (see
* apply-migrations-pglite-spawn.serial.test.ts for the rationale).
*
* Serial: spawns subprocesses + binds a local port + writes tmpdirs.
*/
import { describe, test, expect } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
const REPO = new URL('..', import.meta.url).pathname.replace(/\/$/, '');
const DIMS = 16;
async function runCli(
args: string[],
env: Record<string, string>,
timeoutMs: number,
): Promise<{ exitCode: number; stdout: string; stderr: string }> {
const proc = Bun.spawn(['bun', 'run', `${REPO}/src/cli.ts`, ...args], {
cwd: REPO,
env: { ...process.env, ...env },
stdout: 'pipe',
stderr: 'pipe',
});
const killer = setTimeout(() => {
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
}, timeoutMs);
try {
const [stdout, stderr, exitCode] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
proc.exited,
]);
return { exitCode, stdout, stderr };
} finally {
clearTimeout(killer);
}
}
describe('gbrain embed exit code on failures (#3037)', () => {
test('embed --stale exits non-zero when embedding fails, 0 once it succeeds', async () => {
// Mock OpenAI-compatible embeddings endpoint, flippable between modes.
let mode: 'fail' | 'ok' = 'fail';
const server = Bun.serve({
port: 0,
async fetch(req) {
const url = new URL(req.url);
if (url.pathname.endsWith('/embeddings')) {
if (mode === 'fail') {
return new Response(JSON.stringify({ error: { message: 'mock provider exploded' } }), {
status: 500, headers: { 'Content-Type': 'application/json' },
});
}
const body = await req.json() as { input: string | string[] };
const inputs = Array.isArray(body.input) ? body.input : [body.input];
const vec = Array.from({ length: DIMS }, () => 0.1);
return new Response(JSON.stringify({
data: inputs.map((_, i) => ({ object: 'embedding', index: i, embedding: vec })),
usage: { prompt_tokens: 3, total_tokens: 3 },
}), { status: 200, headers: { 'Content-Type': 'application/json' } });
}
// /v1/models probe shape.
return new Response(JSON.stringify({ data: [{ id: 'test-model' }] }), {
status: 200, headers: { 'Content-Type': 'application/json' },
});
},
});
const home = mkdtempSync(join(tmpdir(), 'gbrain-3037-exit-'));
const notes = mkdtempSync(join(tmpdir(), 'gbrain-3037-notes-'));
try {
mkdirSync(join(home, '.gbrain'), { recursive: true });
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({
engine: 'pglite',
database_path: join(home, '.gbrain', 'brain.pglite'),
embedding_model: 'llama-server:test-model',
embedding_dimensions: DIMS,
}) + '\n',
);
writeFileSync(join(notes, 'note.md'), '# A note\n\nSome content to embed.\n');
const env = {
HOME: home,
GBRAIN_HOME: home,
LLAMA_SERVER_BASE_URL: `http://127.0.0.1:${server.port}/v1`,
};
const init = await runCli(['init', '--migrate-only'], env, 120_000);
expect(init.exitCode).toBe(0);
const imp = await runCli(['import', notes, '--no-embed'], env, 90_000);
expect(imp.exitCode).toBe(0);
// THE #3037 PIN: provider fails every embed call → the run must exit
// non-zero. Pre-fix this exited 0 (result discarded by cli.ts).
const failing = await runCli(['embed', '--stale'], env, 90_000);
if (failing.exitCode === 0) {
console.error('--- failing-embed stdout ---\n' + failing.stdout);
console.error('--- failing-embed stderr ---\n' + failing.stderr);
}
expect(failing.exitCode).not.toBe(0);
expect(failing.stderr).toMatch(/failed to embed/i);
// Same brain, healthy provider: converges and exits 0 (failure exit is
// not sticky; the failed chunks stayed NULL so --stale picks them up).
mode = 'ok';
const healthy = await runCli(['embed', '--stale'], env, 90_000);
if (healthy.exitCode !== 0) {
console.error('--- healthy-embed stdout ---\n' + healthy.stdout);
console.error('--- healthy-embed stderr ---\n' + healthy.stderr);
}
expect(healthy.exitCode).toBe(0);
expect(healthy.stdout + healthy.stderr).toMatch(/Embedded [1-9]\d* chunks/);
} finally {
server.stop(true);
try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ }
try { rmSync(notes, { recursive: true, force: true }); } catch { /* best effort */ }
}
}, 480_000);
});
@@ -0,0 +1,266 @@
/**
* #3037 one oversized/bad chunk must not darken its ENTIRE page, and embed
* failures must be visible on the run result.
*
* Pre-fix, all three embed paths sent a page's chunks in ONE embedBatch call
* inside a try whose catch only logged to stderr: when the batch threw,
* upsertChunks never ran, so EVERY sibling chunk stayed NULL (~8.6x blast
* radius from a single bad chunk), and EmbedResult had no failure field
* `gbrain embed` exited 0 on a total no-op.
*
* Pinned here:
* 1. --stale and --all: a page with ONE bad chunk still embeds its other
* chunks (per-chunk isolation via embedPageTexts), failures are counted
* on result.failures, and the embedding signature is NOT stamped for a
* partially-failed page.
* 2. Cost bounding: a 429 (rate limit) does NOT fan out into N
* single-chunk calls, and neither does an AITransientError (outage)
* isolation only fires for permanent request-shaped failures.
*
* Serial: uses mock.module (leaks across files sharing a bun process).
* The CLI exit-code half of #3037 is pinned by
* test/embed-exit-code-3037.serial.test.ts (real spawned CLI).
*/
import { describe, test, expect, mock, beforeEach, afterEach } from 'bun:test';
import type { BrainEngine } from '../src/core/engine.ts';
import { AITransientError } from '../src/core/ai/errors.ts';
// Track every embedBatch call's shape so tests can assert batch-vs-single
// fan-out behavior.
let embedCalls: string[][] = [];
let embedBatchBehavior: ((texts: string[], opts?: unknown) => Promise<Float32Array[]>) | null = null;
mock.module('../src/core/embedding.ts', () => ({
embedBatch: async (texts: string[], opts?: unknown) => {
embedCalls.push([...texts]);
if (embedBatchBehavior) return embedBatchBehavior(texts, opts);
return texts.map(() => new Float32Array(1536));
},
currentEmbeddingSignature: () => 'test:model:1536',
}));
// Import AFTER mocking.
const { runEmbedCore } = await import('../src/commands/embed.ts');
// Preflight seam (same as test/embed.serial.test.ts): make
// diagnoseEmbedding's fast-path pass without real env vars.
const { __setEmbedTransportForTests } = await import('../src/core/ai/gateway.ts');
__setEmbedTransportForTests(async () => ({ embeddings: [], usage: { tokens: 0 } } as any));
function mockEngine(overrides: Partial<Record<string, any>> = {}): BrainEngine {
const calls: { method: string; args: any[] }[] = [];
const track = (method: string) => (...args: any[]) => {
calls.push({ method, args });
if (overrides[method]) return overrides[method](...args);
return Promise.resolve(null);
};
return new Proxy({} as any, {
get(_, prop: string) {
if (prop === '_calls') return calls;
if (overrides[prop]) return overrides[prop];
return track(prop);
},
});
}
/** Permanent 400-shaped batch failure (e.g. one oversized chunk). */
function permanentBatchError(): Error {
const err = new Error('batch contains an invalid input');
(err as any).cause = { status: 400 };
return err;
}
beforeEach(() => {
embedCalls = [];
embedBatchBehavior = null;
process.env.GBRAIN_EMBED_CONCURRENCY = '1';
});
afterEach(() => {
delete process.env.GBRAIN_EMBED_CONCURRENCY;
});
// Behavior: the whole-page batch 400s; retried per-chunk, only 'BAD' fails.
function oneBadChunkBehavior() {
embedBatchBehavior = async (texts: string[]) => {
if (texts.length > 1) throw permanentBatchError();
if (texts[0] === 'BAD') throw permanentBatchError();
return texts.map(() => new Float32Array(1536));
};
}
const THREE_CHUNKS = [
{ chunk_index: 0, chunk_text: 'good-a', chunk_source: 'compiled_truth' as const, embedded_at: null, token_count: 1 },
{ chunk_index: 1, chunk_text: 'BAD', chunk_source: 'compiled_truth' as const, embedded_at: null, token_count: 1 },
{ chunk_index: 2, chunk_text: 'good-b', chunk_source: 'compiled_truth' as const, embedded_at: null, token_count: 1 },
];
describe('#3037 — one bad chunk no longer darkens its page', () => {
test('--stale: siblings of one bad chunk get embedded; failure counted; signature not stamped', async () => {
oneBadChunkBehavior();
const stale = THREE_CHUNKS.map(c => ({
slug: 'poisoned-page', chunk_index: c.chunk_index, chunk_text: c.chunk_text,
chunk_source: c.chunk_source, model: null, token_count: 1, source_id: 'default', page_id: 1,
}));
const upsertCalls: Array<{ slug: string; chunks: any[] }> = [];
const engine = mockEngine({
countStaleChunks: async () => 3,
listStaleChunks: async () => stale,
getChunks: async () => THREE_CHUNKS,
upsertChunks: async (slug: string, chunks: any[]) => { upsertCalls.push({ slug, chunks }); },
});
const result = await runEmbedCore(engine, { stale: true });
// Pre-fix: the batch threw, upsertChunks never ran, embedded stayed 0.
expect(upsertCalls).toHaveLength(1);
const byIdx = new Map(upsertCalls[0].chunks.map((c: any) => [c.chunk_index, c]));
expect(byIdx.get(0)!.embedding).toBeInstanceOf(Float32Array);
expect(byIdx.get(2)!.embedding).toBeInstanceOf(Float32Array);
expect(byIdx.get(1)!.embedding).toBeUndefined(); // bad chunk stays NULL (re-run picks it up)
expect(result.embedded).toBe(2);
expect(result.failures).toBe(1);
expect(result.failure_samples).toHaveLength(1);
expect(result.failure_samples[0]).toContain('poisoned-page');
// Partially-failed page must NOT be stamped as current provenance.
const stamps = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature');
expect(stamps).toHaveLength(0);
});
test('--all: same isolation on the listPages path', async () => {
oneBadChunkBehavior();
const upsertCalls: Array<{ slug: string; chunks: any[] }> = [];
const engine = mockEngine({
listPages: async () => [{ slug: 'poisoned-page', source_id: 'default' }],
getChunks: async () => THREE_CHUNKS,
upsertChunks: async (slug: string, chunks: any[]) => { upsertCalls.push({ slug, chunks }); },
});
const result = await runEmbedCore(engine, { all: true });
expect(upsertCalls).toHaveLength(1);
const byIdx = new Map(upsertCalls[0].chunks.map((c: any) => [c.chunk_index, c]));
expect(byIdx.get(0)!.embedding).toBeInstanceOf(Float32Array);
expect(byIdx.get(1)!.embedding).toBeUndefined();
expect(result.embedded).toBe(2);
expect(result.failures).toBe(1);
const stamps = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature');
expect(stamps).toHaveLength(0);
});
test('--stale x #3507: fan-out retries the WRAPPED texts and a partially-failed page is not restamped', async () => {
// Composition pin for the #3037 + #3538 merge: the per-chunk isolation
// retry must re-send the contextually WRAPPED text (raw chunk_text here
// would silently strip prefixes on exactly the pages that hit an error),
// and restampIfDemotedToTitleTier must NOT fire when isolation left
// chunks NULL (the page was not fully re-embedded — restamping would
// make contextual_retrieval_mode lie again, the exact #3461 bug).
embedBatchBehavior = async (texts: string[]) => {
if (texts.length > 1) throw permanentBatchError();
if (texts[0].includes('BAD')) throw permanentBatchError();
return texts.map(() => new Float32Array(1536));
};
const stale = THREE_CHUNKS.map(c => ({
slug: 'wrapped-page', chunk_index: c.chunk_index, chunk_text: c.chunk_text,
chunk_source: c.chunk_source, model: null, token_count: 1, source_id: 'default', page_id: 1,
}));
const engine = mockEngine({
countStaleChunks: async () => 3,
listStaleChunks: async () => stale,
getPage: async () => ({
slug: 'wrapped-page', title: 'My Title', compiled_truth: 'x', timeline: '',
source_id: 'default', contextual_retrieval_mode: 'per_chunk_synopsis',
}),
getChunks: async () => THREE_CHUNKS,
upsertChunks: async () => {},
});
const result = await runEmbedCore(engine, { stale: true });
// Every embed call — the failed batch AND each single-chunk retry —
// carries the stored-mode contextual prefix (fenced_code exemption is
// pinned upstream in test/embedding-context.test.ts).
expect(embedCalls.length).toBeGreaterThan(1);
for (const call of embedCalls) {
for (const text of call) expect(text).toStartWith('<context>My Title\n</context>\n');
}
expect(result.embedded).toBe(2);
expect(result.failures).toBe(1);
// Partially-failed page: neither signature-stamped nor CR-restamped.
const calls = (engine as any)._calls as Array<{ method: string }>;
expect(calls.filter(c => c.method === 'setPageEmbeddingSignature')).toHaveLength(0);
expect(calls.filter(c => c.method === 'updatePageContextualRetrievalState')).toHaveLength(0);
});
test('--stale: a fully-failed page is counted on result.failures (no more silent no-op)', async () => {
embedBatchBehavior = async () => { throw permanentBatchError(); };
const stale = THREE_CHUNKS.map(c => ({
slug: 'dark-page', chunk_index: c.chunk_index, chunk_text: c.chunk_text,
chunk_source: c.chunk_source, model: null, token_count: 1, source_id: 'default', page_id: 1,
}));
const engine = mockEngine({
countStaleChunks: async () => 3,
listStaleChunks: async () => stale,
getChunks: async () => THREE_CHUNKS,
upsertChunks: async () => {},
});
const result = await runEmbedCore(engine, { stale: true });
expect(result.embedded).toBe(0);
expect(result.failures).toBe(3);
expect(result.failure_samples[0]).toContain('dark-page');
});
});
describe('#3037 — cost bounding: no per-chunk fan-out on transient failures', () => {
test('sustained 429 does not fan out into single-chunk calls', async () => {
embedBatchBehavior = async () => {
const err = new Error('Rate limit reached. Please try again in 0ms.');
(err as any).cause = { status: 429 };
throw err;
};
const stale = THREE_CHUNKS.map(c => ({
slug: 'rate-limited-page', chunk_index: c.chunk_index, chunk_text: c.chunk_text,
chunk_source: c.chunk_source, model: null, token_count: 1, source_id: 'default', page_id: 1,
}));
const engine = mockEngine({
countStaleChunks: async () => 3,
listStaleChunks: async () => stale,
getChunks: async () => THREE_CHUNKS,
upsertChunks: async () => {},
});
const result = await runEmbedCore(engine, { stale: true });
// Every call must be the full 3-text batch: embedBatchWithBackoff's own
// retries (initial + MAX_RATE_LIMIT_RETRIES), never a 1-text isolation call
// hammering the limiter.
expect(embedCalls.length).toBeGreaterThan(1);
for (const call of embedCalls) expect(call).toHaveLength(3);
expect(result.embedded).toBe(0);
expect(result.failures).toBe(3);
}, 30_000);
test('AITransientError (outage/network) does not fan out', async () => {
embedBatchBehavior = async () => { throw new AITransientError('upstream 502', { status: 502 }); };
const stale = THREE_CHUNKS.map(c => ({
slug: 'outage-page', chunk_index: c.chunk_index, chunk_text: c.chunk_text,
chunk_source: c.chunk_source, model: null, token_count: 1, source_id: 'default', page_id: 1,
}));
const engine = mockEngine({
countStaleChunks: async () => 3,
listStaleChunks: async () => stale,
getChunks: async () => THREE_CHUNKS,
upsertChunks: async () => {},
});
const result = await runEmbedCore(engine, { stale: true });
// Non-429 → no backoff retries; transient → no isolation. Exactly 1 call.
expect(embedCalls).toHaveLength(1);
expect(embedCalls[0]).toHaveLength(3);
expect(result.embedded).toBe(0);
expect(result.failures).toBe(3);
});
});
+53
View File
@@ -0,0 +1,53 @@
// #3447 — resolveEntitySlug's slugify fallback must not corrupt slug-shaped
// input. slugify()'s `[^a-z0-9]+ → '-'` rule rewrites the path separator
// (`people/alice-example` → `people-alice-example`), minting an entity_slug
// no page can ever have. The corruption is self-perpetuating: the flattened
// slug never matches a page, so every re-extraction re-mints it.
//
// Behavior pinned here: a slug-shaped input (contains '/') that fails every
// resolution arm falls back with its path separator INTACT — creating the
// page it names later makes the fact resolvable. Display-name fallback
// (no '/') keeps the existing slugify behavior.
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import {
resolveEntitySlug,
resolveEntitySlugWithSource,
} from '../src/core/entities/resolve.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
// Deliberately NO page at people/zeta-nonexistent — the fallback arm is
// exactly the "well-formed slug, page not created yet" case from #3447.
});
afterAll(async () => {
await engine.disconnect();
});
describe('resolveEntitySlug fallback preserves slug-shaped input (#3447)', () => {
it('returns a well-formed but unresolvable slug untouched', async () => {
const out = await resolveEntitySlug(engine, 'default', 'people/zeta-nonexistent');
expect(out).toBe('people/zeta-nonexistent');
});
it('normalizes a messy path-shaped input per segment, keeping the separator', async () => {
const out = await resolveEntitySlug(engine, 'default', 'People/Zeta Nonexistent');
expect(out).toBe('people/zeta-nonexistent');
});
it('still slugifies display names (no separator) as before', async () => {
const out = await resolveEntitySlug(engine, 'default', 'Zeta Nonexistent Q. Persson');
expect(out).toBe('zeta-nonexistent-q-persson');
});
it('resolveEntitySlugWithSource fallback_slugify agrees with resolveEntitySlug', async () => {
const out = await resolveEntitySlugWithSource(engine, 'default', 'companies/zeta-widgets-nonexistent');
expect(out).toEqual({ slug: 'companies/zeta-widgets-nonexistent', source: 'fallback_slugify' });
});
});
+35
View File
@@ -0,0 +1,35 @@
import { describe, expect, test } from 'bun:test';
import { isModelPriceable } from '../src/core/budget/budget-tracker.ts';
// Regression: extract_atoms applied its DEFAULT cost cap unconditionally.
// BudgetTracker.reserve() hard-fails with BudgetExhausted(reason:'no_pricing')
// when a model is absent from the pricing maps AND a cap is set, so the first
// work item threw, `budgetExhausted` latched, and every remaining item was
// skipped — while the phase still reported status 'ok' with an empty failures[].
// Anthropic users never saw it; every Groq / local-llama / OpenRouter user did.
describe('isModelPriceable', () => {
test('priced Anthropic chat models are priceable', () => {
expect(isModelPriceable('claude-haiku-4-5-20251001', 'chat')).toBe(true);
});
// NOTE: the examples here must be providers with genuinely unknown pricing.
// `ollama` and `llama-server` are NOT: they price at $0 via
// FREE_LOCAL_CHAT_PROVIDERS (local inference costs electricity, not tokens),
// so a cap against them is enforceable and must not be skipped. `litellm` is
// deliberately excluded from that set — a LiteLLM proxy can front a paid
// provider — and `groq` is the paid-but-unpriced case this regression bit.
test('unknown providers are not priceable, so a default cap must be skipped', () => {
expect(isModelPriceable('litellm:gemma4-12b', 'chat')).toBe(false);
expect(isModelPriceable('groq:llama-3.3-70b', 'chat')).toBe(false);
});
test('free local providers ARE priceable at $0, so their cap stays enforced', () => {
expect(isModelPriceable('ollama:gemma3:27b', 'chat')).toBe(true);
expect(isModelPriceable('llama-server:local-model', 'chat')).toBe(true);
});
test('is a pure predicate — no throw on unusual model ids', () => {
expect(() => isModelPriceable('', 'chat')).not.toThrow();
expect(() => isModelPriceable('provider-with-no-colon', 'chat')).not.toThrow();
});
});
+71
View File
@@ -292,6 +292,77 @@ describe('lookupSourceLocalPath', () => {
});
});
describe('writeFactsToFence — row_num survives a fence-less rewrite', () => {
// Regression: row_num uniqueness is enforced by idx_facts_fence_key on
// (source_id, source_markdown_slug, row_num), but the value was derived
// from the markdown fence alone, falling back to 1 when the file has no
// fence. Any write path that rewrites a page without preserving its facts
// fence (put_page write-through, sync, dream-cycle reverse-render) then
// makes the next absorb re-issue an already-taken row_num, and the whole
// batch dies on a duplicate-key error.
test('does not reuse a row_num after the fence is stripped from the file', async () => {
const slug = 'people/carol';
const filePath = join(brainDir, `${slug}.md`);
const first = await writeFactsToFence(
engine,
{ sourceId: 'default', localPath: brainDir, slug },
[baseInput({ fact: 'First fact' }), baseInput({ fact: 'Second fact' })],
);
expect(first.inserted).toBe(2);
// Simulate a non-fence-aware writer replacing the page body. The DB still
// holds row_num 1 and 2; the file now advertises none.
writeFileSync(
filePath,
'---\ntype: person\ntitle: Carol\nslug: people/carol\n---\n\n# Carol\n\nRegenerated without the fence.\n',
'utf-8',
);
expect(readFileSync(filePath, 'utf-8')).not.toContain('First fact');
// Pre-fix this threw: upsertFactRow restarted at 1, colliding with the
// existing rows on idx_facts_fence_key.
const second = await writeFactsToFence(
engine,
{ sourceId: 'default', localPath: brainDir, slug },
[baseInput({ fact: 'Third fact' })],
);
expect(second.inserted).toBe(1);
expect(second.fenceWriteFailed).toBeUndefined();
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
'SELECT row_num, fact FROM facts WHERE source_markdown_slug = $1 ORDER BY row_num',
[slug],
);
const rowNums = rows.rows.map((r: { row_num: number }) => r.row_num);
// Three distinct row_nums, and the new one clears the previous maximum.
expect(new Set(rowNums).size).toBe(3);
expect(Math.max(...rowNums)).toBeGreaterThan(2);
});
test('still writes when the facts table cannot be consulted', async () => {
// The DB seed is a hint, not a hard dependency: a lookup failure must
// degrade to the previous file-derived behaviour rather than making
// fence writes impossible (pre-v51 brains, transient DB errors).
const brokenEngine = Object.create(engine) as typeof engine;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(brokenEngine as any).executeRaw = async (sqlText: string, params: unknown[]) => {
if (sqlText.includes('MAX(row_num)')) throw new Error('simulated lookup failure');
// eslint-disable-next-line @typescript-eslint/no-explicit-any
return (engine as any).executeRaw(sqlText, params);
};
const result = await writeFactsToFence(
brokenEngine,
{ sourceId: 'default', localPath: brainDir, slug: 'people/dave' },
[baseInput({ fact: 'Written despite the failed hint' })],
);
expect(result.inserted).toBe(1);
expect(result.fenceWriteFailed).toBeUndefined();
});
});
// Cleanup any leftover tempdirs after the whole suite.
afterAll(() => {
// No-op: each test cleaned up via the beforeEach; this is a safety net.
+54 -1
View File
@@ -20,7 +20,7 @@
* `afterAll`) per CLAUDE.md test-isolation rules R3 + R4.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync, realpathSync } from 'fs';
import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync, realpathSync, chmodSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -149,6 +149,59 @@ describe('runImport checkpoint resume — v0.33.2 path-based', () => {
});
}, 30_000);
test('interrupted run preserves its tail below the 100-file boundary', async () => {
// The periodic checkpoint save fires on `completed.size % 100 === 0`. With
// fewer than 100 successful files there is no boundary to hit, so before
// the final save every completed file in a run that ends with errors was
// discarded and re-done on the next invocation. On a corpus whose files
// are individually expensive, `completed` can advance ~1 per several
// minutes, putting the next boundary hours away — the run then never
// converges under repeated kills.
await withEnv({ GBRAIN_HOME: workspace }, async () => {
// Three small good files (well under the 100-boundary) plus one that
// exceeds the content-sanity block threshold. That throws, so `errors`
// is non-zero and the checkpoint is PRESERVED rather than cleared —
// note a SLUG_MISMATCH would NOT work here: it is a soft `failures`
// entry that leaves `errors` at 0, so upstream clears the checkpoint.
writeBrainFile('people/alice.md', validMarkdown('people/alice'));
writeBrainFile('people/carol.md', validMarkdown('people/carol'));
writeBrainFile('people/dave.md', validMarkdown('people/dave'));
// A file the reader cannot open raises inside importFile, which is the
// path that increments `errors` (a SLUG_MISMATCH would NOT work: it is
// a soft `failures` entry leaving `errors` at 0, so upstream clears the
// checkpoint rather than preserving it).
writeBrainFile('people/unreadable.md', validMarkdown('people/unreadable'));
chmodSync(join(brainDir, 'people/unreadable.md'), 0o000);
const result = await runImport(engine, [brainDir, '--no-embed']);
expect(result.errors).toBeGreaterThan(0);
// The checkpoint exists AND carries the successful files, even though
// no 100-boundary was ever crossed.
expect(existsSync(cpPath)).toBe(true);
const cp = JSON.parse(readFileSync(cpPath, 'utf8'));
expect(cp.completedPaths).toContain('people/alice.md');
expect(cp.completedPaths).toContain('people/carol.md');
expect(cp.completedPaths).toContain('people/dave.md');
// The failed file must still be absent so the next run retries it.
expect(cp.completedPaths).not.toContain('people/unreadable.md');
});
}, 30_000);
test('clean completion still leaves no checkpoint (final save must not resurrect it)', async () => {
// Guards the ordering of the final save: it runs BEFORE the
// clear/preserve decision and only on the error path, so a fully clean
// run must still end with no checkpoint file.
await withEnv({ GBRAIN_HOME: workspace }, async () => {
writeBrainFile('x.md', validMarkdown('x'));
writeBrainFile('y.md', validMarkdown('y'));
const result = await runImport(engine, [brainDir, '--no-embed']);
expect(result.errors).toBe(0);
expect(existsSync(cpPath)).toBe(false);
});
}, 30_000);
test('failed file does NOT enter completedPaths — next run retries it', async () => {
await withEnv({ GBRAIN_HOME: workspace }, async () => {
// Two healthy files plus one with a path-vs-frontmatter slug mismatch.
@@ -0,0 +1,67 @@
/**
* #1575 the unify-types WORKER registration defaulted `apply` to true,
* while the handler interface documents "Default false (dry-run)". The
* canonical operator invocation
* gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"X"}'
* therefore applied a one-shot destructive taxonomy migration on first
* invocation, with no dry-run checkpoint.
*
* Behavioral pin: invoking the registered worker handler with job.data that
* omits `apply` runs a DRY-RUN (no page mutation, active pack not flipped).
* Consented apply paths pass `apply: true` explicitly (onboard remediation +
* the printed migration command carry it).
*/
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionWorker } from '../src/core/minions/worker.ts';
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
_resetPackCacheForTests();
});
afterAll(async () => {
await engine.disconnect();
});
describe('unify-types worker default (#1575)', () => {
it('omitting apply in job.data runs a dry-run, not a destructive apply', async () => {
await engine.putPage('tweets/default-check', {
title: 'tweets/default-check',
type: 'tweet-single' as never,
compiled_truth: 'body that is sufficiently long for any backstop guards we have in the codebase',
timeline: '',
frontmatter: {},
source_path: 'tweets/default-check.md',
});
const worker = new MinionWorker(engine, { concurrency: 1 });
await registerBuiltinHandlers(worker, engine);
const handler = (worker as unknown as {
handlers: Map<string, (j: unknown) => Promise<unknown>>;
}).handlers.get('unify-types');
if (!handler) throw new Error('unify-types handler not registered');
const result = (await handler({
id: 1,
data: { target_pack: 'gbrain-base-v2' }, // no `apply` — the #1575 trap
updateProgress: async () => {},
})) as { apply: boolean; active_pack_flipped: boolean };
// Handler interface: "Apply mutations. Default false (dry-run)."
expect(result.apply).toBe(false);
expect(result.active_pack_flipped).toBe(false);
// The page was NOT retyped.
const rows = await engine.executeRaw<{ type: string }>(
`SELECT type FROM pages WHERE slug = 'tweets/default-check'`,
);
expect(rows[0]!.type).toBe('tweet-single');
}, 60_000);
});
+40
View File
@@ -0,0 +1,40 @@
import { describe, test, expect } from 'bun:test';
import { openAiCompatV1Hint } from '../src/commands/models.ts';
/**
* `gbrain models doctor` the openai-compatible-proxy `/v1`-suffix hint.
*
* `openAiCompatV1Hint` is pure: it resolves the model's recipe synchronously
* (no network, no engine) to decide whether the provider is an openai-compatible
* proxy, then inspects the passed base URL. These cases pin the four branches
* without any transport stub.
*/
describe('openAiCompatV1Hint', () => {
test('openai-compat proxy without /v1 suffix returns a /v1 hint', () => {
const hint = openAiCompatV1Hint('litellm:gpt-4o', 'http://localhost:4000');
expect(hint).toBeDefined();
expect(hint).toContain('/v1');
});
test('base URL already ending in /v1 returns undefined', () => {
expect(openAiCompatV1Hint('litellm:gpt-4o', 'http://localhost:4000/v1')).toBeUndefined();
});
test('base URL ending in /v1/ (trailing slash) returns undefined', () => {
expect(openAiCompatV1Hint('litellm:gpt-4o', 'http://localhost:4000/v1/')).toBeUndefined();
});
test('native anthropic provider returns undefined', () => {
expect(openAiCompatV1Hint('anthropic:claude-sonnet-4-6', 'https://api.anthropic.com')).toBeUndefined();
});
test('native openai provider returns undefined', () => {
expect(openAiCompatV1Hint('openai:gpt-4o', 'https://api.openai.com')).toBeUndefined();
});
test('missing base URL returns undefined', () => {
expect(openAiCompatV1Hint('litellm:gpt-4o', undefined)).toBeUndefined();
expect(openAiCompatV1Hint('litellm:gpt-4o', null)).toBeUndefined();
expect(openAiCompatV1Hint('litellm:gpt-4o', '')).toBeUndefined();
});
});
+17 -4
View File
@@ -14,10 +14,9 @@
* containing one passing and one failing test, override the discovery
* roots via env-vars, and run with --shards=2.
*
* NOT covered here: the heartbeat (timing-sensitive, not load-bearing
* for correctness) and timeout / WEDGED markers (require synthesizing a
* hung test which is fragile across machines). Those rely on the live
* smoke tests captured in CHANGELOG measurements.
* NOT covered behaviorally here: the heartbeat and a real hung Bun process
* (both timing-sensitive). The timeout escalation wiring is covered as a
* source contract below and exercised separately by a process-leak smoke.
*/
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
@@ -155,6 +154,20 @@ describe('failing-on-purpose', () => {
});
});
describe('run-unit-parallel.sh timeout escalation contract', () => {
it('gives a timed-out shard 30 seconds after TERM, then forces KILL', () => {
const source = readFileSync(PARALLEL_SH_SRC, 'utf-8');
expect(source).toContain('SHARD_KILL_AFTER="${GBRAIN_TEST_SHARD_KILL_AFTER:-30}"');
expect(source).toContain('--signal=TERM --kill-after="${SHARD_KILL_AFTER}s"');
expect(source).toContain('sleep "$SHARD_KILL_AFTER" && kill -KILL "$pid"');
});
it('marks both ordinary timeout and forced-KILL timeout exits as wedged', () => {
const source = readFileSync(PARALLEL_SH_SRC, 'utf-8');
expect(source).toContain('[ "$rc" = "124" ] || [ "$rc" = "137" ]');
});
});
describe('run-unit-parallel.sh no-timeout-binary fallback (rc from shard wait, not watchdog teardown)', () => {
// Forces the no-gtimeout/no-timeout branch by running the wrapper under a
// curated PATH that has every tool the scripts call EXCEPT timeout
+17
View File
@@ -125,6 +125,23 @@ describe('findGbrainRoot', () => {
it('returns null when no gbrain root above', () => {
expect(findGbrainRoot('/tmp/definitely-not-a-gbrain-repo-XYZ')).toBeNull();
});
it('falls back to the module location when cwd has no markers (#1917)', () => {
// Simulate a bun-global install: cwd is an unrelated directory with no
// gbrain markers anywhere above it. The no-arg call must still resolve
// via bundle.ts's own location (which lives in the real repo).
const elsewhere = mkdtempSync(join(tmpdir(), 'skillpack-elsewhere-'));
created.push(elsewhere);
const prevCwd = process.cwd();
try {
process.chdir(elsewhere);
const root = findGbrainRoot();
expect(root).not.toBeNull();
expect(existsSync(join(root!, 'openclaw.plugin.json'))).toBe(true);
expect(existsSync(join(root!, 'src', 'cli.ts'))).toBe(true);
} finally {
process.chdir(prevCwd);
}
});
});
describe('loadBundleManifest', () => {
+92
View File
@@ -0,0 +1,92 @@
/**
* #2830 the `sync_brain` MCP op must thread ctx.sourceId into performSync,
* mirroring the D7 pattern already applied to revert_version / put_page.
*
* Pre-fix: the handler called performSync with no sourceId, so a call with
* no explicit `repo` argument (the normal MCP usage) resolved the sync
* anchor of the DEFAULT source instead of the caller's own source on a
* multi-source brain that silently syncs (or reports "up to date" against)
* the wrong repo's history.
*
* Behavioral pin: with a source whose local_path is a committed git repo,
* calling sync_brain with ctx.sourceId = that source and NO repo param must
* import that repo's pages into that source. On master the anchor lookup
* runs against `default` (no local_path) and the sync errors out zero
* pages land in the source.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runSources } from '../src/commands/sources.ts';
import { operationsByName, type OperationContext } from '../src/core/operations.ts';
const SOURCE = 'srcb-2830';
let engine: PGLiteEngine;
let repoPath: string;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-syncop-'));
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'topics'), { recursive: true });
writeFileSync(join(repoPath, 'topics/anchor-check.md'), [
'---',
'type: concept',
'title: Anchor Check',
'---',
'',
'Body long enough to import cleanly for the sync-op source test.',
'',
].join('\n'));
execSync('git add -A && git commit -m seed', { cwd: repoPath, stdio: 'pipe' });
await runSources(engine, ['add', SOURCE, '--path', repoPath, '--no-federated']);
}, 60_000);
afterAll(async () => {
if (engine) await engine.disconnect();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
}, 60_000);
describe('sync_brain op threads ctx.sourceId (#2830)', () => {
test('no-repo call syncs the caller source own anchor, not default', async () => {
const op = operationsByName['sync_brain']!;
const ctx = {
engine,
config: {},
logger: { info() {}, warn() {}, error() {} },
dryRun: false,
remote: false,
sourceId: SOURCE,
} as unknown as OperationContext;
// No `repo` param — the anchor must resolve from ctx.sourceId.
let result: unknown;
let error: unknown;
try {
result = await op.handler(ctx, { no_embed: true, no_pull: true });
} catch (e) {
error = e;
}
const rows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
[SOURCE],
);
// Pre-fix: performSync resolved the DEFAULT anchor (no local_path) and
// errored — nothing landed in the source. Post-fix: the repo imports
// into the caller's source.
expect({ imported: rows[0]!.n > 0, error: error ? String(error) : null })
.toEqual({ imported: true, error: null });
expect(result).toBeDefined();
}, 60_000);
});
+212
View File
@@ -0,0 +1,212 @@
/**
* issue #1720 autopilot worker crash-loops on pooler CONNECTION_CLOSED.
*
* PRs #2025 (#1491) and #1824 (#1801) fixed the promoteDelayed/claim loops and
* added the supervised db-liveness probe, but three gaps remained:
*
* 1. The stall-detection interval had NO reconnect a dead pool sprayed
* "Stall detection error: write CONNECTION_CLOSED ..." every tick until
* the ~3-minute db_dead exit (the reporter's 20k-line log spray).
* 2. `failJob` was unwrapped when recording a failure threw on the same
* dead pool, the ORIGINAL job error was never logged anywhere (masked),
* and no reconnect happened.
* 3. The startup connect matcher (db.ts isRetryableDbConnectError) and the
* runtime matcher (retry-matcher.ts isRetryableConnError) had drifted:
* the startup list lacked /connection.*closed/ + CONNECTION_CLOSED.
*
* These tests inject the postgres.js error SHAPES (message + code) a real
* Supavisor pooler close is not reproducible in CI (UNVERIFIED-ON-ENVIRONMENT).
*/
import { describe, expect, test } from 'bun:test';
import { MinionWorker } from '../src/core/minions/worker.ts';
import type { BrainEngine } from '../src/core/engine.ts';
import type { MinionJob } from '../src/core/minions/types.ts';
import { isRetryableConnError } from '../src/core/retry-matcher.ts';
/** The exact shape postgres.js throws when a pooler closes the socket. */
function connClosed(): Error {
return Object.assign(
new Error('write CONNECTION_CLOSED db.pooler.example:5432'),
{ code: 'CONNECTION_CLOSED' },
);
}
function makeEngine(counter: { reconnects: number }): BrainEngine {
return {
kind: 'postgres',
reconnect: async () => { counter.reconnects += 1; },
} as unknown as BrainEngine;
}
/** Swap the worker's private queue for a fake (same pattern as worker-promote-reconnect.test.ts). */
function setQueue(worker: MinionWorker, queue: Record<string, unknown>): void {
(worker as unknown as { queue: Record<string, unknown> }).queue = queue;
}
async function withCapturedConsoleError<T>(fn: () => Promise<T>): Promise<{ result: T; lines: string[] }> {
const lines: string[] = [];
const orig = console.error;
console.error = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); };
try {
const result = await fn();
return { result, lines };
} finally {
console.error = orig;
}
}
describe('stall-detection interval reconnect (#1720 gap 1)', () => {
test('a retryable connection error in the stall sweep rebuilds the pool — once per tick across all three sweeps', async () => {
const counter = { reconnects: 0 };
const worker = new MinionWorker(makeEngine(counter), {
pollInterval: 1,
stalledInterval: 10,
healthCheckInterval: 0,
});
worker.register('noop', async () => ({}));
let tickDone = false;
let claims = 0;
const threw = { stalled: false, timeouts: false, wallClock: false };
setQueue(worker, {
ensureSchema: async () => {},
promoteDelayed: async () => [],
claim: async () => {
claims += 1;
// Stop once the first stall tick fully ran (or bail out after a bound
// so an unfixed build terminates and FAILS the assertion, not hangs).
if (tickDone || claims > 2000) worker.stop();
return null;
},
handleStalled: async () => {
if (!threw.stalled) { threw.stalled = true; throw connClosed(); }
return { requeued: [], dead: [] };
},
handleTimeouts: async () => {
if (!threw.timeouts) { threw.timeouts = true; throw connClosed(); }
return [];
},
handleWallClockTimeouts: async () => {
if (!threw.wallClock) { threw.wallClock = true; tickDone = true; throw connClosed(); }
tickDone = true;
return [];
},
});
const { lines } = await withCapturedConsoleError(() => worker.start());
// The tick actually exercised the failing sweeps...
expect(lines.some(l => l.startsWith('Stall detection error:'))).toBe(true);
// ...and the worker repaired its pool in-process: exactly ONE reconnect
// for the tick even though all three sweeps failed (no pooler hammering).
expect(counter.reconnects).toBe(1);
});
});
describe('failJob failure-recording resilience (#1720 gap 2)', () => {
test('when failJob throws a conn error: original job error is surfaced, pool reconnects, recording is retried once', async () => {
const counter = { reconnects: 0 };
const worker = new MinionWorker(makeEngine(counter), {
pollInterval: 1,
stalledInterval: 60_000,
healthCheckInterval: 0,
lockDuration: 60_000,
concurrency: 1,
});
worker.register('explode', async () => { throw new Error('boom: the real job defect'); });
const job = {
id: 42,
name: 'explode',
queue: 'default',
data: {},
status: 'active',
attempts_made: 2,
attempts_started: 3,
max_attempts: 3, // attempts exhausted → newStatus 'dead', no backoff math
backoff_type: 'exponential',
backoff_delay: 1000,
backoff_jitter: false,
timeout_ms: null,
timeout_at: null,
parent_job_id: null,
} as unknown as MinionJob;
let failJobCalls = 0;
let handedOut = false;
let done = false;
let claims = 0;
setQueue(worker, {
ensureSchema: async () => {},
promoteDelayed: async () => [],
handleStalled: async () => ({ requeued: [], dead: [] }),
handleTimeouts: async () => [],
handleWallClockTimeouts: async () => [],
renewLock: async () => true,
claim: async () => {
claims += 1;
if (!handedOut) { handedOut = true; return job; }
if (done || claims > 2000) worker.stop();
return null;
},
failJob: async (_id: number, _tok: string, errorText: string, newStatus: string) => {
failJobCalls += 1;
if (failJobCalls === 1) throw connClosed(); // same outage that failed the job
done = true;
return { ...job, status: newStatus, error_text: errorText };
},
});
const { lines } = await withCapturedConsoleError(() => worker.start());
// Recording was retried after an in-process reconnect (not abandoned).
expect(failJobCalls).toBe(2);
expect(counter.reconnects).toBe(1);
// The ORIGINAL job error must be logged even though recording it threw —
// pre-fix only "executeJob unhandled error ... CONNECTION_CLOSED" survived
// and the real defect was masked.
expect(
lines.some(l => l.includes('boom: the real job defect') && l.includes('CONNECTION_CLOSED')),
).toBe(true);
});
});
describe('retry-matcher unification (#1720 gap 4)', () => {
test('startup connect matcher recognizes a pooler CONNECTION_CLOSED', async () => {
const { isRetryableDbConnectError } = await import('../src/core/db.ts');
expect(isRetryableDbConnectError(new Error('write CONNECTION_CLOSED db.pooler.example:5432'))).toBe(true);
expect(isRetryableDbConnectError(new Error('server closed the connection unexpectedly'))).toBe(true);
});
test('code-only CONNECTION_CLOSED (message rewritten by a wrapper) is retryable', () => {
expect(isRetryableConnError(Object.assign(new Error('socket hang up'), { code: 'CONNECTION_CLOSED' }))).toBe(true);
});
test('drift guard: startup and runtime matchers agree on the full shape corpus', async () => {
const { isRetryableDbConnectError } = await import('../src/core/db.ts');
const shapes: unknown[] = [
// transient / retryable
connClosed(),
new Error('write CONNECTION_CLOSED db.pooler.example:5432'),
new Error('connection closed by server'),
Object.assign(new Error(''), { code: 'CONNECTION_ENDED' }),
new Error('password authentication failed for user "app"'),
new Error('connection refused'),
new Error('the database system is starting up'),
new Error('Connection terminated unexpectedly'),
new Error('read ECONNRESET'),
new Error('No database connection: connect() has not been called'),
new Error('EMAXCONNSESSION'),
new Error('sorry, too many clients already'),
// permanent / non-retryable
new Error('extension "vector" does not exist'),
new Error('relation "pages" does not exist'),
new Error('syntax error at end of input'),
new Error('duplicate key value violates unique constraint "pages_pkey"'),
];
for (const shape of shapes) {
expect(isRetryableDbConnectError(shape)).toBe(isRetryableConnError(shape));
}
});
});